百度统计数据服务器

本文最后更新于:1 年前

摘要(ABSTRACT)

前些天有个小朋友说百度统计不支持实时PV、UV。我去找了一下官方提供的文档,百度统计确实没有实时的浏览量导出功能,因此我就想做一个百度统计的数据接口,以给大家用。
最后通过C#写了一个http服务器,作为网站和百度统计数据源的中间件,以连通两个数据端。最后的测试结果表明这种运行方式是可行的,就是有个问题,本地的请求调试延迟平均2000ms,原因是服务器查询百度的数据延迟较大,因此在服务器上设置缓存,以达到快速响应。同时为保证数据的及时性,每1分钟服务器执行更新缓存方法.

百度API接口

百度提供接口的官方说明:https://tongji.baidu.com/sc-web/10000153831/home/dataapi
以下的接口请求示范均使用postman

百度的服务接口都有都需要通过access_token进行调用,这是一个用户级参数,通过登陆获取。要获取这个参数首先需要登录百度开发者平台控制台:百度开发者中心控制台,新建一个工程,添加安全域名,然后记录下app_idsecrect_key

下一步:通过百度提供的接口,修改相应参数的值用浏览器打开,调起登录;
http://openapi.baidu.com/oauth/2.0/authorize?response_type=code&client_id={CLIENT_ID}&redirect_uri={REDIRECT_URI}&scope=basic&display=popup
参数配置如下图所示:

再下一步,使用获取到的code换取token,API如下图所示,这里可用浏览器打开也可用请求工具进行请求。
http://openapi.baidu.com/oauth/2.0/token?grant_type=authorization_code&code={CODE}&client_id={CLIENT_ID}&client_secret={CLIENT_SECRET}&redirect_uri={REDIRECT_URI}

请求完之后可以获得refresh_token和access_token,保存这两个值。如下图所示:

再再下一步,获取浏览量PV,UV和实时访客,百度提供了测试接口,https://tongji.baidu.com/api/debug/#可在此处调试接口,获取相关参数。

我写的服务器中因为使用了服务器缓存,因此不读取pv数,只每隔一分钟刷新UV和RV。因此不需要很多参数,项目已经上传至github,项目地址:git@github.com:QiQiWan/baidu-tongji.git

设计过程

新建一个.NET Core项目,添加HttpListener服务器,该服务器可选SSL证书,且可编程度较高.开启泛主机名监听:AddDomain("http://*:1234/");微软的文档说这样干可能会有安全问题,因为我的域名没有备案,无法解析,所以先这样用着,否则用localhost他会在IPv6下监听,使得无法通过IP地址访问.

创建多线程接口:myThread,在有http请求之后,压入新的线程,并返回结果,防止阻塞.请求结束后从线程池中删除进程.

添加辅助百度请求类”BaiduWebHelper”,该类处理所有的百度接口调用.这个个web帮助类的源代码在最后

如何使用

首先在服务器上安装dotnet,微软有安装教程:https://docs.microsoft.com/zh-cn/dotnet/core/install/sdk?pivots=os-linux

将项目克隆到本地,进入项目所在目录 $ cd /baidu-tongji/helper.console/

修改配置文件的参数$ vi config.json

启动:$ dotnet run

如果需要后台启动:$ nohup dotnet run &

按回车键即可,日志文件保存在"nobup.out"log.txt

这里明确一下服务器的返回内容,本来设计是JSON直接返回的,因为JavaScript解析JSON格式的数据比C#方便,但是考虑到跨域请求的问题,还是直接改成了外部脚本,因此服务器返回的脚本.只需要在网页中添加服务器的脚本引用即可.`<script src="http://yourDomain.com:1234"></script>

演示

首先在服务器上运行$ dotnet run

然后打开demo.html,就能直接看到结果啦.

项目地址: https://github.com/QiQiWan/baidu-tongji.git

WebHelper.cs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;
using Fleck;


namespace helper.console
{
class WebSocketThread : myThread
{
private Socket sc;
public WebSocketThread(Socket socket)
{
sc = socket;
}
private byte[] buffer = new byte[1024];
public override void Run()
{
Log.WriteLine(Common.GetTime() + "客户端:" + sc.RemoteEndPoint.ToString() + "已连接");

//握手
int length = sc.Receive(buffer);//接受客户端握手信息
sc.Send(SocketPacker.PackHandShakeData(SocketPacker.GetSecKeyAccetp(buffer, length)));
Log.WriteLine(Common.GetTime() + "已经发送握手协议");

//发送数据
string sendMsg = Program.GetStatics();
sc.Send(SocketPacker.PackData(sendMsg));
Log.WriteLine(Common.GetTime() + "已发送:“" + sendMsg);


Log.WriteLine("----------------------------------------------------------------------------------------------------");
this.Abort();
}
public override void BeforeAbort()
{
sc.Close();
Program.GetServerHelper().RemoveSocket(sc);
}
}

class HttpThread: myThread{
private HttpListenerContext result;
public HttpThread(HttpListenerContext result){
this.result = result;
}
public override void Run(){
HttpServer.SetResponse(result);
//Abort();
}
public override void BeforeAbort(){}
}
/// <summary>
/// 获取百度接口的请求类
/// </summary>
class BaiduWebHelper
{
private HttpWebRequest request;
private HttpWebResponse response;
private string ACCESS_TOKEN;
private string siteId;
private string RequestUrl = "https://openapi.baidu.com/rest/2.0/tongji/report/getData?";
public string resultJson = "";
public BaiduWebHelper()
{
this.ACCESS_TOKEN = Common.GetAccessToken();
this.siteId = Common.site_id;
}


/// <summary>
/// 获取百度API返回的JSON
/// </summary>
/// <returns></returns>
public void GetResult()
{
string url = RequestUrl +
"access_token=" + ACCESS_TOKEN + "&" +
"site_id=" + siteId + "&";
GetUVArgus getUVArgus = new GetUVArgus("20200301");
string result = "";

//获取RV
request = HttpWebRequest.Create(url + GetRealVisittor.GetMethod()) as HttpWebRequest;
request.Method = "GET";
response = request.GetResponse() as HttpWebResponse;
result += ReadWebStream(response.GetResponseStream()) + ", ";
//获取UV
url = url + getUVArgus.ToString();
request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
response = request.GetResponse() as HttpWebResponse;
result += ReadWebStream(response.GetResponseStream());

resultJson = result;
}
/// <summary>
/// 读取web响应流
/// </summary>
/// <param name="stream"></param>
/// <returns></returns>
public string ReadWebStream(Stream stream)
{
StreamReader reader = new StreamReader(stream);
string result = reader.ReadToEnd();
stream.Close();
reader.Close();
return result;
}
/// <summary>
/// token过期后换取新的token
/// </summary>
public void RefreshToken()
{
if (Common.GetRefreshToken() == null)
throw new Exception("未给定更新权限!");
string url = "http://openapi.baidu.com/oauth/2.0/token?grant_type=refresh_token&";
url = url + "refresh_token=" + Common.GetRefreshToken() + "&" +
"client_secret=" + Common.client_secret + "&" +
"client_id=" + Common.client_id;
request = HttpWebRequest.Create(url) as HttpWebRequest;
response = request.GetResponse() as HttpWebResponse;
string result = ReadWebStream(response.GetResponseStream());
string REFRESH_TOKEN = Common.GetJsonValue(result, "refresh_token");
this.ACCESS_TOKEN = Common.GetJsonValue(result, "access_token");
Common.UpdateToken(REFRESH_TOKEN, ACCESS_TOKEN);
}
}

class HttpServer
{
private HttpListener server;
private List<string> domainList = new List<string>();
private List<HttpThread> threadPools = new List<HttpThread>();
public HttpServer()
{
server = new HttpListener();
}
public void Start()
{
foreach (var item in domainList)
server.Prefixes.Add(item);
server.Start();
//IAsyncResult result = server.BeginGetContext(new AsyncCallback(SetResponse), server);
}
/// <summary>
/// 阻塞进程
/// </summary>
public void WaitRequest(){
while(true){
HttpListenerContext result = server.GetContext();
HttpThread temp = new HttpThread(result);
AddThread(temp);
temp.Start();
threadPools.Remove(temp);
}
}
static public void SetResponse(HttpListenerContext result){
HttpListenerRequest request = result.Request;
Log.WriteLine(Common.GetTime() + request.RemoteEndPoint.Address + " 已连接");

HttpListenerResponse response = result.Response;
string responseString = Program.GetStatics();
if (responseString.Contains("error"))
Program.UpdateToken();
responseString = Program.GetStatics();
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
output.Close();
}
private void SetResponse(IAsyncResult result)
{
HttpListener listener = (HttpListener)result.AsyncState;
HttpListenerContext context = listener.EndGetContext(result);
HttpListenerRequest request = context.Request;
Log.WriteLine(Common.GetTime() + request.RemoteEndPoint.Address + " 已连接");

HttpListenerResponse response = context.Response;
string responseString = Program.GetStatics();
if (responseString.Contains("error"))
Program.UpdateToken();
responseString = Program.GetStatics();
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
output.Close();

}
public void Stop()
{
server.Stop();
server.Close();
server.Abort();
}
public void AddDomain(string domain) => domainList.Add(domain);
private void AddThread(HttpThread thread){
this.threadPools.Add(thread);
}
}
class WebSocketServerHelper
{
//套接字服务池
private List<Socket> SocketPools = new List<Socket>();
private WebSocketServer server;

Socket Socket;
public WebSocketServerHelper(IpAdress ipadress)
{
string url = "ws://" + ipadress.ToString();
server = new WebSocketServer(url);
IPEndPoint localIEP = new IPEndPoint(IPAddress.Any, ipadress.port);
Socket = new Socket(localIEP.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
Socket.Bind(localIEP);
Socket.Listen(100);//同时允许100个监听套接字
}
public Socket GetSocket() => Socket.Accept();
//配合主程序采用泛型参数
public void RemoveSocket(Socket socket) => SocketPools.Remove((Socket)socket);
}
class SocketPacker
{
static string byte_to_string(byte[] b)
{
string s = "";
foreach (byte _b in b)
{
s += _b.ToString();
}
return s;
}
/// <summary>
/// 打包握手信息
/// </summary>
/// <param name="secKeyAccept">Sec-WebSocket-Accept</param>
/// <returns>数据包</returns>
public static byte[] PackHandShakeData(string secKeyAccept)
{
var responseBuilder = new StringBuilder();
responseBuilder.Append("HTTP/1.1 101 Switching Protocols" + Environment.NewLine);
responseBuilder.Append("Upgrade: websocket" + Environment.NewLine);
responseBuilder.Append("Connection: Upgrade" + Environment.NewLine);
//responseBuilder.Append("Sec-WebSocket-Accept: " + secKeyAccept + Environment.NewLine + Environment.NewLine);
//如果把上一行换成下面两行,才是thewebsocketprotocol-17协议,但居然握手不成功,目前仍没弄明白!
responseBuilder.Append("Sec-WebSocket-Accept: " + secKeyAccept + Environment.NewLine);
responseBuilder.Append("Sec-WebSocket-Protocol: chat");

return Encoding.UTF8.GetBytes(responseBuilder.ToString());
}

/// <summary>
/// 生成Sec-WebSocket-Accept
/// </summary>
/// <param name="handShakeText">客户端握手信息</param>
/// <returns>Sec-WebSocket-Accept</returns>
public static string GetSecKeyAccetp(byte[] handShakeBytes, int bytesLength)
{
string handShakeText = Encoding.UTF8.GetString(handShakeBytes, 0, bytesLength);
string key = string.Empty;
Regex r = new Regex(@"Sec\-WebSocket\-Key:(.*?)\r\n");
Match m = r.Match(handShakeText);
if (m.Groups.Count != 0)
{
key = Regex.Replace(m.Value, @"Sec\-WebSocket\-Key:(.*?)\r\n", "$1").Trim();
}
byte[] encryptionString = SHA1.Create().ComputeHash(Encoding.ASCII.GetBytes(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"));
return Convert.ToBase64String(encryptionString);
}

/// <summary>
/// 解析客户端数据包
/// </summary>
/// <param name="recBytes">服务器接收的数据包</param>
/// <param name="recByteLength">有效数据长度</param>
/// <returns></returns>
public static string AnalyticData(byte[] recBytes, int recByteLength)
{
if (recByteLength < 2) { return string.Empty; }

bool fin = (recBytes[0] & 0x80) == 0x80; // 1bit,1表示最后一帧
if (!fin)
{
return string.Empty;// 超过一帧暂不处理
}

bool mask_flag = (recBytes[1] & 0x80) == 0x80; // 是否包含掩码
if (!mask_flag)
{
return string.Empty;// 不包含掩码的暂不处理
}

int payload_len = recBytes[1] & 0x7F; // 数据长度

byte[] masks = new byte[4];
byte[] payload_data;

if (payload_len == 126)
{
Array.Copy(recBytes, 4, masks, 0, 4);
payload_len = (UInt16)(recBytes[2] << 8 | recBytes[3]);
payload_data = new byte[payload_len];
Array.Copy(recBytes, 8, payload_data, 0, payload_len);

}
else if (payload_len == 127)
{
Array.Copy(recBytes, 10, masks, 0, 4);
byte[] uInt64Bytes = new byte[8];
for (int i = 0; i < 8; i++)
{
uInt64Bytes[i] = recBytes[9 - i];
}
UInt64 len = BitConverter.ToUInt64(uInt64Bytes, 0);

payload_data = new byte[len];
for (UInt64 i = 0; i < len; i++)
{
payload_data[i] = recBytes[i + 14];
}
}
else
{
Array.Copy(recBytes, 2, masks, 0, 4);
payload_data = new byte[payload_len];
Array.Copy(recBytes, 6, payload_data, 0, payload_len);

}

for (var i = 0; i < payload_len; i++)
{
payload_data[i] = (byte)(payload_data[i] ^ masks[i % 4]);
}

return Encoding.UTF8.GetString(payload_data);
}


/// <summary>
/// 打包服务器数据
/// </summary>
/// <param name="message">数据</param>
/// <returns>数据包</returns>
public static byte[] PackData(string message)
{
byte[] contentBytes = null;
byte[] temp = Encoding.UTF8.GetBytes(message);

if (temp.Length < 126)
{
contentBytes = new byte[temp.Length + 2];
contentBytes[0] = 0x81;
contentBytes[1] = (byte)temp.Length;
Array.Copy(temp, 0, contentBytes, 2, temp.Length);
}
else if (temp.Length < 0xFFFF)
{
contentBytes = new byte[temp.Length + 4];
contentBytes[0] = 0x81;
contentBytes[1] = 126;
contentBytes[2] = (byte)(temp.Length & 0xFF);
contentBytes[3] = (byte)(temp.Length >> 8 & 0xFF);
Array.Copy(temp, 0, contentBytes, 4, temp.Length);
}
else
{
// 暂不处理超长内容
}

return contentBytes;
}
}
/// <summary>
/// 获取UV的请求
/// Method为请求保留字
/// </summary>
class GetUVArgus
{

public string Method = "overview/getTimeTrendRpt";
private string StartDate = "20200301";
private string EndDate;
private string Metrics = "visitor_count";
public GetUVArgus(string startDate)
{
this.StartDate = startDate;
EndDate = GetDateString();
}
public string GetDateString()
{
DateTime now = DateTime.Now;
string month = now.Month < 10 ? "0" + now.Month.ToString() : now.Month.ToString();
string day = now.Day < 10 ? "0" + now.Day : now.Day.ToString();
return now.Year + month + day;
}
public override string ToString()
{
string url = "start_date=" + StartDate + "&" +
"end_date=" + EndDate + "&" +
"method=" + Method + "&" +
"metrics=" + Metrics;
return url;
}
}
/// <summary>
/// 不需要加什么,定义一个Method即可
/// </summary>
class GetRealVisittor
{
static public string Method = "trend/latest/a";
static public string GetMethod() => "method=" + Method;
}
/// <summary>
/// 指定请求类型,UV访问总数,RV实时在线
/// </summary>
enum GetWebType { UV, RV };
/// <summary>
/// 格式化的IP地址包括port
/// </summary>
class IpAdress
{
public string ip;
public int port;
public IpAdress(string ip, int port)
{
this.ip = ip;
this.port = port;
}
public override string ToString()
{
return ip + port.ToString();
}
}
enum Protocol { NoSSL, SSL };
}