百度统计数据服务器 ~ web和mechanic技术分享

摘要(ABSTRACT)

前些天有个小朋友说百度统计不支持实时PV、UV。我去找了一下官方提供的文档，百度统计确实没有实时的浏览量导出功能，因此我就想做一个百度统计的数据接口，以给大家用。
最后通过C#写了一个http服务器，作为网站和百度统计数据源的中间件，以连通两个数据端。最后的测试结果表明这种运行方式是可行的，就是有个问题，本地的请求调试延迟平均2000ms,原因是服务器查询百度的数据延迟较大，因此在服务器上设置缓存,以达到快速响应。同时为保证数据的及时性,每1分钟服务器执行更新缓存方法.

百度API接口

百度提供接口的官方说明：https://tongji.baidu.com/sc-web/10000153831/home/dataapi
以下的接口请求示范均使用postman

百度的服务接口都有都需要通过access_token进行调用,这是一个用户级参数,通过登陆获取。要获取这个参数首先需要登录百度开发者平台控制台：百度开发者中心控制台,新建一个工程，添加安全域名，然后记录下app_id和secrect_key。

下一步：通过百度提供的接口，修改相应参数的值用浏览器打开，调起登录；
http://openapi.baidu.com/oauth/2.0/authorize?response_type=code&client_id={CLIENT_ID}&redirect_uri={REDIRECT_URI}&scope=basic&display=popup
参数配置如下图所示：

再下一步，使用获取到的code换取token，API如下图所示,这里可用浏览器打开也可用请求工具进行请求。
http://openapi.baidu.com/oauth/2.0/token?grant_type=authorization_code&code={CODE}&client_id={CLIENT_ID}&client_secret={CLIENT_SECRET}&redirect_uri={REDIRECT_URI}

请求完之后可以获得refresh_token和access_token，保存这两个值。如下图所示：

再再下一步，获取浏览量PV,UV和实时访客，百度提供了测试接口，https://tongji.baidu.com/api/debug/#可在此处调试接口，获取相关参数。

我写的服务器中因为使用了服务器缓存，因此不读取pv数，只每隔一分钟刷新UV和RV。因此不需要很多参数，项目已经上传至github，项目地址：git@github.com:QiQiWan/baidu-tongji.git

设计过程

新建一个.NET Core项目，添加HttpListener服务器,该服务器可选SSL证书,且可编程度较高.开启泛主机名监听:AddDomain("http://*:1234/");微软的文档说这样干可能会有安全问题,因为我的域名没有备案,无法解析,所以先这样用着,否则用localhost他会在IPv6下监听,使得无法通过IP地址访问.

创建多线程接口:myThread,在有http请求之后,压入新的线程,并返回结果,防止阻塞.请求结束后从线程池中删除进程.

添加辅助百度请求类”BaiduWebHelper”,该类处理所有的百度接口调用.这个个web帮助类的源代码在最后

如何使用

首先在服务器上安装dotnet,微软有安装教程:https://docs.microsoft.com/zh-cn/dotnet/core/install/sdk?pivots=os-linux

将项目克隆到本地,进入项目所在目录 $ cd /baidu-tongji/helper.console/

修改配置文件的参数$ vi config.json

启动:$ dotnet run

如果需要后台启动:$ nohup dotnet run &

按回车键即可,日志文件保存在"nobup.out"和log.txt中

这里明确一下服务器的返回内容,本来设计是JSON直接返回的,因为JavaScript解析JSON格式的数据比C#方便,但是考虑到跨域请求的问题,还是直接改成了外部脚本,因此服务器返回的脚本.只需要在网页中添加服务器的脚本引用即可.`<script src="http://yourDomain.com:1234"></script>

演示

首先在服务器上运行$ dotnet run

然后打开demo.html,就能直接看到结果啦.

项目地址: https://github.com/QiQiWan/baidu-tongji.git

WebHelper.cs


using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Security.Cryptography;
using System.Text;
using System.Text.RegularExpressions;
using Fleck;


namespace helper.console
{
    class WebSocketThread : myThread
    {
        private Socket sc;
        public WebSocketThread(Socket socket)
        {
            sc = socket;
        }
        private byte[] buffer = new byte[1024];
        public override void Run()
        {
            Log.WriteLine(Common.GetTime() + "客户端:" + sc.RemoteEndPoint.ToString() + "已连接");

            //握手
            int length = sc.Receive(buffer);//接受客户端握手信息
            sc.Send(SocketPacker.PackHandShakeData(SocketPacker.GetSecKeyAccetp(buffer, length)));
            Log.WriteLine(Common.GetTime() + "已经发送握手协议");

            //发送数据
            string sendMsg = Program.GetStatics();
            sc.Send(SocketPacker.PackData(sendMsg));
            Log.WriteLine(Common.GetTime() + "已发送：“" + sendMsg);


            Log.WriteLine("----------------------------------------------------------------------------------------------------");
            this.Abort();
        }
        public override void BeforeAbort()
        {
            sc.Close();
            Program.GetServerHelper().RemoveSocket(sc);
        }
    }

    class HttpThread: myThread{
        private HttpListenerContext result;
        public HttpThread(HttpListenerContext result){
            this.result = result;
        }
        public override void Run(){
            HttpServer.SetResponse(result);
            //Abort();
        }
        public override void BeforeAbort(){}
    }
    /// <summary>
    /// 获取百度接口的请求类
    /// </summary>
    class BaiduWebHelper
    {
        private HttpWebRequest request;
        private HttpWebResponse response;
        private string ACCESS_TOKEN;
        private string siteId;
        private string RequestUrl = "https://openapi.baidu.com/rest/2.0/tongji/report/getData?";
        public string resultJson = "";
        public BaiduWebHelper()
        {
            this.ACCESS_TOKEN = Common.GetAccessToken();
            this.siteId = Common.site_id;
        }


        /// <summary>
        /// 获取百度API返回的JSON
        /// </summary>
        /// <returns></returns>
        public void GetResult()
        {
            string url = RequestUrl +
                "access_token=" + ACCESS_TOKEN + "&" +
                "site_id=" + siteId + "&";
            GetUVArgus getUVArgus = new GetUVArgus("20200301");
            string result = "";

            //获取RV    
            request = HttpWebRequest.Create(url + GetRealVisittor.GetMethod()) as HttpWebRequest;
            request.Method = "GET";
            response = request.GetResponse() as HttpWebResponse;
            result += ReadWebStream(response.GetResponseStream()) + ", ";
            //获取UV
            url = url + getUVArgus.ToString();
            request = HttpWebRequest.Create(url) as HttpWebRequest;
            request.Method = "GET";
            response = request.GetResponse() as HttpWebResponse;
            result += ReadWebStream(response.GetResponseStream());

            resultJson = result;
        }
        /// <summary>
        /// 读取web响应流
        /// </summary>
        /// <param name="stream"></param>
        /// <returns></returns>
        public string ReadWebStream(Stream stream)
        {
            StreamReader reader = new StreamReader(stream);
            string result = reader.ReadToEnd();
            stream.Close();
            reader.Close();
            return result;
        }
        /// <summary>
        /// token过期后换取新的token
        /// </summary>
        public void RefreshToken()
        {
            if (Common.GetRefreshToken() == null)
                throw new Exception("未给定更新权限!");
            string url = "http://openapi.baidu.com/oauth/2.0/token?grant_type=refresh_token&";
            url = url + "refresh_token=" + Common.GetRefreshToken() + "&" +
                "client_secret=" + Common.client_secret + "&" +
                "client_id=" + Common.client_id;
            request = HttpWebRequest.Create(url) as HttpWebRequest;
            response = request.GetResponse() as HttpWebResponse;
            string result = ReadWebStream(response.GetResponseStream());
            string REFRESH_TOKEN = Common.GetJsonValue(result, "refresh_token");
            this.ACCESS_TOKEN = Common.GetJsonValue(result, "access_token");
            Common.UpdateToken(REFRESH_TOKEN, ACCESS_TOKEN);
        }
    }

    class HttpServer
    {
        private HttpListener server;
        private List<string> domainList = new List<string>();
        private List<HttpThread> threadPools = new List<HttpThread>();
        public HttpServer()
        {
            server = new HttpListener();
        }
        public void Start()
        {
            foreach (var item in domainList)
                server.Prefixes.Add(item);
            server.Start();
            //IAsyncResult result = server.BeginGetContext(new AsyncCallback(SetResponse), server);
        }
        /// <summary>
        /// 阻塞进程
        /// </summary>
        public void WaitRequest(){
            while(true){
                HttpListenerContext result = server.GetContext();
                HttpThread temp = new HttpThread(result);
                AddThread(temp);
                temp.Start();
                threadPools.Remove(temp);
            }
        }
        static public void SetResponse(HttpListenerContext result){
            HttpListenerRequest request = result.Request;
            Log.WriteLine(Common.GetTime() + request.RemoteEndPoint.Address + "  已连接");

            HttpListenerResponse response = result.Response;
            string responseString = Program.GetStatics();
            if (responseString.Contains("error"))
                Program.UpdateToken();
            responseString = Program.GetStatics();
            byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
            response.ContentLength64 = buffer.Length;
            System.IO.Stream output = response.OutputStream;
            output.Write(buffer, 0, buffer.Length);
            output.Close();
        }
        private void SetResponse(IAsyncResult result)
        {
            HttpListener listener = (HttpListener)result.AsyncState;
            HttpListenerContext context = listener.EndGetContext(result);
            HttpListenerRequest request = context.Request;
            Log.WriteLine(Common.GetTime() + request.RemoteEndPoint.Address + "  已连接");

            HttpListenerResponse response = context.Response;
            string responseString = Program.GetStatics();
            if (responseString.Contains("error"))
                Program.UpdateToken();
            responseString = Program.GetStatics();
            byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
            response.ContentLength64 = buffer.Length;
            System.IO.Stream output = response.OutputStream;
            output.Write(buffer, 0, buffer.Length);
            output.Close();

        }
        public void Stop()
        {
            server.Stop();
            server.Close();
            server.Abort();
        }
        public void AddDomain(string domain) => domainList.Add(domain);
        private void AddThread(HttpThread thread){
            this.threadPools.Add(thread);
        }
    }
    class WebSocketServerHelper
    {
        //套接字服务池 
        private List<Socket> SocketPools = new List<Socket>();
        private WebSocketServer server;

        Socket Socket;
        public WebSocketServerHelper(IpAdress ipadress)
        {
            string url = "ws://" + ipadress.ToString();
            server = new WebSocketServer(url);
            IPEndPoint localIEP = new IPEndPoint(IPAddress.Any, ipadress.port);
            Socket = new Socket(localIEP.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
            Socket.Bind(localIEP);
            Socket.Listen(100);//同时允许100个监听套接字
        }
        public Socket GetSocket() => Socket.Accept();
        //配合主程序采用泛型参数
        public void RemoveSocket(Socket socket) => SocketPools.Remove((Socket)socket);
    }
    class SocketPacker
    {
        static string byte_to_string(byte[] b)
        {
            string s = "";
            foreach (byte _b in b)
            {
                s += _b.ToString();
            }
            return s;
        }
        /// <summary>
        /// 打包握手信息
        /// </summary>
        /// <param name="secKeyAccept">Sec-WebSocket-Accept</param>
        /// <returns>数据包</returns>
        public static byte[] PackHandShakeData(string secKeyAccept)
        {
            var responseBuilder = new StringBuilder();
            responseBuilder.Append("HTTP/1.1 101 Switching Protocols" + Environment.NewLine);
            responseBuilder.Append("Upgrade: websocket" + Environment.NewLine);
            responseBuilder.Append("Connection: Upgrade" + Environment.NewLine);
            //responseBuilder.Append("Sec-WebSocket-Accept: " + secKeyAccept + Environment.NewLine + Environment.NewLine);
            //如果把上一行换成下面两行，才是thewebsocketprotocol-17协议，但居然握手不成功，目前仍没弄明白！
            responseBuilder.Append("Sec-WebSocket-Accept: " + secKeyAccept + Environment.NewLine);
            responseBuilder.Append("Sec-WebSocket-Protocol: chat");

            return Encoding.UTF8.GetBytes(responseBuilder.ToString());
        }

        /// <summary>
        /// 生成Sec-WebSocket-Accept
        /// </summary>
        /// <param name="handShakeText">客户端握手信息</param>
        /// <returns>Sec-WebSocket-Accept</returns>
        public static string GetSecKeyAccetp(byte[] handShakeBytes, int bytesLength)
        {
            string handShakeText = Encoding.UTF8.GetString(handShakeBytes, 0, bytesLength);
            string key = string.Empty;
            Regex r = new Regex(@"Sec\-WebSocket\-Key:(.*?)\r\n");
            Match m = r.Match(handShakeText);
            if (m.Groups.Count != 0)
            {
                key = Regex.Replace(m.Value, @"Sec\-WebSocket\-Key:(.*?)\r\n", "$1").Trim();
            }
            byte[] encryptionString = SHA1.Create().ComputeHash(Encoding.ASCII.GetBytes(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"));
            return Convert.ToBase64String(encryptionString);
        }

        /// <summary>
        /// 解析客户端数据包
        /// </summary>
        /// <param name="recBytes">服务器接收的数据包</param>
        /// <param name="recByteLength">有效数据长度</param>
        /// <returns></returns>
        public static string AnalyticData(byte[] recBytes, int recByteLength)
        {
            if (recByteLength < 2) { return string.Empty; }

            bool fin = (recBytes[0] & 0x80) == 0x80; // 1bit，1表示最后一帧  
            if (!fin)
            {
                return string.Empty;// 超过一帧暂不处理 
            }

            bool mask_flag = (recBytes[1] & 0x80) == 0x80; // 是否包含掩码  
            if (!mask_flag)
            {
                return string.Empty;// 不包含掩码的暂不处理
            }

            int payload_len = recBytes[1] & 0x7F; // 数据长度  

            byte[] masks = new byte[4];
            byte[] payload_data;

            if (payload_len == 126)
            {
                Array.Copy(recBytes, 4, masks, 0, 4);
                payload_len = (UInt16)(recBytes[2] << 8 | recBytes[3]);
                payload_data = new byte[payload_len];
                Array.Copy(recBytes, 8, payload_data, 0, payload_len);

            }
            else if (payload_len == 127)
            {
                Array.Copy(recBytes, 10, masks, 0, 4);
                byte[] uInt64Bytes = new byte[8];
                for (int i = 0; i < 8; i++)
                {
                    uInt64Bytes[i] = recBytes[9 - i];
                }
                UInt64 len = BitConverter.ToUInt64(uInt64Bytes, 0);

                payload_data = new byte[len];
                for (UInt64 i = 0; i < len; i++)
                {
                    payload_data[i] = recBytes[i + 14];
                }
            }
            else
            {
                Array.Copy(recBytes, 2, masks, 0, 4);
                payload_data = new byte[payload_len];
                Array.Copy(recBytes, 6, payload_data, 0, payload_len);

            }

            for (var i = 0; i < payload_len; i++)
            {
                payload_data[i] = (byte)(payload_data[i] ^ masks[i % 4]);
            }

            return Encoding.UTF8.GetString(payload_data);
        }


        /// <summary>
        /// 打包服务器数据
        /// </summary>
        /// <param name="message">数据</param>
        /// <returns>数据包</returns>
        public static byte[] PackData(string message)
        {
            byte[] contentBytes = null;
            byte[] temp = Encoding.UTF8.GetBytes(message);

            if (temp.Length < 126)
            {
                contentBytes = new byte[temp.Length + 2];
                contentBytes[0] = 0x81;
                contentBytes[1] = (byte)temp.Length;
                Array.Copy(temp, 0, contentBytes, 2, temp.Length);
            }
            else if (temp.Length < 0xFFFF)
            {
                contentBytes = new byte[temp.Length + 4];
                contentBytes[0] = 0x81;
                contentBytes[1] = 126;
                contentBytes[2] = (byte)(temp.Length & 0xFF);
                contentBytes[3] = (byte)(temp.Length >> 8 & 0xFF);
                Array.Copy(temp, 0, contentBytes, 4, temp.Length);
            }
            else
            {
                // 暂不处理超长内容  
            }

            return contentBytes;
        }
    }
    /// <summary>
    /// 获取UV的请求
    /// Method为请求保留字
    /// </summary>
    class GetUVArgus
    {

        public string Method = "overview/getTimeTrendRpt";
        private string StartDate = "20200301";
        private string EndDate;
        private string Metrics = "visitor_count";
        public GetUVArgus(string startDate)
        {
            this.StartDate = startDate;
            EndDate = GetDateString();
        }
        public string GetDateString()
        {
            DateTime now = DateTime.Now;
            string month = now.Month < 10 ? "0" + now.Month.ToString() : now.Month.ToString();
            string day = now.Day < 10 ? "0" + now.Day : now.Day.ToString();
            return now.Year + month + day;
        }
        public override string ToString()
        {
            string url = "start_date=" + StartDate + "&" +
                "end_date=" + EndDate + "&" +
                "method=" + Method + "&" +
                "metrics=" + Metrics;
            return url;
        }
    }
    /// <summary>
    /// 不需要加什么,定义一个Method即可
    /// </summary>
    class GetRealVisittor
    {
        static public string Method = "trend/latest/a";
        static public string GetMethod() => "method=" + Method;
    }
    /// <summary>
    /// 指定请求类型,UV访问总数,RV实时在线
    /// </summary>
    enum GetWebType { UV, RV };
    /// <summary>
    /// 格式化的IP地址包括port
    /// </summary>
    class IpAdress
    {
        public string ip;
        public int port;
        public IpAdress(string ip, int port)
        {
            this.ip = ip;
            this.port = port;
        }
        public override string ToString()
        {
            return ip + port.ToString();
        }
    }
    enum Protocol { NoSSL, SSL };
}

.NET Core

#学习 #笔记 #百度统计

百度统计数据服务器

https://www.eatrice.cn/post/百度统计数据服务器/

作者

吃白饭-EatRice

发布于

2020年3月5日

许可协议

弹性力学-边界条件和圣维南原理上一篇

土方工程-边坡土方量计算程序设计下一篇