使用网页抓取技术获取自己World Community Grid任务信息的经历

在获取World Community Grid任务页面的时候经历了很多的周折。开始的时候，认为这个页面和其他的页面相似，直接Post登陆信息即可获取到登陆Cookie，但是此方法失败，远端会返回登陆错误的页面。然后就搁置了好长时间，直到发现IE9开发者工具提供了比较友好的网页抓包数据，不过和WireShark获得的数据差不多，只是过滤了和HTTP协议无关的数据，更容易查看。

在捕获登陆操作以后，可以发现浏览器向服务器发送了怎样的请求，同时也可以知道服务器给予了怎样的响应。

上图为跟踪到的登录时的POST请求，可以看到请求的真实地址为/j_security_check，从页面的js代码中也可以看到这一点。然后根据请求的内容将对应的键值对写入request.Header中即可。让我走最多弯路的就是Cookie的处理了，开始我将Cookie的内容赋值给request.Header["Cookie"]，结果始终无法登陆成功，经过一系列的尝试，最后想起来request是使用CookieContainer来存储发送的Cookie的，直接将捕获到的“__utma”等键值对写到CookieContainer对象中，再将此对象赋值给request.CookieContainer，就可以成功登陆并且获取JSESSIONID和访问认证信息等Cookie了，然后就可以访问对应的页面获取想要的内容了。

基本代码如下：

string url = "https://secure.worldcommunitygrid.org/j_security_check"; string resulturl = "https://secure.worldcommunitygrid.org/ms/viewBoincResults.do?filterDevice=0&filterStatus=4&projectId=-1&sortBy=returnedTime&pageNum=1"; HttpWebRequest request = null; HttpWebResponse response = null; Stream requestStream = null; CookieContainer cc = new CookieContainer(); try { string formData = "j_username=zl860628&j_password=" + Constant.PassWord1; ASCIIEncoding encoding = new ASCIIEncoding(); byte[] data = encoding.GetBytes(formData); //将请求时的Cookie加入 cc.Add(new Cookie("__utma", "2464182.1748491084.1301203514.1301203514.1301234717.2", "/", "secure.worldcommunitygrid.org")); cc.Add(new Cookie("__utmz", "2464182.1301203514.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none)", "/", "secure.worldcommunitygrid.org")); cc.Add(new Cookie("__utmc", "2464182", "/", "secure.worldcommunitygrid.org")); cc.Add(new Cookie("__utmb", "2464182", "/", "secure.worldcommunitygrid.org")); request = (HttpWebRequest)WebRequest.Create(url); request.Method = "POST"; request.ContentType = "application/x-www-form-urlencoded"; request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13"; request.Accept = "text/html,application/xhtml+xml,*/*"; request.ContentLength = data.Length; request.CookieContainer = cc; request.KeepAlive = true; request.Timeout = 100 * 1000; request.Headers["Accept-Encoding"] = "gzip, deflate"; request.Headers["Accept-Language"] = "en"; request.Headers["Accept-Charset"] = "GB2312,utf-8"; request.Headers["Cookie"] = "__utma=2464182.1748491084.1301203514.1301203514.1301234717.2; __utmz=2464182.1301203514.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); __utmc=2464182; __utmb=2464182"; //request.Headers["Cookie"] = "__utmb=2464182"; requestStream = request.GetRequestStream(); requestStream.Write(data, 0, data.Length); response = (HttpWebResponse)request.GetResponse(); response.Cookies = cc.GetCookies(request.RequestUri); //获取登陆后的Cookie cc.Add(response.Cookies); //使用得到的Cookie，获取任务结果页面 string html = transportTool.GetAndGetHTML(resulturl, cc, Encoding.Default); } catch (WebException ex) { } finally { if (response != null) response.Close(); if (request != null) request.Abort(); }