因为采集某个网页遇到问题,一直无法获取页面数据。
经过一番排查,发现该网站会检查客户端的Header信息,如果遇到不明确的Header信息就直接否定,返回0数据。
如果Header信息正确,就会返回经过GZip压缩的数据,这样直接获取网页数据的想法破灭了。
实际上.NET已经为我们封装好了GZip和Deflate加解压算法类,都位于 System.IO.Compression 命名空间内。
下面我来演示如何分析并提取经过GZIP压缩的网页数据流。
Imports
System.Net
Imports System.IO
Imports System.IO.Compression
Imports System.Text
Function GZip2Html( ByVal url As String ) As String
' HTTP请求
Dim req As HttpWebRequest = HttpWebRequest.Create(url)
req.Method = " GET " ' GET 或 POST
req.Accept = " */* "
req.UserAgent = " Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;) " ' 这里模拟 XP系统 IE7 的请求字段
req.Referer = " 来源地址 "
' 一些头信息的模拟
req.Headers( " Accept-Language " ) = " zh-cn "
req.Headers( " UA-CPU " ) = " x86 "
req.Headers( " Accept-Encoding " ) = " gzip, deflate "
' HTTP获取
Dim res As HttpWebResponse = req.GetResponse
' 获取数据文本编码
' Dim enc As Encoding = Encoding.GetEncoding(res.CharacterSet) '如果出现乱码请换下面这种方式
Dim enc As Encoding = Encoding.Default
' 创建一个GZip解压流
Dim gz As New GZipStream(res.GetResponseStream, CompressionMode.Decompress)
' 用一个临时内存流来保存解压数据
Dim ms As New MemoryStream
' 缓冲数据
Dim buf( 99 ) As Byte , i As Integer = 0
' 不断从流中解压数据
While True
i = gz.Read(buf, 0 , 100 )
If i = 0 Then Exit While
ms.Write(buf, 0 , i)
End While
' 将数据转换为字符
Dim ret As String = enc.GetString(ms.ToArray)
' 关闭所有的流
gz.Close()
ms.Close()
res.Close()
Return ret
End Function
Imports System.IO
Imports System.IO.Compression
Imports System.Text
Function GZip2Html( ByVal url As String ) As String
' HTTP请求
Dim req As HttpWebRequest = HttpWebRequest.Create(url)
req.Method = " GET " ' GET 或 POST
req.Accept = " */* "
req.UserAgent = " Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;) " ' 这里模拟 XP系统 IE7 的请求字段
req.Referer = " 来源地址 "
' 一些头信息的模拟
req.Headers( " Accept-Language " ) = " zh-cn "
req.Headers( " UA-CPU " ) = " x86 "
req.Headers( " Accept-Encoding " ) = " gzip, deflate "
' HTTP获取
Dim res As HttpWebResponse = req.GetResponse
' 获取数据文本编码
' Dim enc As Encoding = Encoding.GetEncoding(res.CharacterSet) '如果出现乱码请换下面这种方式
Dim enc As Encoding = Encoding.Default
' 创建一个GZip解压流
Dim gz As New GZipStream(res.GetResponseStream, CompressionMode.Decompress)
' 用一个临时内存流来保存解压数据
Dim ms As New MemoryStream
' 缓冲数据
Dim buf( 99 ) As Byte , i As Integer = 0
' 不断从流中解压数据
While True
i = gz.Read(buf, 0 , 100 )
If i = 0 Then Exit While
ms.Write(buf, 0 , i)
End While
' 将数据转换为字符
Dim ret As String = enc.GetString(ms.ToArray)
' 关闭所有的流
gz.Close()
ms.Close()
res.Close()
Return ret
End Function
本人测试解压数据成功,其他环境未进行测试,请自行修改。