一.介绍
在scrapy中,默认不支持Brotli解压,当发现响应乱码时,如何分析确定是由Brotli压缩引起的呢?
1)是看请求头是否有'Accept-Encoding': "gzip, deflate, br" 中的br,如果去掉br 再请求网页,如果响应不成功,则表示服务端只支持br压缩格式,如果成功则看是否乱码。
2) 添加中间件调试查看,使用的是哪种压缩格式,关键代码如下
custom_settings = {
'DOWNLOADER_MIDDLEWARES':{
#加上中间件,主要用于查看是什么类型的压缩
'realtime_python_crawler.middlewares_custom.myCompression_dm.myHtmlCompression':500
}
}
在创建一个myCompression_dm.py文件,在return时打入断点调试进去
from scrapy.downloadermiddlewares.httpcompression import HttpCompressionMiddleware
class myHtmlCompression(HttpCompressionMiddleware):
def process_response(self, request, response, spider):
return super().process_response(request, response, spider)
HttpCompressionMiddleware中间件中源码中解缩代码如下:
def _decode(self, body, encoding):
if encoding == b"gzip" or encoding == b"x-gzip":
body = gunzip(body)
if encoding == b"deflate":
try:
body = zlib.decompress(body)
except zlib.error:
# ugly hack to work with raw deflate content that may
# be sent by microsoft servers. For more information, see:
# http://carsten.codimi.de/gzip.yaws/
# http://www.port80software.com/200ok/archive/2005/10/31/868.aspx
# http://www.gzip.org/zlib/zlib_faq.html#faq38
body = zlib.decompress(body, -15)
if encoding == b"br" and b"br" in ACCEPTED_ENCODINGS:
body = brotli.decompress(body)
if encoding == b"zstd" and b"zstd" in ACCEPTED_ENCODINGS:
# Using its streaming API since its simple API could handle only cases
# where there is content size data embedded in the frame
reader = zstandard.ZstdDecompressor().stream_reader(io.BytesIO(body))
body = reader.read()
return body
解决方法,如果确定是br压缩格式引起的,就安装pip install Brotli, 这样响应就不会乱码