解决python爬虫中文乱码问题
首先网页时可能采用不同编码的,类似这个我爬取的网页
当我直接使用.text函数打印时会出现如下乱码
尝试编码结果
print(strhtml.text.encode(“utf8”))
但发现明显中文被变成了字节
于是看.text的源码
@property
def text(self):
# Try charset from content-type
content = None
encoding = self.encoding
if not self.content:
return str(’’)
# Fallback to auto-detected encoding.
if self.encoding is None:
encoding = self.apparent_encoding
# Decode unicode from given encoding.
try:
content = str(self.content, encoding, errors=‘replace’)
except (LookupError, TypeError):
# A LookupError is raised if the encoding was not found which could
# indicate a misspelling or similar mistake.
# A TypeError can be raised if encoding is None
# So we try blindly encoding.
content = str(self.content, errors=‘replace’)
return content
可以明显的发现此处用于解码的encoding是继承自父类strhtml的,而没有设置过的话父类的默认
的编码方式是“utf8”,但是显然与网页使用的不一样,刚才之所以无法用encode编码,或是解码
是因为.text之后就已经是解码时候的结果了
解决方法
strhtml.encoding=“gb2312”