Python爬虫requests后的html乱码解决(gzip, deflate, br)

1、问题如下
  • 乱码bug集1
  • 前提:resp.encoding编码与网页源码编码一致;本例编码为’utf-8’;
  • 直接输出reponse.text会报异常:UnicodeEncodeError: ‘gbk’ codec can’t encode character ‘\ufffd’ in position 0: illegal multibyte sequence
headers = {
     'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,v=b3",
     'accept-Encoding': "gzip, deflate, br",
     'accept-Language': "zh-CN,zh;q=0.9",
     'connection': "close",
     'Upgrade-Insecure-Requests': '1',
     'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
      }
resp = requests.get(url, headers=headers, proxies=proxy, timeout=20)
resp.encoding = 'utf-8'
print(resp.text)

在这里插入图片描述

  • 修改print(resp.text),出现如下乱码
headers = {
   'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,v=b3",
   'accept-Encoding': "gzip, deflate, br",
   'accept-Language': "zh-CN,zh;q=0.9",
   'connection': "close",
   'Upgrade-Insecure-Requests': '1',
   'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
    }
resp = requests.get(url, headers=headers, proxies=proxy, timeout=20)
resp.encoding = 'utf-8'
print(resp.text.encode('gbk', 'ignore').decode('gbk'))

在这里插入图片描述

2、解决问题
  • 将 ‘accept-Encoding’: "gzip, deflate, br"里面的br去掉即可,或者这一行直接注释掉
headers = {
     'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,v=b3",
     'accept-Encoding': "gzip, deflate",
     'accept-Language': "zh-CN,zh;q=0.9",
     'connection': "close",
     'Upgrade-Insecure-Requests': '1',
     'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
      }
resp = requests.get(url, headers=headers, proxies=proxy, timeout=20)
resp.encoding = 'utf-8'
print(resp.text.encode('gbk', 'ignore').decode('gbk'))

在这里插入图片描述

评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值
>