python3 unicodedecodeerror_python3的UnicodeDecodeError解决方法

爬虫部分解码异常

response.content.decode() # 默认使用 utf-8 出现解码异常

以下是设计的通用解码

通过 text 获取编码

# 通过 text 获取编码

import requests

from lxml import etree

def public_decode():

headers = {

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'

}

response = requests.get('https://blog.csdn.net/a13951206104', headers=headers)

html = etree.HTML(response.text) # response.text 能自动获取编码, 大多乱码

_charset = html.xpath('//@charset') or []

if _charset:

encode_content = response.content.decode(_charset[0].strip().lower(),

errors='replace') # 如果设置为replace，则会用?取代非法字符；

return {'response_text': encode_content, "response_obj": response}

for _charset_ in ['utf-8', 'gbk', 'gb2312'] # 国内主要这3种:

if '�' not in response.content.decode(_charset_, errors='replace'):

return {'response_text': response.content.decode(_charset_, errors='replace'),

"response_obj": response}

else:

# 默认还得是 utf-8

return {'response_text': response.content.decode('utf-8', errors='replace'),

"response_obj": response}

通过数据 来解编码(推荐)

def public_decode(response):

headers = {

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'

}

response = requests.get('https://blog.csdn.net/a13951206104', headers=headers)

html = etree.HTML(response.text)

# 不希望抓下来的数据中有非法字符

item = dict()

result = None

for _charset_ in ['utf-8', 'gbk', 'gb2312']:

if response:

result = response.content.decode(_charset_, errors='replace')

item['content'] = html.xpath('//*[@id="content"]')

if '�' not in result['content'].strip():

result =response.content.decode(_charset_, errors='replace')

break

if not result:

# 默认 utf-8

result = response.content.decode(_charset_, errors='replace')

errors=‘replace' 使解码不报异常, 然后把几个常用的编码一个个试下, 最后要看落下来的数据, 所以最好拿数据 去获取合适的编码

以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持我们。

本文标题: python3的UnicodeDecodeError解决方法

本文地址: http://www.cppcns.com/jiaoben/python/294196.html