第一种方法
import requests
url = 'http://www.***.com'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36'
}
html_temp = requests.get(url, headers=headers)
html_temp .encoding = html_temp .apparent_encoding
html = html_temp .text
print(html)
此时的获取到的HTML源码就是正常的中文文本
第二种方法
import requests
url = 'http://www.***.com'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36'
}
html= requests.get(url, headers=headers).content.decode('utf-8')
print(html)
备注:若是用第二种方法报错说utf-8不能解析网站,则此时就只需要更改编码(具体改成哪种格式根据爬取网站的编码来,一般网站大部分都是UTF-8或者GBK),所以可以改成gbk试下
html= requests.get(url, headers=headers).content.decode('GBK')
第三种方法
import requests
url = 'http://www.***.com'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36'
}
content = requests.get(url, headers=headers)
content.encoding = 'utf-8'
html= content.text
print(html)
Unicode编码转中文
def spider(page):
url = ‘**********’
content = requests.get(url).content.decode('unicode-escape')
print(content)