今天爬网站的内容,在写入TXT文件时,页面总是报
UnicodeEncodeError: 'gbk' codec can't encode character '\ufeff' in position 0: illegal multibyte sequence
错误,网上找了半天也没找到解决办法。
后来终于找到了解决办法,十分简单:在f = open('test.txt','wt',encoding='utf-8') 里加上encoding='utf-8'这个参数就行了。
出错的原因是网页及python的编码都是utf-8,在写进txt时Windows默认转码成gbk,遇到某些gbk不支持的字符就会报错。在打开文件时就声明编码方式为utf-8就能避免这个错误。
附上代码:
import requests
with open('C:\\Users\\HP\\Desktop\\learn_test\\urls.txt') as f:
i = 1
for each_line in f:
url = each_line.strip()
res = requests.get(url)
res.encoding = 'utf-8'
html = res.text
print(type(html))
filename = 'url_' + str(i) + '.txt'
with open(filename,'wt',encoding='utf-8') as f:
f.write(html)
i += 1