python抓取中文网页显示乱码
抓取代码:
import urllib2,urllib,cookielib,threading
import os
import re
url = 'http://www.dugukeji.com/' #抓取的url
req = urllib2.Request(url)
response = urllib2.urlopen(req).read()
print response
研究得知源网页为GBK(gb2312)编码,而python打印为utf8编码,所以需要做一下编码转换
修改后的抓取代码:
import urllib2,urllib,cookielib,threading
import os
import re
url = 'http://www.dugukeji.com/'
req = urllib2.Request(url)
response = urllib2.urlopen(req).read()
response = unicode(response,'GBK').encode('UTF-8')
print response
用unicode函数把GBK编码的网页转换为unicode,再用encode编码成UTF-8输出即可
原文链接http://blog.csdn.net/maverick1990/article/details/8880051