1、SublimeText中在控制台输出unicode字符串时会有问题
如:
import sys
s = u'中国'
print s
会报错:
Traceback (most recent call last):
File "E:\PHPnow\htdocs\kidsview\build\test.py", line 7, in <module>
print s
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)
这是因为ST3在输出编码未定义情况下,会默认使用系统的管道输出编码。而Windows下默认是ansii,而ST3的控制台是需要UTF8才能正确显示,所以报错。
可以通过下面代码查看当前管道输出编码
import sys
print sys.stdout.encoding
解决方案:
在python的Python.sublime-build资源文件里,添加一行"env": {"PYTHONIOENCODING": "utf8"},
如下
{
"shell_cmd": "python -u \"$file\"",
"file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
"selector": "source.python",
"env": {"PYTHONIOENCODING": "utf8"},
"variants":
[
{
"name": "Syntax Check",
"shell_cmd": "python -m py_compile \"${file}\"",
}
]
}
二、使用lxml解析网页
部分关键代码
<span style="font-weight: normal;"> req = urllib2.Request(url)
f = urllib2.urlopen(req)
data = f.read()
encoding = f.info().get("Content-Encoding")
if encoding in ('gzip', 'x-gzip', 'deflate'):
if encoding == "deflate":
data = StringIO.StringIO(zlib.decompress(data))
else:
data = gzip.GzipFile('', 'rb', 9, StringIO(data))
cl = clean.Cleaner()
data = cl.clean_html(data.decode("utf-8"))
parser = etree.HTMLParser()
tree = etree.parse(StringIO(data), parser)
nodelist = tree.xpath("//div[@class='xxx]/a")</span>
注意事项:
<span style="font-weight: normal;"> req = urllib2.Request(url)
f = urllib2.urlopen(req)
data = f.read()
encoding = f.info().get("Content-Encoding")
if encoding in ('gzip', 'x-gzip', 'deflate'):
if encoding == "deflate":
data = StringIO.StringIO(zlib.decompress(data))
else:
data = gzip.GzipFile('', 'rb', 9, StringIO(data))
cl = clean.Cleaner()
data = cl.clean_html(data.decode("utf-8"))
parser = etree.HTMLParser()
tree = etree.parse(StringIO(data), parser)
nodelist = tree.xpath("//div[@class='xxx]/a")</span>
1、lxml里面的大部分的类,像parser,cleaner都要求传递的是unicode字符串,
2、网页获取下来的数据,一般是utf8或者gbk,需要先做解码。有的网站是压缩后的数据,需要先进行解压缩