最近做毕设的时候需要用Python通过XPath从HTML中获取一些值
在大多数网站中是很好用的,今天突然发现在搜狐视频上就获取的不是很正常了
测试代码:
<span style="font-size:18px;">#coding:utf8
from lxml import etree
f = open('sohu.html', 'r')
html = f.read()
f.close()
tree = etree.HTML(html)
container = tree.xpath("//*")
print container</span>
输出结果:
<span style="font-size:18px;">[<Element html at 0x25b0b08>, <Element head at 0x25b0b48>, <Element script at 0x25b0f08>, <Element meta at 0x25b0048>, <Element meta at 0x25b00c8>, <Element meta at 0x25b0108>, <Element meta at 0x25b0208>]</span>
可以明显发现只能获取到几个节点,跟优酷的结果完全不同
一直以为是自己xpath写错了,后来发现是搜狐视频首页的编码是GBK,其他能正常获取节点的编码都是UTF-8
改了一下代码:
<span style="font-size:18px;">#coding:utf8
from lxml import etree
import re
f = open('sohu.html', 'r')
html = f.read()
f.close()
html = re.sub(r'charset=(\w*)', 'charset=UTF-8', html)
tree = etree.HTML(html)
container = tree.xpath("//*")
print len(container)</span>
最后输出:
<span style="font-size:18px;">3216</span>
搞定!~\(≧▽≦)/~