python之xpath爬虫

最新推荐文章于 2024-04-07 08:00:00 发布

lotluck

最新推荐文章于 2024-04-07 08:00:00 发布

阅读量1.3k

点赞数

分类专栏： Python

本文链接：https://blog.csdn.net/lotluck/article/details/48462275

版权

Python 专栏收录该内容

29 篇文章 0 订阅

订阅专栏

<span style="font-size:18px;">#coding=utf-8
from lxml import etree
import sys
import chardet
import  codecs

reload(sys)
sys.setdefaultencoding("utf-8")
html=u'''
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <title>测试常用规范</title>
</head>
<body>
  <div  id="content">
       <ul id="useful">
           <li>546545645645645</li>
           <li>这是第一条信息</li>
           <li>这是第一条信息</li>       
      </ul>
      <ul id="useless">
           <li>不需要信息</li>
           <li>不需要信息</li>
           <li>不需要信息</li>       
      </ul>
        
      <div>
            <a href="http://lotluck.com">lotluck专栏</a>
     
            <a href="http://lotluck/68525233653.com"  title=杜帅加油啊"">linux自学之旅</a>
      </div>

 </div>

</body>
</html>
'''

print html
selector = etree.HTML(html)

content = selector.xpath('//ul[@id="useful"]/li/text()')

 

print type(content)

for each in content:
    print each

 

##提取属性

link = selector.xpath('//a/@href')
for each in link:
    print each


print '开始'
title = selector.xpath('//a/@title')
f = open('5.txt','w')
f.write(title[0])
f.close()

f1 = open('5.txt','r')
fencoding = chardet.detect(f1.read())
print fencoding

print '结束'
print type(title[0])
 
#print title[0].decode('ascii').encode('utf-8')
print "我们是小青哇"


f3=codecs.open("6.htm","r","utf-8")
content1 = f3.read()
f3.close()
tree=etree.HTML(content1)
node= tree.xpath('//ul[@id="useful"]/li/text()')
for each in node:
    print each
   

</span>