python之xpath爬虫

<span style="font-size:18px;">#coding=utf-8
from lxml import etree
import sys
import chardet
import  codecs

reload(sys)
sys.setdefaultencoding("utf-8")
html=u'''
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <title>测试常用规范</title>
</head>
<body>
  <div  id="content">
       <ul id="useful">
           <li>546545645645645</li>
           <li>这是第一条信息</li>
           <li>这是第一条信息</li>       
      </ul>
      <ul id="useless">
           <li>不需要信息</li>
           <li>不需要信息</li>
           <li>不需要信息</li>       
      </ul>
        
      <div>
            <a href="http://lotluck.com">lotluck专栏</a>
     
            <a href="http://lotluck/68525233653.com"  title=杜帅加油啊"">linux自学之旅</a>
      </div>

 </div>

</body>
</html>
'''

print html
selector = etree.HTML(html)

content = selector.xpath('//ul[@id="useful"]/li/text()')

 

print type(content)

for each in content:
    print each

 

##提取属性

link = selector.xpath('//a/@href')
for each in link:
    print each


print '开始'
title = selector.xpath('//a/@title')
f = open('5.txt','w')
f.write(title[0])
f.close()

f1 = open('5.txt','r')
fencoding = chardet.detect(f1.read())
print fencoding

print '结束'
print type(title[0])
 
#print title[0].decode('ascii').encode('utf-8')
print "我们是小青哇"


f3=codecs.open("6.htm","r","utf-8")
content1 = f3.read()
f3.close()
tree=etree.HTML(content1)
node= tree.xpath('//ul[@id="useful"]/li/text()')
for each in node:
    print each
   

</span>


 

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
以下是一个使用PythonXPath爬虫的实例: ```python import requests from lxml import etree if __name__ == '__main__': url = 'https://pic.netbian.com/4kmeinv/' headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36' } page_text = requests.get(url=url, headers=headers).text tree = etree.HTML(page_text) li_list = tree.xpath('//div[@class="slist"]/ul/li/a') if not os.path.exists('./piclibs'): os.mkdir('./piclibs') for li in li_list: detail_url = 'https://pic.netbian.com' + li.xpath('./img/@src')[0] detail_name = li.xpath('./img/@alt')[0 + '.jpg' detail_name = detail_name.encode('iso-8859-1').decode('GBK') detail_path = './piclibs/' + detail_name detail_data = requests.get(url=detail_url, headers=headers).content with open(detail_path, 'wb') as fp: fp.write(detail_data) print(detail_name, 'seccess!!') ``` 这个爬虫实例通过发送网络请求获取页面源码,然后使用lxml库中的etree模块将页面源码转化为可解析的HTML树对象。之后,使用XPath语法来提取页面中的数据,例如获取图片的URL和名称。最后,将图片保存到本地文件夹。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* [四个Python爬虫案例,带你掌握xpath数据解析方法!](https://blog.csdn.net/lyc2016012170/article/details/124507352)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 50%"] - *2* *3* [python中的xpath爬虫实例,新人处女作!!!](https://blog.csdn.net/Musica_l/article/details/120021669)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 50%"] [ .reference_list ]

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值