XPath与多线程爬虫学习

最新推荐文章于 2024-09-18 08:37:42 发布

benhuo931115

最新推荐文章于 2024-09-18 08:37:42 发布

阅读量884

点赞数

分类专栏：机器学习文章标签：多线程爬虫

本文链接：https://blog.csdn.net/benhuo931115/article/details/52819586

版权

机器学习专栏收录该内容

4 篇文章 0 订阅

订阅专栏

XPath与多线程爬虫学习

什么是XPath

XPath是一门语言
XPath可以在XML文档中查找信息
XPath支持HTML
XPath通过元素和属性进行导航
XPath可以用来提取信息
XPath比正则表达式厉害
XPath比正则表达式简单

XPath的使用

XPath与HTML结构

树状结构
逐层展开
逐层定位
寻找独立节点

获取网页元素标签

手动分析法
html->body->div->ul->li
Chrome生成法
打开Chrome浏览器->右键”检查”->选中元素，右键”Copy-Copy XPath”，得到：
//*[@id=”J_IdsSegments”]/ul/li

应用XPath提取内容

//定位根节点
/往下层寻找
提取文本内容：/text()
提取属性内容：/@xxxx

实例：

from lxml import etree

html = '''
<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <title>test</title>
</head>
<body>
<div id="content">
    <ul id="useful">
        <li>1111</li>
        <li>2223</li>
        <li>3333</li>
    </ul>
    <ul id="useless">
        <li>4444</li>
        <li>5555</li>
        <li>6666</li>
    </ul>
    <div>
        <a href="https://www.hao123.com/">好123</a>
    </div>
</div>
</body>
</html>
'''

selector = etree.HTML(html)

# 提取文本
content = selector.xpath('//ul[@id="useful"]/li/text()')
for each in content:
    print(each)

# 提取属性
link = selector.xpath('//a/@href')
for each in link:
    print(each)

XPath的特殊用法

以相同的字符开头

starts-with(@属性名称，属性字符相同部分)，例如得到以下标签的值
```
<div id="test-1">1111</div>
<div id="test-2">2222</div>
<div id="testfault">3333</div>
```
#
selector = etree.HTML(html)
content = selector.xpath(‘//div[starts-with(@id,”test”)]/text()’)
for each in content:
```
    print(each)
```
标签套标签

string(.)，取出以下标签及子标签的值
```
<div id="class3">美女，
    <font color=red>你的QQ是多少？</font>
</div>
```
#
data = selector.xpath(‘//div[@id=”test3”]’)[0]
info = data.xpath(‘string(.)’)
content = info.replace(‘\n’,”).replace(’ ‘,”)
print(content) # 美女，你的QQ是多少？

Python并行化——map的使用

map函数一手包办了序列操作、参数传递和结果保存等一系列的操作

from multiprocessing.dummy import Pool
pool = Pool(4)
results = pool.map(爬取函数，网址列表)
pool.close()
pool.join()

实例：爬取百度贴吧

#-*-coding:utf8-*-
from lxml import etree
from multiprocessing.dummy import Pool as ThreadPool
import requests
import json
import sys

reload(sys)

sys.setdefaultencoding('utf-8')

'''重新运行之前请删除content.txt，因为文件操作使用追加方式，会导致内容太多。'''

def towrite(contentdict):
    f.writelines(u'回帖时间:' + str(contentdict['topic_reply_time']) + '\n')
    f.writelines(u'回帖内容:' + unicode(contentdict['topic_reply_content']) + '\n')
    f.writelines(u'回帖人:' + contentdict['user_name'] + '\n\n')

def spider(url):
    html = requests.get(url)
    selector = etree.HTML(html.text)
    content_field = selector.xpath('//div[@class="l_post l_post_bright "]')
    item = {}
    for each in content_field:
        reply_info = json.loads(each.xpath('@data-field')[0].replace('&quot',''))
        author = reply_info['author']['user_name']
        content = each.xpath('div[@class="d_post_content_main"]/div/cc/div[@class="d_post_content j_d_post_content "]/text()')[0]
        reply_time = reply_info['content']['date']
        print(content)
        print(reply_time)
        print(author)
        item['user_name'] = author
        item['topic_reply_content'] = content
        item['topic_reply_time'] = reply_time
        towrite(item)

if __name__ == '__main__':
    pool = ThreadPool(4)
    f = open('content.txt','a')
    page = []
    for i in range(1,21):
        newpage = 'http://tieba.baidu.com/p/3522395718?pn=' + str(i)
        page.append(newpage)

    results = pool.map(spider, page)
    pool.close()
    pool.join()
    f.close()