Python爬取糗事百科段子

Python爬取糗事百科段子

---

糗事百科是一个原创的糗事笑话分享社区,糗百网友分享的搞笑段子，使用Python爬取段子

翻页操作

http://www.qiushibaike.com/text/
http://www.qiushibaike.com/text/page/2/?s=4964629
http://www.qiushibaike.com/text/page/3/?s=4964629
http://www.qiushibaike.com/text/page/4/?s=4964629

观察可知，我们只需要修改page/{i}参数即可

这里使用正则表达式来提取数据

import requests
import re
from fake_useragent import UserAgent
headers= {'User-Agent':str(UserAgent().random)}

def get_info(url):
    res = requests.get(url,headers=headers)
    ids = re.findall('<h2>(.*?)</h2>',res.text,re.S)
    levels = re.findall('<div class="articleGender \D+Icon">(.*?)</div>',res.text,re.S)
    contents = re.findall('<div class="content">.*?<span>(.*?)</span>',res.text,re.S)
    laughs = re.findall('<span class="stats-vote"><i class="number">(\d+)</i>',res.text,re.S)
    comments = re.findall('<i class="number">(\d+)</i> 评论',res.text,re.S)
    for id,level,sex,content,laugh,comment in zip(ids,levels,sexs,contents,laughs,comments):
        info = {
            'id':id,
            'level':level,
            'content':content,
            'laugh':laugh,
            'comment':comment
        }
        print(info)

if __name__ == '__main__':
    urls = ['http://www.qiushibaike.com/text/page/{}/'.format(str(i)) for i in range(1,15)]
    for url in urls:
        get_info(url)