爬虫批量获取笑话段子

年丰巷简单的探花

于 2022-02-08 22:24:44 发布

阅读量560

点赞数

分类专栏：日常练习文章标签： python

本文链接：https://blog.csdn.net/wh90306/article/details/122832491

版权

日常练习专栏收录该内容

34 篇文章 2 订阅

订阅专栏

# 爬虫批量获取笑话段子
import requests
from lxml import etree

headers = {
	'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64)\
	AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0\
	.2743.116 Safari/537.36',
	'Accept-Language': 'zh-CN,zh;q=0.8'}

url = 'https://ishuo.cn/special/qiushibaike'
response=requests.get(url,headers=headers).text
html = etree.HTML(response)

result = html.xpath('//div[@class="content"]/ul/li/a/@href')

for site in result:
	xurl="https://ishuo.cn/"+site
	response2=requests.get(xurl).text
	html2=etree.HTML(response2)
	result2=html2.xpath("//div[@class='content']")
for i in range(0,len(result2)):
    print(result2[i].text)
    print("-------------------")

# https://ishuo.cn/subject/169