爬取起点中文网

啃西瓜的小煤球

于 2019-09-19 17:15:00 发布

阅读量3k

点赞数 1

分类专栏： Python

本文链接：https://blog.csdn.net/A1010574609/article/details/101028877

版权

Python 专栏收录该内容

23 篇文章

订阅专栏

本文介绍了一种使用Python的requests库和lxml库抓取起点中文网首页小说列表的方法，包括小说名称、作者、分类及简介，并将数据保存到本地文件中。此过程涉及网页请求、HTML解析和数据存储等关键技术。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import requests
from lxml import etree
for page in range(1,15):
    url = 'https://www.qidian.com/all?orderId=&style=1&pageSize=20&siteid=1&pubflag=0&hiddenField=0&page=%s'%page
    response = requests.get(url)
    root = etree.HTML(response.content)
    fiction_list = root.xpath("//ul[@class='all-img-list cf']/li")

    for fiction in fiction_list:
        fiction_name = fiction.xpath("div/h4/a/text()")[0]
        fiction_author = fiction.xpath("div[@class='book-mid-info']/p[@class='author']/a[@class='name']/text()")[0]
        fiction_cla = fiction.xpath("div[@class='book-mid-info']/p[@class='author']/a[@class='go-sub-type']/text()")[0]
        fiction_intro = fiction.xpath("div[@class='book-mid-info']/p[@class='intro']/text()")[0].strip()
        file = open('起点中文网.txt', 'a', encoding='gb18030')
        file.write(fiction_name + '\n' + fiction_author + '\n' + fiction_cla + '\n' + fiction_intro)
        file.write('\n')
        file.write('\n')
        file.close()