爬取纵横中文网---《逐元》全部章节

用python爬取纵横中文网--《逐元》

仅提供学习使用禁止使用在一切非法活动

import requests
from pprint import pprint
import parsel
import os
path = 'xiaoshuo'
os.chdir(path)
url = 'https://bookapi.zongheng.com/api/chapter/getChapterList'
header = {
'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36',
'referer':'https://read.zongheng.com/'
}
data = {'bookId': '1316349'
}
response = requests.post(url=url,headers=header,data=data).json()
# pprint(response)
json_data = response['result']['chapterList'][0]['chapterViewList']
# pprint(json_data)
for i in json_data:
    reag = i['chapterId']
    url = f'https://read.zongheng.com/chapter/1316349/{reag}.html'
    print(url)
    header = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'
    }
    response = requests.get(url=url, headers=header)
    html_data = response.text

    selectr = parsel.Selector(html_data)
    title = selectr.css('.title_txtbox::text').get()
    data1 = selectr.css('.content p').getall()
    for data in data1:
        data2 = data.replace('<p>', '')
        data3 = data2.replace('</p>', '')
        # print(data3)
        with open(title + '.txt', mode='a', encoding='utf-8') as f:
            f.write(data3)
            f.write('\n')

1、requests模块，这个模块是模拟浏览器访问目标网站服务器

2、parsel模块是一个数据解析模块，可以调用xpath、css

3、os模块是一个与本地交互的模块，可以使用该模块调用你的本地文件夹路径

4、还使用了常见的with open用于数据持久化

不懂得可以评论！！！！！