python 20行代码爬取某网站小说全文

小小小菜鸟~

已于 2023-03-15 17:42:58 修改

阅读量359

点赞数

分类专栏： Python 文章标签： python Powered by 金山文档

于 2023-03-08 16:45:47 首次发布

本文链接：https://blog.csdn.net/qq_42011157/article/details/129406701

版权

Python 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

python 20行代码爬取某网站小说全文

import requests
from bs4 import BeautifulSoup
import re

if __name__ == '__main__':
    url = 'https://xxx.xxx.com/'
    headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36'
    }
    main_rsp = requests.get(url=url, headers=headers)
    main_rsp.encoding = 'utf-8'
    soup = BeautifulSoup(main_rsp.text, 'lxml')
    title_list = soup.select('.paiban > .menu-item > a')
    fp = open('./三国演义.txt', 'w', encoding='utf-8')
    for title_a in title_list:
        title = title_a.text
        rsp = requests.get(url=title_a['href'], headers=headers)
        rsp.encoding = 'utf-8'
        soup = BeautifulSoup(re.sub('&nbsp;', '', rsp.text), 'lxml')
        content = soup.find('div', class_='grap').text
        content = re.sub('\\n\\r\\n', '', content)
        fp.write(title + '\n' + content + '\n')
        print(f'章节->{title}，爬取完成！！！')
    fp.close()