利用python的bs4、urllib、re库对笔q阁爬取解析，生成txt文本

最新推荐文章于 2022-11-12 14:23:50 发布

麦滋堡的摸鱼芝士

最新推荐文章于 2022-11-12 14:23:50 发布

阅读量727

点赞数

文章标签： python 爬虫数据挖掘

本文链接：https://blog.csdn.net/aosiweixin/article/details/121683500

版权

该博客介绍了如何使用Python的requests、BeautifulSoup和tqdm库爬取笔趣阁网站的小说内容，并将其保存为txt文本。通过找到内容div，提取章节名和内容，对每个章节进行遍历并写入文件。示例中针对特定书名和URL进行了爬取操作。

摘要由CSDN通过智能技术生成

利用python的bs4、urllib、re库对笔趣阁爬取解析，生成txt文本
博客地址

import requests
from tqdm import tqdm
from bs4 import BeautifulSoup


def get_content(target):
    req = requests.get(url=target)
    req.encoding = 'utf-8'
    html = req.text
    bf = BeautifulSoup(html, 'lxml')
    texts = bf.find('div', id='content')
    content = texts.text.strip().split('\xa0' * 4)
    return content


class spider(object):
    def __init__(self, book_name, target):
        self.__book_name = book_name
        self.__target = target
        self.__server = 'https://www.xsbiquge.com'

    def do_project(self):
        req = requests.get(url=self.__target)
        req.encoding = 'utf-8'
        html = req.text
        chapter_bs = BeautifulSoup(html, 'lxml')
        chapters = chapter_bs.find('div', id='list')
        chapters = chapters.find_all('a')
        for chapter in tqdm(chapters):
            chapter_name = chapter.string
            url = self.__server + chapter.get('href')
            content = get_content(url)
            with open(self.__book_name, 'a', encoding='utf-8') as f:
                f.write(chapter_name)
                f.write('\n')
                f.write('\n'.join(content))
                f.write('\n')


def main():
    spider1 = spider("book_name.txt", "https://www.xsbiquge.com/88_88080/")
    spider1.do_project()
    print("运行结束")


if __name__ == '__main__':
    main()