python多线程爬取某度小说

bqAngus

已于 2024-01-28 10:45:47 修改

阅读量387

点赞数 6

分类专栏： python 文章标签： python

于 2024-01-28 10:44:49 首次发布

本文链接：https://blog.csdn.net/m0_55654781/article/details/135839244

版权

python 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

本文介绍了如何使用Python编程语言，通过调用百度网盘API获取图书《万相之王》的章节内容，并利用异步IO库如aiohttp和asyncio实现批量下载。

摘要由CSDN通过智能技术生成

链接地址：https://dushu.baidu.com/pc/reader?gid=4356290733&cid=1569830905

环境配置：Python 3.8.6

python百度网盘链接：https://pan.baidu.com/s/1I9QkobmvsEAJIjX-8vilWw?pwd=qxdg
提取码：qxdg

工具：pycharm ：PyCharm：JetBrains为专业开发者提供的Python IDE

pip 按照库

pip install aiofiles aiohttp requests asyncio

安装好以后，f12打开网址，发现网址有所有章节的内容

然后点开第一章节，发现连接中包含第一章节内容

获取两个链接url：将%22 修改为" [编码格式问题]

# https://dushu.baidu.com/api/pc/getCatalog?data={"book_id"":"4356290733"} => 所有章节的内容（名称，cid)
# https://dushu.baidu.com/api/pc/getChapterContent?data={"book_id":"4356290733","cid":"4356290733|1569830905","need_bookinfo":1}k开始

# https://dushu.baidu.com/api/pc/getCatalog?data={"book_id"":"4356290733"} => 所有章节的内容（名称，cid)
# https://dushu.baidu.com/api/pc/getChapterContent?data={"book_id":"4356290733","cid":"4356290733|1569830905","need_bookinfo":1}

import json
import os

import aiofiles
import aiohttp
import requests
import asyncio


async def downlod(cid, b_id, title):
    data = {
        'book_id': b_id,
        'cid': f"{b_id}|{cid}",
        'need_bookinfo': 1
    }
    data = json.dumps(data)
    url = f"https://dushu.baidu.com/api/pc/getChapterContent?data={data}"

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            dic = await resp.json()
            folder_path = '万相之王'  # 定义文件夹路径，可以根据需要修改
            os.makedirs(folder_path, exist_ok=True)  # 创建文件夹，如果不存在的话

            file_path = os.path.join(folder_path, title)  # 构建文件路径
            async with aiofiles.open(file_path, mode="w", encoding="utf") as f:
                await f.write(dic['data']['novel']['content'])
            # data = dic['data']['novel']['content']
async def getCatlog(url):
    resp = requests.get(url)
    dic = resp.json()
    tasks = []
    for item in dic['data']['novel']['items']:
        title = item['title']
        cid = item['cid']

        tasks.append(downlod(cid, b_id, title))
    await asyncio.wait(tasks)


if __name__ == '__main__':
    b_id = "4356290733"
    url = 'https://dushu.baidu.com/api/pc/getCatalog?data={"book_id":"' + b_id + '"}'
    asyncio.run(getCatlog(url))

运行程序以后：发现同级目录创建了万相之王的文件夹，下载完成