异步爬取小说所有章节并下载

下载百度APP里面的西游记

1.查找获取所有章节标题的url：

https://dushu.baidu.com/api/pc/getCatalog?data={"book_id":"4306063500"}

进入第一章节，详细内容的url：

https://dushu.baidu.com/api/pc/getChapterContent?data={"book_id":"4306063500","cid":"4306063500|1569782244","need_bookinfo":1}

2.访问getCatalog 拿到所有章节的cid和名称，准备异步下载任务

async def getCatalog(url):
    resp = requests.get(url)
    dic = resp.json()
    tasks = []
    for item in dic['data']['novel']['items']:  # item对应每一个章节的名称和cid
        title = item['title']
        cid = item['cid']
        # 准备异步下载任务
        p = asyncio.create_task(aiodownload(cid, b_id, title))
        tasks.append(p)
    await asyncio.wait(tasks)

3.访问getChapterContent 异步下载所有章节的内容

async def aiodownload(cid, b_id, title):
    data={
        "book_id": f"{b_id}",
        "cid": f"{b_id}|{cid}",
        "need_bookinfo": 1
    }

    data = json.dumps(data)
    url = f"https://dushu.baidu.com/api/pc/getChapterContent?data={data}"

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            dic = await resp.json()

            async with aiofiles.open("西游记/"+title, 'w', encoding="utf-8") as f:
                await f.write(dic['data']['novel']['content'])  # 把小说内容写出

4.main函数

if __name__ == '__main__':
    b_id = "4306063500"
    url = 'https://dushu.baidu.com/api/pc/getCatalog?data={"book_id":"' + b_id + '"}'
    asyncio.run(getCatalog(url))