BeautifulSoup编写PYTHON爬虫案例-下载名著小说

爬虫背景

目的：爬取诗词名著网上的小说
网页：http://www.shicimingju.com/book/sanguoyanyi.html
分析过程：
1、查找当前页面多个章节的URL
2、打开每个章节保存文字内容

代码如下：

import requests
from bs4 import BeautifulSoup

url = 'http://www.shicimingju.com/book/sanguoyanyi.html'
header = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36'}
res = requests.get(url,headers=header).text

soup = BeautifulSoup(res,'html.parser')
mulu = soup.find('div',class_='book-mulu').find_all('a')
book_name = soup.h1.string
chapter_url = ['http://www.shicimingju.com'+x['href'] for x in mulu]

f=open(book_name+'.txt','a',encoding='utf-8')
for i in chapter_url:
    res = requests.get(i,headers=header).text
    soup = BeautifulSoup(res,'html.parser')
    contents = soup.find('div',class_='card bookmark-list').get_text()
    f.write(contents)