BeautifulSoup爬小说

舍友想看小说，但是嫌弃网页版有广告，就用python爬一下

先导入库

import requests
from bs4 import BeautifulSoup

输入要请求的网页，进行UA伪装，为了避免出现乱码，设置一下encoding

url = ("http://www.biaiku.net/b/111/111622/")
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
    }
res = requests.get(url=url,headers=headers)
res.encoding = "gbk"
response = res.text

新建一个.txt文件,用于存储结果。

fp = open('./yiwuandeng.txt','w',encoding='utf-8')

用bs爬取每一章的链接，并对每一章的链接进行requests请求

soup = BeautifulSoup(response,'lxml')
src_list = soup.select('.add-box > ul > li')
for li in src_list:
    title = li.a.string
    detail_url = 'http://www.biaiku.net/b/111/111622/'+li.a['href']
    detail_page = requests.get(url=detail_url, headers=headers)
    detail_page.encoding = detail_page.apparent_encoding
    detail_page_text = detail_page.text
    detail_soup = BeautifulSoup(detail_page_text, 'lxml')
    div_tag = detail_soup.find('div', id="content_div")
    content = div_tag.text
    fp.write(title+':'+content+'\n')
    print(title,'爬取成功！！！')

注：这个链接是爬取的《一屋暗灯》，爬别的小说要根据网页的结构改一下soup.select的内容