舍友想看小说,但是嫌弃网页版有广告,就用python爬一下
先导入库
import requests
from bs4 import BeautifulSoup
输入要请求的网页,进行UA伪装,为了避免出现乱码,设置一下encoding
url = ("http://www.biaiku.net/b/111/111622/")
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
}
res = requests.get(url=url,headers=headers)
res.encoding = "gbk"
response = res.text
新建一个.txt文件,用于存储结果。
fp = open('./yiwuandeng.txt','w',encoding='utf-8')
用bs爬取每一章的链接,并对每一章的链接进行requests请求
soup = BeautifulSoup(response,'lxml')
src_list = soup.select('.add-box > ul > li')
for li in src_list:
title = li.a.string
detail_url = 'http://www.biaiku.net/b/111/111622/'+li.a['href']
detail_page = requests.get(url=detail_url, headers=headers)
detail_page.encoding = detail_page.apparent_encoding
detail_page_text = detail_page.text
detail_soup = BeautifulSoup(detail_page_text, 'lxml')
div_tag = detail_soup.find('div', id="content_div")
content = div_tag.text
fp.write(title+':'+content+'\n')
print(title,'爬取成功!!!')
注:这个链接是爬取的《一屋暗灯》,爬别的小说要根据网页的结构改一下soup.select的内容