python爬虫学习（第三爬）

爬取小说（用BeautifulSoup爬）

url=http://www.bixiange.me/wxxz/3976/

先来常规操作

url = "http://www.bixiange.me/wxxz/3976/index/1.html"
headers = {
    "user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0"
}
resp = requests.get(url,headers=headers)
resp.encoding = resp.apparent_encoding
html = resp.text
print(html)

观察页面源代码

 可以发现小说都是现在标签<div></div>之间

因此用bf提取div标签   并且指定对应class与id

bf = BeautifulSoup(html)
texts = bf.find_all('div',class_="content",id="mycontent") 
print(texts)

得到的代码还是有<br><br>标签

解决方法：在匹配前提前把<br><br>改成换行

完成代码如下：

import requests
import time
from requests.api import head
from lxml import etree
from bs4 import BeautifulSoup

for a in range(1,10):
    url = "http://www.bixiange.me/wxxz/3976/index/{}.html".format(a)
    headers = {
        "user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0"
    }
    resp = requests.get(url,headers=headers)
    resp.encoding = resp.apparent_encoding
    html = resp.text
    html = html.replace("<br><br>",'\n\n')#在匹配前提前把<br><br>改成换行
    bf = BeautifulSoup(html)
    texts = bf.find_all('div',class_="content",id="mycontent") 
    with  open('sample/novel.txt','a') as f:#写入文件中，文件默认和python文件同一级目录
        f.write(texts[0].text.replace('<br><br>','\n\n'))
    print(a)
    time.sleep(0.2)
f.close()