爬虫实战--爬取百度贴吧

一样先看网址：

根据规律来写代码

import requests

kw = "python"
url_sample = "http://tieba.baidu.com/f?kw="+kw+"&ie=utf-8&pn={}"
headers={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"}


for i in range(1,11):
    url = url_sample.format((i-1)*50)
    re = requests.get(url, headers=headers).content.decode()
    file_path = "{}贴吧第{}页.html".format(kw,i)
    with open(file_path, "w", encoding="utf-8") as f:
        f.write(re)
    f.close()

结果：