问题描述:
在爬取百度贴吧的内容时候,一次只爬取一页的时候,没有问题,但是使用for i in range 一次爬取多个页面的时候出现以下问题
代码如下,红框内代码报错:
字符拼接没有问题,两次输出就是相同
https://tieba.baidu.com/f?kw=诗词吧&ie=utf-8&pn=200
https://tieba.baidu.com/f?kw=诗词吧&ie=utf-8&pn=200
想到一个问题,当浏览器中的url地址存在中文时候,在网页跳转时候会自动进行给它url转码,进行测试,查看是否是这个原因
修改后代码如下:
from urllib import request, parse, error
base_url = "https://tieba.baidu.com/f?kw={}&ie=utf-8&pn={}"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
}
def allow(minpage,maxpage, string2):
maxnum = maxpage * 50 + 50 + minpage;
maxnum = maxpage * 50 + minpage;
print(minpage,maxnum)
try:
for i in range(minpage, maxnum, 50):
string1 = base_url.format(parse.quote(string2), i)
url = request.Request(url=string1, headers=headers)
res = request.urlopen(url)
res1 = res.read().decode("utf-8")
with open(string2 + str(i) + ".html", "w", encoding="utf-8") as f:
f.write(res1)
print(string2 + str(i) + ".html")
except error.HTTPError as e:
print(e.code)
except error.URLError as e:
print(e.reason)
else:
print("成功")
string2 = input("请输入贴吧名:")
minpage = int(input("请输入页数开始值:"))
maxpage = int(input("请输入下载页数:"))
print(minpage, maxpage, string2)
allow(minpage, maxpage, string2)
修改代码:
string1 = base_url.format(parse.quote(string2), i)
运行成功,截图如下:
分析:
Url编码
urllib.parse.quote(string) 将字符传进行编码
urllib.parse.unquote(string) 将字符传进行解码