需求:
将虎扑新闻前五页内容保存到本地
1.导入requests模块
import requests
定义请求头
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
}
分析:分页操作:核心->找规律
- 第一页URL:https://voice.hupu.com/
- 第二页URL:https://voice.hupu.com/news?category=all&page=2
- 第三页URL:https://voice.hupu.com/news?category=all&page=3
- 第四页URL:https://voice.hupu.com/news?category=all&page=4
- 第五页URL:https://voice.hupu.com/news?category=all&page=5
定义基础url
base_url = 'https://voice.hupu.com/news'
定义参数:
params = {
'categor':'all'
}
#使用循环动态改变page参数的值
for i in range(1,6):
params['page'] = i
2.发起请求,接收响应
response = requests.get(url=base_url,params=params,headers = headers)
# print(response.url)
URL:统一资源定位符
一个URL只能对应一个页面,但是一个页面可以有多个URL对应
保存文件
with open('hupu{}.html'.format(i),'w',encoding='utf-8') as fp: # 分页
fp.write(response.text)
最终代码:
import requests
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
}
base_url = 'https://voice.hupu.com/news'
params = {
'categor':'all'
}
for i in range(1,6):
params['page'] = i
response = requests.get(url=base_url,params=params,headers = headers)
# print(response.url)
with open('hupu{}.html'.format(i),'w',encoding='utf-8') as fp: # 分页
fp.write(response.text)