Python爬虫:如何爬取分页数据?
REF: https://www.cnblogs.com/duwangdan/p/6898391.html
爬取对象:
有融网理财项目列表页【履约中】状态下的前10页数据,地址:https://www.yrw.com/products/list-all-all-performance-1-createTimeDesc-1.html
编程思路:
1. 寻找分页地址的变动规律 2. 解析网页,获取内容,放入自定义函数中 3. 调用函数,输出分页内容
详细解说:
1. 首先插入用到的库:BeautifulSoup、requests
1 from bs4 import BeautifulSoup 2 import requests
2. 观察地址的变化规律,可以看到,每切换一页时,后面“createTimeDesc-1.html”中的数字1会随着页面的变动而变动,此时我们将地址存放进列表中,后面用format()和for循环来实现多个地址的存储。
1 urls = ['https://www.yrw.com/products/list-direct-all-performance-1-createTimeDesc-{}.html'.format(str(i)) for i in range(1,11)] 2 print(urls)
此时可以先print下,看地址是否正确,这里range(1,11)是前10个页面的地址。
3. 接下来定义解析函数,参数data的初始值为空。函数内用到的内容和上一篇文章中讲到的相同。先请求urls,然后用BeautifulSoup解析,筛选我们想要的项目标题titles的位置,实现输出。
def get_titles(urls,data = None):
web_data = requests.get(urls)
soup = BeautifulSoup(web_data.text, 'lxml')
titles = soup.select(' h3 > a > em > strong')
for title in titles:
data = {
'title': title.get_text()
}
print(data)
4. 最后,我们来调用函数。
1 for titles in urls: 2 get_titles(titles)
完整代码:
from bs4 import BeautifulSoup
import requests
urls = ['https://www.yrw.com/products/list-direct-all-performance-1-createTimeDesc-{}.html'.format(str(i)) for i in range(1,11)]
# print(urls)
def get_titles(urls,data = None):
web_data = requests.get(urls)
soup = BeautifulSoup(web_data.text, 'lxml')
titles = soup.select(' h3 > a > em > strong')
for title in titles:
data = {
'title': title.get_text()
}
print(data)
for titles in urls:
get_titles(titles)
运行结果(只展示部分):
{'title': '资产融ZT321期'}
{'title': '供应链ZT2923期'}
{'title': '租车融ZT335期'}
{'title': '供应链ZT2922期'}
{'title': '供应链ZT2919期'}