腾讯招聘爬取10页（练习翻页）

薛定谔的小笼包·

已于 2024-10-12 17:32:46 修改

阅读量226

点赞数 3

文章标签：爬虫 css 前端

于 2024-10-05 13:37:53 首次发布

本文链接：https://blog.csdn.net/D18346/article/details/142713707

版权

#目标：腾讯招聘爬取10页（练习翻页）
#思考：1.爬取10页，我们需要找每一页的url进行请求
#     2.做翻页，就要找每一页不一样的地方
"""
https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1728103888912&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=&attrId=1&keyword=&pageIndex=1&pageSize=10&language=zh-cn&area=cn
https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1728104088733&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=&attrId=1&keyword=&pageIndex=2&pageSize=10&language=zh-cn&area=cn
"""
#       对比发现两个不一样的地方：timestamp（时间戳，实时改变）和pageIndex→当前页数
#       验证发现：timestamp固定对多页爬取没影响，pageIndex才是最重要的
for i in range(1,11):
    url_=f"https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1728103888912&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=&attrId=1&keyword=&pageIndex={i}&pageSize=10&language=zh-cn&area=cn"
#    print(url)#生成每一页url
    print("===================================================================================================")
    import requests
    import json
    headers_={
    "user-agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36 Edg/129.0.0.0"
    }
    response=requests.get(url_,headers=headers_)
    result=json.loads(response.text)
    data_list=result['Data']['Posts']
    for i in data_list:
        title = i['RecruitPostName']
        location = i['LocationName']
        Time = i['LastUpdateTime']
        Responsibility = i['Responsibility'].replace('\r', '').replace('\n', '')
        print(title, location, Time,Responsibility)
        print("===================================================================")