爬取百度指数行业排行

页面分析

https://insight.baidu.com/base/search/rank/list?pageSize=20&source=0&toFixed=1&filterType=1&dateType=20200511~20200517&dimensionid=37&rateType=1000

Get请求：
parsed:
页面显示数量
pageSize: 20  
source: 0
toFixed: 1
filterType: 1
时间段
dateType: 20200511~20200517
商品类型
dimensionid: 37
rateType: 1000
非必要参数
filterNodes: 
callback: _jsonpsutwe93bymi

请求的url:
https://insight.baidu.com/base/search/rank/list?pageSize=20&dateType=20200511~20200517&dimensionid=37

parsed: 页面显示数量，时间段，商品类型

dataType:可单日期：20100510，可时间段：20200511~20200517

代码：

import requests

from fake_useragent import UserAgent

url='https://insight.baidu.com/base/search/rank/list?'
headers={
    'User-agent':str(UserAgent(verify_ssl=False).random),
    'Cookie':'BAIDUID=14314D9D1182C0F3F18E4D32FB781398:FG=1; BIDUPSID=14314D9D1182C0F3F18E4D32FB781398; PSTM=1589471352; BDUSS=1dPaWxDaFI5akdiSlVpaUN5akR3TmZWRXhNM1V1eGF1V0J3OEhjaklCZ1NqZTFlRVFBQUFBJCQAAAAAABAAAAEAAACfVjf7YXNpbW92MDUxOAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIAxl4SAMZeb; ZD_ENTRY=empty; bdindexid=4gqnrgiqkck0gcbrufvjkka5o2; RT="z=1&dm=baidu.com&si=avs4x0yocsr&ss=kapk89x0&sl=5&tt=3pk&bcn=https%3A%2F%2Ffclog.baidu.com%2Flog%2Fweirwood%3Ftype%3Dperf"',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'zh-CN,zh;q=0.9',
}
parsed={
        # 每页显示的数量
        'pageSize':'20',
        # 日期范围  可单日
        'dateType':'20200511~20200517',
        # 类型id   37：化妆品
        'dimensionid':'37',
}
req=requests.get(url,headers=headers,params=parsed).json()
# 类型id_37 化妆品
req_type=req.get('data').get('dimensionid')
# 前20的品牌
req_results=req.get('data').get('results').get('current')
print(req_type)
print(req_results)