爬虫实战-掌上高考网实战

最新推荐文章于 2024-08-14 15:24:20 发布

XSJ.寒冰之雪

最新推荐文章于 2024-08-14 15:24:20 发布

阅读量619

点赞数 6

文章标签：爬虫

本文链接：https://blog.csdn.net/fasdaegaw/article/details/140782336

版权

'''
1.确定需求：
    爬取什么数据
    爬取大学名称
2.找到数据源地址
    数据在哪个链接中
    https://api.zjzw.cn/web/api/?keyword=&page=1&province_id=&ranktype=&request_type=1&size=20&top_school_id=[3703,2461,659,3117,597,1724]&type=&uri=apidata/api/gkv3/school/lists&signsafe=b6a400d702222a65bbc2fbd7d848933e
3.复制cURL(base)到https://curlconverter.com/拿到爬虫代码
4.解析出需要的数据
5.存储数据
    
'''
import requests
import json


headers = {
    "accept": "application/json, text/plain, */*",
    "accept-language": "zh-CN,zh;q=0.9",
    "content-type": "application/json",
    "origin": "https://www.gaokao.cn",
    "priority": "u=1, i",
    "referer": "https://www.gaokao.cn/",
    "sec-ch-ua": "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Windows\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "cross-site",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
}
url = "https://api.zjzw.cn/web/api/"
params = {
    "keyword": "",
    "page": "1",
    "province_id": "",
    "ranktype": "",
    "request_type": "1",
    "size": "20",
    "top_school_id": "\\[3703,2461,659,3117,597,1724\\]",
    "type": "",
    "uri": "apidata/api/gkv3/school/lists",
    "signsafe": "b6a400d702222a65bbc2fbd7d848933e"
}
data = {
    "keyword": "",
    "page": 1,
    "province_id": "",
    "ranktype": "",
    "request_type": 1,
    "signsafe": "b6a400d702222a65bbc2fbd7d848933e",
    "size": 20,
    "top_school_id": "[3703,2461,659,3117,597,1724]",
    "type": "",
    "uri": "apidata/api/gkv3/school/lists"
}
a=[]
data = json.dumps(data, separators=(',', ':'))
response = requests.post(url, headers=headers, params=params, data=data).text #str
res=eval(response)
for i in res['data']['item']:
    a.append(i['name'])
    
    
with open('存储的文件名称.txt','w') as f:  #encoding='utf-8'
    for i in a:
        f.write(i+'\n')
print('程序运行结束')

配套视频https://www.bilibili.com/video/BV1Vx4y1s73E/?share_source=copy_web&vd_source=9a94d4db7fa8a361a15f7c2717f048fd