爬虫高级之header表头加Host和Upgrade-Insecure-Requests: 1

在爬取51jb时遇到的问题

1.爬出的数据是重复的

2.数据显示不正确

解决方法

headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
            "Host":"search.51job.com",
            "Upgrade-Insecure_Requests":"1"
        }
url="https://js.51jobcdn.com/in/js/2016/layer/area_array_c.js?20200429"

r = requests.get(url, headers=headers)
print(r.content.decode('gbk'))

加入"Host":"search.51job.com"和 “Upgrade-Insecure_Requests”:“1”