Python爬虫学习-反爬虫机制与应对方法

在爬取数据的过程中遇到以下几种反爬手段：
1 .浏览器反爬检测
2 .IP访问限制
对应的解决措施有：
1 .修改hearders。

headers = {
    'authority': 'ditu.amap.com',
    'cache-control': 'max-age=0',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'zh-CN,zh;q=0.9,en;q=0.8,zh-TW;q=0.7',
    'cookie': 'guid=2dcf-1337-74dc-b27b; key=bfe31f4e0fb231d29e1d3ce951e2c780; UM_distinctid=1691af7cd44330-03573b8a5b15a7-1333063-100200-1691af7cd455ff; cna=3qWbFB2RkCcCAXMi8f/QNVwq; CNZZDATA1255626299=343925159-1550935456-%7C1550935456; isg=BJqaOgaPkuSm5B5IstqSUWE460C8yx6lqRRx-qQTQi34FzpRjFobtQJl46Mux5Y9; _uab_collina=155093564321243494443932',
}
demo = requests.get(url,headers=headers)

2 .修改requests中的proxies。

proxies = {

    "http": "http://203.174.112.13:3128"  # 代理ip

}
demo = requests.get(url,proxies=proxies)