菜鸟学习记录
爬虫遇到的问题:connection error :…Max retries exceeded with url:…Failed to establish a new connection:…
据说是访问太频繁导致的,或者ip被封
在此之前我已经有time.sleep
尝试一:关闭多余链接
get里面加上headers={‘Connection’:‘close’}
结果:没用。报错如之前
尝试二:
get里面再加上verify=False
结果:出现警告InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
消除警告:
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
结果:没用。报错如之前
尝试三:增加链接次数
r_s = requests.session()
r_s.keep_alive = False
#增加重试连接次数
r_s.adapters.DEFAULT_RETRIES = 5
结果:没用。报错如之前
尝试四:使用代理
import requests
s = requests.session()
url = “https://mail.163.com/”
s.proxies = {“https”: “47.100.104.247:8080”, “http”: “36.248.10.47:8080”,