前言
在用 PySpider 爬取 https 开头的网站的时候遇到了 HTTP 599: SSL certificate problem: self signed certificate in certificate chain 的错误。
[E 160329 11:32:22 base_handler:194] HTTP 599: SSL certificate problem: self signed certificate in certificate chain
Traceback (most recent call last):
File "D:\Python27\lib\site-packages\pyspider\libs\base_handler.py", line 187, in run_task
result = self._run_task(task, response)
File "D:\Python27\lib\site-packages\pyspider\libs\base_handler.py", line 166, in _run_task
response.raise_for_status()
File "D:\Python27\lib\site-packages\pyspider\libs\response.py", line 183, in raise_for_status
raise http_error
HTTPError: HTTP 599: SSL certificate problem: self signed certificate in certificate chain
错误原因
这个错误会发生在请求 https 开头的网址,SSL 验证错误,证书有误。
报错如下
解决方案
最简单的解决方法是:
在 crawl 方法中加入忽略证书验证的参数,validate_cert=False,即
1 | self.crawl(url, callback=method_name, validate_cert=False) |
以上方法基本可以解决问题。
如果还是出错,重新启动一遍pyspider