INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or no...

爬取豆瓣电影top250，出现以下报错：

2018-08-11 22:02:16 [scrapy.core.engine] INFO: Spider opened
2018-08-11 22:02:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-08-11 22:02:16 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/robots.txt> (referer: None)
2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/top250> (referer: None)
2018-08-11 22:02:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not
allowed
2018-08-11 22:02:17 [scrapy.core.engine] INFO: Closing spider (finished)

 

防止反爬机制，伪装user_agent 

【1】打开豆瓣top250 ：  https://movie.douban.com/top250

【2】F12 打开控制台->刷新页面 ->Network->请求头部找到 User-Agent

在scrapy项目中找到settings.py的  USER_AGENT = ' '  （把注释去掉，加以下内容）

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3493.3 Safari/537.36'

 

重新执行即可

scrapy crawl douban_spider

 

转载于:https://www.cnblogs.com/QW-lzm/p/9461375.html