问题:scrapy shell 请求页面时返回503 Service Unavailablec错误
2018-07-08 15:30:23 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.xicidaili.com/nn/2>
(failed 3 times): 503 Service Unavailable
2018-07-08 15:30:23 [scrapy.core.engine] DEBUG: Crawled (503) <GET http://www.xicidaili.com/nn/2> (referer: None)
2018-07-08 15:30:24 [traitlets] DEBUG: Using default logger
2018-07-08 15:30:24 [traitlets] DEBUG: Using default logger
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x000000F7A1053C18>
[s] item {}
[s] request <GET http://www.xicidaili.com/nn/2>
[s] response <503 http://www.xicidaili.com/nn/2>
[s] settings <scrapy.settings.Settings object at 0x000000F7A10539E8>
[s] spider <DefaultSpider 'default' at 0xf7a12d7518>
补充:常见可能被网站识别返回错误
1、CAPTCHApages (captcha,验证码)
2、Unusualcontent delivery delay (响应时间、速度变慢了)
3、Frequentresponse with HTTP404,301 or 50x errors
(1)301 MovedTemporarily
(2)401unauthorized
(3)403forbidden (aAatch处理的)