python scrapy爬虫遇见301_scrapy 爬虫关闭但实际任务并没完成

2017-10-17 11:48:18 [scrapy.core.engine] INFO: Closing spider (finished)

29957 2017-10-17 11:48:19 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

29958 {'downloader/request_bytes': 2175351,

29959 'downloader/request_count': 6472,

29960 'downloader/request_method_count/GET': 6472,

29961 'downloader/response_bytes': 494012842,

29962 'downloader/response_count': 6472,

29963 'downloader/response_status_count/200': 6419,

29964 'downloader/response_status_count/301': 53,

29965 'dupefilter/filtered': 8,

29966 'finish_reason': 'finished',

29967 'finish_time': datetime.datetime(2017, 10, 17, 11, 48, 19, 59301),

29968 'item_scraped_count': 6417,

29969 'log_count/DEBUG': 12891,

29970 'log_count/ERROR': 1,

29971 &#