CrawlSpider捕捉url对应的响应数据的（线程）运行规律。

最新推荐文章于 2022-01-29 12:49:16 发布

babyjustsaidyes

最新推荐文章于 2022-01-29 12:49:16 发布

阅读量235

点赞数

分类专栏： python具体知识点讲解

本文链接：https://blog.csdn.net/weixin_43256057/article/details/90633913

版权

python具体知识点讲解专栏收录该内容

58 篇文章 1 订阅

订阅专栏

class Watch1Spider(CrawlSpider):
    name = 'watch1'
    allowed_domains = ['amazon.cn']
	    start_urls = ['https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&ref=nb_sb_noss']

    rules = (
        # Rule(LinkExtractor(restrict_xpaths='//li[@class="a-normal"]'), callback='parse_item', follow=True),
        Rule(LinkExtractor(restrict_xpaths='//li[@class="a-last"]//a'), callback='next_page', follow=True),

    )

    def next_page(self,response):
        page = response.xpath('//li[@class="a-selected"]//text()').extract_first()
        print('cccc',page)

我这段代码目的是爬取网站的分页，随后把该页的页码给打印出来：‘cccc’,page，我截取最后的内容放到下面，可以看出，我怀疑系统分配64个线程分别来跑这些程序。这些线程争抢资源，有些比较快就爬到最后的第10页时有些还是爬到第7或8页而已，还有被压箱底的，其它线程都爬完才到它开始从1爬到10.

2019-05-27 22:15:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966428&ref=sr_pg_8> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966264&ref=sr_pg_7)
cccc 9
2019-05-27 22:15:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966482&ref=sr_pg_7> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=7&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966353&ref=sr_pg_6)
cccc 8
2019-05-27 22:15:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966486&ref=sr_pg_7> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=7&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966356&ref=sr_pg_6)
cccc 8
2019-05-27 22:16:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966489&ref=sr_pg_8> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966358&ref=sr_pg_7)
cccc 9
2019-05-27 22:16:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966493&ref=sr_pg_8> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966363&ref=sr_pg_7)
cccc 9
2019-05-27 22:16:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965587&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965511&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:07 [scrapy.extensions.logstats] INFO: Crawled 379 pages (at 18 pages/min), scraped 0 items (at 0 items/min)
2019-05-27 22:16:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965684&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965584&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965805&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965663&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965809&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965668&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965813&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965672&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965817&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965676&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965820&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965680&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965984&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965784&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965987&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965787&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965991&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965790&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965995&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965794&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965999&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965798&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966003&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965802&ref=sr_pg_8)
cccc 10
2019-05-27 22:16:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966554&ref=sr_pg_8> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966482&ref=sr_pg_7)
cccc 9
2019-05-27 22:17:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966557&ref=sr_pg_8> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966486&ref=sr_pg_7)
cccc 9
2019-05-27 22:17:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966177&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965968&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966180&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965973&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:07 [scrapy.extensions.logstats] INFO: Crawled 395 pages (at 16 pages/min), scraped 0 items (at 0 items/min)
2019-05-27 22:17:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966183&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965975&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966187&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558965979&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966320&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966141&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966324&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966145&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966328&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966148&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966331&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966152&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966336&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966156&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966340&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966160&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966344&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966164&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966347&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966168&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966349&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966172&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966431&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966268&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966435&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966272&ref=sr_pg_8)
cccc 10
2019-05-27 22:17:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966439&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966277&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966441&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966279&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966444&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966283&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966449&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966286&ref=sr_pg_8)
2019-05-27 22:18:07 [scrapy.extensions.logstats] INFO: Crawled 412 pages (at 17 pages/min), scraped 0 items (at 0 items/min)
cccc 10
2019-05-27 22:18:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966452&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966290&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966455&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966293&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966459&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966297&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966464&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966300&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966468&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966304&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966472&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966308&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966475&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966312&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966478&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966316&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966496&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966367&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966499&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966371&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966502&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966375&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966506&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966379&ref=sr_pg_8)
cccc 10
2019-05-27 22:18:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966510&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966382&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966514&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966386&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966516&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966390&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:07 [scrapy.extensions.logstats] INFO: Crawled 427 pages (at 15 pages/min), scraped 0 items (at 0 items/min)
2019-05-27 22:19:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966519&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966394&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966522&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966397&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966526&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966401&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966530&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966404&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966533&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966408&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966537&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966413&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966540&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966417&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966544&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966420&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966547&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966423&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966551&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966428&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966559&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966489&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966563&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966493&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966616&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966554&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966620&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966557&ref=sr_pg_8)
cccc 10
2019-05-27 22:19:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&ref=nb_sb_noss> (referer: None)
2019-05-27 22:20:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=2&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966795&ref=sr_pg_1> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&ref=nb_sb_noss)
cccc 2
2019-05-27 22:20:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=3&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966799&ref=sr_pg_2> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=2&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966795&ref=sr_pg_1)
cccc 3
2019-05-27 22:20:07 [scrapy.extensions.logstats] INFO: Crawled 444 pages (at 17 pages/min), scraped 0 items (at 0 items/min)
2019-05-27 22:20:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=4&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966804&ref=sr_pg_3> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=3&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966799&ref=sr_pg_2)
cccc 4
2019-05-27 22:20:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=5&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966808&ref=sr_pg_4> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=4&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966804&ref=sr_pg_3)
cccc 5
2019-05-27 22:20:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=6&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966812&ref=sr_pg_5> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=5&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966808&ref=sr_pg_4)
cccc 6
2019-05-27 22:20:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=7&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966816&ref=sr_pg_6> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=6&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966812&ref=sr_pg_5)
cccc 7
2019-05-27 22:20:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966820&ref=sr_pg_7> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=7&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966816&ref=sr_pg_6)
cccc 8
2019-05-27 22:20:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966824&ref=sr_pg_8> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=8&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966820&ref=sr_pg_7)
cccc 9
2019-05-27 22:20:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=10&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966828&ref=sr_pg_9> (referer: https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&lo=visual_grid&page=9&__mk_zh_CN=%E4%BA%9A%E9%A9%AC%E9%80%8A%E7%BD%91%E7%AB%99&qid=1558966824&ref=sr_pg_8)
cccc 10

后来我把上面的代码只改动了follow=True为follow=False，然后它就只打印一个cccc 2，我惊讶之余又把follow=False改回follow=True，结果如下！惊呆我，说好的64个线程呢？我没改动代码其它地方啊（一万个懵逼）。

python@ubuntu:~/Desktop/python_study/amazon$ scrapy crawl watch1 --nolog
cccc 2
cccc 3
cccc 4
cccc 5
cccc 6
cccc 7
cccc 8
cccc 9
cccc 10

babyjustsaidyes

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
CrawlSpider捕捉url对应的响应数据的（线程）运行规律。

class Watch1Spider(CrawlSpider): name = 'watch1' allowed_domains = ['amazon.cn'] start_urls = ['https://www.amazon.cn/s?k=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&__mk_zh_CN=%E4%BA%9A%E9%...
复制链接

扫一扫

专栏目录