![](https://img-blog.csdnimg.cn/20201014180756754.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
爬虫笔记
RedPintings
记录工作,共同成长
展开
-
scrapy 各组件执行顺序
Scrapy Engine(引擎): 负责Spider、ItemPipeline、Downloader、Scheduler中间的通讯,信号、数据传递等。 Scheduler(调度器): 它负责接受引擎发送过来的Request请求,并按照一定的方式进行整理排列,入队,当引擎需要时,交还给引擎。 Downloader(下载器):负责下载Scrapy Engine(引擎)发送...原创 2018-07-20 10:04:22 · 3651 阅读 · 0 评论 -
Scrapy settings.py配置文件详解
# -*- coding: utf-8 -*- # Scrapy settings for demo1 project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the d...转载 2018-08-17 11:17:44 · 299 阅读 · 0 评论 -
scrapy pipline 将数据存入不同的数据库 下载图片
# -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html from s...原创 2018-10-31 17:41:22 · 925 阅读 · 0 评论