Srapy - Architecture overview【Data flow】

#Data flow

The data flow in Scrapy is controlled by the execution engine, and goes like this:

1. The Engine gets the initial Requests to crawl from the Spider.
2. The Engine schedules the Requests in the Scheduler and asks for the
   next Requests to crawl.
3. The Scheduler returns the next Requests to the Engine.
4. The Engine sends the Requests to the Downloader, passing through the
   Downloader Middleware (requests direction).
5. Once the page finishes downloading the Downloader generates a
   Response (with that page) and sends it to the Engine, passing
   through the Downloader Middleware (response direction).
6. The Engine receives the Response from the Downloader and sends it to
   the Spider for processing, passing through the Spider Middleware
   (input direction).
7. The Spider processes the Response and returns scraped items and new
   Requests (to follow) to the Engine, passing through the Spider
   Middleware (output direction).
8. The Engine sends processed items to Item Pipelines, then send
   processed Requests to the Scheduler and asks for possible next
   Requests to crawl.
9. The process repeats (from step 1) until there are no more requests
   from the Scheduler.

Scrapy的架构初探
https://zhuanlan.zhihu.com/p/21320942