创建Scrapy项目
$ scrapy startproject ZhipinSpider
New Scrapy project 'ZhipinSpider', using template directory '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scrapy/templates/project', created in:
/Users/xxxxxxx/PycharmProjects/ZhipinSpider
You can start your first spider with:
cd ZhipinSpider
scrapy genspider example example.com
Scrapy项目分析
- scrapy.cfg:项目的总配置文件,通常无需修改
- ZhipinSpider/:项目Python模块,程序将从此处导入Python代码
Scrapy概览
Scrapy核心组件
调度器:该组件由Scrapy框架实现
下载器:该组件由Scrapy框架实现
蜘蛛:该组件由开发者实现
Pipeline:该组件有开发者实现