一、了解scrapy架构
二、安装
1.windows
pip3 install scrapy
三、测试
1.查看scrapy对外接口
>>> import scrapy
>>>print(scrapy.all)
>>>[‘version’,
‘version_info’,
‘twisted_version’,
‘Spider’,
‘Request’,
‘FormRequest’,
‘Selector’,
‘Item’,
‘Field’]
2. 创建项目
scrapy startproject myproject [project_dir]
备注:project_dir目录下创建一个Scrapy项目
例:
(project) C:\Desktop\crawl>scrapy startproject boss
New Scrapy project ‘boss’, using template directory ‘c:\virtualenv\project\lib\site-packages\scrapy
templates\project’, created in:
C:\Desktop\crawl\project\boss
You can start your first spider with:
cd boss
scrapy genspider example example.com
3.创建一个spider
分两种方法:
第一种:
手动在spiders文件夹下新建一个.py文件
例:
import scrapy
class QutotesSpider(scrapy.Spider):
name = “quotes”
def start_requests(self):
urls = ['http://quotes.toscrapy.com/page/1/',
'http://qutoes.toscrapy.com/page/2/',
]
for url in urls:
# 必须返回一个可迭代对象
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
pass
第二种:
scrapy genspider quotes quotes.toscrape.com
4.运行爬虫
scrapy crawl myspider