scrapy学习笔记（一）

一、了解scrapy架构

二、安装
1.windows
pip3 install scrapy
三、测试
1.查看scrapy对外接口
>>> import scrapy
>>>print(scrapy.all)
>>>[‘version’,
‘version_info’,
‘twisted_version’,
‘Spider’,
‘Request’,
‘FormRequest’,
‘Selector’,
‘Item’,
‘Field’]
2. 创建项目
scrapy startproject myproject [project_dir]
备注：project_dir目录下创建一个Scrapy项目
例：
(project) C:\Desktop\crawl>scrapy startproject boss
New Scrapy project ‘boss’, using template directory ‘c:\virtualenv\project\lib\site-packages\scrapy
templates\project’, created in:
C:\Desktop\crawl\project\boss

You can start your first spider with:
cd boss
scrapy genspider example example.com

3.创建一个spider
分两种方法：
第一种：
手动在spiders文件夹下新建一个.py文件
例：
import scrapy

class QutotesSpider(scrapy.Spider):
name = “quotes”

def start_requests(self):
    urls = ['http://quotes.toscrapy.com/page/1/',
            'http://qutoes.toscrapy.com/page/2/',
            ]
    for url in urls:
        # 必须返回一个可迭代对象
        yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):
    pass

第二种：
scrapy genspider quotes quotes.toscrape.com

4.运行爬虫
scrapy crawl myspider