scrapy学习笔记（一）

一、安装scrapy（pip  install  scrapy）

 二、在python  shell中导入scrapy

    Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
                  >>> import scrapy
                >>> scrapy.version_info
               (1, 7, 3)
               >>> 

三、在dos下创建第一个爬虫项目example

C:\Users\Administrator>scrapy
Scrapy 1.7.3 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

C:\Users\Administrator>scrapy  startproject  example
New Scrapy project 'example', using template directory 'd:\python\python37\lib\s
ite-packages\scrapy\templates\project', created in:
    C:\Users\Administrator\example

You can start your first spider with:
    cd example
    scrapy genspider example example.com

四、分析目标网站源码http://books.toscrape.com/

五、第一个爬虫存到 C:\Users\Administrator\example\example\spiders\book_spider.py

# 
import scrapy
class BooksSpider(scrapy.Spider):
    # 每一个爬虫的唯一标识
     name = "books"

     # 定义爬虫爬取的起始点，起始点可以是多个，这里只有一个
     start_urls = ['http://books.toscrape.com/']
     def parse(self, response):
         # 提取数据
         # 每一本书的信息在<article class="product_pod">中，我们使用
         # css()方法找到所有这样的article 元素，并依次迭代
         for book in response.css('article.product_pod'):
             # 书名信息在article > h3 > a 元素的title属性里
             # 例如: <a title="A Light in the Attic">A Light in the ...</a>
             name = book.xpath('./h3/a/@title').extract_first()

             # 书价信息在 <p class="price_color">的TEXT中。
             # 例如: <p class="price_color">￡51.77</p>
             price = book.css('p.price_color::text').extract_first()
             yield {
                 'name': name,
                 'price': price,
                 } 
         # 提取链接
         # 下一页的url 在ul.pager > li.next > a 里面
         # 例如: <li class="next"><a href="catalogue/page-2.html">next</a></li>
         next_url = response.css('ul.pager li.next a::attr(href)').extract_first()
         if next_url:
             # 如果找到下一页的URL，得到绝对路径，构造新的Request 对象
             next_url = response.urljoin(next_url)
             yield scrapy.Request(next_url, callback=self.parse)
六、到目录 C:\Users\Administrator\example下运行scrapy crawl books -o books.csv

C:\Users\Administrator\example>scrapy crawl books -o books.csv

生成如下books.csv文件

name	price
A Light in the Attic	£51.77
Tipping the Velvet	£53.74
Soumission	£50.10
Sharp Objects	£47.82
Sapiens: A Brief History of Humankind	£54.23
The Requiem Red	£22.65
The Dirty Little Secrets of Getting Your Dream Job	£33.34