scrapy 笔记一

creating a project：

enter "scrapy startproject tutorial” in your console.

This will create directory named tutorial ,it likes following:

tutorial/
    scrapy.cfg            # deploy configuration file

    tutorial/             # project's Python module, you'll import your code from here
        __init__.py

        items.py          # project items definition file

        middlewares.py    # project middlewares file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders
            __init__.py

Make a file named quotes_spider.py under the tutorial/spiders directory.The code likes fllowing:

import scrapy
class QuotesSpider(scrapy.Spider):
    name = “quotes” #the spider’s name, you will use it while run this spider。
    def start_requests(self):  #must return an iterable。
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)
    def parse(self, response): #parse the response，extracting the scraped data。
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

Our class implements scrapy.Spider and defines some attributes and methods。

To run our spider：

type following code into console and run It。

scrapy crawl quotes  #quotes is spider’s name。we defined it at above code。

Notice：

Instead of implementing a start_request() method, you can just define a start_url class attribute with a list of URLs.like this

start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

The parse（） method will be called to handle each of the requests for these urls.

If you run this spider , It can’t stored scraped data just display them .

If you want to saved the scraped data.

First: 

	You will modify the parse() method.We will use the yield Python keyword in the parse() method.like following code:

    	def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
             }

Second : Type following code in your console and run it.

	scrapy crawl quotes -o quotes.json

That will generate an quotes.json file.The file format is JSON.