Scrapy 笔记本三（Scrapy shell）

Scrapy shell

Scrapy shell is a shell like unit shell, but you can run your scraping code very quickly without having to run the spider.You can debug your code in it.

You can testing your XPath and Css expressions conveniently。 

Configuration shell。

I recommend you install Ipython。If you installed IPython，Scrapy shell will use it。

Launch the shell.

To launch the Scrapy shell you can use the shell command like this.

scrapy shell <url>

The url is you want to scrape.it can be local files.like this scrapy shell ./paht1/path2/file.html.

Available Scrapy objects:

You can use following objects in your testing.

Those objects are:

crawler - the current crawler object.

spider - the Spider which is known to handle the URL, or a Spider object if there is no spider found for the current URL.

request - a Request object of the last fetched page.

response() - Response object containing the last fetched page.

settings - the current Scrapy setting.