scrapy爬虫工具

为什么80%的码农都做不了架构师？>>>   

python 爬虫工具scrapy安装

标签（空格分隔）： scrapy pip python

---

1. 安装

 sudo pip install Scrapy

2.如果提示pip： command not found 需要安装pip 然后再安装scrapy

 sudo easy_install pip 

3.使用scrapy 新建一个项目

 scrapy startproject test

4.如果创建报错 ImportError: cannot import name xmlrpc_client 解决方法

sudo rm -rf /Library/Python/2.7/site-packages/six*

sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six*

sudo pip install six

示例代码：在test/spiders目录下新建quotes_spider.py文件

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

然后执行命令运行脚本 scrapy crawl quotes

相关文档：https://oner-wv.gitbooks.io/scrapy_zh/content/

转载于:https://my.oschina.net/ives/blog/853330