python异步爬虫框架_scrapy: Scrapy 是一套基于基于Twisted的异步处理的超级爬虫框架，纯python实现的爬虫框架，用户只需要定制开发几个模块就可以轻松的实现一个爬虫...

Scrapy

Overview

Scrapy is a fast high-level web crawling and web scraping framework, used to

crawl websites and extract structured data from their pages. It can be used for

a wide range of purposes, from data mining to monitoring and automated testing.

Check the Scrapy homepage at https://scrapy.org for more information,

including a list of features.

Requirements

Python 3.5.1+

Works on Linux, Windows, macOS, BSD

Install

The quick way:

pip install scrapy

See the install section in the documentation at

https://docs.scrapy.org/en/latest/intro/install.html for more details.

Documentation

Documentation is available online at https://docs.scrapy.org/ and in the docs

directory.

Releases

Community (blog, twitter, mail list, IRC)

Contributing

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct

(see https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md).

By participating in this project you agree to abide by its terms.

Please report unacceptable behavior to opensource@scrapinghub.com.

Companies using Scrapy

Commercial Support