scrapy 学习总结

scrapy入门

创建scrapy爬虫

1.新建项目

scrapy startproject xx

2.初始化爬虫实例

scrapy genspider xx "xxxx.com"

3.运行爬虫

scrapy crawl xx

scrapy 中间件

下载中间件 （DOWNLOADER_MIDDLEWARES）

DOWNLOADER_MIDDLEWARES = {
   'opticsjournalSpider.middlewares.RandomUserAgentMiddleware':10,
   'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
    'scrapy_proxies.RandomProxy': 100,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

ip代理池

当爬虫时遇到ip限制可以采用ip代理池的方式进行爬取，因为当scrapy发送请求(REQUEST)和接收响应(RESPONSE)时都会经过中间件，所以在中间件对其进行设置，就能够自动的按照自己配置好的ip去进行爬虫。
今天我使用的ip代理方法是通过scrapy_proxies来实现的。以下是其使用方法：
安装：pip install scrapy_proxies
安装完成之后，通过设置scrapy实例目录下的setting.py文件进行配置
配置方法如下：

# Retry many times since proxies often fail
RETRY_TIMES = 10
# Retry on most error codes since proxies fail for different reasons
RETRY_HTTP_CODES = [500, 503, 504, 400, 403, 404, 408]

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
    'scrapy_proxies.RandomProxy': 100,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

# Proxy list containing entries like
# http://host1:port
# http://username:password@host2:port
# http://host3:port
# ...
PROXY_LIST = '/path/to/proxy/list.txt'

# Proxy mode
# 0 = Every requests have different proxy
# 1 = Take only one proxy from the list and assign it to every requests
# 2 = Put a custom proxy to use in the settings
PROXY_MODE = 0

# If proxy mode is 2 uncomment this sentence :
#CUSTOM_PROXY = "http://host1:port"

scrapy_proxies github地址：https://github.com/aivarsk/scrapy-proxies