python scrapy框架中如何提升爬虫获取数据效率

仅作参考

修改setting.py中的一些线程参数配置，

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 100

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0
# The download delay setting will honor only one of:
CONCURRENT_REQUESTS_PER_DOMAIN = 100
CONCURRENT_REQUESTS_PER_IP = 100

DOWNLOAD_DELAY默认为3

CONCURRENT_REQUESTS，CONCURRENT_REQUESTS_PER_DOMAIN，CONCURRENT_REQUESTS_PER_IP等默认为16，可以根据自己的任务需求来进行修改配置参数。