本文详细解析了Scrapy爬虫框架中DOWNLOAD_DELAY设置的作用，它用于控制从同一网站连续下载页面之间的延迟时间，以避免对服务器造成过大压力。默认情况下，Scrapy会使用0.5到1.5倍DOWNLOAD_DELAY之间的随机间隔来调整请求速度。读者将了解到如何通过此设置以及RANDOMIZE_DOWNLOAD_DELAY选项来调整爬取速率。
The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same spider. This can be used to throttle the crawling speed to avoid hitting servers too hard. Decimal numbers are supported. Example:
DOWNLOAD_DELAY = 0.25 # 250 ms of delay
This setting is also affected by the RANDOMIZE_DOWNLOAD_DELAY setting (which is enabled by default). By default, Scrapy doesn’t wait a fixed amount of time between requests, but uses a random interval between 0.5 and 1.5 * DOWNLOAD_DELAY.
You can also change this setting per spider.
563
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
