scrapy使用布隆过滤器

最新推荐文章于 2024-04-23 22:07:03 发布

cscainiao111

最新推荐文章于 2024-04-23 22:07:03 发布

阅读量950

点赞数 1

分类专栏： python 爬虫

本文链接：https://blog.csdn.net/xx117501/article/details/88865977

版权

python 同时被 2 个专栏收录

16 篇文章 1 订阅

订阅专栏

爬虫

10 篇文章 0 订阅

订阅专栏

pip install scrapy-redis-bloomfilter

在settings中这样配置：

# Ensure use this Scheduler
SCHEDULER = "scrapy_redis_bloomfilter.scheduler.Scheduler"

# Ensure all spiders share same duplicates filter through redis
DUPEFILTER_CLASS = "scrapy_redis_bloomfilter.dupefilter.RFPDupeFilter"

# Redis URL
REDIS_URL = 'redis://localhost:6379/0'

# Number of Hash Functions to use, defaults to 6
BLOOMFILTER_HASH_NUMBER = 6

# Redis Memory Bit of Bloomfilter Usage, 30 means 2^30 = 128MB, defaults to 30
BLOOMFILTER_BIT = 30

# Persist
SCHEDULER_PERSIST = True