Python---爬虫---存储---(布隆过滤器)BloomFilter(数据去重，断点续爬)

 

pip3 install pybloom_live   

from pybloom_live import BloomFilter
import hashlib
 
Bloom_data = BloomFilter(1000000000,0.01)
md5_qa = hashlib.md5(data.encode('utf8')).hexdigest()
md5_qa = bytes(md5_qa, encoding='utf8')
if md5_qa not in Bloom_data:
    Bloom_data.add(md5_qa)

作用：数据去重，断点续爬。