【Python】使用Bloomfilter去重

环境

• python3.5
• pip3 install bitarray-0.8.1-cp35-cp35m-win_amd64.whl
• pip3 install pybloom_live
• 参考：https://github.com/jaybaird/python-bloomfilter

---

使用

• ScalableBloomFilter

from pybloom_live import ScalableBloomFilter

sbf = ScalableBloomFilter(initial_capacity=100, error_rate=0.001, mode=ScalableBloomFilter.LARGE_SET_GROWTH)

url = "www.baidu.com"
url2 = "www.douban,com"

sbf.add(url)

print(url in sbf)   # True
print(url2 in sbf)  # False

• BloomFilter

from pybloom_live import BloomFilter

bf = BloomFilter(capacity=1000)

bf.add("www.baidu.com")

print("www.baidu.com" in bf)   # True
print("www.douban.com" in bf)  # False