布隆过滤器优化算法double—hashing论文原文（二）

引言

本论文来自leveldb源码中bloom.cc下布隆过滤器实现的代码注释中推荐的论文。论证了一种布隆过滤器的优化方式，带有详细的证明。是不可多得的好文章。其中的许多公式会对布隆过滤器相关的文章有所帮助。所以博主将原文和其中的海量公式转化为了可编辑的markdown和Latex，方便大家引用。如有转载，望注明论文出处和本文出处，谢谢！

论文原文出处：论文原文

本文作者：csdn账号，个人空间 - AcWing

受平台字数限制，只好将论文分成多段，望理解

上接布隆过滤器优化算法double—hashing论文原文（一）

6. Rate of Convergence

In the previous sections, we identified a broad class of non-standard Bloom filter schemes that have the same asymptotic false positive probability as a standard Bloom filter. Unfortunately, these results are not particularly compelling in settings with very limited space, since it is reasonable to think that the rate of convergence in the conclusion of Theorem 4.1 might be fairly slow. This problem is compounded by the fact that Bloom filters are particularly attractive in applications where space is extremely limited (for example, see [3]), since they give a fairly small error rate while using only a small constant number of bits per item. Thus, with these applications in mind we provide a detailed analysis of the rate of convergence in Theorem 4.1. Before proceeding with the results, we introduce some useful notation. For functions $f(n)$ and

$g(n)$ , we use $f(n)\sim g(n)$ to denote that ${\lim }_{n\to \infty }f(n)/g(n)=1$ .Similarly, we use $f(n)\lesssim g(n)$ to denote that lim: ${\sup }_{n\to \infty }f(n)/g(n)\le 1$ and $f(n)\gtrsim g(n)$ to denote that $\lim {\inf }_{n\to \infty }f(n)/g(n)\ge 1$ We are now ready to prove the main technical result of this section

Theorem 6.1.Under the same conditions as in Theorem 4.1
$$\mathbf{Pr}(\mathcal{F})-{\left(1-{\mathrm{e}}^{-\lambda /k}\right)}^k\sim nϵ(n),$$
where
$$\begin{array}{rl}ϵ(n)& \stackrel{def}{=}\left(\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k\\ & +\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\\ & +\sum _{j=2}^k\mathbf{Pr}(\Vert C(x)\Vert =j){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-j}.\end{array}$$
Remark. This result is intuitively pleasing, since it says that the portion of the false positive probability represented by the asymptotic error term is essentially the probability that $\Vert C(x,z)\Vert >1$ for exactly one $x\in S$ and Z 's other $k-\Vert C(x,z)\Vert$ hash locations are hit by the other elements of $S$ in the “asymptotic" filter (that is, in the limit as $n-1\to \infty$ ),which happens with probability $(1-{\mathrm{e}}^{-\lambda /k}{)}^{k-\Vert C(x,z)\Vert }$ .(This almost follows from Theorem 4.1. The difference is that now $z$ has only $k-\Vert C(x,z)\Vert$ hash locations, while the elements of S − { x } S-\{x\} S−{x} each have $k$ hash locations; however, it should be clear from the proof of Theorem 4.1 that the limiting false positive probability in this case is $(1-{\mathrm{e}}^{-\lambda /k}{)}^{k-\Vert C(x,z)\Vert }$ )

Proof. We begin along the same lines as in the proof of Theorem 4.1. First, we adopt the convention introduced there that allows us to associate the elements of $H(z)$ (with multiplicity) with the elements of $[k]$ . Next, for $i\in [k]$ and $x\in S$ , we define $X_i(x)=1$ if $i\in C(x)$ and $X_i(x)=0$ otherwise, $X_i\stackrel{\mathrm{def}}{=}{\sum }_{x\in S}{X}_i(x)$ ,and $X\stackrel{\mathrm{def}}{=}(X_0,\dots ,X_{k-1})$ .Finally, we define $P\stackrel{\mathrm{def}}{=}(P_0,\dots ,P_{k-1})$ to be a vector of $k$ independent $\mathrm{Po}(\lambda /k)$ random variables Define
$$\begin{array}{rl}& f(n)\stackrel{\mathrm{def}}{=}\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\\ & g_i(n)\stackrel{\mathrm{def}}{=}\mathbf{Pr}(i\in C(x),\Vert C(x)\Vert =1)-\frac{\lambda }{kn}\mathrm{for}i\in [k]\\ & h_T(n)\stackrel{\mathrm{def}}{=}\mathbf{Pr}(C(x)=f_{H(z)}^{-1}(T))\mathrm{for}T\subseteq [k]:|T|>1,\end{array}$$
and note that they are all $o\left(1/n\right)$ by the hypotheses of the lemma.For $T\subseteq [k]$ ，we may now

---

write

P r ( ⋂ i ∈ T X i = 0 ) = ∏ x ∈ S P r ( { i ∈ [ k ] : i ∈ C ( x ) } ⊆ T ‾ ) = ( P r ( ∥ C ( x ) ∥ = 0 ) + ∑ i ∈ T P r ( i ∈ C ( x ) , ∥ C ( x ) ∥ = 1 ) + ∑ T ≤ T : ∣ T ′ ∣ > 1 P r ( C ( x ) = f H ( z ) − 1 ( T ′ ) ) = ( 1 − λ ∣ T ∣ k n + f ( n ) + ∑ i ∈ T g i ( n ) + ∑ T ′ ⊆ T ∣ T ′ ∣ > 1 h T ′ ( n ) ) n ∼ exp ⁡ [ − λ ∣ T ∣ k + n f ( n ) + ∑ i ∈ T n g i ( n ) + ∑ T ′ ⊆ T : ∣ T ′ ∣ > 1 n h g i ′ ( n ) ] = e − λ ∣ T ∣ k ( exp ⁡ [ n f ( n ) + ∑ i ∈ T n g i ( n ) + ∑ T ′ ⊆ T : ∣ T ′ ∣ > 1 n h g i ′ ( n ) ] ) ∼ e − λ ∣ T ∣ k ( 1 + n f ( n ) + ∑ i ∈ T n g i ( n ) + ∑ T ′ ⊆ T : ∣ T ′ ∣ > 1 n h g i ′ ( n ) ) , \begin{aligned}\mathbf{Pr}\left(\bigcap_{i\in T}X_i=0\right)&=\prod_{x\in S}\mathbf{Pr}\left(\{i\in[k]:i\in C(x)\}\subseteq\overline{T}\right)\\&=\left(\mathbf{Pr}(\|C(x)\|=0)+\sum_{i\in T}\mathbf{Pr}(i\in C(x),\|C(x)\|=1)\right.\\&+\sum_{T\leq T:|T^{\prime}|>1}\mathbf{Pr}(C(x)=f_{H(z)}^{-1}(T^{\prime}))\\&=\left(1-\frac{\lambda|T|}{kn}+f(n)+\sum_{i\in T}g_i(n)+\sum_{T^{\prime}\subseteq T|T^{\prime}|>1}h_{T^{\prime}}(n)\right)^n\\&\sim\exp\left[-\frac{\lambda|T|}k+nf(n)+\sum_{i\in T}ng_i(n)+\sum_{T^{\prime}\subseteq T:|T^{\prime}|>1}nhg_i^{\prime}(n)\right]\\&=\mathrm{e}^{-\frac{\lambda|T|}k}\left(\exp\left[nf(n)+\sum_{i\in T}ng_i(n)+\sum_{T^{\prime}\subseteq T:|T^{\prime}|>1}nhg_i^{\prime}(n)\right]\right)\\&\sim\mathrm{e}^{-\frac{\lambda|T|}k}\left(1+nf(n)+\sum_{i\in T}ng_i(n)+\sum_{T^{\prime}\subseteq T:|T^{\prime}|>1}nhg_i^{\prime}(n)\right),\end{aligned} Pr(i∈T⋂​Xi​=0)​=x∈S∏​Pr({i∈[k]:i∈C(x)}⊆T)=(Pr(∥C(x)∥=0)+i∈T∑​Pr(i∈C(x),∥C(x)∥=1)+T≤T:∣T′∣>1∑​Pr(C(x)=fH(z)−1​(T′))= ​1−knλ∣T∣​+f(n)+i∈T∑​gi​(n)+T′⊆T∣T′∣>1∑​hT′​(n) ​n∼exp ​−kλ∣T∣​+nf(n)+i∈T∑​ngi​(n)+T′⊆T:∣T′∣>1∑​nhgi′​(n) ​=e−kλ∣T∣​ ​exp ​nf(n)+i∈T∑​ngi​(n)+T′⊆T:∣T′∣>1∑​nhgi′​(n) ​ ​∼e−kλ∣T∣​ ​1+nf(n)+i∈T∑​ngi​(n)+T′⊆T:∣T′∣>1∑​nhgi′​(n) ​,​
where the first two steps are obvious, the third step follows from the definition of $f$ ,the $y_i$ 's, and

the $h{T}^{\prime }$ 's, and the fourth and sixth steps follow from the assumption that all of those functions are $o\left(1/n\right)$ (since ${\mathrm{e}}^{t(n)}\sim 1+t(n)$ if $t(n)=o(1)$

Thus, the inclusion/exclusion principle implies that

$$\begin{array}{rl}(\mathcal{F})-\mathbf{Pr}(\forall i:P_i>0)& =-\left(\mathbf{Pr}(\exists i:X_i=0)-\mathbf{Pr}(\exists i:P_i=0)\right)\\ & =-\sum _{\varnothing \subset T\subseteq [k]}(-1{)}^{|T|+1}\left(\mathbf{Pr}\left(\bigcap _{i\in T}{X}_i=0\right)-\mathbf{Pr}\left(\bigcap _{i\in T}{P}_i=0\right)\right)\\ & =\sum _{\varnothing \subset T\subseteq [k]}(-1{)}^{|T|}\left(\mathbf{Pr}\left(\bigcap _{i\in T}{X}_i=0\right)-{\mathrm{e}}^{-\frac{\lambda |T|}{k}}\right)\\ & \sim n\sum _{\varnothing \subset T\subseteq [k]}(-1{)}^{|T|}{\mathrm{e}}^{-\frac{\Delta |T|}{k}}\left(f(n)+\sum _{i\in T}{g}_i(n)+\sum _{T^{\prime }\subseteq \overline{T}:[T^{\prime }]>1}{h}_{T^{\prime }}(n)\right).\end{array}$$

---

Toevaluatethe sum on thelast line,we write

$$\begin{array}{rl}M& \stackrel{\mathrm{def}}{=}\sum _{\varnothing \subset T\subseteq [k]}(-1{)}^{|T|}{\mathrm{e}}^{-\frac{\lambda |T|}{k}}\left(f(n)+\sum _{i\in \overline{T}}{g}_i(n)+\sum _{T^{\prime }\subseteq \overline{T}:|T^{\prime }|>1}{h}_{T^{\prime }}(n)\right)\\ & =\sum _{j=1}^k{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\sum _{T\subseteq [k]:[T|=j}f(n)\\ & +\sum _{j=1}^k{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\sum _{T\subseteq [k]:|T|=j}\sum _{i\in \overline{T}}{g}_i(n)\\ & +\sum _{j=1}^k{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\sum _{T\subseteq [k]:|T|=j}\sum _{T\subseteq \overline{T}:|T^{\prime }|>1}{h}_{T^{\prime }}(n),\end{array}$$

and evaluate each term separately. First, we compute

$$\begin{array}{rl}\sum _{j=1}^k{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\sum _{T\subseteq [k]:[T]=j}f(n)& =f(n)\sum _{j=1}^k\left(\genfrac{}{}{0ex}{}{k}{j}\right){\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\\ & =\left(\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\right)\left({\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k-1\right)\end{array}$$

Next, we see that

∑ j = 1 k ( − e − λ k ) j ∑ T ⊆ [ k ] : [ T ] = j ∑ i ∈ T ‾ g i ( n ) = ∑ j = 1 k ( − e − λ k ) j ∑ i ∈ [ k ] g i ( n ) ∣   { T ⊆ [ k ]   :   ∣ T ∣ = j , i ∉ T } = ( ∑ i ∈ [ k ] g i ( n ) ) ∑ j = 1 k ( k − 1 j ) ( − e − λ k ) j = ( ∑ i ∈ [ k ] g i ( n ) ) ( ( 1 − e − λ k ) k − 1 − 1 ) = ( P r ( ∥ C ( x ) ∥ = 1 ) − λ n ) ( ( 1 − e − λ k ) k − 1 − 1 ) , \begin{aligned}\sum_{j=1}^{k}\left(-\mathrm{e}^{-\frac{\lambda}{k}}\right)^{j}\sum_{T\subseteq[k]:[T]=j}\sum_{i\in\overline{T}}g_{i}(n)&=\sum_{j=1}^{k}\left(-\mathrm{e}^{-\frac{\lambda}{k}}\right)^{j}\sum_{i\in[k]}g_{i}(n)|\:\{T\subseteq[k]\::\:|T|=j,i\not\in T\}\\&=\left(\sum_{i\in[k]}g_{i}(n)\right)\sum_{j=1}^{k}\binom{k-1}{j}\left(-\mathrm{e}^{-\frac{\lambda}{k}}\right)^{j}\\&=\left(\sum_{i\in[k]}g_{i}(n)\right)\left(\left(1-\mathrm{e}^{-\frac{\lambda}{k}}\right)^{k-1}-1\right)\\&=\left(\mathbf{Pr}(\|C(x)\|=1)-\frac{\lambda}{n}\right)\left(\left(1-\mathrm{e}^{-\frac{\lambda}{k}}\right)^{k-1}-1\right),\end{aligned} j=1∑k​(−e−kλ​)jT⊆[k]:[T]=j∑​i∈T∑​gi​(n)​=j=1∑k​(−e−kλ​)ji∈[k]∑​gi​(n)∣{T⊆[k]:∣T∣=j,i∈T}= ​i∈[k]∑​gi​(n) ​j=1∑k​(jk−1​)(−e−kλ​)j= ​i∈[k]∑​gi​(n) ​((1−e−kλ​)k−1−1)=(Pr(∥C(x)∥=1)−nλ​)((1−e−kλ​)k−1−1),​

where we have used the convention that $\left(\genfrac{}{}{0ex}{}{k-1}{k}\right)=0$ .Now, for the last term, we compute

$$\begin{array}{rl}\sum _{T\subseteq [k]:|T|=j}\sum _{T^{\prime }\subseteq \overline{T}:|T^{\prime }|>1}{h}_{T^{\prime }}(n)& =\sum _{\ell =2}^{k-j}\sum _{T^{\prime }\subseteq [k]:|T^{\prime }|=\ell }{h}_{T^{\prime }}(n)|\left\{T\subseteq [k]:|T|=j,T^{\prime }\subseteq \overline{T}\right\}|\\ & =\sum _{\ell =2}^{k-j}\left(\genfrac{}{}{0ex}{}{k-\ell }{j}\right)\mathrm{Pr}(\Vert C(x)\Vert =\ell ),\end{array}$$

---

so

$$\begin{array}{rl}\sum _{j=1}^k{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\sum _{T\subseteq [k]:[T]=j}\sum _{T^{\prime }\subseteq \overline{T}:|T^{\prime }|>1}{h}_{T^{\prime }}(n)& =\sum _{j=1}^k{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\sum _{\ell =2}^{k-j}\left(\genfrac{}{}{0ex}{}{k-\ell }{j}\right)\mathrm{Pr}(\Vert C(x)\Vert =\ell )\\ & =\sum _{j=1}^k\sum _{\ell =2}^{k-j}{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\left(\genfrac{}{}{0ex}{}{k-\ell }{j}\right)\mathrm{Pr}(\Vert C(x)\Vert =\ell )\\ & =\sum _{j=1}^k\sum _{r=j}^{k-2}{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\left(\begin{array}{c}r\\ j\end{array}\right)\mathbf{Pr}(\Vert C(x)\Vert =k-r)\\ & =\sum _{r=1}^{k-2}\sum _{j=1}^r{\left(-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^j\left(\begin{array}{c}r\\ j\end{array}\right)\mathbf{Pr}(\Vert C(x)\Vert =k-r)\\ & =\sum _{r=1}^{k-2}\mathbf{Pr}(\Vert C(x)\Vert =k-r)\sum _{j=1}^r\left(\begin{array}{c}r\\ j\end{array}\right){\left(\begin{array}{c}-{\mathrm{e}}^{-\frac{\lambda }{k}}\end{array}\right)}^j\\ & =\sum _{r=1}^{k-2}\mathbf{Pr}(\Vert C(x)\Vert =k-r)\left({\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^r-1\right)\\ & =\sum _{j=2}^{k-1}\mathbf{Pr}(\Vert C(x)\Vert =j)\left({\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-j}-1\right).\end{array}$$

Adding the terms together gives
$$\begin{array}{rl}\text{M}& =\left(\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\right){\left(1-{\mathbf{e}}^{-\frac{\lambda }{k}}\right)}^k\\ & +\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right){\left(1-{\mathbf{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\\ & +\sum _{j=2}^{k-1}\mathbf{Pr}(\Vert C(x)\Vert =j){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-j}\\ & -\left(\mathbf{Pr}(\Vert C(x)\Vert =0)+\mathbf{Pr}(\Vert C(x)\Vert =1)+\sum _{j=2}^{k-1}\mathbf{Pr}(\Vert C(x)\Vert =j)-1\right).\end{array}$$
Of course,
$$-\left(\mathbf{Pr}(\Vert C(x)\Vert =0)+\mathbf{Pr}(\Vert C(x)\Vert =1)+\sum _{j=2}^{k-1}\mathbf{Pr}(\Vert C(x)\Vert =j)-1\right)=\mathbf{Pr}(\Vert C(x)\Vert =k$$
80
$$\begin{array}{rl}\text{M}& =\left(\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k\\ & +\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\\ & +\sum _{j=2}^{k-1}\mathbf{Pr}(\Vert C(x)\Vert =j){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-j}\\ & +\mathbf{Pr}(\Vert C(x)\Vert =k)\\ & =ϵ(n).\end{array}$$

---

Since
$$\mathbf{Pr}(\mathcal{F})-{\left(1-{\mathrm{e}}^{-\lambda /k}\right)}^k=\mathbf{Pr}(\mathcal{F})-\mathbf{Pr}(\forall i:P_i>0)\sim nM=nϵ(n),$$
the result follows.

Unfortunately,the schemes that we discuss in this paper are often too messy to apply Theo rem 6.1 generally; the values $\mathbf{Pr}(\Vert C(x)\Vert =j$ )depend on the specifics of the hash functions being use. For example, whether the size of the range is prime or not affects $\mathbf{Pr}(\Vert C(x)\Vert =j]$ 0.The result can be applied in cases to examine specific schemes; for example,in the partitioned scheme, when $m^{\prime }$ is prime, $\mathrm{Pr}(\Vert C(x)\Vert =j)=0$ for $j=2,\dots ,k-1$ ,and so the expression becomes easily computable.To achieve general results,we derive some simple bounds that are sufficient to draw some interesting conclusions.

Lemma 6.1. Assume the same conditions as in Theorem 4.1. Furthermore, suppose that fo $x\in S$ ，it is possible to define events $E_0,\dots ,E_{\ell -1}$ such that
$$\begin{array}{rlr}& I.\mathbf{Pr}(\Vert C(x)\Vert \ge 1)=\mathbf{Pr}\left(\bigcup _{i\in [\ell ]}{E}_i\right)& \\ & 2.\sum _{i\in [\ell ]}\mathbf{Pr}(E_i)=\lambda /n& \\ & 3.\mathbf{Pr}(\Vert C(x)\Vert \ge 2)\le \sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j).& \\ & hen& \\ & n\left[\mathbf{Pr}(\Vert C(x)\Vert \equiv k)-{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(1+{\mathrm{e}}^{-\frac{\lambda }{k}}\right)\sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j)\right]& ⩽\mathbf{Pr}(\mathcal{F})-\left(1-{\mathrm{e}}^{-\lambda /k}\right)\\ & & ⩽n\sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j)\end{array}$$

Proof.As in Theorem

6.1

,we define
$$\begin{array}{rl}ϵ(n)& \stackrel{\mathrm{def}}{=}\left(\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k\\ & +\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\\ & +\sum _{j=2}^k\mathbf{Pr}(\Vert C(x)\Vert =j){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-j},\end{array}$$
So that
$$\mathbf{Pr}(\mathcal{F})-{\left(1-{\mathrm{e}}^{-\lambda /k}\right)}^k\sim nϵ(n).$$
Now,
$$\begin{array}{rlrl}& \text{M}& & \stackrel{\mathrm{def}}{=}\left(\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k+\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\\ & & & ={\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(\left(\mathbf{Pr}(\Vert C(x)\Vert =0)-1+\frac{\lambda }{n}\right)\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)+\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right)\right)\\ & & & ={\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}((\mathbf{Pr}(\Vert C(x)\Vert =0)+\mathbf{Pr}(\Vert C(x)\Vert =1)-1)-{\mathrm{e}}^{-\frac{\lambda }{k}}\left((\mathbf{Pr}(\Vert C(x)\Vert =0)-1)+(\mathbf{Pr}(\Vert C(x)\Vert =0)-1)\right)\\ & & & ={\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(-\mathbf{Pr}(\Vert C(x)\Vert \ge 2)-{\mathrm{e}}^{-\frac{\lambda }{k}}\left(-\mathbf{Pr}(\Vert C(x)\Vert \ge 2)-\mathbf{Pr}(\Vert C(x)\Vert =1)+\frac{\lambda }{n}\right)\right)\\ & & & =-{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k\mathbf{Pr}(\Vert C(x)\Vert \ge 2)+{\mathrm{e}}^{-\frac{\lambda }{k}}{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right).\end{array}$$

---

In particular, we have that $M\le 0$ since

$$\mathbf{Pr}(\Vert C(x)\Vert =1)\le \mathbf{Pr}(\Vert C(x)\Vert \ge 1)=\mathbf{Pr}\left(\bigcup _{i\in [\ell ]}{E}_i\right)\le \sum _{i\in [l]}\mathbf{Pr}(E_i)=\lambda /n.$$

Therefore

$$ϵ(n)=M+\sum _{j=2}^k\mathbf{Pr}(\Vert C(x)\Vert =j){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-j}\le \mathbf{Pr}(\Vert C(x)\Vert \ge 2)\le \sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j)$$

establishing the upper bound in thelemma For the lower bound, we note that

$$\begin{array}{rl}\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}& =\mathbf{Pr}(\Vert C(x)\Vert \ge 1)-\mathbf{Pr}(\Vert C(x)\Vert \ge 2)-\frac{\lambda }{n}\\ & =\mathbf{Pr}\left(\bigcup _{i\in [t]}{E}_i\right)-\mathbf{Pr}(\Vert C(x)\Vert \ge 2)-\frac{\lambda }{n}\\ & \ge \sum _{i\in [t]}\mathbf{Pr}(E_i)-\sum _{i<j\in [t]}\mathbf{Pr}(E_i\cap E_j)-\mathbf{Pr}(\Vert C(x)\Vert \ge 2)-\frac{\lambda }{n}\\ & =-\sum _{i<j\in [t]}\mathbf{Pr}(E_i\cap E_j)-\mathbf{Pr}(\Vert C(x)\Vert \ge 2)\\ & \ge -2\sum _{i<j\in [t]}\mathbf{Pr}(E_i\cap E_j),\end{array}$$

SO

$$\begin{array}{rl}\text{M}& =-{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k\mathbf{Pr}(\Vert C(x)\Vert \ge 2)+{\mathrm{e}}^{-\frac{\lambda }{k}}{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(\mathbf{Pr}(\Vert C(x)\Vert =1)-\frac{\lambda }{n}\right)\\ & \ge -{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k\mathbf{Pr}(\Vert C(x)\Vert \ge 2)-{\mathrm{e}}^{-\frac{\lambda }{k}}{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}2\sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j)\\ & \ge -{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^k\sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j)-{\mathrm{e}}^{-\frac{\lambda }{k}}{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}2\sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j)\\ & =-{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(1+{\mathrm{e}}^{-\frac{\lambda }{k}}\right)\sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j).\end{array}$$

Therefore,

$$\begin{array}{rl}ϵ(n)& =\sum _{j=2}^k\mathbf{Pr}(\Vert C(x)\Vert =j){\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-j}+M\\ & \ge \mathbf{Pr}(\Vert C(x)\Vert =k)-{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(1+{\mathrm{e}}^{-\frac{\lambda }{k}}\right)\sum _{i<j\in [\ell ]}\mathbf{Pr}(E_i\cap E_j),\end{array}$$

completing the proof.

Lemma 6.1 is easily applied to the schemes discussed in Sections 5.1 and 5.2

Theorem 6.2. For the partition scheme discussed in Section 5.1

$$\frac{k^2}{c^{2}n}\left[1-{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(1+{\mathrm{e}}^{-\frac{\lambda }{k}}\right)\frac{k^3}{2}\right]\lesssim \mathbf{Pr}(\mathcal{F})-{\left(1-{\mathrm{e}}^{-\lambda /k}\right)}^k\lesssim \frac{k^5}{2c^{2}n}$$

Proof. We wish to apply Lemma 6.1.To this end,we fix $x\in S$ ，and for $i\in [k]$ ，we define $E_i$ to be the event that $i\in C(x)$ (once again, we use the convention introduced in the proof of Theorem 4.1 that allows us to associate the elements of $H(z)$ with the elements of $[k]$ ). Then

$$\mathbf{Pr}(\Vert C(x)\Vert \ge 1)=\mathbf{Pr}\left(\bigcup _{i\in [k]}{E}_i\right).$$

Recall from the proof of Theorem 5.1 that the partition scheme satisfies the conditions of Theo rem 4.1 with $\lambda =k^2/c$ .Furthermore, (as we saw in the proof of Theorem 5.1)

$$\begin{array}{r}\sum _{i\in [k]}\mathbf{Pr}(E_i)=\sum _{i\in [k]}\frac{1}{m^{\prime }}=\frac{\lambda }{n}.\end{array}$$

The proof of Theorem 5.1 also tells us that for $i\ne j\in [k]$

$$\mathbf{Pr}(E_i\cap E_j)\le \frac{k}{(m^{\prime }{)}^2}=\frac{k^3}{c^2{n}^2},$$

SO

$$\mathrm{Pr}(\Vert C(x)\Vert \ge 2)\le \sum _{i<j\in [k]}\mathbf{Pr}(E_i\cap E_j)\le \frac{k^5}{2c^2{n}^2},$$

where we have used the (obvious) fact that every $u\in U$ is assigned $k$ distinct hash locations in the partition scheme. Finally, we note that $\Vert C(x)\Vert =k$ if $h_1(x)=h_1(z)$ and $h_2(x)=h_2(z)$ ，SC

$$\mathbf{Pr}(\Vert C(x)\Vert =k)\ge \frac{1}{(m^{\prime }{)}^2}=\frac{k^2}{c^2{n}^2}.$$

Plugging these bounds into the result from Lemma 6.1 gives the result.

Theorem 6.3. For the double hashing schemes discussed in Section 5.2.

$$\frac{1}{c^{2}n}\left[1-{\left(1-{\mathrm{e}}^{-\frac{\lambda }{k}}\right)}^{k-1}\left(1+{\mathrm{e}}^{-\frac{\lambda }{k}}\right)\frac{k^5}{2}\right]\lesssim \mathbf{Pr}(\mathcal{F})-{\left(1-{\mathrm{e}}^{-\lambda /k}\right)}^k\lesssim \frac{k^5}{2c^{2}n}$$

Proof. We wish to apply Lemma 6.1. First, recall from the proof of Theorem 5.2 that every double hashing scheme satisfies the conditions of Theorem 4.1 with $\lambda =k^2/c$ Now fix $x\in S$ We reintroduce some notation from the proof of Theorem 5.2. For $u\in U$ and $i\in [k]$ ,we define

$$g_i(u)=h_1(u)+i{h}_2(u)+f(i)$$

(where we continue to use the convention that all arithmetic involving the hash functions $h_1$ anc $h_2$ is done modulo $7/l$ ) Proceeding, for $i,j\in [k]$ , we define $E_{i,j}$ to be the event that $g_j(x)=g_i(z)$ .Then

$$\mathbf{Pr}(\Vert C(x)\Vert \ge 1)=\mathbf{Pr}\left(\bigcup _{i,j\in [k]}{E}_{i,j}\right),$$

---

and,as we saw in the proof of Theorem 5.2

$$\sum _{i,j\in [k]}\mathbf{Pr}(E_{i,j})=\sum _{i,j\in [k]}\mathbf{Pr}(g_j(x)=g_i(z))=\sum _{i,j\in [k]}\frac{1}{m}=\frac{\lambda }{n}.$$

Furthermore, fixing any ordering 《 on $[k{]}^2$

P r ( ∥ C ( x ) ∥ ≥ 2 ) = P r ( ∃ i 1 , i 2 , j 1 , j 2 ∈ [ k ] : ∀ ℓ ∈ { 1 , 2 } , g j ℓ ( x ) = g i ℓ ( x ) ) = P r ( ⋃ ( i 1 , j 1 ) < ( i 2 , j 2 ) ∈ [ k ] 2 E i 1 , j 1 ∩ E i 2 , j 2 ) ≤ ∑ ( i 1 , j 1 ) < ( i 2 , j 2 ) ∈ [ k ] 2 P r ( E i 1 , j 1 ∩ E i 2 , j 2 ) , \begin{aligned}\mathbf{Pr}(\|C(x)\|\geq2)&=\mathbf{Pr}(\exists i_{1},i_{2},j_{1},j_{2}\in[k]:\forall\ell\in\{1,2\},g_{j_{\ell}}(x)=g_{i_{\ell}}(x))\\&=\mathbf{Pr}\left(\bigcup_{(i_{1},j_{1})<(i_{2},j_{2})\in[k]^{2}}E_{i_{1},j_{1}}\cap E_{i_{2},j_{2}}\right)\\&\leq\sum_{(i_{1},j_{1})<(i_{2},j_{2})\in[k]^{2}}\mathbf{Pr}(E_{i_{1},j_{1}}\cap E_{i_{2},j_{2}}),\end{aligned} Pr(∥C(x)∥≥2)​=Pr(∃i1​,i2​,j1​,j2​∈[k]:∀ℓ∈{1,2},gjℓ​​(x)=giℓ​​(x))=Pr ​(i1​,j1​)<(i2​,j2​)∈[k]2⋃​Ei1​,j1​​∩Ei2​,j2​​ ​≤(i1​,j1​)<(i2​,j2​)∈[k]2∑​Pr(Ei1​,j1​​∩Ei2​,j2​​),​

so the conditions of Lemma 6.1 are satisfied. To complete the proof, we note that for any $(i_1,j_1),(i_2,j_2)\in [k^2]$

$$\begin{array}{rl}\mathbf{Pr}(E_{i_1,j_1}\cap E_{i_2,j_2})& =\mathbf{Pr}(g_{j_1}(x)=g_{i_1}(z),g_{j_2}(x)=g_{i_2}(z))\\ & \le \frac{1}{m}\cdot \frac{k}{m}\\ & =\frac{k}{c^2{n}^2},\end{array}$$

where the computation in the second step was done in the proof of Theorem 5.2. Therefore

$$\sum _{(i_1,j_1)<(i_2,j_2)\in [k{]}^2}\mathrm{Pr}(E_{i_1,j_1}\cap E_{i_2,j_2})\le \sum _{(i_1,j_1)<(i_2,j_2)\in [k{]}^2}\frac{k}{c^2{n}^2}\le \frac{k^5}{2c^2{n}^2}.$$

Finally,

$$\mathbf{Pr}(\Vert C(x)\Vert =k)\ge \mathbf{Pr}(h_1(x)=h_1(z),h_2(x)=h_2(z))=\frac{1}{m^2}=\frac{1}{c^2{n}^2}.$$

Plugging these bounds into the result of Lemma 6.1 yields the result

It remains to investigate whether the error term analyzed in Theorems 6.2 and 6.3 is negligible in practice.Recall that for all of the schemes considered so far, the asymptotic false positive probability is $(1-\exp [-k/c]{)}^k$ ，the same as for a standard Bloom filter. We would like to minimize this probability.The easiest way to do this is to maximize $t$ given the applicationspecific constraints on the size of the filter, and then optimize $k$ subject to that value of $t$ which results in setting $k=c\ln 2$ (this is a standard result for Bloom filters which is easily obtained using calculus; see, for example,[3l), yielding an asymptotic false positive probability of $2^{-c\ln 2}$ Applying Theorems 6.2 and 6.3,we have that for all of the examined schemes, this setting of $k$ results in
$$\mathbf{Pr}(\mathcal{F})-2^{-c\ln 2}\lesssim \frac{(\ln 2{)}^5}{2}\frac{c^3}{n}\mathrm{as}n\to \infty .$$

We now give a heuristic argument that the above error term is negligible in practice. Suppose that the asymptotic inequality above held for every $7l.$ , and not just in the limit as $Tl\to 0$ .Then

for any $ϵ>0$
$$\begin{array}{rl}\mathrm{Pr}(\mathcal{F})-2^{-c\ln 2}\ge ϵ2^{-c\ln 2}& \Rightarrow \frac{(\ln 2{)}^5}{2}\frac{c^3}{n}\ge ϵ2^{-c\ln 2}\\ & \Rightarrow \frac{(\ln 2{)}^5}{2}\frac{c^3}{n}\ge ϵ2^{-c}\\ & \Rightarrow 2^{c+3\ln c}\ge \frac{2nϵ}{(\ln 2{)}^5}\\ & \Rightarrow 2^{2c+1}\ge \frac{2nϵ}{(\ln 2{)}^5}\\ & \Rightarrow c\ge \frac{1}{2}{\log }_2\left(\frac{nϵ}{(\ln 2{)}^5}\right).\end{array}$$
The first step is the only non-rigorous step, and it follows from the assumption that the asymptotic inequality aboveholds for every $TL$ .The second step holds since $\ln 2<1$ ,the third step is simple algebra, the fourth step follows from the fact that $3\ln c<c+1$ for all $C>0$ ，and the fifth step is also simple algebra. From this heuristic argument, we conclude that the asymptotic error term analyzed above is negligible unless $c\gtrsim {\log }_{2}n$ .In these cases, however, it might be more appropriate to use a hash table or fingerprints rather than a Bloom filter (see, for example, [12, Section 5.5]).