The Maximum Mean Discrepancy

Lemma 1 Let $(\mathcal X,d)$ be a metric space, and let p, q be two Borel probability measures defined on $\mathcal X$. Then p = q if and only if $E_{x}(f(x)) = E_{y}(f(y))$ for all $f \in C(\mathcal X)$ , where $C(\mathcal X)$ is the space of bounded continuous functions on $\mathcal X$.

Because it is not practical to work with such a rich function class in the finite sample setting. We thus define a more general class of statistic, for as yet unspecified function classes $\mathcal F$, to measure the disparity between p and q.

Definition 2 Let $\mathcal F$ be a class of functions $f:\mathcal X\to\mathbb R$ and let p,q,x,y,X,Y be defined as above. We defined the maximum mean discrepancy (MMD) as

$$MMD[\mathcal F,p,q]:= \mathop{sup} \limits_{f\in \mathcal F}(E_{x}[f(x)] - E[f(y)]).$$
In the statistics literature, this is known as an integal probability metric. A biased empirical estimate of the MMD is obtained by replacing the population expectations with empirical expectations computed on the samples X and Y,
$$MMD_b[\mathcal F,X,Y] := \mathop{sup}\limits_{f\in\mathcal F}\left(\frac{1}{m}\sum\limits_{i = 1}^{m}f(x_i)-\frac{1}{n}\sum\limits_{i = 1}^{n}f(y_i)\right).$$
We must therefore identify a function clas that is rich enough to uniquely identify whether p = q, yet restrictive enough to provide useful finite sample estimates.

Lemma 3 If $k(\centerdot,\centerdot)$ is measurable and $E_x\sqrt {k(x,x)}<\infty$ then $\mu_p \in \mathcal H$.

Lemma 4 Assume the condition in Lemma 3 for the existence of the mean embeddings $\mu_p,\mu_q$ is satisfied. Then

$$MMD^2[\mathcal F,p,q] = \left|\left|\mu_p - \mu_q\right|\right|^2_\mathcal H.$$

Theorem 5 Let $\mathcal F$ be a unit ball in a universal RKHS $\mathcal H$, defined on the compact metric space $\mathcal X$, with associated continuous kernel $k(\centerdot,\centerdot)$. Then $MMD[\mathcal F,p,q] = 0$ if and only if p = q.

Lemma 6 Given $x$ and $x^{'}$ independent random variables with distribution p, and $y$ and $y^{'}$ independent random variables with distribution q , the squared population MMD is

$$MMD^2[\mathcal F,X,Y] = E_{x,x^{'}}[k(x,x^{'})] - 2E_{x,y}[k(x,y)]+E_{y,y^{'}}[k(y,y^{'})],$$
where $x^{'}$ is an independent copy of $x$ with the same distribution, and $y^{'}$ is an independent copy of $y$. An unbiased empirical estimate is a sum of two U-statistics and a sample average,
$$MMD^2_u[\mathcal F,X,Y] = \frac{1}{m(m-1)}\sum\limits_{i = 1}^{m}\sum\limits_{j\ne i}^m k(x_i,x_j)+ \frac{1}{n(n-1)}\sum\limits_{i=1}^n\sum\limits_{j\ne i}^nk(y_i.y_j)-\frac{2}{mn}\sum\limits_{i=1}^m\sum\limits_{j=1}^nk(x_i,y_j).$$