Understanding of Hilbert-Schmidt Independence Criterion (HSIC)

• Problem
  Given $$(x_1,y_1),\dots ,(x_n,y_n)\in P(x,y),$$
  determing whether $$P(x,y)=P(x)\times P(y).$$
  Or measure degree of dependence.
  Below is the formal problem setting:
  Let Pxy be a Borel probability measure defined on a domain X x Y, and let Px and Py be the respective marginal distributions on X and Y. Given an I.I.D sample Z := (X, Y) = {(x1, y1), …, (xm, ym)} of size m drawn independently and identically distributed according to Pxy, does Pxy factorize as PxPy.

• Applications

  • Independent component analysis;
  • Dimsionality reduction and feature extraction;
  • Statistical modeling.

• Indirect Approach

  • Perform density estimate of P(x, y)
  • Check whether the estimate approximately factorizes

• Direct Approach

  • Check properties of factorizing distributions
  • e.g. kurtosis, covariance operators.

HSIC is defined as the squared Hilbert-Schmidt (HS) norm of the associated cross-covariance operator Cxy:
$$\mathrm{H}\mathrm{S}\mathrm{I}\mathrm{C}(P_{xy},F,G)=||C_{xy}|{|}_{HS}^2.$$
Above definition of HSIC involves two other definitions. One is cross-covariance operator which is defined:
$$C_{xy}:={\mathrm{E}}_{xy}([f(x)-{\mathrm{E}}_x(f(x))][g(y)-{\mathrm{E}}_y(g(y))]).$$
It is obvious that Cxy is a linear opertor that maps from G to F. The operator itself can be writen in: $$C_{xy}:={\mathrm{E}}_{xy}[(f(x)-{\mu }_x)\otimes (g(y)-{\mu }_y)],$$
where it denotes the tensor product.