独立成分分析 Independent Component Analysis

独立成分分析 Independent Component Analysis

一. 问题引入

盲源分离问题(Blind Source Separation)是ICA算法最著名的应用场景。

• 盲源分离问题：有$L$个信号源，$D$个传感器。$D$个传感器接收了$L$个信号源混叠的信号，采样$m$次，得到一组数据 x = { x ( i ) , i = 1 , 2 , ⋯   , m } x=\{x^{(i)}, i = 1, 2, \cdots, m\} x={x(i),i=1,2,⋯,m}. 问题的目标是从$x$分离出$L$个信号源发出的独立信号 s = { s ( i ) , i = 1 , 2 , ⋯   , m } s=\{s^{(i)}, i = 1, 2, \cdots, m\} s={s(i),i=1,2,⋯,m}. 其中，$x^{(i)}$是$D\times 1$维信号，$s^{(i)}$是$L\times 1$维信号. 混合过程可以用以下方程描述:
  $$X_{D\times m}=A_{D\times L}{S}_{L\times m}$$
  矩阵$A$称为mixing matrix.

二. ICA – 最大似然估计

这里考虑信号源和传感器数量相等的情况，即$D=L$. 此时矩阵$A$为方阵，$A$的逆$W=A^{-1}$称为unmixing matrix. 盲源分离的目标就是找到$W$, 使$S=WX$. 为了方便，这里把$W$的列向量记为$w_i^T,i=1,\cdots ,L$.

2.1 ICA Ambiguities

• ICA算法有一个特性，就是源信号的幅度是无法恢复的.
• 高斯分布的信号是无法盲分离的，因为Gauss分布式旋转对称的，无法分辨旋转变量.
  The reason the Gaussian distribution is disallowed as a source prior in ICA is that it does not permit unique recovery of the sources, as illustrated in Figure 12.20©. This is because the PCA likelihood is invariant to any orthogonal transformation of the sources zt and mixing matrix W. PCA can recover the best linear subspace in which the signals lie, but cannot uniquely recover the signals themselves. – MLAPP

2.2 ICA Algorithm

假设第$i$个信号源的分布为$p_i(s)$，由于源信号各自独立，因此$j$时刻源信号$s_j$的联合概率分布为
$$p(s_j)=\prod _{i=1}^L{p}_i(s_{i,j})$$
则$j$时刻接受信号$x_j$的联合概率分布为
$$p(x_j)=\prod _{i=1}^L{p}_i(w_i^T{x}_j)|W|$$
那么似然函数为:
$$L(W)=\prod _{j=1}^{m}p(x_j)=\prod _{j=1}^m(\prod _{i=1}^L{p}_i(w_i^T{x}_j)|W|)$$
一般用对数似然函数作为优化目标:
$$\begin{array}{rl}J(W)& =\log (L(W))\\ & =\sum _{j=1}^m(\sum _{i=1}^L\log p_i(w_i^T{x}_j)+\log |W|)\end{array}$$
由于我们事先不知道信号源的分布，因此需要做一个假设. 一个合理的假设是认为信号源分布的累积概率密度函数(CDF)是sigmoid函数$F(s)=\frac{1}{1+\exp (-s)}$， 因此概率密度函数(PDF)是:
$$f(s)=\frac{d}{ds}F(s)=F(s)(1-F(s))$$
为了后续计算方便，计算得：
$$f^{{}^{\prime }}(s)=f(s)(1-2F(s))$$

利用随机梯度下降法和公式${\nabla }_W|W|=|W|(W^{-1}{)}^T$进行优化学习计算：
$$\begin{array}{rl}\frac{\partial J(W)}{\partial w_{ij}}& =\frac{\partial {\sum }_{j=1}^m({\sum }_{i=1}^L\log f(w_i^T{x}_j)+\log |W|)}{\partial w_{ij}}\\ & =\frac{1}{f(w_i^T{x}_j)}f(w_i^T{x}_j)(1-2F(w_i^T{x}_j))x_{ij}+\frac{1}{|W|}|W|(W^{-1}{)}_{ij}^T\\ & =(1-2F(w_i^T{x}_t))x_{ij}+(W^{-1}{)}_{ij}^T\end{array}$$
写成向量形式为:
$$\frac{\partial J(W)}{\partial W}=\left[\begin{array}{c}1-2F(w_1^T{x}_j)\\ 1-2F(w_2^T{x}_j)\\ ⋮\\ 1-2F(w_L^T{x}_j)\end{array}\right]x_j^T+(W^{-1}{)}^T$$

梯度下降的公式为:
$$W=W+\alpha \frac{\partial J(W)}{\partial W}$$

参考文献

1. Christopher M. Bishop. Pattern Recognition and Machine Learning
2. Kevin P. Murphy. Machine Learning A Probabilistic Perspective
3. Andrew Ng. http://cs229.stanford.edu/notes/cs229-notes11.pdf