EM算法（2）

1.EM算法
   假定有训练数据集
$$\left\{x^{(1)},x^{(2)},\cdots ,x^{(m)}\right\}$$
   包含m个独立样本，希望从中找出该组数据得模型模型$p(x,z)$得参数。
   取对数似然函数
$$\begin{array}{rl}& l(\theta )=\sum _{i=1}^m\log p(x;\theta )\\ & =\sum _{i=1}^m\log \sum _{z}p(x,z;\theta )\end{array}$$
   z是隐随机变量，不方便直接找到参数估计，使用下面的策略找出：计算$1(\theta )$的下界，求该下界最大值；重复该过程，直到收敛到局部最大值。

   令$Q_i$是z的某一个分布，$Q_i\ge 0$，有：
$$l(\theta )=\sum _{i=1}^m\log \sum _{z}p(x,z;\theta )=\sum _{i=1}^m\log \sum _{z^{(i)}}p\left(x^{(i)},z^{(i)};\theta \right)$$ $$\begin{array}{rl}& =\sum _{i=1}^m\log \sum _{z^{(M}}{Q}_i\left(z^{(i)}\right)\frac{p\left(x^{(i)},z^{(i)};\theta \right)}{Q_i\left(z^{(i)}\right)}\\ & \ge \sum _{i=1}^m\sum _{z^{(i)}}{Q}_i\left(z^{(i)}\right)\log \frac{p\left(x^{(i)},z^{(i)};\theta \right)}{Q_i\left(z^{(i)}\right)}\end{array}$$
   寻找尽量紧的下界，可以令：
$$\frac{p\left(x^{(i)},z^{(i)};\theta \right)}{Q_i\left(z^{(i)}\right)}=c$$
   进一步分析：
$$\begin{array}{c}{Q}_i\left(z^{(i)}\right)\propto p\left(x^{(i)},z^{(i)};\theta \right){\sum }_z{Q}_i\left(z^{(i)}\right)=1\\ Q_i\left(z^{(i)}\right)=\frac{p\left(x^{(i)},z^{(i)};\theta \right)}{{\sum }_{z}p\left(x^{(i)},z^{(i)};\theta \right)}\\ =\frac{p\left(x^{(i)},z^{(i)};\theta \right)}{p\left(x^{(i)};\theta \right)}\\ =p\left(z^{(i)}|x^{(i)};\theta \right)\end{array}$$
   EM算法整体框架：

2.从理论公式推导GMM
   随机变量X是由K个高斯分布混合而成，取各个高斯分布的概率为${\phi }_1{\phi }_2\cdots {\phi }_K$,第i个高斯分布的均值为${\mu }_i$，方差为${\sum }_i$。若观测到随机变量X的一系列样本${\mathrm{x}}_1,{\mathrm{x}}_2,\dots ,{\mathrm{x}}_{\mathrm{n}}$,试估计参数$\phi ,\mathbf{\mu },\mathbf{\Sigma }$。
   E-step
$$w_j^{(i)}=Q_i\left(z^{(i)}=j\right)=P\left(z^{(i)}=j|x^{(i)};\varphi ,\mu ,\Sigma \right)$$
   M-step
   将多项分布和高斯分布的参数带入:
$$\begin{array}{l}{\sum }_{i=1}^m{\sum }_{z^{(i)}}{Q}_i\left(z^{(i)}\right)\log \frac{p\left(x^{(i)},z^{(i)};\varphi ,\mu ,\Sigma \right)}{Q_i\left(z^{(i)}\right)}\\ ={\sum }_{i=1}^m{\sum }_{j=1}^k{Q}_i\left(z^{(i)}=j\right)\log \frac{p\left(x^{(i)}|z^{(i)}=j;\mu ,\Sigma \right)p\left(z^{(i)}=j;\varphi \right)}{Q_i\left(z^{(i)}=j\right)}\\ ={\sum }_{i=1}^m{\sum }_{j=1}^k{w}_j^{(i)}\log \frac{\frac{1}{(2\pi {)}^{n/2}{|{\Sigma }_j|}^{1/2}}\exp \left(-\frac{1}{2}{\left(x^{(i)}-{\mu }_j\right)}^T{\Sigma }_j^{-1}\left(x^{(i)}-{\mu }_j\right)\right)\cdot {\varphi }_j}{w_j^{(i)}}\end{array}$$
   对均值求偏导
$$\begin{array}{l}{\nabla }_{{\mu }_l}{\sum }_{i=1}^m{\sum }_{j=1}^k{w}_j^{(i)}\log \frac{\frac{1}{{(2\pi {)}^{n/2}|{\sum }_j|}^{1/2}}\exp \left(-\frac{1}{2}{\left(x^{(i)}-{\mu }_j\right)}^T{\Sigma }_j^{-1}\left(x^{(i)}-{\mu }_j\right)\right)\cdot {\varphi }_j}{w_j^{(i)}}\\ =-{\nabla }_{{\mu }_l}{\sum }_{i=1}^m{\sum }_{j=1}^k{w}_j^{(i)}\frac{1}{2}{\left(x^{(i)}-{\mu }_j\right)}^T{\Sigma }_j^{-1}\left(x^{(i)}-{\mu }_j\right)\\ =\frac{1}{2}{\sum }_{i=1}^m{w}_l^{(i)}{\nabla }_{{\mu }_l}2{\mu }_l^T{\Sigma }_l^{-1}{x}^{(i)}-{\mu }_l^T{\Sigma }_l^{-1}{\mu }_l\\ ={\sum }_{i=1}^m{w}_l^{(i)}\left({\Sigma }_l^{-1}{x}^{(i)}-{\Sigma }_l^{-1}{\mu }_l\right)\end{array}$$
   令上式等于0，解的均值为：
$${\mu }_l:=\frac{{\sum }_{i=1}^m{w}_l^{(i)}{x}^{(i)}}{{\sum }_{i=1}^m{w}_l^{(i)}}$$
   对方差求偏导，等于0
$${\Sigma }_j=\frac{{\sum }_{i=1}^m{w}_j^{(i)}\left(x^{(i)}-{\mu }_j\right){\left(x^{(i)}-{\mu }_j\right)}^T}{{\sum }_{i=1}^m{w}_j^{(i)}}$$
   多项分布参数，考察M-step的目标函数，对于$\varphi$，删除常数项
$$\sum _{i=1}^m\sum _{j=1}^k{w}_j^{(i)}\log \frac{\frac{1}{(2\pi {)}^{n/2}{|{\Sigma }_j|}^{1/2}}\exp \left(-\frac{1}{2}{\left(x^{(i)}-{\mu }_j\right)}^T{\Sigma }_j^{-1}\left(x^{(i)}-{\mu }_j\right)\right)\cdot {\varphi }_j}{w_j^{(i)}}$$
   得到
$$\sum _{i=1}^m\sum _{j=1}^k{w}_j^{(i)}\log {\varphi }_j$$
   拉格朗日乘子法
   由于多项分布的概率和为1，建立拉格朗日方程
$$\mathcal{L}(\varphi )=\sum _{i=1}^m\sum _{j=1}^k{w}_j^{(i)}\log {\varphi }_j+\beta \left(\sum _{j=1}^k{\varphi }_j-1\right)$$
   求偏导，等于0
$$\begin{array}{c}\frac{\partial }{\partial {\varphi }_j}\mathcal{L}(\varphi )={\sum }_{i=1}^m\frac{w_j^{(i)}}{{\varphi }_j}+\beta \\ -\beta ={\sum }_{i=1}^m{\sum }_{j=1}^k{w}_j^{(i)}={\sum }_{i=1}^{m}1=m\\ {\varphi }_j:=\frac{1}{m}{\sum }_{i=1}^m{w}_j^{(i)}\end{array}$$
   总结，对于所有的数据点，可以看作组份k生成了这些点。组份k是一个标准的高斯分布，利用上面结论： { γ ( i , k ) x i ∣ i = 1 , 2 , ⋯ N } \left\{\gamma(i, k) x_{i} | i=1,2, \cdots N\right\} {γ(i,k)xi​∣i=1,2,⋯N}。
$$\{\begin{array}{l}{\mu }_k=\frac{1}{N_k}{\sum }_{i=1}^N\gamma (i,k)x_i\\ {\Sigma }_k=\frac{1}{N_k}{\sum }_{i=1}^N\gamma (i,k)\left(x_i-{\mu }_k\right){\left(x_i-{\mu }_k\right)}^T\\ {\pi }_k=\frac{1}{N}{\sum }_{i=1}^N\gamma (i,k)\\ N_k=N\cdot {\pi }_k\end{array}$$