【20210916】GMM入门

1. 模型介绍

高斯——高斯分布

从概率密度估计的角度来看 ，从几何角度来看，加权（$\alpha$）平均→多个高斯分布叠加而成
$$p(x)=\sum _{k=1}^k{\alpha }_{k}N\left({\mu }_k,{\Sigma }_k\right),\sum _{k=1}^k{\alpha }_k=1\text{\hspace{1em}(1)}$$
$$\sum {\alpha }_k=1$$
上式中${\alpha }_k$为权重

从混合（生成）模型的角度来看（生成模型）

x：observed variable
z：latent variable

z表示对应的样本x是属于哪一个高斯分布，这是一个离散的随机变量

生成过程，概率图（有向图）如下：（观测图用阴影表示）

联合概率密度可转化为乘法的形式：
$$\begin{array}{rl}p(x)& =\sum _{z}p(x,z)\\ & =\sum _{k=1}^{k}p\left(x,z=c_k\right)\\ & =\sum _{k=1}^{k}p\left(z=c_k\right)\cdot p\left(x\mid z=c_k\right)\\ & =\sum _{k=1}^K{p}_k\cdot N\left(x\mid {\mu }_k{\Sigma }_k\right)\end{array}\text{\hspace{1em}(2)}$$
上式中$p_k$为概率值，可见与式(1)相同

2. 极大似然估计MLE（Maximum Likelihood Estimate）

X：observed data （$X=(x_1,x_2,...,x_N)$）
(X,Z)：complete data
$\theta$：parameter （$\theta$={$p_1,p_2,...,p_k,{\mu }_1,{\mu }_2,...,{\mu }_k,{\Sigma }_1,{\Sigma }_2,...,{\Sigma }_k$}）

$${\hat{\theta }}_{MLE}=\arg \underset{\theta }{\max }\log P(x)$$
样本之间相互独立，上式可写为相乘的形式
$${\hat{\theta }}_{MLE}=\arg \underset{\theta }{\max }\log \prod _{i=1}^{N}P\left(x_i\right)=\arg \max \sum _{i=1}^N\log P\left(x_i\right)$$
将式(2)代入，得
$${\hat{\theta }}_{MLE}=\arg \underset{\theta }{\max }\sum _{i=1}^N\log \sum _{k=1}^k{p}_k\cdot N\left(x_i\mid {\mu }_k,{\Sigma }_k\right)$$

log里面是一个连加的形式，直接用MLE求解GMM，得不到解析解，一般使用EM求出$\theta$

3. EM求解（Expectation-Maximization Algorithm）

(1) EM算法（期望最大算法）公式与收敛性

主要用以估计还有隐变量的参数估计

MLE：$P(x|\theta )$
$${\theta }_{MLE}=\mathrm{argmax}\log P(x\mid \theta )$$
log-likelihood

EM公式如下：
$${\theta }^{(t+1)}=\arg \underset{\theta }{\max }{\int }_z\log p(x,z\mid \theta )\cdot p\left(z\mid x,{\theta }^{(t)}\right)dz\text{\hspace{1em}(3)}$$
第一项为对数联合概率，第二项为后验概率
$$E_{z\mid x{\theta }^{(t)}}[\log P(x,z\mid \theta )]$$

对于收敛性，需要证明：
$$\log P\left(x|{\theta }^{(t)}\right)⩽\log P\left(x\mid {\theta }^{(t+1)}\right)$$
对于
$$\log P(x\mid \theta )=\log P(x,z\mid \theta )-\log P(z\mid x,\theta )$$
对Q进行积分：
$$\begin{array}{rl}\text{ 左边 }& ={\int }_{z}p\left(z\mid x,{\theta }^{(t)}\right)\cdot \log P(x\mid \theta )dz\\ & =\log P(x\mid \theta )\underset{1}{\underbrace{{\int }_{z}P\left(z\mid x,{\theta }^{(t)}\right)dz}}\\ & =\log P(x\mid \theta )\end{array}$$
$$\begin{array}{rl}\text{ 右边 }& =\underset{Q\left(\theta ,{\theta }^{(t)}\right)}{\underbrace{{\int }_{z}P\left(Z\mid x,{\theta }^{(t)}\right)\cdot \log P(x,z\mid \theta )}}dz-\underset{H\left(\theta ,{\theta }^{(t)}\right)}{\underbrace{{\int }_{z}P\left(z\mid x,{\theta }^{(t)}\right)\cdot \log P(Z\mid x,\theta )dz}}\end{array}$$
对于第一项已经得证：
$$\begin{array}{rl}Q\left({\theta }^{(t+1)},{\theta }^{(t)}\right)& ⩾Q\left({\theta }^{(t)},{\theta }^{(t)}\right)\end{array}$$
下面仅需证明：
$$H\left({\theta }^{(t+1)},{\theta }^{(t)}\right)⩽H\left({\theta }^{(t)},{\theta }^{(t)}\right)$$
直接相减得：
$$\begin{array}{rl}& H\left({\theta }^{(t+1)},{\theta }^{(t)}\right)-H\left({\theta }^{(t)}\cdot {\theta }^{(t)}\right)\\ =& {\int }_{z}P\left(z\mid x,{\theta }^{(t)}\right)\cdot \log P\left(z\mid x{\theta }^{(t+1)}\right)dz\\ -& {\int }_{z}P\left(z\mid x,{\theta }^{(t)}\right)\cdot \log P\left(z\mid x,{\theta }^{(t)}\right)dz\\ =& {\int }_{z}P\left(z\mid x,{\theta }^{(t)}\right)\cdot \log \frac{P\left(z\mid x,{\theta }^{(++1)}\right)}{p\left(z\mid x,{\theta }^{(1)}\right)}dz\\ & E[\log x]⩽\log E[x]\\ ⩽& \log \underset{1}{\underbrace{{\int }_{z}p\left(z\mid x,{\theta }^{(t+1)}\right)dz}}=\log 1=0\end{array}$$
最后一步不是很懂

(2) E-step

$$\begin{array}{rl}EM:{\theta }^{(t+1)}& =\arg \max E_{z|x{\theta }^{(i)}}[\log P(x,z\mid \theta )]\\ & =\arg \max (Q(\theta ,{\theta }^t))\end{array}$$
这是一个迭代的方式，逐步去逼近和的最大值，由公式(3)
$$\begin{array}{rl}& Q\left(\theta ,{\theta }^{(t)}\right)={\int }_z\log P(X,z\mid \theta )\cdot P\left(z\mid X,{\theta }^{(t)}\right)dz\\ & =\sum _Z\underbrace{\log \prod _{i=1}^{N}P\left(x_i,z_i\mid \theta \right)}\cdot \prod _{i=1}^{N}P\left(z_i\mid x_i,{\theta }^{(t)}\right)\\ & =\sum _{z_1,z_2,z_N}\sum _{i=1}^N\log P\left(x_i,z_i\mid \theta \right)\cdot \prod _{i=1}^{N}P\left(z_i\mid x_i,{\theta }^{(t)}\right)\\ & =\sum _{z_1,z_2,z_N}\left[\log P\left(x_1,z_1\mid \theta \right)+\log P\left(x_2,z_2\mid \theta \right)+\cdots +\log P\left(x_N,z_N\mid \theta \right)\right]\prod _{i=1}^{N}P\left(z_i\mid x_i,{\theta }^{(t)}\right)\end{array}$$
因为：
$$\begin{array}{rl}P(x,z)& =P(z)\cdot p(x\mid z)\\ & =p_z\cdot N\left(x\mid {\mu }_z,{\Sigma }_z\right)\end{array}$$
$$P(z\mid x)=\frac{p(x,z)}{p(x)}=\frac{p_z\cdot N\left(x\mid {\mu }_z{\Sigma }_z\right)}{{\sum }_{k=1}^k{p}_k\cdot N\left(x\mid {\mu }_k,{\sum }_k\right)}$$
$$\sum _{z_1,z_2,z_N}\log P\left(x_1,z_1\mid \theta \right)\cdot \prod _{i=1}^{N}P\left(z_i\mid x_i,{\theta }^{(t)}\right)=\sum _{z_1}\log P\left(x_1,z_1\mid \theta \right)\cdot p\left(z_1\mid x_1,{\theta }^{(t)}\right)$$
上式：
$$\begin{array}{rl}& =\sum _{z_1}\log P\left(x_1,z_1\mid \theta \right)\cdot P\left(z_1\mid x_1,{\theta }^{(t)}\right)+\cdots +\sum _{Z_N}\log P\left(x_N,z_N\mid \theta \right)\cdot p\left(z_N\mid x_N,{\theta }^{(t)}\right)\\ & =\sum _{i=1}^N\sum _{Z_i}\log P\left(x_i,z_i\mid \theta \right)\cdot P\left(z_i\mid x_i,{\theta }^{(t)}\right)\\ & =\sum _{i=1}^N\sum _{z_i}\log [p_{z_i}\cdot N\left(x_i\mid {\mu }_{z_i},{\Sigma }_{z_i}\right)]\cdot \frac{p_{z_i}\cdot N\left(x_i\mid {\mu }_{z_i,}^{(t)}{\Sigma }_{z_i}^{(t)}\right)}{{\sum }_{k=1}^k{p}_k^{(t)}\cdot N\left(x_i\mid {\mu }_k^{(t)}{\Sigma }_k^{(t)}\right)}\end{array}\text{\hspace{1em}(4)}$$
注意第二项：${\mu }_{z_i}^{(t)},{\Sigma }_{Z_i}^{(t)}$

式(4)可继续写：
$$\begin{array}{rl}& =\sum _{i=1}^N\sum _{z_i}\log \left[p_{z_i}\cdot N\left(x_i\mid {\mu }_{{\epsilon }_i},{\Sigma }_{z_i}\right)\right]\cdot P\left(z_i\mid x_i,{\theta }^{(t)}\right)\\ & =\sum _{z_i}\sum _{i=1}^N\log p_{z_i}\cdot N\left(x_i\mid {\mu }_{z_i},{\Sigma }_{z_i}\right)\cdot P\left(z_i\mid x_i,{\theta }^{(t)}\right)\\ & =\sum _{k=1}^K\sum _{i=1}^N\log \left[p_k\cdot N\left(x_i\mid {\mu }_k,{\Sigma }_k\right)\right]P\left(z_i=c_k\mid x_i,{\theta }^{(t)}\right)\\ & =\sum _{k=1}^k\sum _{i=1}^N\left[\log p_k+\log N\left(x_i\mid {\mu }_k{\Sigma }_k\right)\right]\cdot P\left(z_i=c_k\mid x_i,{\theta }^{\left(t\right)}\right)\end{array}$$

(3) M-step

$${\theta }^{(t+1)}=\underset{\theta }{\mathrm{argmax}}Q\left(\theta ,{\theta }^{(t)}\right)$$
下面以求$p_k^{(t+1)}$为例：
$$p_k^{(t+1)}=\underset{p_k}{\mathrm{argmax}}\sum _{k=1}^k\sum _{i=1}^N\log p_k\cdot P\left(Z_i=C_k\mid x_i,{\theta }^{(t)}\right),\text{ s.t }\sum _{k=1}^K{p}_k=1$$
约束优化问题，使用拉格朗日乘子法，首先定义拉格朗日乘子
$$\mathcal{L}(p,\lambda )=\sum _{k=1}^k\sum _{i=1}^N\log p_k\cdot P\left(Z_i=C_k\mid x_i,{\theta }^{(t)}\right)+\lambda \left(\sum _{k=1}^k{p}_k-1\right)$$
求偏导数令其为0：
$$\frac{\partial f}{\partial p_k}=\sum _{i=1}^N\frac{1}{p_k}\cdot P\left(Z_i=C_k\mid x_i,{\theta }^{(t)}\right)+\lambda \triangleq 0$$
$$\begin{array}{rl}& \Rightarrow \sum _{i=1}^{N}P\left(z_i=C_R\mid x_i,{\theta }^{(t)}\right)+p_{k}i{\lambda }_i=0\\ & \Rightarrow k=1,\cdots ,K\Rightarrow \underset{1}{\underbrace{\sum _{i=1}^N\sum _{k=1}^{K}p(Z_i=C_k\mid x_i,{\theta }^{(t)}}})+\underset{1}{\underbrace{\sum _{k=1}^K{p}_k\lambda }}=0\\ & \Rightarrow N+\lambda =0\end{array}$$
$$\begin{array}{rl}& p_k^{(t+1)}=\frac{1}{N}\sum _{i=1}^{N}P\left(Z_i=C_k\mid x_i,{\theta }^{(t)}\right)\\ & p^{(t+1)}=\left(p_1^{(t+1)},\cdots ,p_k^{(t+1)}\right)\end{array}$$

总结

基本了解了GMM模型是什么，以及如何通过EM算法去求解，手推公式感觉很不友好，开始入门HMM