数学小抄: Gaussian的基础操作

前言:

高斯的基础操作在SLAM和Robot Learning中都有广泛的应用. 这些基础操作包括有: 线性变换,高维高斯分布的边缘分布与条件分布求解, MLE求解均值与方差, EM求解混合模型。 本篇博客参考于白板推导以及SLAM基础知识博客 , 我选择SLAM中的理解思路即$P(a,b)\sim P(b)P(a|b)$ 对条件分布进行推导, 这在PRML中也是采用类似的思路但是需要采用配方法。然后选择白板中引入舒尔补实现推导, 但这里并不会直接采用白板大神在视频里的构造，而是用了两次舒尔补实现推导。

1. MVN (Multivariate Normal Distribution) 多维高斯分布

1.1 线性变换:
对$\mathbf{x}\sim \mathcal{N}(\mathbf{x}|\mathbf{\mu },\mathbf{\Sigma })$作线性变换$\mathbf{y}\sim \mathbf{Ax}+\mathbf{b}$, 有$\mathbf{y}\sim \mathcal{N}(A\mu +b,A\Sigma A^T)$
证明:
$$\begin{array}{c}\begin{array}{r}\mathrm{E}[\mathrm{y}]=\mathrm{E}[\mathrm{Ax}+\mathrm{b}]=\mathrm{AE}[\mathrm{x}]+\mathrm{b}=\mathrm{A}\mu +\mathrm{b}\\ \mathrm{Var}[\mathrm{y}]=\mathrm{Var}[\mathrm{Ax}+\mathrm{b}]=\mathrm{Var}[\mathrm{Ax}]=\mathrm{A}\cdot \mathrm{Var}\cdot {\mathrm{A}}^{\mathrm{T}}\end{array}\end{array}$$

1.2 从线性变换到高维高斯分布$(x_a,x_b)\sim \mathcal{N}(\mu ,\mathbf{\Sigma })$,$\mathbf{\Sigma }=\left[\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right]$的条件分布与边缘分布.

• 已知高维高斯分布求$p(x_a),p(x_b),p(x_a|x_b),p(x_b|x_a)$

• $p(x_a)$与$p(x_b)$:
  $$\begin{array}{rl}{x}_a& =[I,0]\left[\begin{array}{c}{x}_a\\ x_b\end{array}\right]\\ x_b& =[0,I]\left[\begin{array}{c}{x}_a\\ x_b\end{array}\right]\end{array}$$
  $$\begin{array}{r}\mathrm{E}[{\mathrm{x}}_{\mathrm{a}}]=[\mathrm{I},0]\left[\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right]={\mu }_{\mathrm{a}}\\ \mathrm{Var}[{\mathrm{x}}_{\mathrm{a}}]=[\mathrm{I},0]\left[\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right]\left[\begin{array}{c}I\\ 0\end{array}\right]={\mathrm{\Sigma }}_{\mathrm{aa}}\\ \mathrm{E}[{\mathrm{x}}_{\mathrm{b}}]=[0,\mathrm{I}]\left[\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right]={\mu }_{\mathrm{b}}\\ \mathrm{Var}[{\mathrm{x}}_{\mathrm{b}}]=[0,\mathrm{I}]\left[\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right]\left[\begin{array}{c}0\\ I\end{array}\right]={\mathrm{\Sigma }}_{\mathrm{bb}}\end{array}$$
  $x_a\sim \mathcal{N}({\mu }_a,{\Sigma }_{aa})$以及$x_b\sim \mathcal{N}({\mu }_b,{\Sigma }_{bb})$

• $p(x_a|x_b),p(x_b|x_a)$: 需要引入舒尔补, 舒尔补数学小抄

  首先对于高斯分布的表达式，我们可以舍弃其分式前部,只看$\exp (\cdot )$部分.
  $$p(x_a,x_b)\propto exp(-\frac{1}{2}{\left[\begin{array}{c}{x}_a\\ x_b\end{array}\right]}^T{\left[\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right]}^{-1}\left[\begin{array}{c}{x}_a\\ x_b\end{array}\right])$$
  式子中涉及对于协方差矩阵的求逆，舒尔补在这里派上用场.
  下面会是关于舒尔补的一些基础等式
  $$\left[\begin{array}{cc}I& 0\\ -C{A}^{-1}& I\end{array}\right]\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]\left[\begin{array}{cc}I& -A^{-1}B\\ 0& I\end{array}\right]=\left[\begin{array}{cc}A& 0\\ 0& D-C{A}^{-1}B\end{array}\right]$$
  $$\left[\begin{array}{cc}I& -B{D}^{-1}\\ 0& I\end{array}\right]\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]\left[\begin{array}{cc}I& 0\\ -D^{-1}C& I\end{array}\right]=\left[\begin{array}{cc}A-B{D}^{-1}C& 0\\ 0& D\end{array}\right]$$
  $$\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]=\left[\begin{array}{cc}I& 0\\ C{A}^{-1}& I\end{array}\right]\left[\begin{array}{cc}A& 0\\ 0& D-C{A}^{-1}B\end{array}\right]\left[\begin{array}{cc}I& A^{-1}B\\ 0& I\end{array}\right]$$
  $${\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]}^{-1}=\left[\begin{array}{cc}I& -A^{-1}B\\ 0& I\end{array}\right]\left[\begin{array}{cc}{A}^{-1}& 0\\ 0& (D-C{A}^{-1}B{)}^{-1}\end{array}\right]\left[\begin{array}{cc}I& 0\\ -C{A}^{-1}& I\end{array}\right]$$
  $$\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]=\left[\begin{array}{cc}I& B{D}^{-1}\\ 0& I\end{array}\right]\left[\begin{array}{cc}A-B{D}^{-1}C& 0\\ 0& D\end{array}\right]\left[\begin{array}{cc}I& 0\\ D^{-1}C& I\end{array}\right]$$
  $${\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]}^{-1}=\left[\begin{array}{cc}I& 0\\ -D^{-1}C& I\end{array}\right]\left[\begin{array}{cc}(A-B{D}^{-1}C{)}^{-1}& 0\\ 0& D^{-1}\end{array}\right]\left[\begin{array}{cc}I& -B{D}^{-1}\\ 0& I\end{array}\right]$$

  考虑$p(x_a,x_b)$指数幂部分:
  $$\begin{array}{rl}-\frac{1}{2}\left[\begin{array}{cc}{x}_a& x_b\end{array}\right]{\left[\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right]}^{-1}\left[\begin{array}{c}{x}_a\\ x_b\end{array}\right]& =-\frac{1}{2}\left[\begin{array}{cc}{x}_a& x_b\end{array}\right]\left[\begin{array}{cc}{\Lambda }_{aa}& {\Lambda }_{ab}\\ {\Lambda }_{ba}& {\Lambda }_{bb}\end{array}\right]\left[\begin{array}{c}{x}_a\\ x_b\end{array}\right]\\ & =-\frac{1}{2}(\left[\begin{array}{cc}{x}_a& x_b\end{array}\right]\left[\begin{array}{cc}I& 0\\ {\Lambda }_{ba}{\Lambda }_{aa}^{-1}& I\end{array}\right]\left[\begin{array}{cc}{\Lambda }_{aa}& 0\\ 0& {\Lambda }_{bb}-{\Lambda }_{ba}{\Lambda }_{aa}^1{\Lambda }_{ab}\end{array}\right]\left[\begin{array}{cc}I& {\Lambda }_{aa}^{-1}{\Lambda }_{ab}\\ 0& I\end{array}\right]\left[\begin{array}{c}{x}_a\\ x_b\end{array}\right])\\ & =-\frac{1}{2}(x_a+x_b{\Lambda }_{ba}{\Lambda }_{aa}^{-1}){\Lambda }_{aa}(x_a+{\Lambda }_{aa}^{-1}{\Lambda }_{ab}{x}_b)-\frac{1}{2}{x}_b({\Lambda }_{bb}-{\Lambda }_{ba}{\Lambda }_{aa}^{-1}{\Lambda }_{ab})x_b\end{array}$$
  对于精度矩阵$\Lambda$ 可以借用与协方差矩阵之间的关系$\Sigma$，结合舒尔补完成求解:
  $$\begin{array}{rl}\left[\begin{array}{cc}{\Lambda }_{aa}& {\Lambda }_{ab}\\ {\Lambda }_{ba}& {\Lambda }_{bb}\end{array}\right]& ={\left[\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right]}^{-1}\\ & =\left[\begin{array}{cc}I& 0\\ -D^{-1}C& I\end{array}\right]\left[\begin{array}{cc}(A-B{D}^{-1}C{)}^{-1}& 0\\ 0& D^{-1}\end{array}\right]\left[\begin{array}{cc}I& -B{D}^{-1}\\ 0& I\end{array}\right]\\ & =\left[\begin{array}{cc}I& 0\\ -{\Sigma }_{bb}^{-1}{\Sigma }_{ba}& I\end{array}\right]\left[\begin{array}{cc}({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}& 0\\ 0& {\Sigma }_{bb}^{-1}\end{array}\right]\left[\begin{array}{cc}I& -{\Sigma }_{ba}{\Sigma }_{bb}^{-1}\\ 0& I\end{array}\right]\\ & =\left[\begin{array}{cc}({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}& -({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}{\Sigma }_{ab}{\Sigma }_{bb}^{-1}\\ -{\Sigma }_{bb}^{-1}{\Sigma }_{ba}({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}& {\Sigma }_{bb}^{-1}+{\Sigma }_{bb}^{-1}{\Sigma }_{ba}({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}{\Sigma }_{ba}{\Sigma }_{bb}^{-1}\end{array}\right]\end{array}$$

  对于(4)式的后一部分有:
  $$\begin{array}{rl}{\Lambda }_{bb}-{\Lambda }_{ba}{\Lambda }_{aa}^{-1}{\Lambda }_{ab}& ={\Sigma }_{bb}^{-1}+{\Sigma }_{bb}^{-1}{\Sigma }_{ba}({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}{\Sigma }_{ba}{\Sigma }_{bb}^{-1}-{\Sigma }_{bb}^{-1}{\Sigma }_{ba}({\Sigma }_{aa}\\ & -{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba})({\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}{)}^{-1}{\Sigma }_{ba}{\Sigma }_{bb}^{-1}\\ & ={\Sigma }_{bb}^{-1}\end{array}$$
  对于前一部分,方差显然为${\Lambda }_{aa}$, 而均值则: 将相应的部分代入:
  $$x_a+{\Lambda }_{aa}^{-1}{\Lambda }_{ab}{x}_b$$
  中:最后还是得到
  $$x_a-{\Sigma }_{ba}{\Sigma }_{bb}^{-1}{x}_b$$

  注意这里对应的是标准正态分布$(x-\mu {)}^T\Sigma (x-\mu )$中$(x-\mu )$的部分. 将$x_a$写作标准形式$(x_a-{\mu }_a)-{\Sigma }_{ba}{\Sigma }_{bb}^{-1}(x_b-{\mu }_b)$. 这里注意，在白板中会把这个标准形式命名为$x_{a\cdot b}$ ,这真的让我很费解, 直到我悟到一个: $p(a|b)$是在b发生下关于a的概率分布. 因此其标准正态分布形式为: $(x_a-{\mu }_a{)}^T{\Sigma }_a(x_a-{\mu }_a)$. 对照之下可知: 在b发生下a的均值为:${\mu }_a+{\Sigma }_{ba}{\Sigma }_{bb}^{-1}(x_b-{\mu }_b)$
  使用两次舒尔补是我自以为比较有意思的地方，但推导下来发现还是需要用定义去理解不能直接推导得出.看到这里的读者还是选择SLAM那篇博客吧

2. MLE 最大似然估计

2.1 MLE中用到的矩阵微分操作:
证明请见矩阵求导公式
(1) $\frac{\partial w^{T}Aw}{\partial w}=2Aw$
(2) $tr[ABC]=tr[CAB]=tr[BCA]$
(3) $x^{T}Ax=tr[x^{T}Ax]=tr[x{x}^{T}A]$
(4) $\frac{\partial }{\partial A}tr[AB]=B^T$
(5) $\frac{\partial }{\partial A}log|A|=(A^{-1}{)}^T=(A^T{)}^{-1}$
(6) $tr[AB]=tr[BA]$
对于各次服从iid的取样而言，$X^{(i)}\sim \mathcal{N}(\mu ,\Sigma )$, 要每次都取到采样的结果就是${\prod }_{i=1}^m{f}_{{\mathbf{X}}^{(\mathbf{i})}}({\mathbf{x}}^{(\mathbf{i})};\mu ,\mathbf{\Sigma })$对其取log-likehood函数有:
$$\begin{array}{rl}l(\mu ,\Sigma |{\mathbf{x}}^{(\mathbf{i})})& =\log \prod _{i=1}^m{f}_{{\mathbf{X}}^{(\mathbf{i})}}({\mathbf{x}}^{(\mathbf{i})}|\mu ,\mathbf{\Sigma })\\ & =\log \prod _{i=1}^m\frac{1}{(2\pi {)}^{p/2}|\Sigma {|}^{1/2}}\exp \left(-\frac{1}{2}({\mathbf{x}}^{(\mathbf{i})}-\mu {)}^{\mathbf{T}}{\mathbf{\Sigma }}^{-1}({\mathbf{x}}^{(\mathbf{i})}-\mu )\right)\\ & =\sum _{i=1}^m\left(-\frac{p}{2}\log (2\pi )-\frac{1}{2}\log |\Sigma |-\frac{1}{2}({\mathbf{x}}^{(\mathbf{i})}-\mu {)}^{\mathbf{T}}{\mathbf{\Sigma }}^{-1}({\mathbf{x}}^{(\mathbf{i})}-\mu )\right)\end{array}$$
$$\begin{array}{rl}l(\mu ,\Sigma ;)& =-\frac{mp}{2}\log (2\pi )-\frac{m}{2}\log |\Sigma |-\frac{1}{2}\sum _{i=1}^m({\mathbf{x}}^{(\mathbf{i})}-\mu {)}^{\mathbf{T}}{\mathbf{\Sigma }}^{-1}({\mathbf{x}}^{(\mathbf{i})}-\mu )\end{array}$$

$$\begin{array}{rl}\frac{\partial }{\partial \mu }l(\mu ,\Sigma |{\mathbf{x}}^{(\mathbf{i})})& =\sum _{i=1}^m{\mathbf{\Sigma }}^{-1}({\mathbf{x}}^{(\mathbf{i})}-\mu )=0\\ & \text{Since Σ is positive definite}\\ 0& =m\mu -\sum _{i=1}^m{\mathbf{x}}^{(\mathbf{i})}\\ \hat{\mu }& =\frac{1}{m}\sum _{i=1}^m{\mathbf{x}}^{(\mathbf{i})}=\bar{\mathbf{x}}\end{array}$$

$$\frac{\partial }{\partial A}{x}^{T}Ax=\frac{\partial }{\partial A}\mathrm{tr}\left[x{x}^{T}A\right]=[x{x}^T{]}^T={\left(x^T\right)}^T{x}^T=x{x}^T$$
重写log-likehood函数有:
$$\begin{array}{rl}l(\mu ,\Sigma |{\mathbf{x}}^{(\mathbf{i})})& =\text{C}-\frac{m}{2}\log |\Sigma |-\frac{1}{2}\sum _{i=1}^m({\mathbf{x}}^{(\mathbf{i})}-\mu {)}^{\mathbf{T}}{\mathbf{\Sigma }}^{-1}({\mathbf{x}}^{(\mathbf{i})}-\mu )\\ & =\text{C}+\frac{m}{2}\log |{\Sigma }^{-1}|-\frac{1}{2}\sum _{i=1}^m\mathrm{tr}\left[({\mathbf{x}}^{(\mathbf{i})}-\mu )({\mathbf{x}}^{(\mathbf{i})}-\mu {)}^{\mathbf{T}}{\mathbf{\Sigma }}^{-1}\right]\\ \frac{\partial }{\partial {\Sigma }^{-1}}l(\mu ,\Sigma |{\mathbf{x}}^{(\mathbf{i})})& =\frac{m}{2}\Sigma -\frac{1}{2}\sum _{i=1}^m{({\mathbf{x}}^{(\mathbf{i})}-\mu )({\mathbf{x}}^{(\mathbf{i})}-\mu )}^T\text{Since ΣT=Σ}\end{array}$$

$$\begin{array}{rl}0& =m\Sigma -\sum _{i=1}^m{({\mathbf{x}}^{(\mathbf{i})}-\mu )({\mathbf{x}}^{(\mathbf{i})}-\mu )}^T\\ \hat{\Sigma }& =\frac{1}{m}\sum _{i=1}^m{({\mathbf{x}}^{(\mathbf{i})}-\hat{\mu })({\mathbf{x}}^{(\mathbf{i})}-\hat{\mu })}^T\end{array}$$

参考链接

3. EM算法求解GMM中的参数

GMM: k个高斯分布通过线性加权组合而成一个模型. 主要还是几个问题:
(1) 每个高斯成分的参数 $\theta$ 估计;
(2) 权值视为一个隐变量$z_i\sim [{\pi }_1,{\pi }_2,...{\pi }_k]$;
(3) 给定隐变量$z_i$ 计算对应的条件分布$x_i\sim p(x|y_i,{\theta }_{y_i})$
$$p(x;\theta ,\pi )=\sum _{c=1}^k{\pi }_c\mathcal{N}(x;{\mu }_c,{\Sigma }_c)$$

EM算法:
E-step与M-step的导出:
对于一个分布的MLE有:
$$L(\theta )=\ln \mathcal{P}(\mathbf{X}\mid \theta )$$
EM不同于MLE直接给出解析式,而是选择通过迭代的方法实现:
$$L(\theta )>L\left({\theta }_n\right)$$
如果认为$\mathcal{P}(X|\theta )$是受隐变量$\mathbf{z}$的影响则有:
$$\mathcal{P}(\mathbf{X}\mid \theta )=\sum _{\mathbf{z}}\mathcal{P}(\mathbf{X}\mid \mathbf{z},\theta )\mathcal{P}(\mathbf{z}\mid \theta )$$
引入Jensen不等式(回忆log函数, 弧在上，而弦在下)有:
$$\ln \sum _{i=1}^n{\lambda }_i{x}_i\ge \sum _{i=1}^n{\lambda }_i\ln \left(x_i\right)$$
常数${\lambda }_i\ge 0$, ${\sum }_{i=1}^n{\lambda }_i=1$, 考虑隐变量的分布:$\mathcal{P}(\mathbf{z}|X,{\theta }_n)\ge 0$, ${\sum }_{\mathbf{z}}\mathcal{P}(\mathbf{z}|X,{\theta }_n)=1$.

$$\begin{array}{rl}\mathcal{L}& =log\sum _{\mathbf{z}}p(\mathbf{x},\mathbf{z})=log\sum _{\mathbf{z}}q(\mathbf{z})\frac{p(\mathbf{x},\mathbf{z})}{q(\mathbf{z})}\\ & \ge \sum _{\mathbf{z}}q(\mathbf{z})log\frac{p(\mathbf{x},\mathbf{z})}{q(\mathbf{z})}\\ & =\sum _{\mathbf{z}}q(\mathbf{z})logp(\mathbf{x},\mathbf{z})-\sum _{\mathbf{z}}q(\mathbf{z})logq(\mathbf{z})\end{array}$$

$$\mathcal{H}(q)\stackrel{△}{=}-\sum _{\mathbf{z}}q(\mathbf{z})logq(\mathbf{z})$$
$$\mathcal{F}(q,\theta )\stackrel{△}{=}\sum _{\mathbf{z}}q(\mathbf{z})logp(\mathbf{x},\mathbf{z})-\sum _{\mathbf{z}}q(\mathbf{z})logq(\mathbf{z})$$

$\mathcal{F}(q,\theta )$即为这个log-likehood函数的lower bound:
$$\mathcal{L}(\theta )\ge \mathcal{F}(q,\theta )$$

E-step： $q^{(t)}\leftarrow argma{x}_q\mathcal{F}(q,{\theta }^t)$ : 固定${\theta }^t$确定隐变量的分布
M-step: ${\theta }^{(t+1)}\leftarrow argma{x}_{\theta }\mathcal{F}(q^t,\theta )$ : 固定隐变量的分布确定各模型的参数$\theta$

性质: 每一轮都会improve $\mathcal{F}$ 也会improve $\mathcal{L}$.

E-step: 寻找隐变量的分布, 最大化$\mathcal{F}(q,{\theta }^{(t)})$:

$$\begin{array}{rl}\mathcal{F}(q)& =\sum _{\mathbf{z}}q(\mathbf{z})logp(\mathbf{x},\mathbf{z})-\sum _{z}q(\mathbf{z})logq(\mathbf{z})\\ & =\sum _{\mathbf{z}}q(\mathbf{z})log\frac{p(\mathbf{z}|x)p(x)}{q(\mathbf{z})}\\ & =\sum _{\mathbf{z}}q(\mathbf{z})log\frac{p(\mathbf{z}|x)}{q(\mathbf{z})}+\sum _{\mathbf{z}}q(\mathbf{z})logp(\mathbf{x})\\ & =\sum _{\mathbf{z}}q(\mathbf{z})log\frac{p(\mathbf{z}|x)}{q(\mathbf{z})}+logp(\mathbf{x})\\ & =-KL(q(\mathbf{z})|p(\mathbf{z}|\mathbf{x}))+\mathcal{L}\end{array}$$

KL散度部分衡量两个分布之间的距离iff $q=p$时KL=0, 也就是说只有在$q^t(\mathbf{z})=p(\mathbf{z}|x,{\theta }^t)$, $\mathcal{F}(q^t,{\theta }^t)=\mathcal{L}({\theta }^t)$. 因此,当我们在计算$p(\mathbf{z}|\mathbf{x},{\theta }^t)$时就是在最大化$\mathcal{F}(q,{\theta }^t)$.

M-step: 寻找${\theta }^{(t+1)}$最大化$\mathcal{F}(q^{(t)},\theta )$.
$$\mathcal{F}(q^t,\theta )=\sum _{\mathbf{z}}{q}^t(\mathbf{z})\log \mathrm{p}(\mathbf{x},\mathbf{z}|\theta )+\mathcal{H}({\mathrm{q}}^{\mathrm{t}}).$$

$\mathcal{H}(q^t)$在这一步中是常数, 我们只需要计算log-likehood部分即可:
$$\sum _{\mathbf{z}}{q}^t(\mathbf{z})\log \mathrm{p}(\mathbf{x},\mathbf{z}|\theta )$$

在最大化$\mathcal{F}$的时候还会最大化$\mathcal{L}$:
$$\begin{array}{rl}\mathcal{L}({\theta }^{t+1})& \ge \mathcal{F}(q^t,{\theta }^{t+1})\\ & =\underset{\theta }{\max }\mathcal{F}(q^t,\theta )\\ & \ge \mathcal{F}(q^t,{\theta }^t)=\mathcal{L}\end{array}$$

E-step与M-step: EM上的GMM还是太难了. 下面简单抄写白板中的内容

白板中的GMM: $p(x)={\sum }_{k=1}^K{p}_k\mathcal{N}(x|{\mu }_k,{\Sigma }_k)$
$X$: observed data $\to x_1,x_2,...,x_N$
$(X,Z)$: complete data $\to (x_1,z_1),(x_2,z_2),...,(x_N,z_N)$
$x$: observed variable
$z$: latent variable

z	$c_1$	$c_2$	$\dots$	$c_K$
p	$p_1$	$p_2$	$\dots$	$p_K$

${\sum }_{k=1}^K{p}_k=1$; $x|z=c_k\sim \mathcal{N}(x|{\mu }_k,{\Sigma }_k)$ 对于隐变量$z$的理解还是李航老师书里讲得更深刻，即$x$ 的生成需要先通过隐变量确定一个高斯再由该高斯生成, 显然隐变量为一个离散随机变量，因此确定哪个高斯也服从于一个概率分布,作用时再由各个高斯iid.起作用.但就公式的简洁上，白板中的公式更简单.
$$\begin{array}{rl}p(y,\gamma |\theta )& =\prod _{j=1}^{N}p(y_j,{\gamma }_{j1},{\gamma }_{j2},\cdots ,{\gamma }_{jk}|\theta )\\ & =\prod _{k=1}^K\prod _{j=1}^N[{\alpha }_k\varphi (y_j|{\theta }_k){]}^{{\gamma }_{jk}}\\ & 连乘前后交换顺序无关紧要\\ & =\prod _{j=1}^N\prod _{k=1}^K[{\alpha }_k\varphi (y_j|{\theta }_k){]}^{{\gamma }_{jk}}\\ & =\prod _{j=1}^N\underset{K项}{\underbrace{[{\alpha }_j\varphi (y_j|{\theta }_k)]\cdot 1\cdot 1\cdots 1}}\end{array}$$
${\alpha }_j$表示第$j$项观测数据对应的权重(几何角度)或者说是概率值(混合模型角度)

E-step中符号 $argmax{E}_{Z|X,{\theta }^{(t)}}[logP(X,Z|\theta )]$ 的意思是: ${\int }_{Z}logp(X,Z|\theta )p(Z|X,{\theta }^t)dz$ 是一个关于在观测数据已知, 上一步各个高斯分布参数已知的情况下关于隐变量的积分.

$$\begin{array}{rl}Q(\theta ,{\theta }^t)& ={\int }_{Z}logP(X,Z|\theta )p(Z|X,{\theta }^t)dZ\\ & =\sum _{Z}log[\prod _{i=1}^{N}P(x_i,Z_i|\theta )][\prod _{i=1}^{N}P(Z_i|x_i,{\theta }^t)]dz\\ & =\sum _{Z}log[\sum _{i=1}^{N}P(x_i,Z_i|\theta )][\prod _{i=1}^{N}P(Z_i|x_i,{\theta }^t)]dz\\ & =\sum _{Z_1,Z_2,\cdots ,Z_N}\sum _{i=1}^{N}logP(x_i,Z_i|\theta )\prod _{i=1}^{N}P(Z_i|x_i,{\theta }^t)\\ & =\sum _{i=1}^N\sum _{Z_i}logP(x_i,Z_i|\theta )P(Z_i|x_i,{\theta }^t)\end{array}$$
上式中
$$\begin{array}{rl}& \sum _{Z_1,Z_2,\cdots ,z_N}\sum _{i=1}^{N}logP(x_i,Z_i|\theta )\prod _{i=1}^{N}p(Z_i|x_i,{\theta }^t)\\ & =\sum _{Z_1,Z_2,\cdots ,Z_N}[logP(x_1,Z_1|\theta )+logP(x_2,Z_2|\theta )+\cdots +logP(x_N,Z_N|\theta )]\prod _{i=1}^{N}P(Z_i|x_i,{\theta }^t)\\ & 取出第一项有:\\ & =\sum _{Z_1,Z_2,\cdots ,Z_N}logP(x_1,Z_1|\theta )\prod _{i=1}^{N}P(Z_i|x_i,{\theta }^t)\\ & =\sum _{Z_1,Z_2,\cdots ,Z_N}logP(x_1,Z_1|\theta )P(Z_1|x_1,{\theta }^t)\prod _{i=2}^{N}P(Z_i|x_i,{\theta }^t)\\ & =\sum _{Z_1}logP(x_1,Z_1|\theta )P(Z_1|x_1,{\theta }^t)\sum _{Z_2,\cdots ,Z_N}\prod _{i=2}^{N}P(Z_i|x_i,{\theta }^t)\\ & 看后面的连加与连乘符号:\\ & \sum _{Z_2,\cdots ,Z_N}P(Z_2|x_2,{\theta }^t)P(Z_3|x_3,{\theta }^t)P(Z_3|x_3,{\theta }^t)\cdots P(Z_N|x_N,{\theta }^t)\\ & =\sum _{Z_2}P(Z_2|x_2)\sum _{Z_3}P(Z_3|x_3)\sum _{Z_4}P(Z_4|x_4)\cdots \sum _{Z_N}P(Z_N|x_N)\\ & =1\cdot 1\cdots 1\cdot 1\\ & 最后只剩下\\ & \sum _{Z_1}logP(x_1,Z_1|\theta )P(Z_1|x_1,{\theta }^t)\end{array}$$

化简后的$Q(\theta ,{\theta }^t)$代入:
$$P(x,Z)=P(Z)P(x|z)=p_Z\mathcal{N}(x|{\mu }_Z,{\sigma }_Z)$$
$$p(Z|x)=\frac{p(x,Z)}{p(x)}=\frac{p_Z\mathcal{N}(x|{\mu }_Z,{\Sigma }_Z)}{{\sum }_{k=1}^K{p}_k\mathcal{N}(x|{\mu }_k,{\Sigma }_k)}$$
有:
$$Q(\theta ,{\theta }^t)=\sum _{i=1}^N\sum _{Z_i}log[p_{Z_i}\mathcal{N}(x_i|{\mu }_{Z_i},{\Sigma }_{Z_i})]\frac{P_{Z_i}\mathcal{N}(x_i|{\mu }_i,{\Sigma }_{Z_i})}{{\sum }_{k=1}^K{p}_k\mathcal{N}(x_i|{\mu }_k,{\Sigma }_k)}$$
M-step: 求解公式中的参数,
$$Q=\sum _{k=1}^K\sum _{i=1}^N[log{p}_k+log\mathcal{N}(x_i|{\mu }_{Z_i},{\Sigma }_{Z_i})]p(Z_i=k|x_i,{\theta }^t)$$
先求解$p_k^{t+1}$,$p_k^{t+1}$的求解满足约束条件${\sum }_{k=1}^K{p}_k=1$;
$$L=\sum _{k=1}^K\sum _{i=1}^{N}log{p}_{k}p(z_i=k|x_i,{\theta }^t)+\lambda (\sum _{k=1}^K{p}_k-1)$$
没有了$log\mathcal{N}(x_i|{\mu }_{Z_i},{\Sigma }_{Z_i})p(Z_i=k|x_i,{\theta }^t)$项其对于$p_k$而言是常数项。L函数对$p_k$求偏导有:
$$\frac{\partial }{\partial p_k}L=\sum _{i=1}^N\frac{1}{p_k}p(z_i=k|x_i,{\theta }^t)+\lambda =0$$
两边同乘$p_k$以求解$\lambda$有:
$$\sum _{i=1}^{N}p(z_i=k|x_i,{\theta }^t)+\lambda p_k=0$$
利用${\sum }_{k=1}^K{p}_k=1$
$$\begin{array}{rl}& \sum _{k=1}^K\sum _{i=1}^{N}p(z_i=k|x_i,{\theta }^t)+\lambda \sum _{k=1}^K{p}_k=0\\ & \lambda =-\sum _{i=1}^N\underset{1}{\underbrace{\sum _{k=1}^{K}p(z_i=k|x_i,{\theta }^t)}}=-N\end{array}$$
将$\lambda$回代有:
$$p_k^{t+1}=\frac{{\sum }_{i=1}^{N}p(z_i=k|x_i,{\theta }^t)}{N}$$

${\mu }_k$与${\Sigma }_k$的求解则与MLE中的类似(使用矩阵的求导法则,另外注意在广义EM中提到的后一参数用更新后的前一参数更新的思路):
$$\begin{array}{rl}& \sum _{k=1}^K\sum _{i=1}^N[log\mathcal{N}(x_i|{\mu }_{Z_i},{\Sigma }_{Z_i})]p(Z_i=k|x_i,{\theta }^t)\\ & =\sum _{k=1}^K\sum _{i=1}^N[log(\frac{1}{\sqrt{2\pi }})-log|{\Sigma }_k|-\frac{1}{2}(x_n-{\mu }_k{)}^T{\Sigma }_k^{-1}(x_n-{\mu }_k)]p(Z_i=k|x_i,{\theta }^t)\end{array}$$
对${\mu }_k$求偏导有:
$${\mu }_k^{t+1}=\frac{{\sum }_{i=1}^{N}p(Z_i=k|x_i,{\theta }^t)x_i}{{\sum }_{i=1}^{N}p(Z_i=k|x_i,{\theta }^t)}$$
对${\Sigma }_k$求偏导并注意在外层乘上${\sum }_{i=1}^{N}p(z_i=k|x_i,{\theta }^t)$有:
$${\Sigma }_k^{t+1}=\frac{{\sum }_{i=1}^{N}p(z_i=k|x_i,{\theta }^t)(x_i-{\mu }_k^{t+1})(x_i-{\mu }^{t+1}{)}^T}{{\sum }_{i=1}^{N}p(Z_i=k|x_i,{\theta }^t)}$$

GMR: GMM与高斯条件分布的结合:
$${\pi }_{y|x_k}=\frac{{\mathcal{N}}_k(x|{\mu }_{xk},{\Sigma }_{xk})}{{\Sigma }_{l=1}^K{\mathcal{N}}_l(x|{\mu }_{xl},{\Sigma }_{xl})}$$
$$p(y|x)={\Sigma }_{k=1}^K{\pi }_{y|x_k}{\mathcal{N}}_k(y|{\mu }_{y|x_k},{\Sigma }_{y|x_k})$$