简介
:本文主要介绍以下两个方面内容:
公式1
公式2
公式3
公式4
公式5
公式6
最终整理可以得到
的表达式如公式7所示:
公式7
公式8
公式9
公式10
公式11
- 混合高斯(Mixtures of Gaussians)和EM算法
- EM算法相关推导证明
混合高斯和EM算法
关于混合高斯可以看
漫谈 Clustering (3): Gaussian Mixture Model 和
混合高斯模型(Mixtures of Gaussians)和EM算法。观察样本数据X满足公式一概率密度函数:
![](https://i-blog.csdnimg.cn/blog_migrate/1c624fe16494987d30166a56a3c4d05b.gif)
其中
表示第i个Gaussian选中的概率,满足
,数据满足第i个Guassian分布的概率为
,那么公式一中未知参数为
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/26288baa11728346a03c1818e2e21e36.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/353869d6936494b7ea434103ab009ead.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/81669d255262bab99950ce79922a1a96.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/f736c8a653f4c4c042c766b0a76fd0f4.gif)
采用
极大似然估计法maximum-likelihood estimation (MLE)求参数,似然函数如公式2所示:
![](https://i-blog.csdnimg.cn/blog_migrate/2120c9eb284602418f4fd3bd85e650e5.gif)
- 公式2对
求导并令其等于0可得:
![](https://i-blog.csdnimg.cn/blog_migrate/80d0c6c16051bb1983ee92dc05a62236.gif)
设一个
后验概率(Posterior probability
)如公式3所示:
![](https://i-blog.csdnimg.cn/blog_migrate/4b81e539ea48f79a6dd7b8aa55041c96.gif)
可以得出
的表达式如公式4所示:
![](https://i-blog.csdnimg.cn/blog_migrate/e2191e15ada24aa110212210fb06195a.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/e2191e15ada24aa110212210fb06195a.gif%3D%5Cfrac%7B%5Csum_%7Bi%3D1%7D%5E%7Bn%7Dp%28%5Calpha_%7Br%7D%5Cmid%20x_%7Bi%7D%29x_%7Bi%7D%7D%7B%5Csum_%7Bi%3D1%7D%5E%7Bn%7Dp%28%5Calpha_%7Br%7D%5Cmid%20x_%7Bi%7D%29%7D)
- 用拉格朗日乘数法Lagrange multiplier求对
。构造拉格朗日乘子如下:
求导可得
易知
,那么最终可以得到
如公式5所示:
![](https://i-blog.csdnimg.cn/blog_migrate/2dc4b4bfdc83391da0b5c3542ec06cb0.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/4a999394af6a4d31a674e051c1b55650.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/4e3668fb62dd0d0f91637be0106ad987.gif)
- 计算
。这里用到单高斯分布极大似然估计的相关结论(详见Pattern Recognition and Machine Learning 练习2.34)。对于单高斯分布,有如公式6所示结论:
其中
。那么公式2对
求导并令其等于0可得:
![](https://i-blog.csdnimg.cn/blog_migrate/1946abd8a9c8a69bd03c4b71ef914e93.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/28b06b54547d73540c0473762936f447.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/348d944f5451df561360d49d18814872.png)
最终整理可以得到
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/28b06b54547d73540c0473762936f447.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/28b06b54547d73540c0473762936f447.gif%3D%5Cfrac%7Bp%28%5Calpha_%7Br%7D%5Cmid%20x_%7Bi%7D%29%28x_i-%5Cmu_%7Br%7D%29%28x_i-%5Cmu_%7Br%7D%29%5E%7BT%7D%7D%7Bp%28%5Calpha_%7Br%7D%5Cmid%20x_%7Bi%7D%29%7D)
我们得到问题相关参数的表达式如公式4、公式5、公式7所示,但是无法根据样本数据直接获得,因为在计算后验概率
无法直接计算。EM算法给出这类问题的解决方法,即先初始化一组参数
,在E-Step计算后验概率
,在M步骤更新参数
. 如算法一所示:
![](https://i-blog.csdnimg.cn/blog_migrate/c3c008f76bcf41baebd3d2c22f9be4f9.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/31d62727980415e5895d389bfee9b1b8.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/c3c008f76bcf41baebd3d2c22f9be4f9.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/31d62727980415e5895d389bfee9b1b8.gif%5E%7Bnew%7D)
算法1:EM算法求解混合高斯
- Initialize:
,
,
- E-Step:
- M-Step:
这部分主要是利用混合高斯引出EM算法,下部分内容详细讲解EM算法推导及相关证明。
EM算法
EM算法是用极大似然估计法求解存在隐含变量(如混合高斯中的
)问题时的一种有效的方法。下面假定观察样本为X,隐含变量为H,联合概率密度
由参数
决定:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/26288baa11728346a03c1818e2e21e36.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/e1bbc49fb7e7fa93333d06acbd1af3ab.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/31d62727980415e5895d389bfee9b1b8.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/614054b8b122272511be99f9889f9dc1.gif)
我们的目标就是极大化公式8来求参数
:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/31d62727980415e5895d389bfee9b1b8.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/33ff7ce06c08aacc81719d7c8681871a.gif%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7Dln%5C%2C%20p%28x%5Cmid%20%5Ctheta%29%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7Dln%5Cint_%7Bh%7D%5C%2C%20p%28x_%7Bi%7D%2Ch%5Cmid%20%5Ctheta%29%5Cmathrm%7Bd%7Dh)
直接求解公式8有难度,通常如果隐含变量H知道,那么公式8的求解将会容易一些。
EM算法对此类问题提供了一个有效的解决方法:
重复构造
的一个下界(E-Step),优化(提高)这个下界(M-Step)。如果构造和优化下界的任务比较容易,那么我们就间接的极大化公式8求得参数
。
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/33ff7ce06c08aacc81719d7c8681871a.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/31d62727980415e5895d389bfee9b1b8.gif)
对于每一个i,我们定义隐含变量满足某种分布的概率
(即满足
和
),那么我们可以得到公式9:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/f0f3a1018fc556ac3e11bb3e4282cee8.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/69ef10f7d00fcbbae40b66401c98edae.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/f0f3a1018fc556ac3e11bb3e4282cee8.gif%5Cgeq%200)
![](https://i-blog.csdnimg.cn/blog_migrate/02711ff7b03e1a13e02c0389e32933ac.gif)
这里公式9最后一步用到了
Jensen不等式,如过f是一个凹函数(如开口向下的抛物线),X是随机变量,有如下结论:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/cf8ce337c7faa6d04b4235f4bd6b0bef.gif)
等式成立的条件是X=E[X]即X是常量。易知lnx是一个凹函数,那么最后一步推导如下:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/0dcdaffb8493a838bf7240edf222c3b5.gif)
给定一组隐含变量的分布
,公式9给出了
的一个下界。对于给定一组参数
等号成立的条件是
,那么我们可以得到
如公式10所示:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/f0f3a1018fc556ac3e11bb3e4282cee8.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/33ff7ce06c08aacc81719d7c8681871a.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/51b8359f970d2bfe2ad4cdc3ac1aed3c.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/a49943ac45ae2308ff55ca42e005ee68.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/f0f3a1018fc556ac3e11bb3e4282cee8.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/ad3854610fff0371816da17c9362194f.gif)
即
定义为给定样本x和参数
时隐含变量h的后验概率。
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/f0f3a1018fc556ac3e11bb3e4282cee8.gif)
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/51b8359f970d2bfe2ad4cdc3ac1aed3c.gif)
给出这个下界后我们可以通过
来优化这个下界:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/51b8359f970d2bfe2ad4cdc3ac1aed3c.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/fc5d192c1df6d9adb9f34a3b585e440d.gif)
最终我们可以得到EM算法如算法2所示:
算法2:EM算法过程
- Initialize:
- E-Step:
- M-Step:
剩余的最后一个问题是如何证明EM算法会收敛,也就是证明
,证明如公式11所示:
![This is the rendered form of the equation. You can not edit this directly. Right click will give you the option to save the image, and in most browsers you can drag the image onto your desktop or another program.](https://i-blog.csdnimg.cn/blog_migrate/15a13368c3b803264d86a4ce261b819f.gif)
其中第一个不等式利用就是公式9结论,第二个不等式是第t+1次迭代式M-Step的过程,最后一个等号就是t次迭代完的结果。至此,关于EM算法的相关推导证明全部结束。
参考文献
3、Gaussian Mixture Model and EM(Expectation Maximization) Algorithm(清华大学PR课件)