机器学习(七)EM算法

最新推荐文章于 2022-09-09 19:42:46 发布

Dynomite

最新推荐文章于 2022-09-09 19:42:46 发布

阅读量374

点赞数

分类专栏：机器学习文章标签：机器学习 EM算法

本文链接：https://blog.csdn.net/Dynomite/article/details/81007866

版权

机器学习专栏收录该内容

10 篇文章 0 订阅

订阅专栏

机器学习(七)EM算法

7.1 EM
已知X为观测变量，Z为隐变量， $\theta$ 为模型参数，欲对 $\theta$ 做极大似然估计

L L (θ | X, Z) = l n P (X, Z | θ) (7.1.1)

$LL(\theta|X,Z)=lnP(X,Z|\theta)\tag{7.1.1}$
Z为隐变量无法直接求解上式，转为求解：

L L (θ | X) = l n P (X | θ) = l n \sum Z P (X, Z | θ) (7.1.2)

$LL(\theta|X)=lnP(X|\theta)=ln\sum_ZP(X,Z|\theta)\tag{7.1.2}$
原型：
基于

θt θ t $\theta^t$ 推测隐变量Z的期望记为

Zt Z t $Z^t$
基于已观测变量X和

Zt Z t $Z^t$ 对于

θ θ $\theta$ 做极大似然估计，记为

θt+1 θ t + 1 $\theta^{t+1}$
重复上诉步骤直至收敛

E-step：
基于参数 $\theta^t$ 计算隐变量Z的概率分布 $P(Z|X,\theta^t)$ ，计算对数似然函数 $LL(\theta|X,Z)$ 关于Z的期望：

Q (θ | θ t) = E Z | X, θ t L L (θ | X, Z) (7.1.3)

$Q(\theta|\theta^t)=E_{Z|X,\theta^t}LL(\theta|X,Z)\tag{7.1.3}$
M-step，参数最大化：

θ t + 1 = a r g m a x θ Q (θ | θ t) (7.1.3)

$\theta^{t+1}=\mathop{argmax}_\theta Q(\theta|\theta^t)\tag{7.1.3}$

7.2 高斯混合聚类
多元高斯分布定义：

p (x | μ, Σ) = 1 ( 2 π ) n 2 | Σ | 1 2 e - 1 2 (x - μ) T Σ - 1 (x - μ) (7.2.1)

$p(x|\mu,\Sigma)=\frac{1}{(2\pi)^{\frac{n}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}\tag{7.2.1}$
多元高斯混合分布：

p M (x) = \sum i = 1 k α i p (x | μ i, Σ i) (7.2.2)

$p_M(x)=\sum_{i=1}^k\alpha_ip(x|\mu_i,\Sigma_i)\tag{7.2.2}$
假设样本的生成过程如下，首先根据

α1,α2...,αk α 1 , α 2 . . . , α k $\alpha_1,\alpha_2...,\alpha_k$ 定义的先验分布选择在高斯主成分，其中

αi α i $\alpha_i$ 为选择第i个主成分的概率，根据被选择概率密度进行采样，生成相应样本
令

zj∈{1,2,..k} z j ∈ { 1 , 2 , . . k } $z_j\in \{1,2,..k\}$ 表示生成样本

xj x j $x_j$ 的高斯混合主成分，先验概率

P(zj=i)=αi P ( z j = i ) = α i $P(z_j=i)=\alpha_i$
后验分布：

p M (z j = i | x j) = = P ( z j = 1 ) p M ( x j | z j = i ) p M ( x j ) α i p ( x j | μ i , Σ i ) \sum k l = 1 α l p ( x j | μ l , Σ l ) (7.2.3)

$\begin{align} p_M(z_j=i|x_j)=&\frac{P(z_j=1)p_M(x_j|z_j=i)}{p_M(x_j)}\notag\\ =&\frac{\alpha_i p(x_j|\mu_i,\Sigma_i)}{\sum_{l=1}^k\alpha_lp(x_j|\mu_l,\Sigma_l)}\tag{7.2.3} \end{align}$
将

pM(zj=i|xj) p M ( z j = i | x j ) $p_M(z_j=i|x_j)$ 记为

γji γ j i $\gamma_{ji}$
每个样本的

xj x j $x_j$ 簇标记记为

λj λ j $\lambda_j$ ：

λ j = a r g m a x i \in {1, 2, . . . k} γ j i (7.2.4)

$\lambda_j=\mathop{argmax}_{i\in \{1,2,...k\}}\gamma_{ji}\tag{7.2.4}$

对数似然：

L L (D) = = l n (\prod j = 1 m p M (x j)) \sum j = 1 M l n (\sum i = 1 k α i p (x j | μ i, Σ i) (7.2.5)

$\begin{align} LL(D) = &ln (\prod_{j=1}^mp_M(x_j))\notag\\ =&\sum_{j=1}^Mln(\sum_{i=1}^k\alpha_ip(x_j|\mu_i,\Sigma_i)\tag{7.2.5} \end{align}$
采用EM算法进行迭代求解：
对

p(xj|μi,Σi) p ( x j | μ i , Σ i ) $p(x_j|\mu_i,\Sigma_i)$ 进行求导：

\partial | X | \partial X = | x | t r (X - 1 d X)

$\frac{\partial |X|}{\partial X}=|x|tr(X^{-1}dX)$

\partial X - 1 \partial X = - X - 1 d X X - 1

$\frac{\partial X^{-1}}{\partial X}=-X^{-1}dXX^{-1}$

\partial p ( x j | μ i , Σ i ) \partial μ i = p (x j | μ i, Σ i) Σ - 1 i (x j - μ i) (7.2.6)

$\begin{align} \frac{\partial p(x_j|\mu_i,\Sigma_i)}{\partial \mu_i}=p(x_j|\mu_i,\Sigma_i)\Sigma_i^{-1}(x_j-\mu_i)\tag{7.2.6}\\ \end{align}$

d p (x j | μ i, Σ i) = + = + + = + 1 ( 2 π ) n 2 d 1 | Σ i | 1 2 \cdot e - 1 2 (x j - μ i) T Σ - 1 i (x j - μ i) 1 ( 2 π ) n 2 | Σ | 1 2 d e - 1 2 (x j - μ i) T Σ - 1 i (x j - μ i) - | Σ | - 1 2 ( 2 π ) n 2 | Σ | 1 2 | Σ i | t r (Σ - 1 i d Σ i) \cdot e - 1 2 (x j - μ i) T Σ - 1 i (x j - μ i) p (x j | μ i, Σ i) \cdot t r (1 2 Σ - 1 i (x j - μ i) (x j - μ i) T (x j - μ i) Σ - 1 i d Σ i) . . . . - 1 2 p (x j | μ i, Σ i) t r (Σ - 1 i d Σ i) 1 2 p (x j | μ i, Σ i) t r (Σ - 1 i (x j - μ i) (x j - μ i) T (x j - μ i) Σ - 1 i d Σ i) (7.2.7)

$\begin{align} dp(x_j|\mu_i,\Sigma_i)=&\frac{1}{(2\pi)^{\frac{n}{2}}}d\frac{1}{|\Sigma_i|^{\frac{1}{2}}}\cdot e^{-\frac{1}{2}(x_j-\mu_i)^T\Sigma_i^{-1}(x_j-\mu_i)}\notag\\ +&\frac{1}{(2\pi)^{\frac{n}{2}}|\Sigma|^{\frac{1}{2}}}de^{-\frac{1}{2}(x_j-\mu_i)^T\Sigma_i^{-1}(x_j-\mu_i)}\notag\\ =&\frac{-|\Sigma|^{-1}}{2(2\pi)^{\frac{n}{2}}|\Sigma|^{\frac{1}{2}}}|\Sigma_i|tr(\Sigma^{-1}_id\Sigma_i)\cdot e^{-\frac{1}{2}(x_j-\mu_i)^T\Sigma_i^{-1}(x_j-\mu_i)}\notag\\ +&p(x_j|\mu_i,\Sigma_i)\cdot tr(\frac{1}{2}\Sigma_i^{-1}(x_j-\mu_i)(x_j-\mu_i)^T(x_j-\mu_i)\Sigma_i^{-1}d\Sigma_i)\notag\\ +& ....\notag\\ =&-\frac{1}{2}p(x_j|\mu_i,\Sigma_i)tr(\Sigma_i^{-1}d\Sigma_i) \notag \\ +& \frac{1}{2}p(x_j|\mu_i,\Sigma_i)tr(\Sigma_i^{-1}(x_j-\mu_i)(x_j-\mu_i)^T(x_j-\mu_i)\Sigma_i^{-1}d\Sigma_i)\tag{7.2.7} \end{align}$

\partial p ( x j | μ i , Σ i ) \partial Σ i = 1 2 p (x j | μ i, Σ i) (Σ - 1 i (x j - μ i) (x j - μ i) T (x j - μ i) Σ - 1 i - Σ - 1 i) (7.2.6)

$\begin{align} \frac{\partial p(x_j|\mu_i,\Sigma_i)}{\partial \Sigma_i}=\frac{1}{2}p(x_j|\mu_i,\Sigma_i)(\Sigma_i^{-1}(x_j-\mu_i)(x_j-\mu_i)^T(x_j-\mu_i)\Sigma_i^{-1}-\Sigma_i^{-1})\tag{7.2.6}\\ \end{align}$

令偏导为0可得，

\partial L L \partial μ i = \sum j = 1 m α i p ( x j | μ i , Σ i ) \sum k i = 1 α i p ( x j | μ i , Σ i Σ - 1 i (x j - μ i) = 0 (7.2.7)

$\frac{\partial LL}{\partial \mu_i}=\sum_{j=1}^m\frac{\alpha_i p(x_j|\mu_i,\Sigma_i)}{\sum_{i=1}^k\alpha_ip(x_j|\mu_i,\Sigma_i}\Sigma_i^{-1}(x_j-\mu_i)=0\tag{7.2.7}$

μ i = \sum m j = 1 γ j i x j \sum m j = 1 γ j i (7.2.8)

$\mu_i=\frac{\sum_{j=1}^m\gamma_{ji}x_j}{\sum_{j=1}^m\gamma_{ji}}\tag{7.2.8}$

\partial L L \partial Σ i = \sum j = 1 m α i p ( x j | μ i , Σ i ) 2 \sum k i = 1 α i p ( x j | μ i , Σ i (Σ - 1 i (x j - μ i) (x j - μ i) T (x j - μ i) Σ - 1 i - Σ - 1 i) = 0 (7.2.9)

$\frac{\partial LL}{\partial \Sigma_i}=\sum_{j=1}^m\frac{\alpha_i p(x_j|\mu_i,\Sigma_i)}{2\sum_{i=1}^k\alpha_ip(x_j|\mu_i,\Sigma_i}(\Sigma_i^{-1}(x_j-\mu_i)(x_j-\mu_i)^T(x_j-\mu_i)\Sigma_i^{-1}-\Sigma_i^{-1})=0\tag{7.2.9}$

Σ i = \sum m j = 1 γ j i ( x j - μ i ) ( x j - μ i ) T \sum m j = 1 γ j i (7.2.10)

$\Sigma_i=\frac{\sum_{j=1}^m\gamma_{ji}(x_j-\mu_i)(x_j-\mu_i)^T}{\sum_{j=1}^m\gamma_{ji}}\tag{7.2.10}$

对 $\alpha_i$ 求导，引入拉个朗日乘子

L L (D) + λ (\sum i = 1 k α i - 1)

$LL(D) + \lambda(\sum_{i=1}^k \alpha_i - 1)$

\partial L L L \partial α i = \sum j = 1 m p ( x j | μ i , Σ i ) \sum k i = 1 α i p ( x j | μ i , Σ i ) + λ = 0 (7.2.11)

$\frac{\partial LLL}{\partial \alpha_i}=\sum_{j=1}^m\frac{p(x_j|\mu_i,\Sigma_i)}{\sum_{i=1}^k\alpha_ip(x_j|\mu_i,\Sigma_i)}+\lambda=0\tag{7.2.11}$
得到：