EM算法之高斯混合模型(二)

最新推荐文章于 2022-04-07 20:00:39 发布

hadoove

最新推荐文章于 2022-04-07 20:00:39 发布

阅读量317

点赞数

分类专栏：算法 em算法混合高斯模型文章标签：算法 EM

本文链接：https://blog.csdn.net/u013817676/article/details/78617103

版权

算法同时被 3 个专栏收录

7 篇文章 0 订阅

订阅专栏

em算法

2 篇文章 0 订阅

订阅专栏

混合高斯模型

2 篇文章 0 订阅

订阅专栏

EM参数求解

我们将GMM带入 $\theta^{(g+1)}$ 中

θ (g + 1) = a r g m a x θ \int z l n {P (X, z | θ) P (z | X, θ (g))} d z (6.1)

$\theta^{(g+1)} = {argmax}_\theta\int_zln\left\{P(X,z|\theta)P(z|X,\theta^{(g)})\right\}dz \quad(6.1)$
其中

P (Z | X, θ (g)) = \prod i = 1 N P (z i | x i, θ (g)) = P ( x i | z i ) P ( z i ) \sum k i P ( x i | z i ) P ( z i ) = N ( x i | μ z i , Σ z i ) π z i \sum k l N ( x i | μ l , Σ l ) π l (6.2)

$P(Z|X,\theta^{(g)}) = \prod_{i=1}^N P(z_i|x_i,\theta^{(g)}) = \frac{P(x_i|z_i)P(z_i)}{\sum_i^kP(x_i|z_i)P(z_i)} =\frac{N(x_i|\mu_{z_i},\Sigma_{z_i})\pi_{z_i}}{\sum_l^k N(x_i|\mu_l,\Sigma_l)\pi_l} \quad (6.2)$

P (X, z | θ) = \prod i = 1 N P (x i, z i | θ) = \prod i = 1 N P (x i | z i, θ) P (z i | θ) = \prod i = 1 N π i N (μ z i, Σ z i) (6.3)

$P(X,z|\theta) = \prod_{i=1}^N P(x_i,z_i|\theta) = \prod_{i=1}^N P(x_i|z_i,\theta)P(z_i|\theta) =\prod_{i=1}^N \pi_i N(\mu_{z_i},\Sigma_{z_i}) \quad (6.3)$
将6.2和6.3带入6.1
E-step：
确定Q函数，求函数的期望

\sum z 1 = 1 k \sum z 2 = 1 k . . . \sum z N = 1 k \sum (i = 1) N [l n π z i + l n (N (x i | μ z i, Σ z i))] \prod i = 1 N P (z i | x i, θ (g)) (6.4)

$\sum_{z_1=1}^k\sum_{z_2=1}^k...\sum_{z_N=1}^k \sum_{(i=1)}^N[ln\pi_{z_i} + ln(N(x_i|\mu_{z_i},\Sigma_{z_i}))]\prod_{i=1}^NP(z_i|x_i,\theta^{(g)}) \quad (6.4)$
令

f i (z i) = l n π z i + l n (N (x i | μ z i, Σ z i) (6.5)

$f_i(z_i) = ln\pi_{z_i} + ln(N(x_i|\mu_{z_i},\Sigma_{z_i}) \quad (6.5)$

P (Z) = \prod i N P (z i | x i, θ (g)) (6.6)

$P(Z) = \prod_i^N P(z_i|x_i,\theta^{(g)}) \quad (6.6)$
那么式子6.4可以化解为：

\sum z 1 = 1 k \sum z 2 = 1 k . . . \sum z N = 1 k (f 1 (z 1) + f 2 (z 2) + . . . + f N (z N)) P (z 1, z 2, . . . z N) = \sum z 1 = 1 k f 1 (z 1) \sum z 2 = 1 k . . . \sum z N = 1 k P (z 1, z 2, . . ., z N) + . . . = \sum z 1 = 1 k f 1 (z 1) \sum z 2 = 1 k . . . \sum z N = 1 k P (z 1) P (z 2, . . ., z N) + . . . = \sum z 1 = 1 k f 1 (z 1) P (z 1) + . . . . = \sum i = 1 N \sum z 1 = 1 k (l n π i + l n (N (x i | μ i, Σ i))) P (z i | x i, θ (g)) (6.7)

$\sum_{z_1=1}^k\sum_{z_2=1}^k...\sum_{z_N=1}^k(f_1(z_1)+f_2(z_2)+...+f_N(z_N))P(z_1,z_2,...z_N)\\ =\sum_{z_1=1}^kf_1(z_1)\sum_{z_2=1}^k...\sum_{z_N=1}^kP(z_1,z_2,...,z_N) + ...\\ =\sum_{z_1=1}^kf_1(z_1)\sum_{z_2=1}^k...\sum_{z_N=1}^kP(z_1)P(z_2,...,z_N) + ...\\ =\sum_{z_1=1}^kf_1(z_1)P(z_1) + .... \\ =\sum_{i=1}^N\sum_{z_1=1}^k(ln\pi_i+ln(N(x_i|\mu_i,\Sigma_i)))P(z_i|x_i,\theta^{(g)}) \quad(6.7)$

M-step
迭代的M步是求函数 $Q(\theta,\theta^{(i)})$ 对 $\theta$ 求极大值。

(6.7) = \sum i = 1 N \sum l = 1 k (l n π l + l n (N (x l | μ l, Σ l))) P (l | x i, θ (g)) / / z i - > l = \sum i = 1 N \sum l = 1 k l n (π l) P (l | x i, θ (g)) + \sum i = 1 N \sum l = 1 k l n (N (x i | μ l, Σ l) P (l | x i, θ (g))) (6.8)

$(6.7) = \sum_{i=1}^N\sum_{l=1}^k(ln\pi_l+ln(N(x_l|\mu_l,\Sigma_l)))P(l|x_i,\theta^{(g)}) \quad \quad//z_i -> l\\ =\sum_{i=1}^N\sum_{l=1}^kln(\pi_l)P(l|x_i,\theta^{(g)}) + \sum_{i=1}^N\sum_{l=1}^kln(N(x_i|\mu_l,\Sigma_l)P(l|x_i,\theta^{(g)})) \quad(6.8)$

第一部分求解

那我们分别对(6.8)中的两个部分求最大化。

a r g m a x \sum i = 1 N \sum l = 1 k l n (π l) P (l | x i, θ (g)) (6.9) s . t \sum i π i = 1

$argmax\sum_{i=1}^N\sum_{l=1}^kln(\pi_l)P(l|x_i,\theta^{(g)}) \quad(6.9)\\ s.t \sum_i\pi_i = 1$
这边的话，我们拉格朗日的方法求解参数

L M (π 1, π 2, . . . ., π k, λ) = \sum l = 1 k l n (π i) P (l | x i, θ (g)) + λ (\sum l = 1 k π i - 1) (6.10)

$LM(\pi_1,\pi_2,....,\pi_k,\lambda)\\ =\sum_{l=1}^kln(\pi_i)P(l|x_i,\theta^{(g)})+\lambda(\sum_{l=1}^k\pi_i-1) \quad (6.10)$
最后求解得：

π l = 1 N \sum i = 1 N P (l | x i, θ (g)) (6.11)

$\pi_l = \frac 1 N \sum_{i=1}^N P(l|x_i,\theta^{(g)}) \quad (6.11)$
那这个表达式其实就是对(6.2)的值求平均值，就是求某个

xi $x_i$ 属于第

zi $z_i$ 个高斯的权值除以该值属于所有

∑k1zi $\sum_1^kz_i$ 个高斯的权值，然后对k个z求和再求平均值。

第二部分求解

Q (θ, θ (g)) = S (μ l, Σ - l 1) = \sum l = 1 k \sum i = 1 N l n (N (x i | μ l, Σ l)) P (l | x i, θ (g)) = \sum i N - 1 2 l n (| Σ l |) - 1 2 (x i - μ i) T Σ - 1 (x - μ l) P (l | x i, θ (g)) (6.12) = = > S (μ l, Σ) = - T r (Σ - 1 l 2 \sum i = 1 N (x i - μ l) (x - μ l) T P (l | x i, θ (g))) + c o n s t a n t = = > \partial S ( μ l , Σ ) \partial μ l = 2 Σ - 1 - d i a g ( Σ - 1 ) 2 \sum i = 1 N 2 (x i - μ l) P (l | x i, θ (g)) = 0 = = > \sum i = 1 N x i P (l | x i, θ (g)) = μ l \sum i = 1 N P (l | x i, θ (g)) = = > μ l = \sum N i = 1 P ( l | x i , θ ( g ) \sum N i = 1 x i P ( l | x i , θ ( g (6.13) S (μ l, Σ - 1 l) = \sum i = 1 N (- 1 2 l n (| Σ l |) - 1 2 (x i - μ l) T Σ - 1 (x - μ l)) P (l | x i, θ (g)) (6.14)

$Q(\theta,\theta^{(g)})=S(\mu_l,\Sigma_l^-1)=\sum_{l=1}^k\sum_{i=1}^Nln(N(x_i|\mu_l,\Sigma_l))P(l|x_i,\theta^{(g)}) \\ =\sum_i^N-\frac 1 2ln(|\Sigma_l|)-\frac 1 2(x_i-\mu_i)^T\Sigma^{-1}(x-\mu_l)P(l|x_i,\theta^{(g)}) \quad(6.12)\\ ==>S(\mu_l,\Sigma) = -Tr(\frac{\Sigma_l^{-1}}{2}\sum_{i=1}^N(x_i-\mu_l)(x-\mu_l)^TP(l|x_i,\theta^{(g)}))+constant\\ ==>\frac{\partial S(\mu_l,\Sigma)}{\partial \mu_l}=\frac{2\Sigma^{-1}-diag(\Sigma^{-1})}{2} \sum_{i=1}^N2(x_i-\mu_l)P(l|x_i,\theta^{(g)}) =0 \\ ==> \sum_{i=1}^Nx_iP(l|x_i,\theta^{(g)}) = \mu_l\sum_{i=1}^N P(l|x_i,\theta^{(g)}) \\ ==> \mu_l = \frac{\sum_{i=1}^N P(l|x_i,\theta^{(g})}{\sum_{i=1}^N x_i P(l|x_i,\theta^{(g}} \quad(6.13)\\ S(\mu_l,\Sigma_l^{-1}) = \sum_{i=1}^N(-\frac 1 2ln(|\Sigma_l|)-\frac 1 2(x_i-\mu_l)^T\Sigma^{-1}(x-\mu_l))P(l|x_i,\theta^{(g)}) \quad(6.14)$
我们用

Σ $\Sigma$ 代替

Σ−1 $\Sigma^{-1}$ 因为在对

ln(X) $ln(X)$ 求导后，结果为

X−1 $X^{-1}$

(6.14) = (\sum i = 1 N l n (| Σ - 1 l | P (l | x i, θ (g)) - 1 2 t r (Σ - 1 \sum i = 1 N (x i - μ l) (x - μ l) T P (l | x i, θ (g))))

$(6.14) = \left( \sum_{i=1}^Nln(|\Sigma_l^{-1}|P(l|x_i,\theta^{(g)})-\frac 1 2 tr\left(\Sigma^{-1}\sum_{i=1}^N(x_i-\mu_l)(x-\mu_l)^T P(l|x_i,\theta^{(g)})\right)\right)$
令

Ml=∑Ni=1(xi−μl)(x−μl)TP(l|xi,θ(g)) $M_l = \sum_{i=1}^N(x_i-\mu_l)(x-\mu_l)^T P(l|x_i,\theta^{(g)})$

= = > \partial S ( μ l , Σ ) \partial Σ - 1 l = 2 \sum N i = 1 Σ l P ( l | x i , θ ( g ) ) - \sum N i = 1 d i a g ( Σ ) P ( l | x i , θ ( g ) ) 2 - 2 M l - d i a g ( M l ) 2 = 0 = = > 2 (\sum i = 1 N Σ P (l | x i, θ (g)) - M l) - \sum i = 1 N d i a g (Σ P (l | x i, θ (g)) - M l) = 0 = = > \sum i = 1 N Σ P (l | x i, θ (g)) - M l = 0 = = > Σ = \sum N i = 1 M l \sum N i = 1 P ( l | x i , θ ( g ) ) = \sum N i = 1 ( x i - μ l ) ( x - μ l ) T P ( l | x i , θ ( g ) ) \sum N i = 1 P ( l | x i , θ ( g ) )

$==> \frac{\partial S(\mu_l,\Sigma)}{\partial \Sigma_l^{-1}} = \frac{2\sum_{i=1}^N\Sigma_l P(l|x_i,\theta^{(g)})-\sum_{i=1}^Ndiag(\Sigma)P(l|x_i,\theta^{(g)})}{2} - \frac{2M_l-diag(M_l)}{2} = 0\\ ==>2(\sum_{i=1}^N\Sigma P(l|x_i,\theta^{(g)})-M_l) - \sum_{i=1}^N diag(\Sigma P(l|x_i,\theta^{(g)})-M_l) = 0\\ ==>\sum_{i=1}^N\Sigma P(l|x_i,\theta^{(g)})-M_l = 0 \\ ==>\Sigma = \frac{\sum_{i=1}^N M_l}{\sum_{i=1}^NP(l|x_i,\theta^{(g)})} = \frac{\sum_{i=1}^N(x_i-\mu_l)(x-\mu_l)^TP(l|x_i,\theta^{(g)})}{\sum_{i=1}^N P(l|x_i,\theta^{(g)})}$

我们最大化 $\mu$ , $\Sigma$ 意味着更新 $\theta^{(g)}$ -> $\theta^{(g+1)}$ :

π (g + 1) l = 1 N \sum i = 1 N P (l | x i, θ (g)) μ (g + 1) l = \sum N i = 1 x i P ( l | x i , θ ( g ) ) \sum N i = 1 P ( l | x i , θ ( g ) ) Σ (g + 1) l = \sum N i = 1 [ x i - μ i + 1 l ] [ x i - μ i + 1 l ] T P ( l | x i , θ ( g ) ) \sum N i = 1 P ( l | x i , θ ( g ) )

$\pi_l^{(g+1)} = \frac 1 N \sum_{i=1}^N P(l|x_i,\theta^{(g)})\\ \mu_l^{(g+1)} = \frac{\sum_{i=1}^N x_i P(l|x_i,\theta^{(g)})}{\sum_{i=1}^N P(l|x_i,\theta^{(g)})}\\ \Sigma_l^{(g+1)} = \frac{\sum_{i=1}^N[x_i-\mu_l^{i+1}][x_i-\mu_l^{i+1}]^T P(l|x_i,\theta^{(g)})}{\sum_{i=1}^N P(l|x_i,\theta^{(g)})}$
那么我们要计算的是：