Maximum-Likelihood Expectation-Maximization (ML-EM)

最新推荐文章于 2022-06-17 21:05:37 发布

mmgd

最新推荐文章于 2022-06-17 21:05:37 发布

阅读量1.4k

点赞数

分类专栏： Machine Learning

本文链接：https://blog.csdn.net/mingge_deng/article/details/50526769

版权

Machine Learning 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

I. Notations

$X = \{x_1, x_2, ..., x_N\}\quad$ i.i.d. observed variables

$Z = \{z_1, z_2, ..., z_N\}\quad$ latent variables

$\Theta^{(t)} \ \$ The estimate of the parameters at iteration $t$

$\mathcal{l}(\Theta) \$ The marginal log-likelihood $\log p(X|\Theta)$

II. Derivations

1. Maximum-Likelihood:

Θ ̂ = arg max Θ log p (X | Θ) = arg max Θ \sum i N log p (x i | Θ) = arg max Θ \sum i N log \sum k K P (x i, z = k | Θ)

$\hat{\Theta} = \arg\max\limits_{\Theta}\log p(X | \Theta) = \arg\max\limits_{\Theta}\sum_i^N \log p(x_i | \Theta) = \arg\max\limits_{\Theta}\sum_i^N \log \sum_k^K P(x_i, z =k | \Theta)$
which is hard to compute with a gradient method.

l (Θ) = log p (X | Θ) = \sum i N log \sum k K P (x i, z = k | Θ) = \sum i N log \sum k K q (z = k | x i, Θ) P ( x i , z = k | Θ ) q ( z = k | x i , Θ )

$\mathcal{l}(\Theta) = \log p(X | \Theta) = \sum_i^N \log \sum_k^K P(x_i, z = k | \Theta) = \sum_i^N \log \sum_k^K q(z =k \ | \ x_i, \Theta) \frac{P(x_i, z = k | \Theta)}{q(z = k \ | \ x_i, \Theta)}$

\geq \sum i N \sum k K q (z = k | x i, Θ) log P ( x i , z = k | Θ ) q ( z = k | x i , Θ ) \equiv Q (q, Θ)

$\geq \sum_i^N \sum_k^K q(z =k \ | \ x_i, \Theta) \log \frac{P(x_i, z=k | \Theta)}{q(z=k \ | \ x_i, \Theta)} \equiv Q(q, \Theta)$
where

q(z | x,Θ) $q(z \ | \ x, \Theta)$ is an arbitrary density over

Z $Z$ , and the inequality is given by Jessen’s inequality, i.e,

Ef(x)≤f(E(x)) $Ef(x) \leq f(E(x))$ for convex function and

Ef(x)≥f(E(x)) $Ef(x) \geq f(E(x))$ for concave function. Here

f(x)=log(x) $f(x) = log(x)$ is a concave function.

2. Expectation-Maximization:

Thus we have the lower bound of target function $\mathcal{l}(\Theta)$ . Instead of maximizing $\mathcal{l}(\Theta)$ directly, EM maximizes the lower-bound $Q(q, \Theta)$ via coordinate ascent:

E - s t e p : q (t + 1) = arg max q Q (q, Θ t)

$\bf{E-step :} \quad q^{(t+1)} = \arg\max\limits_{q} Q(q, \Theta^{t})$

M - s t e p : Θ (t + 1) = arg max Θ Q (q (t + 1), Θ)

$\bf{M-step :} \quad \Theta^{(t+1)} = \arg\max\limits_{\Theta} Q(q^{(t+1)}, \Theta)$

E-Step: compute $q^{(t+1)} = \arg\max\limits_{q} Q(q, \Theta^{t})$ with constraint $\sum_k^K q(z=k \ | \ x, \Theta) = 1$ (arbitrary density function over $Z$ ), by introducing the lagrange multiplier $\lambda$ , we define

G (q) = λ (1 - \sum k K q (z = k | x, Θ)) + \sum k K q (z = k | x, Θ) log P (x, z = k | Θ) - \sum k K q (z = k | x, Θ) log q (z = k | x, Θ)

\partial G ( q ) \partial q = - λ + log P (x, z = k | Θ) - log q (z = k | x, Θ) - 1 = 0

$\frac{\partial G(q)}{\partial q} = -\lambda + \log P(x,z=k|\Theta) - \log q(z=k \ | \ x, \Theta) - 1 = 0$

q (z = k | x, Θ) \propto P (x, z = k | Θ) = P ( x , z = k | Θ ) \sum K k ' P ( x , z = k ' | Θ ) = P (z = k | x, Θ)

$q(z=k \ | \ x, \Theta) \propto P(x,z=k \ | \ \Theta) = \frac{P(x,z=k |\Theta)}{\sum_{k'}^K P(x,z=k'|\Theta)} = P(z=k | x, \Theta)$

thus, $q = P(z | x, \Theta)$ give the closest lower bound of $\mathcal{l}(\Theta)$

M-Step: update parameters $\theta$ , with

Θ (t + 1) = arg max Θ \sum i N \sum k K P (z = k | x i, Θ (t)) log P ( x i , z = k | Θ ) P ( z = k | x i , Θ ( t ) )

$\Theta^{(t+1)} = \arg\max\limits_{\Theta} \sum_i^N \sum_k^K P(z=k \ | \ x_i, \Theta^{(t)}) \log \frac{P(x_i, z=k | \Theta)}{P(z=k \ | \ x_i, \Theta^{(t)})}$

= arg max Θ \sum i N \sum k K P (z = k | x i, Θ (t)) log P (x i, z = k | Θ)

$= \arg\max\limits_{\Theta} \sum_i^N \sum_k^K P(z=k \ | \ x_i, \Theta^{(t)}) \log P(x_i, z=k | \Theta)$

III. Applications to Gaussian Mixture Models(GMMs)

For general mixture models, we have

P (x | Θ) = \sum k K P (x, z = k | Θ) = \sum k K P (z = k | Θ) P (x | z = k, Θ)

$P(x | \Theta) = \sum_k^K P(x, z=k | \Theta) = \sum_k^K P(z=k | \Theta) P(x | z=k, \Theta)$
For Gaussian Mixture Models (GMMs), we have

P (x | Θ) = \sum k K π k  (x | μ k, Σ k) w i t h \sum k K π k = 1

$P(x | \Theta) = \sum_k^K \pi_k \ \mathcal{N}(x \ | \ \mu_k, \Sigma_k) \quad with \quad \sum_k^K \pi_k = 1$

E-Step for GMMs:

Define $q_{i,k} \equiv P(z=k | x_i, \Theta)$ , then

q i, k = P ( z = k , x i , | Θ ) \sum K k ' P ( z = k ' , x i , | Θ ) = π k  ( x i | μ k , Σ k ) \sum K k ' π k '  ( x i | μ k ' , Σ k ' )

$q_{i,k} = \frac{P(z=k, x_i, | \Theta)}{\sum_{k'}^K P(z=k', x_i, | \Theta)} = \frac{\pi_k \ \mathcal{N}(x_i \ | \ \mu_k, \Sigma_k)}{ \sum_{k'}^K \pi_{k'} \ \mathcal{N}(x_i \ | \ \mu_{k'}, \Sigma_{k'}) }$

M-Step for GMMs:
update the parameters which maximizes the Log-likelihood below

Θ (t + 1) = arg max Θ \sum i N \sum k K q i, k log P (x i, z = k | Θ) = arg max Θ \sum i N \sum k K q i, k log (π k  (x i | μ k, Σ k))

$\Theta^{(t+1)} = \arg\max\limits_{\Theta} \sum_i^N \sum_k^K q_{i,k} \log P(x_i, z=k | \Theta) = \arg\max\limits_{\Theta} \sum_i^N \sum_k^K q_{i,k} \log\Big(\pi_k \ \mathcal{N}(x_i \ | \ \mu_k, \Sigma_k)\Big)$
with subject to

∑Kkπk=1 $\sum_k^K \pi_k = 1$ , introducing the lanrange multiplier into the objective function, we thus have

G (Θ, λ) = λ (1 - \sum k K π k) + \sum i N \sum k K q i, k log π k + \sum i N \sum k K q i, k log  (x i | μ k, Σ k)

$G(\Theta, \lambda) = \lambda(1-\sum_k^K \pi_k) + \sum_i^N \sum_k^K q_{i,k} \log\pi_k + \sum_i^N \sum_k^K q_{i,k} \log \mathcal{N}(x_i \ | \ \mu_k, \Sigma_k)$

\partial G ( Θ , λ ) \partial π k = 0 a n d \sum k K π k = 1 ⟹ π k = \sum N i q i , k \sum K k \sum N i q i , k = \sum N i q i , k N

$\frac{\partial G(\Theta, \lambda)}{\partial \pi_k} = 0 \quad and \quad \sum_k^K \pi_k = 1 \quad \Longrightarrow \pi_k = \frac{\sum_i^N q_{i,k}}{\sum_k^K \sum_i^N q_{i,k}} = \frac{\sum_i^N q_{i,k}}{N}$

\partial G ( Θ , λ ) \partial μ k = 0 ⟹ μ k = \sum N i q i , k x i \sum N i q i , k

$\frac{\partial G(\Theta, \lambda)}{\partial \mu_k} = 0 \quad \Longrightarrow \mu_k = \frac{\sum_i^N q_{i,k} x_i}{\sum_i^N q_{i,k}}$

\partial G ( Θ , λ ) \partial Σ k = 0 ⟹ Σ k = \sum N i q i , k ( x i - μ k ) ( x i - μ k ) T \sum N i q i , k

$\frac{\partial G(\Theta, \lambda)}{\partial \Sigma_k} = 0 \quad \Longrightarrow \Sigma_k = \frac{\sum_i^N q_{i,k} (x_i-\mu_k)(x_i-\mu_k)^T}{\sum_i^N q_{i,k}}$