EM (Expectation–Maximization) Algorithm 思路分析及推导

往期文章链接目录

Jensen’s inequality

• Theorem: Let $f$ be a convex function, and let $X$ be a random variable. Then:

$$E[f(X)]\ge f(E[X])$$

\quad Moreover, if $f$ is strictly convex, then $E[f(X)]=f(E[X])$ holds true if and
only if $X$ is a constant.

• Later in the post we are going to use the following fact from the Jensen’s inequality:
  Suppose ${\lambda }_j\ge 0$ for all $j$ and ${\sum }_j{\lambda }_j=1$, then

$$\log \sum _j{\lambda }_j{y}_j\ge \sum _j{\lambda }_{j}log{y}_j$$

\quad where the $\log$ function is concave.

Overview of Expectation–Maximization (EM) algorithm

In this post, let $Y$ be a set of observed data, $Z$ a set of unobserved latent data, and $\theta$ the unknown parameters.

(After this post, you will be comfortable with the following description about the EM algorithm.)

Expectation–Maximization (EM) algorithm is an iterative method to find (local) maximum likelihood estimation (MLE) of $L(\theta )=p(Y|\theta )$, where the model depends on unobserved latent variables $Z$.

Algorithm:

1. Initialize peremeters ${\theta }_0$.

Iterate between steps 2 and 3 until convergence:

2. an expectation (E) step, which creates a function $Q(\theta ,{\theta }_i)$ for the expectation of the log-likelihood $\log p(Y,Z|\theta )$ evaluated using the current conditional distribution of
   $Z$ given $Y$ and the current estimate of the parameters ${\theta }_i$, where

$$\begin{array}{rl}Q(\theta ,{\theta }_i)& =\sum _{Z}P(Z|Y,{\theta }_i)\cdot \log p(Y,Z|\theta )\\ & =E_{Z\sim P(Z|Y,{\theta }_i)}[\log p(Y,Z|\theta )]\end{array}$$

3. A maximization (M) step, which computes parameters maximizing the expected log-likelihood $Q(\theta ,{\theta }_i)$ found on the $E$ step and then update parameters to ${\theta }_{i+1}$.

These parameter-estimates are then used to determine the distribution of the latent variables in the next $E$ step. We say it converges if the increase in successive iterations is smaller than some tolerance parameter.

In general, multiple maxima may occur, with no guarantee that the global maximum will be found.

Intuition: Why we need EM algorithm

Sometimes maximizing the likelihood $\ell (\theta )$ explicitly might be difficult since there are some unknown latent variables. In such a setting, the EM algorithm gives an efficient method for maximum likelihood estimation.

Complete Case v.s. Incomplete Case

Complete case

$(Y,Z)$ is observable, and the log likelihood can be written as

$$\begin{array}{rl}\ell (\theta )& =\log p(Y,Z|\theta )\\ & =\log p(Z|\theta )\cdot p(Y|Z,\theta )\\ & =\log p(Z|\theta )+\log p(Y|Z,\theta )\end{array}$$

We subdivide our task of maximizing $\ell (\theta )$ into two sub-tasks. Note that in both $\log p(Z|\theta )$ and $\log p(Y|Z,\theta )$, the only unknown parameter is $\theta$. They are just two standard MLE problems which could be easily solved by methods such as gradient descent.

Incomplete case

$(Y)$ is observable, but $(Z)$ is unknown. We need to introduce the marginal distribution of variable $Z$:

$$\begin{array}{rl}\ell (\theta )& =\log p(Y|\theta )\\ & =\log \sum _{Z}p(Y,Z|\theta )\\ & =\log \sum _{Z}p(Z|\theta )\cdot p(Y|Z,\theta )\end{array}$$

Here we have a summation inside the log, so it’s hard to use optimization methods or take derivatives. This is the case where EM algorithm comes into aid.

EM Algorithm Derivation (Using MLE)

Given the observed data $Y$, we want to maximize the likelihood $\ell (\theta )=p(Y|\theta )$, and it’s the same as maximizing the log-likelihood $\log p(Y|\theta )$. Therefore, from now on we will try to maximize the likelihood

$$\ell (\theta )=\log p(Y|\theta )$$

by taking the unknown variable $Z$ into account, we rewrite the objective function as

$$\begin{array}{rl}\ell (\theta )& =\log p(Y|\theta )\\ & =\log \sum _{Z}p(Y,Z|\theta )\\ & =\log \sum _{Z}p(Y|Z,\theta )\cdot p(Z|\theta )\end{array}$$

Note that in the last step, the $\log$ is outside of the $\sum$, which is hard to compute and optimize. Check out my previous post to know why we prefer to have $\log$ inside of $\sum$, instead of outside. So later we would find a way to approximate it (Jensen’s inequality).

Suppose we follow the iteration step 2 (E) and 3 (M) repeatedly, and have updated parameters to ${\theta }_i$, then the difference between $\ell (\theta )$ and our estimate $\ell ({\theta }_i)$ is $\ell (\theta )-\ell ({\theta }_i)$. You can think of this difference as the improvement that later estimate of $\theta$ tries to achieve. So our next step is to find ${\theta }_i$