EM algorithm

If major in ML, you will see the EM algorithm which divide into two steps

1. E step (Exception)
2. M step (Maximum)

You will be confused only through the step of the algorithm. Let’s explore the inner theory.

Goal

The first thing is to get our aim. eg, we have input and output, we should make a model to fit the data. Our aim is to get the parameter of the model.

we denote $\theta$ is the parameter, $X$ is the input, $Y$ is the output, $Z$ is the latent variable.

So we use the condition model to get the $P(Y|\theta )$, we will use maximum likelihood to solve it.
$$L(\theta )=logP(Y|\theta )$$
If there is no latent variable in the model, we can use maximum likelihood directly.

eg: throw a coin for five times, get three up and two down, to calculate the probability of up？

We will use the maximum likelihood to deal. First define the probability of up is $\theta$ and the output(result) is $Y$. So the likelihood function is
$$L(\theta )=P(Y|\theta )$$

We can also get $P(Y|\theta )$
$$P(y_i|\theta )={\theta }^{y_i}+(1-\theta {)}^{1-y_i}$$

To the end,
$$L(\theta )=\prod _{i=1}^n({\theta }^{y_i}+(1-\theta {)}^{1-y_i})$$

call by value, and $max(L(\theta ))$ by making the derivative equal to zero, we get $\theta =\frac{3}{5}$

BUT in EM algorithm, there is a latent variable in model.

Such as
$$\begin{array}{rl}L(\theta )& =P(Y,Z|\theta )\\ & =P(Y|Z,\theta )P(Z|\theta )\end{array}$$

Then
$$P(Y|\theta )=\prod _{i=1}^n[\pi p^{y_i}(1-p{)}^{1-y_i}+(1-\pi )q^{y_i}(1-q{)}^{1-y_i}]$$

Our aim is to
$$\hat{\theta }=\arg \underset{\theta }{\max }logP(Y|\theta )$$

Proof

We have already get
$$L(\theta )=logP(Y,Z|\theta )=logP(Y|Z,\theta )P(Z|\theta )$$

In order to $max(L(\theta ))$, we max it step by step. So we hope through each step, we can always increase the $L(\theta )$

---

Jensen Inequity
Let f be a convex function, and let X be a random variable.Then:
$$E(f(X))\ge f(E(X))$$

---

[1] : 《统计学习方法》 李航
[2] : cs229 Andrew Ng http://cs229.stanford.edu/notes/cs229-notes8.pdf