模式识别 | PRML Chapter 9 Mixture Models and EM

PRML Chapter 9 Mixture Models and EM

9.1 K-means Clustering

Consider the problem of identifying groups or clusters of data points in a multidimensional space.

To describe the assignment of data points to clusters, we define an objective function(distortion measure):

$$J=\sum _{n=1}^N\sum _{k=1}^K{r}_{nk}||x_n-{\mu }_k|{|}^2$$

To find $r_{nk}$ and ${\mu }_k$ so as to minimize $J$, we go through an iterative procedure in which each iteration involves two successive steps corresponding to successive optimizations. The solution will be:

$${\mu }_k=\frac{{\sum }_n{r}_{nk}{x}_n}{{\sum }_n{r}_{nk}}$$

We can also derive an on-line stochastic algorithm by applying the Robbins-Monro procedure to the problem

$${\mu }_k^{new}={\mu }_k^{old}+{\eta }_n(x_n-x_k^{old})$$

We can generalize the K-means algorithm by introducing a more general dissimilarity measure:

$$\tilde{J}=\sum _{n=1}^N\sum _{k=1}^K{r}_{nk}\nu (x_n,{\mu }_k)$$

which gives the K-medoids algorithm.

9.1.1 Image segmentation and compression

9.2 Mixtures of Guassians

We have mentioned the Gaussian mixture model as a simple linear superposition of Gaussian components, we now turn to a formulation of Gaussian mixtures in terms of discrete latent variables.

$$p(x)=\sum _{k=1}^K{\pi }_{k}N(x|{\mu }_k,{\Sigma }_k)$$

The marginal distribution over $z$ is specified in terms of mixing coefficients ${\pi }_k$

$$p(z_k=1)={\pi }_k$$

$$\sum _k{z}_k=1$$

where the parameters { π k } \{\pi_k\} {πk​} must satisfy:

$$0\le {\pi }_k\le 1$$

$$\sum _{k=1}^K{\pi }_k=1$$

in order to be valid probabilities, because $z$ uses a 1-of-K representation, we can also write this distribution in the form:

$$p(z)=\prod _{k=1}^K{\pi }_k^{z_k}$$

Similarly, the conditional distribution of x is a Gaussian:

$$p(x|z)=\prod _{k=1}^K\mathcal{N}(x|{\mu }_k,{\Sigma }_k{)}^{z_k}$$

The joint distribution is given by $p(z)p(x\Vert z)$ and the marginal distribution of $x$ is then obtained by summing the joint distribution over all possible states of $z$ to give:

$$p(x)=\sum _{z}p(z)p(x|z)=\sum _{k=1}^K{\pi }_k\mathcal{N}(x|{\mu }_k,{\Sigma }_k)$$

The corresponding posterior probability will be:

$$\gamma (z_k)\equiv p(z_k=1|x)=\frac{p(z_k=1)p(x|z_k=1)}{{\sum }_{j=1}^{K}p(z_j=1)p(x|z_j=1)}=\frac{{\pi }_k\mathcal{N}(x|{\mu }_k,{\Sigma }_k)}{{\sum }_{j=1}^K{\pi }_j\mathcal{N}(x|{\mu }_j,{\Sigma }_k)}$$

9.2.1 Maximum likelihood

The log of the likelihood function is given by:

$$\ln p(\mathbf{X}|\pi ,\mu ,\Sigma )=\sum _{n=1}^N\ln \left\{\sum _{k=1}^K{\pi }_k\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k)\right\}$$

9.2.2 EM for Gaussian mixtures

An elegant and powerful method for finding maximum likelihood solutions for models with latent variables is called the espectation-maximization algorithm (EM algorithm).

Maximize the likelihood function, we obtain:

$${\mu }_k=\frac{1}{N_k}\sum _{n=1}^N\gamma (z_{nk})x_n$$

where $N_k={\sum }_{n=1}^N\gamma (z_{nk})$.

And for the covariance matrix of a single Gaussian, we obtain:

$${\Sigma }_k=\frac{1}{N_k}\sum _{n=1}^N\gamma (z_{nk})(x_n-{\mu }_k{)}^T$$

For the mixing coefficients (using Lagrange multiplier) we obtain:

$${\pi }_k=\frac{N_k}{N}$$

EM for Gaussian Mixtures

So, given a Gaussian mixture model, the goal is to maximize the likelihood function with respect to the parameters (mean & covariances of the components & mixing coefficients).

• 1. Initialize ${\mu }_k$, $\Sigma$, ${\pi }_k$. Evaluate the initial value of the log likelihood.
• 2. E step. Evaluate the responsibilities using the current parameter values.

$$\gamma (z_{nk})=\frac{{\pi }_k\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k)}{{\sum }_{j=1}^K{\pi }_j\mathcal{N}(x_n|{\mu }_j,{\Sigma }_j)}$$

• 3. M step. Re-estimate the parameters using the current responsibilities.

$${\mu }_k^{new}=\frac{1}{N_k}\sum _{n=1}^N\gamma (z_{nk})x_n$$

$$\sum _k^{new}=\frac{1}{N_k}\sum _{n=1}^N\gamma (z_{nk})(x_n-{\mu }_k^{new})(x_n-{\mu }_k^{new}{)}^T$$

$${\pi }_k^{new}=\frac{N_k}{N}$$

where

$$N_k=\sum _{n=1}^N\gamma (z_{nk})$$

4. Evaluate the log likelihood

$$\ln p(\mathbf{X}|\mu ,\Sigma ,\pi )=\sum _{n=1}^N\ln \left\{\sum _{k=1}^K{\pi }_k\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k)\right\}$$

and check for convergence of either the parameters of the log likelihood. If the convergence criterion is not satisfied return to step 2.

9.3 An Alternative View of EM

The goal of the EM algorithm is to find maximum likelihood solutions for models having latent variables. Set the parameters $\theta$, and the likelihood function is given by:

$$\ln p(\mathbf{X}|\theta )=\ln \left\{\sum _{Z}p(\mathbf{X},\mathbf{Z}|\theta )\right\}$$

In the E step, we use current ${\theta }^{old}$ to find the posterior by which we can find the expectation of the complete-data log likelihood evaluated for some general parameter value $\theta$:

$${\theta }^{new}=\arg \underset{\theta }{\max }\mathcal{Q}(\theta ,{\theta }^{old})$$

In the M step, we determine the revised parameter estimate ${\theta }^{new}$ by maximizing this function:

$$\mathcal{Q}(\theta ,{\theta }^{old})=\sum _{Z}p(\mathcal{Z}|\mathcal{X},{\theta }^{old})\ln p(\mathcal{X},\mathcal{Z}|\theta )$$

9.3.1 Guassian mixtures revisited

The log likelihood function takes the form:

ln ⁡ p ( X , Z ∣ μ , Σ . π ) = ∑ n = 1 N ∑ k = 1 K z n k { ln ⁡ π k + ln ⁡ N ( x n ∣ μ k , Σ k ) } \ln p(\mathbf{X, Z} | \mu, \Sigma. \pi) = \sum_{n=1}^{N}\sum_{k=1}^{K}z_{nk}\{\ln \pi_{k} + \ln\mathcal{N}(x_{n} | \mu_{k}, \Sigma_{k}) \} lnp(X,Z∣μ,Σ.π)=n=1∑N​k=1∑K​znk​{lnπk​+lnN(xn​∣μk​,Σk​)}

Using Bayes’ theorem, we see that the posterior distribution takes the form:

$$p(Z|X,\mu ,\Sigma ,\pi )\propto \prod _{n=1}^N\prod _{k=1}^K[{\pi }_k\mathcal{N}(x_n|{\mu }_k,{\Sigma }_k){]}^{z_{nk}}$$

The expected value of the complete-data log likelihood function is therefore given by:

E Z [ ln ⁡ p ( X , Z ∣ μ , Σ , π ) ] = ∑ n = 1 N ∑ k = 1 K γ ( z n k ) { ln ⁡ π k + ln ⁡ N ( x n ∣ μ k , Σ k ) } E_{Z}[\ln p(X, Z | \mu, \Sigma, \pi)] = \sum_{n=1}^{N}\sum_{k=1}^{K}\gamma(z_{nk})\{\ln \pi_{k} + \ln\mathcal{N}(x_{n} | \mu_{k}, \Sigma_{k}) \} EZ​[lnp(X,Z∣μ,Σ,π)]=n=1∑N​k=1∑K​γ(znk​){lnπk​+lnN(xn​∣μk​,Σk​)}

9.3.2 Relation to K-means

The posterior probabilities for a particular data point $x_n$ are given by:

$$\gamma (z_{nk})=\frac{{\pi }_{k}exp(-||x_n-{\mu }_k|{|}^2/2\epsilon )}{{\sum }_j{\pi }_{j}exp(-||x_n-{\mu }_j|{|}^2/2\epsilon )}$$

9.3.3 Mixtures of Bernoulli distributions

In order to derive the EM algorithm, we first write down the complete-data log likelihood function, which is given by:

$$\ln p(\mathbf{X},\mathbf{Z}|\mu ,\pi )=\sum _{n=1}^N\sum _{k=1}^K{z}_{nk}\left\{\ln {\pi }_k+\sum _{i=1}^D[x_{ni}\ln {\mu }_{ki}+(1-x_{ni})\ln (1-{\mu }_{ki})]\right\}$$

where X = { x n } \mathbf{X}=\{x_n\} X={xn​} and Z = { z n } \mathbf{Z}=\{z_n\} Z={zn​}. Next we take the expectation of the complete-data log likelihood with respect to the posterior distribution of the latent variables to give:

$$E_Z[\ln p(\mathbf{X},\mathbf{Z}|\mu ,\pi )]=\sum _{n=1}^N\sum _{k=1}^K\gamma (z_{nk})\left\{\ln {\pi }_k+\sum _{i=1}^D[x_{ni}\ln {\mu }_{ki}+(1-x_{ni})\ln (1-{\mu }_{ki})]\right\}$$

9.3.4 EM for Bayesian linear regression

The complete-data log likelihood function is then given by:

$$lnp(\mathbf{t},w|\alpha ,\beta )=lnp(\mathbf{t}|w,\beta )+lnp(w|\alpha )$$

Taking the expectation with respect to the posterior distribution of $w$ gives:

$$E[lnp(\mathbf{t},w|\alpha ,\beta )]=\frac{M}{2}ln(\frac{\alpha }{2\pi })-\frac{\alpha }{2}E[w^{T}w]+\frac{N}{2}ln(\frac{\beta }{2\pi })-\frac{\beta }{2}\sum _{n=1}^{N}E[(t_n-w^T{\varphi }_n{)}^2]$$

Setting the derivatives with respect to $\alpha$ or $\beta$ to zero, we obtain the M step re-estimation equation:

$$\alpha =\frac{M}{E[w^{T}w]}=\frac{M}{m_N^T{m}_N+Tr(S_N)}$$

$$\frac{1}{\beta }=\frac{1}{N}(||\mathbf{t}-{\mathbf{\Phi }}_{{\mathbf{m}}_{\mathbf{N}}}|{|}^2+Tr[{\mathbf{\Phi }}^{\mathbf{T}}\mathbf{\Phi }{S}_N])$$

9.4 The EM Algorithm in General

Our goal is to maximize the likelihood function:

$$p(X|\theta )=\sum _{Z}p(X,Z|\theta )$$

Next we introduce a distribution $q(Z)$ defined over the latent variables. For any choice of $q(Z)$, the following decomposition holds:

$$\ln p(X|\theta )=\mathcal{L}(q,\theta )+KL(q||p)$$

where we have defined

$$\mathcal{L}(q,\theta )=\sum _{Z}q(Z)\ln \left\{\frac{p(X,Z|\theta )}{q(Z)}\right\}$$

$$KL(q||p)=-\sum _{Z}q(Z)\ln \left\{\frac{p(Z|X,\theta )}{q(Z)}\right\}$$