GMM & K-means 高斯混合模型和K-means聚类详解

往期文章链接目录

Gaussian mixture model (GMM)

A Gaussian mixture model is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters.

Interpretation from geometry

$p(x)$ is a weighted sum of multiple Gaussian distribution.

$$p(x)=\sum _{k=1}^K{\alpha }_k\cdot \mathcal{N}\left(x|{\mu }_k,{\Sigma }_k\right)$$

Interpretation from mixture model

setup:

• The total number of Gaussian distribution $K$.

• $x$, a sample (observed variable).

• $z$, the distribution of the sample $x$ (a latent variable), where

  • z ∈ { c 1 , c 2 , . . . , c K } z \in \{c_1, c_2, ..., c_K\} z∈{ c1​,c2​,...,cK​}.

  • ${\sum }_{k=1}^{K}p(z=c_k)=1$. We denote $p(z=c_k)$ by $p_k$.

Mixture models are usually generative models, which means new data can be drawn from the distribution of models. Specifically, in the Gaussian Mixture Model (GMM), a new data is generated by first select a class $c_k$ based on the probability distribution of all classes $c$, and then draw a value from the Gaussian distribution of that class. Therefore, we could write $p(x)$ as the following

$$\begin{array}{rl}p(x)& =\sum _{z}p(x,z)\\ & =\sum _{k=1}^{K}p(x,z=c_k)\\ & =\sum _{k=1}^{K}p(z=c_k)\cdot p(x|z=c_k)\\ & =\sum _{k=1}^K{p}_k\cdot \mathcal{N}(x|{\mu }_k,{\Sigma }_k)\end{array}$$

We see that two ways of interpretation reach to the same result.

GMM Derivation

set up

• X: observed data, where $X=(x_1,x_2,...,x_N)$

• $\theta$: parameter of the model, where θ = { p 1 , p 2 , ⋯   , p K , μ 1 , μ 2 , ⋯   , μ K , Σ 1 , Σ 2 , ⋯   , Σ K } \theta=\left\{p_{1}, p_{2}, \cdots, p_{K}, \mu_{1}, \mu_{2}, \cdots, \mu_{K}, \Sigma_{1}, \Sigma_{2}, \cdots, \Sigma_{K}\right\} θ={ p1​,p2​,⋯,pK​,μ1​,μ2​,⋯,μK​,Σ1​,Σ2​,⋯,ΣK​}

• $p(x)={\sum }_{k=1}^K{p}_k\cdot \mathcal{N}(x|{\mu }_k,{\Sigma }_k)$.

• $p(x,z)=p(z)\cdot p(x|z)=p_z\cdot \mathcal{N}(x|{\mu }_z,{\Sigma }_z)$

• $p(z|x)=\frac{p(x,z)}{p(x)}=\frac{p_z\cdot \mathcal{N}(x|{\mu }_z,{\Sigma }_z)}{{\sum }_{k=1}^K{p}_z\cdot \mathcal{N}(x|{\mu }_z,{\Sigma }_z)}$

Solve by MLE

$$\begin{array}{rl}{\hat{\theta }}_{MLE}& =\underset{\theta }{\mathrm{argmax}}p(X)\\ & =\underset{\theta }{\mathrm{argmax}}\log p(X)\\ & =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N\log p\left(x_i\right)\\ & =\underset{\theta }{a}\end{array}$$