ML Note 3 - Unsupervised Learning

无监督学习中，模型的输入不再包括训练样本的期望输出。因此相应 training set 的定义变为
S = { x 1 , x 2 , … , x m } S = \{x_1, x_2, \dots,x_m\} S={x1​,x2​,…,xm​}

本文中其他定义和有监督学习一致。

In this set of notes, the problems are essentially finding clumps and data compression. For finding clumps, we can use k-means to find the center directly, or using EM to model the distribution of each clump. For data compression, we can use factor analysis to estimate the probability of a data being in a specific subspace, or we can find the subspace with PCA.

The k-means Clustering Algorithm

设样本空间有 $k$ 个簇， 算法的目标是找到这些 cluster centroids
$${\mu }_1,{\mu }_2,\dots ,{\mu }_k\in {\mathbb{R}}^n$$

设 $c_i$ 表示样本 $x_i$ 所属簇的下标，定义 distortion function
$$J(c,\mu )=\sum _{i=1}^m||x_i-{\mu }_{c_i}|{|}^2$$

对 $J$ 应用坐标下降可以得到下面的算法
initialize  μ 1 , … , μ k repeat until stable  { for i in 1...m  c i : = arg ⁡ min ⁡ j ∣ ∣ x i − μ j ∣ ∣ 2 for j in 1...k  C j = { k ∣ c k = j } μ j : = ∑ i ∈ C j x i / ∣ C j ∣ } \begin{aligned} & \text{initialize } \mu_1,\dots,\mu_k\\ & \text{repeat until stable } \{\\ & \qquad\text{for i in 1...m } \\ & \qquad\qquad c_i := \arg\min_j ||x_i-\mu_j||^2\\ & \qquad\text{for j in 1...k } \\ & \qquad\qquad C_j = \{k|c_k = j\}\\ & \qquad\qquad\mu_j := \sum_{i \in C_j} x_i / |C_j| \\ & \} \end{aligned} ​initialize μ1​,…,μk​repeat until stable {for i in 1...m ci​:=argjmin​∣∣xi​−μj​∣∣2for j in 1...k Cj​={k∣ck​=j}μj​:=i∈Cj​∑​xi​/∣Cj​∣}​

因为损失函数 $J$ 是非凸的，初始化 $\mu$ 的方法对算法的收敛性有一定影响。常用的方法是选取相距最远的样本作为初始值

1. 选取样本空间中相聚最远的两点，作为两个初始值
2. 当初始值个数不足 $K$ 时，选择第 max ⁡ i min ⁡ { ⟨ x i , x j ⟩ ∣ j = 1 , 2 , … , m } \max\limits_i \min\{ \langle x_i, x_j \rangle \vert j = 1, 2, \dots, m\} imax​min{⟨xi​,xj​⟩∣j=1,2,…,m} 个样本加入初始值

此方法有助于减少迭代次数，但是依赖于核矩阵，因此不适用 $m$ 太大的情形。另一种简单的方法是随机选取 $K$ 个不同的样本作为初始值。

The EM Algorithm

Suppose there exist some latent r.v.s $z^{(1)},\dots ,z^{(m)}$ and we wish to fit the parameters $\theta$ of a model $p(x,z)$ to the training set.

The algorithm can be stated as proof
repeat  { E-step: for i in 1...m  Q i ( z ( i ) ) : = p ( z ( i ) ∣ x ( i ) ; θ ) M-step:  θ : = arg ⁡ max ⁡ θ ∑ i ∑ j Q i ( j ) log ⁡ p ( x ( i ) , z ( i ) = j ; θ ) Q i ( j ) } \begin{aligned} & \text{repeat } \{\\ & \qquad\text{E-step: for i in 1...m } \\ & \qquad\qquad Q_i(z^{(i)}) := p(z^{(i)}|x^{(i)};\theta)\\ & \qquad\text{M-step: } \\ & \qquad\qquad\theta := \arg\max_\theta \sum\limits_i \sum\limits_j Q_i(j) \log \frac{p(x^{(i)},z^{(i)}=j;\theta)}{Q_i(j)}\\ & \} \end{aligned} ​repeat {E-step: for i in 1...m Qi​(z(i)):=p(z(i)∣x(i);θ)M-step: θ:=argθmax​i∑​j∑​Qi​(j)logQi​(j)p(x(i),z(i)=j;θ)​}​

Mixture of Gaussians

Suppose there are $k$ Gaussians and each training data $x^{(i)}$ belongs to one of them. Let $z^{(i)}$ donate the class $x^{(i)}$ belongs to
$$\begin{array}{rcl}{z}^{(i)}& \sim & Multinomia{l}_k(\varphi )\\ x^{(i)}|z^{(i)}=j& \sim & N({\mu }_j,{\Sigma }_j)\end{array}$$

We wish to model the data by specifying $p(x^{(i)},z^{(i)})$.

The algorithm can be stated as proof
repeat  { E-step: for i in 1...m, for j in 1...k w j ( i ) : = ϕ j p ( x ( i ) ∣ z ( i ) = j ; μ , Σ ) ∑ l = 1 k ϕ l p ( x ( i ) ∣ z ( i ) = l ; μ , Σ ) M-step: for j in 1...k ϕ j : = 1 m ∑ i = 1 m w j ( i ) μ j : = ∑ i = 1 m w j ( i ) x ( i ) ∑ i = 1 m w j ( i ) Σ j : = ∑ i = 1 m w j ( i ) ( x ( i ) − μ j ) ( x ( i ) − μ j ) T ∑ i = 1 m w j ( i ) } \begin{aligned} & \text{repeat } \{\\ & \qquad\text{E-step: for i in 1...m, for j in 1...k} \\ & \qquad\qquad w_j^{(i)}:=\frac{\phi_jp(x^{(i)}|z^{(i)}=j;\mu,\Sigma)}{\sum_{l=1}^k\phi_lp(x^{(i)}|z^{(i)}=l;\mu,\Sigma)}\\ & \qquad\text{M-step: for j in 1...k} \\ & \qquad\qquad\phi_j := \frac{1}{m}\sum_{i=1}^m w_j^{(i)}\\ & \qquad\qquad\mu_j := \frac{\sum_{i=1}^m w_j^{(i)}x^{(i)}}{\sum_{i=1}^mw_j^{(i)}}\\ & \qquad\qquad\Sigma_j := \frac{\sum_{i=1}^mw_j^{(i)}(x^{(i)}-\mu_j)(x^{(i)}-\mu_j)^T}{\sum_{i=1}^mw_j^{(i)}}\\ & \} \end{aligned} ​repeat {E-step: for i in 1...m, for j in 1...kwj(i)​:=∑l=1k​ϕl​p(x(i)∣z(i)=l;μ,Σ)ϕj​p(x(i)∣z(i)=j;μ,Σ)​M-step: for j in 1...kϕj​:=m1​i=1∑m​wj(i)​μj​:=∑i=1m​wj(i)​∑i=1m​wj(i)​x(i)​Σj​:=∑i=1m​wj(i)​∑i=1m​wj(i)​(x(i)−μj​)(x(i)−μj​)T​}​

Factor Analysis

Consider the problem in which $n\gg m$. In such a setting, we assume that the data are generated by some latent r.v. $z$
$$x=\mu +\Lambda z+ϵ$$

where
$$\begin{array}{rcccl}{\mathbb{R}}^k& \ni & z& \sim & N(\vec{0},I)\\ {\mathbb{R}}^n& \ni & ϵ& \sim & N(\vec{0},\Psi )\end{array}$$

the value of $k$ is usually chosen to be smaller than $n$. Parameters of our model are

• vector $\mu \in {\mathbb{R}}^n$
• matrix $\Lambda \in {\mathbb{R}}^{n\times k}$
• diagonal matrix $\Psi \in {\mathbb{R}}^{n\times n}$

Since
$$\left[\begin{array}{r}z\\ x\end{array}\right]\sim N(\left[\begin{array}{r}\vec{0}\\ \mu \end{array}\right],\left[\begin{array}{cc}I& {\Lambda }^T\\ \Lambda & \Lambda {\Lambda }^T+\Psi \end{array}\right])$$

parameters for $z^{(i)}|x^{(i)}\sim N({\mu }_{z^{(i)}|x^{(i)}},{\Sigma }_{z^{(i)}|x^{(i)}})$ are
$$\begin{array}{rcl}{\mu }_{z^{(i)}|x^{(i)}}& =& {\Lambda }^T(\Lambda {\Lambda }^T+\Psi {)}^{-1}(x^{(i)}-\mu )\\ {\Sigma }_{z^{(i)}|x^{(i)}}& =& I-{\Lambda }^T(\Lambda {\Lambda }^T+\Psi {)}^{-1}\Lambda \end{array}$$

To use EM algorithm, we have that proof
$$\begin{array}{rcl}{Q}_i(z^{(i)})& :=& p(z^{(i)}|x^{(i)};\mu ,\Lambda ,\Psi )\\ \Lambda & :=& (\sum _{i=1}^m(x^{(i)}-\mu )E[z^{(i)}{]}^T)(\sum _{i=1}^{m}E[z^{(i)}(z^{(i)}{)}^T])\\ \mu & :=& \frac{1}{m}\sum _{i=1}^m{x}^{(i)}\\ \Phi & :=& \frac{1}{m}\sum _{i=1}^{m}E[(x^{(i)}-\Lambda z^{(i)})(x^{(i)}-\Lambda z^{(i)}{)}^T]\end{array}$$

where
$$\begin{array}{rcl}E[z^{(i)}]& =& {\mu }_{z^{(i)}|x^{(i)}}\\ E[z^{(i)}(z^{(i)}{)}^T]& =& {\mu }_{z^{(i)}|x^{(i)}}{\mu }_{z^{(i)}|x^{(i)}}^T+{\Sigma }_{z^{(i)}|x^{(i)}}\\ {\Psi }_{ii}={\Phi }_{ii}& & \end{array}$$

ICA

Independent components analysis finds a new basis in which to represent our data. Suppose some data $s\in {\mathbb{R}}^n$ is generrated via $n$ independent sources, and we observe the overlapping of them
$$x=As$$

where $A$ is the mixing matrix.

Our goal is to find the unmixing matrix $W=A^{-1}$ to recover $s^{(i)}$ according to $x^{(i)}$. Notate
$$W=\left[\begin{array}{c}{w}_1\\ ⋮\\ w_n\end{array}\right]$$

where $w_i\in {\mathbb{R}}^n$ and the $j^{th}$ source can be recovered by
$$s_j^{(i)}=w_j^T{x}^{(i)}$$

Assume that the sources are i.i.d. conforming to logistic distribution
$$p_s(s_i)=g^{\prime }(s_i)$$

where $g(z)=\frac{1}{1+e^{-z}}$. Then the joint distribution
$$p(s)=\prod _{i=1}^n{p}_s(s_i)$$

By transformation $s=Wx$
$$p(x)=\prod _{i=1}^n{p}_s(w_i^{T}x)\cdot |W|$$

Using maximum likelihood
$$l(W)=\sum _{i=1}^m\left(\sum _{j=1}^n\log g^{\prime }(w_j^T{x}^{(i)})+\log |W|\right)$$

we can derive the update rule for stochastic gradient ascent
$$W:=W+\alpha \left(\left[\begin{array}{c}1-2g(w_1^T{x}^{(i)})\\ 1-2g(w_2^T{x}^{(i)})\\ ⋮\\ 1-2g(w_n^T{x}^{(i)})\end{array}\right](x^{(i)}{)}^T+(W^T{)}^{-1}\right)$$

Formula Proof

EM Algorithm

Similar as before, we derive the likelihood
$$\begin{array}{rcl}l(\theta )& =& \sum _{i=1}^m\log p(x^{(i)};\theta )\\ & =& \sum _{i=1}^m\log \sum _{j}p(x^{(i)},z^{(i)}=j;\theta )\end{array}$$

Assume that $z^{(i)}$ have some distribution $Q_i$. Since
$$\begin{array}{rcl}{E}_{z^{(i)}}\left[\frac{p(x^{(i)},z^{(i)};\theta )}{Q_i(z^{(i)})}\right]& =& \sum _j{Q}_i(j)\frac{p(x^{(i)},z^{(i)}=j;\theta )}{Q_i(j)}\\ & =& \sum _{j}p(x^{(i)},z^{(i)}=j;\theta )\end{array}$$

By Jensen’s inequality
$$\begin{array}{rcl}\log E_{z^{(i)}}\left[\frac{p(x^{(i)},z^{(i)};\theta )}{Q_i(z^{(i)})}\right]& \ge & E_{z^{(i)}}\left[\log \frac{p(x^{(i)},z^{(i)};\theta )}{Q_i(z^{(i)})}\right]\\ & & \\ & =& \sum _j{Q}_i(j)\log \frac{p(x^{(i)},z^{(i)}=j;\theta )}{Q_i(j)}\end{array}$$

Therefore
$$l(\theta )\ge \sum _i\sum _j{Q}_i(j)\log \frac{p(x^{(i)},z^{(i)}=j;\theta )}{Q_i(j)}$$

Define the lower bound
$$J(Q,\theta )=\sum _i\sum _j{Q}_i(j)\log \frac{p(x^{(i)},z^{(i)}=j;\theta )}{Q_i(j)}$$

We can apply coordinate ascent to maximize $J$. W.r.t. $Q$, in order for $l(\theta )=J(Q,\theta )$ to hold
$$\frac{p(x^{(i)},z^{(i)};\theta )}{Q_i(z^{(i)})}=c$$

where $c$ does not depend on $z^{(i)}$. Since
$$\sum _z{Q}_i(z)=1$$

we have that
$$c=\sum _{z}p(x^{(i)},z;\theta )=p(x^{(i)};\theta )$$

Therefore
$$Q_i(z^{(i)})=\frac{p(x^{(i)},z^{(i)};\theta )}{p(x^{(i)};\theta )}=p(z^{(i)}|x^{(i)};\theta )$$

Mixture of Gaussians

The E-step is easy
$$\begin{array}{rcl}{w}_j^{(i)}& =& Q_i(z^{(i)}=j)\\ & & \\ & =& P(z^{(i)}=j|x^{(i)};\varphi ,\mu ,\Sigma )\\ & & \\ & =& \frac{{\varphi }_{j}p(x^{(i)}|z^{(i)}=j;\mu ,\Sigma )}{{\sum }_{l=1}^k{\varphi }_{l}p(x^{(i)}|z^{(i)}=l;\mu ,\Sigma )}\end{array}$$

In the M-step
$$J(Q,\theta )=\sum _{i=1}^m\sum _{j=1}^k{w}_j^{(i)}\log \frac{{\varphi }_j\exp (-\frac{1}{2}(x^{(i)}-{\mu }_j{)}^T{\Sigma }_j^{-1}(x^{(i)}-{\mu }_j))}{w_j^{(i)}(2\pi {)}^{n/2}|{\Sigma }_j{|}^{1/2}}$$

To maximize $J$ w.r.t. ${\mu }_j$
$$\begin{array}{rlrl}& \because & {\nabla }_{{\mu }_j}J& =\sum _{i=1}^m{w}_j^{(i)}{\Sigma }_j^{-1}(x^{(i)}-{\mu }_j)\\ & \therefore & {\mu }_j& :=\frac{{\sum }_{i=1}^m{w}_j^{(i)}{x}^{(i)}}{{\sum }_{i=1}^m{w}_j^{(i)}}\end{array}$$

Similarly, we have that
$${\Sigma }_j:=\frac{{\sum }_{i=1}^m{w}_j^{(i)}(x^{(i)}-{\mu }_j)(x^{(i)}-{\mu }_j{)}^T}{{\sum }_{i=1}^m{w}_j^{(i)}}$$

For parameters ${\varphi }_j$
$$J(Q,\theta )=\sum _{i=1}^m\sum _{j=1}^k{w}_j^{(i)}\log {\varphi }_j+c$$

where $c$ does not depend on ${\varphi }_j$. The problem can be stated as
$$\begin{array}{rl}\underset{\varphi }{\min }& \sum _{i=1}^m\sum _{j=1}^k{w}_j^{(i)}\log {\varphi }_j\\ \text{s.t.}& \sum _{j=1}^k{\varphi }_j=1\end{array}$$

Using Lagrangian
$$L(\varphi )=\sum _{i=1}^m\sum _{j=1}^k{w}_j^{(i)}\log {\varphi }_j+\beta (\sum _{j=1}^k{\varphi }_j-1)$$

we have that
$$\frac{\partial }{\partial {\varphi }_j}L=\sum _{i=1}^m\frac{w_j^{(i)}}{{\varphi }_j}+\beta$$

Therefore
$${\varphi }_j\propto \sum _{i=1}^m{w}_j^{(i)}$$

According to the constraint
$${\varphi }_j:=\frac{1}{m}\sum _{i=1}^m{w}_j^{(i)}$$

Factor Analysis

For the M-step, the problem is to
$$\begin{array}{rlr}& \underset{\mu ,\Lambda ,\Psi }{\max }& \sum _{i=1}^m{\int }_{z^{(i)}}{Q}_i(z^{(i)})\log \frac{p(x^{(i)},z^{(i)};\mu ,\Lambda ,\Psi )}{Q_i(z^{(i)})}d{z}^{(i)}\end{array}$$

The object function can be written as
$$\begin{array}{rl}& J(\mu ,\Lambda ,\Psi )\\ & =\sum _{i=1}^{m}E\left[\log \frac{p(x^{(i)},z^{(i)};\mu ,\Lambda ,\Psi )}{Q_i(z^{(i)})}\right]\\ & =\sum _{i=1}^{m}E[\log p(x^{(i)},z^{(i)};\mu ,\Lambda ,\Psi )+\log p(z^{(i)})-\log Q_i(z^{(i)})]\end{array}$$

Getting rid of constants
$$\begin{array}{rl}& J(\mu ,\Lambda ,\Psi )\\ & \equiv \sum _{i=1}^{m}E[\log p(x^{(i)}|z^{(i)};\mu ,\Lambda ,\Psi )]\\ & \equiv -\frac{1}{2}\sum _{i=1}^{m}E[\log |\Psi |+(x^{(i)}-\mu -\Lambda z^{(i)}{)}^T{\Psi }^{-1}(x^{(i)}-\mu -\Lambda z^{(i)})]\end{array}$$

After derivation
$$\begin{array}{rcl}{\nabla }_{\Lambda }J& =& {\Psi }^{-1}\sum _{i=1}^{m}E[(x^{(i)}-\mu -\Lambda z^{(i)})(z^{(i)}{)}^T]\\ {\nabla }_{\mu }J& =& (\Lambda {\Lambda }^T+\Psi {)}^{-1}\sum _{i=1}^m(x^{(i)}-\mu )\\ {\nabla }_{\Psi }J& =& \sum _{i=1}^{m}E[{\Psi }^{-1}-(x^{(i)}-\mu -\Lambda z^{(i)}{)}^T{\Psi }^{-2}(x^{(i)}-\mu -\Lambda z^{(i)})]\end{array}$$