GMM

GMM

• 一个类一个正态分布
• $N({\mu }_k,{\Sigma }_k)$

	有监督	无监督	半监督
目标函数	$$\begin{gathered} L=logp(X_l,Y_l\Vert \theta )={\Sigma }_{i=1}^{l}logp(y_i\Vert \theta )p(x_i\Vert y_i,\theta ) \\ ={\Sigma }_{i=1}^{l}log{\alpha }_{y_i}N(x_i\Vert {\theta }_{y_i}) \end{gathered}$$	$p(x;\theta )={\Pi }_i^N{\Sigma }_{k=1}^K{\pi }_{k}N(x_i\Vert {\mu }_k,{\Sigma }_k)$	$P(x_l,y_l,x_u\Vert \theta )={\Sigma }_{i=1}^{l}log{\alpha }_{y_i}N(x_i\Vert {\theta }_{y_i})+{\Sigma }_{i=l}^{m}log{\Sigma }_{k=1}^N{\alpha }_{k}N(x_i\Vert {\theta }_k)$
E	求导解决	$求{\gamma }_{ik}=p(y_i=k\Vert x_i)=\frac{{\alpha }_{k}N(x_i\Vert {\theta }_k)}{{\Sigma }_{k=1}^N{\alpha }_{k}N(x_i\Vert {\theta }_k)}$	$求{\gamma }_{ik}=p(y_i=k\Vert x_i)=\frac{{\alpha }_{k}N(x_i\Vert {\theta }_k)}{{\Sigma }_{k=1}^N{\alpha }_{k}N(x_i\Vert {\theta }_k)}$
M	$$\begin{gathered} {\mu }_k=\frac{1}{l_k}({\Sigma }_{i\in D_l,y_i=k}{x}_i) \\ {\Sigma }_i=\frac{1}{l_k}({\Sigma }_{i\in D_l,y_i=k}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T) \\ {\alpha }_k=\frac{l_k}{m} \end{gathered}$$	$$\begin{gathered} {\mu }_k=\frac{{\Sigma }_i\gamma (z_{ik})x_i}{\gamma (z_{ik})} \\ {\pi }_k=\frac{{\Sigma }_i\gamma (z_{ik})}{N} \\ {\Sigma }_k=\frac{{\Sigma }_i\gamma (z_{ik})(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T}{\gamma (z_{ik})} \end{gathered}$$	$$\begin{gathered} {\mu }_k=\frac{1}{{\Sigma }_{i=l}^m{\gamma }_{ik}+l_k}({\Sigma }_{i\in D_l,y_i=k}{x}_i+{\Sigma }_{i=l}^m{\gamma }_{ik}{x}_i) \\ {\Sigma }_i=\frac{1}{{\Sigma }_{i=l}^m{\gamma }_{ik}+l_k}({\Sigma }_{i\in D_l,y_i=k}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T+{\Sigma }_{i=l}^m{\gamma }_{ik}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T) \\ {\alpha }_k=\frac{{\Sigma }_{i=l}^m{\gamma }_{ik}+l_k}{m} \end{gathered}$$
			半监督=无监督+有监督

有监督

• 目标函数： $L=logp(X_l,Y_l|\theta )={\Sigma }_{i=1}^{l}logp(y_i|\theta )p(x_i|y_i,\theta ),{\theta }_i={\alpha }_i,{\mu }_i,{\Sigma }_i$
• $$\begin{gathered} ={\Sigma }_{i=1}^{l}log{\alpha }_{y_i}N(x_i|{\theta }_{y_i}) \\ ={\Sigma }_{i=1}^l(log{\alpha }_{y_i}-\frac{n}{2}log(2\pi )-\frac{1}{2}log(|{\Sigma }_{y_i}|)-(x_i-{\mu }_{y_i}{)}^T{\Sigma }_{y_i}^{-1}(x_i-{\mu }_{y_i}) \end{gathered}$$
• 直接求导得到结果
• $$\begin{gathered} {\mu }_k=\frac{1}{l_k}({\Sigma }_{i\in D_l,y_i=k}{x}_i) \\ {\Sigma }_i=\frac{1}{l_k}({\Sigma }_{i\in D_l,y_i=k}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T) \\ {\alpha }_k=\frac{l_k}{m} \end{gathered}$$

无监督

5.2GMM高斯混合模型和EM

• 概率解释: 假设有K个簇，每一个簇服从高斯分布，以概率π𝑘随机选择一个簇 k ，从其分布中采样出一个样本点，如此得到观测数据
• N个样本点𝒙的似然函数(Likelihood)
  • $p(x;\theta )={\Pi }_i^N{\Sigma }_{k=1}^K{\pi }_{k}N(x_i|{\mu }_k,{\Sigma }_k),其中{\Sigma }_k{\pi }_k=1,0\le {\pi }_k\le 1$
  • 引入隐变量，指示所属类,k维独热表示
    • $p(z_k=1)={\pi }_k$
    • $p(x_i|z)={\Pi }_k^{K}N(x_i|{\mu }_k,{\Sigma }_k{)}^{z_k}$
      • $p(x_i|z_k=1)=N(x_i|{\mu }_k,{\Sigma }_k)$
    • $p(x_i)={\Sigma }_{z}p(x_i|z)p(z)={\Sigma }_{k=1}^K{\pi }_{k}N(x_i|{\mu }_k,{\Sigma }_k)$
• 从属度(可以看做，xi属于第k个簇的解释
  • $$\begin{gathered} \gamma (z_{ik}) \\ =p(z_{ik=1}|x_i) \\ =\frac{p(z_{ik}=1)p(x_i|z_k=1)}{{\Sigma }_{k=1}^{K}p(z_{ik}=1)p(x_i|z_k=1)} \\ =\frac{{\pi }_{k}N(x_i|{\mu }_k,{\Sigma }_k)}{{\Sigma }_{k=1}^K{\pi }_{k}N(x_i|{\mu }_k,{\Sigma }_k)} \end{gathered}$$

参数学习：极大似然估计–EM

• 极大似然估计
  • 难：log里面有求和，所有参数耦合
  • 似然函数取最大值时满足的条件：$log(P(x|\theta )对{\mu }_k求导$
    • $0=-{\Sigma }_{i=1}^N\frac{{\pi }_{k}N(x_i|{\mu }_k,{\Sigma }_k)}{{\Sigma }_{k=1}^K{\pi }_{k}N(x_i|{\mu }_k,{\Sigma }_k)}{\Sigma }_k(x_i-{\mu }_k)$
      • ${\mu }_k=\frac{{\Sigma }_i\gamma (z_{ik})x_i}{\gamma (z_{ik})}$
      • ${\pi }_k=\frac{{\Sigma }_i\gamma (z_{ik})}{N}$
      • ${\Sigma }_k=\frac{{\Sigma }_i\gamma (z_{ik})(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T}{\gamma (z_{ik})}$
    • 这不是封闭解–》EM
      • E：给定当前参数估计值，求后验概率$\gamma (z_{ik})=E(z_{ik})$
      • M：依据后验概率$\gamma (z_{ik})$，求参数估计${\mu }_k、{\pi }_k、{\Sigma }_k$
      • 迭代收敛到局部极小

EM

• 通用EM
  • 目标函数：极大似然函数$logP(X|\theta )=log{\Sigma }_{z}P(x,z|\theta )$
  • 用于：不完整数据的对数似然函数
    • 不知Z的数据，只知道Z的后验分布$P(z|x,{\theta }^{old})$
    • 考虑其期望$Q(\theta ,{\theta }^{old})=E_{p(z|x,{\theta }^{old})}(logP(x,z|\theta ))$
    • 最大化期望${\theta }^{new}=argma{x}_{\theta }Q(\theta ,{\theta }^{old})$
  • E：求$P(z|x,{\theta }^{old})$
  • M：${\theta }^{new}=argma{x}_{\theta }Q(\theta ,{\theta }^{old})$
    • why是启发式的，但却存在似然函数？
      • $Q(\theta ,{\theta }^{old})=E_{p(z|x,{\theta }^{old})}(logP(x,z|\theta ))=p(x;\theta )$
  • 完整数据和不完整数据的比较
  • 不完整数据：$logp(x)={\Sigma }_{i}log{\Sigma }_{z}p(x_i|z)p(z)={\Sigma }_{i}log{\Sigma }_{k=1}^K{\pi }_{k}N(x_i|{\mu }_k,{\Sigma }_k)$
    • 不完整数据中，参数之间是耦合的，不存在封闭解
  • 完整数据
    • $logp(x,z|\theta )=logp(z|\theta )p(x|z,\theta )={\Sigma }_i{\Sigma }_k{z}_{ik}(log{\pi }_k+logN(x_i|{\mu }_k,{\Sigma }_k))$
    • $$\begin{gathered} E_z(logp(x,z|\theta )) \\ ={\Sigma }_i{\Sigma }_{k}E(z_{ik})(log{\pi }_k+logN(x_i|{\mu }_k,{\Sigma }_k)) \\ ={\Sigma }_i{\Sigma }_k\gamma (z_{ik})(log{\pi }_k+logN(x_i|{\mu }_k,{\Sigma }_k)) \end{gathered}$$

EM收敛性保证

• 目标：最大化$P(x|\theta )={\Sigma }_{z}p(x,z|\theta )$
  • 直接优化$P(x|\theta )$很困难，但优化完整数据的$p(x,z|\theta )$容易
• 证明
  • 分解
  • 对任意分布q(z),下列分解成立
    • $$\begin{gathered} lnp(x|\theta )=L(q,\theta )+KL(q||p) \\ 其中， \\ L(q,\theta )={\Sigma }_{z}q(z)ln(\frac{p(x,z|\theta )}{q(z)}) \\ KL(q||p)=-{\Sigma }_{z}q(z)ln(\frac{p(z|x,\theta )}{q(z)}) \\ KL(q||p)\ge 0,L(q,\theta )是lnp(x|\theta )的下界 \end{gathered}$$
  • E：$$\begin{gathered} 最大化L(q,\theta ), \\ q(z)=P(z|x,{\theta }^{old}) \end{gathered}$$
  • $M:原来的下界L(q,\theta )={\Sigma }_{z}P(z|x,{\theta }^{old})ln(\frac{p(x,z|\theta )}{q(z)})=Q(\theta ,{\theta }^{old})+const---正好是期望$
  • 下界提升了
    
    
    
    

半监督

• 目标函数： $L=logp(X_l,Y_l,X_u|\theta )={\Sigma }_{i=1}^{l}logp(y_i|\theta )p(x_i|y_i,\theta )+{\Sigma }_{i=l+1}^{m}log({\Sigma }_{k=1}^{N}p(y_i=k|\theta )p(x_i|y_i=k,\theta )),{\theta }_i={\alpha }_i,{\mu }_i,{\Sigma }_i$
• = Σ i = 1 l l o g α y i N ( x i ∣ θ y i ) + Σ i = l m l o g Σ k = 1 N α k N ( x i ∣ θ k ) = Σ i = 1 l ( l o g α y i − n 2 l o g ( 2 π ) − 1 2 l o g ( ∣ Σ y i ∣ ) − ( x i − μ y i ) T Σ y i − 1 ( x i − μ y i ) + Σ i = l m l o g ( Σ k = 1 N ( α k 1 ( 2 π ) n / 2 ∣ Σ k ∣ 1 / 2 e x p { − 1 2 ( x i − μ k ) T Σ k − 1 ( x i − μ k ) } ) ) =\Sigma_{i=1}^llog \alpha_{y_i}N(x_i|\theta_{y_i})+\Sigma_{i=l}^mlog\Sigma_{k=1}^N\alpha_kN(x_i|\theta_k) \\=\Sigma_{i=1}^l(log\alpha_{y_i}-\frac{n}{2}log(2\pi)-\frac{1}{2}log(|\Sigma_{y_i}|)-(x_i-\mu_{y_i})^T\Sigma_{y_i}^{-1}(x_i-\mu_{y_i})+\Sigma_{i=l}^mlog(\Sigma_{k=1}^N(\alpha_k{{1} \over {(2\pi)^{n/2}|\Sigma_k|^{1/2}}} exp\{ -{{1} \over {2}}(x_i-\mu_k)^T{\Sigma_k}^{-1}(x_i-\mu_k)\})) =Σi=1l​logαyi​​N(xi​∣θyi​​)+Σi=lm​logΣk=1N​αk​N(xi​∣θk​)=Σi=1l​(logαyi​​−2n​log(2π)−21​log(∣Σyi​​∣)−(xi​−μyi​​)TΣyi​−1​(xi​−μyi​​)+Σi=lm​log(Σk=1N​(αk​(2π)n/2∣Σk​∣1/21​exp{−21​(xi​−μk​)TΣk​−1(xi​−μk​)}))
• E：$求{\gamma }_{ik}=p(y_i=k|x_i)=\frac{{\alpha }_{k}N(x_i|{\theta }_k)}{{\Sigma }_{k=1}^N{\alpha }_{k}N(x_i|{\theta }_k)}$
• M：$$\begin{gathered} {\mu }_k=\frac{1}{{\Sigma }_{i=l}^m{\gamma }_{ik}+l_k}({\Sigma }_{i\in D_l,y_i=k}{x}_i+{\Sigma }_{i=l}^m{\gamma }_{ik}{x}_i) \\ {\Sigma }_i=\frac{1}{{\Sigma }_{i=l}^m{\gamma }_{ik}+l_k}({\Sigma }_{i\in D_l,y_i=k}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T+{\Sigma }_{i=l}^m{\gamma }_{ik}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^T) \\ {\alpha }_k=\frac{{\Sigma }_{i=l}^m{\gamma }_{ik}+l_k}{m} \end{gathered}$$