【机器学习算法】高斯判别分析GDA

高斯判别分析

  高斯判别分析（Gaussian discriminative analysis ）是一个较为直观的模型，属于生成模型的一种，采用一种软分类的思路，所谓软分类就是我们对一个样本决定它的类别时使用概率模型来决定，而不是直接由函数映射到某一类上。生成模型通过求解联合概率来求解$P(y|x)$。它假设
$$\begin{gathered} y\sim Bernoulli(\varphi ) \\ x|y=1\sim N({\mu }_1,\Sigma ) \\ x|y=0\sim N({\mu }_2,\Sigma ) \end{gathered}$$
  则有
$$\begin{array}{rl}& P(y)={\varphi }^y(1-\varphi {)}^{1-y}\\ & P(x|y)=N({\mu }_1,\Sigma {)}^y\cdot N({\mu }_2,\Sigma {)}^{1-y}\end{array}$$
  模型的参数为
$$\theta =({\mu }_1,{\mu }_2,\Sigma ,\varphi )$$
  对于生成模型，我们要求解的目标函数是
y ^ = arg ⁡ max ⁡ y ∈ { 0 , 1 } p ( y ∣ x ) = arg ⁡ max ⁡ y p ( y ) p ( x ∣ y ) \hat y=\arg \max_{y \in \{0,1\}}p(y|x)=\arg \max_yp(y)p(x|y) y^​=argy∈{0,1}max​p(y∣x)=argymax​p(y)p(x∣y)
  定义似然函数，则
$$\begin{array}{rl}\hat{\theta }& =\arg \underset{\theta }{\max }l(\theta )\\ & =\arg \underset{\theta }{\max }\log \prod _{i=1}^{N}p(x_i,y_i)\\ & =\arg \underset{\theta }{\max }\log \prod _{i=1}^{N}p(y_i)p(x_i|y_i)\\ & =\arg \underset{\theta }{\max }\sum _{i=1}^N(\log N({\mu }_1,\Sigma {)}^{y_i}\\ & +\log N({\mu }_2,\Sigma {)}^{1-y_i}+\log {\varphi }^{y_i}(1-\varphi {)}^{1-y_i})\end{array}$$

• 求$\varphi$：
  $$\begin{array}{rl}& \frac{\partial l(\theta )}{\partial \varphi }=\sum _{i=1}^N{y}_i\frac{1}{\varphi }-(1-y_i)\frac{1}{1-\varphi }=0\\ & ⟺\sum _{i=1}^N{y}_i(1-\varphi )-(1-y_i)\varphi =0\\ & ⟺\sum _{i=1}^N(y_i-\varphi )=0\\ & ⟺\sum _{i=1}^N{y}_i-N\varphi =0\\ & ⟺\hat{\varphi }=\frac{1}{N}\sum _{i=1}^N{y}_i=\frac{N_1}{N}\end{array}$$
• 求${\mu }_1,{\mu }_2$：
    两个的求解过程其实是相同的，所以我们直接求解${\mu }_1$，由于我们只对${\mu }_1$求解，所以原式可以化简为
  $$\begin{array}{rl}& \sum _{i=1}^N{y}_i\log \frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp (-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1))\\ & =\sum _{i=1}^N{y}_i\log \frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp (-\frac{1}{2}(x_i^T{\Sigma }^{-1}-{\mu }_1^T{\Sigma }^{-1})(x_i-{\mu }_1))\\ & =\sum _{i=1}^N{y}_i\log \frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp (-\frac{1}{2}(x_i^T{\Sigma }^{-1}{x}_i-2{\mu }_1^T{\Sigma }^{-1}{x}_i+{\mu }_1^T{\Sigma }^{-1}{\mu }_1))\end{array}$$
    对上式求导并令导数为0，有
  $$\begin{array}{rl}& -\frac{1}{2}\sum _{i=1}^N{y}_i(-2{\Sigma }^{-1}{x}_i+2{\Sigma }^{-1}{\mu }_1)=0\\ & ⟺\sum _{i=1}^N{y}_i({\Sigma }^{-1}{\mu }_1-{\Sigma }^{-1}{x}_i)=0\\ & ⟺\sum _{i=1}^N{y}_i({\mu }_1-x_i)=0\\ & ⟺\sum _{i=1}^N{y}_i{\mu }_1=\sum _{i=1}^N{y}_i{x}_i\\ & ⟺{\hat{\mu }}_1=\frac{\sum _{i=1}^N{y}_i{x}_i}{\sum _{i=1}^N{y}_i}=\frac{\sum _{i=1}^N{y}_i{x}_i}{N_1}\end{array}$$
    同理可得
  $${\hat{\mu }}_2=\frac{\sum _{i=1}^N(1-y_i)x_i}{\sum _{i=1}^N(1-y_i)}=\frac{\sum _{i=1}^N(1-y_i)x_i}{N_2}$$
• 求$\Sigma$:
    尝试对通项$\log N(\mu ,\Sigma )$进行化简，有
  $$\begin{array}{rl}\sum _{i=1}^N\log N(\mu ,\Sigma )& =\sum _{i=1}^N\log \frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp (-\frac{1}{2}(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu ))\\ & =\sum _{i=1}^N(\log \frac{1}{(2\pi {)}^{\frac{p}{2}}}+|\Sigma {|}^{-\frac{1}{2}}-\frac{1}{2}(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu ))\\ & =\sum _{i=1}^N(C-\frac{1}{2}\log |\Sigma |-\frac{1}{2}(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu ))\\ & =C-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}tr(\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu ))\\ & =C-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}tr(\sum _{i=1}^N(x_i-\mu )(x_i-\mu {)}^T{\Sigma }^{-1})\\ & =-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}tr(S{\Sigma }^{-1})+C\end{array}$$
    由于只需要对$\Sigma$求解，所以对似然函数化简为
  $$\begin{array}{rl}& \sum _{i=1}^N(y_i\log N({\mu }_1,\Sigma )+(1-y_i)\log N({\mu }_2,\Sigma ))\\ & =\sum _{x_i\in c_1}\log N({\mu }_1,\Sigma )+\sum _{x_i\in c_2}\log N({\mu }_2,\Sigma )\\ & =-\frac{1}{2}{N}_1\log |\Sigma |-\frac{1}{2}tr(S_1{\Sigma }^{-1})-\frac{1}{2}{N}_2\log |\Sigma |-\frac{1}{2}{N}_{2}tr(S_2{\Sigma }^{-1})+C\\ & =-\frac{1}{2}(N_1\log |\Sigma |+N_{1}tr(S_1{\Sigma }^{-1})+N_2\log |\Sigma |+N_{2}tr(S_2{\Sigma }^{-1}))+C\end{array}$$
    根据tr的求导公式
  $$\begin{array}{rl}& \frac{\partial tr(AB)}{\partial A}=B^{-1}\\ & \frac{\partial tr(|A|)}{\partial A}=|A|\cdot A^{-1}\\ & tr(AB)=tr(BA)\end{array}$$
    对上面化简后的式子进行求导并令导数为0，有
  $$\begin{array}{rl}& -\frac{1}{2}(N\frac{1}{|\Sigma |}|\Sigma |{\Sigma }^{-1}+N_1\frac{\partial tr({\Sigma }^{-1}{S}_1)}{\partial {\Sigma }^{-1}}\frac{\partial tr({\Sigma }^{-1})}{\partial \Sigma }+N_2\frac{\partial tr({\Sigma }^{-1}{S}_2)}{\partial {\Sigma }^{-1}}\frac{\partial tr({\Sigma }^{-1})}{\partial \Sigma })=0\\ & ⟺N\frac{1}{|\Sigma |}|\Sigma |{\Sigma }^{-1}-N_1{S}_1^T{\Sigma }^{-2}-N_1{S}_2^T{\Sigma }^{-2}=0\\ & ⟺N{\Sigma }^{-1}-N_1{S}_1{\Sigma }^{-2}-N_1{S}_2{\Sigma }^{-2}=0\\ & ⟺N\Sigma -N_1{S}_1-N_1{S}_2=0\\ & ⟺N\Sigma -N_1{S}_1-N_1{S}_2=0\\ & ⟺\hat{\Sigma }=\frac{1}{N}(N_1{S}_1+N_2{S}_2)\end{array}$$