机器学习-白板推导 P4_5 (高斯判别分析)

高斯判别分析 Gaussian Discriminant Analysis

定义

y ^ = arg ⁡ max ⁡ y ∈ { 0 , 1 } p ( y ∣ x ) \hat{y} = \arg \max_{y \in \lbrace 0,1 \rbrace} p(y|x) y^​=argmaxy∈{0,1}​p(y∣x)

借助贝叶斯定理：$p(y|x)=\frac{p(x|y)p(y)}{p(x)}$

生成式模型并不是求两个值的大小，只要能够比较出两个值的大小就行

因为$p(x)$是一个定值

所以 $p(y|x)\propto p(x|y)p(y)$

所以： y ^ ∝ arg ⁡ max ⁡ y ∈ { 0 , 1 } p ( x ∣ y ) p ( y ) \hat{y} \propto \arg \max_{y \in \lbrace 0,1 \rbrace} p(x|y) p(y) y^​∝argmaxy∈{0,1}​p(x∣y)p(y)

假设$y$服从伯努力分布

$y\propto Bernoulli(\varphi )$

高斯判别分析，假设条件概率服从高斯分布,均值不同,方差相同

$x|y=1\propto N({\mu }_1,\Sigma )$

$x|y=0\propto N({\mu }_2,\Sigma )$

公式

$log-likelihood$:
$$\begin{array}{rl}L(\theta )& =\sum _{i=1}^N\log (p(x_i|y_i)p(y_i))\\ & =\sum _{i=1}^N(\log (p(x_i|y_i)+\log (p(y_i))\\ & =\sum _{i=1}^N(\log N({\mu }_1,\Sigma {)}^{y_i}N({\mu }_2,\Sigma {)}^{1-y_i}+\log {\varphi }^{y_i}(1-\varphi {)}^{1-y_i})\\ & =\sum _{i=1}^N(\log N({\mu }_1,\Sigma {)}^{y_i}+\log N({\mu }_2,\Sigma {)}^{1-y_i}+\log {\varphi }^{y_i}(1-\varphi {)}^{1-y_i})\end{array}$$

$\theta =({\mu }_1,{\mu }_2,\Sigma ,\varphi )$

$\hat{w}=arg{\max }_{\theta }L(\theta )$

求值

把样本分为2类：
$y=1:N_1$
$y=0:N_2$
$N=N_1+N_2$

求$\varphi$:

定义：

$L(\theta {)}_3=\log {\varphi }^{y_i}(1-\varphi {)}^{1-y_i}$

$\frac{\partial L(\theta {)}_3}{\partial \varphi }={\sum }_{i=1}^N(y_i\frac{1}{\varphi }+(1-y_i)\frac{1}{1-\varphi }(-1))=0$

${\sum }_{i=1}^N(y_i(1-\varphi )-(1-y_i)\varphi )=0$

${\sum }_{i=1}^N(y_i-\varphi )=0$

${\sum }_{i=1}^N{y}_i-N\varphi =0$

所以：

$\hat{\varphi }=\frac{1}{N}{\sum }_{i=1}^N{y}_i=\frac{N_1}{N}$

求${\mu }_1$:
定义：

$L(\theta {)}_1={\sum }_{i=1}^N(\log N({\mu }_1,\Sigma {)}^{y_i})={\sum }_{i=1}^N{y}_i\log \frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}exp(-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1))$

${\mu }_1=arg{\max }_{{\mu }_1}L(\theta {)}_1=arg{\max }_{{\mu }_1}{\sum }_{i=1}^N{y}_i(-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1))$

$$\begin{array}{rl}\Delta & =\sum _{i=1}^N{y}_i(-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1))\\ & =-\frac{1}{2}\sum _{i=1}^N{y}_i((x_i^T{\Sigma }^{-1}-{\mu }_1^T{\Sigma }^{-1})(x_i-{\mu }_1))\\ & =-\frac{1}{2}\sum _{i=1}^N{y}_i(x_i^T{\Sigma }^{-1}{x}_i-{\mu }_1^T{\Sigma }^{-1}{\mu }_1-x_i^T{\Sigma }^{-1}{x}_i+{\mu }_1^T{\Sigma }^{-1}{\mu }_1)\\ & =-\frac{1}{2}\sum _{i=1}^N{y}_i(x_i^T{\Sigma }^{-1}{x}_i-2{\mu }_1^T{\Sigma }^{-1}{x}_1+{\mu }_1^T{\Sigma }^{-1}{\mu }_1)\end{array}$$

$\frac{\partial \Delta }{\partial {\mu }_1}=-\frac{1}{2}{\sum }_{i=1}^N{y}_i(-2{\Sigma }^{-1}{x}_i+2{\Sigma }^{-1}{\mu }_1)=0$

${\sum }_{i=1}^N{y}_i({\Sigma }^{-1}{\mu }_1-{\Sigma }^{-1}{x}_i)=0$

${\sum }_{i=1}^N{y}_i({\mu }_1-x_i)=0$

${\sum }_{i=1}^N{y}_i{\mu }_1={\sum }_{i=1}^N{y}_i{x}_i$

$\widehat{{\mu }_1}=\frac{{\sum }_{i=1}^N{y}_i{x}_i}{{\sum }_{i=1}^N{y}_i}=\frac{{\sum }_{i=1}^N{y}_i{x}_i}{N_1}$

求$\Sigma$:

补充公式：
$\frac{\partial tr(AB)}{\partial A}=B^T$
$\frac{\partial |A|}{\partial A}=|A|.A^{-1}$
$tr(AB)=tr(BA)$
$tr(ABC)=tr(CAB)=tr(BCA)$

$L(\theta {)}_2={\sum }_{i=1}^N(\log N({\mu }_1,\Sigma {)}^{y_i}+\log N({\mu }_2,\Sigma {)}^{1-y_i})$

$$\begin{array}{rl}\log N(\mu ,\Sigma )& =\log \frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}exp(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))\\ & =\log \frac{1}{(2\pi {)}^{\frac{p}{2}}}+\log |\Sigma {|}^{-\frac{1}{2}}-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))\\ & =C-\frac{1}{2}\log |\Sigma |-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))\end{array}$$

$$\begin{array}{rl}\sum _{i=1}^N\log N(\mu ,\Sigma )& =\sum _{i=1}^N\left(C-\frac{1}{2}\log |\Sigma |-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))\right)\\ & =C-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}\sum _{i=1}^N\log N(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))\end{array}$$
$(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )$维度为1

$(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )=tr((x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))$

样本方差$S=\frac{1}{N}{\sum }_{i=1}^N(x-\mu )(x-\mu {)}^T$
$$\begin{array}{rl}\sum _{i=1}^{N}tr\left((x-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right)& =\sum _{i=1}^{N}tr\left((x-\mu )(x-\mu {)}^T{\Sigma }^{-1}\right)\\ & =tr\left(\sum _{i=1}^N(x-\mu )(x-\mu {)}^T{\Sigma }^{-1}\right)\\ & =Ntr(S{\Sigma }^{-1})\end{array}$$
所以：
$$\begin{array}{rl}\sum _{i=1}^N\log N(\mu ,\Sigma )& =C-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}Ntr(S{\Sigma }^{-1})\end{array}$$

$$\begin{array}{rl}L(\theta {)}_2& =\sum _{i=1}^N(\log N({\mu }_1,\Sigma {)}^{y_i}+\log N({\mu }_2,\Sigma {)}^{1-y_i})\\ & =-\frac{1}{2}{N}_1\log |\Sigma |-\frac{1}{2}{N}_{1}tr(S_1{\Sigma }^{-1})-\frac{1}{2}{N}_2\log |\Sigma |-\frac{1}{2}{N}_{2}tr(S_2{\Sigma }^{-1})+C\\ & =-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}{N}_{1}tr(S_1{\Sigma }^{-1})-\frac{1}{2}{N}_{2}tr(S_2{\Sigma }^{-1})+C\\ & =-\frac{1}{2}\left(N\log |\Sigma |+N_{1}tr(S_1{\Sigma }^{-1})+N_{2}tr(S_2{\Sigma }^{-1})\right)+C\end{array}$$

$$\begin{array}{rl}\frac{\partial L(\theta {)}_2}{\partial \Sigma }& =-\frac{1}{2}\left(N\frac{1}{|\Sigma |}|\Sigma |{\Sigma }^{-1}-N_1{S}_1{\Sigma }^{-2}-N_2{S}_2{\Sigma }^{-2}\right)\\ & =-\frac{1}{2}\left(N{\Sigma }^{-1}-N_1{S}_1{\Sigma }^{-2}-N_2{S}_2{\Sigma }^{-2}\right)\\ & =0\end{array}$$

$N{\Sigma }^{-1}=N_1{S}_1{\Sigma }^{-2}+N_2{S}_2{\Sigma }^{-2}$

$N\Sigma =N_1{S}_1+N_2{S}_2$

$\hat{\Sigma }=\frac{1}{N}(N_1{S}_1+N_2{S}_2)$