【白板推导系列笔记】线性分类-高斯判别分析（Gaussian Discriminant Analysis）-模型求解（求期望）&模型求解（求协方差）

$$L({\mu }_1,{\mu }_2,\Sigma ,\varphi )=\sum _{i=1}^N[\underset{(1)}{\underbrace{\log N({\mu }_1,\Sigma {)}^{y_i}}}+\underset{(2)}{\underbrace{\log N({\mu }_2,\Sigma {)}^{1-y_i}}}+\underset{(3)}{\underbrace{\log {\varphi }^{y_i}(1-\varphi {)}^{1-y_i}}}]$$
求$\varphi$，显然只有$(3)$与$\varphi$相关
$$\begin{array}{rl}(3)& =\sum _{i=1}^N\log {\varphi }^{y_i}(1-\varphi {)}^{1-y_i}\\ & =\sum _{i=1}^N[y_i\log \varphi +(1-y_i)\log (1-\varphi )]\\ \frac{\partial (3)}{\partial \varphi }& =\sum _{i=1}^N\left[y_i\cdot \frac{1}{\varphi }-\left(1-y_i\right)\frac{1}{1-\varphi }\right]=0\\ 0& =\sum _{i=1}^N[y_i\cdot (1-\varphi )-(1-y_i)\varphi ]\\ 0& =\sum _{i=1}^N(y_i-y_i\varphi -\varphi +y_i\varphi )\\ 0& =\sum _{i=1}^N(y_i-\varphi )\\ 0& =\sum _{i=1}^N{y}_i+N\varphi \\ \hat{\varphi }& =\frac{\sum _{i=1}^N{y}_i}{N}\end{array}$$
求${\mu }_1$，显然只有$(1)$与${\mu }_1$相关。对于${\mu }_2$类似于${\mu }_1$，只需要$1-y_i$替换$y_i$即可
$$\begin{array}{rl}(1)& =\sum _{i=1}^N\log N({\mu }_1,\Sigma {)}^{y_i}\\ & =\sum _{i=1}^N{y}_i\log \frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}\text{exp}\left[-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1)\right]\\ {\mu }_1& =\underset{{\mu }_1}{\mathrm{argmax}}(1)\\ & =\underset{{\mu }_1}{\mathrm{argmax}}\sum _{i=1}^N{y}_i\left[-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1)\right]\\ & =\underset{{\mu }_1}{\mathrm{argmax}}-\frac{1}{2}\sum _{i=1}^N{y}_i(x_i^T{\Sigma }^{-1}-{\mu }_1^T{\Sigma }^{-1})(x_i-{\mu }_1)\\ & =\underset{{\mu }_1}{\mathrm{argmax}}-\frac{1}{2}\sum _{i=1}^N{y}_i(\underset{\in \mathbb{R}}{\underbrace{x_i^T{\Sigma }^{-1}{x}_i}}-\underset{1\times 1}{\underbrace{x_i^T{\Sigma }^{-1}{\mu }_1}}-\underset{1\times 1}{\underbrace{{\mu }_1^T{\Sigma }^{-1}{x}_i}}+{\mu }_1^T{\Sigma }^{-1}{\mu }_1)\\ & =\underset{{\mu }_1}{\mathrm{argmax}}\underset{\Delta }{\underbrace{-\frac{1}{2}\sum _{i=1}^N{y}_i(x_i^T{\Sigma }^{-1}{x}_i-2{\mu }_1^T{\Sigma }^{-1}{x}_i+{\mu }_1^T{\Sigma }^{-1}{\mu }_1)}}\\ \frac{\partial \Delta }{\partial {\mu }_1}& =-\frac{1}{2}\sum _{i=1}^N{y}_i(-2{\Sigma }^{-1}{x}_i+2{\Sigma }^{-1}{\mu }_1)=0\\ 0& =\sum _{i=1}^N{y}_i({\Sigma }^{-1}{\mu }_1-{\Sigma }^{-1}{x}_i)\\ 0& =\sum _{i=1}^N{y}_i({\mu }_1-x_i)\\ \sum _{i=1}^N{y}_i{\mu }_1& =\sum _{i=1}^N{y}_i{x}_i\\ \widehat{{\mu }_1}& =\frac{\sum _{i=1}^N{y}_i{x}_i}{\sum _{i=1}^N{y}_i}\end{array}$$
这里我们设
C 1 = { x i ∣ y i = 1 , i = 1 , 2 , ⋯   , N } , ∣ C 1 ∣ = N 1 C 0 = { x i ∣ y i = 0 , i = 1 , 2 , ⋯   , N } , ∣ C 0 ∣ = N 0 N = N 1 + N 0 \begin{aligned} C_{1}&=\left\{x_{i}|y_{i}=1,i=1,2,\cdots,N\right\},|C_{1}|=N_{1}\\ C_{0}&=\left\{x_{i}|y_{i}=0,i=1,2,\cdots,N\right\},|C_{0}|=N_{0}\\ N&=N_{1}+N_{0} \end{aligned} C1​C0​N​={xi​∣yi​=1,i=1,2,⋯,N},∣C1​∣=N1​={xi​∣yi​=0,i=1,2,⋯,N},∣C0​∣=N0​=N1​+N0​​
因此
$$\widehat{{\mu }_1}=\frac{\sum _{i=1}^N{y}_i{x}_i}{N_1}$$
再用$1-y_i$替换$y_i$得$\widehat{{\mu }_2}$
$$\widehat{{\mu }_2}=\frac{\sum _{i=1}^N(1-y_i)x_i}{\sum _{i=1}^N(1-y_i)}=\frac{\sum _{i=1}^N(1-y_i)x_i}{N-N_1}=\frac{\sum _{i=1}^N(1-y_i)x_i}{N_0}$$
求$\Sigma$，显然只有$(1),(2)$与$\Sigma$相关
$$\begin{array}{rl}(1)+(2)& =\sum _{i=1}^N{y}_i\log N({\mu }_1,\Sigma )+\sum _{i=1}^N(1-y_i)\log N({\mu }_2,\Sigma )\\ & =\sum _{x_i\in C_1}\log ({\mu }_1,\Sigma )+\sum _{x_i\in C_2}\log N({\mu }_2,\Sigma )\\ \sum _{i=1}^N\log N(\mu ,\Sigma )& =\sum _{i=1}^N\frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}\text{exp}\left[-\frac{1}{2}(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )\right]\\ & =\sum _{i=1}^N\left[\log \frac{1}{{\left(2\pi \right)}^{\frac{p}{2}}}+\log |\Sigma {|}^{\frac{1}{2}}+\left(-\frac{1}{2}(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )\right)\right]\\ & =\sum _{i=1}^N\left[C-\frac{1}{2}\log |\Sigma |-\frac{1}{2}(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )\right]\\ & =C-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}\underset{\in \mathbb{R}}{\underbrace{\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )}}\\ \sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )& =\sum _{i=1}^N\text{tr }[(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )]\\ & =\sum _{i=1}^N\text{tr }[(x_i-\mu )(x_i-\mu {)}^T{\Sigma }^{-1}]\\ & =\text{tr }\left[\underset{x_i的方差S}{\underbrace{\sum _{i=1}^N(x_i-\mu )(x_i-\mu {)}^T}}{\Sigma }^{-1}\right]\\ & 设S=\frac{1}{N}\sum _{i=1}^N(x_i-\mu )(x_i-\mu {)}^T\\ & =N\cdot \text{tr }(S{\Sigma }^{-1})\\ & 带回\sum _{i=1}^N\log N(\mu ,\Sigma )\\ \sum _{i=1}^N\log N(\mu ,\Sigma )& =C-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )\\ & =-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}N\cdot \text{tr }(S\cdot {\Sigma }^{-1})+C\\ & 带回(1)+(2)\\ (1)+(2)& =-\frac{1}{2}{N}_1\log |\Sigma |-\frac{1}{2}N\cdot \text{tr }(S\cdot {\Sigma }^{-1})-\frac{1}{2}{N}_2\log |\Sigma |-\frac{1}{2}N\cdot \text{tr }(S_2{\Sigma }^{-1})+C\\ & =-\frac{1}{2}N\log |\Sigma |-\frac{1}{2}N\cdot \text{tr }(S_2{\Sigma }^{-1})-\frac{1}{2}N\cdot \text{tr }(S\cdot {\Sigma }^{-1})+C\\ & =-\frac{1}{2}[N\log |\Sigma |+N_1\text{tr }(S_1{\Sigma }^{-1})+N_2\text{tr }(S_2{\Sigma }^{-1})]+C\\ \frac{\partial (1)+(2)}{\partial \Sigma }& =-\frac{1}{2}(N\cdot \frac{1}{|\Sigma |}|\Sigma |{\Sigma }^{-1}-N_1{S}_1{\Sigma }^{-1}{\Sigma }^{-1}-N_2{S}_2{\Sigma }^{-1}{\Sigma }^{-1})=0\\ N\Sigma -N_1{S}_1-N_2{S}_2& =0\\ \hat{\Sigma }& =\frac{1}{N}(N_1{S}_1+N_2{S}_2)\end{array}$$

迹的性质
$$\begin{array}{rl}\text{tr }(AB)& =\text{tr }(BA)\\ \text{tr }(ABC)& =\text{tr }(CAB)=\text{tr }(BCA)\end{array}$$
矩阵求导
$$\begin{array}{rl}\frac{\partial \text{tr }(AB)}{\partial A}& =B^{-1}\\ \frac{\partial |A|}{\partial A}& =|A|\cdot A^T\end{array}$$

图中圆圈代表正样本，叉号代表负样本，直线p(y = 1|x) = 0.5代表分类边界(decision boundary)。因为Σ相同所以两个形状相同，但是具有不同的μ 。

作者：张文翔
链接：Andrew Ng Stanford机器学习公开课 总结（5） - 张文翔的博客 | BY ZhangWenxiang (demmon-tju.github.io)