高斯判别分析

高斯判别分析（Gaussian discriminative analysis）属于概率生成式模型，并不是直接计算p(y|x)的概率，而是基于bayes,比较p(y=1|x)和p(y=0|x)的大小，从而确定分类
贝叶斯公式：
$p(y|x)=\frac{p(x|y)p(y)}{p(x)}$
p(x)项和p(y)没有关系，所以可以去掉，原式可以写为基于联合概率建模,形式
$argmaxp(y|x)=argmaxp(x|y)p(y)=argmaxp(x,y)$
这里$p(y)是先验概率，p(y|x)是后验概率，p(x|y)是似然函数$

假定：
$p(y)\sim B(1,p)$
$p(x|y=1)\sim N({\mu }_1,\sigma )$
$p(x|y=0)\sim N({\mu }_2,\sigma )$
令$y=1$
则有：
$p(y)={\rho }^y(1-\rho {)}^{1-y}$
$p(x|y)=N({\mu }_1,\sigma {)}^{y}N({\mu }_2,\sigma {)}^{1-y}$
建立似然函数有
$L(\theta )=\log \prod P(x|y)p(y)$
$=\sum \log P(x|y)p(y)$
$=\sum \log (N({\mu }_1,\sigma {)}^{y}N({\mu }_2,\sigma {)}^{1-y}{\rho }^y(1-\rho {)}^{1-y})$
$=\sum \log (N({\mu }_1,\sigma {)}^{y}N({\mu }_2,\sigma {)}^{1-y})+\log ({\rho }^y(1-\rho {)}^{1-y})$
$=\sum \log (N({\mu }_1,\sigma {)}^y)+\log (N({\mu }_2,\sigma {)}^{1-y})+\log ({\rho }^y(1-\rho {)}^{1-y})$
所以$\theta =({\mu }_1,{\mu }_2,\sigma ,\rho )$
最后求解$\hat{\theta }=argmaxL(\theta )$
1：求$\rho$
$\frac{\partial L(\theta )}{\partial \rho }=d\sum (\log {\rho }^y+\log (1-\rho {)}^{1-y})$
$=\sum (y\frac{1}{\rho }+(1-y)\frac{1}{1-\rho }(-1))=0$
$=\sum (y(1-\rho )-(1-y)\rho )=0$
$=\sum (\rho -y\rho -y+y\rho )=0$
$=\sum (\rho -y)=0$
所以有
$\sum y=\sum \rho$
因为：
$\sum =N$
y=1的个数有
$\sum y=N1$
y=0的个数有
$\sum (1-y)=N2$
$N1+N2=N$
所以
$N1=N\rho$
$\rho =\frac{N_1}{N}$

2：求${\mu }_1$
$\frac{\partial L(\theta )}{\partial {\mu }_1}=d\sum \log (N({\mu }_1,\sigma {)}^y)$
定义：
${\sum }_{}^{}=\left[\begin{array}{cccc}{\sigma }_1^2& 0& \cdots & 0\\ 0& {\sigma }_2^2& \cdots & 0\\ ⋮& \cdots & \cdots & ⋮\\ 0& 0& \cdots & {\sigma }_n^2\end{array}\right]$
$\sum$代表协方差矩阵， i行j列的元素值表示不同元素的协方差

因为现在变量之间是相互独立的，所以只有对角线上 (i = j)存在非0元素，其他地方都等于0，且元素与它本身的协方差就等于方差
∑是一个对角阵，根据对角矩阵的性质，它的逆矩阵表示为：
$({\sum }_{}^{}{)}^{-1}=\left[\begin{array}{cccc}\frac{1}{{\sigma }_1^2}& 0& \cdots & 0\\ 0& \frac{1}{{\sigma }_2^2}& \cdots & 0\\ ⋮& \cdots & \cdots & ⋮\\ 0& 0& \cdots & \frac{1}{{\sigma }_n^2}\end{array}\right]$
对角矩阵的行列式 = 对角元素的乘积
${\sigma }_z={\left|{\sum }_{}^{}\right|}^{\frac{1}{2}}={\sigma }_1{\sigma }_2.....{\sigma }_n$

展开有
$\frac{\partial L(\theta )}{\partial {\mu }_1}=d\sum y\log (\frac{1}{\sqrt{2\pi }{)}^n|\sum {|}^{\frac{1}{2}}}exp(-\frac{1}{2}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1))$
$\frac{\partial L(\theta )}{\partial {\mu }_1}=d\sum y\log (\frac{1}{\sqrt{2\pi }{)}^n|\sum {|}^{\frac{1}{2}}})-y\frac{1}{2}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)$
这里的第一个$\sum$是求和符号

第一项和${\mu }_1$无关，所以也就是
$-\frac{1}{2}{d}_{\mu }\sum y(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)=0$
$-\frac{1}{2}{d}_{\mu }\sum y(x^T{\sum }^{-1}-{\mu }_1^T{\sum }^{-1})(x-{\mu }_1)=0$
$-\frac{1}{2}{d}_{\mu }\sum y(x^T{\sum }^{-1}x-x^T{\sum }^{-1}{\mu }_1-{\mu }_1^T{\sum }^{-1}x+{\mu }_1^T{\sum }^{-1}{\mu }_1)=0$
$-\frac{1}{2}{d}_{\mu }\sum y(x^T{\sum }^{-1}x-x^T{\sum }^{-1}{\mu }_1-{\mu }_1^T{\sum }^{-1}x+{\mu }_1^T{\sum }^{-1}{\mu }_1)=0$
也就是
$-\frac{1}{2}\sum y(-2x^T{\sum }^{-1}+2{\sum }^{-1}{\mu }_1）=0$
$\sum y(x^T{\sum }^{-1}-{\sum }^{-1}{\mu }_1）=0$
$\sum y(x-{\mu }_1）=0$
$\sum xy=\sum y{\mu }_1$
${\mu }_1=\frac{\sum xy}{\sum y}=\frac{\sum xy}{N1}$

求$\sum$
矩阵的迹相关定理：
$tr(A)=\sum A_{ii}$
$tr(AB)=tr(BA)$
$tr(ABC)=tr(CBA)$
$\frac{\partial tr(AB)}{\partial A}=B^T$
|A|表示矩阵A的行列式
$\frac{\partial |A|}{\partial A}=|A|.A^{-1}$
如果a∈实数，则有tr(a)=a
令：
C 1 = { x i ∣ y = 1 ; x i ∈ 1... n } C1=\{x_i |y=1;x_i \in 1...n\} C1={xi​∣y=1;xi​∈1...n}
C 2 = { x i ∣ y = 0 ; x i ∈ 1... n } C2=\{x_i |y=0;x_i \in 1...n\} C2={xi​∣y=0;xi​∈1...n}
$|C1|=N1$
$|C2|=N2$
$N1+N2=N$
原函数对$\sum$ 求偏导有
$\frac{\partial J(\theta )}{\partial \sum }=d(\sum _{x_i\in C1}\log (N({\mu }_1,\sum )+\sum _{x_i\in C2}\log (N({\mu }_2,\sum ))=0$
令：
$f({\mu }_1)=\sum _{x_i\in C1}\log (N({\mu }_1,\sum )$
$f({\mu }_1)={\sum }_{x_i\in C1}\log (\frac{1}{\sqrt{2\pi }{)}^n|\sum {|}^{\frac{1}{2}}}exp(-\frac{1}{2}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1))$
$f({\mu }_1)={\sum }_{x_i\in C1}\log \frac{1}{\sqrt{2\pi }{)}^n|\sum {|}^{\frac{1}{2}}}-\frac{1}{2}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)$
$f({\mu }_1)={\sum }_{x_i\in C1}\log \frac{1}{\sqrt{2\pi }{)}^n}-\frac{1}{2}\log |\sum |-\frac{1}{2}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)$
把求和符号带人有
$f({\mu }_1)={\sum }_{x_i\in C1}\log \frac{1}{\sqrt{2\pi }{)}^n}-\frac{1}{2}{\sum }_{x_i\in C1}\log |\sum |-\frac{1}{2}{\sum }_{x_i\in C1}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)$

$\sum _{x_i\in C1}\log \frac{1}{\sqrt{2\pi }{)}^n}和\sum 无关，记作常识C3$

$-\frac{1}{2}\sum _{x_i\in C1}\log |\sum |=-\frac{1}{2}N1\log |\sum |$

由于 $(x-{\mu }_1{)}^T$是(1xn)维
${\sum }^{-1}$是pxp 维
$(x-{\mu }_1)$是px1维
所以$(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)$结果是实数
也就可以表示为
$(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)=tr((x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1))$
$=tr((x-{\mu }_1{)}^T(x-{\mu }_1){\sum }^{-1})$
${\sum }_{x_i\in C1}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)={\sum }_{x_i\in C1}tr((x-{\mu }_1{)}^T(x-{\mu }_1){\sum }^{-1})$
${\sum }_{x_i\in C1}(x-{\mu }_1{)}^T{\sum }^{-1}(x-{\mu }_1)=tr({\sum }_{x_i\in C1}(x-{\mu }_1{)}^T(x-{\mu }_1){\sum }^{-1})$
因为有：方差矩阵
$S1=\frac{1}{N1}(\sum _{x_i\in C1}(x-{\mu }_1{)}^T(x-{\mu }_1))$
所以
${\sum }_{x_i\in C1}(x-{\mu }_1{)}^T|\sum {|}^{-1}(x-{\mu }_1)=N1tr(S1{\sum }^{-1})$

$f({\mu }_1)=-\frac{1}{2}(C3+N1\log |\sum |+N1tr(S1{\sum }^{-1})$
同理：
$f({\mu }_2)=-\frac{1}{2}(C4+N2\log |\sum |+N2tr(S2{\sum }^{-1})$
对原函数求导可以写为
$\frac{\partial J(\theta )}{\partial \sum }=d(f({\mu }_1)+f({\mu }_2))=0$
$\frac{\partial J(\theta )}{\partial \sum }=d(-\frac{1}{2}(N1\log |\sum |+N1tr(S1{\sum }^{-1})-\frac{1}{2}(N2\log |\sum |+N2tr(S2{\sum }^{-1}))=0$
$\frac{\partial J(\theta )}{\partial \sum }=d(-\frac{1}{2}(N\log |\sum |+N1tr(S1{\sum }^{-1})+N2tr(S2{\sum }^{-1})))=0$
$\frac{\partial J(\theta )}{\partial \sum }=-\frac{1}{2}(N\frac{1}{|\sum |}|\sum |{\sum }^{-1}+N1tr({\sum }^{-1}S1)+N2tr({\sum }^{-1}S2))=0$
$\frac{\partial J(\theta )}{\partial \sum }=-\frac{1}{2}(N{\sum }^{-1}+N1tr({\sum }^{-1}S1)+N2tr({\sum }^{-1}S2))=0$
$\frac{\partial J(\theta )}{\partial \sum }=-\frac{1}{2}(N{\sum }^{-1}-N1S{1}^T{\sum }^{-2}-N2S{2}^T{\sum }^{-2})=0$
两边乘以${\sum }^2$有
$N\sum =N1S{1}^T+N2S{2}^T$
$\sum =\frac{N1S{1}^T+N2S{2}^T}{N}$
由于方差矩阵的对称型，所以可写为
$\sum =\frac{N1S1+N2S2}{N}$