高斯判别分析（GDA）公式推导

解：将概率分布代入对数似然函数，

$l(\psi ,{\mu }_0,{\mu }_1,\sum )={\sum }_{i=1}^{m}log{p}_{X|Y}(x^{(i)}|y^{(i)};{\mu }_0,{\mu }_1,\sum )+{\sum }_{i=1}^{m}log{p}_Y(y^{(i)};\psi )$

$={\sum }_{i=1}^m(1-y^{(i)})log\frac{1}{(2\pi {)}^{n/2}|\sum {|}^{1/2}}exp(\frac{1}{2}(x^{(i)}-{\mu }_0{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_0))$

$+{\sum }_{i=1}^m{y}^{(i)}log\frac{1}{(2\pi {)}^{n/2}|\sum {|}^{1/2}}exp(\frac{1}{2}(x^{(i)}-{\mu }_1{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_1))$

$+{\sum }_{i=1}^{m}log{\psi }^{y^{(i)}}(1-\psi {)}^{1-y^{(i)}}$

求取$l(\psi ,{\mu }_0,{\mu }_1,\sum )$的最大值，令

$\frac{\partial }{\partial \psi }l(\psi ,{\mu }_0,{\mu }_1,\sum )=0$ （1）

${\nabla }_{{\mu }_0}l(\psi ,{\mu }_0,{\mu }_1,\sum )=0$ （2）

${\nabla }_{{\mu }_1}l(\psi ,{\mu }_0,{\mu }_1,\sum )=0$ （3）

${\nabla }_{\sum }l(\psi ,{\mu }_0,{\mu }_1,\sum )=0$ （4）

对于（1）式：

$\frac{\partial }{\partial \psi }{\sum }_{i=1}^m{y}^{(i)}log\psi +(1-y^{(i)})log(1-\psi )=0$

${\sum }_{i=1}^m\frac{y^{(i)}}{\psi }+\frac{1-y^{(i)}}{1-\psi }=0$

${\sum }_{i=1}^m{y}^{(i)}(1-\psi )+(1-y^{(i)})\psi =0$

${\sum }_{i=1}^m{y}^{(i)}=m\psi$

ψ = ∑ i = 1 m 1 { y ( i ) = 1 } m \psi=\frac{\sum^m_{i=1}1\{y^{(i)}=1\}}{m} ψ=m∑i=1m​1{y(i)=1}​

对于（2）式：

${\nabla }_{{\mu }_0}{\sum }_{i=1}^m(1-y^{(i)})(x^{(i)}-{\mu }_0{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_0)=0$

${\sum }_{i=1}^m(1-y^{(i)})(x^{(i)}-{\mu }_0{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_0)=0$

${\sum }_{i=1}^m(1-y^{(i)})[{\sum }^{-1}(x^{(i)}-{\mu }_0)d(x^{(i)}-{\mu }_0{)}^T+(x^{(i)}-{\mu }_0{)}^T{\sum }^{-1}d(x^{(i)}-{\mu }_0)]=0$

${\sum }_{i=1}^m(1-y^{(i)}){\sum }^{-1}(x^{(i)}-{\mu }_0)=0$

${\sum }_{i=1}^m(1-y^{(i)})(x^{(i)}-{\mu }_0)=0$

${\sum }_{i=1}^m(1-y^{(i)})x^{(i)}={\sum }_{i=1}^m(1-y^{(i)}){\mu }_0$

μ 0 = ∑ i = 1 m 1 { y ( i ) = 0 } x ( i ) / ∑ i = 1 m 1 { y ( i ) = 0 } \mu_0=\sum^m_{i=1}1\{y^{(i)}=0\}x^{(i)}/\sum^m_{i=1}1\{y^{(i)}=0\} μ0​=∑i=1m​1{y(i)=0}x(i)/∑i=1m​1{y(i)=0}

对于（3）式，类同（2）式：

μ 0 = ∑ i = 1 m 1 { y ( i ) = 1 } x ( i ) / ∑ i = 1 m 1 { y ( i ) = 1 } \mu_0=\sum^m_{i=1}1\{y^{(i)}=1\}x^{(i)}/\sum^m_{i=1}1\{y^{(i)}=1\} μ0​=∑i=1m​1{y(i)=1}x(i)/∑i=1m​1{y(i)=1}

对于（4）式：

${\nabla }_{\sum }(-\frac{m}{2}log|\sum |)-\frac{1}{2}{\sum }_{i=1}^m(1-y^{(i)})(x^{(i)}-{\mu }_0{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_0)-\frac{1}{2}{\sum }_{i=1}^m{y}^{(i)}(x^{(i)}-{\mu }_1{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_1)=0$

${\nabla }_{\sum }(mlog|\sum |)+{\nabla }_{\sum }{\sum }_{i=1}^m(1-y^{(i)})(x^{(i)}-{\mu }_0{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_0)+{\nabla }_{\sum }{\sum }_{i=1}^m{y}^{(i)}(x^{(i)}-{\mu }_1{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_1)=0$

已知协方差矩阵$S_i=\frac{1}{m}{\sum }_{i=1}^m(x^{(i)}-{\mu }_i)(x^{(i)}-{\mu }_i{)}^T$，将通过$S_i$简化表达上式

${\nabla }_{\sum }{\sum }_{i=1}^m(x^{(i)}-{\mu }_i{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_i)$

$={\nabla }_{\sum }tr({\sum }_{i=1}^m(x^{(i)}-{\mu }_i{)}^T{\sum }^{-1}(x^{(i)}-{\mu }_i))$

$={\nabla }_{\sum }tr({\sum }_{i=1}^m(x^{(i)}-{\mu }_i)(x^{(i)}-{\mu }_i{)}^T{\sum }^{-1})$

$={\nabla }_{\sum }tr(m_i{S}_i{\sum }^{-1})$

其中 m i = ∑ k = 1 m 1 { y ( k ) = i } m_i=\sum^m_{k=1}1\{y^{(k)}=i\} mi​=∑k=1m​1{y(k)=i}，

${\nabla }_{\sum }tr(m_i{S}_i{\sum }^{-1})=-m_i{S}_i^T{\sum }^{-2}$，

而${\nabla }_{\sum }(mlog|\sum |)=m\frac{1}{|\sum |}|\sum |{\sum }^{-1}=m{\sum }^{-1}$，

因此，（4）式可简化为

$m{\sum }^{-1}-{\sum }_i^2{m}_i{S}_i^T{\sum }^{-2}=0$

$\sum =\frac{1}{m}{\sum }_i^2{m}_i{S}_i^T$

$\sum =\frac{1}{m}{\sum }_{i=1}^m(x^{(i)}-{\mu }_{y^{(i)}}{)}^T(x^{(i)}-{\mu }_{y^{(i)}})$