LDA推导

个人笔记，非教程

$$W:D\times d$$
$$Z:d\times N$$
$$X:D\times N$$

类内散度矩阵

$$S_W=\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)(x_j^{(i)}-{\mu }_i{)}^2$$

投影后的类内散度矩阵

$$\begin{array}{rl}{S}_W^{\prime \prime \prime }& =\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)(W^T{x}_j^{(i)}-W^T{\mu }_i{)}^2\\ & =\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)(W^T(x_j^{(i)}-{\mu }_i){)}^2\\ & =\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)(W^T(x_j^{(i)}-{\mu }_i)(x_j^{(i)}-{\mu }_i{)}^{T}W)\\ & =W^T\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)(x_j^{(i)}-{\mu }_i)(x_j^{(i)}-{\mu }_i{)}^{T}W\\ & =W^T{S}_{W}W\end{array}$$

类间散度矩阵

$$S_B=\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)({\mu }_i-{\mu }_j{)}^2$$

投影后的类间散度矩阵

$$\begin{array}{rl}{S}_B^{\prime \prime \prime }& =\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)({\mu }_i-{\mu }_j{)}^2\\ & =\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)(W^T({\mu }_i-{\mu }_j){)}^2\\ & =\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)(W^T({\mu }_i-{\mu }_j)({\mu }_i-{\mu }_j{)}^{T}W)\\ & =W^T\sum _{i=1}^C\sum _{j=1}^{M_i}p(i,j)({\mu }_i-{\mu }_j)({\mu }_i-{\mu }_j{)}^{T}W\\ & =W^T{S}_{B}W\end{array}$$

最大化投影后的类间散度和最小化投影后的类内散度

$$\max |W^T{S}_{B}W|$$
$$\min |W^T{S}_{W}W|$$

等价于

$$\max \frac{|W^T{S}_{B}W|}{|W^T{S}_{W}W|}$$

不失一般性我们可以令$|W^T{S}_{W}W|=1$，则

$$\max |W^T{S}_{B}W|$$
$$s.t.|W^T{S}_{W}W|=1$$

拉格朗日

$$L(W,\lambda )=W^T{S}_{B}W+\lambda (W^T{S}_{W}W-1)$$
$$\frac{\partial L(W,\lambda )}{\partial W}=2S_{B}W+2\lambda S_{W}W$$

令$\frac{\partial L(W,\lambda )}{\partial W}=0$

$$S_{B}W=\lambda S_{W}W$$
$$S_W^{-1}{S}_{B}W=\lambda W$$

$W$即$S_W^{-1}{S}_B$的特征向量

$$Z=W^{T}X$$