Kernel Method: 3.线性判别分析与广义判别分析

3. LDA and GDA

3.1 Linear Discriminant Analysis

寻找一个方向向量满足：

• 投影后的各类均值距离最大
• 投影后每一类的样本与均值的距离最小

即增大类均值距离，增大每一类的样本聚集程度。目的是降低样本投影之间的重叠部分，增大可分性

$L$：样本类别数目；$N_i$：第$i$类样本的数目；$N$全部样本数目；${\mathbf{x}}_j^{(i)}$：第$j$类中的第$i$个样本

将所有的样本投影到方向向量$\mathbf{v}$上，${\mathbf{v}}^T{\mathbf{x}}_1^{(1)},\cdots ,{\mathbf{v}}^T{\mathbf{x}}_{N_1}^{(1)};{\mathbf{v}}^T{\mathbf{x}}_2^{(2)},\cdots ,{\mathbf{v}}^T{\mathbf{x}}_{N_2}^{(2)};\cdots ;{\mathbf{v}}^T{\mathbf{x}}_1^{(L)},\cdots ,{\mathbf{v}}^T{\mathbf{x}}_{N_L}^{(L)}$。

各类的均值为
$${\overline{\mathbf{m}}}_i=\frac{1}{N_i}\sum _{j=1}^{N_i}{\mathbf{v}}^T{\mathbf{x}}_j^{(i)}={\mathbf{v}}^T\left(\frac{1}{N_i}\sum _{j=1}^{N_i}{\mathbf{x}}_j^{(i)}\right)={\mathbf{v}}^T{\mathbf{m}}_i$$
其中$m_i$为原空间内第$i$类的均值。

然后计算每一类均值之间距离的权重平方和为
$$\begin{array}{rl}\sum _{i=1}^{L-1}\sum _{j=i+1}^L\frac{N_i}{N}\frac{N_j}{N}({\overline{\mathbf{m}}}_i-{\overline{\mathbf{m}}}_j{)}^2& =\sum _{i=1}^{L-1}\sum _{j=i+1}^L\frac{N_i}{N}\frac{N_j}{N}({\overline{\mathbf{m}}}_i-{\overline{\mathbf{m}}}_j)({\overline{\mathbf{m}}}_i-{\overline{\mathbf{m}}}_j{)}^T\\ & =\sum _{i=1}^{L-1}\sum _{j=i+1}^L\frac{N_i}{N}\frac{N_j}{N}({\mathbf{v}}^T{\mathbf{m}}_i-{\mathbf{v}}^T{\mathbf{m}}_j)({\mathbf{v}}^T{\mathbf{m}}_i-{\mathbf{v}}^T{\mathbf{m}}_j{)}^T\\ & =\sum _{i=1}^{L-1}\sum _{j=i+1}^L\frac{N_i}{N}\frac{N_j}{N}{\mathbf{v}}^T({\mathbf{m}}_i-{\mathbf{m}}_j)({\mathbf{m}}_i-{\mathbf{m}}_j{)}^T\mathbf{v}\\ & ={\mathbf{v}}^T\left(\sum _{i=1}^{L-1}\sum _{j=i+1}^L\frac{N_i}{N}\frac{N_j}{N}({\mathbf{m}}_i-{\mathbf{m}}_j)({\mathbf{m}}_i-{\mathbf{m}}_j{)}^T\right)\mathbf{v}\\ & ={\mathbf{v}}^T{S}_b^{LDA}\mathbf{v}\end{array}$$

$$\begin{array}{rl}{S}_b^{LDA}& =\sum _{i=1}^{L-1}\sum _{j=i+1}^L\frac{N_i}{N}\frac{N_j}{N}({\mathbf{m}}_i-{\mathbf{m}}_j)({\mathbf{m}}_i-{\mathbf{m}}_j{)}^T\\ & =\frac{1}{2}\sum _{i=1}^L\sum _{j=1}^L\frac{N_i}{N}\frac{N_j}{N}({\mathbf{m}}_i-{\mathbf{m}}_j)({\mathbf{m}}_i-{\mathbf{m}}_j{)}^T\\ & =\frac{1}{2}\sum _{i=1}^L\sum _{j=1}^L\frac{N_i}{N}\frac{N_j}{N}({\mathbf{m}}_i{\mathbf{m}}_i^T-{\mathbf{m}}_i{\mathbf{m}}_j^T-{\mathbf{m}}_j{\mathbf{m}}_i^T+{\mathbf{m}}_j{\mathbf{m}}_j^T)\\ & =\frac{1}{2}\left(\sum _{i=1}^L\sum _{j=1}^L\frac{N_i}{N}\frac{N_j}{N}{\mathbf{m}}_i{\mathbf{m}}_i^T-\sum _{i=1}^L\sum _{j=1}^L\frac{N_i}{N}\frac{N_j}{N}{\mathbf{m}}_i{\mathbf{m}}_j^T-\sum _{i=1}^L\sum _{j=1}^L\frac{N_i}{N}\frac{N_j}{N}{\mathbf{m}}_j{\mathbf{m}}_i^T+\sum _{i=1}^L\sum _{j=1}^L\frac{N_i}{N}\frac{N_j}{N}{\mathbf{m}}_j{\mathbf{m}}_j^T\right)\\ & =\frac{1}{2}\left(\sum _{i=1}^L\frac{N_i}{N}{\mathbf{m}}_i{\mathbf{m}}_i^T\sum _{j=1}^L\frac{N_j}{N}-\sum _{i=1}^L\frac{N_i}{N}{\mathbf{m}}_i\sum _{j=1}^L\frac{N_j}{N}{\mathbf{m}}_j^T-\sum _{j=1}^L\frac{N_j}{N}{\mathbf{m}}_j\sum _{i=1}^L\frac{N_i}{N}{\mathbf{m}}_i^T+\sum _{i=1}^L\frac{N_i}{N}\sum _{j=1}^L\frac{N_j}{N}{\mathbf{m}}_j^T{\mathbf{m}}_j^T\right)\\ & =\frac{1}{2}\left(\sum _{i=1}^L\frac{N_i}{N}{\mathbf{m}}_i{\mathbf{m}}_i^T-{\mathbf{m}}_0{\mathbf{m}}_0^T-{\mathbf{m}}_0{\mathbf{m}}_0^T+\sum _L^{j=1}\frac{N_j}{N}{\mathbf{m}}_j{\mathbf{m}}_j^T\right)\\ & =\sum _L^{i=1}\frac{N_i}{N}{\mathbf{m}}_i{\mathbf{m}}_i^T-{\mathbf{m}}_0{\mathbf{m}}_0^T\\ & =\sum _{i=1}^L\frac{N_i}{N}({\mathbf{m}}_i-{\mathbf{m}}_0)({\mathbf{m}}_i-{\mathbf{m}}_0{)}^T(与E[(x-\bar{x}{)}^2]=E[x^2]-{\bar{x}}^2相似)\end{array}$$

其中，
$${\mathbf{m}}_0=\sum _L^{i=1}\frac{N_i}{N}{\mathbf{m}}_i=\sum _L^{i=1}\frac{N_i}{N}\sum _{k=1}^{N_i}\frac{1}{N_i}{\mathbf{x}}_k^{(i)}=\sum _L^{i=1}\sum _{N_i}^{k=1}\frac{1}{N}{\mathbf{x}}_k^{(i)}$$
综上，组间分散矩阵为：
$$S_b^{LDA}=\sum _{i=1}^{L-1}\sum _{j=i+1}^L\frac{N_i}{N}\frac{N_j}{N}({\mathbf{m}}_i-{\mathbf{m}}_j)({\mathbf{m}}_i-{\mathbf{m}}_j{)}^T=\sum _{i=1}^L\frac{N_i}{N}({\mathbf{m}}_i-{\mathbf{m}}_0)({\mathbf{m}}_i-{\mathbf{m}}_0{)}^T$$
相当与每个集群的形心到整个集群的形心之间的距离乘上质量权重。

类方差和为
$$\begin{array}{rl}\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}({\mathbf{v}}^T{x}_J^{(i)}-{\overline{\mathbf{m}}}_i{)}^2& =\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}({\mathbf{v}}^T{\mathbf{x}}_j^{(i)}-{\mathbf{v}}^T{\mathbf{m}}_i)({\mathbf{v}}^T{\mathbf{x}}_j^{(i)}-{\mathbf{v}}^T{\mathbf{m}}_i{)}^T\\ & ={\mathbf{v}}^T\left(\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}({\mathbf{x}}_j^{(i)}-{\mathbf{m}}_i)({\mathbf{x}}_j^{(i)}-{\mathbf{m}}_i{)}^T\right)\mathbf{v}\\ & ={\mathbf{v}}^T{S}_w^{LDA}\mathbf{v}\end{array}$$
所以，组内分散矩阵为：
$$S_w^{LDA}=\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}({\mathbf{x}}_j^{(i)}-{\mathbf{m}}_i)({\mathbf{x}}_j^{(i)}-{\mathbf{m}}_i{)}^T$$
第一主元向量可以由以下计算：
$$\mathbf{v}={\mathrm{argmax}}_{\mathbf{v}\in {\mathbb{R}}^d}\frac{{\mathbf{v}}^T{S}_b^{LDA}\mathbf{v}}{{\mathbf{v}}^T{S}_w^{LDA}\mathbf{v}}={\mathrm{argmax}}_{{\mathbf{v}}^T{S}_b^{LDA}\mathbf{v}=1}{\mathbf{v}}^T{S}_b^{LDA}\mathbf{v}.$$
由Lagrangian方法可得，
$$f(\mathbf{v},\lambda )={\mathbf{v}}^T{S}_b^{LDA}\mathbf{v}-\lambda ({\mathbf{v}}^T{S}_w^{LDA}\mathbf{v}-1)$$

$$\begin{array}{rl}\frac{\partial f}{\partial \mathbf{v}}& =2S_b^{LDA}\mathbf{v}-2\lambda S_w^{LDA}\mathbf{v}\iff (S_w^{LDA}{)}^{-1}{S}_b^{LDA}\mathbf{v}=\lambda \mathbf{v}\\ \frac{\partial f}{\partial \lambda }& ={\mathbf{v}}^T{S}_w^{LDA}\mathbf{v}-1=0\iff {\mathbf{v}}^T{S}_w^{LDA}\mathbf{v}=1\end{array}$$

当满足以上条件时，${\mathbf{v}}^T{S}_b^{LDA}\mathbf{v}=\lambda {\mathbf{v}}^T{S}_w^{LDA}\mathbf{v}=\lambda$。

综上，求解第一主元等价于求解下列最大广义特征值，
$$S_b^{LDA}\mathbf{u}=\lambda S_w^{LDA}\mathbf{u},\mathbf{v}=\frac{1}{\sqrt{{\mathbf{u}}^T{S}_w^{LDA}\mathbf{u}}}\mathbf{u}$$
其中后一项保证${\mathbf{v}}^T{S}_b^{LDA}\mathbf{v}=1$。

3.2 Generalized Discriminant Analysis

$L$：样本类别数目；

$N_i$：第$i$类样本的数目；

$N$全部样本数目；

$\varphi ({\mathbf{x}}_j^{(i)})$：第$j$类中的第$i$个样本；

$X_i^T=[\varphi ({\mathbf{x}}_1^{(i)}),\cdots ,\varphi ({\mathbf{x}}_{N_i}^{(i)})]$；

$X^T=[X_1^T,\cdots ,X_L^T]$。

假设在空间$H$内样本均值为零：${\mathbf{m}}_0=0$

则组间分散矩阵为：
$$S_b^{GDA}=\sum _{i=1}^L\frac{N_i}{N}({\mathbf{m}}_i-{\mathbf{m}}_0)({\mathbf{m}}_i-{\mathbf{m}}_0{)}^T=\sum _{i=1}^L\frac{N_i}{N}{\mathbf{m}}_i{\mathbf{m}}_i^T$$
组内分散矩阵为：
$$S_w^{GDA}=\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}\varphi ({\mathbf{x}}_j^{(i)})\varphi ({\mathbf{x}}_j^{(i)}{)}^T$$

$${\mathbf{m}}_i=\frac{1}{N_i}\sum _{j=1}^{N_i}\varphi ({\mathbf{x}}_j^{(i)})=\frac{1}{N_i}[\varphi ({\mathbf{x}}_1^{(i)}),\cdots ,\varphi ({\mathbf{x}}_{N_i}^{(i)})]\left[\begin{array}{c}1\\ ⋮\\ 1\end{array}\right]=\frac{1}{N_i}{X}_i^T{1}_{N_i\times 1}$$

$${\mathbf{m}}_i{\mathbf{m}}_i^T=\frac{1}{N_i^2}{X}_i^T{1}_{N_i\times 1}{1}_{1\times N_i}{X}_i=\frac{1}{N_i}{X}_i^T{B}_i{X}_i$$

其中，$B_i=\frac{1}{N_i}{1}_{N_i\times N_i}$。组间分散矩阵为：
$$S_b^{GDA}=\sum _{i=1}^L\frac{N_i}{N}{\mathbf{m}}_i{\mathbf{m}}_i^T=\frac{1}{N}\sum _{i=1}^L{X}_i^T{B}_i{X}_i=\frac{1}{N}\left[\begin{array}{ccc}{X}_1^T& \cdots & X_L^T\end{array}\right]\left[\begin{array}{ccc}{B}_1& & 0\\ & \ddots & \\ 0& & B_L\end{array}\right]\left[\begin{array}{c}{X}_i\\ ⋮\\ X_L\end{array}\right]=\frac{1}{N}{X}^{T}BX$$
组内分散矩阵为：
$$\begin{array}{rl}{S}_w^{GDA}& =\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}\varphi ({\mathbf{x}}_j^{(i)})\varphi ({\mathbf{x}}_j^{(i)}{)}^T\\ & =\frac{1}{N}\sum _{i=1}^L\left[\begin{array}{ccc}\varphi ({\mathbf{x}}_1^{(i)})& \cdots & \varphi ({\mathbf{x}}_{N_i}^{(i)})\end{array}\right]\left[\begin{array}{c}\varphi ({\mathbf{x}}_1^{(i)}{)}^T\\ ⋮\\ \varphi ({\mathbf{x}}_{N_i}^{(i)}{)}^T\end{array}\right]\\ & =\frac{1}{N}\sum _{i=1}^L{X}_i^T{X}_i\\ & =\frac{1}{N}\left[\begin{array}{ccc}{X}_1^T& \cdots & X_L^T\end{array}\right]\left[\begin{array}{c}{X}_1\\ ⋮\\ X_L\end{array}\right]\\ & =\frac{1}{N}{X}^{T}X\end{array}$$
同理，
$$\begin{gathered} S_b^{GDA}\mathbf{v}=\lambda S_w^{GDA}\mathbf{v} \\ i.e.(\frac{1}{N}{X}^{T}BX)\mathbf{v}=\lambda (\frac{1}{N}{X}^{T}X)\mathbf{v}(X未知) \end{gathered}$$
假设$v$可以由样本的线性组合表示，即
$$\mathbf{v}=\sum _{i=1}^L\sum _{j=1}^{N_i}{\alpha }_j^{(i)}\varphi ({\mathbf{x}}_j^{(i)})=X^T\mathbf{\alpha }.$$
将假设代入上式，
$$\begin{array}{rl}& \Rightarrow X^{T}BX{X}^T\mathbf{\alpha }=\lambda X^{T}X{X}^T\mathbf{\alpha }\\ & \Rightarrow X{X}^{T}BX{X}^T\mathbf{\alpha }=\lambda X{X}^{T}X{X}^T\mathbf{\alpha }\\ & \Rightarrow (KBK)\mathbf{\alpha }=\lambda (KK)\mathbf{\alpha }\end{array}$$
计算上式可获得$\mathbf{\alpha }$，将测试样本投影到$\mathbf{v}=X^T\mathbf{\alpha }$上，
$${\mathbf{v}}^T\varphi (\mathbf{x})=(X^T\mathbf{\alpha }{)}^T\varphi (\mathbf{x})={\mathbf{\alpha }}^T\left[\begin{array}{c}\varphi ({\mathbf{x}}_1{)}^T\\ ⋮\\ \varphi ({\mathbf{x}}_N{)}^T\end{array}\right]\varphi (\mathbf{x})={\mathbf{\alpha }}^T\left[\begin{array}{c}\kappa ({\mathbf{x}}_1,\mathbf{x})\\ ⋮\\ \kappa ({\mathbf{x}}_N,\mathbf{x})\end{array}\right]$$
Ex: 在GDA中，组内分散矩阵为：
$$S_w^{GDA}=\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}\varphi ({\mathbf{x}}_j^{(i)})\varphi ({\mathbf{x}}_j^{(i)}{)}^T$$
而在LDA中，
$$\begin{gathered} S_w^{LDA}=\sum _{i=1}^L\sum _{j=1}^{N_i}\frac{1}{N}(\varphi ({\mathbf{x}}_j^{(i)})-{\mathbf{m}}_i)(\varphi ({\mathbf{x}}_j^{(i)})-{\mathbf{m}}_i{)}^T \\ {\mathbf{m}}_i=\frac{1}{N_i}\sum _{j=1}^{N_i}\varphi ({\mathbf{x}}_j^{(i)}) \end{gathered}$$
能否由LDA推导GDA组内分散矩阵，并且找到$W$，使得$S_w^{LDA}=X^{T}WX$.