【机器学习基础】第二十一课：支持向量机之核方法

【机器学习基础】系列博客为参考周志华老师的《机器学习》一书，自己所做的读书笔记。

1.表示定理

无论SVM还是SVR，学得的模型总能表示成核函数$\kappa (\mathbf{x},{\mathbf{x}}_i)$的线性组合。不仅如此，事实上我们有下面这个称为“表示定理”（representer theorem）的更一般的结论：

表示定理： 令$\mathbb{H}$为核函数$\kappa$对应的再生核希尔伯特空间，$\Vert \Vert h\Vert {\Vert }_{\mathbb{H}}$表示$\mathbb{H}$空间中关于$h$的范数，对于任意单调递增函数$\Omega :[0,\infty ]\mapsto \mathbb{R}$和任意非负损失函数$\ell :{\mathbb{R}}^m\mapsto [0,\infty ]$，优化问题

$$\underset{h\in \mathbb{H}}{\min }F(h)=\Omega (||h|{|}_{\mathbb{H}})+\ell (h({\mathbf{x}}_1),h({\mathbf{x}}_2),...,h({\mathbf{x}}_m))\text{\hspace{1em}(1)}$$

的解总可写为

$$h^{\ast }(\mathbf{x})=\sum _{i=1}^m{\alpha }_i\kappa (\mathbf{x},{\mathbf{x}}_i)\text{\hspace{1em}(2)}$$

表示定理对损失函数没有限制，对正则化项$\Omega$仅要求单调递增，甚至不要求$\Omega$是凸函数，意味着对于一般的损失函数和正则化项，优化问题(1)的最优解$h^{\ast }(\mathbf{x})$都可表示为核函数$\kappa (\mathbf{x},{\mathbf{x}}_i)$的线性组合；这显示出核函数的巨大威力。

2.核方法

人们发展出一系列基于核函数的学习方法，统称为 “核方法”（kernel methods） 。最常见的，是通过“核化”（即引入核函数）来将线性学习器拓展为非线性学习器。下面以线性判别分析为例来演示如何通过核化来对其进行非线性拓展，从而得到 “核线性判别分析”（Kernelized Linear Discriminant Analysis，简称KLDA）。

我们先假设可通过某种映射$\varphi :\chi \mapsto \mathbb{F}$将样本映射到一个特征空间$\mathbb{F}$，然后在$\mathbb{F}$中执行线性判别分析，以求得：

$$h(\mathbf{x})={\mathbf{w}}^T\varphi (\mathbf{x})\text{\hspace{1em}(3)}$$

KLDA的学习目标是：

$$\underset{\mathbf{w}}{\max }J(\mathbf{w})=\frac{{\mathbf{w}}^T{\mathbf{S}}_b^{\varphi }\mathbf{w}}{{\mathbf{w}}^T{\mathbf{S}}_w^{\varphi }\mathbf{w}}\text{\hspace{1em}(4)}$$

其中${\mathbf{S}}_b^{\varphi }$和${\mathbf{S}}_w^{\varphi }$分别为训练样本在特征空间$\mathbb{F}$中的类间散度矩阵和类内散度矩阵。令${\mathbf{X}}_i$表示第 i ∈ { 0 , 1 } i \in \{0,1\} i∈{0,1}类样本的集合，其样本数为$m_i$；总样本数$m=m_0+m_1$。第$i$类样本在特征空间$\mathbb{F}$中的均值为：

$${\mu }_i^{\varphi }=\frac{1}{m_i}\sum _{\mathbf{x}\in {\mathbf{X}}_i}\varphi (\mathbf{x})\text{\hspace{1em}(5)}$$

两个散度矩阵分别为：

$${\mathbf{S}}_b^{\varphi }=({\mu }_1^{\varphi }-{\mu }_0^{\varphi })({\mu }_1^{\varphi }-{\mu }_0^{\varphi }{)}^T\text{\hspace{1em}(6)}$$

$${\mathbf{S}}_w^{\varphi }=\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}(\varphi (\mathbf{x})-{\mu }_i^{\varphi })(\varphi (\mathbf{x})-{\mu }_i^{\varphi }{)}^T\text{\hspace{1em}(7)}$$

通常我们难以知道映射$\varphi$的具体形式，因此使用核函数$\kappa (\mathbf{x},{\mathbf{x}}_i)=\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})$来隐式地表达这个映射和特征空间$\mathbb{F}$。把$J(\mathbf{w})$作为式(1)中的损失函数$\ell$，再令$\Omega \equiv 0$，由表示定理，函数$h(\mathbf{x})$可写为：

$$h(\mathbf{x})=\sum _{i=1}^m{\alpha }_i\kappa (\mathbf{x},{\mathbf{x}}_i)\text{\hspace{1em}(8)}$$

$\equiv$为恒等于，即无论条件如何变化，等式始终保持不变。

因为有式(3)等于式(8)：

$${\mathbf{w}}^T\varphi (\mathbf{x})=\sum _{i=1}^m{\alpha }_i\kappa (\mathbf{x},{\mathbf{x}}_i)\text{\hspace{1em}(9)}$$

将$\kappa (\mathbf{x},{\mathbf{x}}_i)=\varphi (\mathbf{x}{)}^T\varphi ({\mathbf{x}}_i)$代入式(9)：

$${\mathbf{w}}^T\varphi (\mathbf{x})=\sum _{i=1}^m{\alpha }_i\varphi (\mathbf{x}{)}^T\varphi ({\mathbf{x}}_i)\text{\hspace{1em}(10)}$$

$${\mathbf{w}}^T\varphi (\mathbf{x})=\varphi (\mathbf{x}{)}^T\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\text{\hspace{1em}(11)}$$

由于${\mathbf{w}}^T\varphi (\mathbf{x})$的计算结果为标量，而标量的转置等于其本身，所以：

$${\mathbf{w}}^T\varphi (\mathbf{x})=({\mathbf{w}}^T\varphi (\mathbf{x}){)}^T=\varphi (\mathbf{x}{)}^T\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\text{\hspace{1em}(12)}$$

$${\mathbf{w}}^T\varphi (\mathbf{x})=\varphi (\mathbf{x}{)}^T\mathbf{w}=\varphi (\mathbf{x}{)}^T\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\text{\hspace{1em}(13)}$$

$$\mathbf{w}=\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\text{\hspace{1em}(14)}$$

令$\mathbf{K}\in {\mathbb{R}}^{m\times m}$为核函数$\kappa$所对应的核矩阵，$(\mathbf{K}{)}_{ij}=\kappa ({\mathbf{x}}_i,{\mathbf{x}}_j)$。令 l i ∈ { 1 , 0 } m × 1 \mathbf l_i \in \{ 1,0 \}^{m\times 1} li​∈{1,0}m×1为第$i$类样本的指示向量，即${\mathbf{l}}_i$的第$j$个分量为1当且仅当${\mathbf{x}}_j\in {\mathbf{X}}_i$，否则${\mathbf{l}}_i$的第$j$个分量为0。再令：

$${\hat{\mu }}_0=\frac{1}{m_0}{\mathbf{Kl}}_0\text{\hspace{1em}(15)}$$

$${\hat{\mu }}_1=\frac{1}{m_1}{\mathbf{Kl}}_1\text{\hspace{1em}(16)}$$

$$\mathbf{M}=({\hat{\mu }}_0-{\hat{\mu }}_1)({\hat{\mu }}_0-{\hat{\mu }}_1{)}^T\text{\hspace{1em}(17)}$$

$$\mathbf{N}={\mathbf{KK}}^T-\sum _{i=0}^1{m}_i{\hat{\mu }}_i{\hat{\mu }}_i^T\text{\hspace{1em}(18)}$$

于是，式(4)等价为：

$$\underset{\alpha }{\max }J(\alpha )\frac{{\alpha }^T\mathbf{M}\alpha }{{\alpha }^T\mathbf{N}\alpha }\text{\hspace{1em}(19)}$$

显然，使用线性判别分析求解方法即可得到$\alpha$，进而可由式(8)得到投影函数$h(\mathbf{x})$。

2.1.式(15)、式(16)的推导

为了详细地说明此公式的计算原理，下面首先先举例说明，然后再在例子的基础上延展出其一般形式。假设此时仅有4个样本，其中第1和第3个样本的标记为0，第2和第4个样本的标记为1，那么此时：

$$m=4\text{\hspace{1em}(20)}$$

$$m_0=2,m_1=2\text{\hspace{1em}(21)}$$

X 0 = { x 1 , x 3 } , X 1 = { x 2 , x 4 } (22) X_0=\{\mathbf x_1,\mathbf x_3 \},X_1=\{\mathbf x_2,\mathbf x_4 \} \tag{22} X0​={x1​,x3​},X1​={x2​,x4​}(22)

$$\mathbf{K}=\left[\begin{array}{cccc}\kappa ({\mathbf{x}}_1,{\mathbf{x}}_1)& \kappa ({\mathbf{x}}_1,{\mathbf{x}}_2)& \kappa ({\mathbf{x}}_1,{\mathbf{x}}_3)& \kappa ({\mathbf{x}}_1,{\mathbf{x}}_4)\\ \kappa ({\mathbf{x}}_2,{\mathbf{x}}_1)& \kappa ({\mathbf{x}}_2,{\mathbf{x}}_2)& \kappa ({\mathbf{x}}_2,{\mathbf{x}}_3)& \kappa ({\mathbf{x}}_2,{\mathbf{x}}_4)\\ \kappa ({\mathbf{x}}_3,{\mathbf{x}}_1)& \kappa ({\mathbf{x}}_3,{\mathbf{x}}_2)& \kappa ({\mathbf{x}}_3,{\mathbf{x}}_3)& \kappa ({\mathbf{x}}_3,{\mathbf{x}}_4)\\ \kappa ({\mathbf{x}}_4,{\mathbf{x}}_1)& \kappa ({\mathbf{x}}_4,{\mathbf{x}}_2)& \kappa ({\mathbf{x}}_4,{\mathbf{x}}_3)& \kappa ({\mathbf{x}}_4,{\mathbf{x}}_4)\end{array}\right]\in {\mathbb{R}}^{4\times 4}\text{\hspace{1em}(23)}$$

$${\mathbf{l}}_0=\left[\begin{array}{c}1\\ 0\\ 1\\ 0\end{array}\right]\in {\mathbb{R}}^{4\times 1}\text{\hspace{1em}(24)}$$

$${\mathbf{l}}_1=\left[\begin{array}{c}0\\ 1\\ 0\\ 1\end{array}\right]\in {\mathbb{R}}^{4\times 1}\text{\hspace{1em}(25)}$$

所以：

$${\hat{\mu }}_0=\frac{1}{m_0}{\mathbf{Kl}}_0=\frac{1}{2}\left[\begin{array}{c}\kappa ({\mathbf{x}}_1,{\mathbf{x}}_1)+\kappa ({\mathbf{x}}_1,{\mathbf{x}}_3)\\ \kappa ({\mathbf{x}}_2,{\mathbf{x}}_1)+\kappa ({\mathbf{x}}_2,{\mathbf{x}}_3)\\ \kappa ({\mathbf{x}}_3,{\mathbf{x}}_1)+\kappa ({\mathbf{x}}_3,{\mathbf{x}}_3)\\ \kappa ({\mathbf{x}}_4,{\mathbf{x}}_1)+\kappa ({\mathbf{x}}_4,{\mathbf{x}}_3)\end{array}\right]\in {\mathbb{R}}^{4\times 1}\text{\hspace{1em}(26)}$$

$${\hat{\mu }}_1=\frac{1}{m_1}{\mathbf{Kl}}_1=\frac{1}{2}\left[\begin{array}{c}\kappa ({\mathbf{x}}_1,{\mathbf{x}}_2)+\kappa ({\mathbf{x}}_1,{\mathbf{x}}_4)\\ \kappa ({\mathbf{x}}_2,{\mathbf{x}}_2)+\kappa ({\mathbf{x}}_2,{\mathbf{x}}_4)\\ \kappa ({\mathbf{x}}_3,{\mathbf{x}}_2)+\kappa ({\mathbf{x}}_3,{\mathbf{x}}_4)\\ \kappa ({\mathbf{x}}_4,{\mathbf{x}}_2)+\kappa ({\mathbf{x}}_4,{\mathbf{x}}_4)\end{array}\right]\in {\mathbb{R}}^{4\times 1}\text{\hspace{1em}(27)}$$

根据此结果易得${\hat{\mu }}_0,{\hat{\mu }}_1$的一般形式为：

$${\hat{\mu }}_0=\frac{1}{m_0}{\mathbf{Kl}}_0=\frac{1}{m_0}\left[\begin{array}{c}{\sum }_{\mathbf{x}\in X_0}\kappa ({\mathbf{x}}_1,\mathbf{x})\\ {\sum }_{\mathbf{x}\in X_0}\kappa ({\mathbf{x}}_2,\mathbf{x})\\ ⋮\\ {\sum }_{\mathbf{x}\in X_0}\kappa ({\mathbf{x}}_m,\mathbf{x})\end{array}\right]\in {\mathbb{R}}^{m\times 1}\text{\hspace{1em}(28)}$$

$${\hat{\mu }}_1=\frac{1}{m_1}{\mathbf{Kl}}_1=\frac{1}{m_1}\left[\begin{array}{c}{\sum }_{\mathbf{x}\in X_1}\kappa ({\mathbf{x}}_1,\mathbf{x})\\ {\sum }_{\mathbf{x}\in X_1}\kappa ({\mathbf{x}}_2,\mathbf{x})\\ ⋮\\ {\sum }_{\mathbf{x}\in X_1}\kappa ({\mathbf{x}}_m,\mathbf{x})\end{array}\right]\in {\mathbb{R}}^{m\times 1}\text{\hspace{1em}(29)}$$

2.2.式(19)的推导

首先将式(14)代入式(4)的分子可得：

$$\begin{array}{rl}{\mathbf{w}}^T{\mathbf{S}}_b^{\varphi }\mathbf{w}& ={\left(\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\right)}^T\cdot {\mathbf{S}}_b^{\varphi }\cdot \sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\\ & =\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\cdot {\mathbf{S}}_b^{\varphi }\cdot \sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\end{array}\text{\hspace{1em}(30)}$$

其中：

$$\begin{array}{rl}{\mathbf{S}}_b^{\varphi }& =({\mu }_1^{\varphi }-{\mu }_0^{\varphi })({\mu }_1^{\varphi }-{\mu }_0^{\varphi }{)}^T\\ & =\left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\varphi (\mathbf{x})-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x})\right){\left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\varphi (\mathbf{x})-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x})\right)}^T\\ & =\left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\varphi (\mathbf{x})-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x})\right)\left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\varphi (\mathbf{x}{)}^T-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x}{)}^T\right)\end{array}\text{\hspace{1em}(31)}$$

将式(31)代入式(30)：

$$\begin{array}{rl}{\mathbf{w}}^T{\mathbf{S}}_b^{\varphi }\mathbf{w}& =\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\cdot \left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\varphi (\mathbf{x})-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x})\right)\left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\varphi (\mathbf{x}{)}^T-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x}{)}^T\right)\cdot \sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\\ & =\left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})\right)\left(\frac{1}{m_1}\sum _{\mathbf{x}\in X_1}\sum _{i=1}^m{\alpha }_i\varphi (\mathbf{x}{)}^T\varphi ({\mathbf{x}}_i)-\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\sum _{i=1}^m{\alpha }_i\varphi (\mathbf{x}{)}^T\varphi ({\mathbf{x}}_i)\right)\end{array}\text{\hspace{1em}(32)}$$

由于$\kappa ({\mathbf{x}}_i,\mathbf{x})=\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})$为标量，所以其转置等于本身，也即$\kappa ({\mathbf{x}}_i,\mathbf{x})=\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})=(\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x}){)}^T=\varphi (\mathbf{x}{)}^T\varphi ({\mathbf{x}}_i)=\kappa ({\mathbf{x}}_i,\mathbf{x}{)}^T$，将其代入式(32)可得：

$${\mathbf{w}}^T{\mathbf{S}}_b^{\varphi }\mathbf{w}=\left(\frac{1}{m_1}\sum _{i=1}^m\sum _{\mathbf{x}\in X_1}{\alpha }_i\kappa ({\mathbf{x}}_i,\mathbf{x})-\frac{1}{m_0}\sum _{i=1}^m\sum _{\mathbf{x}\in X_0}{\alpha }_i\kappa ({\mathbf{x}}_i,\mathbf{x})\right)\left(\frac{1}{m_1}\sum _{i=1}^m\sum _{\mathbf{x}\in X_1}{\alpha }_i\kappa ({\mathbf{x}}_i,\mathbf{x})-\frac{1}{m_0}\sum _{i=1}^m\sum _{\mathbf{x}\in X_0}{\alpha }_i\kappa ({\mathbf{x}}_i,\mathbf{x})\right)\text{\hspace{1em}(33)}$$

令$\alpha =({\alpha }_1;{\alpha }_2;\cdots ;{\alpha }_m{)}^T\in {\mathbb{R}}^{m\times 1}$，同时代入式(28)和式(29)，则式(33)可化简为：

$$\begin{array}{rl}{\mathbf{w}}^T{\mathbf{S}}_b^{\varphi }\mathbf{w}& =({\alpha }^T{\hat{\mu }}_1-{\alpha }^T{\hat{\mu }}_0)\cdot ({\hat{\mu }}_1^T\alpha -{\hat{\mu }}_0^T\alpha )\\ & ={\alpha }^T\cdot ({\hat{\mu }}_1-{\hat{\mu }}_0)\cdot ({\hat{\mu }}_1^T-{\hat{\mu }}_0^T)\cdot \alpha \\ & ={\alpha }^T\cdot ({\hat{\mu }}_1-{\hat{\mu }}_0)\cdot ({\hat{\mu }}_1-{\hat{\mu }}_0{)}^T\cdot \alpha \\ & ={\alpha }^T\mathbf{M}\alpha \end{array}\text{\hspace{1em}(34)}$$

以上便是式(19)分子部分的推导，下面继续推导式(19)的分母部分。将式(14)代入式(4)的分母可得：

$$\begin{array}{rl}{\mathbf{w}}^T{\mathbf{S}}_w^{\varphi }\mathbf{w}& ={\left(\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\right)}^T\cdot {\mathbf{S}}_w^{\varphi }\cdot \sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\\ & =\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\cdot {\mathbf{S}}_w^{\varphi }\cdot \sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\end{array}\text{\hspace{1em}(35)}$$

其中：

$$\begin{array}{rl}{\mathbf{S}}_w^{\varphi }& =\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}\left(\varphi (\mathbf{x})-{\mu }_i^{\varphi }\right){\left(\varphi (\mathbf{x})-{\mu }_i^{\varphi }\right)}^T\\ & =\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}\left(\varphi (\mathbf{x})-{\mu }_i^{\varphi }\right)\left(\varphi (\mathbf{x}{)}^T-{\left({\mu }_i^{\varphi }\right)}^T\right)\\ & =\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}\left(\varphi (\mathbf{x})\varphi (\mathbf{x}{)}^T-\varphi (\mathbf{x}){\left({\mu }_i^{\varphi }\right)}^T-{\mu }_i^{\varphi }\varphi (\mathbf{x}{)}^T+{\mu }_i^{\varphi }{\left({\mu }_i^{\varphi }\right)}^T\right)\\ & =\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}\varphi (\mathbf{x})\varphi (\mathbf{x}{)}^T-\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}\varphi (\mathbf{x}){\left({\mu }_i^{\varphi }\right)}^T-\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}{\mu }_i^{\varphi }\varphi (\mathbf{x}{)}^T+\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}{\mu }_i^{\varphi }{\left({\mu }_i^{\varphi }\right)}^T\end{array}\text{\hspace{1em}(36)}$$

由于：

$$\begin{array}{rl}\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}\varphi (\mathbf{x}){\left({\mu }_i^{\varphi }\right)}^T& =\sum _{\mathbf{x}\in {\mathbf{X}}_0}\varphi (\mathbf{x}){\left({\mu }_0^{\varphi }\right)}^T+\sum _{\mathbf{x}\in {\mathbf{X}}_1}\varphi (\mathbf{x}){\left({\mu }_1^{\varphi }\right)}^T\\ & =m_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T+m_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T\end{array}\text{\hspace{1em}(37)}$$

$$\begin{array}{rl}\sum _{i=0}^1\sum _{\mathbf{x}\in {\mathbf{X}}_i}{\mu }_i^{\varphi }\varphi (\mathbf{x}{)}^T& =\sum _{i=0}^1{\mu }_i^{\varphi }\sum _{\mathbf{x}\in {\mathbf{X}}_i}\varphi (\mathbf{x}{)}^T\\ & ={\mu }_0^{\varphi }\sum _{\mathbf{x}\in {\mathbf{X}}_0}\varphi (\mathbf{x}{)}^T+{\mu }_1^{\varphi }\sum _{\mathbf{x}\in {\mathbf{X}}_1}\varphi (\mathbf{x}{)}^T\\ & =m_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T+m_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T\end{array}\text{\hspace{1em}(38)}$$

将式(37)和式(38)代入式(36)：

$$\begin{array}{rl}{\mathbf{S}}_w^{\varphi }& =\sum _{\mathbf{x}\in D}\varphi (\mathbf{x})\varphi (\mathbf{x}{)}^T-2\left[m_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T+m_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T\right]+m_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T+m_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T\\ & =\sum _{\mathbf{x}\in D}\varphi (\mathbf{x})\varphi (\mathbf{x}{)}^T-m_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T-m_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T\end{array}\text{\hspace{1em}(39)}$$

将式(39)带回式(35)：

$$\begin{array}{rl}{\mathbf{w}}^T{\mathbf{S}}_w^{\varphi }\mathbf{w}& =\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\cdot {\mathbf{S}}_w^{\varphi }\cdot \sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\\ & =\sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\cdot \left(\sum _{\mathbf{x}\in D}\varphi (\mathbf{x})\varphi (\mathbf{x}{)}^T-m_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T-m_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T\right)\cdot \sum _{i=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i)\\ & =\sum _{i=1}^m\sum _{j=1}^m\sum _{\mathbf{x}\in D}{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})\varphi (\mathbf{x}{)}^T{\alpha }_j\varphi ({\mathbf{x}}_j)-\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T{m}_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T{\alpha }_j\varphi ({\mathbf{x}}_j)-\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T{m}_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T{\alpha }_j\varphi ({\mathbf{x}}_j)\end{array}\text{\hspace{1em}(40)}$$

其中，式(40)的第一项可化简为：

$$\begin{array}{rl}\sum _{i=1}^m\sum _{j=1}^m\sum _{\mathbf{x}\in D}{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})\varphi (\mathbf{x}{)}^T{\alpha }_j\varphi ({\mathbf{x}}_j)& =\sum _{i=1}^m\sum _{j=1}^m\sum _{\mathbf{x}\in D}{\alpha }_i{\alpha }_j\kappa ({\mathbf{x}}_i,\mathbf{x})\kappa ({\mathbf{x}}_j,\mathbf{x})\\ & ={\alpha }^T{\mathbf{KK}}^T\alpha \end{array}\text{\hspace{1em}(41)}$$

式(40)的第二项可化简为：

$$\begin{array}{rl}\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T{m}_0{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T{\alpha }_j\varphi ({\mathbf{x}}_j)& =m_0\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j\varphi ({\mathbf{x}}_i{)}^T{\mu }_0^{\varphi }{\left({\mu }_0^{\varphi }\right)}^T\varphi ({\mathbf{x}}_j)\\ & =m_0\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j\varphi ({\mathbf{x}}_i{)}^T\left[\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x})\right]{\left[\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x})\right]}^T\varphi ({\mathbf{x}}_j)\\ & =m_0\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j\left[\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi ({\mathbf{x}}_i{)}^T\varphi (\mathbf{x})\right]\left[\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\varphi (\mathbf{x}{)}^T\varphi ({\mathbf{x}}_j)\right]\\ & =m_0\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j\left[\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\kappa ({\mathbf{x}}_i,\mathbf{x})\right]\left[\frac{1}{m_0}\sum _{\mathbf{x}\in X_0}\kappa ({\mathbf{x}}_j,\mathbf{x})\right]\\ & =m_0{\alpha }^T{\hat{\mu }}_0{\hat{\mu }}_0^T\alpha \end{array}\text{\hspace{1em}(42)}$$

同理，式(40)的第三项可化简为：

$$\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i\varphi ({\mathbf{x}}_i{)}^T{m}_1{\mu }_1^{\varphi }{\left({\mu }_1^{\varphi }\right)}^T{\alpha }_j\varphi ({\mathbf{x}}_j)=m_1{\alpha }^T{\hat{\mu }}_1{\hat{\mu }}_1^T\alpha \text{\hspace{1em}(43)}$$

将式(41)、式(42)、式(43)带回到式(40)：

$$\begin{array}{rl}{\mathbf{w}}^T{\mathbf{S}}_w^{\varphi }\mathbf{w}& ={\alpha }^T{\mathbf{KK}}^T\alpha -m_0{\alpha }^T{\hat{\mu }}_0{\hat{\mu }}_0^T\alpha -m_1{\alpha }^T{\hat{\mu }}_1{\hat{\mu }}_1^T\alpha \\ & ={\alpha }^T\cdot \left({\mathbf{KK}}^T-m_0{\hat{\mu }}_0{\hat{\mu }}_0^T-m_1{\hat{\mu }}_1{\hat{\mu }}_1^T\right)\cdot \alpha \\ & ={\alpha }^T\cdot \left({\mathbf{KK}}^T-\sum _{i=0}^1{m}_i{\hat{\mu }}_i{\hat{\mu }}_i^T\right)\cdot \alpha \\ & ={\alpha }^T\mathbf{N}\alpha \end{array}\text{\hspace{1em}(44)}$$

3.参考资料

1. 南瓜书PumpkinBook

---

想要获取最新文章推送或者私聊谈人生，请关注我的个人微信公众号：⬇️x-jeff的AI工坊⬇️

个人博客网站：https://shichaoxin.com

GitHub：https://github.com/x-jeff

---