支持向量机—SMO算法详细总结汇总

引言

面对这样的优化问题: $$\begin{array}{rl}& \underset{\alpha }{\min }\frac{1}{2}\sum _{i=1}^m\sum _{j=1}^m{\alpha }^{(i)}{\alpha }^{(j)}K(x^{(i)}\cdot x^{(j)})-\sum _{i=1}^m{\alpha }^{(i)}\\ & s.t.\sum _{i=1}^m{\alpha }^{(i)}{y}^{(i)}=0,0\le {\alpha }^{(i)}\le C,i=1,2,\cdots ,m\end{array}$$

参数:$\alpha$是拉格朗日乘子构成的变量，有$\alpha =({\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_N{)}^T$，每一个拉格朗日乘子对应一个样本点，例如:${\alpha }_1\to (x_1,y_1)$。

1.坐标下降法

每次只完成一个参数的更新。

我们接下来举一个简单的例子来说明:
$$\arg \underset{x_1,x_2}{\min }f(x_1,x_2)=x_1^2+2x_2^2-x_1{x}_2+1$$

:初始值$(x_1^{(0)},x_2^{(0)}{)}^T$

:选择其中一个进行更新

例如选择$x_1^{(0)}$，固定$x_2^{(0)}$，使得问题转化为:
$$\arg \underset{x_1,x_2}{\min }f(x_1,x_2^{(0)})$$

我们采用费马原理如下:
$$\frac{\partial f}{\partial x_1}=2x_1-x_2^{(0)}=0\Rightarrow x_2=\frac{x_2^0}{2}$$

:更新$x_2$，固定$x_1=x_1^1$，求解$x_2$

$$\arg \underset{x_1,x_2}{\min }f(x_1^{(1)},x_2)$$

$$\frac{\partial f}{\partial x_2}=4x_2-x_1^{(1)}\Rightarrow x_2^{(1)}=\frac{x_1^1}{4}$$

:重复上面的$\to$直到收敛为止。

坐标下降法可以应用在非线性支持向量机吗？

不妨选取${\alpha }_1$，固定${\alpha }_2,{\alpha }_3,\cdots ,{\alpha }_N$。

:初始值${\alpha }^{(0)}=({\alpha }_1^{(0)},{\alpha }_2^{(0)},\cdots ,{\alpha }_N^{(0)})$

:在固定${\alpha }_2,{\alpha }_3,\cdots ,{\alpha }_N$下，求${\alpha }_1$

使得: $$\begin{array}{rl}& \underset{\alpha }{\min }W({\alpha }_1,{\alpha }_2^{(0)},{\alpha }_3^{(0)},\cdots ,{\alpha }_N^{(0)})\\ & s.t.{\alpha }_i{y}_i=-\sum _{i=2}^N{\alpha }_i^{(0)}{y}_i,0\le {\alpha }_i\le C\end{array}$$

此时${\alpha }_1$可以直接由约束条件得到具体的值，无法进行更新。

所以坐标下降法用于非线性支持向量机的方法失败了。

但是我们可以换个思路，固定剩余的$N-2$个变量，求两个变量，这就是接下来要讲的序列最小最优算法的最初想法。

2.SMO算法

SMO算法要解决如下问题: $$\begin{array}{rl}& \underset{\alpha }{\min }\frac{1}{2}\sum _{i=1}^m\sum _{j=1}^m{\alpha }^{(i)}{\alpha }^{(j)}K(x^{(i)}\cdot x^{(j)})-\sum _{i=1}^m{\alpha }^{(i)}\\ & \text{s.t.}\sum _{i=1}^m{\alpha }^{(i)}{y}^{(i)}=0,0\le {\alpha }^{(i)}\le C,i=1,2,\cdots ,m\end{array}$$

我们选择两个变量，${\alpha }_1,{\alpha }_2$，其他变量固定，于是SMO的最优化问题的子问题为:
$$\begin{array}{rl}\underset{{\alpha }_1,{\alpha }_2}{\min }W({\alpha }^{(1)},{\alpha }^{(2)})& =\frac{1}{2}{K}_{11}{{\alpha }^{(1)}}^2+\frac{1}{2}{K}_{22}{{\alpha }^{(2)}}^2+y^{(1)}{y}^{(2)}{K}_{12}{\alpha }^{(1)}{\alpha }^{(2)}\\ & -({\alpha }^{(1)}+{\alpha }^{(2)})+y^{(1)}{\alpha }^{(1)}\sum _{i=3}^m{y}^{(i)}{\alpha }^{(i)}{K}_{i1}+y^{(2)}{\alpha }^{(2)}\sum _{i=3}^m{y}^{(i)}{\alpha }^{(i)}{K}_{i2}\\ \text{s.t.}{\alpha }^{(1)}{y}^{(1)}+& {\alpha }^{(2)}{y}^{(2)}=-\sum _{i=3}^m{y}^{(i)}{\alpha }^{(i)}=\zeta ,0\le {\alpha }^{(i)}\le C,i=1,2\end{array}$$

其中，$K_{ij}=K(x_i,x_j),i,j=1,2,\cdots ,N$，$\zeta$是常数，目标函数中省略了不含${\alpha }^{(1)},{\alpha }^{(2)}$项。

为了叙述简单，记: $$\begin{array}{rl}& g(x)=\sum _{i=1}^m{\alpha }^{(i)}{y}^{(i)}K(x^{(i)},x)+b\\ & E_i=g(x^{(i)})-y^{(i)}=(\sum _{j=1}^m{\alpha }^{(j)}{y}^{(j)}K(x^{(j)},x^{(i)})+b)-y^{(i)}\\ & V_i=\sum _{j=3}^m{\alpha }^{(j)}{y}^{(j)}K(x^{(j)},x^{(i)})=g(x^{(i)})-\sum _{j=1}^2{\alpha }^{(j)}{y}^{(j)}K(x^{(j)},x^{(i)})-b\end{array}$$

目标函数可写成:
$$W({\alpha }^{(1)},{\alpha }^{(2)})=\frac{1}{2}{K}_{11}{{\alpha }^{(1)}}^2+\frac{1}{2}{K}_{22}{{\alpha }^{(2)}}^2+y^{(1)}{y}^{(2)}{K}_{12}{\alpha }^{(1)}{\alpha }^{(2)}-({\alpha }^{(1)}+{\alpha }^{(2)})+y^{(1)}{\alpha }^{(1)}{v}_1+y^{(2)}{\alpha }^{(2)}{v}_2$$

我们的表示方法如下: $$\begin{array}{rl}& K_{11}=K(x_1,x_2),K_{22}=K(x_2,x_2)\\ & K_{12}=K(x_1,x_2),K_{1j}=K(x_1,x_j)\\ & K_{2j}=K(x_2,x_j)\end{array}$$

由${\alpha }^{(1)}{y}^{(1)}=\zeta -{\alpha }^{(2)}{y}^{(2)}$可将${\alpha }^{(1)}$表示为:
$${\alpha }^{(1)}=(\zeta -{\alpha }^{(2)}{y}^{(2)})y^{(1)}$$ 且${y^{(i)}}^2=1$。

$$\begin{array}{rl}W({\alpha }^{(2)})& =\frac{1}{2}{K}_{11}(\zeta -{\alpha }^{(2)}{y}^{(2)}{)}^2+\frac{1}{2}{K}_{22}{{\alpha }^{(2)}}^2+y^{(2)}{K}_{12}(\zeta -{\alpha }^{(2)}{y}^{(2)}){\alpha }^{(2)}\\ & -((\zeta -{\alpha }^{(2)}{y}^{(2)})y^{(1)}+{\alpha }^{(2)})+(\zeta -{\alpha }^{(2)}{y}^{(2)})v_1+y^{(2)}{\alpha }^{(2)}{v}_2\end{array}$$

对${\alpha }^{(2)}$求导:
$$\frac{\partial W}{\partial {\alpha }^{(2)}}=K_{11}{\alpha }^{(2)}+K_{22}{\alpha }^{(2)}-2K_{12}{\alpha }^{(2)}-K_{11}\zeta y^{(2)}+K_{12}\zeta y^{(2)}+y^{(1)}{y}^{(2)}-1-v_1{y}^{(2)}+y^{(2)}{v}_2$$

令其为0，得到: $$\begin{array}{rl}& (K_{11}+K_{22}-2K_{12}){\alpha }^{(2)}=y^{(2)}(y^{(2)}-y^{(1)}+\zeta K_{11}-\zeta K_{12}+v_1-v_2)\\ & =y^{(2)}(y^{(2)}-y^{(1)}+\zeta K_{11}-\zeta K_{12}+(g(x_1)-\sum _{j=1}^2{\alpha }^{(j)}{y}^{(j)}{K}_{1j}-b)-(g(x_2)-\sum _{j=1}^2{\alpha }^{(j)}{y}^{(j)}{K}_{2j}-b))\end{array}$$

将$\zeta ={\alpha }_{old}^{(1)}{y}^{(1)}+{\alpha }_{old}^{(2)}{y}^{(2)}$代入，得到:
$$\begin{array}{rl}(K_{11}+K_{22}-2K_{12}){\alpha }_{new,unc}^{(2)}& =y^{(2)}((K_{11}+K_{22}-2K_{12}){\alpha }_{old}^{(2)}{y}^{(2)}+y^{(2)}-y^{(1)}+g(x_1)-g(x_2))\\ & =(K_{11}+K_{22}-2K_{12}){\alpha }_{old}^{(2)}+y^{(2)}(E_1-E_2)\end{array}$$

令$\eta =K_{11}+K_{22}-2K_{12}$代入，得到:
$${\alpha }_{new,unc}^{(2)}={\alpha }_{old}^{(2)}+\frac{y^{(2)}(E_1-E_2)}{\eta }$$

上面的结果我们求得的是无约束的解，我们需要看一个经过约束条件后的迭代条件。条件如下:
$$\{\begin{array}{rl}& {\alpha }_1{y}_1+{\alpha }_2{y}_2=\zeta \\ & 0\le {\alpha }_1\le C\\ & 0\le {\alpha }_2\le C\end{array}$$

我们分情况讨论:

$y_1=y_2$: $${\alpha }_1+{\alpha }_2=y_1\zeta =k$$

$y_1\ne y_2$: $${\alpha }_1-{\alpha }_2=y_1\zeta =k$$

我们对于第一种情况:

此时的区间为: $$\begin{array}{rl}& L=\max (0,\zeta -C)=\max (0,{\alpha }_{old}^{(2)}+{\alpha }_{old}^{(1)}-C)\\ & H=\min (C,\zeta )=\min (C,{\alpha }_{old}^{(2)}+{\alpha }_{old}^{(1)})\end{array}$$

我们对于第二种情况:

此时的区间为: $$\begin{array}{rl}& L=\max (0,-\zeta )=\max (0,{\alpha }_{old}^{(2)}-{\alpha }_{old}^{(1)})\\ & H=\min (C,C-\zeta )=\min (C,C+{\alpha }_{old}^{(2)}-{\alpha }_{old}^{(1)})\end{array}$$

我们的${\alpha }^{(2)}$的区间为: $$L\le {\alpha }^{(2)}\le H$$

因此我们得到的最终的${\alpha }^{(2)}$的解为: $${\alpha }_{new}^{(2)}=\{\begin{array}{rl}& H,{\alpha }_{new,unc}^{(2)}>H\\ & {\alpha }_{new,unc}^{(2)},L\le {\alpha }_{new,unc}^{(2)}\le H\\ & L,{\alpha }_{new,unc}^{(2)}<L\end{array}$$

求得${\alpha }_{new}^{(1)}$为:
$${\alpha }_{new}^{(1)}={\alpha }_{old}^{(1)}+y^{(1)}{y}^{(2)}({\alpha }_{old}^{(2)}-{\alpha }_{new}^{(2)})$$

我们接下来是计算$b$值，我们有: $$\begin{array}{rl}& g(x)=\sum _{i=1}^m{\alpha }^{(i)}{y}^{(i)}K(x^{(i)},x)+b\\ & E_i=g(x^{(i)})-y^{(i)}=(\sum _{j=1}^m{\alpha }^{(j)}{y}^{(j)}K(x^{(j)},x^{(i)})+b)-y^{(i)}\end{array}$$

(1)当$0<{\alpha }_{new}^{(1)}<C$时，有:
$$\sum _{i=1}^m{y}^{(i)}{\alpha }^{(i)}{K}_{i1}+b=y^{(1)}$$

因此:
$$b_{new}^{(1)}=y^{(1)}-\sum _{i=3}^m{y}^{(i)}{\alpha }^{(i)}{K}_{i1}-{\alpha }_{new}^{(1)}{y}^{(1)}{K}_{11}-{\alpha }_{new}^{(2)}{y}^{(2)}{K}_{21}$$

由$E_1$定义可知:
$$E_1=\sum _{i=3}^m{y}^{(i)}{\alpha }^{(i)}{K}_{i1}+{\alpha }_{old}^{(1)}{y}^{(1)}{K}_{11}+{\alpha }_{old}^{(2)}{y}^{(2)}{K}_{21}+b_{old}-y^{(1)}$$

变形得:
$$y^{(1)}-\sum _{i=3}^m{y}^{(i)}{\alpha }^{(i)}{K}_{i1}=-E_1+{\alpha }_{old}^{(1)}{y}^{(1)}{K}_{11}+{\alpha }_{old}^{(2)}{y}^{(2)}{K}_{21}+b_{old}$$

代入$b_{new}^{(1)}=y^{(1)}-{\sum }_{i=3}^m{y}^{(i)}{\alpha }^{(i)}{K}_{i1}-{\alpha }_{new}^{(1)}{y}^{(1)}{K}_{11}-{\alpha }_{new}^{(2)}{y}^{(2)}{K}_{21}$得:
$$b_{new}^{(1)}=-E_1-y^{(1)}{K}_{11}({\alpha }_{new}^{(1)}-{\alpha }_{old}^{(1)})-y^{(2)}{K}_{21}({\alpha }_{new}^{(2)}-{\alpha }_{old}^{(2)})+b_{old}$$

(2)同理若$0<{\alpha }_{new}^{(2)}<C$，可得:
$$b_{new}^{(2)}=-E_2-y^{(1)}{K}_{12}({\alpha }_{new}^{(1)}-{\alpha }_{old}^{(1)})-y^{(2)}{K}_{22}({\alpha }_{new}^{(2)}-{\alpha }_{old}^{(2)})+b_{old}$$

(3)若${\alpha }_{new}^{(1)}$和${\alpha }_{new}^{(2)}$同时满足$0<{\alpha }_{new}^{(i)}<C$，则:
$$b_{new}^{(1)}=b_{new}^{(2)}$$

若${\alpha }_{new}^{(1)}$和${\alpha }_{new}^{(2)}$是0或者$C$，则:
$$b_{new}=\frac{b_{new}^{(1)}+b_{new}^{(2)}}{2}$$

3.SMO算法推导结果

$$\begin{array}{rl}& g(x)=\sum _{i=1}^m{\alpha }^{(i)}{y}^{(i)}K(x^{(i)},x)+b\\ & E_i=g(x^{(i)})-y^{(i)}=(\sum _{j=1}^m{\alpha }^{(j)}{y}^{(j)}K(x^{(j)},x^{(i)})+b)-y^{(i)}\\ & \eta =K_{11}+K_{22}-2K_{12}\\ & {\alpha }_{new,unc}^{(2)}={\alpha }_{old}^{(2)}+\frac{y^{(2)}(E_1-E_2)}{\eta }\end{array}$$

若$y^{(1)}\ne y^{(2)}$: $$\begin{array}{rl}& L=\max (0,-\zeta )=\max (0,{\alpha }_{old}^{(2)}-{\alpha }_{old}^{(1)})\\ & H=\min (C,C-\zeta )=\min (C,C+{\alpha }_{old}^{(2)}-{\alpha }_{old}^{(1)})\end{array}$$

若$y^{(1)}=y^{(2)}$: $$\begin{array}{rl}& L=\max (0,\zeta -C)=\max (0,{\alpha }_{old}^{(2)}+{\alpha }_{old}^{(1)}-C)\\ & H=\min (C,\zeta )=\min (C,{\alpha }_{old}^{(2)}+{\alpha }_{old}^{(1)})\end{array}$$

$${\alpha }_{new}^{(2)}=\{\begin{array}{rl}& H,{\alpha }_{new,unc}^{(2)}>H\\ & {\alpha }_{new,unc}^{(2)},L\le {\alpha }_{new,unc}^{(2)}\le H\\ & L,{\alpha }_{new,unc}^{(2)}<L\end{array}$$

$$\begin{array}{rl}& {\alpha }_{new}^{(1)}={\alpha }_{old}^{(1)}+y^{(1)}{y}^{(2)}({\alpha }_{old}^{(2)}-{\alpha }_{new}^{(2)})\\ & b_{new}^{(1)}=-E_1-y^{(1)}{K}_{11}({\alpha }_{new}^{(1)}-{\alpha }_{old}^{(1)})-y^{(2)}{K}_{21}({\alpha }_{new}^{(2)}-{\alpha }_{old}^{(2)})\\ & b_{new}^{(2)}=-E_2-y^{(1)}{K}_{12}({\alpha }_{new}^{(1)}-{\alpha }_{old}^{(1)})-y^{(2)}{K}_{22}({\alpha }_{new}^{(2)}-{\alpha }_{old}^{(2)})\end{array}$$

若$0<{\alpha }_{new}^{(1)}<C$，则: $$b=b_{new}^{(1)}$$

若$0<{\alpha }_{new}^{(2)}<C$，则: $$b=b_{new}^{(2)}$$

其他情况: $$b_{new}=\frac{b_{new}^{(1)}+b_{new}^{(2)}}{2}$$
)}K_{22}(\alpha_{new}^{{(2)}-\alpha_{old}}{(2)})
\end{split}$$

若$0<{\alpha }_{new}^{(1)}<C$，则: $$b=b_{new}^{(1)}$$

若$0<{\alpha }_{new}^{(2)}<C$，则: $$b=b_{new}^{(2)}$$

其他情况: $$b_{new}=\frac{b_{new}^{(1)}+b_{new}^{(2)}}{2}$$