SVM原理-2

SVM (support vector machine)

本文着眼于SMO原理和非线性分类器。

SMO

网页。
网页。
platt论文 Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines。

非线性分类器

一个思路是将低维数据投射到高维数据，在高维空间中寻找超平面。

则代价函数由$\overrightarrow{x_i}\cdot \overrightarrow{x_j}$变为$⟨\varphi (x^{(i)}),\varphi (x^{(j)})⟩$，那么对于$k$维，其时间复杂度是$O(k^2)$（需要转化为高维空间），难以接受。一个可行$trick$是使用$kernelfunction$，其高维点积为低维点积转换后相乘，$\overrightarrow{h(x_i)}\cdot \overrightarrow{h(x_j)}$，则时间复杂度为$O(k)$。

常用$kernel$：

Kernel	expression
Linear	$K(x,y)=x^{T}y+c$
Polynomial	$K(x,y)=(a{x}^{T}y+c{)}^d,(a,c⩾0)$
Radial Basis	$K(x,y)=exp(-\gamma \Vert x-y{\Vert }^2),(\gamma ⩾0)$
Gaussiaan	$K(x,y)=exp(-\frac{\Vert x-y{\Vert }^2}{2{\sigma }^2})$

Valid Kernel: 半正定对称矩阵。证明。

SMO derivation(concrete)

We have $\sum {\alpha }_i{y}_i=0$, so we have to change ${\alpha }_i,{\alpha }_j$ simultaneously. Assume we choose ${\alpha }_1,{\alpha }_2$，then ${\alpha }_1{y}_1+{\alpha }_2{y}_2=-{\sum }_{i=3}{\alpha }_i{y}_i=\zeta$.

target:
$minL=\frac{1}{2}{\sum }_{i,j}{\alpha }_i{\alpha }_j{y}_i{y}_j<\overrightarrow{x_i},\overrightarrow{x_j}>-\sum {\alpha }_i$
$$\begin{array}{rlr}L=& \frac{1}{2}{\alpha }_1^2{K}_{11}+\frac{1}{2}{\alpha }_2^2{K}_{22}+{\alpha }_1{\alpha }_2{y}_1{y}_2{K}_{12}+{\alpha }_1{y}_1\sum _{i=3}{\alpha }_i{y}_i{K}_{i1}+{\alpha }_2{y}_2\sum _{i=3}{\alpha }_i{y}_i{K}_{i2}-({\alpha }_1+{\alpha }_2)+const& \\ {\alpha }_1=& y_1\zeta -y_1{y}_2{\alpha }_2& \\ \frac{\partial {\alpha }_1}{\partial {\alpha }_2}=& -y_1{y}_2& \\ \frac{\partial L}{\partial {\alpha }_2}=& {\alpha }_1{K}_{11}\frac{\partial {\alpha }_1}{\partial {\alpha }_2}+{\alpha }_2{K}_{22}+y_1{y}_2{K}_{12}\frac{\partial {\alpha }_1{\alpha }_2}{\partial {\alpha }_2}+\frac{\partial {\alpha }_1}{\partial {\alpha }_2}{y}_1\sum _{i=3}{\alpha }_i{y}_i{K}_{i1}+y_2\sum _{i=3}{\alpha }_i{y}_i{K}_{i2}-1-\frac{\partial {\alpha }_1}{\partial {\alpha }_2}& \\ =& -y_1{y}_2{\alpha }_1{K}_{11}+{\alpha }_2{K}_{22}+y_1{y}_2{K}_{12}({\alpha }_1-y_1{y}_2{\alpha }_2)-y_2\sum _{i=3}{\alpha }_i{y}_i{K}_{i1}+y_2\sum _{i=3}{\alpha }_i{y}_i{K}_{i2}+y_1{y}_2-1& \\ =& (K_{11}+K_{22}-2K_{12}){\alpha }_2-y_2{K}_{11}\zeta +y_2{K}_{12}\zeta +y_1{y}_2-1-y_2\sum _{i=3}{\alpha }_i{y}_i(K_{i1}-K_{i2})& \end{array}$$

let $\frac{\partial L}{\partial {\alpha }_2}=0$, then $(K_{11}+K_{22}-2K_{12}){\alpha }_2=y_2((K_{11}-K_{12})\zeta +y_2-y_1+{\sum }_{i=3}{\alpha }_i{y}_i(K_{i1}-K_{i2}))$

$$\begin{array}{rl}(K_{11}+K_{22}-2K_{12}){\alpha }_2=& y_2((K_{11}-K_{12})\zeta +y_2-y_1+\sum _{i=3}{\alpha }_i{y}_i(K_{i1}-K_{i2}))\\ (K_{11}+K_{22}-2K_{12}){\alpha }_2=& y_2(\sum {\alpha }_i{y}_i{K}_{i1}-\sum {\alpha }_i{y}_i{K}_{i2}+y_2-y_1+{\alpha }_2{y}_2(K_{11}+K_{22}-2K_{12}))\\ (K_{11}+K_{22}-2K_{12}){\alpha }_2^{\ast }=& (K_{11}+K_{22}-2K_{12}){\alpha }_2+y_2((\sum {\alpha }_i{y}_i{K}_{i1}-y_1)-(\sum {\alpha }_i{y}_i{K}_{i2}-y_2))\end{array}$$

let $E_i={\sum }_j{\alpha }_i{y}_i{K}_{ij}+b-y_i$, $\eta =K_{11}+K_{22}-2K_{12}$
${\alpha }_2^{\ast }={\alpha }_2+\frac{y_2(E_1-E_2)}{\eta }$

${\alpha }_2^{\ast }$ also needs to satisfy $[L,H]$

${\alpha }_2^{new}=\{\begin{array}{cc}H,& (H<{\alpha }_2^{\ast })\\ {\alpha }_2^{\ast },& (L⩽{\alpha }_2^{\ast }⩽H)\\ L,& ({\alpha }_2^{\ast }<L)\end{array}$

${\alpha }_1^{new}=y_1(\eta -y_2{\alpha }_2^{new})=y_1(y_1{\alpha }_1+y_2{\alpha }_2-y_2{\alpha }_2^{new})={\alpha }_1+y_1{y}_2({\alpha }_2-{\alpha }_2^{new})$

iterations choice

$u=\sum y_j{\alpha }_{j}K(\overrightarrow{x_j},\overrightarrow{x})-b$
KKT condition of QP problem:
$$\begin{array}{r}{\alpha }_i=0\iff y_i{u}_i⩾1\\ 0<{\alpha }_i<C\iff y_i{u}_i=1\\ {\alpha }_i=C\iff y_i{u}_i⩽1\end{array}$$

first choice the point which violates KKT, then choice the point of max $\Vert E_2-E_1\Vert$.