# 一、原理

## 1. 线性可分支持向量机

X = {${x}_{1},{x}_{2},...,{x}_{n}$$x_1,x_2,...,x_n$}
Y = {+1, -1}

##### 算法:
###### 原始算法：

(2)得到分离超平面 ${w}^{\ast }\cdot x+b=0$$w^*\cdot x+b=0$，决策函数 $f\left(x\right)=sign\left({\omega }^{\ast }\cdot x+{b}^{\ast }\right)$$f(x)=sign(\omega^*\cdot x+b^*)$

###### 对偶算法：

a. 对$\omega ,b$$\omega,b$ 求偏导数，并令其等于0，【${\mathrm{\nabla }}_{\omega }L\left(\omega ,b,\alpha \right)=\omega -\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}{x}_{i}=0$$\nabla_\omega L(\omega,b,\alpha)=\omega-\sum^N_{i=1}\alpha_i y_ix_i=0$ 】, 【${\mathrm{\nabla }}_{b}L\left(\omega ,b,\alpha \right)=\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}$$\nabla_bL(\omega,b,\alpha)=\sum_{i=1}^N\alpha_iy_i$】，得到【$\omega =\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}{x}_{i},\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}=0$$\omega=\sum^N_{i=1}\alpha_i y_ix_i,\sum_{i=1}^N\alpha_iy_i=0$

b. 将a中得到的结果代入拉格朗日函数，化简得到
$mi{n}_{\omega ,b}L\left(\omega ,b,\alpha \right)=-\frac{1}{2}\sum _{i=1}^{N}\sum _{j=1}^{N}{\alpha }_{i}{\alpha }_{j}{y}_{i}{y}_{j}\left({x}_{i}\cdot {x}_{j}\right)+\sum _{i=1}^{N}{\alpha }_{i}$$min_{\omega,b}L(\omega,b,\alpha)=-\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i\alpha_jy_iy_j(x_i\cdot x_j)+\sum_{i=1}^N\alpha_i$

c. 接下来求【$ma{x}_{\alpha }-\frac{1}{2}\sum _{i=1}^{N}\sum _{j=1}^{N}{\alpha }_{i}{\alpha }_{j}{y}_{i}{y}_{j}\left({x}_{i}\cdot {x}_{j}\right)+\sum _{i=1}^{N}{\alpha }_{i}$$max_{\alpha}-\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i\alpha_jy_iy_j(x_i\cdot x_j)+\sum_{i=1}^N\alpha_i$,s.t $\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}=0$$\sum_{i=1}^{N}\alpha_i y_i=0$, ${\alpha }_{i}\ge 0$$\alpha_i\geq0$】, 【$i=1,2,\cdots ,N$$i=1,2,\cdots,N$

d. 转化为求【$mi{n}_{\alpha }\frac{1}{2}\sum _{i=1}^{N}\sum _{j=1}^{N}{\alpha }_{i}{\alpha }_{j}{y}_{i}{y}_{j}\left({x}_{i}\cdot {x}_{j}\right)-\sum _{i=1}^{N}{\alpha }_{i}$$min_{\alpha}\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i\alpha_jy_iy_j(x_i\cdot x_j)-\sum_{i=1}^N\alpha_i$, s.t$\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}=0$$\sum_{i=1}^{N}\alpha_i y_i=0$, ${\alpha }_{i}\ge 0，i=1,2,\cdots ,N$$\alpha_i\geq0，i=1,2,\cdots,N$

${\mathrm{\nabla }}_{\omega }L\left(\omega ,b,\alpha \right)=0$$\nabla_\omega L(\omega,b,\alpha)=0$
${\alpha }_{i}\ge 0$$\alpha_i\geq0$
${y}_{i}f\left({x}_{i}\right)-1\ge 0$$y_if(x_i)-1\geq0$
${\alpha }_{i}\left({y}_{i}f\left({x}_{i}\right)-1\right)=0$$\alpha_i(y_if(x_i)-1)=0$

e. 由KKT条件，${\omega }^{\ast }=\sum _{i=1}^{N}{\alpha }^{\ast }{y}_{i}{x}_{i}$$\omega^*=\sum^N_{i=1}\alpha^*y_ix_i$ , 由于$\omega \ne 0$$\omega \not=0$可知存在下标 j，使得${\alpha }_{j}^{\ast }>0$$\alpha_j^*>0$ , 那么${y}_{i}\left(\omega \cdot {x}_{i}+b\right)=0,$$y_i(\omega\cdot x_i+b)=0,$$\omega$$\omega$ 代入，得到 ${b}^{\ast }={y}_{j}-\sum _{i=1}^{N}{\alpha }_{i}^{\ast }{y}_{i}\left({x}_{i}\cdot {x}_{j}\right)$$b^*=y_j-\sum^N_{i=1}\alpha_i^*y_i(x_i\cdot x_j)$

f.计算出 $\alpha$$\alpha$ 之后，可以得到$\omega ,b$$\omega,b$,分离超平面$\sum _{i=1}^{N}{\alpha }_{i}^{\ast }{y}_{i}\left(x\cdot {x}_{i}\right)+{b}^{\ast }$$\sum^N_{i=1}\alpha^*_iy_i(x\cdot x_i)+b^*$ , 分类决策$f\left(x\right)=sign\left(\sum _{i=1}^{N}{\alpha }_{i}^{\ast }{y}_{i}\left(x\cdot {x}_{i}\right)+{b}^{\ast }\right)$$f(x)=sign(\sum_{i=1}^N\alpha_i^*y_i(x\cdot x_i)+b^*)$

### 2. 线性支持向量机

##### 策略:

$min\frac{1}{2}||\omega |{|}^{2}+C\sum _{i}^{N}{\xi }_{i}$$min \frac{1}{2}||\omega||^2+C\sum_i^N\xi_i$
$s.t.{y}_{i}\left(\omega \cdot {x}_{i}+b\right)\ge 1-{\xi }_{i}$$s.t.y_i(\omega\cdot x_i+b)\geq1-\xi_i$
$\xi \ge 0$$\xi\geq0$

##### 算法：

$a.L\left(\omega ,b,\xi ,\alpha ,\mu \right)=\frac{1}{2}||\omega |{|}^{2}+C\sum _{i}^{N}{\xi }_{i}+\sum _{i}^{N}{\alpha }_{i}\left[1-{\xi }_{i}-{y}_{i}\left(\omega \cdot {x}_{i}+b\right)\right]+\sum _{i}^{N}\left(-{\mu }_{i}{\xi }_{i}\right)$$a.L(\omega,b,\xi,\alpha,\mu)=\frac{1}{2}||\omega||^2+C\sum_i^N\xi_i+\sum_i^N\alpha_i[1-\xi_i-y_i(\omega\cdot x_i+b)]+\sum_i^N(-\mu_i\xi_i)$

$L\left(\omega ,b,\xi ,\alpha ,\mu \right)=\frac{1}{2}||\omega |{|}^{2}+C\sum _{i}^{N}{\xi }_{i}+\sum _{i}^{N}{\alpha }_{i}-\sum _{i}^{N}{\alpha }_{i}{\xi }_{i}-\sum _{i}^{N}{\alpha }_{i}{y}_{i}\omega {x}_{i}-\sum _{i}^{N}{\alpha }_{i}{y}_{i}b-\sum _{i}^{N}{\mu }_{i}{\xi }_{i}$$L(\omega,b,\xi,\alpha,\mu)=\frac{1}{2}||\omega||^2+C\sum_i^N\xi_i+\sum_i^N\alpha_i-\sum_i^N\alpha_i\xi_i-\sum_i^N\alpha_iy_i\omega x_i-\sum_i^N\alpha_iy_ib-\sum_i^N\mu_i\xi_i$

b.原始问题的对偶问题$\omega ,b,\xi$$\omega,b,\xi$求偏导数，得到
${\mathrm{\nabla }}_{\omega }L=\omega -\sum _{i}^{N}{\alpha }_{i}{y}_{i}{x}_{i}$$\nabla_\omega L=\omega-\sum_i^N\alpha_iy_i x_i$
${\mathrm{\nabla }}_{b}L=-\sum _{i}^{N}{\alpha }_{i}{y}_{i}$$\nabla_b L=-\sum_i^N\alpha_iy_i$
${\mathrm{\nabla }}_{{\xi }^{i}}L=C-{\alpha }_{i}-{\mu }_{i}$$\nabla_{\xi^i} L=C-\alpha_i-\mu_i$

$\omega =\sum _{i}^{N}{\alpha }_{i}{y}_{i}{x}_{i}$$\omega=\sum_i^N\alpha_iy_i x_i$
$\sum _{i}^{N}{\alpha }_{i}{y}_{i}=0$$\sum_i^N\alpha_iy_i=0$
$C-{\alpha }_{i}-{\mu }_{i}=0$$C-\alpha_i-\mu_i=0$

c.将b中的结果代入拉格朗日函数，得到对偶问题
$mi{n}_{\omega ,b}L\left(\omega ,b,\alpha \right)=-\frac{1}{2}\sum _{i=1}^{N}\sum _{j=1}^{N}{\alpha }_{i}{\alpha }_{j}{y}_{i}{y}_{j}\left({x}_{i}\cdot {x}_{j}\right)+\sum _{i=1}^{N}{\alpha }_{i}$$min_{\omega,b}L(\omega,b,\alpha)=-\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i\alpha_jy_iy_j(x_i\cdot x_j)+\sum_{i=1}^N\alpha_i$

d. 接下来求
$ma{x}_{\alpha }-\frac{1}{2}\sum _{i=1}^{N}\sum _{j=1}^{N}{\alpha }_{i}{\alpha }_{j}{y}_{i}{y}_{j}\left({x}_{i}\cdot {x}_{j}\right)+\sum _{i=1}^{N}{\alpha }_{i}$$max_{\alpha}-\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i\alpha_jy_iy_j(x_i\cdot x_j)+\sum_{i=1}^N\alpha_i$,s.t $\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}=0$$\sum_{i=1}^{N}\alpha_i y_i=0$, ${\alpha }_{i}\ge 0$$\alpha_i\geq0$】, 【$i=1,2,\cdots ,N$$i=1,2,\cdots,N$

e. 转化为求
$mi{n}_{\alpha }\frac{1}{2}\sum _{i=1}^{N}\sum _{j=1}^{N}{\alpha }_{i}{\alpha }_{j}{y}_{i}{y}_{j}\left({x}_{i}\cdot {x}_{j}\right)-\sum _{i=1}^{N}{\alpha }_{i}$$min_{\alpha}\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i\alpha_jy_iy_j(x_i\cdot x_j)-\sum_{i=1}^N\alpha_i$,
s.t $\sum _{i=1}^{N}{\alpha }_{i}{y}_{i}=0$$\sum_{i=1}^{N}\alpha_i y_i=0$,
${\alpha }_{i}\ge 0，i=1,2,\cdots ,N$$\alpha_i\geq0，i=1,2,\cdots,N$
${\mu }_{i}\ge 0$$\mu_i\geq0$
${\alpha }_{i}\left({y}_{i}\left(\omega \cdot {x}_{i}+b\right)-1+{\xi }_{i}\right)=0$$\alpha_i(y_i(\omega\cdot x_i+b)-1+\xi_i)=0$

f.计算出 $\alpha$$\alpha$ 之后，可以得到$\omega ,b$$\omega,b$,分离超平面$\sum _{i=1}^{N}{\alpha }_{i}^{\ast }{y}_{i}\left(x\cdot {x}_{i}\right)+{b}^{\ast }$$\sum^N_{i=1}\alpha^*_iy_i(x\cdot x_i)+b^*$ , 分类决策$f\left(x\right)=sign\left(\sum _{i=1}^{N}{\alpha }_{i}^{\ast }{y}_{i}\left(x\cdot {x}_{i}\right)+{b}^{\ast }\right)$$f(x)=sign(\sum_{i=1}^N\alpha_i^*y_i(x\cdot x_i)+b^*)$

### 3. 非线性支持向量机

$K\left({x}_{i},{x}_{j}\right)=\varphi \left({x}_{i}\right)\cdot \varphi \left({x}_{j}\right)$$K(x_i,x_j)=\phi(x_i)\cdot \phi(x_j)$

SMO算法
1.选出两个优化变量${\alpha }_{i},{\alpha }_{j}$$\alpha_i,\alpha_j$之后，通过代数方法求解二次规划问题
2.选出一个变量${\alpha }_{i}$$\alpha_i$,如何选择第二个变量${\alpha }_{j}$$\alpha_j$