Support Vector Machine 支持向量机

Support Vector Machine 支持向量机

Vapnik（前苏联人）
适合样本数较小时的预测
由听浙江大学的研究生课程整理得来，链接

0. No Free Lunch Therome

(随手放到这里)
如果不对特征空间有先验假设，所有算法平均表现一致。
我们认为：特征差距小的样本更有可能是一类，所以机器学习不是白学di!

---

SVM简介

图片来自百度百科

可以证明，对于一个 线性可分(linear separable) 的样本空间有无数条线可以分开，那么怎么算最好的？使 间隔(margin) d最大，且$d_1=d_2=\frac{d}{2}$的线唯一且最佳。图中在虚线上的的样本点称为 支持向量(support vector)

定义：
①训练数据和标签 $(x_1,y_1),(x_2,y_2)...(x_n,y_n)$ 。$x_i$是向量，代表该样本各个特征(Feature)的取值，$y_i$是标签(label)，在SVM中一般取+1，-1（为什么不是1，0？后面会解释）
②线性模型 $(\mathbf{\omega },b)$代表了${\mathbf{\omega }}^T\mathbf{x}+b=0$的超平面(hyperplane)

要求什么？

当然是要求d(margin)的最大值了，但是显然不太好求，需要转化为：（一律先给结果再解释）
$$\{\begin{array}{l}\min \frac{1}{2}\Vert \mathbf{\omega }{\Vert }^2\\ s.t.(subjectto)y_i[{\mathbf{\omega }}^T{\mathbf{x}}_{\mathbf{i}}+b]\ge 1\end{array}$$
（这是一个凸优化理论中的二次规划问题）
考虑到以下的事实：
事实1：${\mathbf{\omega }}^T\mathbf{x}+b=0$与$a{\mathbf{\omega }}^T\mathbf{x}+ab=0(a\in R^{+})$是同一个超平面，即$(\mathbf{\omega },b)=(a\mathbf{\omega },ab)$
事实2：向量$x_0$和超平面$(\mathbf{\omega },b)$的距离（类比点到直线的距离）$d=\frac{|{\mathbf{\omega }}^T{\mathbf{x}}_0+b|}{\Vert \mathbf{\omega }\Vert }$
下面开始分析：
对超平面$({\mathbf{\omega }}_0,b_0)\stackrel{a}{⟶}(\mathbf{\omega },b)$进行变换，一定存在a使得$|{\mathbf{\omega }}^T\mathbf{x}+b|=1$，此时$d=\frac{1}{\Vert \mathbf{\omega }\Vert }$（所以要$\min \frac{1}{2}\Vert \mathbf{\omega }{\Vert }^2$, $\frac{1}{2}$是为了求导方便）
For non-support vector $\frac{|{\mathbf{\omega }}^T\mathbf{x}+b|}{\Vert \mathbf{\omega }\Vert }=d>\frac{1}{\Vert \mathbf{\omega }\Vert }$, so $|{\mathbf{\omega }}^T\mathbf{x}+b|>1$,. Now we need to drop the absolute value sign. We notice that $y_i$ has the same sign with $|{\mathbf{\omega }}^T\mathbf{x}+b|>1$, so $y_i[{\mathbf{\omega }}^T\mathbf{x}+b]>1$. I don’t why the teacher use [] instead of ()😦
For support vector $y_i[{\mathbf{\omega }}^T\mathbf{x}+b]=1$
Finally, we get the restrictive condition $y_i[{\mathbf{\omega }}^T\mathbf{x}+b]\ge 1$

PS When you set is non-linear separable the ineuqlity $y_i[{\mathbf{\omega }}^T\mathbf{x}+b]\ge 1$ has no solution.
PPS In fact, you can say $y_i[{\mathbf{\omega }}^T\mathbf{x}+b]$ is no less than any positive number, but 1 is much more simple.

How do we use SVM dealing with non-separable data?

$$\{\begin{array}{l}\min \frac{1}{2}\Vert \mathbf{\omega }{\Vert }^2+C\sum _{i=1}^N{\xi }_i\\ s.t.(subjectto)\{\begin{array}{l}s.t.(subjectto)y_i[{\mathbf{\omega }}^T{\mathbf{x}}_{\mathbf{i}}+b]\ge 1-{\xi }_i\\ {\xi }_i\ge 0\end{array}\end{array}$$
We introduce ξ to make the ineqlity hold. At the same time, we have to make sure $\xi$ is not too big, so we add regulation term to the first equation. (Regulation term is pretty common in the machine learning field). C is a parameter we have already set. In practice, we usually set up upper bound, lower bound and step and try everyone to get the best C.
One of the most importent differnece between SVM and other algorithms is how they deal with non-linear separable data. Other algorithms try to use circles or rectangles etc to find the boundry. SVM tries to find a linear boundary in a higher dimension space.
$\mathbf{X}\stackrel{\varphi }{⟶}\varphi (\mathbf{X})$
example:
$$\mathbf{X}=\left[\begin{array}{c}a\\ b\end{array}\right]\to \varphi (\mathbf{X})=\left[\begin{array}{c}{a}^2\\ b^2\\ a\\ b\\ ab\end{array}\right]$$
Warning:The dimension of $\mathbf{\omega }$ changes too.
We have already proved that the possibility of linear separable grow as the dimension grows. In a infinity dimension space, all the data sets are linear separable.

Kernel Function

We don’t have to know the expilicit expression of φ(x) if we have kernel function.
$K(x_1,x_2)=\varphi (x_1{)}^T\varphi (x_2)$
Here are some common kernel function:

1. Gaussian kernel function：$K(x_1,x_2)=exp(-\frac{\mid \mid x_1-x_2|{|}^2}{2{\sigma }^2})$
2. Polynomial kernel function：$K(x_1,x_2)=(\gamma \varphi (x_1{)}^T\varphi (x_2)+c{)}^n$
3. Linear kernel function(no kernel function)$K(x_1,x_2)=\varphi (x_1{)}^T\varphi (x_2)$
4. Sigmoid kernel function：$K(x_1,x_2)=tanh(\gamma \varphi (x_1{)}^T\varphi (x_2)+c)$
5. Laplace kernel function：$K(x_1,x_2)=exp(-\frac{||x_1-x_2||}{\sigma })$

Kernel function must follow the following rules:
①$K(x_1,x_2)=K(x_2,x_1)$
②半正定性 $\forall C_i(parameter),{\mathbf{X}}_{\mathbf{i}}(vector),\exists \sum _{i=1}^N\sum _{j=1}^N{C}_i{C}_{j}K({\mathbf{x}}_{\mathbf{i}},{\mathbf{x}}_{\mathbf{j}})$

Use prime problem and dual problem to avoid $\varphi (x)$

What is prime problem and dual problem? Check this
Prime problem:
$$\{\begin{array}{l}minimizef(\mathbf{\omega })\\ s.t.\{\begin{array}{l}{g}_i(\mathbf{\omega })\le (i=1\sim K)\\ h_i(\mathbf{\omega })=0(i=1\sim N)\end{array}\end{array}$$
Dual Problem:
{ Θ ( α , β ) = min ⁡ a l l   ω { L ( ω , α , β ) } s . t .   α i ≥ 0 \left\{\begin{array}{l} \Theta(\boldsymbol\alpha, \boldsymbol\beta) = \min\limits_{all \ ω}\{L(\boldsymbol\omega, \boldsymbol\alpha, \boldsymbol\beta)\}\\ s.t. \ \boldsymbol\alpha_i \geq0 \end{array}\right. {Θ(α,β)=all ωmin​{L(ω,α,β)}s.t. αi​≥0​
We need to find:
$$\{\begin{array}{l}\min \frac{1}{2}\Vert \mathbf{\omega }{\Vert }^2+C\sum _{i=1}^N{\xi }_i\\ s.t.(subjectto)\{\begin{array}{l}s.t.(subjectto)y_i[{\mathbf{\omega }}^T{\mathbf{x}}_{\mathbf{i}}+b]\ge 1-{\xi }_i\\ {\xi }_i\ge 0\end{array}\end{array}$$
Considerate it as prime problem, we can get dual problem:
{ Θ ( α ) = min ⁡ a l l   ω ,   ξ i ,   b { 1 2 ∥ ω ∥ 2 − C ∑ i = 1 N ξ i + ∑ i = 1 N β i ξ i + ∑ i = 1 N α i [ 1 + ξ i − y i ω T ϕ ( x i ) − y i b ] } s . t .   α i ≥ 0 ,   β i ≥ 0 \left\{\begin{array}{l} \Theta(\boldsymbol\alpha) = \min\limits_{all \ \omega, \ \xi_i, \ b} \{ \frac{1}{2}\|\boldsymbol{\omega}\|^2 - C \sum\limits_{i=1}^N\xi_i + \sum\limits_{i=1}^N\beta_i\xi_i + \sum\limits_{i=1}^N\alpha_i[1 + \xi_i - y_i\boldsymbol\omega^T \phi(x_i) - y_ib] \}\\ s.t. \ \boldsymbol\alpha_i \geq0,\ \boldsymbol\beta_i \geq 0\end{array}\right. ⎩⎨⎧​Θ(α)=all ω, ξi​, bmin​{21​∥ω∥2−Ci=1∑N​ξi​+i=1∑N​βi​ξi​+i=1∑N​αi​[1+ξi​−yi​ωTϕ(xi​)−yi​b]}s.t. αi​≥0, βi​≥0​
$\mathbf{\omega }\to \mathbf{\omega },{\mathbf{\xi }}_{\mathbf{i}},\mathbf{b};\mathbf{\alpha }\to {\mathbf{\alpha }}_{\mathbf{i}},{\mathbf{\beta }}_{\mathbf{i}},\mathbf{\beta }\to \mathbf{\varphi }$
($\alpha$ controlls the inequality, $\beta$ controlls the euqality)

Let’s start!
$$\begin{gathered} L=\frac{1}{2}\Vert \mathbf{\omega }{\Vert }^2-C\sum _{i=1}^N{\xi }_i+\sum _{i=1}^N{\beta }_i{\xi }_i+\sum _{i=1}^N{\alpha }_i[1+{\xi }_i-y_i{\mathbf{\omega }}^T\varphi (x_i)-y_{i}b] \\ \{\begin{array}{l}\frac{\partial L}{\partial \omega }=0\\ \frac{\partial L}{\partial {\xi }_i}=0\\ \frac{\partial L}{\partial b}=0\end{array} \end{gathered}$$
Solve these three equation and we can get:
$$\{\begin{array}{l}\omega =\sum _{i=1}^N{\alpha }_i{y}_i\varphi (x_i)\\ {\alpha }_i+{\beta }_i=C\\ \sum _{i=0}^N{\alpha }_i{y}_i=0\end{array}$$
By using these three euqations we can solve $\Theta (\alpha )$. Remember what we want? We want to use $K(x_i,x_2)$ to replace $\varphi (x_i)$.
$$\begin{gathered} L_{min}=\frac{1}{2}{\mathbf{\omega }}^T\mathbf{\omega }+\sum _{i=1}^N{\alpha }_i-\sum _{i=1}^N{\alpha }_i{y}_i{\mathbf{\omega }}^T\varphi (x_i) \\ {L}_{min}=\sum _{i=1}^N{\alpha }_i+\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j\varphi (x_i{)}^T\varphi (x_j)-\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j\varphi (x_j{)}^T\varphi (x_i) \\ attention!K(x_i,x_j)=\varphi (x_i{)}^T\varphi (x_i) \\ {L}_{min}=\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_{j}K(x_i,x_j) \end{gathered}$$
Finally, we get our optimize target:
$$\{\begin{array}{l}\max \Theta (\mathbf{\alpha })=\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_{j}K(x_i,x_j)\\ s.t.0\le {\mathbf{\alpha }}_i\le C,\sum _{i=1}^N{\alpha }_i{y}_i=0\end{array}$$
This is also a convex optimization problem, but we can solve it. (We know what K is!). There are different algorithms to slove convex optimization problem. We won’t talk about that here. SMO is one of the most common algorithms.

evaluation $\omega andb$

$\mathbf{\omega }$:
Remember how do we predict y?
if ${\mathbf{\omega }}^T\varphi ({\mathbf{x}}_{\mathbf{i}})+b\ge 0$, then $y=1$
if ${\mathbf{\omega }}^T\varphi ({\mathbf{x}}_{\mathbf{i}})+b<0$, then $y=-1$
Actually we don’t need to know the exact value of $\mathbf{\omega }$ if we know ${\mathbf{\omega }}^T\varphi ({\mathbf{x}}_{\mathbf{i}})$.
${\mathbf{\omega }}^T\varphi ({\mathbf{x}}_{\mathbf{i}})=\sum _{i=1}^N{\alpha }_i{y}_{i}K(x_i,x_j)$

$b$:
We need to use KKT condition: $\forall i=1\sim K,\alpha =0org(\omega )=0$
$$\begin{gathered} {\beta }_i=0,{\alpha }_i=0 \\ or \\ {\xi }_i=0,1+{\xi }_i-y_i{\mathbf{\omega }}^T\varphi (x_i)-y_{i}b=0 \end{gathered}$$
Choose a $0<{\alpha }_i<C$ randomly, then:
$$\begin{gathered} \{\begin{array}{l}{\xi }_i=0\\ 1+{\xi }_i-y_i{\mathbf{\omega }}^T\varphi (x_i)-y_{i}b=0\end{array} \\ b=\frac{1-y_i\sum _{i=1}^N{\alpha }_i{y}_{i}K(x_i,x_j)}{y_i} \end{gathered}$$
In parctice, we often choose many ${\alpha }_i$ and evaluate mean b as the final output parameter.

---

Summary

①train the model

• input { ( x i , y i ) } i = 1 ∼ N \{(x_i,y_i)\}_{i=1\sim N} {(xi​,yi​)}i=1∼N​
• solve the optimization problem (SMO, etc.)$$\{\begin{array}{l}\max \Theta (\mathbf{\alpha })=\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_{j}K(x_i,x_j)\\ s.t.0\le {\mathbf{\alpha }}_i\le C,\sum _{i=1}^N{\alpha }_i{y}_i=0\end{array}$$
• evaluate b

②test model

• input x

$$\{\begin{array}{l}if\sum _{i=1}^N{\alpha }_i{y}_{i}K(x_i,x_j)+b\ge 0,theny=1\\ if\sum _{i=1}^N{\alpha }_i{y}_{i}K(x_i,x_j)+b<0,theny=-1\end{array}$$

• ouput y