Support Vector Machine

这是我的特征提取的大作业，总结了支持向量机学习算法 。由于作业是英文的，我也就懒得翻译成中文的。

A support vector machine is a supervised learning model used for pattern recognition and regression analysis. An SVM model sets a clear gap to separate different categories according to data learning. Using a method called kernel trick, SVMs can efficiently perform a non-linear classification, implicitly mapping inputs into high-dimensional space.

3.1 Basic Theory
A SVM constructs a hyperplane in a high-dimensional space to classify the inputs. The vectors on hyperplane is called support vectors. That is reason why this method is called support vector machine.
Now a very simple problem is given in Figure 3.1. How to classify the black dots and white dots? I give an easy solution. Following variables are defined:

x =

$$\begin{cases} - 1,{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} for{\kern 1pt} {\kern 1pt} {\kern 1pt} the{\kern 1pt} {\kern 1pt} {\kern 1pt} black{\kern 1pt} {\kern 1pt} {\kern 1pt} dots \\ 1,{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} for{\kern 1pt} {\kern 1pt} {\kern 1pt} the{\kern 1pt} {\kern 1pt} {\kern 1pt} white{\kern 1pt} {\kern 1pt} {\kern 1pt} dots \\ \end{cases}$$

$f(x) = {w^T}x + b$

where w, b are the parametres to be learned. We can adopt a sgn(f(x)) to forecast the result if an input needs classification. If f(x) > 0, sgn(f(x))= 1; if f(x) < 0, sgn(f(x)) = -1.We obtain a variety of lines to classify the data, showed in Figure 3.2. But which line is best? We choose the maximum-margin line in Figure 3.3. The maximum-margin line has the following equation:
$M = \frac{2 } {\sqrt{w^T w}}$
Therefore, our aim is to maximize the M. That is