SVM (支持向量机推导)

SVM (支持向量机推导)

Background

1. Hyperplane Definition

A hyperplane can be defined by two vectors $\mathbf{w}=\left(\begin{array}{c}-b\\ -a\\ 1\end{array}\right),\mathbf{x}=\left(\begin{array}{c}1\\ x\\ y\end{array}\right)$, because

• A hyperplane can be defined as $y=ax+b⟹y-ax-b=0$
• And we have ${\mathbf{w}}^T\mathbf{x}=-b\ast 1+(-a)\ast x+1\ast y=y-ax-b$

2. Vector subtraction

3. Dot product to calculate projection length

• Vector dot product: $\mathbf{a}\cdot \mathbf{b}=|\mathbf{a}|\times |\mathbf{b}|\times \cos (\theta )$
• $\text{Length }(|\mathbf{a}|\text{’s projection on }|\mathbf{b}|)=|\mathbf{a}|\times \cos (\theta )=\frac{\mathbf{a}\cdot \mathbf{b}}{|\mathbf{b}|}$
  

4. The distance from a point to a line

• $d=\frac{|A{x}_0+B{y}_0+C|}{\sqrt{A^2+B^2}}$

---

What is Support Vector Machines

• Intuition: The objective of the support vector machine algorithm is to find a hyperplane in an N-dimensional space (N - the number of features) that distinctly classifies the data points (the hyperplane with maximum margin).
• Decision boundaries: hyperplanes that help classify the data points.
• Support vectors
  • Data points that are closer to the hyperplane and influence the position and orientation of the hyperplane.
  • Using these support vectors, we maximize the margin of the classifier.
  • Delete the support vectors will change the position of the hyperplane.

---

Objective function

$$\underset{\mathbf{w},b}{\min }\frac{||\mathbf{w}|{|}^2}{2},\text{s.t. }{y}_i(w^T{x}_i+b)\ge 1,i=1,2,\dots ,m$$

• With above function, we can find the hyperplane $(\mathbf{w},b)$ with the largest margin, where $\mathbf{w}$ is the weight vector and b is the bias of the hyperplane.
• Assumpe the hyperplane $(\mathbf{w},b)$ can classify all the data correctly, i.e. for any $(x_i,y_i)\in D$, ${\mathbf{w}}^T{\mathbf{x}}_{\mathbf{i}}+b>0$ if $y_i=+1$; ${\mathbf{w}}^T{\mathbf{x}}_{\mathbf{i}}+b<0$ if $y_i=-1$. Let,
  $$\{\begin{array}{ll}{\mathbf{w}}^T{\mathbf{x}}_{\mathbf{i}}+b\ge 1,& y_i=+1\\ {\mathbf{w}}^T{\mathbf{x}}_{\mathbf{i}}+b\le -1,& y_i=-1\end{array}$$
• For the two support vectors (the closest data points on the margin line), we have ${\mathbf{w}}^T{\mathbf{x}}^{+}+b=+1,{\mathbf{w}}^T{\mathbf{x}}^{-}+b=-1$ (e.q. 1)

1. Proof with dot product

• $\hat{\mathbf{w}}=\frac{\mathbf{w}}{||\mathbf{w}||}$ is the unit vector that are orthogonal to hyperplane.

  • why $\hat{\mathbf{w}}$ is orthogonal to ${\mathbf{w}}^T\mathbf{x}+b=0$? Can prove for a line $y=kx+b$ in 2D axis.

• Then use dot product to calculate projected length of $\mathbf{c}$ alongside $\hat{\mathbf{w}}$, $\text{margin}=||\mathbf{c}||\times \cos \theta =\frac{\mathbf{c}\cdot \mathbf{w}}{||\mathbf{w}||}=\mathbf{c}\cdot \hat{\mathbf{w}}=({\mathbf{x}}^{+}-{\mathbf{x}}^{-})\cdot \hat{\mathbf{w}}=({\mathbf{x}}^{+}-{\mathbf{x}}^{-})\cdot \frac{\mathbf{w}}{||\mathbf{w}||}={\mathbf{x}}^{+}\cdot \frac{\mathbf{w}}{||\mathbf{w}||}-{\mathbf{x}}^{-}\cdot \frac{\mathbf{w}}{||\mathbf{w}||}$

  • $\mathbf{c}$ is the vector starting at $x^{-}$ and ending at $x^{+}$

• With (e.q. 1) , we have $\text{margin}={\mathbf{x}}^{+}\cdot \frac{\mathbf{w}}{||\mathbf{w}||}-{\mathbf{x}}^{-}\cdot \frac{\mathbf{w}}{||\mathbf{w}||}=\frac{1-b}{||\mathbf{w}||}-\frac{-1-b}{||\mathbf{w}||}=\frac{2}{||\mathbf{w}||}$

• All in one
  

2. Proof with distance formula

• Using formula to calculate distance from a point to a line
  $\text{margin}=r^{+}+r^{-}=\frac{|w^T{x}^{+}+b|}{||w||}+\frac{|w^T{x}^{-}+b|}{||w||}$
• With (e.q. 1) , we have $\text{margin}=r^{+}+r^{-}=\frac{|w^T{x}^{+}+b|}{||w||}+\frac{|w^T{x}^{-}+b|}{||w||}=\frac{|1|}{||w||}+\frac{|-1|}{||w||}=\frac{|2|}{||w||}$

Our objective function is then: $\max \frac{2}{||\mathbf{w}||}⟹\max \frac{1}{||\mathbf{w}||}⟹\min ||\mathbf{w}||⟹\min \frac{||\mathbf{w}|{|}^2}{2}$

---

Soft Margin SVM

• Margin: The distance of the vectors from the hyperplane.
  • Hard margin: all data are separated correctly
  • Soft margin: allow some margin violation to occur
    
• It’s not always plausible to classify all the data points correctly with a hyperplane, so we need to tolerate some error classification which not satisfy the restriction: $y_i({\mathbf{w}}^T{\mathbf{x}}_{\mathbf{i}}+b)\ge 1$ (e.q. 2) .We call this soft margin SVM.

Objective function of soft margin SVM

$$\underset{\mathbf{w},b}{\min }\frac{||\mathbf{w}|{|}^2}{2}+C\sum _{i=1}^m{\ell }_{0/1}(y_i({\mathbf{w}}^T{\mathbf{x}}_{\mathbf{i}}+b)),{\ell }_{0/1}=\{\begin{array}{ll}1,& \text{if }z<0;\\ 0,& \text{otherwise.}\end{array}$$

• When $C=+\infty$, objective function force all the data subject to our restriction e.q.2. Otherwise, the function tolerates some error classification.
• Because ${\ell }_{0/1}$ is not a convex and continuous function, we usually use other function to take place of ${\ell }_{0/1}$, we call them “surrogate loss”.
  • hinge loss: ${\ell }_{hinge}(z)=\max (0,1-z)$
  • exponential loss: ${\ell }_{exp}(z)=\exp (-z)$
  • logistic loss: ${\ell }_{hinge}(z)=\log (1+\exp (-z))$

---

Kernel Tricks -TBD

If ata points are not separable in low dimensional space, use kernel function to map them to higher dimensional space

---

Solve objective function -TBD

• Lagrange Multiplier

Reference：A Top Machine Learning Algorithm Explained: Support Vector Machines (SVMs)