支持向量机（Support Vector Machine）

支持向量机（Support Vector Machine）

---

1 Introduction

• SVM = hinge loss + Kernel method
  

• Hinge loss：$L(f(x_n), \hat y_n) = max(0, 1-\hat y_nf(x))$

---

2 Linear SVM

• step 1：$f(x) = \sum\limits_{i}w_ix_i +b = \left[ \begin{matrix} w\\b \end{matrix} \right] \left[ \begin{matrix} x\\1 \end{matrix} \right] = w\cdot x$

• step 2：$Hinge Loss + \lambda ||w||_2$ 凸函数

• step 3：梯度下降

《Deep Learning using Linear Support Vector Machine》——ICML2013

• $\varepsilon_n$：松弛变量
• Quadratic Programming

---

3 Dual Representation

• step 1：$f(x) = w^Tx = \alpha ^TX^Tx = \sum\limits_{n}\alpha_n(x_nx) = \sum\limits_{n}\alpha_nK(x_n,x)$ 核技巧
• step 2、3：Find$\{\alpha_1^*,…,\alpha_N^*\}$ minimizing loss function

---

4 Kernel Trick

• $K(x,z) = \phi(x)\phi(z)$
• Radial Basis Function Kernel：$K(x,z) = exp(-\frac{1}{2}||x-z||_2) = \phi(x)\phi(z)$ 无穷多维的空间
• Sigmoid Kernel：$K(x,z) = tanh(x\cdot z)$