SVM_CS229PartV_学习笔记

1. margins and the idea of separating data with a large “gap”
2. optimal margin classifier
3. kernels, which give a way to apply SVMs in very high dimensional feature spaces
4. SMO algorithm, an implementation of SVMs

Notation

Use y ∈ {−1, 1} (instead of {0, 1}) to denote the class labels.
Use parameters $w,b$, and write classifier as $h_{w,b}(x)=g(w^{T}x+b)$, where $g(z)=1$ if $z>=0$, and $g(z)=-1$ otherwise.
Drop the convention of letting $x_0=1$ be an extra coordinate in the input feature vector. Thus, $b$ takes ${\theta }_0$'s role, and $w$ is $[{\theta }_1\cdots {\theta }_n{]}^T$.

Functional and geometric margins

Given a training example $(x^{(i)},y^{(i)})$, we define the functional margin of $(w,b)$ with respect to the training example as ${\gamma }^{(i)}=y^{(i)}(w^{T}x+b)$.
A large functional margin means a confident and correct prediction.
Given a training example $(x^{(i)},y^{(i)})$, we define the geometric margin of $(w,b)$ with respect to the training example as ${\gamma }^{(i)}=y^{(i)}((w/||w||{)}^T{x}^{(i)}+b/||w||)$.
The geometric margin is invariant to rescaling of the parameters; i.e., if we replace w with 2w and b with 2b, then the geometric margin does not change. Thus, when trying to fit w and b to training data, we can impose an arbitrary scaling constraint on w without changing anything important.
Finally, given a training set $S=(x^{(i)},y^{(i)});i=1,...,m$, we define the geometric margin of $(w,b)$ with respect to S to be the smallest of the geometric margins on the individual training examples: $\gamma =min{\gamma }^{(i)}$.

Optimal margin classifier

Goal: find a decision boundary that maximizes the geometric margin, since this would reflect a very confident set of predictions on the training set and a good “fit” to the training data.

Lagrange duality

This will lead us to optimization problem’s dual form, which solves constrained optimization problems.

Kernels

A function to present the similarity between two examples.
Gaussian Kernel: $K(x,z)=exp(-{||x-z||}^2/2{\sigma }^2)$
If $K$ is a valid kernel (also called a Mercer kernel)(i.e., if it corresponds to some feature mapping $\varphi$), then the corresponding Kernel matrix K is symmetric positive semidefinite.
Theorem (Mercer): Let K : $R^{n}x{R}^n$→ R be given. Then for K to be a valid (Mercer) kernel, it is necessary and sufficient that for any {x(1),…,x(m)}, (m < ∞), the corresponding kernel matrix is symmetric positive semi-definite.
If you have any learning algorithm that you can write in terms of only inner products ⟨x, z⟩ between input attribute vectors, then by replacing this with K(x, z) where K is a kernel, you can “magically” allow your algorithm to work efficiently in the high dimensional feature space corresponding to K.

Regularization and non-separable case

To make the algorithm work for non-linearly separable datasets as well as be less sensitive to outliers, regularization should be used to reduce the overfitting.

SMO algorithm

Sequential Minimal Optimization, an efficient way of solving the dual problem arising from the derivation of the SVM.