机器学习系列之coursera week 7 Support Vector Machine

目录

1. Large margin classification

1.1 Optimization objective

1.2 Large Margin Intuition

1.3 Mathematic Behind Large Margin classification

2. Kernels

2.1 Kernels I

2.2 Kernels II

3. SVMs in practice

---

1. Large margin classification

1.1 Optimization objective

Alternative view of LR:

fig. 1

(引自coursera week 7 Optimization objective)

If y = 1, we want h(x) approximately equal to 1, θ^T * x >> 0

If y = 1, we want h(x) approximately equal to 0, θ^T * x << 0

if y = 1, want θ^T * x >> 0

fig. 2

(引自coursera week 7 Optimization objective)

if y = 0, want θ^T * x << 0

fig. 3

(引自coursera week 7 Optimization objective)

Support Vector Machine:

LR:

SVM:

hypothesis:

1.2 Large Margin Intuition

SVM:

fig. 4

(引自coursera week 7 Large Margin Intuition)

SVM Decision Boundary:

whenever y^i = 1:

θ^T * x^i  >= 1

whenever y^i = 0:

θ^T * x^i  <= -1

linearly separable case:

fig. 5

(引自coursera week 7 Large Margin Intuition)

SVM会选择黑色的decision boundary， 因为其分类性能会更好，也有很大的margin。 SVM有时被称为 Large Margin classifier

Large Margin Classifier in presence of outliers(异常值):

fig. 6

(引自coursera week 7 Large Margin Intuition)

1.3 Mathematic Behind Large Margin classification

Vector Inner Product:

inner product = u^T * v

SVM Decision Boundary:

simplication: θ_0 = 0, n = 2

 fig. 7

(引自coursera week 7 Mathematic Behind Large Margin classification)

where p^i is the project of x^i onto the vector θ. simplification: θ_0 = 0, 决策边界过原点

E.g.

fig 8

(引自coursera week 7 Mathematic Behind Large Margin classification)

fig. 9

(引自coursera week 7 Mathematic Behind Large Margin classification)

SVM会选择fig 9 的边界.

 

2. Kernels

2.1 Kernels I

Non-linear Decision Boundary:

fig. 10

(引自coursera week 7 Kernels I)

Is there a different/better choice of the features???

Kenels:

Given x, compute new feature depending on proximity to landmarks l_1, l_2, l_3

fig. 11

(引自coursera week 7 Kernels I)

Given x:

exp(...) is called Gaussian Kernel

note: if x approximately equal to l_1, then f1 approximately equal 1

if x far from l_1, then f1 approximately equal 0

fig. 12

(引自coursera week 7 Kernels I)

2.2 Kernels II

Choose the landmarks:

choose l_(i) = x_(i)

Given example x:

fig. 13

(引自coursera week 7 Kernels II)

given (x1, y1), (x2, y2), ..., (xm, ym)

for training example (xi, yi), 

Hypothesis: given x, compute features f(belongs to R^m+1), predict "y = 1", if θ^T * f >= 0 (θ belongs to R^m+1)

Training:

note: 正则化项的的项数从n变成了m

SVM parameters:

C:

large C: lower bias, high variance

small C: higher bias, low variance

σ^2:

large σ^2: feature f very more smoothly, then higher bias, lower variace

small σ^2: feature f very less smoothly, then lower bias, higher variace

 

3. SVMs in practice

Using a SVM:

Need to specify:

-choice of parameter C

-choice of Kermel

E.g. No kernel("linear kernel") ---- n large, m small

Gaussian kernel: choose σ^2 ---- n small, m large

note: Do perform feature scaling before using the Gaussian kernel

other choice of kernel:

note: Not all similarity functions similarity (x, l) make valid kernels. (need to satisfy technical condition called "Mercer's  Theorem" to make sure SVM package's optimizations run correctly and do not diverge).

many off-the-self kernels available:

-Polynomial kernel: k(x, l) = (x^t * x + r)^d

- More esoteric: string kernel, chi-square kernel, histogram intersection kernel, ....

Multi-class classification:

fig. 14

(引自coursera week 7 SVMs in practice)

LR Vs. SVMs:

n = # features, m = # training examples

if n is large(relative to m), then use LR or SVm without a kernel

if n is small, m is intermediate, then use SVM with Gaussian kernel

if n is small, m is large, then create more features, then us LR or SVM without kernel