支持向量机
分离超平面:
w
T
x
+
b
=
0
w^Tx+b=0
wTx+b=0
点到直线距离:
r
=
∣
w
T
x
+
b
∣
∣
∣
w
∣
∣
2
r=\frac{|w^Tx+b|}{||w||_2}
r=∣∣w∣∣2∣wTx+b∣
∣
∣
w
∣
∣
2
||w||_2
∣∣w∣∣2为2-范数:
∣
∣
w
∣
∣
2
=
∑
i
=
1
m
w
i
2
2
||w||_2=\sqrt[2]{\sum^m_{i=1}w_i^2}
∣∣w∣∣2=2∑i=1mwi2
直线为超平面,样本可表示为:
w
T
x
+
b
≥
+
1
w^Tx+b\ \geq+1
wTx+b ≥+1
w
T
x
+
b
≤
+
1
w^Tx+b\ \leq+1
wTx+b ≤+1
margin:
函数间隔:
l
a
b
e
l
(
w
T
x
+
b
)
o
r
y
i
(
w
T
x
+
b
)
label(w^Tx+b)\ or\ y_i(w^Tx+b)
label(wTx+b) or yi(wTx+b)
几何间隔:
r
=
l
a
b
e
l
(
w
T
x
+
b
)
∣
∣
w
∣
∣
2
r=\frac{label(w^Tx+b)}{||w||_2}
r=∣∣w∣∣2label(wTx+b),当数据被正确分类时,几何间隔就是点到超平面的距离
为了求几何间隔最大,SVM基本问题可以转化为求解:(
r
∗
∣
∣
w
∣
∣
\frac{r^*}{||w||}
∣∣w∣∣r∗为几何间隔,(
r
∗
{r^*}
r∗为函数间隔)
max
r
∗
∣
∣
w
∣
∣
\max\ \frac{r^*}{||w||}
max ∣∣w∣∣r∗
(
s
u
b
j
e
c
t
t
o
)
y
i
(
w
T
x
i
+
b
)
≥
r
∗
,
i
=
1
,
2
,
.
.
,
m
(subject\ to)\ y_i({w^T}x_i+{b})\geq {r^*},\ i=1,2,..,m
(subject to) yi(wTxi+b)≥r∗, i=1,2,..,m
分类点几何间隔最大,同时被正确分类。但这个方程并非凸函数求解,所以要先①将方程转化为凸函数,②用拉格朗日乘子法和KKT条件求解对偶问题。
①转化为凸函数:
先令 r ∗ = 1 {r^*}=1 r∗=1,方便计算(参照衡量,不影响评价结果)
max
1
∣
∣
w
∣
∣
\max\ \frac{1}{||w||}
max ∣∣w∣∣1
s
.
t
.
y
i
(
w
T
x
i
+
b
)
≥
1
,
i
=
1
,
2
,
.
.
,
m
s.t.\ y_i({w^T}x_i+{b})\geq {1},\ i=1,2,..,m
s.t. yi(wTxi+b)≥1, i=1,2,..,m
再将
max
1
∣
∣
w
∣
∣
\max\ \frac{1}{||w||}
max ∣∣w∣∣1转化成
min
1
2
∣
∣
w
∣
∣
2
\min\ \frac{1}{2}||w||^2
min 21∣∣w∣∣2求解凸函数,1/2是为了求导之后方便计算。
min
1
2
∣
∣
w
∣
∣
2
\min\ \frac{1}{2}||w||^2
min 21∣∣w∣∣2
s
.
t
.
y
i
(
w
T
x
i
+
b
)
≥
1
,
i
=
1
,
2
,
.
.
,
m
s.t.\ y_i(w^Tx_i+b)\geq 1,\ i=1,2,..,m
s.t. yi(wTxi+b)≥1, i=1,2,..,m
②用拉格朗日乘子法和KKT条件求解最优值:
min
1
2
∣
∣
w
∣
∣
2
\min\ \frac{1}{2}||w||^2
min 21∣∣w∣∣2
s
.
t
.
−
y
i
(
w
T
x
i
+
b
)
+
1
≤
0
,
i
=
1
,
2
,
.
.
,
m
s.t.\ -y_i(w^Tx_i+b)+1\leq 0,\ i=1,2,..,m
s.t. −yi(wTxi+b)+1≤0, i=1,2,..,m
整合成:
L
(
w
,
b
,
α
)
=
1
2
∣
∣
w
∣
∣
2
+
∑
i
=
1
m
α
i
(
−
y
i
(
w
T
x
i
+
b
)
+
1
)
L(w, b, \alpha) = \frac{1}{2}||w||^2+\sum^m_{i=1}\alpha_i(-y_i(w^Tx_i+b)+1)
L(w,b,α)=21∣∣w∣∣2+i=1∑mαi(−yi(wTxi+b)+1)
推导:
min
f
(
x
)
=
min
max
L
(
w
,
b
,
α
)
≥
max
min
L
(
w
,
b
,
α
)
\min\ f(x)=\min \max\ L(w, b, \alpha)\geq \max \min\ L(w, b, \alpha)
min f(x)=minmax L(w,b,α)≥maxmin L(w,b,α)
根据KKT条件:
∂
∂
w
L
(
w
,
b
,
α
)
=
w
−
∑
α
i
y
i
x
i
=
0
,
w
=
∑
α
i
y
i
x
i
\frac{\partial }{\partial w}L(w, b, \alpha)=w-\sum\alpha_iy_ix_i=0,\ w=\sum\alpha_iy_ix_i
∂w∂L(w,b,α)=w−∑αiyixi=0, w=∑αiyixi
∂
∂
b
L
(
w
,
b
,
α
)
=
∑
α
i
y
i
=
0
\frac{\partial }{\partial b}L(w, b, \alpha)=\sum\alpha_iy_i=0
∂b∂L(w,b,α)=∑αiyi=0
带入
L
(
w
,
b
,
α
)
L(w, b, \alpha)
L(w,b,α)
min
L
(
w
,
b
,
α
)
=
1
2
∣
∣
w
∣
∣
2
+
∑
i
=
1
m
α
i
(
−
y
i
(
w
T
x
i
+
b
)
+
1
)
\min\ L(w, b, \alpha)=\frac{1}{2}||w||^2+\sum^m_{i=1}\alpha_i(-y_i(w^Tx_i+b)+1)
min L(w,b,α)=21∣∣w∣∣2+∑i=1mαi(−yi(wTxi+b)+1)
=
1
2
w
T
w
−
∑
i
=
1
m
α
i
y
i
w
T
x
i
−
b
∑
i
=
1
m
α
i
y
i
+
∑
i
=
1
m
α
i
\qquad\qquad\qquad=\frac{1}{2}w^Tw-\sum^m_{i=1}\alpha_iy_iw^Tx_i-b\sum^m_{i=1}\alpha_iy_i+\sum^m_{i=1}\alpha_i
=21wTw−∑i=1mαiyiwTxi−b∑i=1mαiyi+∑i=1mαi
=
1
2
w
T
∑
α
i
y
i
x
i
−
∑
i
=
1
m
α
i
y
i
w
T
x
i
+
∑
i
=
1
m
α
i
\qquad\qquad\qquad=\frac{1}{2}w^T\sum\alpha_iy_ix_i-\sum^m_{i=1}\alpha_iy_iw^Tx_i+\sum^m_{i=1}\alpha_i
=21wT∑αiyixi−∑i=1mαiyiwTxi+∑i=1mαi
= ∑ i = 1 m α i − 1 2 ∑ i = 1 m α i y i w T x i \qquad\qquad\qquad=\sum^m_{i=1}\alpha_i-\frac{1}{2}\sum^m_{i=1}\alpha_iy_iw^Tx_i =∑i=1mαi−21∑i=1mαiyiwTxi
= ∑ i = 1 m α i − 1 2 ∑ i , j = 1 m α i α j y i y j ( x i x j ) \qquad\qquad\qquad=\sum^m_{i=1}\alpha_i-\frac{1}{2}\sum^m_{i,j=1}\alpha_i\alpha_jy_iy_j(x_ix_j) =∑i=1mαi−21∑i,j=1mαiαjyiyj(xixj)
再把max问题转成min问题:
max ∑ i = 1 m α i − 1 2 ∑ i , j = 1 m α i α j y i y j ( x i x j ) = min 1 2 ∑ i , j = 1 m α i α j y i y j ( x i x j ) − ∑ i = 1 m α i \max\ \sum^m_{i=1}\alpha_i-\frac{1}{2}\sum^m_{i,j=1}\alpha_i\alpha_jy_iy_j(x_ix_j)=\min \frac{1}{2}\sum^m_{i,j=1}\alpha_i\alpha_jy_iy_j(x_ix_j)-\sum^m_{i=1}\alpha_i max ∑i=1mαi−21∑i,j=1mαiαjyiyj(xixj)=min21∑i,j=1mαiαjyiyj(xixj)−∑i=1mαi
s . t . ∑ i = 1 m α i y i = 0 , s.t.\ \sum^m_{i=1}\alpha_iy_i=0, s.t. ∑i=1mαiyi=0,
α i ≥ 0 , i = 1 , 2 , . . . , m \alpha_i \geq 0,i=1,2,...,m αi≥0,i=1,2,...,m
以上为SVM对偶问题的对偶形式
kernel
在低维空间计算获得高维空间的计算结果,也就是说计算结果满足高维(满足高维,才能说明高维下线性可分)。
soft margin & slack variable
引入松弛变量
ξ
≥
0
\xi\geq0
ξ≥0,对应数据点允许偏离的functional margin 的量。
目标函数:
min
1
2
∣
∣
w
∣
∣
2
+
C
∑
ξ
i
\min\ \frac{1}{2}||w||^2+C\sum\xi_i\qquad
min 21∣∣w∣∣2+C∑ξi
s . t . y i ( w T x i + b ) ≥ 1 − ξ i s.t.\ y_i(w^Tx_i+b)\geq1-\xi_i s.t. yi(wTxi+b)≥1−ξi
对偶问题:
max ∑ i = 1 m α i − 1 2 ∑ i , j = 1 m α i α j y i y j ( x i x j ) = min 1 2 ∑ i , j = 1 m α i α j y i y j ( x i x j ) − ∑ i = 1 m α i \max\ \sum^m_{i=1}\alpha_i-\frac{1}{2}\sum^m_{i,j=1}\alpha_i\alpha_jy_iy_j(x_ix_j)=\min \frac{1}{2}\sum^m_{i,j=1}\alpha_i\alpha_jy_iy_j(x_ix_j)-\sum^m_{i=1}\alpha_i max i=1∑mαi−21i,j=1∑mαiαjyiyj(xixj)=min21i,j=1∑mαiαjyiyj(xixj)−i=1∑mαi s . t . C ≥ α i ≥ 0 , i = 1 , 2 , . . . , m ∑ i = 1 m α i y i = 0 , s.t.\ C\geq\alpha_i \geq 0,i=1,2,...,m\quad \sum^m_{i=1}\alpha_iy_i=0, s.t. C≥αi≥0,i=1,2,...,mi=1∑mαiyi=0,
Sequential Minimal Optimization
首先定义特征到结果的输出函数:
u
=
w
T
x
+
b
u=w^Tx+b
u=wTx+b.
因为
w
=
∑
α
i
y
i
x
i
w=\sum\alpha_iy_ix_i
w=∑αiyixi
有
u
=
∑
y
i
α
i
K
(
x
i
,
x
)
−
b
u=\sum y_i\alpha_iK(x_i, x)-b
u=∑yiαiK(xi,x)−b
max ∑ i = 1 m α i − 1 2 ∑ i = 1 m ∑ j = 1 m α i α j y i y j < ϕ ( x i ) T \max\ \sum^m_{i=1}\alpha_i-\frac{1}{2}\sum^m_{i=1}\sum^m_{j=1}\alpha_i\alpha_jy_iy_j<\phi(x_i)^T max ∑i=1mαi−21∑i=1m∑j=1mαiαjyiyj<ϕ(xi)T
s . t . ∑ i = 1 m α i y i = 0 , s.t.\ \sum^m_{i=1}\alpha_iy_i=0, s.t. ∑i=1mαiyi=0,
α i ≥ 0 , i = 1 , 2 , . . . , m \alpha_i \geq 0,i=1,2,...,m αi≥0,i=1,2,...,m
SMO算法是一种启发式算,其基本思路是:如果所有变量的解都满足此最优化问题的KKT条件,那么这个最优化的问题的解就得到了。
否则,选择两个变量,固定其他变量,针对这两个变量构建一个二次规划问题。
整个SMO算法包括两个部分:求解两个变量二次规划的解析方法和选择变量的启发式方法。
求解两个变量的二次规划问题:首先分析约束条件,然后在此约束条件下求极小。
class SVM:
def __init__(self, max_iter=100, kernel='linear'):
self.max_iter = max_iter
self._kernel = kernel
def init_args(self, features, labels):
self.m, self.n = features.shape
self.X = features
self.Y = labels
self.b = 0.0
# 将Ei保存在一个列表里
self.alpha = np.ones(self.m)
self.E = [self._E(i) for i in range(self.m)]
# 松弛变量
self.C = 1.0
def _KKT(self, i):
y_g = self._g(i)*self.Y[i]
if self.alpha[i] == 0:
return y_g >= 1
elif 0 < self.alpha[i] < self.C:
return y_g == 1
else:
return y_g <= 1
# g(x)预测值,输入xi(X[i])
def _g(self, i):
r = self.b
for j in range(self.m):
r += self.alpha[j]*self.Y[j]*self.kernel(self.X[i], self.X[j])
return r
# 核函数
def kernel(self, x1, x2):
if self._kernel == 'linear':
return sum([x1[k]*x2[k] for k in range(self.n)])
elif self._kernel == 'poly':
return (sum([x1[k]*x2[k] for k in range(self.n)]) + 1)**2
return 0
# E(x)为g(x)对输入x的预测值和y的差
def _E(self, i):
return self._g(i) - self.Y[i]
def _init_alpha(self):
# 外层循环首先遍历所有满足0<a<C的样本点,检验是否满足KKT
index_list = [i for i in range(self.m) if 0 < self.alpha[i] < self.C]
# 否则遍历整个训练集
non_satisfy_list = [i for i in range(self.m) if i not in index_list]
index_list.extend(non_satisfy_list)
for i in index_list:
if self._KKT(i):
continue
E1 = self.E[i]
# 如果E1是+,选择最小的;如果E1是负的,选择最大的
if E1 >= 0:
j = min(range(self.m), key=lambda x: self.E[x])
else:
j = max(range(self.m), key=lambda x: self.E[x])
return i, j
def _compare(self, _alpha, L, H):
if _alpha > H:
return H
elif _alpha < L:
return L
else:
return _alpha
#SMO算法
def fit(self, features, labels):
#初始化参数
self.init_args(features, labels)
#循环训练
for t in range(self.max_iter):
# train
#选择变量 检验训练样本点是否满足KKT条件
i1, i2 = self._init_alpha()
# 边界
#求解两个变量的二次规划问题
if self.Y[i1] == self.Y[i2]:
L = max(0, self.alpha[i1]+self.alpha[i2]-self.C)
H = min(self.C, self.alpha[i1]+self.alpha[i2])
else:
L = max(0, self.alpha[i2]-self.alpha[i1])
H = min(self.C, self.C+self.alpha[i2]-self.alpha[i1])
E1 = self.E[i1]
E2 = self.E[i2]
# eta=K11+K22-2K12
eta = self.kernel(self.X[i1], self.X[i1]) + self.kernel(self.X[i2], self.X[i2]) - 2*self.kernel(self.X[i1], self.X[i2])
if eta <= 0:
# print('eta <= 0')
continue
alpha2_new_unc = self.alpha[i2] + self.Y[i2] * (E1 - E2) / eta#此处有修改,根据书上应该是E1 - E2,书上130-131页
alpha2_new = self._compare(alpha2_new_unc, L, H)
alpha1_new = self.alpha[i1] + self.Y[i1] * self.Y[i2] * (self.alpha[i2] - alpha2_new)
b1_new = -E1 - self.Y[i1] * self.kernel(self.X[i1], self.X[i1]) * (alpha1_new-self.alpha[i1]) - self.Y[i2] * self.kernel(self.X[i2], self.X[i1]) * (alpha2_new-self.alpha[i2])+ self.b
b2_new = -E2 - self.Y[i1] * self.kernel(self.X[i1], self.X[i2]) * (alpha1_new-self.alpha[i1]) - self.Y[i2] * self.kernel(self.X[i2], self.X[i2]) * (alpha2_new-self.alpha[i2])+ self.b
if 0 < alpha1_new < self.C:
b_new = b1_new
elif 0 < alpha2_new < self.C:
b_new = b2_new
else:
# 选择中点
b_new = (b1_new + b2_new) / 2
# 更新参数
self.alpha[i1] = alpha1_new
self.alpha[i2] = alpha2_new
self.b = b_new
self.E[i1] = self._E(i1)
self.E[i2] = self._E(i2)
return 'train done!'
def predict(self, data):
r = self.b
for i in range(self.m):
r += self.alpha[i] * self.Y[i] * self.kernel(data, self.X[i])
return 1 if r > 0 else -1
def score(self, X_test, y_test):
right_count = 0
for i in range(len(X_test)):
result = self.predict(X_test[i])
if result == y_test[i]:
right_count += 1
return right_count / len(X_test)
def _weight(self):
# linear model
yx = self.Y.reshape(-1, 1)*self.X
self.w = np.dot(yx.T, self.alpha)
return self.w