# 2 SVM原理

## 2.1 线性回归

J(w,b)=i(wTxi+byi)2

minw,bJ(w,b)

yi(wTxi+b)>0

# 3 公式推导

## 3.1 点到直线的距离

wTx+b=0

wTx′′+b=0

wT(xx′′)=0

dist(x,h)=|wT||w||(xx)|=|wTx+b|||w||

yi(wTxi+b)>0

dist(xi,h)=yi(wTxi+b)||w||

maxb,w=margin(b,w)使, yi(wTxi+b)>0margin(b,w)=mini=1,...Ndist(xi,h)

1=yi(wTx+b)$1 = y_i(w^Tx+b)$ 。 注意，因为我们要找到最小的 dist(xi,h)$dist(x_i,h)$，对w和b 的放缩不会影响谁是最近的点这个结果。也不会影响超平面的位置因为
wTx+b=0$w^Tx+b=0$(wTx/t+b/t)=0$(w^Tx/t+b/t) = 0$并无分别。

yi(wTxi+b)>=1

margin(b,w)=1||w||

maxb,w1||w||使yi(wTxi+b)>=1

minb,w||w||使yi(wTxi+b)>=1

minb,w12wTw使yi(wTxi+b)>=1

# 转化为二次规划的标准型

uQP(H,f,A,b)minu12uTHu+fTu,aTmu<=bm

u=[bw]

H=[00K0KIK]
, K为w的维度，H是(K+1)*(K+1)的矩阵，除了（1，1）点，其他对角线均为1

H = [
0 0 0 ;
0 1 0 ;
0 0 1
]

f=0KRK×1

yi(wTxi+b)>=1$y_i(w^Tx_i+b) >= 1$修改为 yi(wTxi+b)<=1$-y_i(w^Tx_i+b) <= -1$

Ai=yi[1,xi]$A_i = - y_i [ 1 , x_i]$ , 使得bi=1$b_i = -1$

problem:

         min 0.5*x'*H*x + f'*x   subject to:  A*x <= b
x

X = quadprog(H,f,A,b,Aeq,beq) solves the problem above while
additionally satisfying the equality constraints Aeq*x = beq.

X = quadprog(H,f,A,b,Aeq,beq,LB,UB) defines a set of lower and upper
bounds on the design variables, X, so that the solution is in the
range LB <= X <= UB. Use empty matrices for LB and UB if no bounds
exist. Set LB(i) = -Inf if X(i) is unbounded below; set UB(i) = Inf if
X(i) is unbounded above.

X = quadprog(H,f,A,b,Aeq,beq,LB,UB,X0) sets the starting point to X0.

X = quadprog(H,f,A,b,Aeq,beq,LB,UB,X0,OPTIONS) minimizes with the
default optimization parameters replaced by values in the structure
OPTIONS, an argument created with the OPTIMSET function. See OPTIMSET
for details.

X = quadprog(PROBLEM) finds the minimum for PROBLEM. PROBLEM is a
structure with matrix 'H' in PROBLEM.H, the vector 'f' in PROBLEM.f,
the linear inequality constraints in PROBLEM.Aineq and PROBLEM.bineq,
the linear equality constraints in PROBLEM.Aeq and PROBLEM.beq, the
lower bounds in PROBLEM.lb, the upper bounds in PROBLEM.ub, the start
point in PROBLEM.x0, the options structure in PROBLEM.options, and
solver name 'quadprog' in PROBLEM.solver. Use this syntax to solve at
the command line a problem exported from OPTIMTOOL. The structure
PROBLEM must have all the fields.

[X,FVAL] = quadprog(H,f,A,b) returns the value of the objective
function at X: FVAL = 0.5*X'*H*X + f'*X.

[X,FVAL,EXITFLAG] = quadprog(H,f,A,b) returns an EXITFLAG that
describes the exit condition of quadprog. Possible values of EXITFLAG
and the corresponding exit conditions are

All algorithms:
1  First order optimality conditions satisfied.
0  Maximum number of iterations exceeded.
-2  No feasible point found.
-3  Problem is unbounded.
Interior-point-convex only:
-6  Non-convex problem detected.
Trust-region-reflective only:
3  Change in objective function too small.
-4  Current search direction is not a descent direction; no further
Active-set only:
4  Local minimizer found.
-7  Magnitude of search direction became too small; no further
conditioned.

[X,FVAL,EXITFLAG,OUTPUT] = quadprog(H,f,A,b) returns a structure
OUTPUT with the number of iterations taken in OUTPUT.iterations,
maximum of constraint violations in OUTPUT.constrviolation, the
type of algorithm used in OUTPUT.algorithm, the number of conjugate
gradient iterations (if used) in OUTPUT.cgiterations, a measure of
first order optimality (large-scale algorithm only) in
OUTPUT.firstorderopt, and the exit message in OUTPUT.message.

[X,FVAL,EXITFLAG,OUTPUT,LAMBDA] = quadprog(H,f,A,b) returns the set of
Lagrangian multipliers LAMBDA, at the solution: LAMBDA.ineqlin for the
linear inequalities A, LAMBDA.eqlin for the linear equalities Aeq,
LAMBDA.lower for LB, and LAMBDA.upper for UB.

Reference page in Help browser


## matlab求解SVM

function test_svm()
t1  = 5+4*randn(2,10);
t2 = 20+4*randn(2,10);
X = [t1 t2]
X = [t1 t2];
Y = [ones(10,1) ; -ones(10,1)]

plot(t1(1,:),t1(2,:),'ro');
hold on;
plot(t2(1,:),t2(2,:),'bx');
u = svm(X,Y)
x = [-u(1)/u(2) , 0];
y = [0 , -u(1)/u(3)];
plot(x,y);

end

function u = svm( X,Y )
%SVM Summary of this function goes here
%   Detailed explanation goes here
[K,N] = size(X);
u0= rand(K+1,1); %  u= [b ; w];
A = - repmat(Y,1,K+1).*[ones(N,1) X'];
b = -ones(N,1);

H = eye(K);
H = [zeros(1,K);H];
H = [zeros(K+1,1) H];

p = zeros(K+1,1);
lb = -10*ones(K+1,1);
rb = 10*ones(K+1,1);

options = optimset;    % Options是用来控制算法的选项参数的向量
options.LargeScale = 'off';
options.Display = 'off';
options.Algorithm = 'active-set';

end


ans =

-0.7964    6.3340
6.5654    6.8067
4.4789    5.7348
3.0954    8.4481
-0.4468    6.8201
1.6052    3.6605
7.2111    9.1564
0.5294   10.0426
7.6406    4.7285
4.2191    4.1296
18.7876   20.0922
20.2052   23.3043
26.1079   21.8677
19.1611   22.5008
20.7329   15.8809
23.7969   21.2282
20.5407   22.0610
21.0456   16.2341
19.3506   19.4158
17.8720   26.7284

Y =

1
1
1
1
1
1
1
1
1
1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1

Optimization terminated.

u =

2.3951
-0.1186
-0.0590

val =

0.0088