1. SVM 代码解析
损失函数的计算公式如下:
L
(
W
)
=
1
N
∑
i
=
1
N
L
i
(
f
(
x
i
,
W
)
,
y
i
)
+
λ
R
(
W
)
L(W) = \frac{1}{N} \sum_{i=1}^N L_i(f(x_i,W),y_i)+\lambda R(W)
L(W)=N1∑i=1NLi(f(xi,W),yi)+λR(W)
而损失函数关于权重
W
W
W的求导公式如下:
∇
W
L
(
W
)
=
1
N
∑
i
=
1
N
∇
W
L
i
(
x
i
,
y
i
,
W
)
+
λ
∇
W
R
(
W
)
=
1
N
∑
i
=
1
N
x
i
∗
y
i
+
W
\nabla_W L(W)=\frac{1}{N} \sum_{i=1}^{N}\nabla_W L_i(x_i,y_i,W) + \lambda \nabla_W R(W) \\ = \frac{1}{N} \sum_{i=1}^{N} x_i * y_i + W
∇WL(W)=N1∑i=1N∇WLi(xi,yi,W)+λ∇WR(W)=N1∑i=1Nxi∗yi+W
对于Hinge Loss, L i ( f ( x i , W ) , y i ) = ∑ j ≠ y i m a x ( 0 , ( W ∗ x i ) j − ( W ∗ x i ) y i + Δ ) L_i(f(x_i,W),y_i) = \sum_{j\neq y_i}max(0, (W*x_i)_j - (W*x_i)_{y_i} + \Delta) Li(f(xi,W),yi)=∑j=yimax(0,(W∗xi)j−(W∗xi)yi+Δ)
在实验中我们取 Δ = 1 \Delta = 1 Δ=1, 正则化项 λ R ( W ) = 1 2 ∣ ∣ W ∣ ∣ F 2 \lambda R(W) = \frac{1}{2} ||W||_F^2 λR(W)=21∣∣W∣∣F2, 预测结果 s c o r e i = W ∗ x i ∈ R c score_{i} = W*x_i \in R^c scorei=W∗xi∈Rc,c为数据的类别数。
代码中的margin就是第 i i i 个样本在第 j j j 类的loss,一个内循环结束后就得到该样本的损失值。
Hinge loss 导数:
∂
L
∂
W
=
\frac{\partial L}{\partial W} =
∂W∂L=
Softmax Loss 导数:
def svm_loss_naive(W, X, y, reg):
"""
Structured SVM loss function, naive implementation (with loops).
Inputs have dimension D, there are C classes, and we operate on minibatches
of N examples.
Inputs:
- W: A numpy array of shape (D, C) containing weights.
- X: A numpy array of shape (N, D) containing a minibatch of data.
- y: A numpy array of shape (N,) containing training labels; y[i] = c means
that X[i] has label c, where 0 <= c < C.
- reg: (float) regularization strength
Returns a tuple of:
- loss as single float
- gradient with respect to weights W; an array of same shape as W
"""
dW = np.zeros(W.shape) # initialize the gradient as zero
# compute the loss and the gradient
num_classes = W.shape[1]
num_train = X.shape[0]
loss = 0.0
for i in range(num_train):
scores = X[i].dot(W)
correct_class_score = scores[y[i]]
for j in range(num_classes):
if j == y[i]:
continue
margin = scores[j] - correct_class_score + 1 # note delta = 1
if margin > 0:
loss += margin
# Right now the loss is a sum over all training examples, but we want it
# to be an average instead so we divide by num_train.
loss /= num_train
# Add regularization to the loss.
loss += reg * np.sum(W * W)
dW = (W - h)/h
return loss, dW
-
Hinge loss ,softmax loss梯度计算
-
Svm ,softmax训练的损失函数及测试结果 权重可视化结果