cs231n assignment1 SVM

上一篇：cs231n assignment1 knn

SVM

支持向量机的损失函数为
$$L_i=\sum _{j!=y_i}\max \left(0,s_j-s_{y_i}+△\right)$$
其中$s_j$是其他分类得分，$s_{yi}$是当前图片真实分类的得分，加入正则化后的公式为
$$L=\frac{1}{N}\sum _i\sum _{j!=y_i}\max \left(0,f{\left(x_i;W\right)}_j-f{\left(x_i;W\right)}_{y_i}+△\right)+\lambda \sum _k\sum _l{W}_{k,l}^2$$

在本次作业中，还需要计算梯度，然鹅CS231N视频中，老师并未讲解如何计算梯度，故参考了别人的文章：SVM损失函数及梯度矩阵的计算

linear_svm.py中的svm_loss_naive(W, X, y, reg)函数：

	dW = np.zeros(W.shape) # initialize the gradient as zero

    # compute the loss and the gradient
    num_classes = W.shape[1]
    num_train = X.shape[0]
    loss = 0.0
    for i in range(num_train):
        scores = X[i].dot(W)
        # 真实类别的分数
        correct_class_score = scores[y[i]]
        for j in range(num_classes):
            if j == y[i]:
                continue
            margin = scores[j] - correct_class_score + 1 # note delta = 1
            if margin > 0:
                loss += margin
                dW[:,y[i]] += -X[i]
                dW[:,j] += X[i]

    # Right now the loss is a sum over all training examples, but we want it
    # to be an average instead so we divide by num_train.
    loss /= num_train
    dW /= num_train

    # Add regularization to the loss.
    loss += reg * np.sum(W * W)
    dW += 2 * reg * W

接下来需要完成完全向量化的代码，损失函数计算时可以完全按照公式来走，一看代码就能明白，梯度计算时参考【cs231n】SVM与Softmax的梯度下降

linear_svm.py中的svm_loss_vectorized(W, X, y, reg)函数

	# 计算得分矩阵
    scores = X.dot(W)
    # 矩阵行数
    num_train = X.shape[0]

    # 取出每一行的真实分类的那一列的得分,并且reshape成(num_train * 1)
    current_score = scores[np.arange(num_train), y].reshape(num_train, 1)
    # 损失函数计算
    margins = np.maximum(0, scores - current_score + 1)
    # 将其中真实分类对应列的值置为0
    margins[np.arange(num_train), y] = 0
    # 对所有的列求和，并对所有行求和
    loss = np.sum(margins)
    loss /= num_train
    # 正则化
    loss += reg * np.sum(W ** 2)

    # 计算梯度
    dS = np.zeros_like(scores)
    # 找到大于0的坐标，二维坐标
    idx = np.where(scores - current_score + 1 > 0)
    dS[idx] = 1
    dS[np.arange(num_train), y] = -1 * (np.sum(scores - current_score + 1 > 0, axis=1) - 1)
    dW = X.T.dot(dS)
    dW /= num_train
    dW += 2 * reg * W

---

接下来需要实现随机梯度下降算法
linear_classifier.py中train()函数

# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
idx = np.random.choice(range(num_train),batch_size,replace=True)
X_batch = X[idx,:]
y_batch = y[idx]
pass
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
self.W = self.W - learning_rate * grad
pass
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

接下来实现预测，很简单
linear_classifier.py中的predict()函数

scores = X.dot(self.W)
y_pred = np.argmax(scores,axis=1)
pass

---

继续svm文件代码阅读，紧接着是自己写代码实现最好超参选择，我在代码中把损失函数画出来，然后通过最终的结果比较选择参数，可以参考我选择的参数，最终结果见后图。
svm文件

num_iters = 1500
for lr in learning_rates:
    for rs in regularization_strengths:
        svm = LinearSVM()
        tic = time.time()
        # 关闭训练时损失值的输出
        loss_hist = svm.train(X_train,y_train,learning_rate=lr,reg=rs,num_iters=num_iters,verbose=False)
        toc = time.time()
        plt.plot(loss_hist)
        plt.xlabel('Iteration number')
        plt.ylabel('Loss value')
        plt.show()
        y_train_pred = svm.predict(X_train)
        acc_train = np.mean(y_train == y_train_pred)
        y_val_pred = svm.predict(X_val)
        acc_val = np.mean(y_val == y_val_pred)
        
        # 更新准确率
        if best_val < acc_val:
            best_val = acc_val
            best_svm = svm
        
        # 完成记录
        results[(lr,rs)] = (acc_train,acc_val)
        
pass

最终准确率达到了39%

Inline Question

Inline Question 1

It is possible that once in a while a dimension in the gradcheck will not match exactly. What could such a discrepancy be caused by? Is it a reason for concern? What is a simple example in one dimension where a gradient check could fail? How would change the margin affect of the frequency of this happening? Hint: the SVM loss function is not strictly speaking differentiable

$\text{Y}ourAnswer:$ fill this in.
程序中的梯度是通过数值计算得到的，因为max函数在0处连续但不可导，可能导致计算值与实际值不一样。

Inline question 2

Describe what your visualized SVM weights look like, and offer a brief explanation for why they look they way that they do.

$\text{Y}ourAnswer:$ fill this in
每一类的权重可视化图像，都大致展示该类物体的形状以及背景颜色。当图片中出现了类似该类的形状或者背景颜色时，有很大的概率被归类为这一类。例如：船、汽车、青蛙等。