cs231n assignment svm公式推导

简要记录学习理解。

1. svm 是一个线性分类器

直接由权重矩阵和图片(input x) 得到一个值，通过svm算法，不断更新W，使 正确的标签的一栏得分最高。

2. Loss function

Loss function 就是评价目前这个W矩阵有’多坏‘。 其值越高，则分类效果越差， 值越低，则效果越好

3. Regularization

作用：简而言之，防止过拟合。

所以总的Loss function 为

4. 公式推导 & 代码解析

svm_loss_naive 函数

def svm_loss_naive(W, X, y, reg):
    """
    Structured SVM loss function, naive implementation (with loops).

    Inputs have dimension D, there are C classes, and we operate on minibatches
    of N examples.

    Inputs:
    - W: A numpy array of shape (D, C) containing weights.
    - X: A numpy array of shape (N, D) containing a minibatch of data.
    - y: A numpy array of shape (N,) containing training labels; y[i] = c means
      that X[i] has label c, where 0 <= c < C.
    - reg: (float) regularization strength

    Returns a tuple of:
    - loss as single float
    - gradient with respect to weights W; an array of same shape as W
    """
    dW = np.zeros(W.shape) # initialize the gradient as zero

    # compute the loss and the gradient
    num_classes = W.shape[1]
    num_train = X.shape[0]
    loss = 0.0
    for i in range(num_train):
        scores = X[i].dot(W)
        correct_class_score = scores[y[i]]  #correct_label_score
        for j in range(num_classes):
            if j == y[i]:
                continue
            margin = scores[j] - correct_class_score + 1 # note delta = 1
           
            if margin > 0:
                loss += margin
                dW[:,j] += X[i,:]
                dW[:,y[i]] -= X[i,:]



    # Right now the loss is a sum over all training examples, but we want it
    # to be an average instead so we divide by num_train.
    loss /= num_train
    dW /= num_train
    # Add regularization to the loss.
    loss += reg * np.sum(W * W)
    dW += reg * W
    return loss, dW.

Xi 即为输入矩阵的第i 个图片， 假设这张图片的正确标签为最后一个， 则score的最后一个就为Syi. 则j 范围1-9个（本例中10 个类别)

 loss = 0.0
    for i in range(num_train):
        scores = X[i].dot(W)
        correct_class_score = scores[y[i]]  #correct_label_score
        for j in range(num_classes):
            if j == y[i]:
                continue
            margin = scores[j] - correct_class_score + 1 # note delta = 1
           
            if margin > 0:
                loss += margin
                dW[:,j] += X[i,:]
                dW[:,y[i]] -= X[i,:]

当margin > 0 时， 被计入loss 里面。 对于每一个特定的 j 列， 参与求score 的为 Wj 和 Wyi, 所以求导值需要 关注 第 j 列和 第 y[i] 列。 (y[i] 为 Xi 对应的label）。 而他们的倒数就为 Xi.
应该注意的是，对输入的第 i 个图像（Xi) 来说， yi 是不变的， 则其dW[:,y[i]] 就减去了 margin > 0 的次数 * Xi。 这是下面 svm_loss_vectorized 函数要用到的思路。

svm_loss_vectorized 函数

上面的函数时对输入的图片（Xi) 逐个处理， 这个是把所有的图片一起处理，就是输入矩阵X，每一行代表一张图片。 顺着上面的函数思路，利用numpy 特性处理。

def svm_loss_vectorized(W, X, y, reg):
    """
    Structured SVM loss function, vectorized implementation.

    Inputs and outputs are the same as svm_loss_naive.
    """
    loss = 0.0
    dW = np.zeros(W.shape) # initialize the gradient as zero
    num_train = X.shape[0]
    #############################################################################
    # TODO:                                                                     #
    # Implement a vectorized version of the structured SVM loss, storing the    #
    # result in loss.                                                           #
    #############################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    
    scores = X.dot(W)  
    correct_class_score = scores[np.arange(num_train),y]
    correct_class_score = np.reshape(correct_class_score,(num_train,-1))
    margin = scores - correct_class_score + 1
    margin = np.maximum(0, margin)
    margin[np.arange(num_train),y] = 0
    loss += np.sum(margin) / num_train
    loss += reg * np.sum(W * W) 
    pass

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    #############################################################################
    # TODO:                                                                     #
    # Implement a vectorized version of the gradient for the structured SVM     #
    # loss, storing the result in dW.                                           #
    #                                                                           #
    # Hint: Instead of computing the gradient from scratch, it may be easier    #
    # to reuse some of the intermediate values that you used to compute the     #
    # loss.                                                                     #
    #############################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    margin[margin > 0] = 1    
    row_sum = np.sum(margin, axis = 1)   #记录对于某个特定的i，margin > 0 的个数
    margin[np.arange(num_train),y] = -row_sum
    dW += np.dot(X.T, margin) / num_train + reg * W
  

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return loss, dW