svm-loss 关于权重矩阵W的导数（cs231n Assignment 1）

svm-loss 关于权重矩阵W的导数（cs231n Assignment 1）

先给出相应习题的代码，各位可以自行领会一下：

def svm_loss_vectorized(W, X, y, reg):
  """
  Structured SVM loss function, vectorized implementation.

  Inputs and outputs are the same as svm_loss_naive.
  """
  loss = 0.0
  dW = np.zeros(W.shape) # initialize the gradient as zero
  scores = X.dot(W)
  num_train = X.shape[0]
  num_type = W.shape[1]
  print(num_type)
  #############################################################################
  # TODO:                                                                     #
  # Implement a vectorized version of the structured SVM loss, storing the    #
  # result in loss.                                                           #
  #############################################################################
  correct_scores = scores[range(num_train), y].reshape(-1, 1)
  pre_loss = scores + 1 - correct_scores
  loss = (np.sum(np.maximum(pre_loss, 0)) - num_train) / num_train + reg * np.sum(W * W)
  #############################################################################
  #                             END OF YOUR CODE                              #
  #############################################################################

  #############################################################################
  # TODO:                                                                     #
  # Implement a vectorized version of the gradient for the structured SVM     #
  # loss, storing the result in dW.                                           #
  #                                                                           #
  # Hint: Instead of computing the gradient from scratch, it may be easier    #
  # to reuse some of the intermediate values that you used to compute the     #
  # loss.                                                                     #
  #############################################################################
  mask = np.ones(scores.shape)
  cnt = pre_loss > 0
  mask[range(num_train), y] = 1 - np.sum(cnt, axis = 1)
  dW = X.T.dot(mask * (pre_loss > 0)) / num_train + 2 * reg * W
  #############################################################################
  #                             END OF YOUR CODE                              #
  #############################################################################

  return loss, dW

本题是cs231n Assignment 1中关于svm-loss向量化方法的一个思路，并不是严格的数学证明。

首先给出SVM-loss的表达式
设输入矩阵为$X$，$X\in R^{N\times D}$,$y$为样本标签集$y\in R^{N \times 1}$,权重矩阵为$W$， $W\in R^{D\times C}$,
其中$N$为训练样本的个数，$D$为样本的维数, $C$为标签的种类数 ，
设$S$为SVM的输出矩阵 $S = XW \in R^{N \times C}$，

$$L = \dfrac{1}{N}\sum_{i=1}^N\sum_{j=1,j\not=y_i}^Cmax(S_{i,j}-S_{i,y_i}+1, 0)$$
关于svm-loss函数此处不再赘述，详细请参加CS231n的课程内容

在CS231n 的Assignment 1中要求向量化svm-loss函数对于权重矩阵W的导数， 即$\dfrac{dL}{dW}$

这里我们将$L$的表达式稍做变形, 由于对任意给定的$i$ 有且只有一个$j$, 使得$j=y_i$(因为$y_i$唯一)
且对于该$j=y_i$， $max(S_{i,j}-S_{i,y_i}+1, 0)$恒等于1，从而，$L$可以变形为:

$$\begin{split} L &= \dfrac{1}{N}\sum_{i=1}^N\bigg[\sum_{j=1}^C\big(max(S_{i,j}-S_{i,y_i}+1, 0)\big)-1\bigg]\\ &=\dfrac{1}{N}\sum_{i=1}^N\sum_{j=1}^C\big(max(S_{i,j}-S_{i,y_i}+1, 0)\big)-1 \end{split}$$
max函数不方便处理，所以我们考虑消去max，由于 $max(S_{i,j}-S_{i,y_i}+1, 0)$在 S