cs231n - assignment1 - softmax 梯度推导

Softmax exercise

Complete and hand in this completed worksheet (including its outputs and any supporting code outside of the worksheet) with your assignment submission. For more details see the assignments page on the course website.

This exercise is analogous to the SVM exercise. You will:

- implement a fully-vectorized loss function for the Softmax classifier 
- implement the fully-vectorized expression for its analytic gradient 
- check your implementation with numerical gradient 
- use a validation set to tune the learning rate and regularization strength 
- optimize the loss function with SGD 
- visualize the final learned weights

和linear_svm一样，主要难点是求导操作，不过softmax的求导更简单一些。 
首先还是给出 Loss 的公式： 

L=1N∑iLi+λR(W)−−−（1）

其中共有 N 个样本，每个样本带来的 Loss 是 Li: 

Li=−logpyi=−log(efyi∑jefj)=−fyi+log∑jefj−−−（2）

对于每一个样本 Xi ， 由于 softmax 的分母对所有的 fj 进行了累积求和， 所以 Li 对 W的导数对 W的每一列都又贡献， 即 ∂LiWj 对所有的 j 都不为 0: 
当 j！=yi 时： 

∂Li∂Wj=efyi∑jefj∂fi∂Wj=efyi∑jefjXTi−−−（3）

当 j==yi 时： 

∂Li∂Wj=efyi∑jefj∂fi∂Wj=efyi∑jefjXTi−XTi−−−（4）

对所有样本都求出对应的Loss, 累积求和，并加上正则项即可以得到最终要求的Loss了。

---

上面求导数过程是把 Loss 对于 W 的导数显示的写出来，然后直接对 W 求导数，在这个简单的例子中可以这样，但是一旦网络变得复杂了，就很难直接写出Loss 对于要求的表达式的导数了。一种比较好的方式是利用 chain rule 逐级的求导数： 

pk=efk∑jefj,Li=−logpyi−−−（5）

这里 fk 是 softmax 层的输出，由上面公式 (2） 可以求出 Loss 对 fk 的导数为： 

∂Li∂fk=pk−1(yi=k)−−−（6）

该式子表明 Loss 对 softmax 层的输出的导数为 pk，并且当 k=yi 时导数项还要减去1。 
把式（6） 改写为向量形式： 

∂Li∂f=p−[0...1...](第yi维为1)−−−（6a）

现在考虑第二层 fully connected layer，也就是紧连着 softmax 的那一层全连接层，这一层的输入是隐藏层的输出 hiddenlayer[1×H]， 所以softmax的输入 f=hidden_layer.dot(W2)+b2， 检查一下维度， f 为 C 维向量， W2 为 H×C 的二维矩阵， b2 为 C 维向量，没问题。 现在就可以来求 f 对 W2 的导数了： 

∂f∂W=hidden_layer.T(维度为H×1)−−−（7）

可以看到，∂f∂W的是全连接层的输入向量。

综合以上结果就可以求得： 

∂Li∂W=∂f∂W∂Li∂f−−−(8)

---

最后对所有 N 个样本写成矩阵形式： 

∂L∂f=p[N×C]−MaskMat[N×C]−−−（6m）

∂f∂W=hiddenlayerT[N×H]−−−（7m）

∂L∂W=∂f∂W∂L∂f[H×C]−−−(8m)

其中(6m)中的 MskMat为 N 个(6a)中向量组成，具体形式可以参见如下python代码：

 # compute the gradient on scores
  dscores = probs
  dscores[range(num_examples),y] -= 1
1
2
3

---

# softmax.py
import numpy as np
from random import shuffle

def softmax_loss_naive(W, X, y, reg):
  """
  Softmax loss function, naive implementation (with loops)

  Inputs have dimension D, there are C classes, and we operate on minibatches
  of N examples.

  Inputs:
  - W: A numpy array of shape (D, C) containing weights.
  - X: A numpy array of shape (N, D) containing a minibatch of data.
  - y: A numpy array of shape (N,) containing training labels; y[i] = c means
    that X[i] has label c, where 0 <= c < C.
  - reg: (float) regularization strength

  Returns a tuple of:
  - loss as single float
  - gradient with respect to weights W; an array of same shape as W
  """
  # Initialize the loss and gradient to zero.
  loss = 0.0
  dW = np.zeros_like(W)

  #############################################################################
  # TODO: Compute the softmax loss and its gradient using explicit loops.     #
  # Store the loss in loss and the gradient in dW. If you are not careful     #
  # here, it is easy to run into numeric instability. Don't forget the        #
  # regularization!                                                           #
  #############################################################################
  num_train = X.shape[0]
  num_classes = W.shape[1]
  for i in xrange(num_train):
    scores = X[i].dot(W) 
    scores -= np.max(scores) #prevents numerical instability
    correct_class_score = scores[y[i]]

    exp_sum = np.sum(np.exp(scores))
    loss += np.log(exp_sum) - correct_class_score

    dW[:, y[i]] -= X[i]
    for j in xrange(num_classes):
      dW[:,j] += (np.exp(scores[j]) / exp_sum) * X[i]

  loss /= num_train
  loss += 0.5 * reg * np.sum( W*W )
  dW /= num_train
  dW += reg * W

  #############################################################################
  #                          END OF YOUR CODE                                 #
  #############################################################################

  return loss, dW


def softmax_loss_vectorized(W, X, y, reg):
  """
  Softmax loss function, vectorized version.

  Inputs and outputs are the same as softmax_loss_naive.
  """
  # Initialize the loss and gradient to zero.
  loss = 0.0
  dW = np.zeros_like(W)

  #############################################################################
  # TODO: Compute the softmax loss and its gradient using no explicit loops.  #
  # Store the loss in loss and the gradient in dW. If you are not careful     #
  # here, it is easy to run into numeric instability. Don't forget the        #
  # regularization!                                                           #
  #############################################################################
  num_train = X.shape[0]
  num_classes = W.shape[1]

  scores = X.dot(W)
  scores -= np.max(scores, axis = 1)[:, np.newaxis]
  exp_scores = np.exp(scores)
  sum_exp_scores = np.sum(exp_scores, axis = 1)
  correct_class_score = scores[range(num_train), y]

  loss = np.sum(np.log(sum_exp_scores)) - np.sum(correct_class_score)

  exp_scores = exp_scores / sum_exp_scores[:,np.newaxis]

  # maybe here can be rewroten into matrix operations 
  for i in xrange(num_train):
    dW += exp_scores[i] * X[i][:,np.newaxis]
    dW[:, y[i]] -= X[i]

  loss /= num_train
  loss += 0.5 * reg * np.sum( W*W )
  dW /= num_train
  dW += reg * W
  #############################################################################
  #                          END OF YOUR CODE                                 #
  #############################################################################

  return loss, dW
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

# softmax.ipynb
# Use the validation set to tune hyperparameters (regularization strength and
# learning rate). You should experiment with different ranges for the learning
# rates and regularization strengths; if you are careful you should be able to
# get a classification accuracy of over 0.35 on the validation set.
from cs231n.classifiers import Softmax
results = {}
best_val = -1
best_softmax = None
learning_rates = [5e-6, 1e-7, 5e-7]
regularization_strengths = [1e4, 5e4, 1e5]

################################################################################
# TODO:                                                                        #
# Use the validation set to set the learning rate and regularization strength. #
# This should be identical to the validation that you did for the SVM; save    #
# the best trained softmax classifer in best_softmax.                          #
################################################################################
params = [(x,y) for x in learning_rates for y in regularization_strengths ]
for lrate, regular in params:
    softmax = Softmax()
    loss_hist = softmax.train(X_train, y_train, learning_rate=lrate, reg=regular,
                             num_iters=700, verbose=True)
    y_train_pred = softmax.predict(X_train)
    accuracy_train = np.mean( y_train == y_train_pred)
    y_val_pred = softmax.predict(X_val)
    accuracy_val = np.mean(y_val == y_val_pred)
    results[(lrate, regular)] = (accuracy_train, accuracy_val)
    if(best_val < accuracy_val):
        best_val = accuracy_val
        best_softmax = softmax
################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print 'lr %e reg %e train accuracy: %f val accuracy: %f' % (
                lr, reg, train_accuracy, val_accuracy)

print 'best validation accuracy achieved during cross-validation: %f' % best_val