cs231n - assignment1 - softmax 梯度推导

Softmax exercise

Complete and hand in this completed worksheet (including its outputs and any supporting code outside of the worksheet) with your assignment submission. For more details see the assignments page on the course website.

This exercise is analogous to the SVM exercise. You will:

- implement a fully-vectorized loss function for the Softmax classifier
- implement the fully-vectorized expression for its analytic gradient
- check your implementation with numerical gradient
- use a validation set to tune the learning rate and regularization strength
- optimize the loss function with SGD
- visualize the final learned weights

和linear_svm一样，主要难点是求导操作，不过softmax的求导更简单一些。
首先还是给出 Loss 的公式：

$$L = { \frac{1}{N} \sum_i L_i }+ { \lambda R(W) } ---（1）$$
其中共有 $N$ 个样本，每个样本带来的 Loss 是 $L_i$:
$$L_i =-\log{p_{y_i}}= -\log\left(\frac{e^{f_{y_i}}}{ \sum_j e^{f_j} }\right) = -f_{y_i} + \log\sum_j e^{f_j} ---（2）$$

对于每一个样本 $X_i$ ， 由于 softmax 的分母对所有的 $f_j$ 进行了累积求和， 所以 $L_i$ 对 $W$的导数对 $W$的每一列都又贡献， 即 $\frac{\partial{L_i}}{W_j}$ 对所有的 j 都不为 0:
当 j！=y