3. 前向/反向传播——Softmax

参考资料

cs231n Course Materials: Backprop
Derivatives, Backpropagation, and Vectorization
cs231n Lecture 4:Neural Networks and Backpropagation
cs231n Assignment 2
笔记: Batch Normalization及其反向传播

3. SoftMax 损失函数

"""
Inputs:
    - X: Input data, of shape (N, C) where x[i, j] is the score for the jth
      class for the ith input.
    - Y: Vector of labels, of shape (N,) where y[i] is the label for x[i] and
      0 <= y[i] < C

Returns a tuple of:
	- L: Scalar giving the loss
	- dx: Gradient of the loss with respect to x
"""

$$L_i=-\log \frac{e^{X_{i,y_i}}}{{\sum }_j{e}^{X_{i,j}}}\text{\hspace{1em}(3.1)}$$
$$L=\frac{1}{N}\sum _i{L}_i\text{\hspace{1em}(3.2)}$$
为了防止数值溢出，一般在实现时进行如下变形：
L i = − log ⁡ e X i , y i ∑ j e X i , j = − log ⁡ e max ⁡ { e i , ⋅ X } e ( X i , y i − max ⁡ { e i , ⋅ X } ) e max ⁡ { e i , ⋅ X } ∑ j e ( X i , j − max ⁡ { e i , ⋅ X } ) = − log ⁡ e ( X i , y i − max ⁡ { e i , ⋅ X } ) ∑ j e ( X i , j − max ⁡ { e i , ⋅ X } ) (3.3) \begin{aligned}L_i&=-\log{\frac{e^{X_{i,y_i}}}{\sum_{j}e^{X_{i,j}}}}\\&=-\log{\frac{e^{\max{\{e^X_{i,\cdot}\}}}e^{\left(X_{i,y_i}-\max{\{e^X_{i,\cdot}\}}\right)}}{e^{\max{\{e^X_{i,\cdot}\}}}\sum_{j}e^{\left(X_{i,j}-\max{\{e^X_{i,\cdot}\}}\right)}}}\\&=-\log{\frac{e^{\left(X_{i,y_i}-\max{\{e^X_{i,\cdot}\}}\right)}}{\sum_{j}e^{\left(X_{i,j}-\max{\{e^X_{i,\cdot}\}}\right)}}}\end{aligned}\tag{3.3} Li​​=−log∑j​eXi,j​eXi,yi​​​=−logemax{ei,⋅X​}∑j​e(Xi,j​−max{ei,⋅X​})emax{ei,⋅X​}e(Xi,yi​​−max{ei,⋅X​})​=−log∑j​e(Xi,j​−max{ei,⋅X​})e(Xi,yi​​−max{ei,⋅X​})​​(3.3)

关于反向传播，推导如下：
由于式(3.3)的变形并不影响函数的值，所以可以使用变形前的形式进行推导。
分成两种情况进行讨论：
(1) 对$X_{i,y_i}$求梯度
$$\begin{array}{rl}\frac{\partial L}{\partial X_{i,y_i}}& =\frac{1}{N}\frac{\partial L_i}{\partial X_{i,y_i}}\\ & =-\frac{1}{N}\frac{{\sum }_j{e}^{X_{i,j}}}{e^{X_{i,y_i}}}\frac{e^{X_{i,y_i}}{\sum }_j{e}^{X_{i,j}}-{\left(e^{X_{i,y_i}}\right)}^2}{{\left({\sum }_j{e}^{X_{i,j}}\right)}^2}\\ & =\frac{1}{N}\frac{e^{X_{i,y_i}}-{\sum }_j{e}^{X_{i,j}}}{{\sum }_j{e}^{X_{i,j}}}\\ & =\frac{1}{N}\left(\frac{e^{X_{i,y_i}}}{{\sum }_j{e}^{X_{i,j}}}-1\right)\end{array}\text{\hspace{1em}(3.4)}$$

(2) 对$X_{i,k}(k\ne y_i)$求梯度
$$\begin{array}{rl}\frac{\partial L}{\partial X_{i,k}}& =\frac{1}{N}\frac{\partial L_i}{\partial X_{i,k}}\\ & =\frac{1}{N}\frac{{\sum }_j{e}^{X_{i,j}}}{e^{X_{i,y_i}}}\frac{e^{X_{i,y_i}}{e}^{X_{i,k}}}{{\left({\sum }_j{e}^{X_{i,j}}\right)}^2}\\ & =\frac{1}{N}\frac{e^{X_{i,k}}}{{\sum }_j{e}^{X_{i,j}}}\end{array}\text{\hspace{1em}(3.5)}$$

令
$$p_{i,k}=\frac{e^{X_{i,k}}}{{\sum }_j{e}^{X_{i,j}}}\text{\hspace{1em}(3.6)}$$
则有
$$\frac{\partial L}{\partial X_{i,y_i}}=\frac{1}{N}\left(p_{i,y_i}-1\right)\text{\hspace{1em}(3.7)}$$
$$\frac{\partial L}{\partial X_{i,k}}=\frac{1}{N}{p}_{i,k}\text{\hspace{1em}(3.8)}$$