Softmax及其损失函数求导推导过程

Softmax函数介绍

在深度学习领域，多分类问题的激活函数常常使用softmax函数，它将多个神经元的输出，映射到（0,1）区间内，可以看成概率来理解，从而来进行多分类！
以前学机器学习时了解过softmax，但没仔细推导过，这段时间在学CS224n课程，要自己推导，看了一些网上的推导，始终有些地方不明白的，后来终于把softmax求梯度过程弄明白了，下面把自己的心得和推导过程记录一下
先看一下Softmax函数：
$$Softmax(\theta i)=\frac{e^{\theta i}}{{\sum }_k{e}^{\theta }}$$

Softmax的损失函数：交叉熵

$$Loss=\sum _k{y}_i\mathrm{l}\mathrm{n}{a}_i$$

Softmax求导过程

Softmax函数输入$\theta$是有k个元素的向量，每个$\theta$都会对应着一个softmax，因此，每个softmax都要分别对每个$\theta$求导，对$\theta$求导结果就是一个k*k的雅可比矩阵
$$\left[\begin{array}{cccc}\frac{\partial S_0}{\partial {\theta }_0}& \frac{\partial S_1}{\partial {\theta }_0}& ...& \frac{\partial S_k}{\partial {\theta }_0}\\ \frac{\partial S_0}{\partial {\theta }_1}& \frac{\partial S_1}{\partial {\theta }_1}& ...& \frac{\partial S_k}{\partial {\theta }_0}\\ ...& ...& ...& ...\\ \frac{\partial S_0}{\partial {\theta }_k}& \frac{\partial S_1}{\partial {\theta }_k}& ...& \frac{\partial S_k}{\partial {\theta }_k}\end{array}\right]$$

代入softmax公式，求偏导公式为：
$$\frac{\partial S_i}{\partial {\theta }_j}=\frac{\partial \frac{e^{{\theta }_i}}{\sum e^{\theta }}}{\partial {\theta }_j}$$
令$u=e^{{\theta }_i}$, $v=\sum e^{\theta }$,则根据复合函数求导法则，$\frac{u}{v}$导数为$\frac{u^{\prime }v-u{v}^{\prime }}{v^2}$：
$$\begin{array}{rl}\frac{\partial S_i}{\partial {\theta }_j}& =\frac{\partial \frac{e^{{\theta }_i}}{\sum e^{\theta }}}{\partial {\theta }_j}\\ & =\frac{\frac{\partial e^{{\theta }_i}}{\partial {\theta }_j}\sum e^{\theta }-e^{{\theta }_i}\frac{\partial \sum e^{\theta }}{\partial {\theta }_j}}{(\sum e^{\theta }{)}^2}\end{array}$$
此时，需要考虑$i=j和i̸=j$的情况
当$i=j$时：
$$\begin{array}{rl}\frac{\partial e^{{\theta }_i}}{\partial {\theta }_j}& =e^{{\theta }_i}\\ \frac{\partial \sum e^{\theta }}{\partial {\theta }_j}& =e^{{\theta }_i}\\ 则\frac{\partial S_i}{\partial {\theta }_j}& =\frac{e^{{\theta }_i}\sum e^{\theta }-e^{{\theta }_i}{e}^{{\theta }_i}}{(\sum e^{\theta }{)}^2}\\ & =\frac{e^{{\theta }_i}}{\sum e^{\theta }}\cdot \frac{\sum e^{\theta }-e^{{\theta }_i}}{\sum e^{\theta }}\\ & =S_i\cdot (1-S_i)\end{array}$$
当$i̸=j$时：
$$\begin{array}{rl}\frac{\partial e^{{\theta }_i}}{\partial {\theta }_j}& =0\\ \frac{\partial \sum e^{\theta }}{\partial {\theta }_j}& =e^{{\theta }_j}\\ 则\frac{\partial S_i}{\partial {\theta }_j}& =\frac{0\cdot \sum e^{\theta }-e^{{\theta }_i}{e}^{{\theta }_j}}{(\sum e^{\theta }{)}^2}\\ & =-\frac{e^{{\theta }_i}}{\sum e^{\theta }}\cdot \frac{e^{{\theta }_j}}{\sum e^{\theta }}\\ & =-S_i\cdot S_j\end{array}$$

损失函数求导

根据链式求导法则，损失函数对$\theta$求导可进行分解：
$$\begin{array}{rl}\frac{dL}{d\theta }& =\frac{dL}{dS}\cdot \frac{dS}{d\theta }\\ & =-\frac{d\sum y_i\mathrm{l}\mathrm{n}{S}_i}{d{S}_i}\cdot \frac{d{S}_i}{d{\theta }_i}\\ & =-\sum \frac{y_i}{S_i}\cdot \frac{d{S}_i}{d{\theta }_j}\end{array}$$
代入上面的softmax求导结果，将$i=j$和$i̸=j$结果相加：
$$\begin{array}{rl}\frac{dL}{d\theta }& =-\frac{y_i}{S_i}\cdot S_i(1-S_i)-\sum _{i̸=j}\frac{y_i}{S_j}\cdot (-S_i\cdot S_j)\\ & =-y_i\cdot (1-S_i)+\sum _{i̸=j}{y}_i\cdot S_i\\ & =S_i\sum y_i-y_i\end{array}$$
因为y_i相加的结果等于1，因此最后的结果就是
$$\frac{dL}{d\theta }=S_i-y_i$$