Softmax与Cross-entropy的求导

引言

在多分类问题中，一般会把输出结果传入到softmax函数中，得到最终结果。并且用交叉熵作为损失函数。本来就来分析下以交叉熵为损失函数的情况下，softmax如何求导。

对softmax求导

softmax函数为:

$${\hat{y}}_i=\frac{e^{z_i}}{{\sum }_{k=1}^K{e}^{z_k}}$$

这里$K$是类别的总数，接下来求${\hat{y}}_i$对某个输出$z_j$的导数,
$$\frac{\partial {\hat{y}}_i}{\partial z_j}=\frac{\partial \frac{e^{z_i}}{{\sum }_{k=1}^K{e}^{z_k}}}{\partial z_j}$$

这里要分两种情况,分别是$i=j$与$i\ne j$。当$i=j$时,$e^{z_i}$对$z_j$的导数为$e^{z_i}$，否则当$i\ne j$时，导数为$0$。

当$i=j$，
$$\begin{array}{rl}\frac{\partial {\hat{y}}_i}{\partial z_j}& =\frac{e^{z_i}\cdot {\sum }_{k=1}^K{e}^{z_k}-e^{z_i}\cdot e^{z_j}}{({\sum }_{k=1}^m{e}^{z_k}{)}^2}\\ & =\frac{e^{z_i}}{{\sum }_{k=1}^m{e}^{z_k}}-\frac{e^{z_i}}{{\sum }_{k=1}^m{e}^{z_k}}\cdot \frac{e^{z_j}}{{\sum }_{k=1}^m{e}^{z_k}}\\ & ={\hat{y}}_i-{\hat{y}}_i^2={\hat{y}}_i(1-{\hat{y}}_i)\end{array}$$

当$i\ne j$，
$$\begin{array}{rl}\frac{\partial {\hat{y}}_i}{\partial z_j}& =\frac{0\cdot {\sum }_{k=1}^K{e}^{z_k}-e^{z_i}\cdot e^{z_j}}{({\sum }_{k=1}^m{e}^{z_k}{)}^2}\\ & =-\frac{e^{z_i}}{{\sum }_{k=1}^m{e}^{z_k}}\cdot \frac{e^{z_j}}{{\sum }_{k=1}^m{e}^{z_k}}\\ & =-{\hat{y}}_i{\hat{y}}_j\end{array}$$

对cross-entropy求导

损失函数$L$为:

$$L=-\sum _k{y}_k\log {\hat{y}}_k$$

其中$y_k$是真实类别，相当于一个常数，接下来求$L$对$z_j$的导数

$$\begin{array}{rl}\frac{\partial L}{\partial z_j}& =\frac{\partial -({\sum }_k{y}_k\log {\hat{y}}_k)}{z_j}\\ & =\frac{\partial -({\sum }_k{y}_k\log {\hat{y}}_k)}{\partial {\hat{y}}_k}\frac{\partial {\hat{y}}_k}{\partial z_j}\\ & =-\sum _k{y}_k\frac{1}{{\hat{y}}_k}\frac{\partial {\hat{y}}_k}{z_j}\\ & ={\left(-y_k\cdot {\hat{y}}_k(1-{\hat{y}}_k)\frac{1}{{\hat{y}}_k}\right)}_{k=j}-\sum _{k\ne j}{y}_k\frac{1}{{\hat{y}}_k}(-{\hat{y}}_k{\hat{y}}_j)\\ & =-y_j(1-{\hat{y}}_j)-\sum _{k\ne j}{y}_k(-{\hat{y}}_j)\\ & =-y_j+y_j{\hat{y}}_j+\sum _{k\ne j}{y}_k({\hat{y}}_j)\\ & =-y_j+\sum _k{y}_k({\hat{y}}_j)\\ & =-y_j+{\hat{y}}_j\\ & ={\hat{y}}_j-y_j\end{array}$$

这里用到了${\sum }_k{y}_k=1$

可以看到，求导结果非常简单，如果不推导都不敢信。