softmax求导

softmax函数：
$$S_i=\frac{e^{a_i}}{{\sum }_j{e}^{a_j}}$$

softmax求导

$$\frac{\partial S_i}{\partial a_j}=\frac{\frac{\partial e^{a_i}}{\partial a_j}\cdot \Sigma -\frac{\partial \Sigma }{\partial a_j}\cdot e^{a_i}}{{\sum }^2}$$

当$i=j$时：
$$\frac{\partial S_i}{\partial a_j}=\frac{e^{a_i}\cdot \Sigma -e^{a_j}{e}^{a_i}}{{\Sigma }^2}=\frac{e^{a_i}}{\Sigma }\cdot \frac{\Sigma -e^{a_j}}{\Sigma }=S_i\cdot \left(1-S_j\right)$$

当$i\ne j$时：
$$\frac{\partial S_i}{\partial a_j}=-\frac{e^{{\alpha }_j}\cdot e^{a_i}}{{\Sigma }^2}=-S_i\cdot S_j$$

交叉熵损失函数

$$L=-\sum y_i\log S_i$$

$$\frac{\partial L}{\partial S_i}=-y_i\cdot \frac{1}{S_i}$$

对softmax的输入求导：

$$\begin{gathered} \frac{\partial L}{\partial a_i}=\sum _j\frac{\partial L}{\partial S_j}\cdot \frac{\partial S_j}{\partial a_i}=\frac{\partial L}{\partial S_i}\cdot \frac{\partial S_i}{\partial a_i}+\sum _{j\ne i}\frac{\partial L}{\partial S_j}\cdot \frac{\partial S_j}{\partial a_i} \\ =-y_i\cdot \frac{1}{S_i}\cdot S_i\left(1-S_i\right)+\sum _{j\ne i}\frac{-y_j}{S_j}\cdot (-1)S_i{S}_j \\ =-y_i\left(1-S_i\right)+\sum _{j\ne i}{y}_j\cdot S_i \\ =-y_i+y_i{S}_i+\sum _{j\ne i}{y}_j\cdot S_i \\ =S_i-y_i \end{gathered}$$