softmax函数求导

softmax函数简写为$S_i=\frac{e^i}{{\sum }_j{e}^j}$
交叉熵损失函数$L=-{\sum }_k{t}_{k}ln(y=k)$。其中目标类的$t_k$为1，其余类的$t_k$为0.
当预测为第i个时，可以认为$t_i=1$，$y_i$表示求出的softmax值。
此时损失函数变成了：$Los{s}_i=-ln{y}_i$。
根据定义：$y_i=\frac{e^i}{{\sum }_j{e}^j}$。我们已经将数值映射到了0-1之间，并且和为1，则有:$$\frac{e^i}{{\sum }_j{e}^j}=1-\frac{{\sum }_{j\ne i}{e}^j}{{\sum }_j{e}^j}$$
接下来对Loss求导：$$\frac{\partial Los{s}_i}{{\partial }_i}=-\frac{\partial ln{y}_i}{{\partial }_i}=\frac{\partial (-ln\frac{e^i}{{\sum }_j{e}^J})}{{\partial }_i}$$$$=-\frac{1}{\frac{e^i}{{\sum }_j{e}^J}}\cdot \frac{\partial (\frac{e^i}{{\sum }_j{e}^J})}{{\partial }_i}$$$$=-\frac{{\sum }_j{e}^j}{e^i}\cdot \frac{\partial (1-\frac{{\sum }_{j\ne i}{e}^j}{{\sum }_j{e}^j})}{{\partial }_i}$$ $$=-\frac{{\sum }_j{e}^j}{e^i}\cdot (-\sum _{j\ne i}{e}^j)\cdot \frac{\partial (\frac{1}{{\sum }_j{e}^j})}{{\partial }_i}$$ $$=\frac{{\sum }_j{e}^j\cdot {\sum }_{j\ne i}{e}^j)}{e^i}\cdot (-1)\cdot \frac{e^i}{({\sum }_j{e}^j{)}^2}$$$$=-\frac{{\sum }_{j\ne i}{e}^j}{{\sum }_j{e}^j}$$$$=\frac{e^i}{{\sum }_j{e}^j}-1$$$$=y_i-1$$
所以我们只需要正向求出$y_i$，将结果减1就是反向更新的梯度。