Softmax交叉熵损失函数 反向传播公式推导

Softmax交叉熵损失函数 反向传播公式推导

标签（空格分隔）： Caffe源代码

---

Softmax交叉熵损失函数：

$$\begin{array}{l} J = - \sum\limits_{i = 1}^K {{y_i}\ln ({z_i})} \\ {z_i} = \frac{{{e^{{x_i}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }} \end{array}$$
现在我们的目的是求 $\frac{{\partial J}}{{\partial {x_k}}}$

在训练集中，假设 $y_{s}=1$，其余$y_{k \ne s} = 0$

我们分为两种情况：
（1）当 $k=s$ 时:

$$\frac{{\partial J}}{{\partial {x_k}}} = \frac{{\partial ( - {y_s}ln(\frac{{{e^{{x_s}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }}))}}{{\partial {x_s}}} = - {y_s} \times \frac{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }}{{{e^{{x_s}}}}} \times \frac{{{e^{{x_s}}} \times \sum\limits_{j = 1}^K {{e^{{x_j}}}} - {e^{2{x_s}}}}}{{{{(\sum\limits_{j = 1}^K {{e^{{x_j}}}} )}^2}}} = - {y_s} \times \frac{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} - {e^{{x_s}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }} = - {y_s}(1 - {z_s})$$

(2)当$k \ne s$ 时：

$$\frac{{\partial J}}{{\partial {x_k}}} = \frac{{\partial ( - {y_s}ln(\frac{{{e^{{x_s}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }}))}}{{\partial {x_k}}} = - {y_s} \times \frac{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }}{{{e^{{x_s}}}}} \times \frac{{ - {e^{{x_s}}} \times {e^{{x_k}}}}}{{{{(\sum\limits_{j = 1}^K {{e^{{x_j}}}} )}^2}}} = - {y_s}\frac{{{e^{{x_k}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }} = - {y_s}{z_k}$$

总结：

$$\frac{{\partial J}}{{\partial {x_k}}} = z_{k} - y_{k}$$