深度学习中交叉熵函数的导数:(极简)

---

前言

另一个博主有更详细的推导https://blog.csdn.net/chaipp0607/article/details/101946040

---

一.交叉熵函数的导数

• softmax:令一条数据最后的输出为[z1,z2,z3,z4,…,z10],这里令输出层的神经元数量为10
  $$pi=\frac{e^{z_i}}{{\sum }_{j=1}^{10}{e}^{z_j}}$$
• cross_entropy:
  $$L=-\sum _{i=1}^{10}{y}_i\times log(pi)$$
• 链式法则:
  $$\frac{\partial L}{\partial z_i}=\sum _{j=1}^{10}\frac{\partial L}{\partial p_j}\frac{\partial p_j}{\partial z_i}$$
• 逐个击破
  • $$\frac{\partial L}{\partial p_j}=-y_j\times \frac{1}{p_j}$$
  • $\frac{\partial p_j}{\partial z_i}$有两种情况,这里直接跳过中间步骤(链式求导)
    • 1.i!=j
      $$\frac{\partial p_j}{\partial z_i}=-p_i\times p_j$$
      $$\frac{\partial L}{\partial p_j}\frac{\partial p_j}{\partial z_i}=p_i\times y_j$$
    • i==j
      $$\frac{\partial p_j}{\partial z_i}=p_i\times (1-p_i)$$
      $$\frac{\partial L}{\partial p_j}\frac{\partial p_j}{\partial z_i}=(p_i-1)\times y_i$$

---

$$\begin{gathered} 综上,\frac{\partial L}{\partial z_i}=\sum _{j=1}^{10}\frac{\partial L}{\partial p_j}\frac{\partial p_j}{\partial z_i}=\sum _{j\ne i}\frac{\partial L}{\partial p_j}\frac{\partial p_j}{\partial z_i}+(p_i-1)\times y_i= \\ \sum _{j\ne i}{p}_i\times y_j+(p_i-1)\times y_i \end{gathered}$$
$$且已知\sum _{j=1}^{10}{y}_i=1,所以\frac{\partial L}{\partial z_i}=p_i-y_i!$$

二.Z,y为有n条数据的矩阵

$$z=\left[\begin{array}{c}{z}_{10},z_{11},...,z_{19}\\ ...\\ z_{n0},z_{n1},...,z_{n9}\end{array}\right]$$
$$y=\left[\begin{array}{c}{y}_{10},y_{11},...,y_{19}\\ ...\\ y_{n0},y_{n1},...,y_{n9}\end{array}\right]$$
$$\frac{\partial L}{\partial z}=softmax(z)-y//这里用到numpy的广播机制$$
$$\frac{\partial L}{\partial z}=\left[\begin{array}{c}softmax(z_{10}-y_1),softmax(z_{11}-y_1),...,softmax(z_{19}-y_1)\\ ...\\ softmax(z_{n0}-y_n),softmax(z_{n1}-y_n),...,softmax(z_{n9}-y_n)\end{array}\right]$$