深度神经网络中的反向传播算法以及对softmax函数求导

神经网络的模型结构

$$\begin{array}{c}{a}_1^{(2)}=\sigma ({\mathbf{W}}_{10}^{(1)}{x}_0+{\mathbf{W}}_{11}^{(1)}{x}_1+{\mathbf{W}}_{12}^{(1)}{x}_2+{\mathbf{W}}_{13}^{(1)}{x}_3)\\ a_2^{(2)}=\sigma ({\mathbf{W}}_{20}^{(1)}{x}_0+{\mathbf{W}}_{21}^{(1)}{x}_1+{\mathbf{W}}_{22}^{(1)}{x}_2+{\mathbf{W}}_{23}^{(1)}{x}_3)\\ a_3^{(2)}=\sigma ({\mathbf{W}}_{30}^{(1)}{x}_0+{\mathbf{W}}_{31}^{(1)}{x}_1+{\mathbf{W}}_{32}^{(1)}{x}_2+{\mathbf{W}}_{33}^{(1)}{x}_3)\end{array}$$

$$\begin{array}{c}{a}_0^{(3)}=\sigma ({\mathbf{W}}_{10}^{(2)}{a}_0^{(2)}+{\mathbf{W}}_{11}^{(2)}{a}_1^{(2)}+{\mathbf{W}}_{12}^{(2)}{a}_2^{(2)}+{\mathbf{W}}_{13}^{(2)}{a}_3^{(2)})\end{array}$$

基本符号

${\mathbf{W}}^{(l)}$ 表示第 $l$ 层的参数矩阵,

${\mathbf{z}}^{(l)}$ 表示第 $l$ 层在激活函数之前的输入,

$\sigma$ 表示激活函数,

${\mathbf{a}}^{(l)}$ 表示第 $l$ 层的输出, ${\mathbf{a}}^{(l)}=\sigma ({\mathbf{z}}^{(l)})$ ,也可以看成第 $l+1$ 层的输入,

${\mathbf{a}}^L$ 表示最终神经网络的输出结果.

${\mathbf{a}}^L$ 一般在计算 $\text{Loss}$ 时,还需要经过 $\text{softmax}$ 操作才可以计算损失.

数据在神经网络中的流动路线

$$\begin{array}{rl}{\mathbf{a}}^{(1)}& =\mathbf{x}\\ {\mathbf{z}}^{(2)}& ={\mathbf{W}}^{(1)}{\mathbf{a}}^{(1)}\\ {\mathbf{a}}^{(2)}& =\sigma ({\mathbf{z}}^{(2)})\\ {\mathbf{z}}^{(3)}& ={\mathbf{W}}^{(2)}{\mathbf{a}}^{(2)}\\ {\mathbf{a}}^{(3)}& =\sigma ({\mathbf{z}}^{(3)})\\ {\mathbf{z}}^{(4)}& ={\mathbf{W}}^{(3)}{\mathbf{a}}^{(3)}\\ & \cdots \\ {\mathbf{z}}^{(L)}& ={\mathbf{W}}^{(L)}{\mathbf{a}}^{(L-1)}\\ {\mathbf{a}}^{(L)}& =\sigma ({\mathbf{z}}^{(L)})\end{array}$$

损失函数

$$\begin{array}{rl}\text{Loss}& =-{\mathbf{y}}^T\ln \hat{\mathbf{y}}\\ & =-\sum _{i=1}^{\text{K}}{y}_i\ln {\hat{y}}_i\end{array}$$
其中 y ∈ { 0 , 1 } C \boldsymbol{y} \in\{0,1\}^{C} y∈{0,1}C 为标签, $\hat{\mathbf{y}}$ 为神经网络的预测值.

$\text{softmax}$ 函数

$${\hat{y}}_k=\text{softmax}({\mathbf{a}}^{(L)}{)}_k=\frac{e^{a_k^{(L)}}}{{\sum }_{i=1}^K{e}^{a_i^{(L)}}}$$

分子布局与分母布局

矩阵微积分的表示通常有两种符号约定: 分子布局 ( Numerator Layout) 和分母布局 ( Denominator Layout). 两者的区别是一个标量关于一个向量的导数是写成列向量还是行向量.标量关于向量的偏导数, 对于 $M$ 维向量 $\mathbf{x}\in {\mathbb{R}}^M$ 和函数 $y=f(\mathbf{x})\in \mathbb{R}$, 则 $y$ 关于 $\mathbf{x}$ 的偏导数为

$$\begin{array}{rlr}\text{分母布局 }\frac{\partial y}{\partial x}& ={\left[\frac{\partial y}{\partial x_1},\cdots ,\frac{\partial y}{\partial x_M}\right]}^T& \in {\mathbb{R}}^{M\times 1},\\ \text{分子布局 }\frac{\partial y}{\partial x}& =\left[\frac{\partial y}{\partial x_1},\cdots ,\frac{\partial y}{\partial x_M}\right]& \in {\mathbb{R}}^{1\times M}.\end{array}$$

在分母布局中, $\frac{\partial y}{\partial x}$ 为列向量;而在分子布局中, $\frac{\partial y}{\partial x}$ 为行向量. 向量关于标量的偏导数, 对于标量 $x\in \mathbb{R}$ 和函数 $\mathbf{y}=f(x)\in {\mathbb{R}}^N$, 则 $\mathbf{y}$ 关于 $x$ 的 偏导数为

$$\begin{gathered} \text{分母布局 }\frac{\partial \mathbf{y}}{\partial x}=\left[\frac{\partial y_1}{\partial x},\cdots ,\frac{\partial y_N}{\partial x}\right]\in {\mathbb{R}}^{1\times N} \\ \text{分子布局 }\frac{\partial \mathbf{y}}{\partial x}={\left[\frac{\partial y_1}{\partial x},\cdots ,\frac{\partial y_N}{\partial x}\right]}^T\in {\mathbb{R}}^{N\times 1} \end{gathered}$$
在分母布局中, $\frac{\partial y}{\partial x}$ 为行向量;而在分子布局中, $\frac{\partial y}{\partial }$ 为列向量.

因为 $\frac{\partial \mathcal{L}(\mathbf{y}\hat{\mathbf{y}})}{\partial {\mathbf{W}}^{(l)}}$ 的计算涉及向量对矩阵的微分, 十分繁琐, 因此我们先计算 $\mathcal{L}(\mathbf{y},\hat{\mathbf{y}})$ 关于参数矩阵中每
个元素的偏导数.

反向传播算法

不失一般性， 对第𝑙 层中的参数 ${\mathbf{W}}^{(l)}$ 和 ${\mathbf{b}}^{(l)}$ 计算偏导数． 这里使用向量或矩阵来表示多变量函数的偏导数， 并使用分子布局表示， 根据链式法则，
$$\begin{array}{l}\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial w_{ij}^{(l)}}=\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}\frac{\partial {\mathbf{z}}^{(l)}}{\partial w_{ij}^{(l)}},\\ \frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{b}}^{(l)}}=\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}\frac{\partial {\mathbf{z}}^{(l)}}{\partial {\mathbf{b}}^{(l)}},\end{array}$$

上述两个公式中的第二项都是目标函数关于第 $l$ 层的神经元 ${\mathbf{z}}^{(l)}$
的偏导数， 称为误差项， 可以一次计算得到． 这样我们只需要计算三个偏导数， 分别为 $\frac{\partial {\mathbf{z}}^{(l)}}{\partial w_{ij}^{(l)}},\frac{\partial {\mathbf{z}}^{(l)}}{\partial {\mathbf{b}}^{(l)}}$ 和 $\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}$

$(1)$ 计算偏导数 $\frac{\partial {\mathbf{z}}^{(l)}}{\partial w_{ij}^{(l)}}$ 因 $z^{(l)}={\mathbf{W}}^{(l)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l)}$, 偏导数

$$\begin{array}{rl}\frac{\partial {\mathbf{z}}^{(l)}}{\partial w_{ij}^{(l)}}& =[\frac{\partial z_1^{(l)}}{\partial w_{ij}^{(l)}},\cdots ,\frac{\partial z_i^{(l)}}{\partial w_{ij}^{(l)}},\cdots ,\frac{\partial z_{M_l}^{(l)}}{\partial w_{ij}^{(l)}}]\\ & =[0,\cdots ,\frac{({\mathbf{w}}_{i:}^{(l)}{\mathbf{a}}^{(l-1)}+b_i^{(l)})}{\partial w_{ij}^{(l)}}\cdots ,0]\\ & =[0,\dots ,a_j^{(l-1)},\dots ,0{]}^T\end{array}$$

其中 ${\mathbf{w}}_{i:}^{(l)}$ 为权重矩阵 ${\mathbf{W}}^{(l)}$ 的第 $i$ 行.

$(2)$ 计算偏导数 $\frac{\partial {\mathbf{z}}^{(l)}}{\partial b^{(l)}}$ 因为 ${\mathbf{z}}^{(l)}$ 和 ${\mathbf{b}}^{(l)}$ 的函数关系为 ${\mathbf{z}}^{(l)}=W^{(l)}{\mathbf{a}}^{(l-1)}+$
${\mathbf{b}}^{(l)}$, 因此偏导数

$$\frac{\partial {\mathbf{z}}^{(l)}}{\partial {\mathbf{b}}^{(l)}}={\mathbf{I}}_{M_l}\in {\mathbb{R}}^{M_l\times M_l}$$

为 $M_l\times M_l$ 的单位矩阵.

$(3)$ 计算偏导数 $\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}$ 偏导数 $\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}$ 表示第 $l$ 层神经元对最终损失
的影响,也反映了最终损失对第 $l$ 层神经元的敏感程度, 因此一般称为第 $l$ 层神经
元的误差项,用 ${\delta }^{(l)}$ 来表示.

$${\delta }^{(l)}\triangleq \frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}\in {\mathbb{R}}^{M_l}$$

误差项 ${\delta }^{(l)}$ 也间接反映了不同神经元对网络能力的贡献程度, 从而比较好地解决
了贡献度分配问题 ( Credit Assignment Problem, CAP ).
根据 ${\mathbf{z}}^{(l+1)}={\mathbf{W}}^{(l+1)}{\mathbf{a}}^{(l)}+{\mathbf{b}}^{(l+1)}$, 有(这里采用的是分母布局)

$$\frac{\partial {\mathbf{z}}^{(l+1)}}{\partial {\mathbf{a}}^{(l)}}={\mathbf{W}}^{(l+1)}\in {\mathbb{R}}^{M_l\times M_{l+1}}$$

根据 ${\mathbf{a}}^{(l)}={\sigma }_l\left({\mathbf{z}}^{(l)}\right)$,其中 ${\sigma }_l(\cdot )$ 为第 $l$ 层神经网络中按位计算的激活函数, 因此有

$$\begin{array}{rl}\frac{\partial {\mathbf{a}}^{(l)}}{\partial {\mathbf{z}}^{(l)}}& =\frac{\partial {\sigma }_l\left({\mathbf{z}}^{(l)}\right)}{\partial {\mathbf{z}}^{(l)}}\\ & =\left[\begin{array}{r}\frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_1}{\partial z_1^{(l)}}\frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_1}{\partial z_2^{(l)}}\dots \frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_1}{\partial z_{m^l}^{(l)}}\\ \frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_2}{\partial z_1^{(l)}}\frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_2}{\partial z_2^{(l)}}\dots \frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_2}{\partial z_{m^l}^{(l)}}\\ ⋮⋮\ddots ⋮\\ \frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_{m^l}}{\partial z_1^{(l)}}\frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_{m^l}}{\partial z_2^{(l)}}\dots \frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_{m^l}}{\partial z_{m^l}^{(l)}}\end{array}\right]\\ & =\left[\begin{array}{r}\frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_1}{\partial z_1^{(l)}}0\dots 0\\ 0\frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_2}{\partial z_2^{(l)}}\dots 0\\ ⋮⋮\ddots ⋮\\ 00\cdots \frac{\partial {\sigma }_l{\left(z^{(l)}\right)}_{m^l}}{\partial z_{m^l}^{(l)}}\end{array}\right]\\ & =\text{diag}\left({\sigma }_l^{\prime }\left({\mathbf{z}}^{(l)}\right)\right)\in {\mathbb{R}}^{M_l\times M_l}\end{array}$$

因此,根据链式法则 ,第 $l$ 层的误差项为

$$\begin{array}{rl}1\times M_l& =(1\times M_{l+1})(M_{l+1}\times M_l)(M_l\times M_l)\\ {\delta }^{(l)}& =\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}\\ & =\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l+1)}}.\frac{\partial {\mathbf{z}}^{l+1}}{\partial {\mathbf{a}}^{(l)}}.\frac{\partial {\mathbf{a}}^l}{\partial {\mathbf{z}}^{(l)}}\\ & ={\delta }^{(l+1)}.{\mathbf{W}}^{(l+1)}.\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(l)}))\end{array}$$

从公式 ${\delta }^{(l)}={\delta }^{(l+1)}.{\mathbf{W}}^{(l+1)}.\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(l)}))$ 可以看出，第 $l$ 层的误差项可以通过第 $l+1$ 层的误差项计算得到, 这就是误差的反向传播(Back Propagation， BP). 反向传播算法的含义是：第 $l$ 层的一个神经元的误差项(或敏感性)是所有与该神经元相连的第$l+1$层的神经元的误差项的权重和. 然后,再乘上该神经元激活函数的梯度. 由于得到 ${\delta }^{(l)}$ 的值需要得到 ${\delta }^{(l+1)}$ 的值,而想要得到 ${\delta }^{(l+1)}$ 的值又需要得到 ${\delta }^{(l+2)}$, 如此一直反向传播下去(与递归过程很类似),直到最后一个 ${\delta }^{(L)}$, ${\delta }^{(L)}$ 是损失函数对输出层(即最后一层)的求导结果.则有:

$$\begin{array}{rl}1\times M_l& =(1\times M_{l+1})(M_{l+1}\times M_l)(M_l\times M_l)\\ {\delta }^{(l)}& ={\delta }^{(l+1)}.{\mathbf{W}}^{(l+1)}.\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(l)}))\\ & ={\delta }^{(l+2)}.{\mathbf{W}}^{(l+2)}.\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(l+1)})).{\mathbf{W}}^{(l+1)}.\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(l)}))\\ & ={\delta }^{(L)}\prod _{i=L}^{l+1}{\mathbf{W}}^{(i)}.\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(i-1)}))\end{array}$$

对最后一层进行求导 $\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(L)}}$

${\delta }^{(L)}$ 的不同形式如下:

$$(1)\begin{array}{rl}{\delta }^{(L)}& =\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{-{\sum }_{i=1}^K{y}_i\ln {\hat{y}}_i}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{-{\sum }_{i=1}^K{y}_i\ln {\left(\text{softmax}({\mathbf{a}}^{(L)})\right)}_i}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{-{\sum }_{i=1}^K{y}_i\ln {\left(\text{softmax}(\sigma ({\mathbf{z}}^{(L)}))\right)}_i}{\partial {\mathbf{z}}^{(L)}}\end{array}$$

---

$$(2)\begin{array}{rl}{\delta }^{(L)}& =\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{\mathbf{y}(\ln \hat{\mathbf{y}}{)}^T}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{\mathbf{y}\ln \text{softmax}({\mathbf{a}}^{(L)}{)}^T}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{\mathbf{y}\ln \text{softmax}(\sigma ({\mathbf{z}}^{(L)}){)}^T}{\partial {\mathbf{z}}^{(L)}}\end{array}$$

---

$$(3)\begin{array}{rl}{\delta }^{(L)}& =\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{(L)}}\\ & =\frac{\partial \mathcal{L}}{\partial \hat{\mathbf{y}}}\frac{\partial \hat{\mathbf{y}}}{\partial {\mathbf{a}}^{(L)}}\frac{\partial {\mathbf{a}}^{(L)}}{\partial {\mathbf{z}}^{(L)}}\end{array}$$

公式	大 小	分子分母
$\frac{\partial \mathcal{L}}{\partial \hat{\mathbf{y}}}$	1*K	标量/矢量
$\frac{\partial \hat{\mathbf{y}}}{\partial {\mathbf{a}}^{(L)}}$	K*K	矢量/矢量
$\frac{\partial {\mathbf{a}}^{(L)}}{\partial {\mathbf{z}}^{(L)}}$	K*K	矢量/矢量

计算 $\frac{\partial \mathcal{L}}{\partial \hat{\mathbf{y}}}$

$$\begin{array}{rl}\frac{\partial \mathcal{L}}{\partial \hat{\mathbf{y}}}& =\frac{\partial \mathbf{y}\ln {\hat{\mathbf{y}}}^T}{\partial \hat{\mathbf{y}}}\\ & =\left[y_1\frac{1}{\widehat{y_1}},y_2\frac{1}{\widehat{y_2}},\cdots ,y_K\frac{1}{{\hat{y}}_K}\right]\end{array}$$

计算 $\frac{\partial \hat{\mathbf{y}}}{\partial {\mathbf{a}}^{(L)}}$

$$\frac{\partial \hat{\mathbf{y}}}{\partial {\mathbf{a}}^{(L)}}=\left[\begin{array}{r}\frac{\partial {\hat{y}}_1}{\partial a_1^{(L)}}\frac{\partial {\hat{y}}_1}{\partial a_2^{(L)}}\dots \frac{\partial {\hat{y}}_1}{\partial a_K^{(L)}}\\ \frac{\partial {\hat{y}}_2}{\partial a_1^{(L)}}\frac{\partial {\hat{y}}_2}{\partial a_2^{(L)}}\dots \frac{\partial {\hat{y}}_2}{\partial a_K^{(L)}}\\ ⋮⋮\ddots ⋮\\ \frac{\partial {\hat{y}}_K}{\partial a_1^{(L)}}\frac{\partial {\hat{y}}_K}{\partial a_2^{(L)}}\dots \frac{\partial {\hat{y}}_K}{\partial a_K^{(L)}}\end{array}\right]$$

$ifj=i:$
$$\begin{array}{rl}\frac{\partial {\hat{y}}_j}{\partial a_i^{(L)}}& =\frac{\partial \left(\frac{e^{a_j^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}\right)}{\partial a_i^{(L)}}\\ & =\frac{\partial \left(\frac{e^{a_j^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}\right)}{\partial a_j^{(L)}}\\ & =\frac{{\left(e^{a_j^{(L)}}\right)}^{\prime }\cdot {\sum }_k{e}^{a_k^{(L)}}-e^{a_j^{(L)}}\cdot e^{a_j^{(L)}}}{{\left({\sum }_k{e}^{a_k^{(L)}}\right)}^2}\\ & =\frac{e^{a_j^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}-\frac{e^{a_j^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}\cdot \frac{e^{a_j^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}={\hat{y}}_j\left(1-{\hat{y}}_j\right)\end{array}$$

$ifj\ne i:$
$$\begin{array}{rl}\frac{\partial {\hat{y}}_j}{\partial a_i^{(L)}}& =\frac{\partial \left(\frac{e^{a_j^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}\right)}{\partial a_i^{(L)}}\\ & =\frac{0\cdot {\sum }_k{e}^{a_k^{(L)}}-e^{a_i^{(L)}}\cdot e^{a_j^{(L)}}}{{\left({\sum }_k{e}^{a_k^{(L)}}\right)}^2}\\ & =-\frac{e^{a_i^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}\cdot \frac{e^{a_j^{(L)}}}{{\sum }_k{e}^{a_k^{(L)}}}=-{\hat{y}}_i{\hat{y}}_j\end{array}$$

计算 $\frac{\partial {\mathbf{a}}^{(L)}}{\partial {\mathbf{z}}^{(L)}}$

由于这里是属于激活函数的部分即
$${\mathbf{a}}^{(L)}=\sigma ({\mathbf{z}}^{(L)})$$
并且激活函数是逐元素进行运算,则有
$$\frac{\partial {\mathbf{a}}^{(L)}}{\partial {\mathbf{z}}^{(L)}}=\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(L)}))$$

最终的 ${\delta }^{(L)}$ 结果

则总的表达式如下
$$\begin{array}{rl}(1\times M_L)& =(1\times M_L)(M_L\times M_L)(M_l\times M_L)=1\times M_L\\ {\delta }^{(L)}& =\frac{\partial \mathcal{L}}{\partial \hat{\mathbf{y}}}\frac{\partial \hat{\mathbf{y}}}{\partial {\mathbf{a}}^{(L)}}\frac{\partial {\mathbf{a}}^{(L)}}{\partial {\mathbf{z}}^{(L)}}\\ & =\left[y_1\frac{1}{\widehat{y_1}},y_2\frac{1}{\widehat{y_2}},\cdots ,y_K\frac{1}{{\hat{y}}_K}\right]\left[\begin{array}{r}\frac{\partial {\hat{y}}_1}{\partial a_1^{(L)}}\frac{\partial {\hat{y}}_1}{\partial a_2^{(L)}}\dots \frac{\partial {\hat{y}}_1}{\partial a_K^{(L)}}\\ \frac{\partial {\hat{y}}_2}{\partial a_1^{(L)}}\frac{\partial {\hat{y}}_2}{\partial a_2^{(L)}}\dots \frac{\partial {\hat{y}}_2}{\partial a_K^{(L)}}\\ ⋮⋮\ddots ⋮\\ \frac{\partial {\hat{y}}_K}{\partial a_1^{(L)}}\frac{\partial {\hat{y}}_K}{\partial a_2^{(L)}}\dots \frac{\partial {\hat{y}}_K}{\partial a_K^{(L)}}\end{array}\right]\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(L)}))\end{array}$$

最终对 $\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial w_{ij}^{(l)}}$ 的求导结果

因此,当我们将上述所有结果汇总即可得到最终对 $\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial w_{ij}^{(l)}}$ 的求导结果

$$\begin{array}{rl}(1\times 1)& =(1\times M_l)(M_l\times 1)=(1\times 1)=1\\ \frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial w_{ij}^{(l)}}& =\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}\frac{\partial {\mathbf{z}}^{(l)}}{\partial w_{ij}^{(l)}}\\ & =\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial {\mathbf{z}}^{(l)}}\frac{\partial {\mathbf{z}}^{(l)}}{\partial w_{ij}^{(l)}}\\ & ={\delta }^{(L)}\left(\prod _{i=L}^{l+1}{\mathbf{W}}^{(i)}.\text{diag}({\sigma }^{\prime }({\mathbf{z}}^{(i-1)}))\right)[0,\dots ,a_j^{(l-1)},\dots ,0{]}^T\end{array}$$

由丄式可知,在求导 $\frac{\partial \mathcal{L}(\mathbf{y},\hat{\mathbf{y}})}{\partial w_{ij}^{(l)}}$ 时,距离最后一层越近( $l$ 越接近 $L$ ),则求导所需的计算量越少, 并且计算靠近输入层的参数需要靠近输出层的一些数据,这就是反向传播.

参考资料

• [1] 《神经网络与深度学习》
• [２] 《李宏毅人工智能》