Neural Networks and Deep Learning 2

本章我做的矩阵乘法的地方与这本书使用相同，不过我所有向量都是列向量

Ch02 How the backpropagation algorithm works

在线书籍http://neuralnetworksanddeeplearning.com/chap2.html

目录

The four fundamental equations behind backpropagation

• Alternate presentation of the equations of backpropagation: I’ve stated the equations of backpropagation (notably (BP1¹) and (BP2²)) using the Hadamard product. This presentation may be disconcerting if you’re unused to the Hadamard product. There’s an alternative approach, based on conventional matrix multiplication, which some readers may find enlightening. (1) Show that (BP1) may be rewritten as
  $${\delta }^L={\Sigma }^{\prime }(z^L){\nabla }_{a}C,$$
  where ${\Sigma }^{\prime }(z^L)$ is a square matrix whose diagonal entries are the values ${\sigma }^{\prime }(z_j^L)$, and whose off-diagonal entries are zero. Note that this matrix acts on ${\nabla }_{a}C$ by conventional matrix multiplication. (2) Show that (BP2) may be rewritten as
  $${\delta }^l={\Sigma }^{\prime }(z^l)(w^{l+1}{)}^T{\delta }^{l+1}$$
  (3) By combining observations (1) and (2) show that
  $${\delta }^l={\Sigma }^{\prime }(z^l)(w^{l+1}{)}^T...{\Sigma }^{\prime }(z^{L-1})(w^L{)}^T{\Sigma }^{\prime }(z^L){\nabla }_{a}C$$
  For readers comfortable with matrix multiplication this equation may be easier to understand than (BP1) and (BP2). The reason I’ve focused on (BP1) and (BP2) is because that approach turns out to be faster to implement numerically.
  证明：(BP1)
  Expected node of symbol group type, but got node of type cr
  即${\delta }^L={\left(\frac{\partial C}{\partial a_1^L}{\sigma }^{\prime }(z_1^L),\frac{\partial C}{\partial a_2^L}{\sigma }^{\prime }(z_2^L),...,\frac{\partial C}{\partial a_k^L}{\sigma }^{\prime }(z_k^L)\right)}^T$
  (BP2)
  ${\delta }^l=\left({\left(w^{l+1}\right)}^T{\delta }^{l+1}\right)\odot {\sigma }^{\prime }(z^l)$
  $=\left[\begin{array}\
  w_{11}^{l+1} & w_{12}^{l+1} & … & w_{1k}^{l+1} \
  w_{21}^{l+1} & w_{22}^{l+1} & … & w_{2k}^{l+1} \
  … & … & …& … \
  w_{j1}^{l+1} & w_{j2}^{l+1} & … & w_{jk}^{l+1} \\
  \end{array}\right]^T
  \cdot \left[\begin{array}\
  \delta_1^{l+1} \
  \delta_2^{l+1} \
  … \
  \delta_j^{l+1}
  \end{array}\right] \odot\left[\begin{array}\
  \sigma’(z_1^l) \
  \sigma’(z_2^l) \
  … \
  \sigma’(z_k^l)
  \end{array}\right] $
  $=\left[\begin{array}\
  \sum_j (w_{j1}^{{l+1}\delta_j}{l+1})\cdot \sigma’(z_1^l) \
  \sum_j (w_{j2}^{{l+1}\delta_j}{l+1})\cdot \sigma’(z_2^l) \
  … \
  \sum_j (w_{jk}^{{l+1}\delta_j}{l+1})\cdot \sigma’(z_k^l)
  \end{array}\right] $
  ${\Sigma }^{\prime }(z^l)(w^{l+1}{)}^T{\delta }^{l+1}$
  $ = \left[\begin{array} \
  \sigma’(z_1^L) & 0 & … & 0 \
  0 & \sigma’(z_2^L) & … & 0 \
  … & … & … & …\
  0 & … & … & \sigma’(z_k^L)
  \end{array}\right] \cdot \left[\begin{array}\
  w_{11}^{l+1} & w_{12}^{l+1} & … & w_{1k}^{l+1} \
  w_{21}^{l+1} & w_{22}^{l+1} & … & w_{2k}^{l+1} \
  … & … & …& … \
  w_{j1}^{l+1} & w_{j2}^{l+1} & … & w_{jk}^{l+1} \\
  \end{array}\right]^T\cdot \left[\begin{array}\
  \delta_1^{l+1} \
  \delta_2^{l+1} \
  … \
  \delta_j^{l+1}
  \end{array}\right]$
  $ = \left[\begin{array}\
  \sigma’(z_1^l) \cdot \sum_j (w_{j1}^{l+1} \delta_j^{l+1}) \
  \sigma’(z_2^l) \cdot \sum_j (w_{j2}^{l+1} \delta_j^{l+1}) \
  … \
  \sigma’(z_k^l) \cdot \sum_j (w_{jk}^{l+1} \delta_j^{l+1})
  \end{array}\right] = \delta^l$
  其中第$l$层有$k$个神经元，第$l+1$层有$j$个神经元，$w_{jk}^{l+1}$表示第$l$层第$k$个神经元对第$l+1$层第$j$个神经元的作用。
  ${\delta }^l={\Sigma }^{\prime }(z^l)(w^{l+1}{)}^T...{\Sigma }^{\prime }(z^{L-1})(w^L{)}^T{\Sigma }^{\prime }(z^L){\nabla }_{a}C$
  将${\delta }^{l+1}$迭代为${\delta }^{l+1}={\Sigma }^{\prime }(z^{l+1})(w^{l+2}{)}^T{\delta }^{l+2}$代入即可。

Proof of the four fundamental equations (optional)

• Prove Equations (BP3³) and (BP4⁴).
  证明：(BP3)由链式求导
  $\frac{\partial C}{\partial b_j^l}=\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial z_j^l}\frac{\partial z_j^l}{\partial b_j^l}$
  由于$\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial z_j^l}=\frac{\partial C}{\partial a_j^l}{\sigma }^{\prime }(z_j$