反向传播

• 一些笔记（未完待续）

文章中的英文描述，公式以及图片，均来自吴恩达深度学习课程的课后作业

$\frac{\partial \mathcal{J}}{\partial z_2^{(i)}}=\frac{1}{m}(a^{[2](i)}-y^{(i)})$

$\frac{\partial \mathcal{J}}{\partial W_2}=\frac{\partial \mathcal{J}}{\partial z_2^{(i)}}{a}^{[1](i)T}$

$\frac{\partial \mathcal{J}}{\partial b_2}={\sum }_i\frac{\partial \mathcal{J}}{\partial z_2^{(i)}}$

$\frac{\partial \mathcal{J}}{\partial z_1^{(i)}}=W_2^T\frac{\partial \mathcal{J}}{\partial z_2^{(i)}}\ast (1-a^{[1](i)2})$

$\frac{\partial \mathcal{J}}{\partial W_1}=\frac{\partial \mathcal{J}}{\partial z_1^{(i)}}{X}^T$

$\frac{\partial {\mathcal{J}}_i}{\partial b_1}={\sum }_i\frac{\partial \mathcal{J}}{\partial z_1^{(i)}}$

下图是反向传播时的梯度计算，输出层激活函数为Sigmoid函数，隐藏层的激活函数是tanh()，右侧是对应的向量化实现。

• Note that $\ast$ denotes elementwise multiplication.

• The notation you will use is common in deep learning coding:

  • dW1 = $\frac{\partial \mathcal{J}}{\partial W_1}$
  • db1 = $\frac{\partial \mathcal{J}}{\partial b_1}$
  • dW2 = $\frac{\partial \mathcal{J}}{\partial W_2}$
  • db2 = $\frac{\partial \mathcal{J}}{\partial b_2}$

• Tips:

  • To compute dZ1 you’ll need to compute $g^{[1{]}^{\prime }}(Z^{[1]})$. Since $g^{[1]}(.)$ is the tanh activation function, if $a=g^{[1]}(z)$ then $g^{[1{]}^{\prime }}(z)=1-a^2$. So you can compute
    $g^{[1{]}^{\prime }}(Z^{[1]})$ using (1 - np.power(A1, 2)).