DNN的BP推导

最新推荐文章于 2022-10-13 14:20:42 发布

X丶

最新推荐文章于 2022-10-13 14:20:42 发布

阅读量374

点赞数

文章标签：机器学习

本文链接：https://blog.csdn.net/qq742762377/article/details/89703367

版权

向前传播：

第一层

$z^{(2)} = x \cdot w^{(1)}$

$a^{(2)} = g(z^{(2)})$

第二层

$z^{(3)} = a^{(2)} \cdot w^{(2)}$

$a^{(3)} = g(z^{(3)})$

第三层

$z^{(4)} = a^{(3)} \cdot w^{(3)}$

$a^{(4)} = g(z^{(4)})$

计算代价：

$J = (a^{(4)} - y)^{2}$

反向传播：

这是一个求偏导数的线路图

$\frac{\partial J}{\partial a^{(4)}} =\frac{\partial J}{\partial z^{(4)}}= 2 \cdot (a^{(4)} - y)$

$\frac{\partial J}{\partial z^{(4)}} = \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}}$

$\frac{\partial J}{\partial a^{(3)}} = \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial a^{(3)}}$

$\frac{\partial J}{\partial z^{(3)}} = \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial a^{(3)}} \cdot \frac{\partial a^{(3)}}{\partial z^{(3)}}$

$\frac{\partial J}{\partial a^{(2)}} = \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial a^{(3)}} \cdot \frac{\partial a^{(3)}}{\partial z^{(3)}} \cdot \frac{\partial z^{(3)}}{\partial a^{(2)}}$

$\frac{\partial J}{\partial z^{(2)}} = \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial a^{(3)}} \cdot \frac{\partial a^{(3)}}{\partial z^{(3)}} \cdot \frac{\partial z^{(3)}}{\partial a^{(2)}} \cdot \frac{\partial a^{(2)}}{\partial z^{(2)}}$

我们会发现，这都是有规律的: 每一列都是一样的，也就是我们可以通过从结果J反向求偏导数，累积偏导数来反向递推计算每一层的 $a^{(l)}$ 和 $z^{(l)}$

也就是我们可以根据上一项的梯度(grad)来进一步求出下一项的梯度(grad)

既然我们可以求出每一层 $z^{(l)}$ 的梯度(grad) ，那么，我们自然就可以根据 $z^{(l)}$ 进一步求出 $w^{(l-1)}$ 的梯度(grad)

$\frac{\partial J}{\partial w^{(3)}} = \frac{\partial J}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial w^{(3)}} = \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial w^{(3)}}$

$\frac{\partial J}{\partial w^{(2)}} = \frac{\partial J}{\partial z^{(3)}} \cdot \frac{\partial z^{(3)}}{\partial w^{(2)}}= \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial a^{(3)}} \cdot \frac{\partial a^{(3)}}{\partial z^{(3)}} \cdot \frac{\partial z^{(3)}}{\partial w^{(2)}}$

$\frac{\partial J}{\partial w^{(1)}} = \frac{\partial J}{\partial z^{(2)}} \cdot \frac{\partial z^{(2)}}{\partial w^{(1)}}= \frac{\partial J}{\partial a^{(4)}} \cdot \frac{\partial a^{(4)}}{\partial z^{(4)}} \cdot \frac{\partial z^{(4)}}{\partial a^{(3)}} \cdot \frac{\partial a^{(3)}}{\partial z^{(3)}} \cdot \frac{\partial z^{(3)}}{\partial a^{(2)}} \cdot \frac{\partial a^{(2)}}{\partial z^{(2)}} \cdot \frac{\partial z^{(2)}}{\partial w^{(1)}}$