[deeplearning-004] 一般形式的反向传导算法BP最简推导-2

最新推荐文章于 2022-01-26 09:43:56 发布

未济2019

最新推荐文章于 2022-01-26 09:43:56 发布

阅读量456

点赞数

分类专栏：深度学习

本文链接：https://blog.csdn.net/lizhe_dashuju/article/details/78360649

版权

深度学习专栏收录该内容

18 篇文章

订阅专栏

本文详细介绍了深度学习中反向传播算法（BP）的最简推导，从梯度下降法出发，解析了神经网络中权重矩阵和偏置矩阵的一阶偏导数计算，逐步展开推导出一般形式，为优化网络参数提供理论基础。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

3.2 梯度下降
训练神经网络，就是让 $J(\mathbf W, \mathbf b)$ 的值最小。解法就是梯度下降，或者随机梯度下降。
梯度下降法的迭代公式是：

(W l j, i) s + 1 = (W l j, i) s - η \partial J ( W , b ) \partial W l j , i ∣ W l j, i = (W l j, i) s

$(\mathbf W^l_{j,i})_{s+1} =(\mathbf W^l_{j,i})_s-\eta \frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^l_{j,i}}\lvert_{\mathbf W^l_{j,i} =(\mathbf W^l_{j,i})_s}$

(b l j) s + 1 = (b l j) s - η \partial J ( W , b ) \partial b l j ∣ b l i = (b l j) s

$(\mathbf b^l_{j})_{s+1}=(\mathbf b^l_{j})_s-\eta \frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf b^l_{j}}\lvert_{\mathbf b^l_{i}=(\mathbf b^l_{j})_s}$
其中，

s $s$ 是迭代次数，

η $\eta$ 是学习速率。

上两式的一阶偏导又可以写成：

\partial J ( W , b ) \partial W l j , i = \sum k = 1 m \partial \partial W l j , i J (W, b; x k, y k) + λ W l j, i

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^l_{j,i}}=\sum_{k=1}^m\frac{\partial}{\partial \mathbf W^l_{j,i}}J(\mathbf W, \mathbf b;\mathbf x^k,y^k)+\lambda\mathbf W^l_{j,i}$

\partial J ( W , b ) \partial b l j = \sum k = 1 m \partial \partial b l j J (W, b; x k, y k)

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf b^l_j}=\sum_{k=1}^m\frac{\partial}{\partial \mathbf b^l_j}J(\mathbf W, \mathbf b;\mathbf x^k,\mathbf y^k)$

3.2.1 输出层 $L^p$ 和隐层 $L^{p-1}$ 之间的权重矩阵 $\mathbf W^{p-1}$ 一阶偏导求解并推广到一般情形

我们可以把神经网络视为一个复杂一点的函数，自变量 $\mathbf x$ 经过一系列函数的操作，输出因变量 $\mathbf y$ ， $\mathbf y$ 是由 $\mathbf W$ 和 $\mathbf b$ 决定的，优化输出结果，需要调整 $\mathbf W$ 和 $\mathbf b$ 。

这里，先推导一个具体的权重，也就是最后一个隐层的第二个神经元跟输出层的第一个神经元之间的权重，然后再扩展到一般形式：

\partial J ( W , b ) \partial W p - 1 1 , 2 = \sum k = 1 m \partial \partial W p - 1 1 , 2 J (W, b; x k, y k) + λ W p - 1 1, 2 = 1 2 \sum k = 1 m \partial \partial W p - 1 1 , 2 (\sum q = 1 d p (y k q - f (z p, k q)) 2) + λ W p - 1 1, 2 = \sum k = 1 m \sum q = 1 d p (- (y k q - f (z p, k q)) f' (z p, k q) \partial z p , k q \partial W p - 1 1 , 2) + λ W p - 1 1, 2 = \sum k = 1 m \sum q = 1 d p (δ p, k q \partial z p , k q \partial W p - 1 1 , 2) + λ W p - 1 1, 2 = \sum k = 1 m δ p, k 1 a p - 1, k 2 + λ W p - 1 1, 2

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^{p-1}_{1,2}} =\sum_{k=1}^m\frac{\partial}{\partial \mathbf W^{p-1}_{1,2}}J(\mathbf W, \mathbf b;\mathbf x^k,y^k)+\lambda\mathbf W^{p-1}_{1,2} =\frac{1}{2}\sum_{k=1}^m\frac{\partial}{\partial \mathbf W^{p-1}_{1,2}}(\sum_{q=1}^{d^p}(\mathbf y^{k}_q-f(z^{p,k}_q))^2)+\lambda\mathbf W^{p-1}_{1,2} =\sum_{k=1}^m\sum_{q=1}^{d^p}(-(\mathbf y^k_q-f(z^{p,k}_q))f^{'}(z^{p,k}_q)\frac{\partial z^{p,k}_q}{\partial \mathbf W^{p-1}_{1,2}})+\lambda\mathbf W^{p-1}_{1,2} =\sum_{k=1}^m\sum_{q=1}^{d^p}(\delta^{p,k}_{q} \frac{\partial z^{p,k}_q}{\partial \mathbf W^{p-1}_{1,2}})+\lambda\mathbf W^{p-1}_{1,2} =\sum_{k=1}^m\delta^{p,k}_{1}a^{p-1,k}_2 +\lambda\mathbf W^{p-1}_{1,2}$

其中， $\delta^{p,k}_{q}=-(\mathbf y^k_q-f(z^{p,k}_q))f^{'}(z^{p,k}_q)$ 是简便写法。

进一步地， $\mathbf W^{p-1}$ 的所有权重系数的一阶偏导如下：

\partial J ( W , b ) \partial W p - 1 j , i = \sum k = 1 m δ p, k j a p - 1, k i + λ W p - 1 j, i

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^{p-1}_{j,i}} =\sum_{k=1}^m\delta^{p,k}_{j}a^{p-1,k}_i +\lambda\mathbf W^{p-1}_{j,i}$

神经网络的最后一个 $W^l$ 跟其他层有一点区别在于其 $\delta^{p,k}_{q}$ 是直接的，其他层不是如此，需要推导一般形式对 $\mathbf W^l_{j,i}$ 一阶偏导，也就是，推导其和后一层的各值的关系。先从 $\mathbf W^{p-2}$ 开始：注意，考虑到误差最小时应当设置 $D=d^p$ ，以免引起对 $p$ 的不必要推导误会。

\partial J ( W , b ) \partial W p - 2 j , i = \sum k = 1 m \partial \partial W p - 2 j , i J (W, b; x k, y k) + λ W p - 2 j, i = 1 2 \sum k = 1 m \partial \partial W p - 2 j , i (\sum q = 1 D (y k q - f (z p, k q)) 2) + λ W p - 2 j, i = \sum k = 1 m \sum q = 1 D (- (y k q - f (z p, k q)) f' (z p, k q) \partial z p , k q \partial W p - 2 j , i) + λ W p - 2 j, i = \sum k = 1 m \sum q = 1 D (δ p, k q \partial z p , k q \partial W p - 2 j , i) + λ W p - 2 j, i = \sum k = 1 m \sum q = 1 D (δ p, k q \partial z p , k q \partial z p - 1 , k j \partial z p - 1 , k j \partial W p - 2 j , i) + λ W p - 2 j, i = \sum k = 1 m \sum q = 1 D (δ p, k q \partial z p , k q \partial z p - 1 , k j a p - 2, k i) + λ W p - 2 j, i = \sum k = 1 m a p - 2, k i (\sum q = 1 D (δ p, k q \partial z p , k q \partial z p - 1 , k j)) + λ W p - 2 j, i = \sum k = 1 m a p - 2, k i (\sum q = 1 D (δ p, k q f' (z p - 1, k j) W p - 1 q, j)) + λ W p - 2 j, i = \sum k = 1 m a p - 2, k i f' (z p - 1, k j) (\sum q = 1 D (δ p, k q W p - 1 q, j)) + λ W p - 2 j, i

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^{p-2}_{j,i}} =\sum_{k=1}^m\frac{\partial}{\partial \mathbf W^{p-2}_{j,i}}J(\mathbf W, \mathbf b;\mathbf x^k,y^k)+\lambda\mathbf W^{p-2}_{j,i} =\frac{1}{2}\sum_{k=1}^m\frac{\partial}{\partial \mathbf W^{p-2}_{j,i}}(\sum_{q=1}^{D}(\mathbf y^k_q-f(z^{p,k}_q))^2)+\lambda\mathbf W^{p-2}_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(-(\mathbf y^k_q-f(z^{p,k}_q))f^{'}(z^{p,k}_q)\frac{\partial z^{p,k}_q}{\partial \mathbf W^{p-2}_{j,i}})+\lambda\mathbf W^{p-2}_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(\delta^{p,k}_{q} \frac{\partial z^{p,k}_q}{\partial \mathbf W^{p-2}_{j,i}})+\lambda\mathbf W^{p-2}_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(\delta^{p,k}_q\frac{\partial z^{p,k}_q}{\partial z^{p-1,k}_j}\frac{\partial z^{p-1,k}_j}{\partial \mathbf W^{p-2}_{j,i}})+\lambda\mathbf W^{p-2}_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(\delta^{p,k}_q\frac{\partial z^{p,k}_q}{\partial z^{p-1,k}_j}a^{p-2,k}_i)+\lambda\mathbf W^{p-2}_{j,i} =\sum_{k=1}^ma^{p-2,k}_i(\sum_{q=1}^{D}(\delta^{p,k}_q\frac{\partial z^{p,k}_q}{\partial z^{p-1,k}_j}))+\lambda\mathbf W^{p-2}_{j,i} =\sum_{k=1}^ma^{p-2,k}_i(\sum_{q=1}^{D}(\delta^{p,k}_qf^{'}(z^{p-1,k}_j)\mathbf W^{p-1}_{q,j}))+\lambda\mathbf W^{p-2}_{j,i} =\sum_{k=1}^ma^{p-2,k}_if^{'}(z^{p-1,k}_j)(\sum_{q=1}^{D}(\delta^{p,k}_q\mathbf W^{p-1}_{q,j}))+\lambda\mathbf W^{p-2}_{j,i}$

更进一步地推导 $W^l$ 和 $W^{l+1}$ 的一阶偏导关系：

\partial J ( W , b ) \partial W l j , i = \sum k = 1 m \partial \partial W l j , i J (W, b; x k, y k) + λ W l j, i = 1 2 \sum k = 1 m \partial \partial W l j , i (\sum q = 1 D (y k q - f (z p, k q)) 2) + λ W l j, i = \sum k = 1 m \sum q = 1 D (- (y k q - f (z p, k q)) f' (z p, k q) \partial z p , k q \partial W l j , i) + λ W l j, i = \sum k = 1 m \sum q = 1 D (δ p, k q \partial z p , k q \partial W l j , i) + λ W l j, i = \sum k = 1 m \sum q = 1 D (δ p, k q \partial z p , k q \partial z l + 1 , k j \partial z l + 1 , k j \partial W l j , i) + λ W l j, i = \sum k = 1 m \sum q = 1 D (δ p, k q \partial z p , k q \partial z l + 1 , k j a l, k i) + λ W l j, i = \sum k = 1 m a l, k i (\sum q = 1 D (δ p, k q \partial z p , k q \partial z l + 1 , k j)) + λ W l j, i

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^l_{j,i}} =\sum_{k=1}^m\frac{\partial}{\partial \mathbf W^l_{j,i}}J(\mathbf W, \mathbf b;\mathbf x^k,y^k)+\lambda\mathbf W^l_{j,i} =\frac{1}{2}\sum_{k=1}^m\frac{\partial}{\partial \mathbf W^l_{j,i}}(\sum_{q=1}^{D}(\mathbf y^k_q-f(z^{p,k}_q))^2)+\lambda\mathbf W^l_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(-(\mathbf y^k_q-f(z^{p,k}_q))f^{'}(z^{p,k}_q)\frac{\partial z^{p,k}_q}{\partial \mathbf W^l_{j,i}})+\lambda\mathbf W^l_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(\delta^{p,k}_{q} \frac{\partial z^{p,k}_q}{\partial \mathbf W^l_{j,i}})+\lambda\mathbf W^l_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(\delta^{p,k}_q\frac{\partial z^{p,k}_q}{\partial z^{l+1,k}_j}\frac{\partial z^{l+1,k}_j}{\partial \mathbf W^l_{j,i}})+\lambda\mathbf W^l_{j,i} =\sum_{k=1}^m\sum_{q=1}^{D}(\delta^{p,k}_q\frac{\partial z^{p,k}_q}{\partial z^{l+1,k}_j}a^{l,k}_i)+\lambda\mathbf W^l_{j,i} =\sum_{k=1}^ma^{l,k}_i(\sum_{q=1}^{D}(\delta^{p,k}_q\frac{\partial z^{p,k}_q}{\partial z^{l+1,k}_j}))+\lambda\mathbf W^l_{j,i}$
当

l=p−2 $l=p-2$ ，结论同上文已经推导的一致。
当

l<p−2 $l<p-2$ ，又有所不同：

\partial J ( W , b ) \partial W l j , i = \sum k = 1 m a l, k i (\sum q = 1 D (δ p, k q \partial z p , k q \partial z l + 1 , k j)) + λ W l j, i = \sum k = 1 m a l, k i (\sum q = 1 D (δ p, k q \sum r = 1 d l + 2 \partial z p , k q \partial z l + 2 , k r)) + λ W l j, i

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^l_{j,i}} =\sum_{k=1}^ma^{l,k}_i(\sum_{q=1}^{D}(\delta^{p,k}_q\frac{\partial z^{p,k}_q}{\partial z^{l+1,k}_j}))+\lambda\mathbf W^l_{j,i} =\sum_{k=1}^ma^{l,k}_i(\sum_{q=1}^{D}(\delta^{p,k}_q \sum_{r=1}^{d^{l+2}} \frac{\partial z^{p,k}_q}{\partial z^{l+2,k}_r}))+\lambda\mathbf W^l_{j,i}$

更进一步地， $\mathbf W^{l}$ 的权重系数一阶偏导如下：

\partial J ( W , b ) \partial W l j , i = \sum k = 1 m δ l + 1, k j a l, k i + λ W l j, i

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf W^l_{j,i}} =\sum_{k=1}^m\delta^{l+1,k}_{j}a^{l,k}_i +\lambda \mathbf W^l_{j,i}$

3.2.2 输出层 $L^p$ 和隐层 $L^{p-1}$ 之间的bias矩阵 $\mathbf b^{p-1}$ 一阶偏导求解并推广到一般情形
先推导一个具体的bias，也就是最后一个隐层的第二个神经元跟输出层的第一个神经元之间的bias，然后再扩展到一般形式：

\partial J ( W , b ) \partial b p - 1 1 = \sum k = 1 m \partial \partial b p - 1 1 J (W, b; x k, y k) = \sum k = 1 m \partial \partial b p - 1 1 (1 2 \sum q = 1 d p (y k q - f (z p, k q)) 2) = \sum k = 1 m \sum q = 1 d p (- (y k q - f (z p, k q)) f' (z k q) \partial z k q \partial b p - 1 1) = \sum k = 1 m (- (y k q - f (z p, k 1)) f' (z k 1) \partial z k 1 \partial b p - 1 1) = \sum k = 1 m δ p, k 1

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf b^{p-1}_1} =\sum_{k=1}^m\frac{\partial}{\partial \mathbf b^{p-1}_1}J(\mathbf W, \mathbf b;\mathbf x^k,y^k) =\sum_{k=1}^m\frac{\partial}{\partial \mathbf b^{p-1}_1}(\frac{1}{2}\sum_{q=1}^{d^p}(\mathbf y^k_q-f(z^{p,k}_q))^2) =\sum_{k=1}^m\sum_{q=1}^{d^p}(-(\mathbf y^k_q-f(z^{p,k}_q))f^{'}(z^k_q)\frac{\partial z^k_q}{\partial \mathbf b^{p-1}_1}) =\sum_{k=1}^m(-(\mathbf y^k_q-f(z^{p,k}_1))f^{'}(z^k_1)\frac{\partial z^k_1}{\partial \mathbf b^{p-1}_1}) =\sum_{k=1}^m\delta^{p,k}_1$

进一步地，输出层和隐层之间的bias矩阵一阶偏导的一般形式是：

\partial J ( W , b ) \partial b p - 1 j = \sum k = 1 m δ p, k j

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf b^{p-1}_j} =\sum_{k=1}^m\delta^{p,k}_j$

更进一步地，所有层的bias矩阵一阶偏导的一般形式是：

\partial J ( W , b ) \partial b l j = \sum k = 1 m δ l + 1, k j

$\frac{\partial J(\mathbf W, \mathbf b)}{\partial \mathbf b^l_j} =\sum_{k=1}^m\delta^{l+1,k}_j$

3.2.3 梯度下降法最终迭代公式

(W l j, i) s + 1 = (W l j, i) s - η (\sum k = 1 m δ l + 1, k j a l, k i + λ W l j, i) ∣ W l j, i = (W l j, i) s

$(\mathbf W^l_{j,i})_{s+1}=(\mathbf W^l_{j,i})_s-\eta (\sum_{k=1}^m\delta^{l+1,k}_{j}a^{l,k}_i +\lambda \mathbf W^l_{j,i})\lvert_{\mathbf W^l_{j,i}=(\mathbf W^l_{j,i})_s}$