【Python实现卷积神经网络】：反向传播推导卷积层对输入数据的求导

0.前言

　　通过之前的学习【Python实现卷积神经网络】：卷积层的正向传播与反向传播+python实现代码，我们知道卷积层的反向传播有三个梯度要求：

1.对输入数据的求导

2.对W的求导

3.对b的求导

---

　　这篇博客推导第一个公式：对输入数据求导。如下公式是怎么来的：

1.【对输入数据求导】计算方法一

　　我在之前的博客中举了正向传播输入数据不带pad，它的反向传播对输入数据求导的例子。

　　这里我们通过举另外一个输入数据带pad的正向卷积，然后反向传播的例子。

　　假设我们现在已经可以递推出上层的梯度误差$δ^{l+1}$了；卷积层输出z，输入a和W,b的关系为：

$$z^l = a^{l}*W^l +b$$
　　因此本层残差（ $δ^{l}$）和上层残差（ $δ^{l+1}$）的递推关系为：

$$伪码： δ^{l}=\frac{\partial J(W,b)}{\partial a^{l}} = \frac{\partial J(W,b)}{\partial z^{l}}\frac{\partial z^{l}}{\partial a^{l}} =\delta^{l+1} *W^{l}$$

　　上边伪码是为了便于推导理解，事实上公式是：

$$δ^{l}=\frac{\partial J(W,b)}{\partial a^{l}} = \frac{\partial J(W,b)}{\partial z^{l}}\frac{\partial z^{l}}{\partial a^{l}} =pad(\delta^{l+1}) \otimes rot180(W^{l})$$

　　假设我们输入a是2x2的矩阵，且加入pad=1卷积核W是3x3的矩阵，输出z是2x2的矩阵，那么反向传播的z的梯度误差δ也是2x2的矩阵。我们列出a,W,z的矩阵表达式如下：
　　

$$\left( \begin{array}{ccc} 0&0&0&0 \\ 0&a_{11}&a_{12}&0\\ 0&a_{21}&a_{22} &0\\ 0&0&0 &0\end{array} \right) \otimes \left( \begin{array}{ccc} w_{11}&w_{12}&w_{13}\\ w_{21}&w_{22}&w_{23}\\ w_{31}&w_{32}&w_{33} \end{array} \right) = \left( \begin{array}{ccc} z_{11}&z_{12}\\ z_{21}&z_{22} \end{array} \right)$$
　　反向传播的 $z$的梯度误差$δ^{l+1}$是：
　　 $$\left( \begin{array}{ccc} δ_{11}&δ_{12}\\ δ_{21}&δ_{22} \end{array} \right)$$
　　利用卷积的定义，很容易得出：
$$z_{11} = 0 + 0 + 0 + 0 + a_{11}w_{22}+ a_{12}w_{23}+ 0 + a_{21}w_{32}+ a_{22}w_{33}\\ z_{12} = 0 + 0 + 0 + a_{11}w_{21} + a_{12}w_{22}+ 0 + a_{21}w_{31} + a_{22}w_{32}+ 0\\ z_{21} = 0 + a_{11}w_{12} + a_{12}w_{13} + 0 + a_{21}w_{22}+ a_{22}w_{23} + 0 + 0 + 0\\ z_{22} = a_{11}w_{11} + a_{12}w_{12} + 0 + a_{21}w_{21} + a_{22}w_{22} + 0 + 0 + 0+ 0$$

　　那么根据上面的式子，我们有：

$$\frac{\partial J(W,b)}{\partial a_{11}^{l}} = w_{22}\delta_{11}+w_{21}\delta_{12}+w_{12}\delta_{21}+w_{11}\delta_{22} \\ \frac{\partial J(W,b)}{\partial a_{12}^{l}} = w_{23}\delta_{11}+w_{22}\delta_{12}+w_{13}\delta_{21}+w_{12}\delta_{22}\\ \frac{\partial J(W,b)}{\partial a_{21}^{l}} = w_{32}\delta_{11}+w_{31}\delta_{12}+w_{22}\delta_{21}+w_{21}\delta_{22}\\ \frac{\partial J(W,b)}{\partial a_{22}^{l}} = w_{33}\delta_{11}+w_{32}\delta_{12}+w_{23}\delta_{21}+w_{22}\delta_{22}$$
　　最终我们可以一共得到4个式子。整理成矩阵形式后可得：
$$\frac{\partial J(W,b)}{\partial a^{l}} =\left( \begin{array}{ccc} 0&0&0&0 \\ 0&\delta_{11}& \delta_{12}&0 \\ 0&\delta_{21}&\delta_{22}&0 \\ 0&0&0&0 \end{array} \right) \otimes \left( \begin{array}{ccc} w_{33}&w_{32}&w_{31}\\ w_{23}&w_{22}&w_{21}\\ w_{13}&w_{12}&w_{11} \end{array} \right)$$
　　从这个例子证明了刚才的公式的正确性：
　　 $$δ^{l}=\frac{\partial J(W,b)}{\partial a^{l}} = \frac{\partial J(W,b)}{\partial z^{l}}\frac{\partial z^{l}}{\partial a^{l}} =pad(\delta^{l+1}) \otimes rot180(W^{l})$$
　

当然，这个仅仅是对输入数据求导的计算公式1，如果我们有别的计算方法能够得出同样的结果，那么我们也可以总结为对输入数据求导的计算公式2。有没有呢？当然有，稍后再表。

1.1.代码

residual_pad = np.pad(residual, ((0,), (0,), (pad_diff_H,), (pad_diff_W,)), mode='constant', constant_values=0)
for i in range(H_out):
    for j in range(W_out):
        residual_pad_masked = residual_pad[:, :, i*stride:i*stride+HH, j*stride:j*stride+WW]        
        for h in range(C):
            dx_2[:, h , i, j] = np.sum(residual_pad_masked[:,:,:,:] * rot_w[:, h, :, :], axis=(1,2,3))

注意：
这里的pad大小是由正向传播卷积核与正向传播pad共同决定的，不是1。我总结的公式是：

$$pad_{diff} = kernel_{size} - (1+pad_{foward})$$
至于这个公式是怎么来的，请读者将上边儿我举的例子中3X3的核变成5X5或者7X7的核，然后推导一边就总结出来了。

2.【对输入数据求导】计算方法二

　　还是上边儿的例子，这次我们换种计算方法，看最终结果和上边最终结果一样不。
　　
　　我们假设：
　　

$$\frac{\partial J(W,b)}{\partial a^{l}} =pool(d)=pool(\left( \begin{array}{ccc} d_{11}&d_{12}&d_{13}&d_{14} \\ d_{21}&d_{22}&d_{23}&d_{24} \\ d_{31}&d_{32}&d_{33}&d_{34}\\ d_{41}&d_{42}&d_{43}&d_{44} \end{array} \right))=\left( \begin{array}{ccc} d_{22}&d_{23}\\d_{32}&d_{33} \end{array} \right)$$

注：这里pool（）池化的意思，在这里表示去掉d的上下左右各pad=1的数字，剩下的部分。
　　

$$d^{11}=\left( \begin{array}{ccc} d_{11}&d_{12}&d_{13} \\ d_{21}&d_{22}&d_{23} \\ d_{31}&d_{32}&d_{33} \end{array} \right)= \left( \begin{array}{ccc} w_{11}&w_{12}&w_{13}\\ w_{21}&w_{22}&w_{23}\\ w_{31}&w_{32}&w_{33} \end{array} \right)*\delta_{11}=\left( \begin{array}{ccc} w_{11}*\delta_{11}&w_{12}*\delta_{11}&w_{13}*\delta_{11}\\ w_{21}*\delta_{11}&w_{22}*\delta_{11}&w_{23}*\delta_{11}\\ w_{31}*\delta_{11}&w_{32}*\delta_{11}&w_{33}*\delta_{11} \end{array} \right)$$

---

$$d^{12}=\left( \begin{array}{ccc} d_{12}&d_{13}&d_{14} \\ d_{22}&d_{23}&d_{24} \\ d_{32}&d_{33}&d_{34} \end{array} \right)= \left( \begin{array}{ccc} w_{11}&w_{12}&w_{13}\\ w_{21}&w_{22}&w_{23}\\ w_{31}&w_{32}&w_{33} \end{array} \right)*\delta_{12}=\left( \begin{array}{ccc} w_{11}*\delta_{12}&w_{12}*\delta_{12}&w_{13}*\delta_{12}\\ w_{21}*\delta_{12}&w_{22}*\delta_{12}&w_{23}*\delta_{12}\\ w_{31}*\delta_{12}&w_{32}*\delta_{12}&w_{33}*\delta_{12} \end{array} \right)$$

---

$$d^{21}=\left( \begin{array}{ccc} d_{21}&d_{22}&d_{23} \\ d_{31}&d_{32}&d_{33} \\ d_{41}&d_{42}&d_{43} \end{array} \right)= \left( \begin{array}{ccc} w_{11}&w_{12}&w_{13}\\ w_{21}&w_{22}&w_{23}\\ w_{31}&w_{32}&w_{33} \end{array} \right)*\delta_{21}=\left( \begin{array}{ccc} w_{11}*\delta_{21}&w_{12}*\delta_{21}&w_{13}*\delta_{21}\\ w_{21}*\delta_{21}&w_{22}*\delta_{21}&w_{23}*\delta_{21}\\ w_{31}*\delta_{21}&w_{32}*\delta_{21}&w_{33}*\delta_{21} \end{array} \right)$$

---

$$d^{22}=\left( \begin{array}{ccc} d_{22}&d_{23}&d_{24} \\ d_{32}&d_{33}&d_{34} \\ d_{42}&d_{43}&d_{44} \end{array} \right)= \left( \begin{array}{ccc} w_{11}&w_{12}&w_{13}\\ w_{21}&w_{22}&w_{23}\\ w_{31}&w_{32}&w_{33} \end{array} \right)*\delta_{22}=\left( \begin{array}{ccc} w_{11}*\delta_{22}&w_{12}*\delta_{22}&w_{13}*\delta_{22}\\ w_{21}*\delta_{22}&w_{22}*\delta_{22}&w_{23}*\delta_{22}\\ w_{31}*\delta_{22}&w_{32}*\delta_{22}&w_{33}*\delta_{22} \end{array} \right)$$

---

然后，将$d^{11},d^{12},d^{21},d^{22}$中相应$d_{i,j}$的位置相加，得到：

$$d_{11}=w_{11}*\delta_{11}$$
$$...$$
$$d_{22}=w_{22}*\delta_{11}+w_{21}*\delta_{12}+w_{12}*\delta_{21}+w_{11}*\delta_{22}$$
$$d_{23}=w_{23}*\delta_{11}+w_{22}*\delta_{12}+w_{13}*\delta_{21}+w_{12}*\delta_{22}$$
$$d_{32}=w_{32}*\delta_{11}+w_{31}*\delta_{12}+w_{22}*\delta_{21}+w_{21}*\delta_{22}$$
$$d_{23}=w_{33}*\delta_{11}+w_{32}*\delta_{12}+w_{23}*\delta_{21}+w_{22}*\delta_{22}$$
$$...$$
$$d_{44}=w_{33}*\delta_{22}$$

可以看出，我们的计算结果与第一个公式一样：

$$\frac{\partial J(W,b)}{\partial a^{l}} =\left( \begin{array}{ccc} \frac{\partial J(W,b)}{\partial a_{11}^{l}}&\frac{\partial J(W,b)}{\partial a_{12}^{l}} \\ \frac{\partial J(W,b)}{\partial a_{21}^{l}}&\frac{\partial J(W,b)}{\partial a_{22}^{l}} \end{array} \right) = \left( \begin{array}{ccc} d_{22}&d_{23}\\d_{32}&d_{33} \end{array} \right)$$

注意：计算方法二不需要rot180（w）

2.1.代码：

      for i in range(H_out):
            for j in range(W_out):
                x_pad_masked = x_pad[:, :, i * stride:i * stride + HH, j * stride:j * stride + WW]
                for k in range(F):  # compute dw
                    dw[k, :, :, :] += np.sum(x_pad_masked * (residual[:, k, i, j])[:, None, None, None], axis=0)  
                    # dw=pad（bottom_data）* top_diff_ij
                for n in range(N):  # compute dx_pad
                    dx_pad[n, :, i * stride:i * stride + HH, j * stride:j * stride + WW] += np.sum((self.w[:, :, :, :] * (residual[n, :, i,j])[:, None, None, None]), axis=0)
                    # dx = (w)* （top_diff_ij）