【Python实现卷积神经网络】：推导卷积层的W,b的梯度

0.前言

　　通过之前的学习【Python实现卷积神经网络】：卷积层的正向传播与反向传播+python实现代码，我们知道卷积层的反向传播有三个梯度要求：

1.对输入数据的求导

2.对W的求导

3.对b的求导

1.正确的推导

　　我们看这里篇文章：卷积神经网络(CNN)反向传播算法

　　它对这部分做了推导，我摘录如下：

---

　　我们现在已经可以递推出每一层的梯度误差$δl$了，现在求卷积层对的W,b的梯度。

注意到卷积层输出z，输入a（同时也是上层输出，所以是$(l-1)$层）和W,b的关系为：

$$z^l = a^{l-1}*W^l +b$$
　　因此我们有：
$$\frac{\partial J(W,b)}{\partial W^{l}} = \frac{\partial J(W,b)}{\partial z^{l}}\frac{\partial z^{l}}{\partial W^{l}} =a^{l-1} *\delta^l$$
注意到此时卷积核并没有反转，主要是此时是层内的求导，而不是反向传播到上一层的求导。具体过程我们可以分析一下。

　　一个简化的例子，这里输入是矩阵，不是张量，那么对于第$l$层，某个卷积核矩阵W的导数可以表示如下：

$$\frac{\partial J(W,b)}{\partial W_{pq}^{l}} = \sum\limits_i\sum\limits_j(\delta_{ij}^lx_{i+p-1,j+q-1}^{l-1})$$

　　首先，我们举一个卷积的小例子。假设我们输入a是4x4的矩阵，卷积核W是3x3的矩阵，输出z是2x2的矩阵，那么反向传播的$z$的梯度误差$δ$也是2x2的矩阵。我们列出a,W,z的矩阵表达式如下：

$$\left( \begin{array}{ccc} a_{11}&a_{12}&a_{13}&a_{14} \\ a_{21}&a_{22}&a_{23}&a_{24}\\ a_{31}&a_{32}&a_{33} &a_{34}\\ a_{41}&a_{42}&a_{43} &a_{44}\end{array} \right) \otimes \left( \begin{array}{ccc} w_{11}&w_{12}&w_{13}\\ w_{21}&w_{22}&w_{23}\\ w_{31}&w_{32}&w_{33} \end{array} \right) = \left( \begin{array}{ccc} z_{11}&z_{12}\\ z_{21}&z_{22} \end{array} \right)$$
　　反向传播的 $z$的梯度误差$δ$是：
$$\left( \begin{array}{ccc} δ_{11}&δ_{12}\\ δ_{21}&δ_{22} \end{array} \right)$$
　　利用卷积的定义，很容易得出：
$$z_{11} = a_{11}w_{11} + a_{12}w_{12} + a_{13}w_{13} + a_{21}w_{21} + a_{22}w_{22}+ a_{23}w_{23}+ a_{31}w_{31} + a_{32}w_{32}+ a_{33}w_{33}\\ z_{12} = a_{12}w_{11} + a_{13}w_{12} + a_{14}w_{13} + a_{22}w_{21} + a_{23}w_{22}+ a_{24}w_{23}+ a_{32}w_{31} + a_{33}w_{32}+ a_{34}w_{33}\\ z_{21} = a_{21}w_{11} + a_{22}w_{12} + a_{23}w_{13} + a_{31}w_{21} + a_{32}w_{22}+ a_{33}w_{23}+ a_{41}w_{31} + a_{42}w_{32}+ a_{43}w_{33}\\ z_{22} = a_{22}w_{11} + a_{23}w_{12} + a_{24}w_{13} + a_{32}w_{21} + a_{33}w_{22}+ a_{34}w_{23}+ a_{42}w_{31} + a_{43}w_{32}+ a_{44}w_{33}$$
　　那么根据上面的式子，我们有：
　
$$\frac{\partial J(W,b)}{\partial W_{11}^{l}} = a_{11}\delta_{11} + a_{12}\delta_{12} + a_{21}\delta_{21} + a_{22}\delta_{22}$$
$$\frac{\partial J(W,b)}{\partial W_{12}^{l}} = a_{12}\delta_{11} + a_{13}\delta_{12} + a_{22}\delta_{21} + a_{23}\delta_{22}$$
$$\frac{\partial J(W,b)}{\partial W_{13}^{l}} = a_{13}\delta_{11} + a_{14}\delta_{12} + a_{23}\delta_{21} + a_{24}\delta_{22}$$
$$\frac{\partial J(W,b)}{\partial W_{21}^{l}} = a_{21}\delta_{11} + a_{22}\delta_{12} + a_{31}\delta_{21} + a_{32}\delta_{22}$$

　　最终我们可以一共得到9个式子。整理成矩阵形式后可得：

$$\frac{\partial J(W,b)}{\partial W^{l}} =\left( \begin{array}{ccc} a_{11}&a_{12}&a_{13}&a_{14} \\ a_{21}&a_{22}&a_{23}&a_{24} \\ a_{31}&a_{32}&a_{33}&a_{34} \\ a_{41}&a_{42}&a_{43}&a_{44} \end{array} \right) \otimes \left( \begin{array}{ccc} \delta_{11}& \delta_{12} \\ \delta_{21}&\delta_{22} \end{array} \right)$$

　　从而可以清楚的看到这次我们为什么没有反转的原因。

　　而对于$b$,则稍微有些特殊，因为$δl$是三维张量，而b只是一个向量，不能像DNN那样直接和$δl$相等。通常的做法是将$δl$的各个子矩阵的项分别求和，得到一个误差向量，即为b的梯度：

$$\frac{\partial J(W,b)}{\partial b^{l}} = \sum\limits_{u,v}(\delta^l)_{u,v}$$

---

所以，对w的求导：

1.2.扩展：

上边小例子是正向卷积输入数据没有pad的情况，那么输入数据加pad呢？

我们这里以另外一个小例子为例。假设我们输入a是2x2的矩阵，且加入pad=1卷积核W是3x3的矩阵，输出z是2x2的矩阵，那么反向传播的$z$的梯度误差$δ$也是2x2的矩阵。我们列出a,W,z的矩阵表达式如下：

$$\left( \begin{array}{ccc} 0&0&0&0 \\ 0&a_{11}&a_{12}&0\\ 0&a_{21}&a_{22} &0\\ 0&0&0 &0\end{array} \right) \otimes \left( \begin{array}{ccc} w_{11}&w_{12}&w_{13}\\ w_{21}&w_{22}&w_{23}\\ w_{31}&w_{32}&w_{33} \end{array} \right) = \left( \begin{array}{ccc} z_{11}&z_{12}\\ z_{21}&z_{22} \end{array} \right)$$
反向传播的 $z$的梯度误差$δ$是：
$$\left( \begin{array}{ccc} δ_{11}&δ_{12}\\ δ_{21}&δ_{22} \end{array} \right)$$
　　利用卷积的定义，很容易得出：
$$z_{11} = 0 + 0 + 0 + 0 + a_{11}w_{22}+ a_{12}w_{23}+ 0 + a_{21}w_{32}+ a_{22}w_{33}\\ z_{12} = 0 + 0 + 0 + a_{11}w_{21} + a_{12}w_{22}+ 0 + a_{21}w_{31} + a_{22}w_{32}+ 0\\ z_{21} = 0 + a_{11}w_{12} + a_{12}w_{13} + 0 + a_{21}w_{22}+ a_{22}w_{23} + 0 + 0 + 0\\ z_{22} = a_{11}w_{11} + a_{12}w_{12} + 0 + a_{21}w_{21} + a_{22}w_{22} + 0 + 0 + 0+ 0$$
　　那么根据上面的式子，我们有：
$$\frac{\partial J(W,b)}{\partial W_{11}^{l}} = a_{11}\delta_{22} \\ \frac{\partial J(W,b)}{\partial W_{12}^{l}} = a_{11}\delta_{21} + a_{12}\delta_{22}\\ \frac{\partial J(W,b)}{\partial W_{13}^{l}} = a_{12}\delta_{21}\\ \frac{\partial J(W,b)}{\partial W_{21}^{l}} = a_{11}\delta_{12} + a_{21}\delta_{22}\\ \frac{\partial J(W,b)}{\partial W_{22}^{l}} = a_{11}\delta_{11} + a_{12}\delta_{12} + a_{21}\delta_{21} + a_{22}\delta_{22}$$

　　最终我们可以一共得到9个式子。整理成矩阵形式后可得：

$$\frac{\partial J(W,b)}{\partial W^{l}} =\left( \begin{array}{ccc} 0&0&0&0 \\ 0&a_{11}&a_{12}&0 \\ 0&a_{21}&a_{22}&0 \\ 0&0&0&0 \end{array} \right) \otimes \left( \begin{array}{ccc} \delta_{11}& \delta_{12} \\ \delta_{21}&\delta_{22} \end{array} \right)$$

需要注意的是：
1.卷积左边是输入数据，右边是残差。
2.输入数据做pad。

1.3.扩展二

我们扩展一是举了加pad的例子，那么我们再看批处理的时候这个公式是怎么实现的？

这里有两个方法用python实现：

1.直接按照公式进行卷积

        for m in range(HH):
            for n in range(WW):
                x_pad_masked_d = x_pad[:, :, m * stride:m * stride + H_out, n * stride:n * stride + W_out]
                for k in range(F):
                    for p in range(C):
                        dw[k, p, m, n] = np.sum(x_pad_masked_d[:,p,:,:] * residual[:, k, :, :], axis=(0,1,2)) 

从上边代码可以看出：卷积过程大体与正向传播时的卷积过程一样，有一点需要注意：for p in range（C）这个循环是[k 0 m n]这个位置=sum32[10 14 14]*[10 14 14]求出来的。

2.不卷积的求法：

首先，上代码

        residual_pad = np.pad(residual, ((0,), (0,), (pad,), (pad,)), mode='constant', constant_values=0)
        for i in range(H_out):
            for j in range(W_out):
                x_pad_masked = x_pad[:, :, i * stride:i * stride + HH, j * stride:j * stride + WW]
                for k in range(F):  # compute dw
                    dw[k, :, :, :] += np.sum(x_pad_masked * (residual[:, k, i, j])[:, None, None, None], axis=0)  
                for n in range(N):  # compute dx_pad
                    dx_pad[n, :, i * stride:i * stride + HH, j * stride:j * stride + WW] += np.sum((rot_w[:, :, :, :] * (residual_pad[n, :, i,j])[:, None, None, None]), axis=0)

这里边求dw的代码在：

dw[k, :, :, :] += np.sum(x_pad_masked * (residual[:, k, i, j])[:, None, None, None], axis=0)  

这行代码怎么理解呢？
首先，做简单展开：（注意，这里展开i，j不展开k）

dw[k,:,:,:] = 
np.sum(x_pad_masked * (residual[:, k, i=0, j=0])[:, None, None, None], axis=0)  \\
+np.sum(x_pad_masked * (residual[:, k, i=0, j=1])[:, None, None, None], axis=0)  \\
+np.sum(x_pad_masked * (residual[:, k, i=0, j=2])[:, None, None, None], axis=0)  

我们接着上边儿例子看：（这里x对简单例子中的a ；residual对应简单例子中的$\delta$;i=2,j=2）
按照代码我们的公式为：

$$\frac{\partial J(W,b)}{\partial W^{l}} =（d_{1}+d_{2}+d_{3}+d_{4}）$$
其中：
$$d_{1} =\left( \begin{array}{ccc} 0&0&0 \\ 0&a_{11}&a_{12} \\ 0&a_{21}&a_{22} \end{array} \right) * \delta_{11}=\left( \begin{array}{ccc} 0&0&0 \\ 0&a_{11}* \delta_{11}&a_{12}* \delta_{11} \\ 0&a_{21}* \delta_{11}&a_{22}* \delta_{11} \end{array} \right)$$

$$d_{2} =\left( \begin{array}{ccc} 0&0&0 \\ a_{11}&a_{12}&0 \\ a_{21}&a_{22}&0 \end{array} \right) * \delta_{12}=\left( \begin{array}{ccc} 0&0&0 \\ a_{11}* \delta_{12}&a_{12}* \delta_{12}&0 \\ a_{21}* \delta_{12}&a_{22}* \delta_{12}&0 \end{array} \right)$$

$$d_{3} =\left( \begin{array}{ccc} 0&a_{11}&a_{12} \\ 0&a_{21}&a_{22} \\ 0&0&0 \end{array} \right) * \delta_{21}=\left( \begin{array}{ccc} 0&a_{11}* \delta_{21}&a_{12}* \delta_{21} \\ 0&a_{21}* \delta_{21}&a_{22}* \delta_{21} \\ 0&0&0 \end{array} \right)$$

$$d_{4} =\left( \begin{array}{ccc} a_{11}&a_{12}&0 \\ a_{21}&a_{22}&0 \\ 0&0&0 \end{array} \right) * \delta_{22}=\left( \begin{array}{ccc} a_{11}* \delta_{22}&a_{12}* \delta_{22}&0 \\ a_{21}* \delta_{22}&a_{22}* \delta_{22}&0 \\ 0&0&0 \end{array} \right)$$

那么$d_{1}$$d_{2}$$d_{3}$$d_{4}$对应位置相加结果是：

$$d_{11}=a_{11}\delta_{22} \\ d_{12}=a_{11}\delta_{21} + a_{12}\delta_{22} \\ d_{13}=a_{12}\delta_{21} \\ d_{21}=a_{11}\delta_{12} + a_{21}\delta_{22} \\ d_{22}=a_{11}\delta_{11} + a_{12}\delta_{12} + a_{21}\delta_{21} + a_{22}\delta_{22}$$
这里我们设：
$$\frac{\partial J(W,b)}{\partial W^{l}} =d=\left( \begin{array}{ccc} d_{11}&d_{12}&d_{13} \\ d_{21}&d_{22}&d_{23} \\ d_{31}&d_{32}&d_{33} \end{array} \right)$$

对比之前1.2.扩展推导，说明我们的结果是一样的：

$$\frac{\partial J(W,b)}{\partial W_{11}^{l}} = a_{11}\delta_{22} =d_{11} \\ \frac{\partial J(W,b)}{\partial W_{12}^{l}} = a_{11}\delta_{21} + a_{12}\delta_{22}=d_{12}\\ \frac{\partial J(W,b)}{\partial W_{13}^{l}} = a_{12}\delta_{21}=d_{13}\\ \frac{\partial J(W,b)}{\partial W_{21}^{l}} = a_{11}\delta_{12} + a_{21}\delta_{22}=d_{21}\\ \frac{\partial J(W,b)}{\partial W_{22}^{l}} = a_{11}\delta_{11} + a_{12}\delta_{12} + a_{21}\delta_{21} + a_{22}\delta_{22}=d_{22}$$

---

2.错误的写法：

　　当时，我写2.对w求导时，参考的是这篇文章：CNN卷积神经网络和反向传播，而他的对w求导是需要将卷积层的输入数据做180度翻转，下图是摘自这个博客：

---

---

　　所以，我依照这个说法，做出了如下公式：

---