卷积神经网络误差反向传播公式-卷积层

本文给出卷积神经网络中卷积层的误差反向传播公式.

卷积层包括卷积与激活运算.一般卷积神经网络的卷积层包含数个输入通道与输出通道,每一层卷积层包含若干卷积核,所以本文先给出在单输入单输出的情况下的公式,然后给出多输入多输出情况下的公式.关于卷积运算,本文针对valid与full两种模式进行讨论.

一、单输入通道单输出通道公式

在单输入单输出情况下,卷积核$K^l$,输入$O^{l-1}$输出$ne{t}^l$均为二维矩阵.

1.$ne{t}^l=K^l{\ast }_{valid}{O}^{l-1}+B^l$

卷积核$K^l\in {\mathbb{R}}^{n\times n},$
输入$O^{l-1}\in {\mathbb{R}}^{m\times m},$
输出$ne{t}^l\in {\mathbb{R}}^{m-n+1\times m-n+1}.$

1.1. $\frac{\partial e}{\partial K^l}$与$\frac{\partial e}{\partial B^l}$

$$损失函数:$$$$e=g\left(ne{t}^l\right)$$$$第l层净输入:$$$$ne{t}^l=K^l{\ast }_{valid}{O}^{l-1}+B^l$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot dne{t}^l\right)$$$$净输入的微分:$$$$dne{t}^l=d{K}^l{\ast }_{valid}{O}^{l-1}+d{B}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{valid}{O}^{l-1}+d{B}^l\right)\right)$$$$=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{valid}{O}^{l-1}\right)+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$$$=tr\left(\left({\frac{\partial e}{\partial ne{t}^l}}^T{\ast }_{valid}{\left(O^{l-1}\right)}_{rot}^T\right)\cdot d{K}^l+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$

$$损失函数对l层卷积核K^l的偏导:$$$$\frac{\partial e}{\partial K^l}={\left({\frac{\partial e}{\partial ne{t}^l}}^T{\ast }_{valid}{\left(O^{l-1}\right)}_{rot}^T\right)}^T$$$$=\frac{\partial e}{\partial ne{t}^l}{\ast }_{valid}{O}_{rot}^{l-1}$$$$损失函数对l层偏移B^l的偏导:$$$$\frac{\partial e}{\partial B^l}=\frac{\partial e}{\partial ne{t}^l}$$

1.2. $\frac{\partial e}{\partial ne{t}^l}$递推公式

$$损失函数:$$$$e=g(ne{t}^{l+1})$$$$第l+1层净输入:$$$$ne{t}^{l+1}=K^{l+1}{\ast }_{valid}{O}^l+B^{l+1}$$$$第l层净输出$$$$O^l=f(ne{t}^l)$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot dne{t}^{l+1}\right)$$$$净输入的微分:$$$$dne{t}^{l+1}=K^{l+1}{\ast }_{valid}d{O}^l$$$$净输出的微分:$$$$d{O}^l=f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot \left(K^{l+1}{\ast }_{valid}\left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)\right)$$$$=tr\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_{rot}^T\right)\cdot \left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)$$$$=tr\left(\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_{rot}^T\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)\cdot dne{t}^l\right)$$

$$\frac{\partial e}{\partial ne{t}^l}={\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_{rot}^T\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)}^T$$$$={\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_{rot}^T\right)}^T\odot f^{{}^{\prime }}(ne{t}^l)$$$$=\left(\frac{\partial e}{\partial ne{t}^{l+1}}{\ast }_{full}{K}_{rot}^{l+1}\right)\odot f^{{}^{\prime }}(ne{t}^l)$$

2.$ne{t}^l=K^l{\ast }_{full}{O}^{l-1}+B^l$

卷积核$K^l\in {\mathbb{R}}^{n\times n},$
输入$O^{l-1}\in {\mathbb{R}}^{m\times m},$
输出$ne{t}^l\in {\mathbb{R}}^{m+n-1\times m+n-1}.$

2.1. $\frac{\partial e}{\partial K^l}$与$\frac{\partial e}{\partial B^l}$

$$损失函数:$$$$e=g\left(ne{t}^l\right)$$$$第l层净输入:$$$$ne{t}^l=K^l{\ast }_{full}{O}^{l-1}+B^l$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot dne{t}^l\right)$$$$净输入的微分:$$$$dne{t}^l=d{K}^l{\ast }_{full}{O}^{l-1}+d{B}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{full}{O}^{l-1}+d{B}^l\right)\right)$$$$=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{full}{O}^{l-1}\right)+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$$$=tr\left(\left({\frac{\partial e}{\partial ne{t}^l}}^T{\ast }_{valid}{\left(O^{l-1}\right)}_{rot}^T\right)\cdot d{K}^l+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$

$$损失函数对l层卷积核K^l的偏导:$$$$\frac{\partial e}{\partial K^l}={\left({\frac{\partial e}{\partial ne{t}^l}}^T{\ast }_{valid}{\left(O^{l-1}\right)}_{rot}^T\right)}^T$$$$=\frac{\partial e}{\partial ne{t}^l}{\ast }_{valid}{O}_{rot}^{l-1}$$$$损失函数对l层偏移B^l的偏导:$$$$\frac{\partial e}{\partial B^l}=\frac{\partial e}{\partial ne{t}^l}$$

2.2. $\frac{\partial e}{\partial ne{t}^l}$递推公式

$$损失函数:$$$$e=g(ne{t}^{l+1})$$$$第l+1层净输入:$$$$ne{t}^{l+1}=K^{l+1}{\ast }_{full}{O}^l+B^{l+1}$$$$第l层净输出$$$$O^l=f(ne{t}^l)$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot dne{t}^{l+1}\right)$$$$净输入的微分:$$$$dne{t}^{l+1}=K^{l+1}{\ast }_{full}d{O}^l$$$$净输出的微分:$$$$d{O}^l=f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot \left(K^{l+1}{\ast }_{full}\left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)\right)$$$$=tr\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_{rot}^T\right)\cdot \left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)$$$$=tr\left(\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_{rot}^T\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)\cdot dne{t}^l\right)$$

$$\frac{\partial e}{\partial ne{t}^l}={\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_{rot}^T\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)}^T$$$$={\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_{rot}^T\right)}^T\odot f^{{}^{\prime }}(ne{t}^l)$$$$=\left(\frac{\partial e}{\partial ne{t}^{l+1}}{\ast }_{valid}{K}_{rot}^{l+1}\right)\odot f^{{}^{\prime }}(ne{t}^l)$$

二、多输入通道多输出通道公式

在多输入多输出情况下,卷积核$K^l$,输入$O^{l-1}$,输出$ne{t}^l$均为四维矩阵.

四维矩阵$A$可以理解为元素类型为二维矩阵的二维矩阵.
$A_r^t$运算的意义为对矩阵$A$中每个元素进行旋转与转置操作.
$A^T$运算的意义为对矩阵$A$进行转置操作，及对矩阵$A$和$A$中每个元素进行转置.

1.$ne{t}^l=K^l{\ast }_{valid}{O}^{l-1}+B^l$

1.1. $\frac{\partial e}{\partial K^l}$与$\frac{\partial e}{\partial B^l}$

卷积核$K^l\in {\mathbb{R}}^{m\times p\times i\times i},$
输入$O^{l-1}\in {\mathbb{R}}^{p\times n\times j\times j},$
输出$ne{t}^l\in {\mathbb{R}}^{m\times n\times j-i+1\times j-i+1},$
输入有n批,每批包含p个通道,每个通道的大小为$\text{j*j},$
输出有n批,每批包含m个通道,每个通道的大小为$\text{j-i+1*j-i+1},$
卷积核有输出有$\text{m*p}$个, 每个卷积核的大小为$\text{i*i}.$

$$损失函数:$$$$e=g\left(ne{t}^l\right)$$$$第l层净输入:$$$$ne{t}^l=K^l{\ast }_{valid}{O}^{l-1}+B^l$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot dne{t}^l\right)$$$$净输入的微分:$$$$dne{t}^l=d{K}^l{\ast }_{valid}{O}^{l-1}+d{B}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{valid}{O}^{l-1}+d{B}^l\right)\right)$$$$=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{valid}{O}^{l-1}\right)+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$$$=tr\left(\left({\left(O^{l-1}\right)}_r^t{\ast }_{valid}{\frac{\partial e}{\partial ne{t}^l}}^T\right)\cdot d{K}^l+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$

$$损失函数对l层卷积核K^l的偏导:$$$$\frac{\partial e}{\partial K^l}={\left({\left(O^{l-1}\right)}_r^t{\ast }_{valid}{\frac{\partial e}{\partial ne{t}^l}}^T\right)}^T$$$$=\frac{\partial e}{\partial ne{t}^l}{\ast }_{valid}{\left({\left(O^{l-1}\right)}_r^t\right)}^T$$$$损失函数对l层偏移B^l的偏导:$$$$\frac{\partial e}{\partial B^l}=\frac{\partial e}{\partial ne{t}^l}$$

1.2. $\frac{\partial e}{\partial ne{t}^l}$递推公式

$$损失函数:$$$$e=g(ne{t}^{l+1})$$$$第l+1层净输入:$$$$ne{t}^{l+1}=K^{l+1}{\ast }_{valid}{O}^l+B^{l+1}$$$$第l层净输出$$$$O^l=f(ne{t}^l)$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot dne{t}^{l+1}\right)$$$$净输入的微分:$$$$dne{t}^{l+1}=K^{l+1}{\ast }_{valid}d{O}^l$$$$净输出的微分:$$$$d{O}^l=f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot \left(K^{l+1}{\ast }_{valid}\left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)\right)$$$$=tr\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_r^t\right)\cdot \left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)$$$$=tr\left(\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_r^t\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)\cdot dne{t}^l\right)$$

$$\frac{\partial e}{\partial ne{t}^l}={\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_r^t\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)}^T$$$$={\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{full}{\left(K^{l+1}\right)}_r^t\right)}^T\odot f^{{}^{\prime }}(ne{t}^l)$$$$=\left({\left({\left(K^{l+1}\right)}_r^t\right)}^T{\ast }_{full}\frac{\partial e}{\partial ne{t}^{l+1}}\right)\odot f^{{}^{\prime }}(ne{t}^l)$$

2.$ne{t}^l=K^l{\ast }_{full}{O}^{l-1}+B^l$

卷积核$K^l\in {\mathbb{R}}^{m\times p\times i\times i},$
输入$O^{l-1}\in {\mathbb{R}}^{p\times n\times j\times j},$
输出$ne{t}^l\in {\mathbb{R}}^{m\times n\times j+i-1\times j+i-1},$
输入有n批,每批包含p个通道,每个通道的大小为$\text{j*j},$
输出有n批,每批包含m个通道,每个通道的大小为$\text{j+i-1*j+i-1},$
卷积核有输出有$\text{m*p}$个, 每个卷积核的大小为$\text{i*i}.$

2.1. $\frac{\partial e}{\partial K^l}$与$\frac{\partial e}{\partial B^l}$

$$损失函数:$$$$e=g\left(ne{t}^l\right)$$$$第l层净输入:$$$$ne{t}^l=K^l{\ast }_{full}{O}^{l-1}+B^l$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot dne{t}^l\right)$$$$净输入的微分:$$$$dne{t}^l=d{K}^l{\ast }_{full}{O}^{l-1}+d{B}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{full}{O}^{l-1}+d{B}^l\right)\right)$$$$=tr\left({\frac{\partial e}{\partial ne{t}^l}}^T\cdot \left(d{K}^l{\ast }_{full}{O}^{l-1}\right)+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$$$=tr\left(\left({\left(O^{l-1}\right)}_r^t{\ast }_{valid}{\frac{\partial e}{\partial ne{t}^l}}^T\right)\cdot d{K}^l+{\frac{\partial e}{\partial ne{t}^l}}^T\cdot d{B}^l\right)$$

$$损失函数对l层卷积核K^l的偏导:$$$$\frac{\partial e}{\partial K^l}={\left({\left(O^{l-1}\right)}_r^t{\ast }_{valid}{\frac{\partial e}{\partial ne{t}^l}}^T\right)}^T$$$$=\frac{\partial e}{\partial ne{t}^l}{\ast }_{valid}{\left({\left(O^{l-1}\right)}_r^t\right)}^T$$$$损失函数对l层偏移B^l的偏导:$$$$\frac{\partial e}{\partial B^l}=\frac{\partial e}{\partial ne{t}^l}$$

2.2. $\frac{\partial e}{\partial ne{t}^l}$递推公式

$$损失函数:$$$$e=g(ne{t}^{l+1})$$$$第l+1层净输入:$$$$ne{t}^{l+1}=K^{l+1}{\ast }_{full}{O}^l+B^{l+1}$$$$第l层净输出$$$$O^l=f(ne{t}^l)$$$$损失函数的微分:$$$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot dne{t}^{l+1}\right)$$$$净输入的微分:$$$$dne{t}^{l+1}=K^{l+1}{\ast }_{full}d{O}^l$$$$净输出的微分:$$$$d{O}^l=f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l$$

$$de=tr\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T\cdot \left(K^{l+1}{\ast }_{full}\left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)\right)$$$$=tr\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_r^t\right)\cdot \left(f^{{}^{\prime }}(ne{t}^l)\odot dne{t}^l\right)\right)$$$$=tr\left(\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_r^t\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)\cdot dne{t}^l\right)$$

$$\frac{\partial e}{\partial ne{t}^l}={\left(\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_r^t\right)\odot {\left(f^{{}^{\prime }}(ne{t}^l)\right)}^T\right)}^T$$$$={\left({\frac{\partial e}{\partial ne{t}^{l+1}}}^T{\ast }_{valid}{\left(K^{l+1}\right)}_r^t\right)}^T\odot f^{{}^{\prime }}(ne{t}^l)$$$$=\left({\left({\left(K^{l+1}\right)}_r^t\right)}^T{\ast }_{valid}\frac{\partial e}{\partial ne{t}^{l+1}}\right)\odot f^{{}^{\prime }}(ne{t}^l)$$