正向传播
这里用输入为
3
∗
3
3*3
3∗3矩阵
A
l
−
1
A^{l-1}
Al−1,步长为
1
1
1,卷积核为
2
∗
2
2*2
2∗2矩阵
W
l
W^l
Wl,输出为
2
∗
2
2*2
2∗2矩阵
Z
l
Z^l
Zl的卷积层为例。矩阵
A
l
−
1
A^{l-1}
Al−1可以是整个神经网络的输入,也可以是池化层的输出。这个模型简化为输入层
A
l
−
1
A^{l-1}
Al−1经过卷积计算得到特征图
Z
l
Z^l
Zl,
Z
l
Z^l
Zl经过激活函数
σ
(
x
)
\sigma(x)
σ(x)得到输出层
A
l
A^l
Al(实际上在现实工程中很多时候不用激活函数)。对于第
l
l
l层,有下列表达式:
[
a
1
a
2
a
3
a
4
a
5
a
6
a
7
a
8
a
9
]
l
−
1
⇒
[
ω
1
ω
2
ω
3
ω
4
]
l
⇒
[
z
1
z
2
z
3
z
4
]
l
⇒
[
a
1
a
2
a
3
a
4
]
l
(1)
\left[ \begin{array}{r} a_1&a_2&a_3\\ a_4&a_5&a_6\\ a_7&a_8&a_9\\ \end{array} \right]^{l-1} \Rightarrow \left[ \begin{array}{r} \omega_1&\omega_2\\ \omega_3&\omega_4\\ \end{array} \right]^l \Rightarrow \left[ \begin{array}{r} z_1&z_2\\ z_3&z_4\\ \end{array} \right]^l \Rightarrow \left[ \begin{array}{r} a_1&a_2\\ a_3&a_4\\ \end{array} \right]^l\tag{1}
⎣⎡a1a4a7a2a5a8a3a6a9⎦⎤l−1⇒[ω1ω3ω2ω4]l⇒[z1z3z2z4]l⇒[a1a3a2a4]l(1)
{
a
1
l
=
∑
(
[
a
1
a
2
a
4
a
5
]
l
−
1
⋅
[
ω
1
ω
2
ω
3
ω
4
]
l
)
=
σ
(
ω
1
a
1
l
−
1
+
ω
2
a
2
l
−
1
+
ω
3
a
3
l
−
1
+
ω
4
a
4
l
−
1
+
b
l
)
=
σ
(
z
1
l
)
a
2
l
=
∑
(
[
a
2
a
3
a
5
a
6
]
l
−
1
⋅
[
ω
1
ω
2
ω
3
ω
4
]
l
)
=
σ
(
ω
1
a
2
l
−
1
+
ω
2
a
3
l
−
1
+
ω
3
a
5
l
−
1
+
ω
4
a
6
l
−
1
+
b
l
)
=
σ
(
z
2
l
)
a
3
l
=
∑
(
[
a
4
a
5
a
7
a
8
]
l
−
1
⋅
[
ω
1
ω
2
ω
3
ω
4
]
l
)
=
σ
(
ω
1
a
4
l
−
1
+
ω
2
a
5
l
−
1
+
ω
3
a
7
l
−
1
+
ω
4
a
8
l
−
1
+
b
l
)
=
σ
(
z
3
l
)
a
4
l
=
∑
(
[
a
5
a
6
a
8
a
9
]
l
−
1
⋅
[
ω
1
ω
2
ω
3
ω
4
]
l
)
=
σ
(
ω
1
a
5
l
−
1
+
ω
2
a
6
l
−
1
+
ω
3
a
8
l
−
1
+
ω
4
a
9
l
−
1
+
b
l
)
=
σ
(
z
4
l
)
(2)
\left\{ \begin{array}{r} a_1^l=\sum( \left[ \begin{array}{r} a_1&a_2\\ a_4&a_5\\ \end{array} \right]^{l-1} \cdot \left[ \begin{array}{r} \omega_1&\omega_2\\ \omega_3&\omega_4\\ \end{array} \right]^l)=\sigma(\omega_1a_1^{l-1}+\omega_2a_2^{l-1}+\omega_3a_3^{l-1}+\omega_4a_4^{l-1}+b^l)=\sigma(z_1^l) \\ a_2^l=\sum( \left[ \begin{array}{r} a_2&a_3\\ a_5&a_6\\ \end{array} \right]^{l-1} \cdot \left[ \begin{array}{r} \omega_1&\omega_2\\ \omega_3&\omega_4\\ \end{array} \right]^l)=\sigma(\omega_1a_2^{l-1}+\omega_2a_3^{l-1}+\omega_3a_5^{l-1}+\omega_4a_6^{l-1}+b^l)=\sigma(z_2^l) \\ a_3^l=\sum( \left[ \begin{array}{r} a_4&a_5\\ a_7&a_8\\ \end{array} \right]^{l-1} \cdot \left[ \begin{array}{r} \omega_1&\omega_2\\ \omega_3&\omega_4\\ \end{array} \right]^l)=\sigma(\omega_1a_4^{l-1}+\omega_2a_5^{l-1}+\omega_3a_7^{l-1}+\omega_4a_8^{l-1}+b^l)=\sigma(z_3^l) \\ a_4^l=\sum( \left[ \begin{array}{r} a_5&a_6\\ a_8&a_9\\ \end{array} \right]^{l-1} \cdot \left[ \begin{array}{r} \omega_1&\omega_2\\ \omega_3&\omega_4\\ \end{array} \right]^l)=\sigma(\omega_1a_5^{l-1}+\omega_2a_6^{l-1}+\omega_3a_8^{l-1}+\omega_4a_9^{l-1}+b^l)=\sigma(z_4^l) \end{array} \right.\tag{2}
⎩⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎧a1l=∑([a1a4a2a5]l−1⋅[ω1ω3ω2ω4]l)=σ(ω1a1l−1+ω2a2l−1+ω3a3l−1+ω4a4l−1+bl)=σ(z1l)a2l=∑([a2a5a3a6]l−1⋅[ω1ω3ω2ω4]l)=σ(ω1a2l−1+ω2a3l−1+ω3a5l−1+ω4a6l−1+bl)=σ(z2l)a3l=∑([a4a7a5a8]l−1⋅[ω1ω3ω2ω4]l)=σ(ω1a4l−1+ω2a5l−1+ω3a7l−1+ω4a8l−1+bl)=σ(z3l)a4l=∑([a5a8a6a9]l−1⋅[ω1ω3ω2ω4]l)=σ(ω1a5l−1+ω2a6l−1+ω3a8l−1+ω4a9l−1+bl)=σ(z4l)(2)
简单来说,卷积过程就是对应的位置代入函数之后相加求和,不同的函数有不同的参数
w
w
w和
b
b
b,我们需要训练的是卷积核参数,所以这个公式还可以写做
Z
l
=
W
l
∗
A
l
−
1
+
b
l
Z^l=W^l*A^{l-1}+b^l
Zl=Wl∗Al−1+bl,
σ
(
x
)
\sigma(x)
σ(x)是激活函数,我们假设是ReLU函数,求导比较好求,所以我们接下来的计算忽略了对激活层的求导。
σ
(
x
)
=
{
0
,
x
<
0
x
,
x
>
=
0
(3)
\sigma(x)=\left\{ \begin{array}{r} 0&,&x<0\\ x&,&x>=0 \end{array} \right.\tag{3}
σ(x)={0x,,x<0x>=0(3)
反向传播
假设矩阵
Z
L
Z^L
ZL是最后一层(第
L
L
L层),那么
Z
L
Z^L
ZL会被展开为
1
∗
4
1*4
1∗4的矩阵之后通过分类器进行分类,这个分类器一般用全连接层替代(DNN),DNN的反向传播很简单,不是本文的重点。通过全连接层的误差函数
J
(
W
,
b
,
x
,
y
)
J(W,b,x,y)
J(W,b,x,y)来计算标签与输出结果的误差(
x
x
x是输出,
y
y
y是标签),根据误差我们可以计算出相对应的梯度,之后通过全连接层的反向传播将梯度
Δ
L
\Delta^L
ΔL传到
Z
L
Z^L
ZL矩阵这一。
Δ
L
=
[
δ
1
δ
2
δ
3
δ
4
]
=
∂
J
(
W
L
,
b
L
,
x
,
y
)
∂
Z
L
(4)
\Delta^L= \left[ \begin{array}{r} \delta_1&\delta_2\\ \delta_3&\delta_4 \end{array} \right] =\frac{\partial J(W^L,b^L,x,y)}{\partial Z^L}\tag{4}
ΔL=[δ1δ3δ2δ4]=∂ZL∂J(WL,bL,x,y)(4)
对于倒数第二层(第
l
−
1
l-1
l−1层),对应于前文的正向传播公式
Z
l
=
Z
L
,
l
=
L
Z^l=Z^L,l=L
Zl=ZL,l=L,接下来要做的是将梯度传给输入层(也就是计算出输入层的梯度),计算输入层
Z
l
−
1
Z^{l-1}
Zl−1的梯度。
Δ
l
−
1
=
∂
J
(
W
l
,
b
l
,
x
,
y
)
∂
A
l
−
1
=
(
∂
Z
L
∂
A
l
−
1
)
T
∂
J
(
W
l
,
b
l
,
x
,
y
)
∂
Z
L
=
(
∂
Z
l
∂
A
l
−
1
)
T
∂
J
(
W
l
,
b
l
,
x
,
y
)
∂
Z
l
(5)
\Delta^{l-1}=\frac{\partial J(W^l,b^l,x,y)}{\partial A^{l-1}}=(\frac{\partial Z^L}{\partial A^{l-1}})^T\frac{\partial J(W^l,b^l,x,y)}{\partial Z^L}=(\frac{\partial Z^l}{\partial A^{l-1}})^T\frac{\partial J(W^l,b^l,x,y)}{\partial Z^l}\tag{5}
Δl−1=∂Al−1∂J(Wl,bl,x,y)=(∂Al−1∂ZL)T∂ZL∂J(Wl,bl,x,y)=(∂Al−1∂Zl)T∂Zl∂J(Wl,bl,x,y)(5)
将
(
4
)
(4)
(4)代入
(
5
)
(5)
(5)中
Δ
l
−
1
=
(
∂
Z
l
∂
A
l
−
1
)
T
Δ
L
(6)
\Delta^{l-1}=(\frac{\partial Z^l}{\partial A^{l-1}})^T\Delta^L\tag{6}
Δl−1=(∂Al−1∂Zl)TΔL(6)
因为卷积层的输入和输出不是最终计算结果,而是下一层传递的误差,所以卷积层误差函数是
J
(
W
,
b
)
J(W,b)
J(W,b),将这个公式扩展到每一个卷积层,可得
Δ
l
−
1
=
(
∂
Z
l
∂
A
l
−
1
)
T
∂
J
(
W
l
,
b
l
)
∂
Z
l
=
(
∂
Z
l
∂
A
l
−
1
)
T
Δ
l
(7)
\Delta^{l-1}=(\frac{\partial Z^l}{\partial A^{l-1}})^T\frac{\partial J(W^l,b^l)}{\partial Z^l}=(\frac{\partial Z^l}{\partial A^{l-1}})^T\Delta^l \tag{7}
Δl−1=(∂Al−1∂Zl)T∂Zl∂J(Wl,bl)=(∂Al−1∂Zl)TΔl(7)
这样就得到了反向传播的公式,这个公式唯一不确定的是
∂
Z
l
∂
A
l
−
1
\frac{\partial Z^l}{\partial A^{l-1}}
∂Al−1∂Zl。
我们接下来的重点放在如何计算
∂
Z
l
∂
A
l
−
1
\frac{\partial Z^l}{\partial A^{l-1}}
∂Al−1∂Zl,从正向传播中我们已知
Z
l
=
W
l
∗
A
l
−
1
+
b
l
(8)
Z^l=W^l*A^{l-1}+b^l\tag{8}
Zl=Wl∗Al−1+bl(8)
{
z
1
l
=
ω
1
a
1
l
−
1
+
ω
2
a
2
l
−
1
+
ω
3
a
3
l
−
1
+
ω
4
a
4
l
−
1
+
b
l
z
2
l
=
ω
1
a
2
l
−
1
+
ω
2
a
3
l
−
1
+
ω
3
a
5
l
−
1
+
ω
4
a
6
l
−
1
+
b
l
z
3
l
=
ω
1
a
4
l
−
1
+
ω
2
a
5
l
−
1
+
ω
3
a
7
l
−
1
+
ω
4
a
8
l
−
1
+
b
l
z
4
l
=
ω
1
a
5
l
−
1
+
ω
2
a
6
l
−
1
+
ω
3
a
8
l
−
1
+
ω
4
a
9
l
−
1
+
b
l
(9)
\left\{ \begin{array}{r} z_1^l=\omega_1a_1^{l-1}+\omega_2a_2^{l-1}+\omega_3a_3^{l-1}+\omega_4a_4^{l-1}+b^l\\ z_2^l=\omega_1a_2^{l-1}+\omega_2a_3^{l-1}+\omega_3a_5^{l-1}+\omega_4a_6^{l-1}+b^l\\ z_3^l=\omega_1a_4^{l-1}+\omega_2a_5^{l-1}+\omega_3a_7^{l-1}+\omega_4a_8^{l-1}+b^l\\ z_4^l=\omega_1a_5^{l-1}+\omega_2a_6^{l-1}+\omega_3a_8^{l-1}+\omega_4a_9^{l-1}+b^l \end{array} \right.\tag{9}
⎩⎪⎪⎨⎪⎪⎧z1l=ω1a1l−1+ω2a2l−1+ω3a3l−1+ω4a4l−1+blz2l=ω1a2l−1+ω2a3l−1+ω3a5l−1+ω4a6l−1+blz3l=ω1a4l−1+ω2a5l−1+ω3a7l−1+ω4a8l−1+blz4l=ω1a5l−1+ω2a6l−1+ω3a8l−1+ω4a9l−1+bl(9)
设第
l
−
1
l-1
l−1层的梯度为
Δ
l
−
1
=
[
δ
1
δ
2
δ
3
δ
4
δ
5
δ
6
δ
7
δ
8
δ
9
]
l
−
1
(10)
\Delta^{l-1}= \left[ \begin{array}{r} \delta_1&\delta_2&\delta_3\\ \delta_4&\delta_5&\delta_6\\ \delta_7&\delta_8&\delta_9\\ \end{array} \right]^{l-1}\tag{10}
Δl−1=⎣⎡δ1δ4δ7δ2δ5δ8δ3δ6δ9⎦⎤l−1(10)
对于
(
9
)
(9)
(9),
z
l
z^l
zl对
a
l
−
1
a^{l-1}
al−1逐一求偏导,那么可得到16个公式
{
∂
z
1
l
∂
a
1
l
−
1
=
ω
1
∂
z
1
l
∂
a
2
l
−
1
=
ω
2
.
.
.
(11)
\left\{ \begin{array}{r} \frac{\partial z^l_1}{\partial a^{l-1}_1}&=&\omega_1\\ \frac{\partial z^l_1}{\partial a^{l-1}_2}&=&\omega_2\\ ... \end{array} \right.\tag{11}
⎩⎪⎪⎨⎪⎪⎧∂a1l−1∂z1l∂a2l−1∂z1l...==ω1ω2(11)
将
(
11
)
(11)
(11)代入
(
7
)
(7)
(7)和
(
10
)
(10)
(10)中,得到9个公式
{
δ
1
l
−
1
=
∂
z
1
l
∂
a
1
l
−
1
δ
1
l
=
ω
1
δ
1
l
δ
2
l
−
1
=
∂
z
1
l
∂
a
2
l
−
1
δ
1
l
+
∂
z
2
l
∂
a
2
l
−
1
δ
2
l
=
ω
2
δ
1
l
+
ω
1
δ
2
l
.
.
.
(12)
\left\{ \begin{array}{r} \delta_1^{l-1}&=&\frac{\partial z^l_1}{\partial a^{l-1}_1}\delta_1^l&=&\omega_1\delta_1^l\\ \delta_2^{l-1}&=&\frac{\partial z^l_1}{\partial a^{l-1}_2}\delta_1^l+\frac{\partial z^l_2}{\partial a^{l-1}_2}\delta_2^l&=&\omega_2\delta_1^l+\omega_1\delta_2^l\\ ... \end{array} \right.\tag{12}
⎩⎪⎪⎨⎪⎪⎧δ1l−1δ2l−1...==∂a1l−1∂z1lδ1l∂a2l−1∂z1lδ1l+∂a2l−1∂z2lδ2l==ω1δ1lω2δ1l+ω1δ2l(12)
观察公式,其实就是同一层的下标相对应,比如求
l
−
1
l-1
l−1层的
δ
2
\delta_2
δ2,那么分母也是
l
−
1
l-1
l−1层,下标就都是
2
2
2,分母是
l
l
l层的
z
z
z,那么相乘的
l
l
l层
δ
\delta
δ下标也和
z
z
z相同。
列出所有公式后,可以发现一个规律(读者可以自己尝试列出剩余的公式),其中
r
o
t
90
rot90
rot90是将矩阵逆时针旋转90度。
Δ
l
−
1
=
[
0
0
0
0
0
δ
1
δ
2
0
0
δ
3
δ
4
0
0
0
0
0
]
l
∗
r
o
t
90
(
[
ω
1
ω
2
ω
3
ω
4
]
l
)
=
[
0
0
0
0
0
δ
1
δ
2
0
0
δ
3
δ
4
0
0
0
0
0
]
l
∗
[
ω
4
ω
3
ω
2
ω
1
]
l
(13)
\Delta^{l-1}= \left[ \begin{array}{r} 0&0&0&0\\ 0&\delta_1&\delta_2&0\\ 0&\delta_3&\delta_4&0\\ 0&0&0&0\\ \end{array} \right]^l* rot90( \left[ \begin{array}{r} \omega_1&\omega_2\\ \omega_3&\omega_4\\ \end{array} \right]^l) =\left[ \begin{array}{r} 0&0&0&0\\ 0&\delta_1&\delta_2&0\\ 0&\delta_3&\delta_4&0\\ 0&0&0&0\\ \end{array} \right]^l*\left[ \begin{array}{r} \omega_4&\omega_3\\ \omega_2&\omega_1\\ \end{array} \right]^l\tag{13}
Δl−1=⎣⎢⎢⎡00000δ1δ300δ2δ400000⎦⎥⎥⎤l∗rot90([ω1ω3ω2ω4]l)=⎣⎢⎢⎡00000δ1δ300δ2δ400000⎦⎥⎥⎤l∗[ω4ω2ω3ω1]l(13)
反向传播的误差传递结束。
参数更新
更新 W l W^l Wl
根据反向传播可知
Z
l
=
W
l
A
l
−
1
+
b
l
(14)
Z^l=W^lA^{l-1}+b^l\tag{14}
Zl=WlAl−1+bl(14)
上式对两边求
W
l
W^l
Wl的偏导可得
∂
Z
l
∂
W
l
=
A
l
−
1
(15)
\frac{\partial Z^l}{\partial W^l}=A^{l-1}\tag{15}
∂Wl∂Zl=Al−1(15)
用误差对
W
l
W^l
Wl求偏导来计算
W
l
W^l
Wl的梯度
∂
J
(
W
l
,
b
l
)
∂
W
l
=
∂
J
(
W
l
,
b
l
)
∂
Z
l
∂
Z
l
∂
W
l
=
Δ
l
A
l
−
1
(16)
\frac{\partial J(W^l,b^l)}{\partial W^l}=\frac{\partial J(W^l,b^l)}{\partial Z^l}\frac{\partial Z^l}{\partial W^l}=\Delta^{l}A^{l-1}\tag{16}
∂Wl∂J(Wl,bl)=∂Zl∂J(Wl,bl)∂Wl∂Zl=ΔlAl−1(16)
W
l
W^l
Wl的更新就为学习率
L
R
LR
LR与梯度相乘
W
n
e
w
l
=
W
o
l
d
l
−
Δ
l
A
l
−
1
L
R
(17)
W_{new}^l=W^l_{old}-\Delta^{l}A^{l-1}LR\tag{17}
Wnewl=Woldl−ΔlAl−1LR(17)
更新 b l b^l bl
而对于
b
l
b^l
bl,则稍微有些特殊,因为
Δ
l
\Delta^l
Δl是高维张量(矩阵),而
b
l
b^l
bl只是一个向量(一个数字),不能像DNN那样直接和
Δ
l
\Delta^l
Δl相等。通常的做法是将
Δ
l
\Delta^l
Δl的各个子矩阵的项分别求和,得到一个误差向量,即为
b
b
b的梯度:
∂
J
(
W
l
,
b
l
)
∂
b
l
=
∑
Δ
l
(18)
\frac{\partial J(W^l,b^l)}{\partial b^l}=\sum\Delta^{l}\tag{18}
∂bl∂J(Wl,bl)=∑Δl(18)
b
n
e
w
l
=
b
o
l
d
l
−
L
R
∑
Δ
l
(19)
b_{new}^l=b_{old}^l-LR\sum\Delta^{l}\tag{19}
bnewl=boldl−LR∑Δl(19)