文章目录
回顾
我们设 n [ l ] n^{[l]} n[l] 为第 l l l 层的单元数
则它们的维数
w
[
l
]
,
d
w
:
(
n
[
l
]
,
n
[
l
−
1
]
)
w^{[l]}, dw :(n^{[l]},n^{[l-1]})\\
w[l],dw:(n[l],n[l−1])
b [ l ] , d b : ( n [ l ] , 1 ) b^{[l]}, db :(n^{[l]},1)\\ b[l],db:(n[l],1)
z [ l ] , a l : ( n [ l ] , 1 ) z^{[l]},a^{l}:(n^{[l]},1)\\ z[l],al:(n[l],1)
Z l , A l , d Z , d A : ( n [ l ] , m ) Z^{l},A^{l},dZ,dA:(n^{[l]},m) Zl,Al,dZ,dA:(n[l],m)
反向传播公式为:
d
z
[
l
]
=
d
a
[
l
]
∗
g
[
l
]
′
(
z
[
l
]
)
dz^{[l]} = da^{[l]} * g^{[l]'}(z^{[l]})\\
dz[l]=da[l]∗g[l]′(z[l])
d w [ l ] = d z [ l ] ∗ a [ l − 1 ] dw^{[l]} = dz^{[l]} * a^{[l-1]}\\ dw[l]=dz[l]∗a[l−1]
d b [ l ] = d z [ l ] db^{[l]} = dz^{[l]}\\ db[l]=dz[l]
d a [ l − 1 ] = w [ l ] T ∗ d z [ l ] da^{[l-1]} = w^{[l]^{T}}*dz^{[l]} da[l−1]=w[l]T∗dz[l]
推导
首先我们知道
z
[
l
]
=
w
[
l
]
∗
a
[
l
−
1
]
+
b
[
l
]
z^{[l]} = w^{[l]}*a^{[l-1]} + b^{[l]}
z[l]=w[l]∗a[l−1]+b[l]
a [ l ] = g [ l ] ( z [ l ] ) a^{[l]} = g^{[l]}(z^{[l]}) a[l]=g[l](z[l])
ȷ ( a , y ) = − y l o g a − ( 1 − y ) l o g ( 1 − a ) \jmath(a,y) = - yloga-(1-y)log(1-a)\\ ȷ(a,y)=−yloga−(1−y)log(1−a)
接下来开始推导过程:
d z [ l ] dz^{[l]} dz[l]的证明
由公式 $\jmath(a,y) = - yloga-(1-y)log(1-a)\$ 对
a
a
a 求导得:
d
ȷ
(
a
[
l
]
,
y
)
d
a
[
l
]
=
−
y
a
[
l
]
+
1
−
y
1
−
a
[
l
]
\frac{d\jmath(a^{[l]},y)}{da^{[l]}} = -\frac{y}{a^{[l]}} + \frac{1-y}{1-a^{[l]}}\\
da[l]dȷ(a[l],y)=−a[l]y+1−a[l]1−y
d ȷ ( a [ l ] , y ) d z [ l ] = d ȷ ( a [ l ] , y ) d a [ l ] ∗ d a [ l ] d z [ l ] \frac{d\jmath(a^{[l]},y)}{dz^{[l]}} = \frac{d\jmath(a^{[l]},y)}{da^{[l]}}*\frac{da^{[l]}}{dz^{[l]}} dz[l]dȷ(a[l],y)=da[l]dȷ(a[l],y)∗dz[l]da[l]
而
d
a
[
l
]
d
z
[
l
]
=
g
[
l
]
′
(
z
[
l
]
)
\frac{da^{[l]}}{dz^{[l]}} = g^{[l]'}(z^{[l]})
dz[l]da[l]=g[l]′(z[l])
所以代入公式 中得:
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
=
d
a
[
l
]
∗
g
[
l
]
′
(
z
[
l
]
)
\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} =da^{[l]}*g^{[l]'}(z^{[l]})
dz[l]dȷ(a[l],y)=da[l]∗g[l]′(z[l])
注意:上式子中的
d
a
[
l
]
da^{[l]}
da[l] 为简写,实际上是:
d
ȷ
(
a
[
l
]
,
y
)
d
a
[
l
]
\frac{d\jmath(a^{[l]},y)}{da^{[l]}}
da[l]dȷ(a[l],y)
后面的
d
z
[
l
]
,
d
w
[
l
]
,
d
b
[
l
]
dz^{[l]}, dw^{[l]}, db^{[l]}
dz[l],dw[l],db[l] 同理
即证明了公式 $dz^{[l]} = da^{[l]} * g{[l]’}(z{[l]})\$
d w [ l ] dw^{[l]} dw[l]的证明
由
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
=
d
a
[
l
]
∗
g
[
l
]
′
(
z
[
l
]
)
\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} =da^{[l]}*g^{[l]'}(z^{[l]})
dz[l]dȷ(a[l],y)=da[l]∗g[l]′(z[l])
的证明结果,我们来推一下
d
w
[
l
]
dw^{[l]}
dw[l]
d
ȷ
(
a
[
l
]
,
y
)
d
w
[
l
]
=
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
∗
d
z
[
l
]
d
w
[
l
]
\frac{d\jmath(a^{[l]},y)}{dw^{[l]}} =\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} * \frac{dz^{[l]}}{dw^{[l]}}
dw[l]dȷ(a[l],y)=dz[l]dȷ(a[l],y)∗dw[l]dz[l]
因为:
z
[
l
]
=
w
[
l
]
∗
a
[
l
−
1
]
+
b
[
l
]
z^{[l]} = w^{[l]}*a^{[l-1]} + b^{[l]}\\
z[l]=w[l]∗a[l−1]+b[l]
所以:
d
z
[
l
]
d
w
[
l
]
=
a
[
l
−
1
]
\frac{dz^{[l]}}{dw^{[l]}} = a^{[l-1]}
dw[l]dz[l]=a[l−1]
故:
d
ȷ
(
a
[
l
]
,
y
)
d
w
[
l
]
=
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
∗
d
z
[
l
]
d
w
[
l
]
=
d
z
[
l
]
∗
a
[
l
−
1
]
\frac{d\jmath(a^{[l]},y)}{dw^{[l]}} =\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} * \frac{dz^{[l]}}{dw^{[l]}}=dz^{[l]}*a^{[l-1]}
dw[l]dȷ(a[l],y)=dz[l]dȷ(a[l],y)∗dw[l]dz[l]=dz[l]∗a[l−1]
d b [ l ] db^{[l]} db[l] 的证明
由
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
=
d
a
[
l
]
∗
g
[
l
]
′
(
z
[
l
]
)
\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} =da^{[l]}*g^{[l]'}(z^{[l]})
dz[l]dȷ(a[l],y)=da[l]∗g[l]′(z[l])
的证明结果,我们来推一下
d
b
[
l
]
db^{[l]}
db[l]
d
ȷ
(
a
[
l
]
,
y
)
d
b
[
l
]
=
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
∗
d
z
[
l
]
d
b
[
l
]
\frac{d\jmath(a^{[l]},y)}{db^{[l]}} =\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} * \frac{dz^{[l]}}{db^{[l]}}
db[l]dȷ(a[l],y)=dz[l]dȷ(a[l],y)∗db[l]dz[l]
因为:
z
[
l
]
=
w
[
l
]
∗
a
[
l
−
1
]
+
b
[
l
]
z^{[l]} = w^{[l]}*a^{[l-1]} + b^{[l]}\\
z[l]=w[l]∗a[l−1]+b[l]
所以:
d
z
[
l
]
d
b
[
l
]
=
1
\frac{dz^{[l]}}{db^{[l]}} = 1
db[l]dz[l]=1
故:
d
ȷ
(
a
[
l
]
,
y
)
d
b
[
l
]
=
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
∗
d
z
[
l
]
d
b
[
l
]
=
d
z
[
l
]
\frac{d\jmath(a^{[l]},y)}{db^{[l]}} =\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} * \frac{dz^{[l]}}{db^{[l]}}=dz^{[l]}
db[l]dȷ(a[l],y)=dz[l]dȷ(a[l],y)∗db[l]dz[l]=dz[l]
d a [ l − 1 ] da^{[l-1]} da[l−1] 的证明
由
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
=
d
a
[
l
]
∗
g
[
l
]
′
(
z
[
l
]
)
\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} =da^{[l]}*g^{[l]'}(z^{[l]})
dz[l]dȷ(a[l],y)=da[l]∗g[l]′(z[l])
的证明结果,我们来推一下
d
a
[
l
−
1
]
da^{[l-1]}
da[l−1]
d
ȷ
(
a
[
l
]
,
y
)
d
a
[
l
−
1
]
=
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
∗
d
z
[
l
]
d
a
[
l
−
1
]
\frac{d\jmath(a^{[l]},y)}{da^{[l-1]}} =\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} * \frac{dz^{[l]}}{da^{[l-1]}}
da[l−1]dȷ(a[l],y)=dz[l]dȷ(a[l],y)∗da[l−1]dz[l]
因为:
z
[
l
]
=
w
[
l
]
∗
a
[
l
−
1
]
+
b
[
l
]
z^{[l]} = w^{[l]}*a^{[l-1]} + b^{[l]}\\
z[l]=w[l]∗a[l−1]+b[l]
所以:
d
z
[
l
]
d
a
[
l
−
1
]
=
w
[
l
]
\frac{dz^{[l]}}{da^{[l-1]}} = w^{[l]}
da[l−1]dz[l]=w[l]
故:
d
ȷ
(
a
[
l
]
,
y
)
d
a
[
l
−
1
]
=
d
ȷ
(
a
[
l
]
,
y
)
d
z
[
l
]
∗
d
z
[
l
]
d
a
[
l
−
1
]
=
w
[
l
]
T
∗
d
z
[
l
]
\frac{d\jmath(a^{[l]},y)}{da^{[l-1]}} =\frac{d\jmath(a^{[l]},y)}{dz^{[l]}} * \frac{dz^{[l]}}{da^{[l-1]}}=w^{[l]^{T}}*dz^{[l]}
da[l−1]dȷ(a[l],y)=dz[l]dȷ(a[l],y)∗da[l−1]dz[l]=w[l]T∗dz[l]
会不会觉得很奇怪?
为什么 w [ l ] T w^{[l]^{T}} w[l]T 是个转置?
w [ l ] , d w : ( n [ l ] , n [ l − 1 ] ) w^{[l]}, dw :(n^{[l]},n^{[l-1]})\\ w[l],dw:(n[l],n[l−1])
b [ l ] , d b : ( n [ l ] , 1 ) b^{[l]}, db :(n^{[l]},1)\\ b[l],db:(n[l],1)
z [ l ] , a l : ( n [ l ] , 1 ) z^{[l]},a^{l}:(n^{[l]},1)\\ z[l],al:(n[l],1)
因为我们还要考虑维度的问题,
w [ l ] w^{[l]} w[l] 维度为 ( n [ l ] , n [ l − 1 ] ) (n^{[l]},n^{[l-1]}) (n[l],n[l−1])
z [ l ] z^{[l]} z[l]维度为 ( n [ l ] , 1 ) (n^{[l]},1) (n[l],1)
d a [ l − 1 ] da^{[l-1]} da[l−1] 维度为 ( n [ l ] , 1 ) (n^{[l]},1) (n[l],1)
故要使得 w [ l ] w^{[l]} w[l] 与 z [ l ] z^{[l]} z[l] 的积等于 d a [ l − 1 ] da^{[l-1]} da[l−1] ,我们需要将 w [ l ] w^{[l]} w[l] 转置,转置后的 w [ l ] T w^{[l]^{T}} w[l]T 维度为 ( n [ l ] , n [ l − 1 ] ) (n^{[l]},n^{[l-1]}) (n[l],n[l−1]) ,才可以使得等式成立,且维度一致。