Backpropagation简称BP，翻译为反向传播，是一种与最优化方法（如梯度下降法）结合使用的训练人工神经网络的方法。该方法会对网络中所有weights计算loss function的梯度，用来更新这些weights以minimize loss function。
h
1
=
i
1
w
1
+
i
2
w
2
+
b
1
h
2
=
i
1
w
3
+
i
2
w
4
+
b
1
o
1
=
h
1
w
5
+
h
2
w
6
+
b
2
o
2
=
h
1
w
7
+
h
2
w
8
+
b
2
h_1=i_1w_1+i_2w_2+b_1\\ h_2=i_1w_3+i_2w_4+b_1\\ o_1=h_1w_5+h_2w_6+b_2\\ o_2=h_1w_7+h_2w_8+b_2\\
h1=i1w1+i2w2+b1h2=i1w3+i2w4+b1o1=h1w5+h2w6+b2o2=h1w7+h2w8+b2
每个神经元后用激活函数sigmoid函数激活，图中未表示出。上述就是前向传播的过程，第一次使用的参数为random initialzation得到，后续计算第一次输出与实际值的总误差L（这里使用均方误差作为损失函数）。
L
=
L
o
s
s
F
u
n
c
t
i
o
n
=
∑
i
=
1
2
1
2
(
t
a
r
g
e
t
o
1
−
o
1
)
2
L
=
1
2
(
t
a
r
g
e
t
o
1
−
o
1
)
2
+
1
2
(
t
a
r
g
e
t
o
2
−
o
2
)
2
L=LossFunction=\sum_{i=1}^2\frac{1}{2}(target_{o_1}-o_1)^2\\ L=\frac{1}{2}(target_{o_1}-o_1)^2+\frac{1}{2}(target_{o_2}-o_2)^2
L=LossFunction=i=1∑221(targeto1−o1)2L=21(targeto1−o1)2+21(targeto2−o2)2
计算出总误差后，我们需要得知每个weight对误差的影响，并对weight进行修正。
∂
L
∂
w
1
=
∂
h
1
∂
w
1
∂
o
1
∂
h
1
∂
L
∂
o
1
+
∂
h
1
∂
w
1
∂
o
2
∂
h
1
∂
L
∂
o
2
∂
L
∂
w
2
=
∂
h
1
∂
w
2
∂
o
1
∂
h
1
∂
L
∂
o
1
+
∂
h
1
∂
w
2
∂
o
2
∂
h
1
∂
L
∂
o
2
∂
L
∂
w
3
=
∂
h
2
∂
w
3
∂
o
1
∂
h
2
∂
L
∂
o
1
+
∂
h
2
∂
w
3
∂
o
2
∂
h
2
∂
L
∂
o
2
∂
L
∂
w
4
=
∂
h
2
∂
w
4
∂
o
1
∂
h
2
∂
L
∂
o
1
+
∂
h
2
∂
w
4
∂
o
2
∂
h
2
∂
L
∂
o
2
∂
L
∂
w
5
=
∂
o
1
∂
w
5
∂
L
∂
o
1
∂
L
∂
w
6
=
∂
o
1
∂
w
6
∂
L
∂
o
1
∂
L
∂
w
7
=
∂
o
2
∂
w
7
∂
L
∂
o
2
∂
L
∂
w
8
=
∂
o
2
∂
w
8
∂
L
∂
o
2
\frac{\partial L}{\partial w_1}=\frac{\partial h_1}{\partial w_1}\frac{\partial o_1}{\partial h_1}\frac{\partial L}{\partial o_1}+\frac{\partial h_1}{\partial w_1}\frac{\partial o_2}{\partial h_1}\frac{\partial L}{\partial o_2}\\ \frac{\partial L}{\partial w_2}=\frac{\partial h_1}{\partial w_2}\frac{\partial o_1}{\partial h_1}\frac{\partial L}{\partial o_1}+\frac{\partial h_1}{\partial w_2}\frac{\partial o_2}{\partial h_1}\frac{\partial L}{\partial o_2}\\ \frac{\partial L}{\partial w_3}=\frac{\partial h_2}{\partial w_3}\frac{\partial o_1}{\partial h_2}\frac{\partial L}{\partial o_1}+\frac{\partial h_2}{\partial w_3}\frac{\partial o_2}{\partial h_2}\frac{\partial L}{\partial o_2}\\ \frac{\partial L}{\partial w_4}=\frac{\partial h_2}{\partial w_4}\frac{\partial o_1}{\partial h_2}\frac{\partial L}{\partial o_1}+\frac{\partial h_2}{\partial w_4}\frac{\partial o_2}{\partial h_2}\frac{\partial L}{\partial o_2}\\ \frac{\partial L}{\partial w_5}=\frac{\partial o_1}{\partial w_5}\frac{\partial L}{\partial o_1}\\ \frac{\partial L}{\partial w_6}=\frac{\partial o_1}{\partial w_6}\frac{\partial L}{\partial o_1}\\ \frac{\partial L}{\partial w_7}=\frac{\partial o_2}{\partial w_7}\frac{\partial L}{\partial o_2}\\ \frac{\partial L}{\partial w_8}=\frac{\partial o_2}{\partial w_8}\frac{\partial L}{\partial o_2}\\
∂w1∂L=∂w1∂h1∂h1∂o1∂o1∂L+∂w1∂h1∂h1∂o2∂o2∂L∂w2∂L=∂w2∂h1∂h1∂o1∂o1∂L+∂w2∂h1∂h1∂o2∂o2∂L∂w3∂L=∂w3∂h2∂h2∂o1∂o1∂L+∂w3∂h2∂h2∂o2∂o2∂L∂w4∂L=∂w4∂h2∂h2∂o1∂o1∂L+∂w4∂h2∂h2∂o2∂o2∂L∂w5∂L=∂w5∂o1∂o1∂L∂w6∂L=∂w6∂o1∂o1∂L∂w7∂L=∂w7∂o2∂o2∂L∂w8∂L=∂w8∂o2∂o2∂L
若每次都通过前向传播计算weight对loss function的影响，则计算复杂度过高，所以采取从后往前计算的反向传播方法。
∂
L
∂
o
1
∂
L
∂
o
2
∂
o
1
∂
h
1
∂
o
1
∂
h
2
∂
o
2
∂
h
1
∂
o
2
∂
h
2
∂
o
1
∂
w
5
∂
o
1
∂
w
6
∂
o
2
∂
w
7
∂
o
2
∂
w
8
∂
h
1
∂
w
1
∂
h
1
∂
w
2
∂
h
2
∂
w
3
∂
h
2
∂
w
4
\frac{\partial L}{\partial o_1}\quad\frac{\partial L}{\partial o_2}\\\\ \frac{\partial o_1}{\partial h_1}\quad\frac{\partial o_1}{\partial h_2}\quad\frac{\partial o_2}{\partial h_1}\quad\frac{\partial o_2}{\partial h_2}\quad\frac{\partial o_1}{\partial w_5}\quad\frac{\partial o_1}{\partial w_6}\quad\frac{\partial o_2}{\partial w_7}\quad\frac{\partial o_2}{\partial w_8}\quad\\\\ \frac{\partial h_1}{\partial w_1}\quad\frac{\partial h_1}{\partial w_2}\quad\frac{\partial h_2}{\partial w_3}\quad\frac{\partial h_2}{\partial w_4}\quad
∂o1∂L∂o2∂L∂h1∂o1∂h2∂o1∂h1∂o2∂h2∂o2∂w5∂o1∂w6∂o1∂w7∂o2∂w8∂o2∂w1∂h1∂w2∂h1∂w3∂h2∂w4∂h2
使用反向传播，只需计算以上的梯度，相比于前向传播计算可以避免很多重复梯度计算，大大降低了计算复杂度。
387
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
