说到神经网络，大家看到这个图应该不陌生：
这是典型的三层神经网络的基本构成，Layer L1是输入层，Layer L2是隐含层，Layer L3是隐含层，我们现在手里有一堆数据{x1,x2,x3,…,xn},输出也是一堆数据{y1,y2,y3,…,yn},现在要他们在隐含层做某种变换，让你把数据灌进去后得到你期望的输出。如果你希望你的输出和原始输入一样，那么就是最常见的自编码模型（Auto-Encoder）。可能有人会问，为什么要输入输出都一样呢？有什么用啊？其实应用挺广的，在图像识别，文本分类等等都会用到，我会专门再写一篇Auto-Encoder的文章来说明，包括一些变种之类的。如果你的输出和原始输入不一样，那么就是很常见的人工神经网络了，相当于让原始数据通过一个映射来得到我们想要的输出数据，也就是我们今天要讲的话题。
本文直接举一个例子，带入数值演示反向传播法的过程（注：本文假设你已经懂得基本的神经网络构成，如果完全不懂，可以参考Poll写的笔记：[Mechine Learning & Algorithm] 神经网络基础）
假设，你有这样一个网络层：
第一层是输入层，包含两个神经元i1，i2，和截距项b1；第二层是隐含层，包含两个神经元h1,h2和截距项b2，第三层是输出o1,o2，每条线上标的wi是层与层之间连接的权重，激活函数我们默认为sigmoid函数。
现在对他们赋上初值，如下图：
其中，输入数据 i1=0.05，i2=0.10;
输出数据 o1=0.01,o2=0.99;
初始权重 w1=0.15,w2=0.20,w3=0.25,w4=0.30;
w5=0.40,w6=0.45,w7=0.50,w8=0.55
目标：给出输入数据i1,i2(0.05和0.10)，使输出尽可能与原始输出o1,o2(0.01和0.99)接近。
step1 前向传播
1.输入层---->隐含层：
计算神经元h1的输入加权和：
n e t h 1 = w 1 ∗ i 1 + w 2 ∗ i 2 + b 1 ∗ 1 n e t h 1 = 0.15 ∗ 0.05 + 0.2 ∗ 0.1 + 0.35 ∗ 1 = 0.3775 \begin{aligned}net_{h1}&=w_1*i_1+w_2*i_2+b_1*1\\\\net_{h1}&=0.15*0.05+0.2*0.1+0.35*1=0.3775\end{aligned} neth1neth1=w1∗i1+w2∗i2+b1∗1=0.15∗0.05+0.2∗0.1+0.35∗1=0.3775
神经元h1的输出outh1:(此处用到激活函数为sigmoid函数)：
o u t h 1 = 1 1 + e − n e t h 1 = 1 1 + e − 0.3775 = 0.593269992 out_{h1}=\frac1{1+e^{-net_{h1}}}=\frac1{1+e^{-0.3775}}=0.593269992 outh1=1+e−neth11=1+e−0.37751=0.593269992
同理，可计算出神经元h2的输出outh2：
o u t h 2 = 0.596884378 out_{h2}=0.596884378 outh2=0.596884378
2.隐含层---->输出层：
计算输出层神经元o1和o2的值：
n e t o 1 = w 5 ∗ o u t h 1 + w 6 ∗ o u t h 2 + b 2 ∗ 1 o u t o 1 = 1 1 + e − n e t o 1 = 1 1 + e − 1.105905967 = 0.75136507 \begin{aligned} &net_{o1} =w_{5}*out_{h1}+w_{6}*out_{h2}+b_{2}*1 \\ &out_{o1} =\frac1{1+e^{-net_o1}}=\frac1{1+e^{-1.105905967}}=0.75136507 \end{aligned} neto1=w5∗outh1+w6∗outh2+b2∗1outo1=1+e−neto11=1+e−1.1059059671=0.75136507
o u t o 2 = 0.772928465 out_{o2}=0.772928465 outo2=0.772928465
这样前向传播的过程就结束了，我们得到输出值为[0.75136079 , 0.772928465]，与实际值[0.01 , 0.99]相差还很远，现在我们对误差进行反向传播，更新权值，重新计算输出。
Step 2 反向传播
1.计算总误差
总误差：(square error)
E t o t a l = ∑ 1 2 ( t a r g e t − o u t p u t ) 2 E_{total}=\sum\frac12(target-output)^2 Etotal=∑21(target−output)2
但是有两个输出，所以分别计算o1和o2的误差，总误差为两者之和：
E o 1 = 1 2 ( t a r g e t o 1 − o u t o 1 ) 2 = 1 2 ( 0.01 − 0.75136507 ) 2 = 0.274811083 E_{o1}=\frac12(target_{o1}-out_{o1})^2=\frac12(0.01-0.75136507)^2=0.274811083 Eo1=21(targeto1−outo1)2=21(0.01−0.75136507)2=0.274811083
E o 2 = 0.023560026 E_{o2}=0.023560026 Eo2=0.023560026
E t o t a l = E o 1 + E o 2 = 0.274811083 + 0.023560026 = 0.298371109 E_{total}=E_{o1}+E_{o2}=0.274811083+0.023560026=0.298371109 Etotal=Eo1+Eo2=0.274811083+0.023560026=0.298371109
2.隐含层---->输出层的权值更新：
以权重参数w5为例，如果我们想知道w5对整体误差产生了多少影响，可以用整体误差对w5求偏导求出：（链式法则）
∂ E t o t a l ∂ w 5 = ∂ E t o t a l ∂ o u t o 1 ∗ ∂ o u t o 1 ∂ n e t o 1 ∗ ∂ n e t o 1 ∂ w 5 \frac{\partial E_{total}}{\partial w_{5}}=\frac{\partial E_{total}}{\partial out_{o1}}*\frac{\partial out_{o1}}{\partial net_{o1}}*\frac{\partial net_{o1}}{\partial w_{5}} ∂w5∂Etotal=∂outo1∂Etotal∗∂neto1∂outo1∗∂w5∂neto1
下面的图可以更直观的看清楚误差是怎样反向传播的：
现在我们来分别计算每个式子的值：
E t o t a l = 1 2 ( t a r g e t o 1 − o u t o 1 ) 2 + 1 2 ( t a r g e t o 2 − o u t o 2 ) 2 ∂ E t o t a l ∂ o u t o 1 = 2 ∗ 1 2 ( t a r g e t o 1 − o u t o 1 ) 2 − 1 ∗ − 1 + 0 ∂ E t o t a l ∂ o u t o 1 = − ( t a r g e t o 1 − o u t o 1 ) = − ( 0.01 − 0.75136507 ) = 0.74136507 \begin{gathered} \begin{aligned}E_{total}=\frac{1}{2}(target_{o1}-out_{o1})^2+\frac{1}{2}(target_{o2}-out_{o2})^2\end{aligned} \\ \frac{\partial E_{total}}{\partial out_{o1}}=2*\frac{1}{2}(target_{o1}-out_{o1})^{2-1}*-1+0 \\ \frac{\partial E_{total}}{\partial out_{o1}}=-(target_{o1}-out_{o1})=-(0.01-0.75136507)=0.74136507 \end{gathered} Etotal=21(targeto1−outo1)2+21(targeto2−outo2)2∂outo1∂Etotal=2∗21(targeto1−outo1)2−1∗−1+0∂outo1∂Etotal=−(targeto1−outo1)=−(0.01−0.75136507)=0.74136507
o u t o 1 = 1 1 + e o 1 − n e t ∂ o u t o 1 ∂ n e t o 1 = o u t o 1 ( 1 − o u t o 1 ) = 0.75136507 ( 1 − 0.75136507 ) = 0.186815602 \begin{aligned}out_{o1}&=\frac{1}{1+e^{-net}_{o1}}\\\\\frac{\partial out_{o1}}{\partial net_{o1}}&=out_{o1}(1-out_{o1})=0.75136507(1-0.75136507)=0.186815602\end{aligned} outo1∂neto1∂outo1=1+eo1−net1=outo1(1−outo1)=0.75136507(1−0.75136507)=0.186815602
（这一步实际上就是对sigmoid函数求导，比较简单，可以自己推导一下）
n e t o 1 = w 5 ∗ o u t h 1 + w 6 ∗ o u t h 2 + b 2 ∗ 1 ∂ n e t o 1 ∂ w 5 = 1 ∗ o u t h 1 ∗ w 5 ( 1 − 1 ) + 0 + 0 = o u t h 1 = 0.593269992 \begin{aligned}net_{o1}&=w_5*out_{h1}+w_6*out_{h2}+b_2*1\\\\\frac{\partial net_{o1}}{\partial w_5}&=1*out_{h1}*w_5^{(1-1)}+0+0=out_{h1}=0.593269992\end{aligned} neto1∂w5∂neto1=w5∗outh1+w6∗outh2+b2∗1=1∗outh1∗w5(1−1)+0+0=outh1=0.593269992
最后三者相乘：
∂ E t o t a l ∂ w 5 = ∂ E t o t a l ∂ o u t o 1 ∗ ∂ o u t o 1 ∂ n e t o 1 ∗ ∂ n e t o 1 ∂ w 5 ∂ E t o t a l ∂ w 5 = 0.74136507 ∗ 0.186815602 ∗ 0.593269992 = 0.082167041 \begin{aligned}\frac{\partial E_{total}}{\partial w_5}&=\frac{\partial E_{total}}{\partial out_{o1}}*\frac{\partial out_{o1}}{\partial net_{o1}}*\frac{\partial net_{o1}}{\partial w_5}\\\\\frac{\partial E_{total}}{\partial w_5}&=0.74136507*0.186815602*0.593269992=0.082167041\end{aligned} ∂w5∂Etotal∂w5∂Etotal=∂outo1∂Etotal∗∂neto1∂outo1∗∂w5∂neto1=0.74136507∗0.186815602∗0.593269992=0.082167041
这样我们就计算出整体误差E(total)对w5的偏导值。
回过头来再看看上面的公式，我们发现： ∂ E t o t a l ∂ w 5 = − ( t a r g e t o 1 − o u t o 1 ) ∗ o u t o 1 ( 1 − o u t o 1 ) ∗ o u t h 1 \frac{\partial E_{total}}{\partial w_5}=-(target_{o1}-out_{o1})*out_{o1}(1-out_{o1})*out_{h1} ∂w5∂Etotal=−(targeto1−outo1)∗outo1(1−outo1)∗outh1
为了表达方便，用 δ o 1 \delta_{o1} δo1来表示输出层的误差：
δ o 1 = ∂ E t o t a l ∂ o u t o 1 ∗ ∂ o u t o 1 ∂ n e t o 1 = ∂ E t o t a l ∂ n e t o 1 δ o 1 = − ( t a r g e t o 1 − o u t o 1 ) ∗ o u t o 1 ( 1 − o u t o 1 ) \begin{aligned} &\delta_{o1} =\frac{\partial E_{total}}{\partial out_{o1}}*\frac{\partial out_{o1}}{\partial net_{o1}}=\frac{\partial E_{total}}{\partial net_{o1}} \\ &\delta_{o1} =-(target_{o1}-out_{o1})*out_{o1}(1-out_{o1}) \end{aligned} δo1=∂outo1∂Etotal∗∂neto1∂outo1=∂neto1∂Etotalδo1=−(targeto1−outo1)∗outo1(1−outo1)
因此，整体误差E(total)对w5的偏导公式可以写成： ∂ E t o t a l ∂ w 5 = δ o 1 o u t h 1 \frac{\partial E_{total}}{\partial w_5}=\delta_{o1}out_{h1} ∂w5∂Etotal=δo1outh1
如果输出层误差计为负的话，也可以写成： ∂ E t o t a l ∂ w 5 = − δ o 1 o u t h 1 \frac{\partial E_{total}}{\partial w_5}=-\delta_{o1}out_{h1} ∂w5∂Etotal=−δo1outh1
最后我们来更新w5的值： w 5 + = w 5 − η ∗ ∂ E t o t a l ∂ w 5 = 0.4 − 0.5 ∗ 0.082167041 = 0.35891648 w_5^+=w_5-\eta*\frac{\partial E_{total}}{\partial w_5}=0.4-0.5*0.082167041=0.35891648 w5+=w5−η∗∂w5∂Etotal=0.4−0.5∗0.082167041=0.35891648
（其中， η \eta η是学习速率，这里我们取0.5)
同理，可更新w6,w7,w8:
w 6 + = 0.408666186 w 7 + = 0.511301270 w 8 + = 0.561370121 \begin{aligned}w_6^+&=0.408666186\\\\w_7^+&=0.511301270\\\\w_8^+&=0.561370121\end{aligned} w6+w7+w8+=0.408666186=0.511301270=0.561370121
####3.隐含层---->隐含层的权值更新：
方法其实与上面说的差不多，但是有个地方需要变一下，在上文计算总误差对w5的偏导时，是从out(o1)---->net(o1)---->w5,但是在隐含层之间的权值更新时，是out(h1)---->net(h1)---->w1,而out(h1)会接受E(o1)和E(o2)两个地方传来的误差，所以这个地方两个都要计算。
∂ E t o t a l ∂ o u t h 1 = ∂ E o 1 ∂ o u t h 1 + ∂ E o 2 ∂ o u t h 1 \frac{\partial E_{total}}{\partial out_{h1}}=\frac{\partial E_{o1}}{\partial out_{h1}}+\frac{\partial E_{o2}}{\partial out_{h1}} ∂outh1∂Etotal=∂outh1∂Eo1+∂outh1∂Eo2
∂ E o 1 ∂ o u t h 1 = ∂ E o 1 ∂ n e t o 1 ∗ ∂ n e t o 1 ∂ o u t h 1 \frac{\partial E_{o1}}{\partial out_{h1}}=\frac{\partial E_{o1}}{\partial net_{o1}}*\frac{\partial net_{o1}}{\partial out_{h1}} ∂outh1∂Eo1=∂neto1∂Eo1∗∂outh1∂neto1
∂ E o 1 ∂ n e t o 1 = ∂ E o 1 ∂ o u t o 1 ∗ ∂ o u t o 1 ∂ n e t o 1 = 0.74136507 ∗ 0.186815602 = 0.138498562 \frac{\partial E_{o1}}{\partial net_{o1}}=\frac{\partial E_{o1}}{\partial out_{o1}}*\frac{\partial out_{o1}}{\partial net_{o1}}=0.74136507*0.186815602=0.138498562 ∂neto1∂Eo1=∂outo1∂Eo1∗∂neto1∂outo1=0.74136507∗0.186815602=0.138498562
n e t o 1 = w 5 ∗ o u t h 1 + w 6 ∗ o u t h 2 + b 2 ∗ 1 ∂ n e t o 1 ∂ o u t h 1 = w 5 = 0.40 \begin{aligned}net_{o1}&=w_5*out_{h1}+w_6*out_{h2}+b_2*1\\\\\frac{\partial net_{o1}}{\partial out_{h1}}&=w_5=0.40\end{aligned} neto1∂outh1∂neto1=w5∗outh1+w6∗outh2+b2∗1=w5=0.40
∂ E o 1 ∂ o u t h 1 = ∂ E o 1 ∂ n e t o 1 ∗ ∂ n e t o 1 ∂ o u t h 1 = 0.138498562 ∗ 0.40 = 0.055399425 \frac{\partial E_{o1}}{\partial out_{h1}}=\frac{\partial E_{o1}}{\partial net_{o1}}*\frac{\partial net_{o1}}{\partial out_{h1}}=0.138498562*0.40=0.055399425 ∂outh1∂Eo1=∂neto1∂Eo1∗∂outh1∂neto1=0.138498562∗0.40=0.055399425
同理，计算出：
∂ E o 2 ∂ o u t h 1 = − 0.019049119 \frac{\partial E_{o2}}{\partial out_{h1}}=-0.019049119 ∂outh1∂Eo2=−0.019049119
∂ E t o t a l ∂ o u t h 1 = ∂ E o 1 ∂ o u t h 1 + ∂ E o 2 ∂ o u t h 1 = 0.055399425 + − 0.019049119 = 0.036350306 \frac{\partial E_{total}}{\partial out_{h1}}=\frac{\partial E_{o1}}{\partial out_{h1}}+\frac{\partial E_{o2}}{\partial out_{h1}}=0.055399425+-0.019049119=0.036350306 ∂outh1∂Etotal=∂outh1∂Eo1+∂outh1∂Eo2=0.055399425+−0.019049119=0.036350306
在计算 ∂ o u t h 1 ∂ n e t h 1 \frac{\partial out_{h1}}{\partial net_{h1}} ∂neth1∂outh1
o u t h 1 = 1 1 + e − n e t h 1 ∂ o u t h 1 ∂ n e t h 1 = o u t h 1 ( 1 − o u t h 1 ) = 0.59326999 ( 1 − 0.59326999 ) = 0.241300709 \begin{aligned}out_{h1}&=\frac{1}{1+e^{-net}h1}\\\\\frac{\partial out_{h1}}{\partial net_{h1}}&=out_{h1}(1-out_{h1})=0.59326999(1-0.59326999)=0.241300709\end{aligned} outh1∂neth1∂outh1=1+e−neth11=outh1(1−outh1)=0.59326999(1−0.59326999)=0.241300709
n e t h 1 = w 1 ∗ i 1 + w 2 ∗ i 2 + b 1 ∗ 1 ∂ n e t h 1 ∂ w 1 = i 1 = 0.05 \begin{aligned}net_{h1}&=w_1*i_1+w_2*i_2+b_1*1\\\\\frac{\partial net_{h1}}{\partial w_1}&=i_1=0.05\end{aligned} neth1∂w1∂neth1=w1∗i1+w2∗i2+b1∗1=i1=0.05
最后，三者相乘：
∂ E t o t a l ∂ w 1 = ∂ E t o t a l ∂ o u t h 1 ∗ ∂ o u t h 1 ∂ n e t h 1 ∗ ∂ n e t h 1 ∂ w 1 ∂ E t o t a l ∂ w 1 = 0.036350306 ∗ 0.241300709 ∗ 0.05 = 0.000438568 \begin{aligned}\frac{\partial E_{total}}{\partial w_1}&=\frac{\partial E_{total}}{\partial out_{h1}}*\frac{\partial out_{h1}}{\partial net_{h1}}*\frac{\partial net_h1}{\partial w_1}\\\\\frac{\partial E_{total}}{\partial w_1}&=0.036350306*0.241300709*0.05=0.000438568\end{aligned} ∂w1∂Etotal∂w1∂Etotal=∂outh1∂Etotal∗∂neth1∂outh1∗∂w1∂neth1=0.036350306∗0.241300709∗0.05=0.000438568
为了简化公式，用sigma(h1)表示隐含层单元h1的误差：
∂ E t o t a l ∂ w 1 = ( ∑ o ∂ E t o t a l ∂ o u t o ∗ ∂ o u t o ∂ n e t o ∗ ∂ n e t o ∂ o u t h 1 ) ∗ ∂ o u t h 1 ∂ n e t h 1 ∗ ∂ n e t h 1 ∂ w 1 ∂ E t o t a l ∂ w 1 = ( ∑ o δ o ∗ w h o ) ∗ o u t h 1 ( 1 − o u t h 1 ) ∗ i 1 ∂ E t o t a l ∂ w 1 = δ h 1 i 1 \begin{aligned} &\frac{\partial E_{total}}{\partial w_{1}} =\left(\sum_o\frac{\partial E_{total}}{\partial out_o}*\frac{\partial out_o}{\partial net_o}*\frac{\partial net_o}{\partial out_{h1}}\right)*\frac{\partial out_{h1}}{\partial net_{h1}}*\frac{\partial net_{h1}}{\partial w_1} \\ &\frac{\partial E_{total}}{\partial w_{1}} \begin{aligned}=(\sum_o\delta_o*w_{ho})*out_{h1}(1-out_{h1})*i_1\end{aligned} \\ &\frac{\partial E_{total}}{\partial w_{1}} =\delta_{h1}i_{1} \end{aligned} ∂w1∂Etotal=(o∑∂outo∂Etotal∗∂neto∂outo∗∂outh1∂neto)∗∂neth1∂outh1∗∂w1∂neth1∂w1∂Etotal=(o∑δo∗who)∗outh1(1−outh1)∗i1∂w1∂Etotal=δh1i1
最后，更新w1的权值：
w 1 + = w 1 − η ∗ ∂ E t o t a l ∂ w 1 = 0.15 − 0.5 ∗ 0.000438568 = 0.149780716 w_1^+=w_1-\eta*\frac{\partial E_{total}}{\partial w_1}=0.15-0.5*0.000438568=0.149780716 w1+=w1−η∗∂w1∂Etotal=0.15−0.5∗0.000438568=0.149780716
同理，额可更新w2,w3,w4的权值：
w 2 + = 0.19956143 w 3 + = 0.24975114 w 4 + = 0.29950229 \begin{aligned}w_2^+&=0.19956143\\\\w_3^+&=0.24975114\\\\w_4^+&=0.29950229\end{aligned} w2+w3+w4+=0.19956143=0.24975114=0.29950229
这样误差反向传播法就完成了，最后我们再把更新的权值重新计算，不停地迭代，在这个例子中第一次迭代之后，总误差E(total)由0.298371109下降至0.291027924。迭代10000次后，总误差为0.000035085，输出为0.015912196,0.984065734,证明效果还是不错的。
附代码：
#coding:utf-8
import random
import math
#
# 参数解释：
# "pd_" ：偏导的前缀
# "d_" ：导数的前缀
# "w_ho" ：隐含层到输出层的权重系数索引
# "w_ih" ：输入层到隐含层的权重系数的索引
class NeuralNetwork:
LEARNING_RATE = 0.5
def __init__(self, num_inputs, num_hidden, num_outputs, hidden_layer_weights = None, hidden_layer_bias = None, output_layer_weights = None, output_layer_bias = None):
self.num_inputs = num_inputs
self.hidden_layer = NeuronLayer(num_hidden, hidden_layer_bias)
self.output_layer = NeuronLayer(num_outputs, output_layer_bias)
self.init_weights_from_inputs_to_hidden_layer_neurons(hidden_layer_weights)
self.init_weights_from_hidden_layer_neurons_to_output_layer_neurons(output_layer_weights)
def init_weights_from_inputs_to_hidden_layer_neurons(self, hidden_layer_weights):
weight_num = 0
for h in range(len(self.hidden_layer.neurons)):
for i in range(self.num_inputs):
if not hidden_layer_weights:
self.hidden_layer.neurons[h].weights.append(random.random())
else:
self.hidden_layer.neurons[h].weights.append(hidden_layer_weights[weight_num])
weight_num += 1
def init_weights_from_hidden_layer_neurons_to_output_layer_neurons(self, output_layer_weights):
weight_num = 0
for o in range(len(self.output_layer.neurons)):
for h in range(len(self.hidden_layer.neurons)):
if not output_layer_weights:
self.output_layer.neurons[o].weights.append(random.random())
else:
self.output_layer.neurons[o].weights.append(output_layer_weights[weight_num])
weight_num += 1
def inspect(self):
print('------')
print('* Inputs: {}'.format(self.num_inputs))
print('------')
print('Hidden Layer')
self.hidden_layer.inspect()
print('------')
print('* Output Layer')
self.output_layer.inspect()
print('------')
def feed_forward(self, inputs):
hidden_layer_outputs = self.hidden_layer.feed_forward(inputs)
return self.output_layer.feed_forward(hidden_layer_outputs)
def train(self, training_inputs, training_outputs):
self.feed_forward(training_inputs)
# 1. 输出神经元的值
pd_errors_wrt_output_neuron_total_net_input = [0] * len(self.output_layer.neurons)
for o in range(len(self.output_layer.neurons)):
# ∂E/∂zⱼ
pd_errors_wrt_output_neuron_total_net_input[o] = self.output_layer.neurons[o].calculate_pd_error_wrt_total_net_input(training_outputs[o])
# 2. 隐含层神经元的值
pd_errors_wrt_hidden_neuron_total_net_input = [0] * len(self.hidden_layer.neurons)
for h in range(len(self.hidden_layer.neurons)):
# dE/dyⱼ = Σ ∂E/∂zⱼ * ∂z/∂yⱼ = Σ ∂E/∂zⱼ * wᵢⱼ
d_error_wrt_hidden_neuron_output = 0
for o in range(len(self.output_layer.neurons)):
d_error_wrt_hidden_neuron_output += pd_errors_wrt_output_neuron_total_net_input[o] * self.output_layer.neurons[o].weights[h]
# ∂E/∂zⱼ = dE/dyⱼ * ∂zⱼ/∂
pd_errors_wrt_hidden_neuron_total_net_input[h] = d_error_wrt_hidden_neuron_output * self.hidden_layer.neurons[h].calculate_pd_total_net_input_wrt_input()
# 3. 更新输出层权重系数
for o in range(len(self.output_layer.neurons)):
for w_ho in range(len(self.output_layer.neurons[o].weights)):
# ∂Eⱼ/∂wᵢⱼ = ∂E/∂zⱼ * ∂zⱼ/∂wᵢⱼ
pd_error_wrt_weight = pd_errors_wrt_output_neuron_total_net_input[o] * self.output_layer.neurons[o].calculate_pd_total_net_input_wrt_weight(w_ho)
# Δw = α * ∂Eⱼ/∂wᵢ
self.output_layer.neurons[o].weights[w_ho] -= self.LEARNING_RATE * pd_error_wrt_weight
# 4. 更新隐含层的权重系数
for h in range(len(self.hidden_layer.neurons)):
for w_ih in range(len(self.hidden_layer.neurons[h].weights)):
# ∂Eⱼ/∂wᵢ = ∂E/∂zⱼ * ∂zⱼ/∂wᵢ
pd_error_wrt_weight = pd_errors_wrt_hidden_neuron_total_net_input[h] * self.hidden_layer.neurons[h].calculate_pd_total_net_input_wrt_weight(w_ih)
# Δw = α * ∂Eⱼ/∂wᵢ
self.hidden_layer.neurons[h].weights[w_ih] -= self.LEARNING_RATE * pd_error_wrt_weight
def calculate_total_error(self, training_sets):
total_error = 0
for t in range(len(training_sets)):
training_inputs, training_outputs = training_sets[t]
self.feed_forward(training_inputs)
for o in range(len(training_outputs)):
total_error += self.output_layer.neurons[o].calculate_error(training_outputs[o])
return total_error
class NeuronLayer:
def __init__(self, num_neurons, bias):
# 同一层的神经元共享一个截距项b
self.bias = bias if bias else random.random()
self.neurons = []
for i in range(num_neurons):
self.neurons.append(Neuron(self.bias))
def inspect(self):
print('Neurons:', len(self.neurons))
for n in range(len(self.neurons)):
print(' Neuron', n)
for w in range(len(self.neurons[n].weights)):
print(' Weight:', self.neurons[n].weights[w])
print(' Bias:', self.bias)
def feed_forward(self, inputs):
outputs = []
for neuron in self.neurons:
outputs.append(neuron.calculate_output(inputs))
return outputs
def get_outputs(self):
outputs = []
for neuron in self.neurons:
outputs.append(neuron.output)
return outputs
class Neuron:
def __init__(self, bias):
self.bias = bias
self.weights = []
def calculate_output(self, inputs):
self.inputs = inputs
self.output = self.squash(self.calculate_total_net_input())
return self.output
def calculate_total_net_input(self):
total = 0
for i in range(len(self.inputs)):
total += self.inputs[i] * self.weights[i]
return total + self.bias
# 激活函数sigmoid
def squash(self, total_net_input):
return 1 / (1 + math.exp(-total_net_input))
def calculate_pd_error_wrt_total_net_input(self, target_output):
return self.calculate_pd_error_wrt_output(target_output) * self.calculate_pd_total_net_input_wrt_input();
# 每一个神经元的误差是由平方差公式计算的
def calculate_error(self, target_output):
return 0.5 * (target_output - self.output) ** 2
def calculate_pd_error_wrt_output(self, target_output):
return -(target_output - self.output)
def calculate_pd_total_net_input_wrt_input(self):
return self.output * (1 - self.output)
def calculate_pd_total_net_input_wrt_weight(self, index):
return self.inputs[index]
# 文中的例子:
nn = NeuralNetwork(2, 2, 2, hidden_layer_weights=[0.15, 0.2, 0.25, 0.3], hidden_layer_bias=0.35, output_layer_weights=[0.4, 0.45, 0.5, 0.55], output_layer_bias=0.6)
for i in range(10000):
nn.train([0.05, 0.1], [0.01, 0.09])
print(i, round(nn.calculate_total_error([[[0.05, 0.1], [0.01, 0.09]]]), 9))
#另外一个例子，可以把上面的例子注释掉再运行一下:
# training_sets = [
# [[0, 0], [0]],
# [[0, 1], [1]],
# [[1, 0], [1]],
# [[1, 1], [0]]
# ]
# nn = NeuralNetwork(len(training_sets[0][0]), 5, len(training_sets[0][1]))
# for i in range(10000):
# training_inputs, training_outputs = random.choice(training_sets)
# nn.train(training_inputs, training_outputs)
# print(i, nn.calculate_total_error(training_sets))
365
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
