前向传播和反向传播

前向传播和反向传播

---

---

前言

前向传播就是神经网络的每一层经过权重乘以输入求和再加上偏置项，通过激活函数在得到输出，输出再做为下一层的输入，反复直至的到最后输出的过程。
反向传播就是神经网络通过最终的输出计算每一层的权重对最终输出的影响（用偏导数衡量），再经过梯度下降原理，从当前权值减去学习率乘以偏导数，从而达到更新权值的过程。
下面我通过一个例子来演示这两个过程。

假设我有一个三层的神经网络：

i1、i2是输入层，h1、h2是隐藏层，o1、o2是输出层，b1、b2是输入层到隐藏层和隐藏层到输出层的偏置，w是每一层到下一层的权值。每一层都是两个神经元，激活函数使用Sigmoid函数。然后我给每个参数赋予初值。

其中，$$\begin{array}{rl}输入数据：& i1=0.05，i2=0.10\\ 输出数据：& o1=0.01，o2=0.99\\ 初始权重：& w1=0.15，w2=0.20，w3=0.25，w4=0.30\\ & w5=0.40，w6=0.45，w7=0.50，w8=0.55\end{array}$$
目标：给出输入数据i1=0.05，i2=0.10，使最终输出尽可能与目标输出o1=0.01，o2=0.99接近。

前向传播

1. 输入层----->隐藏层
   计算隐藏层h1的输入：
   $$\begin{array}{rl}ne{t}_{h1}& =i1\ast w1+i2\ast w2+b1\ast 1\\ & =0.05\ast 0.15+0.10\ast 0.20+0.35\ast 1\\ & =0.3775\end{array}$$
   神经元h1的输出：
   $$ou{t}_{h1}=\frac{1}{1+e^{-ne{t}_{h1}}}=\frac{1}{1+e^{-0.3775}}=0.59326999$$
   同理，算出神经元h2的输出为：
   $$ou{t}_{h2}=0.59688438$$
2. 隐藏层----->输出层
   计算输出层o1的输入：
   $$\begin{array}{rl}ne{t}_{o1}& =ou{t}_{h1}\ast w5+ou{t}_{h2}\ast w6+b2\ast 1\\ & =0.59326999\ast 0.4+0.59688438\ast 0.45+0.60\ast 1\\ & =1.10590597\end{array}$$
   神经元o1的输出：
   $$ou{t}_{o1}=\frac{1}{1+e^{-ne{t}_{o1}}}=\frac{1}{1+e^{-1.10590597}}=0.75136507$$
   同理，算出神经元o2的输出为：
   $$ou{t}_{o2}=0.77292847$$
   至此前向传播结束，得到了输出值 o1=0.75136507，o2=0.77292847，与目标输出o1=0.01，o2=0.99相差较多。我们接着对误差反向传播，更新权值，再重新计算输出。

反向传播

1. 计算总误差
   总误差等于所有误差之和：
   $$\begin{array}{rl}{E}_{total}& =\sum \frac{1}{2}(target-output{)}^2=E_{o1}+E_{o2}\\ E_{o1}& =\frac{1}{2}(targe{t}_{o1}-ou{t}_{o1}{)}^2\\ & =\frac{1}{2}(0.01-0.75136507{)}^2\\ & =0.27481108\\ E_{o2}& =0.02356003\\ E_{total}& =E_{o1}+E_{o2}=0.29837111\end{array}$$

2. 隐藏层----->输出层的权值更新
   以权重$w7$为例，计算$w7$对整个结果的影响（使用链式法则分开求导）：
   下图明确了误差是如何反向传播的：
   

   $$\begin{gathered} \begin{array}{r} \\ \\ \\ \frac{\partial E_{total}}{\partial w_7}=\frac{\partial E_{total}}{\partial ou{t}_{02}}\times \frac{\partial ou{t}_{02}}{\partial ne{t}_{02}}\times \frac{\partial ne{t}_{o2}}{\partial w_7}\end{array} \end{gathered}$$

   计算$\frac{\partial E_{total}}{\partial ou{t}_{02}}$：
   $$\begin{array}{rl}{E}_{total}& =\sum \frac{1}{2}(target-output{)}^2\\ & =\frac{1}{2}(targe{t}_{o1}-ou{t}_{o1}{)}^2+\frac{1}{2}(targe{t}_{o2}-outpu{t}_{o2}{)}^2\\ & \\ \frac{\partial E_{total}}{\partial ou{t}_{02}}& =\frac{1}{2}\times 2\times (targe{t}_{o2}-outpu{t}_{o2})\times (-1)\\ & =-(targe{t}_{o2}-ou{t}_{o2})\\ & =-(0.99-0.77292847)\\ & =-0.2170753\end{array}$$
   计算$\frac{\partial ou{t}_{o2}}{ne{t}_{o2}}$：

   $$\begin{array}{rl}ou{t}_{o2}& =\frac{1}{1+e^{-ne{t}_{o2}}}\\ & \\ \frac{\partial ou{t}_{o2}}{\partial ne{t}_{o2}}& =ou{t}_{o2}(1-ou{t}_{o2})(这就是Sigmoid函数的导数)\\ & =0.77292847\times (1-0.77292847)\\ & =0.17551005\end{array}$$
   计算$\frac{\partial ne{t}_{o2}}{\partial w_7}$:
   $$\begin{array}{rl}ne{t}_{o1}& =ou{t}_{h1}\times w_7+ou{t}_{h2}\times w_8+b_2\times 1\\ \frac{\partial ne{t}_{o1}}{\partial w_7}& =ou{t}_{h1}\\ & =0.59326999\end{array}$$

   由此我们求得：
   $$\begin{array}{rl}\frac{\partial E_{total}}{\partial w_7}& =\frac{\partial E_{total}}{\partial ou{t}_{02}}\times \frac{\partial ou{t}_{02}}{\partial ne{t}_{02}}\times \frac{\partial ne{t}_{o2}}{\partial w_7}\\ & \\ & =-0.2170753\times 0.17551005\times 0.59326999\\ & \\ & =-0.02260294\end{array}$$
   这样我们就求得整体误差$E_{total}$对$w_7$的偏导数。

   再看上面的公式，我们发现：
   $$\begin{array}{rl}\frac{\partial E_{total}}{\partial w_7}& =-(targe{t}_{o2}-ou{t}_{o2})\ast ou{t}_{o2}(1-ou{t}_{o2})\ast ou{t}_{h1}\end{array}$$
   现在我们用${\delta }_{o2}$来表示输出层的误差：
   $$\begin{array}{rl}{\delta }_{o2}& =\frac{\partial E_{total}}{\partial ou{t}_{02}}\times \frac{\partial ou{t}_{02}}{\partial ne{t}_{02}}\\ & \\ & =\frac{\partial E_{total}}{\partial ne{t}_{o2}}\\ & \\ {\delta }_{o2}& =-(targe{t}_{o2}-ou{t}_{o2})\ast ou{t}_{o2}(1-ou{t}_{o2})\\ & \\ & 所以，整体误差E_{total}对w_7的偏导数简写为：\\ & \\ \frac{\partial E_{total}}{\partial w_7}& ={\delta }_{o1}\ast ou{t}_{h1}\end{array}$$

   然后我们就可以更新 $w7$ 的值了，假设 $\eta =0.5$
   $$\begin{array}{rl}{w}_7^{\ast }& =w_7-\eta \times \frac{\partial E_{total}}{\partial w_7}\\ & =0.5-0.5\times (-0.02260294)\\ & =0.51130147\end{array}$$
   同理，对$w_5，w_6，w_8，$也进行更新：
   $$\begin{array}{rl}{w}_5^{\ast }& =0.35891648\\ & \\ w_6^{\ast }& =0.40866619\\ & \\ w_8^{\ast }& =0.56137012\\ & \end{array}$$

3. 输入层----->隐藏层的权值更新 （假设更新$w_3$）
   输入层----->隐藏层的权值更新跟上面 隐藏层----->输出层的权值更新思路差不多，但是隐藏 层----->输出层的权值更新是从 $ou{t}_{o2}->ne{t}_{o2}->w_7$，输入层----->隐藏层的权值更新是从 $ou{t}_{h2}->ne{t}_{h2}->w_3$，但是 $ou{t}_{h2}$ 会接受 $E_{o1},E_{o2}$ 两个方向传来的误差，所以都要计算。下图说明了误差影响的方向。
   
   $$\begin{array}{rl}\frac{\partial E_{total}}{\partial w_3}& =\frac{\partial E_{total}}{\partial ou{t}_{h2}}\ast \frac{\partial ou{t}_{h2}}{\partial ne{t}_{h2}}\ast \frac{\partial ne{t}_{h2}}{\partial w_3}\\ & \end{array}$$

$\frac{\partial E_{total}}{\partial ou{t}_{h2}}$:
$$\begin{array}{rl}{E}_{total}& =E_{o1}+E_{o2}\\ & \\ \frac{\partial E_{total}}{\partial ou{t}_{h2}}& =\frac{\partial E_{o1}}{\partial ou{t}_{h2}}+\frac{\partial E_{o2}}{\partial ou{t}_{h2}}\end{array}$$
再计算$\frac{\partial E_{o1}}{\partial ou{t}_{h2}},\frac{\partial E_{o2}}{\partial ou{t}_{h2}}$
$$\begin{gathered} \begin{array}{rl} \\ \frac{\partial E_{o1}}{\partial ou{t}_{h2}}& =\frac{\partial E_{o1}}{\partial ou{t}_{o1}}\ast \frac{\partial ou{t}_{o1}}{\partial ne{t}_{o1}}\ast \frac{\partial ne{t}_{o1}}{\partial ou{t}_{h2}}\\ & \\ 分别计算\frac{\partial E_{o1}}{\partial ou{t}_{o1}}、\frac{\partial ou{t}_{o1}}{\partial ne{t}_{o1}}、\frac{\partial ne{t}_{o1}}{\partial ou{t}_{h2}}& \\ & \\ E_{o1}& =\frac{1}{2}\ast (targe{t}_{o1}-ou{t}_{o1}{)}^2\\ & \\ \frac{\partial E_{o1}}{\partial ou{t}_{o1}}& =\frac{1}{2}\ast 2\ast (targe{t}_{o1}-ou{t}_{o1})\ast (-1)\\ & \\ & =-(0.01-0.75136507)\\ & \\ & =0.74136507\\ & \\ ou{t}_{o1}& =\frac{1}{1+e^{-ne{t}_{o1}}}\\ & \\ \frac{\partial ou{t}_{o1}}{\partial ne{t}_{o1}}& =ou{t}_{o1}\ast (1-ou{t}_{o1})\\ & \\ & =0.75136507\ast (1-0.75136507)\\ & \\ & =0.18681560\\ & \\ ne{t}_{o1}& =ou{t}_{h1}\ast w_5+ou{t}_{h2}\ast w_6+1\ast b_2\\ & \\ \frac{\partial ne{t}_{o1}}{\partial ou{t}_{h2}}& =w_6=0.45\\ & \\ 所以，\frac{\partial E_{o1}}{\partial ou{t}_{h2}}& =\frac{\partial E_{o1}}{\partial ou{t}_{o1}}\ast \frac{\partial ou{t}_{o1}}{\partial ne{t}_{o1}}\ast \frac{\partial ne{t}_{o1}}{\partial ou{t}_{h2}}\\ & \\ & =0.74136507\ast 0.18681560\ast 0.45\\ & \\ & =0.06232435\end{array} \end{gathered}$$

同理得：$$\frac{\partial E_{o2}}{\partial ou{t}_{h2}}=0.17551005$$
所以
$$\begin{array}{rl}\frac{\partial E_{total}}{\partial ou{t}_{h2}}& =\frac{\partial E_{o1}}{\partial ou{t}_{h2}}+\frac{\partial E_{o2}}{\partial ou{t}_{h2}}\\ & \\ & =0.06232435+0.17551005\\ & \\ & =0.23783440\\ & \end{array}$$

$\frac{\partial ou{t}_{h2}}{ne{t}_{h2}}$:
$$\begin{array}{rl}ou{t}_{h2}& =\frac{1}{1+e^{-ne{t}_{h2}}}\\ & \\ \frac{\partial ou{t}_{h2}}{ne{t}_{h2}}& =ou{t}_{h2}\ast (1-ou{t}_{h2})\\ & \\ & =0.59688438\ast (1-0.59688438)\\ & \\ & =0.24061342\end{array}$$

$\frac{\partial ne{t}_{h2}}{\partial w_3}$:
$$\begin{array}{rl}ne{t}_{h2}& =i_1\ast w_3+i_2\ast w_4+1\ast b_1\\ & \\ \frac{\partial ne{t}_{h2}}{\partial w_3}& =i_1=0.05\end{array}$$
所以
$$\begin{array}{rl}\frac{\partial E_{total}}{\partial w_3}& =\frac{\partial E_{total}}{\partial ou{t}_{h2}}\ast \frac{\partial ou{t}_{h2}}{\partial ne{t}_{h2}}\ast \frac{\partial ne{t}_{h2}}{\partial w_3}\\ & \\ & =0.23783440\ast 0.24061306\ast 0.05\\ & \\ & =0.00286130\end{array}$$
为了简记，用${\delta }_{h1}$代表隐藏层$h_1$的误差：
$$\begin{array}{rl}\frac{\partial E_{total}}{\partial w_3}& =\frac{\partial E_{total}}{\partial ou{t}_{h2}}\ast \frac{\partial ou{t}_{h2}}{\partial ne{t}_{h2}}\ast \frac{\partial ne{t}_{h2}}{\partial w_3}\\ & \\ & =(\frac{\partial E_{o1}}{\partial ou{t}_{h2}}+\frac{\partial E_{o2}}{\partial ou{t}_{h2}})\ast ou{t}_{h2}\ast (1-ou{t}_{h2})\ast i_1\\ & \\ & =(\frac{\partial E_{o1}}{\partial ou{t}_{o1}}\ast \frac{\partial ou{t}_{o1}}{\partial ne{t}_{o1}}\ast \frac{\partial ne{t}_{o1}}{\partial ou{t}_{h2}}+\frac{\partial E_{o2}}{\partial ou{t}_{o2}}\ast \frac{\partial ou{t}_{o2}}{\partial ne{t}_{o2}}\ast \frac{\partial ne{t}_{o2}}{\partial ou{t}_{h2}})\\ & \\ & \ast ou{t}_{h2}\ast (1-ou{t}_{h2})\ast i_1\\ & \\ & =((ou{t}_{o1}-targe{t}_{o1})\ast (ou{t}_{o1}\ast (1-ou{t}_{o1}))\ast w_6\\ & \\ & +(ou{t}_{o2}-targe{t}_{o2})\ast (ou{t}_{o2}\ast (1-ou{t}_{o2}))\ast w_6)\\ & \\ & \ast ou{t}_{h2}\ast (1-ou{t}_{h2})\ast i_1\\ & \\ & ={\delta }_{h1}\ast i_1\end{array}$$
最后更新$w_3$的值：
$$w_3^{\ast }=w_3-\eta \ast \frac{\partial E_{total}}{\partial w_3}=0.24856935$$
同样的，更新$w_1、w_2、w_4$的权值：
$$\begin{array}{rl}{w}_1^{\ast }& =0.14978072\\ & \\ w_2^{\ast }& =0.19956143\\ & \\ w_4^{\ast }& =0.29950229\\ & \end{array}$$
这样误差反向传播法就完成了，然后把新的权值代入，在进行误差计算，到最后接近目标输出就算完成了。

#coding:utf-8
import random
import math


#   参数解释：
#   "pd_" ：偏导的前缀
#   "d_" ：导数的前缀
#   "w_ho" ：隐含层到输出层的权重系数索引
#   "w_ih" ：输入层到隐含层的权重系数的索引

class NeuralNetwork:
    LEARNING_RATE = 0.5

    def __init__(self, num_inputs, num_hidden, num_outputs, hidden_layer_weights = None, hidden_layer_bias = None, output_layer_weights = None, output_layer_bias = None):
        self.num_inputs = num_inputs

        self.hidden_layer = NeuronLayer(num_hidden, hidden_layer_bias)
        self.output_layer = NeuronLayer(num_outputs, output_layer_bias)

        self.init_weights_from_inputs_to_hidden_layer_neurons(hidden_layer_weights)
        self.init_weights_from_hidden_layer_neurons_to_output_layer_neurons(output_layer_weights)

    def init_weights_from_inputs_to_hidden_layer_neurons(self, hidden_layer_weights):
        weight_num = 0
        for h in range(len(self.hidden_layer.neurons)):
            for i in range(self.num_inputs):
                if not hidden_layer_weights:
                    self.hidden_layer.neurons[h].weights.append(random.random())
                else:
                    self.hidden_layer.neurons[h].weights.append(hidden_layer_weights[weight_num])
                weight_num += 1

    def init_weights_from_hidden_layer_neurons_to_output_layer_neurons(self, output_layer_weights):
        weight_num = 0
        for o in range(len(self.output_layer.neurons)):
            for h in range(len(self.hidden_layer.neurons)):
                if not output_layer_weights:
                    self.output_layer.neurons[o].weights.append(random.random())
                else:
                    self.output_layer.neurons[o].weights.append(output_layer_weights[weight_num])
                weight_num += 1

    def inspect(self):
        print('------')
        print('* Inputs: {}'.format(self.num_inputs))
        print('------')
        print('Hidden Layer')
        self.hidden_layer.inspect()
        print('------')
        print('* Output Layer')
        self.output_layer.inspect()
        print('------')

    def feed_forward(self, inputs):
        hidden_layer_outputs = self.hidden_layer.feed_forward(inputs)
        return self.output_layer.feed_forward(hidden_layer_outputs)

    def train(self, training_inputs, training_outputs):
        self.feed_forward(training_inputs)

        # 1. 输出神经元的值
        pd_errors_wrt_output_neuron_total_net_input = [0] * len(self.output_layer.neurons)
        for o in range(len(self.output_layer.neurons)):

            # ∂E/∂zⱼ
            pd_errors_wrt_output_neuron_total_net_input[o] = self.output_layer.neurons[o].calculate_pd_error_wrt_total_net_input(training_outputs[o])

        # 2. 隐含层神经元的值
        pd_errors_wrt_hidden_neuron_total_net_input = [0] * len(self.hidden_layer.neurons)
        for h in range(len(self.hidden_layer.neurons)):

            # dE/dyⱼ = Σ ∂E/∂zⱼ * ∂z/∂yⱼ = Σ ∂E/∂zⱼ * wᵢⱼ
            d_error_wrt_hidden_neuron_output = 0
            for o in range(len(self.output_layer.neurons)):
                d_error_wrt_hidden_neuron_output += pd_errors_wrt_output_neuron_total_net_input[o] * self.output_layer.neurons[o].weights[h]

            # ∂E/∂zⱼ = dE/dyⱼ * ∂zⱼ/∂
            pd_errors_wrt_hidden_neuron_total_net_input[h] = d_error_wrt_hidden_neuron_output * self.hidden_layer.neurons[h].calculate_pd_total_net_input_wrt_input()

        # 3. 更新输出层权重系数
        for o in range(len(self.output_layer.neurons)):
            for w_ho in range(len(self.output_layer.neurons[o].weights)):

                # ∂Eⱼ/∂wᵢⱼ = ∂E/∂zⱼ * ∂zⱼ/∂wᵢⱼ
                pd_error_wrt_weight = pd_errors_wrt_output_neuron_total_net_input[o] * self.output_layer.neurons[o].calculate_pd_total_net_input_wrt_weight(w_ho)

                # Δw = α * ∂Eⱼ/∂wᵢ
                self.output_layer.neurons[o].weights[w_ho] -= self.LEARNING_RATE * pd_error_wrt_weight

        # 4. 更新隐含层的权重系数
        for h in range(len(self.hidden_layer.neurons)):
            for w_ih in range(len(self.hidden_layer.neurons[h].weights)):

                # ∂Eⱼ/∂wᵢ = ∂E/∂zⱼ * ∂zⱼ/∂wᵢ
                pd_error_wrt_weight = pd_errors_wrt_hidden_neuron_total_net_input[h] * self.hidden_layer.neurons[h].calculate_pd_total_net_input_wrt_weight(w_ih)

                # Δw = α * ∂Eⱼ/∂wᵢ
                self.hidden_layer.neurons[h].weights[w_ih] -= self.LEARNING_RATE * pd_error_wrt_weight

    def calculate_total_error(self, training_sets):
        total_error = 0
        for t in range(len(training_sets)):
            training_inputs, training_outputs = training_sets[t]
            self.feed_forward(training_inputs)
            for o in range(len(training_outputs)):
                total_error += self.output_layer.neurons[o].calculate_error(training_outputs[o])
        return total_error

class NeuronLayer:
    def __init__(self, num_neurons, bias):

        # 同一层的神经元共享一个截距项b
        self.bias = bias if bias else random.random()

        self.neurons = []
        for i in range(num_neurons):
            self.neurons.append(Neuron(self.bias))

    def inspect(self):
        print('Neurons:', len(self.neurons))
        for n in range(len(self.neurons)):
            print(' Neuron', n)
            for w in range(len(self.neurons[n].weights)):
                print('  Weight:', self.neurons[n].weights[w])
            print('  Bias:', self.bias)

    def feed_forward(self, inputs):
        outputs = []
        for neuron in self.neurons:
            outputs.append(neuron.calculate_output(inputs))
        return outputs

    def get_outputs(self):
        outputs = []
        for neuron in self.neurons:
            outputs.append(neuron.output)
        return outputs

class Neuron:
    def __init__(self, bias):
        self.bias = bias
        self.weights = []

    def calculate_output(self, inputs):
        self.inputs = inputs
        self.output = self.squash(self.calculate_total_net_input())
        return self.output

    def calculate_total_net_input(self):
        total = 0
        for i in range(len(self.inputs)):
            total += self.inputs[i] * self.weights[i]
        return total + self.bias

    # 激活函数sigmoid
    def squash(self, total_net_input):
        return 1 / (1 + math.exp(-total_net_input))


    def calculate_pd_error_wrt_total_net_input(self, target_output):
        return self.calculate_pd_error_wrt_output(target_output) * self.calculate_pd_total_net_input_wrt_input();

    # 每一个神经元的误差是由平方差公式计算的
    def calculate_error(self, target_output):
        return 0.5 * (target_output - self.output) ** 2

    
    def calculate_pd_error_wrt_output(self, target_output):
        return -(target_output - self.output)

    
    def calculate_pd_total_net_input_wrt_input(self):
        return self.output * (1 - self.output)


    def calculate_pd_total_net_input_wrt_weight(self, index):
        return self.inputs[index]


# 文中的例子:

nn = NeuralNetwork(2, 2, 2, hidden_layer_weights=[0.15, 0.2, 0.25, 0.3], hidden_layer_bias=0.35, output_layer_weights=[0.4, 0.45, 0.5, 0.55], output_layer_bias=0.6)
for i in range(10000):
    nn.train([0.05, 0.1], [0.01, 0.09])
    print(i, round(nn.calculate_total_error([[[0.05, 0.1], [0.01, 0.09]]]), 9))


#另外一个例子，可以把上面的例子注释掉再运行一下:

# training_sets = [
#     [[0, 0], [0]],
#     [[0, 1], [1]],
#     [[1, 0], [1]],
#     [[1, 1], [0]]
# ]

# nn = NeuralNetwork(len(training_sets[0][0]), 5, len(training_sets[0][1]))
# for i in range(10000):
#     training_inputs, training_outputs = random.choice(training_sets)
#     nn.train(training_inputs, training_outputs)
#     print(i, nn.calculate_total_error(training_sets))

作者：Charlotte77
出处：http://www.cnblogs.com/charlotte77/