BPN算法
学习 BPN 算法可以分成以下两个过程:
正向传播:输入被馈送到网络,信号从输入层通过隐藏层传播到输出层。在输出层,计算误差和损失函数。
反向传播:在反向传播中,首先计算输出层神经元损失函数的梯度,然后计算隐藏层神经元损失函数的梯度。接下来用梯度更新权重。
终于理清了epoch、batch_size、输入输出的shape表示的含义。调参、训练模型从此再也不迷糊了。
Result
2020-01-15 21:30:10.556053: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
Epoch: 0 accuracy train%:9.748333333333333 accuracy test%:10.55
Epoch: 1000 accuracy train%:81.77333333333333 accuracy test%:89.64
Epoch: 2000 accuracy train%:83.83 accuracy test%:91.26
Epoch: 3000 accuracy train%:84.73333333333333 accuracy test%:91.96
Epoch: 4000 accuracy train%:85.51666666666667 accuracy test%:92.95
Epoch: 5000 accuracy train%:86.02 accuracy test%:93.38
Epoch: 6000 accuracy train%:86.46 accuracy test%:93.59
Epoch: 7000 accuracy train%:86.68666666666667 accuracy test%:93.49
Epoch: 8000 accuracy train%:86.965 accuracy test%:93.88
Epoch: 9000 accuracy train%:87.11833333333334 accuracy test%:93.71
Epoch: 10000 accuracy train%:87.335 accuracy test%:94.15
Epoch: 11000 accuracy train%:87.39166666666667 accuracy test%:94.06
Epoch: 12000 accuracy train%:87.65 accuracy test%:94.34
Process finished with exit code 0
BPN的网络结构
代码:亲自敲一敲代码,理解每一步的执行原理,让你可以很清楚知道训练网络每一步是如何进行的,前向传播,隐藏层计算,后向传播,更新权重等是怎样工作的。
输入层:x_in的矩阵结构60x784的数据矩阵
W_h为784x30的权重矩阵
b_h为1x30的偏置矩阵
W_o为30x10的输出层权重矩阵
b_o为1x30的输出层偏置
图示如下:
X_in与w_h之间为矩阵乘法。即x_in的每一行元素乘以w_h每一列的对应元素,再累加和,为结果矩阵的对应位置上的值。
以上表示前向传播的分解示意。
Epoch等于几,就表示同样的步骤,再进行几轮训练,更新几轮权重网络。
网络实现:
def multilayer_perception(x, weights, biases):
# 隐藏层的激活函数RELU
h_layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['h1'])
h_layer_out = tf.sigmoid(h_layer_1)
out_layer = tf.matmul(h_layer_out, weights['out']) + biases['out']
return tf.sigmoid(out_layer),out_layer,h_layer_out,h_layer_1
反向传播
损失函数J(w):
J对权重各参数求偏导数,相当于J到y_hat到输出激活,到out_layer到h_layer_out到h_layer_1,依次链式求导。
还不清楚,请参考高数。或者教程tensorflow
学习使用的激活函数是sigmoid,它的导数:
# 导函数:sigmoid(x)[1.0-sigmoid(x)]
def sigmaprime(x):
return tf.multiply(tf.sigmoid(x),tf.subtract(tf.constant(1.0), tf.sigmoid(x)))
向后传播实现:
# 向后传播
delta_2 = tf.multiply(err,sigmaprime(h_2))
# transpose 转置运算
delta_w_2 = tf.matmul(tf.transpose(o_1),delta_2)
wtd_error = tf.matmul(delta_2, tf.transpose(weight['out']))
delta_1 = tf.multiply(wtd_error, sigmaprime(h_1))
delta_w_1 = tf.matmul(tf.transpose(x_in), delta_1)
然后是根据梯度下降的方向结合学习率更新权重。
# update weights
# tf.assign(ref, value, validate_shape=None, use_locking=None, name=None)
# 函数完成了将value赋值给ref的作用。其中:ref 必须是tf.Variable创建的tensor
step = [
tf.assign(weight['h1'], tf.subtract(weight['h1'],tf.multiply(eta,delta_w_1))),
tf.assign(bias['h1'], tf.subtract(bias['h1'],tf.multiply(eta,tf.reduce_mean(delta_1,axis=[0])))),
tf.assign(weight['out'], tf.subtract(weight['out'],tf.multiply(eta,delta_w_2))),
tf.assign(bias['out'], tf.subtract(bias['out'],tf.multiply(eta,tf.reduce_mean(delta_2,axis=[0]))))
]
附上亲自实践成功的代码
#!/usr/bin/python
# -*- coding = utf-8 -*-
#author:beauthy
#date:2020.1.15
#version:1.0
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
# 加载数据集 使用独热编码标签
mnist = input_data.read_data_sets("MNIST_data",one_hot=True)
# 28 X 28 = 784 标签分为10个类0-9
# 超参数包括:学习率、最大迭代周期、每次批量训练的批量大小、以及每次隐藏层中的神经元数量
n_input = 784
n_classes = 10
max_epochs = 12001
learning_rate = 0.15
batch_size = 60
seed = 0
# number of neurons in the hidden layer
n_hidden = 30
# 使用sigmoid函数的导数进行权重更新
# 导函数:sigmoid(x)[1.0-sigmoid(x)]
def sigmaprime(x):
return tf.multiply(tf.sigmoid(x),tf.subtract(tf.constant(1.0), tf.sigmoid(x)))
def multilayer_perception(x, weights, biases):
# 隐藏层的激活函数RELU
h_layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['h1'])
h_layer_out = tf.sigmoid(h_layer_1)
out_layer = tf.matmul(h_layer_out, weights['out']) + biases['out']
return tf.sigmoid(out_layer),out_layer,h_layer_out,h_layer_1
def train_mnist_model():
x_in = tf.placeholder(tf.float32, [None,n_input])
y = tf.placeholder(tf.float32, [None,n_classes])
weight = {
'h1': tf.Variable(tf.random_normal([n_input, n_hidden],seed=seed)),
'out': tf.Variable(tf.random_normal([n_hidden,n_classes],seed=seed))
}
bias = {
'h1': tf.Variable(tf.random_normal([1, n_hidden],seed=seed)),
'out': tf.Variable(tf.random_normal([1, n_classes],seed=seed))
}
# 前向传播
y_hat, h_2, o_1,h_1 = multilayer_perception(x_in,weight,bias)
# 误差
err = y_hat - y
# 向后传播
delta_2 = tf.multiply(err,sigmaprime(h_2))
# transpose 转置运算
delta_w_2 = tf.matmul(tf.transpose(o_1),delta_2)
wtd_error = tf.matmul(delta_2, tf.transpose(weight['out']))
delta_1 = tf.multiply(wtd_error, sigmaprime(h_1))
delta_w_1 = tf.matmul(tf.transpose(x_in), delta_1)
eta = tf.constant(learning_rate)
# update weights
# tf.assign(ref, value, validate_shape=None, use_locking=None, name=None)
# 函数完成了将value赋值给ref的作用。其中:ref 必须是tf.Variable创建的tensor
step = [
tf.assign(weight['h1'], tf.subtract(weight['h1'],tf.multiply(eta,delta_w_1))),
tf.assign(bias['h1'], tf.subtract(bias['h1'],tf.multiply(eta,tf.reduce_mean(delta_1,axis=[0])))),
tf.assign(weight['out'], tf.subtract(weight['out'],tf.multiply(eta,delta_w_2))),
tf.assign(bias['out'], tf.subtract(bias['out'],tf.multiply(eta,tf.reduce_mean(delta_2,axis=[0]))))
]
# accuracy
# tf.argmax(input)用途:返回最大的那个数值所在的下标
# 1表示按行比较返回最大值的索引
# equal(x, y, name=None)
# equal,相等的意思。顾名思义,就是判断,x, y 是不是相等,它的判断方法不是整体判断,
# 逐个元素判断,相等就是True,不相等就是False。
acct_mat = tf.equal(tf.argmax(y_hat,1), tf.argmax(y,1))
accuracy = tf.reduce_sum(tf.cast(acct_mat,tf.float32))
init = tf.global_variables_initializer()
# 计算图
with tf.Session() as sess:
sess.run(init)
for epoch in range(max_epochs):
batch_xs, batch_ys = mnist.train.next_batch(batch_size)
sess.run([y_hat, h_2, o_1, h_1], feed_dict={x_in:batch_xs})
sess.run(step, feed_dict={
x_in: batch_xs,
y:batch_ys
})
if epoch%1000 == 0:
acc_test = sess.run(accuracy,feed_dict={
x_in: mnist.test.images,
y: mnist.test.labels
}) #测试数据集的大小是 1000
acc_train = sess.run(accuracy,feed_dict={
x_in: mnist.train.images,
y: mnist.train.labels
})
print('Epoch: {0} accuracy train%:{1} accuracy test%:{2}'
.format(epoch,acc_train/600,acc_test/100))
if __name__ == "__main__":
train_mnist_model()
pass