神经网络学习系列（二）自编码器以及栈自编码器 附python代码（minst手写体识别）
最近刚开始接触神经网络，想把自己找到的一些好的文章和代码以及自己的一些理解整理一下，一方面和广大学者交流，一方面巩固自己对神经网络的理解，若大家有什么其他更好的见解，欢迎留言讨论
参考资料：吴岸城《神经网络与深度学习》
理论分析
自编码器
自编码器（AutoEncoder）的工作就是深度学习中的“预训练”环节。
自编码器 模拟大脑的学习和重构，看下面这个例子：
小明在学习英语单词“Easy”，通过老师讲解他知道了这个单词对应的汉语是容易。第二天，老师开始检查小明的学习成果，小明在黑板上写下“容易”——“easy”。
请大家注意，这里小明写出的“easy”和昨天学到的“Easy”有偏差（误差），就是首字母大写变成小写了！小明将学习到的“Easy”记下来 ，再次反馈出来时，没有严格地区分首字母大小写，写出了“easy”这个单词。严格意义上，两个单词是有差异的。
小明昨天学习的过程就是编码，今天将整个单词还原出来就是解码过程，解码出的“easy”和“Easy”虽有偏差，但这个偏差是可以接受的，这就是自编码器的思想：学习对应编码，重构对应解码！
单看自编码器是是不需要标签的，因为他的输出就是输入，我们通过调整自编码器中的参数使得输出尽可能的与输入相同。
这样的自编码器是无法实现分类的，他只是学习到了一个可以代表输入的特征，这个特征在最大程度上代表原输入信号，（实现了数据的降维！！！），为了实现分类，可以在自编码器最顶的编码层添加一个分类器（例如，罗杰斯特回归、SVM等），然后通过标准的多层神经网络的监督训练方法（梯度下降法）去训练。小编在下面的代码中是直接采用了前向多层网络BP算法分类。
栈自编码器
栈自编码器是一个由多层自编码器组成的神经网络。
实战代码
一个自编码器（两层隐含层）+分类（只使用了最后提取的特征）
自编码器代码参考链接：
TensorFlow上实现AutoEncoder自编码器
外加自己添加的分类器
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# Visualize decoder setting
# Parameters
learning_rate = 0.015
learning_rate2 = 0.1
batch_size = 256
display_step = 1
examples_to_show = 10
# Network Parameters
n_input = 784 # 28x28 pix，即 784 Features
n_output = 10
# hidden layer settings
n_hidden_1 = 128 # 经过第一个隐藏层压缩至256个
n_hidden_2 = 49 # 经过第二个压缩至128个
n_hidden_3 = 36
n_hidden_4 = 16
# tf Graph input (only pictures)
X = tf.placeholder("float", [None, n_input])
OUT = tf.placeholder("float", [None, n_output])
XX = tf.placeholder("float", [None, n_hidden_2])
# output
# 两个隐藏层的 weights 和 biases 的定义
weights = {
'encoder_h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
'encoder_h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
'decoder_h1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_1])),
'decoder_h2': tf.Variable(tf.random_normal([n_hidden_1, n_input])),
}
biases = {
'encoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
'encoder_b2': tf.Variable(tf.random_normal([n_hidden_2])),
'decoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
'decoder_b2': tf.Variable(tf.random_normal([n_input])),
}
classify_weights = {
'layer1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3])),
'layer2': tf.Variable(tf.random_normal([n_hidden_3, n_hidden_4])),
'layer3': tf.Variable(tf.random_normal([n_hidden_4, n_output])),
}
classify_biases = {
'layer1': tf.Variable(tf.random_normal([n_hidden_3])),
'layer2': tf.Variable(tf.random_normal([n_hidden_4])),
'layer3': tf.Variable(tf.random_normal([n_output])),
}
# Building the encoder
def encoder(x):
# Encoder Hidden layer 使用的 Activation function 是 sigmoid #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['encoder_h1']), biases['encoder_b1']))
# Decoder Hidden layer with sigmoid activation #2
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['encoder_h2']), biases['encoder_b2']))
return layer_2
# Building the decoder
def decoder(x):
# Encoder Hidden layer with sigmoid activation #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['decoder_h1']), biases['decoder_b1']))
# Decoder Hidden layer with sigmoid activation #2
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['decoder_h2']), biases['decoder_b2']))
return layer_2
def classify(x):
out_layer1 = tf.nn.sigmoid(tf.add(tf.matmul(x, classify_weights['layer1']), classify_biases['layer1']))
out_layer2 = tf.nn.sigmoid(tf.add(tf.matmul(out_layer1, classify_weights['layer2']), classify_biases['layer2']))
out_layer3 = tf.nn.sigmoid(tf.add(tf.matmul(out_layer2, classify_weights['layer3']), classify_biases['layer3']))
return out_layer3
# Construct model
encoder_op = encoder(X)
decoder_op = decoder(encoder_op)
classify_op = classify(XX)
# Prediction
y_pred1 = decoder_op
y_pred2 = classify_op
# Targets (Labels) are the input data.
y_true1 = X
y_true2 = OUT
# Define loss and optimizer, minimize the squared error
# 比较原始数据与还原后的拥有 784 Features 的数据进行 cost 的对比，
# 根据 cost 来提升我的 Autoencoder 的准确率
loss1 = tf.reduce_mean(tf.pow(y_true1 - y_pred1, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
# loss2 = tf.reduce_mean(tf.pow(y_true2 - y_pred2, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
loss2 = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=OUT, logits=y_pred2))
# loss1 = tf.reduce_mean(tf.square(y_true1 - y_pred1))
optimizer1 = tf.train.AdamOptimizer(learning_rate).minimize(loss1)
optimizer2 = tf.train.AdamOptimizer(learning_rate).minimize(loss2)
# optimizer2 = tf.train.GradientDescentOptimizer(0.05).minimize(loss2)
# Launch the graph
with tf.Session() as sess:
init = tf.global_variables_initializer()
sess.run(init)
total_batch = int(mnist.train.num_examples / batch_size)
print(total_batch)
training_epochs = 20
training_epochs2 = 30
# Training cycle autoencoder
for epoch in range(training_epochs): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i in range(total_batch):
batch_xs1, batch_ys1 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# Run optimization op (backprop) and cost op (to get loss value)
_, c = sess.run([optimizer1, loss1], feed_dict={X: batch_xs1})
# Display logs per epoch step
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))
print("First Optimization Finished!")
for epoch2 in range(training_epochs2): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i2 in range(total_batch):
batch_xs2, batch_ys2 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# print(batch_ys2)
# Run optimization op (backprop) and cost op (to get loss value)
encoder_result = sess.run(encoder_op, feed_dict={X: batch_xs2})
_, c = sess.run([optimizer2, loss2], feed_dict={XX: encoder_result, OUT: batch_ys2})
# Display logs per epoch step
if epoch2 % display_step == 0:
test_output = sess.run(fetches=classify_op, feed_dict={XX: encoder_result})
test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(batch_ys2, 1)))
test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
print("Epoch:", '%04d' % (epoch2 + 1), "cost=", "{:.9f}".format(c))
print(test_accaracy) # 打印当前测试集的精度
print("Second Optimization Finished!")
# Applying encode and decode over test set
encode_decode = sess.run(y_pred1, feed_dict={X: mnist.test.images[:examples_to_show]})
# Compare original images with their reconstructions
f, a = plt.subplots(2, 10, figsize=(10, 2))
for i in range(examples_to_show):
a[0][i].imshow(np.reshape(mnist.test.images[i], (28, 28)))
a[1][i].imshow(np.reshape(encode_decode[i], (28, 28)))
plt.show()
# encoder_result = sess.run(encoder_op, feed_dict={X: mnist.test.images})
# sc = plt.scatter(encoder_result[:, 0], encoder_result[:, 1], c=mnist.test.labels) #散点图
# plt.colorbar(sc) #scatter设置颜色渐变条colorbar
# plt.show()
# with tf.Session() as sess:
# init = tf.global_variables_initializer()
# sess.run(init)
# total_batch0 = int(mnist.train.num_examples / batch_size)
# training_epochs = 25
# # Training cycle classify
# for epoch2 in range(training_epochs): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# # Loop over all batches
# for i2 in range(total_batch0):
# batch_xs2, batch_ys2 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# # print(batch_ys2)
# # Run optimization op (backprop) and cost op (to get loss value)
# encoder_result = sess.run(encoder_op, feed_dict={X: batch_xs2})
# _, c = sess.run([optimizer2, loss2], feed_dict={XX: encoder_result, OUT: batch_ys2})
# # Display logs per epoch step
# if epoch2 % display_step == 0:
# test_output = sess.run(fetches=classify_op, feed_dict={XX: encoder_result})
# test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(batch_ys2, 1)))
# test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
# print("Epoch:", '%04d' % (epoch2 + 1), "cost=", "{:.9f}".format(c))
# print(test_accaracy) # 打印当前测试集的精度
# print("Second Optimization Finished!")
# 计算准确率
# encoder_result2 = sess.run(encoder_op, feed_dict={X: mnist.test.images[:examples_to_show]})
test_ax, test_ay = mnist.test.next_batch(1000) # 则使用测试集对当前网络进行测试
encoder_result2 = sess.run(encoder_op, feed_dict={X: test_ax})
print(test_ay)
test_output = sess.run(fetches=classify_op, feed_dict={XX: encoder_result2})
print(test_output)
test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(test_ay, 1)))
print(test_acc)
test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
print(test_accaracy) # 打印当前测试集的精度
栈自编码器（两个自编码器）+分类（只用了最后提取的特征）
#Two Autoencoders + classify minst
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# Visualize decoder setting
# Parameters
learning_rate = 0.01
learning_rate2 = 0.1
batch_size = 100
display_step = 1
examples_to_show = 10
# Network Parameters
n_input = 784 # 28x28 pix，即 784 Features
n_output = 10
# hidden layer settings
n_hidden_1 = 128 # 经过第一个隐藏层压缩至256个
n_hidden_2 = 49 # 经过第二个压缩至128个
n_hidden_3 = 36
n_hidden_4 = 16
# tf Graph input (only pictures)
X1 = tf.placeholder("float", [None, n_input])
X2 = tf.placeholder("float", [None, n_hidden_1])
OUT = tf.placeholder("float", [None, n_output])
XX = tf.placeholder("float", [None, n_hidden_2])
# output
# 两个隐藏层的 weights 和 biases 的定义
weights1 = {
'encoder_h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
'decoder_h1': tf.Variable(tf.random_normal([n_hidden_1, n_input])),
}
biases1 = {
'encoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
'decoder_b1': tf.Variable(tf.random_normal([n_input])),
}
weights2 = {
'encoder_h1': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
'decoder_h1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_1])),
}
biases2 = {
'encoder_b1': tf.Variable(tf.random_normal([n_hidden_2])),
'decoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
}
classify_weights = {
'layer1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3])),
'layer2': tf.Variable(tf.random_normal([n_hidden_3, n_hidden_4])),
'layer3': tf.Variable(tf.random_normal([n_hidden_4, n_output])),
}
classify_biases = {
'layer1': tf.Variable(tf.random_normal([n_hidden_3])),
'layer2': tf.Variable(tf.random_normal([n_hidden_4])),
'layer3': tf.Variable(tf.random_normal([n_output])),
}
# Building the encoder
def encoder1(x):
# Encoder Hidden layer 使用的 Activation function 是 sigmoid #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights1['encoder_h1']), biases1['encoder_b1']))
return layer_1
def encoder2(x):
# Encoder Hidden layer 使用的 Activation function 是 sigmoid #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights2['encoder_h1']), biases2['encoder_b1']))
return layer_1
# Building the decoder
def decoder1(x):
# Encoder Hidden layer with sigmoid activation #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights1['decoder_h1']), biases1['decoder_b1']))
return layer_1
def decoder2(x):
# Encoder Hidden layer with sigmoid activation #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights2['decoder_h1']), biases2['decoder_b1']))
return layer_1
def classify(x):
out_layer1 = tf.nn.sigmoid(tf.add(tf.matmul(x, classify_weights['layer1']), classify_biases['layer1']))
out_layer2 = tf.nn.sigmoid(tf.add(tf.matmul(out_layer1, classify_weights['layer2']), classify_biases['layer2']))
out_layer3 = tf.nn.sigmoid(tf.add(tf.matmul(out_layer2, classify_weights['layer3']), classify_biases['layer3']))
return out_layer3
# Construct model
encoder_op1 = encoder1(X1)
decoder_op1 = decoder1(encoder_op1)
encoder_op2 = encoder2(X2)
decoder_op2 = decoder2(encoder_op2)
classify_op = classify(XX)
# Prediction
y_pred1 = decoder_op1
y_pred1_2 = decoder_op2
y_pred2 = classify_op
# Targets (Labels) are the input data.
y_true1 = X1
y_true1_2 = X2
y_true2 = OUT
# Define loss and optimizer, minimize the squared error
# 比较原始数据与还原后的拥有 784 Features 的数据进行 cost 的对比，
# 根据 cost 来提升我的 Autoencoder 的准确率
loss1 = tf.reduce_mean(tf.pow(y_true1 - y_pred1, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
loss1_2 = tf.reduce_mean(tf.pow(y_true1_2 - y_pred1_2, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
# loss2 = tf.reduce_mean(tf.pow(y_true2 - y_pred2, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
loss2 = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=OUT, logits=y_pred2))
# loss1 = tf.reduce_mean(tf.square(y_true1 - y_pred1))
optimizer1 = tf.train.AdamOptimizer(learning_rate).minimize(loss1)
optimizer1_2 = tf.train.AdamOptimizer(learning_rate).minimize(loss1_2)
optimizer2 = tf.train.AdamOptimizer(learning_rate).minimize(loss2)
# optimizer2 = tf.train.GradientDescentOptimizer(0.05).minimize(loss2)
# Launch the graph
with tf.Session() as sess:
init = tf.global_variables_initializer()
sess.run(init)
total_batch = int(mnist.train.num_examples / batch_size)
print(total_batch)
training_epochs = 10
training_epochs1 = 10
training_epochs2 = 10
# Training cycle autoencoder
for epoch in range(training_epochs): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i in range(total_batch):
batch_xs1, batch_ys1 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# Run optimization op (backprop) and cost op (to get loss value)
_, c = sess.run([optimizer1, loss1], feed_dict={X1: batch_xs1})
# Display logs per epoch step
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))
print("First Optimization Finished!")
for epoch in range(training_epochs1): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i in range(total_batch):
batch_xs1_2, batch_ys1_2 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# Run optimization op (backprop) and cost op (to get loss value)
encoder_result1 = sess.run(encoder_op1, feed_dict={X1: batch_xs1_2})
_, c = sess.run([optimizer1_2, loss1_2], feed_dict={X2: encoder_result1})
# Display logs per epoch step
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))
print("First second Optimization Finished!")
for epoch in range(training_epochs2): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i in range(total_batch):
batch_xs2, batch_ys2 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# print(batch_ys2)
encoder_result1 = sess.run(encoder_op1, feed_dict={X1: batch_xs2})
encoder_result2 = sess.run(encoder_op2, feed_dict={X2: encoder_result1})
_, c = sess.run([optimizer2, loss2], feed_dict={XX: encoder_result2, OUT: batch_ys2})
# Display logs per epoch step
if epoch % display_step == 0:
test_output = sess.run(fetches=classify_op, feed_dict={XX: encoder_result2})
test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(batch_ys2, 1)))
test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))
print(test_accaracy) # 打印当前测试集的精度
print("Second Optimization Finished!")
# Applying encode and decode over test set
encode_decode = sess.run(y_pred1, feed_dict={X1: mnist.test.images[:examples_to_show]})
# Compare original images with their reconstructions
f, a = plt.subplots(2, 10, figsize=(10, 2))
for i in range(examples_to_show):
a[0][i].imshow(np.reshape(mnist.test.images[i], (28, 28)))
a[1][i].imshow(np.reshape(encode_decode[i], (28, 28)))
plt.show()
# encoder_result = sess.run(encoder_op, feed_dict={X: mnist.test.images})
# sc = plt.scatter(encoder_result[:, 0], encoder_result[:, 1], c=mnist.test.labels) #散点图
# plt.colorbar(sc) #scatter设置颜色渐变条colorbar
# plt.show()
# with tf.Session() as sess:
# init = tf.global_variables_initializer()
# sess.run(init)
# total_batch0 = int(mnist.train.num_examples / batch_size)
# training_epochs = 25
# # Training cycle classify
# for epoch2 in range(training_epochs): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# # Loop over all batches
# for i2 in range(total_batch0):
# batch_xs2, batch_ys2 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# # print(batch_ys2)
# # Run optimization op (backprop) and cost op (to get loss value)
# encoder_result = sess.run(encoder_op, feed_dict={X: batch_xs2})
# _, c = sess.run([optimizer2, loss2], feed_dict={XX: encoder_result, OUT: batch_ys2})
# # Display logs per epoch step
# if epoch2 % display_step == 0:
# test_output = sess.run(fetches=classify_op, feed_dict={XX: encoder_result})
# test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(batch_ys2, 1)))
# test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
# print("Epoch:", '%04d' % (epoch2 + 1), "cost=", "{:.9f}".format(c))
# print(test_accaracy) # 打印当前测试集的精度
# print("Second Optimization Finished!")
# 计算准确率
# encoder_result2 = sess.run(encoder_op, feed_dict={X: mnist.test.images[:examples_to_show]})
test_ax, test_ay = mnist.test.next_batch(5000) # 则使用测试集对当前网络进行测试
print(test_ay)
encoder_result1 = sess.run(encoder_op1, feed_dict={X1: test_ax})
encoder_result2 = sess.run(encoder_op2, feed_dict={X2: encoder_result1})
test_output = sess.run(fetches=classify_op, feed_dict={XX: encoder_result2})
print(test_output)
test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(test_ay, 1)))
print(test_acc)
test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
print(test_accaracy) # 打印当前测试集的精度
栈自编码器（两个自编码器）+分类（从输入开始进行BP算法）
#Two Autoencoders + classify minst
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# Visualize decoder setting
# Parameters
learning_rate = 0.01
learning_rate2 = 0.1
batch_size = 100
display_step = 1
examples_to_show = 10
# Network Parameters
n_input = 784 # 28x28 pix，即 784 Features
n_output = 10
# hidden layer settings
n_hidden_1 = 128 # 经过第一个隐藏层压缩至256个
n_hidden_2 = 49 # 经过第二个压缩至128个
n_hidden_3 = 36 # 分类器的第一个隐含层
n_hidden_4 = 16 # 分类器的第二个隐含层
# tf Graph input (only pictures)
X1 = tf.placeholder("float", [None, n_input])
X2 = tf.placeholder("float", [None, n_hidden_1])
OUT = tf.placeholder("float", [None, n_output])
# output
# 两个隐藏层的 weights 和 biases 的定义
weights1 = {
'encoder_h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
'decoder_h1': tf.Variable(tf.random_normal([n_hidden_1, n_input])),
}
biases1 = {
'encoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
'decoder_b1': tf.Variable(tf.random_normal([n_input])),
}
weights2 = {
'encoder_h1': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
'decoder_h1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_1])),
}
biases2 = {
'encoder_b1': tf.Variable(tf.random_normal([n_hidden_2])),
'decoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
}
classify_weights = {
'layer1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3])),
'layer2': tf.Variable(tf.random_normal([n_hidden_3, n_hidden_4])),
'layer3': tf.Variable(tf.random_normal([n_hidden_4, n_output])),
}
classify_biases = {
'layer1': tf.Variable(tf.random_normal([n_hidden_3])),
'layer2': tf.Variable(tf.random_normal([n_hidden_4])),
'layer3': tf.Variable(tf.random_normal([n_output])),
}
# Building the encoder
def encoder1(x):
# Encoder Hidden layer 使用的 Activation function 是 sigmoid #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights1['encoder_h1']), biases1['encoder_b1']))
return layer_1
def encoder2(x):
# Encoder Hidden layer 使用的 Activation function 是 sigmoid #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights2['encoder_h1']), biases2['encoder_b1']))
return layer_1
# Building the decoder
def decoder1(x):
# Encoder Hidden layer with sigmoid activation #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights1['decoder_h1']), biases1['decoder_b1']))
return layer_1
def decoder2(x):
# Encoder Hidden layer with sigmoid activation #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights2['decoder_h1']), biases2['decoder_b1']))
return layer_1
def classify(x):
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights1['encoder_h1']), biases1['encoder_b1']))
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights2['encoder_h1']), biases2['encoder_b1']))
out_layer1 = tf.nn.sigmoid(tf.add(tf.matmul(layer_2, classify_weights['layer1']), classify_biases['layer1']))
out_layer2 = tf.nn.sigmoid(tf.add(tf.matmul(out_layer1, classify_weights['layer2']), classify_biases['layer2']))
out_layer3 = tf.nn.sigmoid(tf.add(tf.matmul(out_layer2, classify_weights['layer3']), classify_biases['layer3']))
return out_layer3
# Construct model
encoder_op1 = encoder1(X1)
decoder_op1 = decoder1(encoder_op1)
encoder_op2 = encoder2(X2)
decoder_op2 = decoder2(encoder_op2)
classify_op = classify(X1)
# Prediction
y_pred1 = decoder_op1
y_pred1_2 = decoder_op2
y_pred2 = classify_op
# Targets (Labels) are the input data.
y_true1 = X1
y_true1_2 = X2
y_true2 = OUT
# Define loss and optimizer, minimize the squared error
# 比较原始数据与还原后的拥有 784 Features 的数据进行 cost 的对比，
# 根据 cost 来提升我的 Autoencoder 的准确率
loss1 = tf.reduce_mean(tf.pow(y_true1 - y_pred1, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
loss1_2 = tf.reduce_mean(tf.pow(y_true1_2 - y_pred1_2, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
# loss2 = tf.reduce_mean(tf.pow(y_true2 - y_pred2, 2)) # 进行最小二乘法的计算(y_true - y_pred)^2
loss2 = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=OUT, logits=y_pred2))
# loss1 = tf.reduce_mean(tf.square(y_true1 - y_pred1))
optimizer1 = tf.train.AdamOptimizer(learning_rate).minimize(loss1)
optimizer1_2 = tf.train.AdamOptimizer(learning_rate).minimize(loss1_2)
optimizer2 = tf.train.AdamOptimizer(learning_rate).minimize(loss2)
# optimizer2 = tf.train.GradientDescentOptimizer(0.05).minimize(loss2)
# Launch the graph
with tf.Session() as sess:
init = tf.global_variables_initializer()
sess.run(init)
total_batch = int(mnist.train.num_examples / batch_size)
print(total_batch)
training_epochs = 10
training_epochs1 = 10
training_epochs2 = 10
# Training cycle autoencoder
for epoch in range(training_epochs): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i in range(total_batch):
batch_xs1, batch_ys1 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# Run optimization op (backprop) and cost op (to get loss value)
_, c = sess.run([optimizer1, loss1], feed_dict={X1: batch_xs1})
# Display logs per epoch step
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))
print("First Optimization Finished!")
# Applying encode and decode over test set
encode_decode = sess.run(y_pred1, feed_dict={X1: mnist.test.images[:examples_to_show]})
# Compare original images with their reconstructions
f, a = plt.subplots(2, 10, figsize=(10, 2))
for i in range(examples_to_show):
a[0][i].imshow(np.reshape(mnist.test.images[i], (28, 28)))
a[1][i].imshow(np.reshape(encode_decode[i], (28, 28)))
plt.show()
for epoch in range(training_epochs1): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i in range(total_batch):
batch_xs1_2, batch_ys1_2 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# Run optimization op (backprop) and cost op (to get loss value)
encoder_result1 = sess.run(encoder_op1, feed_dict={X1: batch_xs1_2})
_, c = sess.run([optimizer1_2, loss1_2], feed_dict={X2: encoder_result1})
# Display logs per epoch step
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))
print("First second Optimization Finished!")
for epoch in range(training_epochs2): # 到好的的效果，我们应进行10 ~ 20个 Epoch 的训练
# Loop over all batches
for i in range(total_batch):
batch_xs2, batch_ys2 = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# print(batch_ys2)
_, c = sess.run([optimizer2, loss2], feed_dict={X1: batch_xs2, OUT: batch_ys2})
# Display logs per epoch step
if epoch % display_step == 0:
test_output = sess.run(fetches=classify_op, feed_dict={X1: batch_xs2})
test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(batch_ys2, 1)))
test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))
print(test_accaracy) # 打印当前测试集的精
print("Second Optimization Finished!")
# encoder_result = sess.run(encoder_op, feed_dict={X: mnist.test.images})
# sc = plt.scatter(encoder_result[:, 0], encoder_result[:, 1], c=mnist.test.labels) #散点图
# plt.colorbar(sc) #scatter设置颜色渐变条colorbar
# plt.show()
# 计算准确率
# encoder_result2 = sess.run(encoder_op, feed_dict={X: mnist.test.images[:examples_to_show]})
test_ax, test_ay = mnist.test.next_batch(5000) # 则使用测试集对当前网络进行测试
print(test_ay)
test_output = sess.run(fetches=classify_op, feed_dict={X1: test_ax})
print(test_output)
test_acc = sess.run(tf.equal(tf.argmax(test_output, 1), tf.argmax(test_ay, 1)))
print(test_acc)
test_accaracy = sess.run(tf.reduce_mean(tf.cast(test_acc, dtype=tf.float32))) # 求出精度的准确率进行打印
print(test_accaracy) # 打印当前测试集的精度
最后，小编为了节省时间，缩短了训练的迭代次数，导致准确率不高，大家可以调整迭代次数提高准确率。
1352
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
