概述
长短期记忆网络(LSTM,Long Short-Term Memory)是一种时间循环神经网络,是为了解决一般的RNN(循环神经网络)存在的长期依赖问题而专门设计出来的。LSTM可以解决RNN的梯度爆炸问题,时间序列可以比RNN更长。想详细了解循环神经网络的原理可以参考:
- https://zhuanlan.zhihu.com/p/45289691
- https://www.jianshu.com/p/5f39b91bff11
- https://www.cnblogs.com/lovychen/p/9294624.html
- https://www.cnblogs.com/wzdLY/p/10071962.html
LSTM可以对时间序列选择性遗忘、选择性保存,在此,写一个简单的例子对此进行验证。
验证
训练数据为:
train_x = [[[1], [2], [5], [6]],
[[5], [7], [7], [8]],
[[3], [4], [5], [7]]]
train_y = [[1, 3, 7, 11],
[5, 12, 14, 15],
[3, 7, 9, 12]]
此处的对应关系是:train_y元素为train_x相邻元素之和。1=0+1、3=1+2、7=2+5、11=5+6等。
利用train_x和train_y对模型进行训练,使用训练好的模型对test_x进行预测,看LSTM网络能否找出train_x和train_y之间的规律。
具体实现代码rnn-example.py:
import tensorflow as tf
from tensorflow.contrib import rnn
class SeriesPredictor:
def __init__(self, input_dim, seq_size, hidden_dim=10):
# Hyperparameters
self.input_dim = input_dim
self.seq_size = seq_size
self.hidden_dim = hidden_dim
# Weight variables and input placeholders
self.W_out = tf.Variable(tf.random_normal([hidden_dim, 1]), name='W_out')
self.b_out = tf.Variable(tf.random_normal([1]), name='b_out')
# (batchsize, 4个阶段, 每个阶段1个值)
self.x = tf.placeholder(tf.float32, [None, seq_size, input_dim])
# 四个标签
self.y = tf.placeholder(tf.float32, [None, seq_size])
# Cost optimizer
self.cost = tf.reduce_mean(tf.square(self.model() - self.y))
self.train_op = tf.train.AdamOptimizer().minimize(self.cost)
# Auxiliary ops
self.saver = tf.train.Saver()
def model(self):
"""
:param x: Input of size [T, batch_size, input_size]
:param W: Matrix of fully-connected output layer weights
:param b: Vector of fully-connected output layer biases
"""
cell = rnn.BasicLSTMCell(self.hidden_dim)
# 初始化cell单元。outputs维度(?, 4, 10)
outputs, states = tf.nn.dynamic_rnn(cell, self.x, dtype=tf.float32)
# 当前样本个数
num_examples = tf.shape(self.x)[0]
# tf.expand_dims,增加一个新的维度
# W_repeated = tf.tile(tf.expand_dims(self.W_out, 0), [num_examples, 1, 1])
# tf_expand维度(1,10,1)
tf_expand = tf.expand_dims(self.W_out, 0)
# tf_tile维度(?,10,1)
tf_tile = tf.tile(tf_expand, [num_examples, 1, 1])
# out维度(?, 4, 1)
out = tf.matmul(outputs, tf_tile) + self.b_out
# tf.squeeze 删除所有维度是1的
out = tf.squeeze(out)
return out
def train(self, train_x, train_y):
with tf.Session() as sess:
tf.get_variable_scope().reuse_variables()
sess.run(tf.global_variables_initializer())
# 迭代次数
for i in range(1000):
_, mse = sess.run([self.train_op, self.cost], feed_dict={self.x: train_x, self.y: train_y})
if i % 100 == 0:
print(i, mse)
save_path = self.saver.save(sess, './model')
print('Model saved to {}'.format(save_path))
def test(self, test_x):
with tf.Session() as sess:
tf.get_variable_scope().reuse_variables()
self.saver.restore(sess, './model')
output = sess.run(self.model(), feed_dict={self.x: test_x})
return output
if __name__ == '__main__':
predictor = SeriesPredictor(input_dim=1, seq_size=4, hidden_dim=10)
train_x = [[[1], [2], [5], [6]],
[[5], [7], [7], [8]],
[[3], [4], [5], [7]]]
train_y = [[1, 3, 7, 11],
[5, 12, 14, 15],
[3, 7, 9, 12]]
predictor.train(train_x, train_y)
test_x = [[[1], [2], [3], [4]],
[[4], [5], [6], [7]]]
actual_y = [[1, 3, 5, 7],
[4, 9, 11, 13]]
pred_y = predictor.test(test_x)
print("\nlets run some tests!\n")
for i, x in enumerate(test_x):
print('when the inputs is {}'.format(x))
print('the ground truth output should be {}'.format(actual_y[i]))
print('and the model thinks it is {}\n'.format(pred_y[i]))
结果:
0 93.94602
100 37.57643
200 12.853694
300 5.2746544
400 3.4352033
500 2.7549746
600 2.1920292
700 1.6691321
800 1.2889558
900 0.9742842
Model saved to ./model
lets run some tests!
when the inputs is [[1], [2], [3], [4]]
the ground truth output should be [1, 3, 5, 7]
and the model thinks it is [0.86799127 2.593877 4.958681 6.8895497 ]
when the inputs is [[4], [5], [6], [7]]
the ground truth output should be [4, 9, 11, 13]
and the model thinks it is [ 4.114984 9.15321 11.73832 12.63355 ]
由运行结果可知,预测的值和标签值比较接近,由于测试数据集比较小,所以预测值还是有一定误差的,但是可以看出LSTM已经找到训练数据的规律了。