[LSTM学习笔记5]How to Develop Stacked LSTMs

一.Stacked LSTM概述

Stacked LSTM是有多个隐藏LSTM层，每个LSTM层有包含多个memory cell的LSTM。这样做的目的时加深网络，从而预测更加复杂的问题。上层的LSTM结构输出一个序列而不是一个单一的值到下一个LSTM结构。在《Speech Recogition with Deep Recurent Neural Network》这篇文章中指出，LSTM层级的堆叠比LSTM单元内部memory cell增加起到的效果更大。

默认输入LSTM是3D数据，输出的是最后的值,是一个2D shape array

# Example of one output for whole sequence
from keras.models import Sequential
from keras.layers import LSTM
from numpy import array
#define model where LSTM is also output layer
model = Sequential()
model.add(LSTM(1, input_shape=(3,1)))
model.compile(optimizer='adam', loss='mse')
# input time steps
data = array([0.1, 0.2, 0.3]).reshape((1,3,1))
# make and show prediction
print(model.predict(data))

输出：
[[ 0.04022677]]
使用stacked LSTM，下级LSTM接收的是上一级输出的序列而不是值，需要使用return_sequences=True，返回输出的3D array。

# Example of one output for each input time step
from keras.models import Sequential
from keras.layers import LSTM
from numpy import array
# define model where LSTM is also output layer
model = Sequential()
model.add(LSTM(1, return_sequences=True, input_shape=(3,1)))
model.compile(optimizer='adam', loss='mse')
# input time steps
data = array([0.1, 0.2, 0.3]).reshape((1,3,1))
# make and show prediction
print(model.predict(data))

输出：
[[[-0.02033241]
[-0.04997678]
[-0.08230399]]]

####二.Damped Sine Wave Prediction实例
标准正弦波： $$\sin [2\pi f(\frac{i}{length})]$$

from math import sin
from math import pi
import matplotlib.pyplot as plt
# create sequence
length = 100
freq = 5
sequence = [sin(2 * pi * freq * (i/length)) for i in range (length)]
# plot sequence
plt.plot(sequence)
plt.show()

输出：

阻尼正弦波：$$0.5+0.5\ast \sin (2\ast \pi \ast \frac{i}{peroid})\ast {\exp }^{-decay\ast i}$$

from math import pi,exp,sin
import matplotlib.pyplot as plt
# create sequence
length = 100
peroid = 10
decay = 0.05
sequence = [0.5+0.5*sin(2 * pi * i/peroid)*exp(-decay * i) for i in range (length)]
# plot sequence
plt.plot(sequence)
plt.show()

输出：

下面的例子保留序列的最后几个点，用模型来进行预测：

from random import randint
from random import uniform
# generate damped sine wave in [0,1]
def generate_sequence(length, period, decay):
    return [0.5 + 0.5 * sin(2 * pi * i / period) * exp(-decay * i) for i in range(length)]

#将生成的序列进行分离，分离出序列最后的n time steps用作预测（output）
# generate input and output pairs of damped sine waves
def generate_examples(length, n_patterns, output):
    X, y = list(), list()
    for _ in range(n_patterns):
        #随机选择period和decay
        p = randint(10, 20)
        d = uniform(0.01, 0.1)
        sequence = generate_sequence(length + output, p, d)
        X.append(sequence[:-output])
        y.append(sequence[-output:])
    X = array(X).reshape(n_patterns, length, 1)
    y = array(y).reshape(n_patterns, output)
    return X, y

# 测试数据生成，生成5个不同的阻尼正弦波，每个25time steps，其中最后5个留作预测使用
X, y = generate_examples(20, 5, 5)
for i in range(len (X)):
    plt.plot([x for x in X[i, :, 0]] + [x for x in y[i]],'-o' )
plt.show()

输出：

该问题将被建模为many-to-one模型，一次预测输出全部序列而不是预测出一个time step,类似于：

下面使用有两层隐藏LSTM层的stacked LSTM：

• 每一层有20个memory cell，输入为1feature,20 time step。
• 输出为5个值组成的向量，解释为5个time steps。输出层使用线性激活函数，这是当没有声明激活函数时的默认选项。
• 使用平均绝对误差（MAE）和Adam Optimizer

import keras
from keras.layers import Dense,LSTM
from keras.models import Sequential
# configure problem
length = 20
output = 5
# define model
model = Sequential()
model.add(LSTM(20, return_sequences=True, input_shape=(length, 1)))
model.add(LSTM(20))
model.add(Dense(output))
model.compile(loss='mae', optimizer='adam')
print (model.summary())

#evaluate model
X, y = generate_examples(length, 1000, output)
loss = model.evaluate(X, y, verbose=0)
print('MAE : %f'% loss)

输出：
MAE : 0.018124

import  matplotlib.pyplot as plt
# prediction on new data
X, y = generate_examples(length, 1, output)
yhat = model.predict(X, verbose=0)
plt.plot(y[0], label='y')
plt.plot(yhat[0], label='yhat')
plt.legend()
plt.show()

输出：