RNN、LSTM、GRU学习笔记

传统神经网络结构的缺陷
输出信号只与输入信号有关，而与输入信号的先后顺序无关

循环神经网络RNN
信息的传递往往会因为时间间隔太长而逐渐衰减。对于信息的长期依赖问题没有很好的处理办法

原文链接：https://blog.csdn.net/shakehands2012/article/details/109241628
长短期记忆网络（LSTM）

LSTM中的三个门
(1)forget 遗忘门: 神经网络层、按位乘操作
遗忘门的原理有点类似于注意力机制

激活函数为sigmoid

图中符号的理解
t:时间
x:输入
c:胞元状态
h:历史记忆（胞元状态与历史记忆的关系？）
f:是一个0~1的值决定上一个胞中哪些信息保留

图中公式的理解
[h,x] 将两个向量拼接
W*[h,x] 矩阵乘以向量
W*[h,x]+b 向量加法

(2)input 输入门: 与tanh神经网络层和一个按位乘操作构成

(3)output 输出门: 以及按位乘操作共同作用将细胞状态和输入信号传递到输出端。

GRU（ Gated Recurrent Unit，LSTM变体）

#GRU Demo # https://www.kaggle.com/sehandev/ny-stock-price-prediction-gru-tensorflow-2-x
import numpy as np
import pandas as pd

# import all stock prices
df = pd.read_csv("./prices-split-adjusted.csv", index_col = 0)

# choose one stock
df_stock = df[df.symbol == 'EQIX'].copy()
df_stock.drop(['symbol'], 1, inplace=True)
df_stock.drop(['volume'], 1, inplace=True)#只保留4个指标：日期和高、中、低股值 #(1762, 4)

# create train, test data
def load_data(stock, seq_len):# function to create train, validation, test data given stock data and sequence length
    data_raw = stock.values # convert to numpy array
    data = []

    # create all possible sequences of length seq_len
    for index in range(len(data_raw) - seq_len):
        data.append(data_raw[index: index + seq_len])

    data = np.array(data)
    valid_set_size = int(np.round(10/100*data.shape[0]))#valid_set_size_percentage = 10
    test_set_size = int(np.round(10/100*data.shape[0]))#test_set_size_percentage = 10
    train_set_size = data.shape[0] - (valid_set_size + test_set_size)

    x_train = data[:train_set_size,:-1,:]#(1394, 19, 4) (1394, 4) # 输入：n*seq*dim  输出：n*dim
    y_train = data[:train_set_size,-1,:]
    x_valid = data[train_set_size:train_set_size+valid_set_size,:-1,:]#(174, 19, 4) (174, 4)
    y_valid = data[train_set_size:train_set_size+valid_set_size,-1,:]
    x_test = data[train_set_size+valid_set_size:,:-1,:]#(174, 19, 4) (174, 4)
    y_test = data[train_set_size+valid_set_size:,-1,:]
    return [x_train, y_train, x_valid, y_valid, x_test, y_test]
seq_len = 20 # choose sequence length
x_train, y_train, x_valid, y_valid, x_test, y_test = load_data(df_stock, seq_len)

# Build the GRU model - tensorflow 2
from tensorflow import keras
def build_model(n_steps,n_inputs,n_neurons,n_outputs,n_layers):
    model = keras.Sequential()
    model.add(keras.layers.GRU(n_neurons, activation='relu', input_shape=(n_steps, n_inputs), return_sequences=True))# GRU Layers
    for i in range(n_layers - 1):#GRU可以多层嵌套
        model.add(keras.layers.GRU(n_neurons, activation='relu', input_shape=(n_steps, n_inputs), return_sequences=False))
    model.add(keras.layers.Dense(n_outputs))# Output Layer
    model.summary()#似乎用于输出
    return model
model = build_model(seq_len-1, 4, 100, 4, 2)
model.compile(
    loss="mean_squared_error",#采用均方误差
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),#输入的是初始学习率，训练过程中学习率逐渐衰减
    metrics=['accuracy'],
)
model.fit(
    x_train, y_train, # 输入：n*seq*dim  输出：n*dim
    validation_data=(x_valid, y_valid),
    batch_size=50, #将数据分为多少组 #只分为1组速度反而下降了
    epochs=20,#100  #数据使用多少遍
    verbose=1,#是否显示进度条
)

y_pred = model.predict(x_test)
## show predictions
import matplotlib.pyplot as plt
ft = 0 # 0 = open, 1 = close, 2 = highest, 3 = lowest
plt.plot(np.arange(y_test.shape[0]),y_test[:, ft], color='black', label='target'    )
plt.plot(np.arange(y_test.shape[0]),y_pred[:, ft], color='red'  , label='prediction')
plt.show()