002_wz_NLP_LSTM和LSTMCell

LSTM介绍

先去看RNN

LSTM使用“门”（sigmoid函数）解决了RNN不能记忆长距离信息的问题，使用累加方式的损失解决了RNN梯度爆炸/弥散的问题

LSTM中有三个门，分别是：
1.forget gate（遗忘门)
遗忘门用来控制记住以前记忆（$C_{t-1}$）的程度

2.input gate（输入门）
输入门用来控制记忆此时刻输入数据（$X_t$）的程度

此时在经过了遗忘门和输入门，我们可以拿到本次的记忆数据$C_t$

这里介绍一下，$f_t$是遗忘门，$i_t$是输入门，$C_{t-1}$相当于RNN中的$h_t{W}_{hh}$,~$C_t$相当于$x_t{W}_{ih}$，门在这里是起一个选择的作用，LSTM后面还需要再经过输出门，得到$h_t$

3.output gate（输出门）
输出门用来控制输出$h_t$的程度

下面是LSTM的简化整体流程

迭代公式

不同门关闭与开启，造成的效果

torch.nn.LSTM

输入参数：
input_size:即对数据做embedding的数据维度feature_len
hidden_size:LSTM的隐层维度
num_layers:LSTM网络的层数，默认为1层

LSTM的前向传播

out, (h_t, c_t) = lstm(x, (h_t0, c_t0)

x:输入数据，维度为(seq_len, batch_size, feature_len)
h/c:上一隐层的输出，维度为(num_layers, batch_size, hidden_size)
out:为每一时刻隐层输出的列表集合，形如[h_1, h_2, …, h_t]，维度为(seq_len, batch_size, hidden_size)

LSTM代码验证

import torch

lstm = torch.nn.LSTM(input_size=100, hidden_size=20, num_layers=4)

x = torch.randn(10, 3, 100)
h_0 = torch.zeros(4, 3, 20)
c_0 = torch.zeros(4, 3, 20)

out, (h_t, c_t) = lstm(x, (h_0, c_0))

print(out.shape, h_t.shape, c_t.shape)

torch.Size([10, 3, 20]) torch.Size([4, 3, 20]) torch.Size([4, 3, 20])

torch.nn.LSTMCell

输入与LSTM相同，但是没有num_layers

LSTMCell的前向传播

h_t, c_t = lstmcell(x_t, (h_t0, c_t0))

x_t:单个输入数据，维度为(batch_size, feature_len)
h_t/c_t:隐层输出，维度为(batch_size, hidden_size)

LSTM代码验证

import torch

lstmcell = torch.nn.LSTMCell(input_size=100, hidden_size=20)

x = torch.randn(10, 3, 100)
h_0 = torch.zeros(3, 20)
c_0 = torch.zeros(3, 20)

for x_t in x:
    h_t, c_t = lstmcell(x_t, (h_0, c_0))

print(h_t.shape, c_t.shape)

torch.Size([3, 20]) torch.Size([3, 20])

import torch

lstmcell1 = torch.nn.LSTMCell(input_size=100, hidden_size=20)
lstmcell2 = torch.nn.LSTMCell(input_size=20, hidden_size=10)

x = torch.randn(10, 3, 100)
h_0 = torch.zeros(3, 20)
c_0 = torch.zeros(3, 20)

h_1 = torch.zeros(3, 10)
c_1 = torch.zeros(3, 10)

for x_t in x:
    h_0, c_0 = lstmcell1(x_t, (h_0, c_0))
    h_1, c_1 = lstmcell2(h_0, (h_1, c_1))

print(h_1.shape, c_1.shape)

torch.Size([3, 10]) torch.Size([3, 10])