长短期记忆网络LSTM-读书笔记

取个名字真难呐

已于 2022-02-08 10:37:21 修改

阅读量1.6k

点赞数

分类专栏： pytorch 文章标签：神经网络深度学习 pytorch

于 2022-01-28 17:15:49 首次发布

本文链接：https://blog.csdn.net/scar2016/article/details/122734129

版权

pytorch 专栏收录该内容

148 篇文章 28 订阅

订阅专栏

文章目录

1. 长短期记忆网络 LSTM
2. 深度循环神经网络
3. 双向循环神经网络

1. 长短期记忆网络 LSTM

1.1 原理概述

长短期记忆网络门：
（1）忘记门 $F_t$ ：将值朝0减少
$F_t=\sigma(X_tW_{xf}+H_{t-1}W_{hf}+b_f)\tag 1$
（2）输入门 $I_t$ ：决定是不是忽略掉输入数据
$I_t=\sigma(X_tW_{xi}+H_{t-1}W_{hi}+b_i)\tag 2$
（3）输出门 $O_t$ ：决定是不是使用隐状态
$O_t=\sigma(X_tW_{xo}+H_{t-1}W_{ho}+b_o)\tag 3$
（4）候选记忆单元 $\widetilde{C}_t$ ：存储记忆作用
$\widetilde{C}_t=\tanh(X_tW_{xc}+H_{t-1}W_{hc}+b_c)\tag4$
（5）记忆单元 $C_t$ ：
$C_t=F_t \odot C_{t-1}+I_t \odot \widetilde{C}_t\tag5$
（6）隐状态 $H_t$ :
$H_t=O_t\odot \tanh(C_t)\tag6$

1.2 图解结构

在这里插入图片描述

1.3 代码描述

1.3.1 lstm 从零开始实现

代码

# -*- coding: utf-8 -*-
# @Project: zc
# @Author: zc
# @File name: LSTM_test
# @Create time: 2022/1/30 22:48

import torch
from torch import nn
from d2l import torch as d2l
import matplotlib.pyplot as plt

batch_size, num_steps = 32, 35
train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)


def get_lstm_params(vocab_size, num_hiddens, device):
	num_inputs = num_outputs = vocab_size

	def normal(shape):
		return torch.randn(size=shape, device=device) * 0.01

	def three():
		return (normal((num_inputs, num_hiddens)),
				normal((num_hiddens, num_hiddens)),
				torch.zeros(num_hiddens, device=device))

	W_xi, W_hi, b_i = three()  # 输入门参数
	W_xf, W_hf, b_f = three()  # 遗忘门参数
	W_xo, W_ho, b_o = three()  # 输出门参数
	W_xc, W_hc, b_c = three()  # 候选记忆单元参数

	# 输出层参数
	W_hq = normal((num_hiddens, num_outputs))
	b_q = torch.zeros(num_outputs, device=device)

	params = [W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc,
			  b_c, W_hq, b_q]
	for param in params:
		param.requires_grad_(True)
	return params


def init_lstm_state(batch_size, num_hiddens, device):
	return (torch.zeros((batch_size, num_hiddens), device=device),
			torch.zeros((batch_size, num_hiddens), device=device))


def lstm(inputs, state, params):
	[W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc, b_c,
	 W_hq, b_q] = params
	(H, C) = state
	outputs = []
	for X in inputs:
		I = torch.sigmoid((X @ W_xi) + (H @ W_hi) + b_i)
		F = torch.sigmoid((X @ W_xf) + (H @ W_hf) + b_f)
		O = torch.sigmoid((X @ W_xo) + (H @ W_ho) + b_o)
		C_tilda = torch.tanh((X @ W_xc) + (H @ W_hc) + b_c)
		C = F * C + I * C_tilda
		H = O * torch.tanh(C)
		Y = (H @ W_hq) + b_q
		outputs.append(Y)
	return torch.cat(outputs, dim=0), (H, C)


vocab_size,num_hiddens,device = len(vocab),256,d2l.try_gpu()
num_epochs,lr = 500,1
model = d2l.RNNModelScratch(len(vocab),num_hiddens,device,get_lstm_params,
							init_lstm_state,lstm)
d2l.train_ch8(model,train_iter,vocab,lr,num_epochs,device)
plt.show()

结果

perplexity 1.2, 29691.7 tokens/sec on cuda:0
time traveller after the thing tine wionerswer for sowey bucklex
traveller afcenthe epenthe of ho gersfor ton this it ar wis

在这里插入图片描述

1.3.2 lstm 简洁实现

代码

# -*- coding: utf-8 -*-
# @Project: zc
# @Author: zc
# @File name: lstm-concise
# @Create time: 2022/2/6 22:06
import torch
from torch import nn
from d2l import torch as d2l
import matplotlib.pyplot as plt

batch_size, num_steps = 32, 35
train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)

vocab_size, num_hiddens, device = len(vocab), 256, d2l.try_gpu()
num_epochs, lr = 500, 1

num_inputs = vocab_size
lstm_layer = nn.LSTM(num_inputs, num_hiddens)
model = d2l.RNNModel(lstm_layer, len(vocab))
model = model.to(device)
d2l.train_ch8(model,train_iter,vocab,lr,num_epochs,device)
plt.show()

结果

perplexity 1.1, 359347.9 tokens/sec on cuda:0
time travelleryou can show black is white by argument said filby
travelleryou can show black is white by argument said filby

在这里插入图片描述

1.4 网络小结

（1）长短期记忆网络有三种类型的门：输入门，遗忘门和输出门
（2）长短期记忆网络的隐藏层输出包括"隐状态"和"记忆单元"。只有隐状态会传递到输出层，而记忆元完全属于内部信息
（3）长短期记忆网络可以缓解梯度消失和梯度爆炸

2. 深度循环神经网络

2.1 循环神经网络模型结构

在这里插入图片描述

$h_t$ 相关网络：
$h_t=\phi(W_{hh}h_{t-1}+W_{hx}x_{t-1}+b_h)\tag1$
注1： $h_t$ 来自于 $h_{t-1}$ 和 $x_{t-1}$ 的影响； $O_t$ 来自于 $h_t$ 的影响； $W_{hh}$ 存储所有的时序信息
$O_t=\phi(W_{ho}h_t+b_o)\tag2$
注2：损失的计算是 $loss =O_t-X_t$ ;因为 $O_t$ 相当于 Y_hat,而X_t 相当于标签 Y，那么我们就能得到损失值

2.2 深度循环神经网络结构图

在这里插入图片描述
$H_t^{(l)}=\phi(H_t^{(l-1)}W_{xh}^{(l)}+H_{t-1}^{(l)}W_{hh}^{(l)}+b_h^{(l)})\tag{3}$
$O_t=H_t^{(L)}W_{hq}+b_q\tag{4}$

2.3 区别

由上述可以看出，区别在于深度神经网络将隐藏层 $H_t$ 改变成了 $H_1,H_2,....,H_L]$

2.4 代码

代码

import torch
from torch import nn
from d2l import torch as d2l
import matplotlib.pyplot as plt


batch_size, num_steps = 32, 35
train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)
vocab_size, num_hiddens, num_layers = len(vocab), 256, 2
num_inputs = vocab_size
device = d2l.try_gpu()
lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers)
model = d2l.RNNModel(lstm_layer, len(vocab))
model = model.to(device)
num_epochs, lr = 500, 2
d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)

结果

perplexity 1.0, 195541.7 tokens/sec on cuda:0
time travelleryou can show black is white by argument said filby
travelleryou can show black is white by argument said filby

3. 双向循环神经网络

3.1 结构图

注：双向循环神经网络只是在原来的一个单向隐藏层中新增一个反向的隐藏层来进行传播，具体传播路径如下：

3.2 公式

隐藏层-正向传播
$\mathop{H_t}\limits ^{\rightarrow}=\phi(X_tW_{xh}^{(f)}+\mathop{H_{t-1}}\limits ^{\rightarrow}W_{hh}^{(f)}+b_h^{(f)})\tag 1$
隐藏层-反向传播
$\mathop{H_t}\limits ^{\leftarrow}=\phi(X_tW_{xh}^{(b)}+\mathop{H_{t+1}}\limits ^{\leftarrow}W_{hh}^{(b)}+b_h^{(b)})\tag 2$
输出层
$H_t=[\mathop{H_t}\limits ^{\rightarrow},\mathop{H_t}\limits ^{\leftarrow}]\tag3$
$O_t=H_tW_{hq}+b_q\tag4$

3.3 代码

代码

import torch
from torch import nn
from d2l import torch as d2l
# 加载数据
batch_size, num_steps, device = 32, 35, d2l.try_gpu()
train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)
# 通过设置“bidirective=True”来定义双向LSTM模型
vocab_size, num_hiddens, num_layers = len(vocab), 256, 2
num_inputs = vocab_size
lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers, bidirectional=True)
model = d2l.RNNModel(lstm_layer, len(vocab))
model = model.to(device)
# 训练模型
num_epochs, lr = 500, 1
d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)

结果

perplexity 1.1, 86529.1 tokens/sec on cuda:0
time travellerererererererererererererererererererererererererer
travellerererererererererererererererererererererererererer

在这里插入图片描述