系列文章目录
机器学习笔记——梯度下降、反向传播
机器学习笔记——用pytorch实现线性回归
机器学习笔记——pytorch实现逻辑斯蒂回归Logistic regression
机器学习笔记——多层线性(回归)模型 Multilevel (Linear Regression) Model
深度学习笔记——pytorch构造数据集 Dataset and Dataloader
深度学习笔记——pytorch解决多分类问题 Multi-Class Classification
深度学习笔记——pytorch实现卷积神经网络CNN
深度学习笔记——卷积神经网络CNN进阶
深度学习笔记——循环神经网络 RNN
深度学习笔记——pytorch实现GRU
前言
一、RNN
在全连接网络中,有大量的线性映射关系,于是产生了大量的权重。当层数很多时,即深度神经网络(Deep Neural Networks, DNN),就会有大量的权重。
循环神经网络(Recurrent Neural Networks, RNN),共享权重,主要应用于时序相关的问题,如自然语言。因为会将前一次输出的结果h,加入到下一次的运算中。当前输出由前一次的计算结果h和当前输入x共同影响。
图里的RNN Cell是同一层神经网络,因此每一次的输入x共享神经网络的权重。
RNN Cell计算方式,激活函数通常选用tanh。
在pytorch中RNN Cell的代码,我们需要确定输入x的向量维度,和h的维度
输出hidden的维度和输入hidden的维度相同
在pytorch中RNN的实现代码
可以有多层的结构
二、示例
1.示例
用RNN Cell实现将hello输出为ohlol
先建立一个词库/字典Dictionary,将输入的序列映射为向量。
结构
2.RNN Cell实现代码
#!/user/bin/env python3
# -*- coding: utf-8 -*-
import torch
input_size = 4
hidden_size = 4
batch_size = 1
idx2char = ['e', 'h', 'l', 'o']
x_data = [1, 0, 2, 2, 3] # hello
y_data = [3, 1, 2, 3, 2] # ohlol
one_hot_lookup = [[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data] # 将x_data中的数据映射为矩阵
inputs = torch.Tensor(x_one_hot).view(-1, batch_size, input_size)
labels = torch.LongTensor(y_data).view(-1, 1)
# 神经网络模型
class Net(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size):
super(Net, self).__init__()
self.input_size = input_size
self.hidden_size = hidden_size
self.batch_size = batch_size
self.cell = torch.nn.RNNCell(input_size=self.input_size, hidden_size=self.hidden_size)
def forward(self, inputs, hidden):
hidden = self.cell(inputs, hidden)
return hidden
# 定义初始的hidden
def init_hidden(self):
return torch.zeros(self.batch_size, self.hidden_size)
model = Net(input_size, hidden_size, batch_size)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
for epoch in range(10):
loss = 0
optimizer.zero_grad()
hidden = model.init_hidden()
print('predicted string:', end='')
for input, label in zip(inputs, labels):
hidden = model(input, hidden)
loss += criterion(hidden, label)
_, idx = hidden.max(dim=1)
print(idx2char[idx.item()], end='')
loss.backward()
optimizer.step()
print(',epoch[%d/10] loss=%.4f' % (epoch + 1, loss.item()))
运行结果:
3.RNN代码实现
#!/user/bin/env python3
# -*- coding: utf-8 -*-
import torch
input_size = 4
hidden_size = 4
batch_size = 1
seq_len = 5
idx2char = ['e', 'h', 'l', 'o']
x_data = [1, 0, 2, 2, 3] # hello
y_data = [3, 1, 2, 3, 2] # ohlol
one_hot_lookup = [[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data] # 将x_data中的数据映射为矩阵
inputs = torch.Tensor(x_one_hot).view(seq_len, batch_size, input_size)
labels = torch.LongTensor(y_data)
class Net(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size, num_layers=1):
super(Net, self).__init__()
self.num_layers = num_layers
self.input_size = input_size
self.hidden_size = hidden_size
self.batch_size = batch_size
self.rnn = torch.nn.RNN(input_size=self.input_size, hidden_size=self.hidden_size, num_layers=self.num_layers)
def forward(self, input):
hidden = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
out, _ = self.rnn(input, hidden)
return out.view(-1, self.hidden_size)
model = Net(input_size, hidden_size, batch_size)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
for epoch in range(15):
optimizer.zero_grad()
output = model(inputs)
loss = criterion(output, labels)
loss.backward()
optimizer.step()
_, idx = output.max(dim=1)
idx = idx.data.numpy()
print('predicted:', ''.join([idx2char[x] for x in idx]), end='')
print(',epoch[%d/10] loss=%.4f' % (epoch + 1, loss.item()))
输出结果:
三、embedding
独热向量存在维度高计算难度大、过于稀疏导致浪费资源、难以编码等问题。
因此提出了嵌入层(embedding)
Embedding层,在某种程度上,就是用来降维的,降维的原理就是矩阵乘法。
embedding将高维、稀疏、难以编码的空间映射到低维、稠密、可学习的空间。
embedding的输入是(序列长度,批次数量batchsize),输出是(序列长度,批次数量batchsize,隐层维度)
因此,我们的模型就变成了这样。数据经过embedding层映射,经过RNN计算,最后通过线性层计算并输出结果。
代码:
#!/user/bin/env python3
# -*- coding: utf-8 -*-
import torch
input_size = 4
embedding_size = 10
hidden_size = 8
num_layers = 2
num_class = 4
seq_len = 5
batch_size = 1
idx2char = ['e', 'h', 'l', 'o']
x_data = [1, 0, 2, 2, 3] # hello
y_data = [3, 1, 2, 3, 2] # ohlol
inputs = torch.LongTensor(x_data)
labels = torch.LongTensor(y_data)
class model(torch.nn.Module):
def __init__(self):
super(model, self).__init__()
self.bed = torch.nn.Embedding(input_size, embedding_size)
self.rnn = torch.nn.RNN(input_size=embedding_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
self.fc = torch.nn.Linear(in_features=hidden_size, out_features=num_class)
def forward(self, x):
hidden = torch.zeros(num_layers, x.size(0), hidden_size)
x = self.bed(x)
x, _ = self.rnn(x)
x = self.fc(x)
return x.view(-1, num_class)
net = model()
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(),lr=0.05)
for epoch in range(15):
optimizer.zero_grad()
output = net(inputs)
loss = criterion(output, labels)
loss.backward()
optimizer.step()
_, idx = output.max(dim=1)
idx = idx.data.numpy()
print('predicted:', ''.join([idx2char[x] for x in idx]), end='')
print(',epoch[%d/15] loss=%.4f' % (epoch + 1, loss.item()))
结果: