【PyTorch】刘二大人的Pytorch深度学习实践学习笔记

松堂飞萤

已于 2024-10-08 11:52:52 修改

阅读量1.3k

点赞数 31

分类专栏：深度学习代码深度学习基础文章标签：深度学习 pytorch 笔记

于 2024-07-11 20:21:32 首次发布

本文链接：https://blog.csdn.net/weixin_49119476/article/details/140361184

版权

深度学习基础同时被 2 个专栏收录

3 篇文章 1 订阅

订阅专栏

深度学习代码

1 篇文章 0 订阅

订阅专栏

【PyTorch】刘二大人的Pytorch深度学习实践学习笔记

写在前面

博主在去年暑假已经看过一次刘二大人的《Pytorch深度学习实践》，但是没有做任何笔记。现在突发奇想打算再过一次视频来夯实一下代码基础，顺便做下笔记来供自己以后复习使用。

01. Overview

之前基于rule的系统是通过人工设定规则实现，到了传统的机器学习方法则是通过手动设计特征（如把一段语音变成向量），最后通过一个mapping函数对应到输出。

表示学习则希望feature的提取也是通过学习的方法来得到（而不是手工获取），但是其中feature的提取和mapping映射是分开完成的。

深度学习是一种端到端的（end2end）的算法，之前的表示学习可能需要一个设计特征提取器来获取特征，而深度学习直接设计一个很大的模型完成数据从输入到输出的整体过程。

SVM的劣势。

02. 线性模型

四步：数据集，设计模型，训练模型，推理（预测）

训练集train set：训练模型的数据集；

测试集test set：预测步骤，测试模型的准确性。

训练集可能会存在过拟合问题，比如学习到图形的噪声，我们要求模型有好的泛化能力。比如训练集的猫狗图像可能是艺术照片（美颜加猫狗在中间），而实际上可能用户上传后照片是随手一拍。由此我们将训练集分为两块，将另外一块用于评估，称之为验证集valid set。

评估模型，称之为损失（loss）

loss function：对于一个样本的损失

cost function：对于所有样本（整个训练集）的平均损失。如图所示是MSE损失。

03. 梯度下降算法

观察法：维度过高时候过于困难，程序复杂度和数据量过高；

分治法：不是凸函数的情况下容易不容易得到最优值。而且维度过高也难以划分。

由此提出梯度下降算法。贪心算法，容易陷入局部最优点。但是在深度网络里面，人们发现局部最优点比较少，所以可以使用。

但是深度学习里面鞍点比较多，（比如马鞍面），也是需要解决的最大的问题。

随机梯度下降：用一个数据（或者单个样本）代替原来所有的数据计算损失。这样有助于跳过鞍点。

但是随机梯度下降无法实现并行，因为权值w有依赖关系。

在深度学习中选择折中，因为梯度下降效率高但性能差，随机梯度下降效率低但性能好。所以采用批次的随机梯度下降法。

04. 反向传播

理论

在之前讲解过程中我们很容易求得梯度的解析式，如下所示。

但是在深度学习中再求取解析式的话，过于复杂，而且嵌套了很多复合函数，例子如下。由此设计了反向传播算法，利用计算图通过链式法则来得到梯度。

关于激活函数的作用，主要是得到非线性的变化。

链式法则的步骤：

前馈获取loss。

局部梯度。这儿的f是输入x和权重w的函数。

获取loss对于z的偏导。（从前面的传播过程获得）

反向传播，从而获得L对x、w的导数，其中z对x、对w的导数，是在计算f的函数的时候算出的。

举例说明：

正向得到loss为1后，通过正向计算过程中算得的梯度，反向传播获取最终的偏导。

这儿的loss一般会保存，用来评估模型。

代码

forward函数里面的乘法被重载了。
w需要计算梯度，后面的也需要计算梯度（图中蓝色方框）。
调用一次loss，动态产生计算图。

#训练代码
for epoch in range(100):
    for x,y in zip(x_data,y_data):
        l = loss(x,y) #前向传播，得到loss，构造计算图。同时将梯度存到w中
        l.backward()#反向传播，释放计算图（每次反向传播释放计算图可以方便drop out)
        
        print("grad:",x,y,w.grad.item())#w.grad.item()原理同w.grad.data
        w.data = w.data - 0.01 * w.grad.data#w.grad是一个tensor，如果直接用它是在构造计算图，因此需要用.data，这样可以不需要构造计算图
        w.grad.data.zero_()#梯度清零，因为梯度在反向传播中是累加的，需要手动清零

05. pytorch实现线性回归

深度学习四步。

广播机制，如下所示，这里的乘是矩阵元素对应位置相乘，不是矩阵乘。

在nn.Linear中，包括了权重w和偏置b。它也是继承自Module，所以可以自动反向传播求导。

对应完整代码如下所示：

import numpy as np
import torch

x_data = torch.Tensor([[1.0],[2.0],[3.0]])
y_data = torch.Tensor([[2.0],[4.0],[6.0]])

#nn.Module是所有神经网络的父类
class LinearModel(torch.nn.Module):
#至少需要两个函数：构造函数和forward函数。前者用来初始化，后者用来计算前缀forward过程
#如果觉得pytorch中封装的反向传播算法效率不高，可以自己写function类
    def __init__(self):
        super(LinearModel,self).__init__()#调用父类的构造函数
        self.linear = torch.nn.Linear(1,1)#linear是一个nn.Linear的对象

    def forward(self,x):
        y_pred = self.linear(x)#可调用的对象(__call__)
        return y_pred

def train(model):
    criterion = torch.nn.MSELoss(size_average=False)  # MSEloss继承自nn.Module
    optimizer = torch.optim.SGD(model.parameters(), lr=0.05)  # 不会构建计算图，model.parameters()记录了需要更新的参数
    for epoch in range(100):
        #step 1：得到y_hat
        y_pred = model(x_data)
        #step 2: 计算loss
        loss = criterion(y_pred, y_data)
        print(epoch, loss.item())
        #step 3：求取梯度
        optimizer.zero_grad()#梯度清零
        loss.backward()#反向传播
        #step 4: 权重更新
        optimizer.step()#更新

    print('w= ',model.linear.weight.item())
    print('b= ',model.linear.bias.item())

    x_test = torch.Tensor([[4.0]])
    y_test = model(x_test)
    print('y_pred = ',y_test.data)

if __name__ == '__main__':
    model = LinearModel()
    train(model)

06-08. Logistic Regression

sigmoid函数

下面都是sigmoid函数，但是由于logistic function： $\frac{1}{1+e^{-x}}$ 比较出名，因此后面sigmoid函数便就是logistic function $\frac{1}{1+e^{-x}}$ 。

交叉熵损失：输出的是分布，而不是实数距离。

8个feature为例，如下。

mini-batch实例

epoch：所有样本经过一次前馈和反向传播，便为一个epoch。

batch_size：进行一次前馈和反向传播的样本数量。

iteration：每一次epoch执行了多少个batch_size。

Dataset需要满足：1.下标访问； 2.数据集长度。

由此dataloader可以对dataset进行shuffle，然后批次输入并可以利用for循环。

详细代码如下。

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset#Dataset是一个抽象类
from torch.utils.data import DataLoader#帮助进行shuffle等操作


class MyDataset(Dataset):
    def __init__(self,file_path):
        xy = np.loadtxt(file_path,delimiter=',',dtype=np.float32)
        self.len = xy.shape[0]#(N,9)
        self.x_data = torch.from_numpy(xy[:,:-1])#取前八列
        self.y_data = torch.from_numpy(xy[:,[-1]])#取最后一列，并取矩阵形式

    def __getitem__(self, index):
        return self.x_data[index],self.y_data[index]

    def __len__(self):#len()
        return self.len


class LogisticRegressionModel(torch.nn.Module):
    def __init__(self):
        super(LogisticRegressionModel, self).__init__()
        self.FFN = nn.Sequential(
            nn.Linear(8,4),
            nn.Sigmoid(),
            nn.Linear(4,2),
            nn.Sigmoid(),
            nn.Linear(2,1),
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.FFN(x)


def train(model):
    criterion = torch.nn.BCELoss(reduction='mean')
    optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
    for epoch in range(100):
        for i,data in enumerate(train_loader,0):
            #prepare data
            inputs,labels = data
            #forward
            y_pred=model(inputs)
            loss = criterion(y_pred,labels)
            print(epoch,i,loss.item())
            #backward
            optimizer.zero_grad()
            loss.backward()
            #update
            optimizer.step()


if __name__ == '__main__':
    #load data
    dataset = MyDataset('data/diabetes/diabetes.csv.gz')
    train_loader = DataLoader(dataset=dataset,batch_size=32,shuffle=True,num_workers=2)
    #model
    model = LogisticRegressionModel()
    train(model)

作业

后续再补。。

09. 多分类问题

softmax函数

softmax实现多分类：保证概率和为1，每类的概率大于等于0。

举例如下。

交叉熵损失函数，如下所示。NLLLoss需要要求先对Y_hat求对数。

为方便，利用CrossEntrpyLoss，将Softmax、求对数过程、NLLLoss全部封装在其中。

这时候要求y（label）是LongTensor类型。

代码对应举例如下所示：

MNIST数据集

详细代码如下。

import torch
import torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

def load_data():
    transform = transforms.Compose([
        transforms.ToTensor(),  # 把0-255转为0-1的张量，（28,28）-》(1,28,28)，即 channel * w * h
        transforms.Normalize((0.1307,), (0.3081,))  # 均值 mean 和标准差 std
    ])
    train_set = datasets.MNIST(root='./data/mnist', train=True, download=True, transform=transform)
    test_set = datasets.MNIST(root='./data/mnist', train=False, download=True, transform=transform)
    return train_set, test_set

class Mymodel(nn.Module):
    def __init__(self):
        super(Mymodel, self).__init__()
        self.FFN = nn.Sequential(
            nn.Linear(28*28, 512), nn.LeakyReLU(),
            nn.Linear(512, 256), nn.LeakyReLU(),
            nn.Linear(256, 128), nn.LeakyReLU(),
            nn.Linear(128, 64), nn.LeakyReLU(),
            nn.Linear(64, 10)
        )

    def forward(self, x):
        x = x.view(-1, 784)  # -1 表示自动计算第一个维度，784 是一张图片大小为 28 * 28
        return self.FFN(x)

def train_valid(model, train_loader, test_loader):
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

    for epoch in range(10):
        model.train()  # 确保模型处于训练模式
        running_loss = 0.0
        for batch_idx, data in enumerate(train_loader):
            inputs, targets = data
            optimizer.zero_grad()

            outputs = model(inputs)
            loss = criterion(outputs, targets)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            if batch_idx % 300 == 299:
                print("[{:d}, {:5d}] loss: {:.3f}".format(epoch + 1, batch_idx + 1, running_loss / 300))
                running_loss = 0.0
        vaild(model, test_loader)

def vaild(model, test_loader):
    correct = 0
    total = 0
    model.eval()  # 确保模型处于评估模式
    with torch.no_grad():
        for data in test_loader:
            images, labels = data
            outputs = model(images)  # (batch_size, 10)
            _, predicted = torch.max(outputs.data, dim=1)  # 返回最大值和最大值的下标
            total += labels.size(0)  # (batch_size, 1)
            correct += (predicted == labels).sum().item()
    print('Accuracy on test set: %d %%' % (100 * correct / total))

if __name__ == '__main__':
    batch_size = 64
    train_set, test_set = load_data()

    train_loader = DataLoader(train_set, shuffle=True, batch_size=batch_size)
    test_loader = DataLoader(test_set, shuffle=False, batch_size=batch_size)
    model = Mymodel()

    train_valid(model, train_loader, test_loader)

PS:这儿出现了一个问题，原函数名叫test，让pycharm误以为是测试用例，修改为valid后正常。

输出结果如下。

[1,   300] loss: 2.234
[1,   600] loss: 1.042
[1,   900] loss: 0.445
Accuracy on test set: 89 %
[2,   300] loss: 0.327
[2,   600] loss: 0.284
[2,   900] loss: 0.234
Accuracy on test set: 93 %
[3,   300] loss: 0.197
[3,   600] loss: 0.180
[3,   900] loss: 0.164
Accuracy on test set: 95 %
[4,   300] loss: 0.134
[4,   600] loss: 0.133
[4,   900] loss: 0.126
Accuracy on test set: 96 %
[5,   300] loss: 0.107
[5,   600] loss: 0.098
[5,   900] loss: 0.099
Accuracy on test set: 96 %
[6,   300] loss: 0.079
[6,   600] loss: 0.080
[6,   900] loss: 0.081
Accuracy on test set: 97 %
[7,   300] loss: 0.065
[7,   600] loss: 0.066
[7,   900] loss: 0.065
Accuracy on test set: 97 %
[8,   300] loss: 0.050
[8,   600] loss: 0.054
[8,   900] loss: 0.055
Accuracy on test set: 97 %
[9,   300] loss: 0.043
[9,   600] loss: 0.042
[9,   900] loss: 0.043
Accuracy on test set: 97 %
[10,   300] loss: 0.034
[10,   600] loss: 0.034
[10,   900] loss: 0.034
Accuracy on test set: 97 %

作业

后续再补。。

10. 卷积神经网络(CNN)基础

卷积理论

特征提取层：卷积和下采样。

下采样：减少数据量，降低运算需求。

单通道卷积过程如下所示。

多通道的卷积过程。

例子如下，输入通道数=一个卷积核的通道数，卷积核的数量等于输出的通道数。这里有一个卷积核，对应输出数据的通道数为1（银白色矩阵），一个卷积核有三个通道，对应输入数据的通道数为3。

由此可以构造卷积核。

卷积操作（padding等）

padding：对输入填充圈数，来改变输出的宽和高。

如图所示，padding=1。

stride：步长。卷积核的中心步长。

MaxPooling：最大池化，通道数量不变，大小改变。

代码

关于GPU迁移：首先指定device，然后对于model和数据都迁移到GPU即可。

import torch
import torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

batch_size = 64

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("torch.cuda.is_available()",torch.cuda.is_available())

def load_data():
    transform = transforms.Compose([
        transforms.ToTensor(),  # 把0-255转为0-1的张量，（28,28）-》(1,28,28)，即 channel * w * h
        transforms.Normalize((0.1307,), (0.3081,))  # 均值 mean 和标准差 std
    ])
    train_set = datasets.MNIST(root='./data/mnist', train=True, download=True, transform=transform)
    test_set = datasets.MNIST(root='./data/mnist', train=False, download=True, transform=transform)
    return train_set, test_set

class CNNmodel(nn.Module):
    def __init__(self):
        super(CNNmodel, self).__init__()
        self.CNN = nn.Sequential(
            nn.Conv2d(1,8,kernel_size=5),nn.MaxPool2d(2),nn.ReLU(),
            nn.Conv2d(8,16,kernel_size=3),nn.MaxPool2d(2),nn.ReLU(),
            nn.Conv2d(16,24,kernel_size=3),
        )
        self.FC = nn.Sequential(
            nn.Linear(216,128),nn.ReLU(),
            nn.Linear(128,64),nn.ReLU(),
            nn.Linear(64,10)
        )
    def forward(self, x):
        x = self.CNN(x)
        x = x.view(x.size(0),-1)
        x = self.FC(x)
        return x
    
def train_valid(model, train_loader, test_loader):
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

    for epoch in range(20):
        model.train()  # 确保模型处于训练模式
        running_loss = 0.0
        for batch_idx, data in enumerate(train_loader):
            inputs, targets = data
            inputs, targets = inputs.to(device), targets.to(device)
            optimizer.zero_grad()

            outputs = model(inputs)
            loss = criterion(outputs, targets)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            if batch_idx % 300 == 299:
                print("[{:d}, {:5d}] loss: {:.3f}".format(epoch + 1, batch_idx + 1, running_loss / 300))
                running_loss = 0.0
        vaild(model, test_loader)

def vaild(model, test_loader):
    correct = 0
    total = 0
    model.eval()  # 确保模型处于评估模式
    with torch.no_grad():
        for data in test_loader:
            images, labels = data
            images, labels = images.to(device), labels.to(device)
            outputs = model(images)  # (batch_size, 10)
            _, predicted = torch.max(outputs.data, dim=1)  # 返回最大值和最大值的下标
            total += labels.size(0)  # (batch_size, 1)
            correct += (predicted == labels).sum().item()
    print('Accuracy on test set: %d %% [%d/%d]' % (100 * correct / total, correct, total))

if __name__ == '__main__':
    train_set, test_set = load_data()

    train_loader = DataLoader(train_set, shuffle=True, batch_size=batch_size)
    test_loader = DataLoader(test_set, shuffle=False, batch_size=batch_size)
    model = CNNmodel()
    model.to(device)
    train_valid(model, train_loader, test_loader)

对应输出如下：

torch.cuda.is_available() True
[1,   300] loss: 1.807
[1,   600] loss: 0.335
[1,   900] loss: 0.211
Accuracy on test set: 94 % [9497/10000]
[2,   300] loss: 0.149
[2,   600] loss: 0.129
[2,   900] loss: 0.116
Accuracy on test set: 97 % [9720/10000]
[3,   300] loss: 0.097
[3,   600] loss: 0.090
[3,   900] loss: 0.082
Accuracy on test set: 97 % [9745/10000]
[4,   300] loss: 0.076
[4,   600] loss: 0.068
[4,   900] loss: 0.071
Accuracy on test set: 98 % [9828/10000]
[5,   300] loss: 0.060
[5,   600] loss: 0.058
[5,   900] loss: 0.059
Accuracy on test set: 98 % [9832/10000]
[6,   300] loss: 0.054
[6,   600] loss: 0.054
[6,   900] loss: 0.050
Accuracy on test set: 97 % [9796/10000]
[7,   300] loss: 0.047
[7,   600] loss: 0.045
[7,   900] loss: 0.048
Accuracy on test set: 98 % [9858/10000]
[8,   300] loss: 0.043
[8,   600] loss: 0.040
[8,   900] loss: 0.041
Accuracy on test set: 98 % [9858/10000]
[9,   300] loss: 0.035
[9,   600] loss: 0.043
[9,   900] loss: 0.038
Accuracy on test set: 98 % [9846/10000]
[10,   300] loss: 0.033
[10,   600] loss: 0.034
[10,   900] loss: 0.038
Accuracy on test set: 98 % [9879/10000]
[11,   300] loss: 0.032
[11,   600] loss: 0.028
[11,   900] loss: 0.034
Accuracy on test set: 98 % [9875/10000]
[12,   300] loss: 0.025
[12,   600] loss: 0.031
[12,   900] loss: 0.030
Accuracy on test set: 98 % [9861/10000]
[13,   300] loss: 0.025
[13,   600] loss: 0.026
[13,   900] loss: 0.029
Accuracy on test set: 98 % [9864/10000]
[14,   300] loss: 0.025
[14,   600] loss: 0.023
[14,   900] loss: 0.026
Accuracy on test set: 98 % [9870/10000]
[15,   300] loss: 0.023
[15,   600] loss: 0.023
[15,   900] loss: 0.023
Accuracy on test set: 98 % [9885/10000]
[16,   300] loss: 0.020
[16,   600] loss: 0.023
[16,   900] loss: 0.021
Accuracy on test set: 98 % [9888/10000]
[17,   300] loss: 0.019
[17,   600] loss: 0.021
[17,   900] loss: 0.018
Accuracy on test set: 98 % [9890/10000]
[18,   300] loss: 0.018
[18,   600] loss: 0.019
[18,   900] loss: 0.017
Accuracy on test set: 98 % [9865/10000]
[19,   300] loss: 0.017
[19,   600] loss: 0.019
[19,   900] loss: 0.020
Accuracy on test set: 98 % [9885/10000]
[20,   300] loss: 0.016
[20,   600] loss: 0.018
[20,   900] loss: 0.016
Accuracy on test set: 98 % [9885/10000]

11. 卷积神经网络(CNN)进阶

GoogleNet

1*1卷积核作用：

跨越输入的不同通道融合输入通道的值，完成信息融合。
改变通道数量。
降低运算量。

关于Inception Module：考虑不知道选哪个卷积核好，由此设计多个分支，可以得到最好的情况。

最后代码如下。

class InceptionA(nn.Module):
    def __init__(self,in_channels):
        super(InceptionA, self).__init__()
        self.branch1_1 = nn.Conv2d(in_channels,16,kernel_size=1)
        self.branch5_5 = nn.Sequential(
            nn.Conv2d(in_channels,16,kernel_size=1),
            nn.Conv2d(16,24,kernel_size=5,padding=2),
        )
        self.branch3_3 = nn.Sequential(
            nn.Conv2d(in_channels,16,kernel_size=1),
            nn.Conv2d(16,24,kernel_size=3,padding=1),
            nn.Conv2d(24,24,kernel_size=3,padding=1),
        )
        self.branch_pool = nn.Sequential(
            nn.AvgPool2d(kernel_size=3, stride=1, padding=1),
            nn.Conv2d(in_channels, 24, kernel_size=1),
        )

    def forward(self,x):
        branch1_1 = self.branch1_1(x)
        branch5_5 = self.branch5_5(x)
        branch3_3 = self.branch3_3(x)
        branch_pool = self.branch_pool(x)

        output = [branch1_1,branch5_5,branch3_3,branch_pool]
        return torch.cat(output,dim=1)

class Google_Net(nn.Module):
    def __init__(self):
        super(Google_Net, self).__init__()
        self.FNN = nn.Sequential(
            nn.Conv2d(1, 10, kernel_size=5),nn.MaxPool2d(2),nn.ReLU(),
            InceptionA(10),
            nn.Conv2d(88,20,kernel_size=5),nn.MaxPool2d(2),nn.ReLU(),
            InceptionA(20),
        )
        self.FFN = nn.Sequential(
            nn.Linear(1408,512),nn.ReLU(),
            nn.Linear(512,256),nn.ReLU(),
            nn.Linear(256,128),nn.ReLU(),
            nn.Linear(128,10)
        )

    def forward(self,x):
        x = self.FNN(x)
        x = x.view(x.size(0),-1)
        return self.FFN(x)

跑10.CNN中的例子，结果如下：

torch.cuda.is_available() True
[1,   300] loss: 2.301
[1,   600] loss: 2.297
[1,   900] loss: 2.004
Accuracy on test set: 67 % [6775/10000]
[2,   300] loss: 0.486
[2,   600] loss: 0.275
[2,   900] loss: 0.197
Accuracy on test set: 95 % [9534/10000]
[3,   300] loss: 0.147
[3,   600] loss: 0.124
[3,   900] loss: 0.112
Accuracy on test set: 96 % [9672/10000]
[4,   300] loss: 0.095
[4,   600] loss: 0.080
[4,   900] loss: 0.078
Accuracy on test set: 97 % [9772/10000]
[5,   300] loss: 0.066
[5,   600] loss: 0.066
[5,   900] loss: 0.057
Accuracy on test set: 98 % [9814/10000]
[6,   300] loss: 0.051
[6,   600] loss: 0.050
[6,   900] loss: 0.051
Accuracy on test set: 98 % [9841/10000]
[7,   300] loss: 0.041
[7,   600] loss: 0.043
[7,   900] loss: 0.043
Accuracy on test set: 98 % [9849/10000]
[8,   300] loss: 0.037
[8,   600] loss: 0.035
[8,   900] loss: 0.038
Accuracy on test set: 98 % [9860/10000]
[9,   300] loss: 0.029
[9,   600] loss: 0.033
[9,   900] loss: 0.032
Accuracy on test set: 98 % [9859/10000]
[10,   300] loss: 0.025
[10,   600] loss: 0.029
[10,   900] loss: 0.027
Accuracy on test set: 98 % [9879/10000]
[11,   300] loss: 0.020
[11,   600] loss: 0.024
[11,   900] loss: 0.026
Accuracy on test set: 98 % [9884/10000]
[12,   300] loss: 0.019
[12,   600] loss: 0.019
[12,   900] loss: 0.024
Accuracy on test set: 98 % [9872/10000]
[13,   300] loss: 0.016
[13,   600] loss: 0.019
[13,   900] loss: 0.019
Accuracy on test set: 98 % [9879/10000]
[14,   300] loss: 0.013
[14,   600] loss: 0.015
[14,   900] loss: 0.018
Accuracy on test set: 98 % [9875/10000]
[15,   300] loss: 0.011
[15,   600] loss: 0.014
[15,   900] loss: 0.019
Accuracy on test set: 98 % [9859/10000]
[16,   300] loss: 0.012
[16,   600] loss: 0.010
[16,   900] loss: 0.013
Accuracy on test set: 98 % [9883/10000]
[17,   300] loss: 0.009
[17,   600] loss: 0.010
[17,   900] loss: 0.012
Accuracy on test set: 98 % [9882/10000]
[18,   300] loss: 0.009
[18,   600] loss: 0.010
[18,   900] loss: 0.013
Accuracy on test set: 98 % [9882/10000]
[19,   300] loss: 0.008
[19,   600] loss: 0.010
[19,   900] loss: 0.011
Accuracy on test set: 98 % [9886/10000]
[20,   300] loss: 0.005
[20,   600] loss: 0.008
[20,   900] loss: 0.007
Accuracy on test set: 98 % [9880/10000]

ResNet

代码如下：

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super(ResidualBlock, self).__init__()
        self.block = nn.Sequential(
            nn.Conv2d(channels, channels, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        )

    def forward(self, x):
        return F.relu(x+ self.block(x))

class ResNet(nn.Module):
    def __init__(self):
        super(ResNet, self).__init__()
        self.RN = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5),nn.MaxPool2d(2),nn.ReLU(),
            ResidualBlock(16),
            nn.Conv2d(16, 32, kernel_size=5),nn.MaxPool2d(2),nn.ReLU(),
            ResidualBlock(32),
        )
        self.FCN = nn.Sequential(
            nn.Linear(512,128),nn.ReLU(),
            nn.Linear(128,10)
        )

    def forward(self, x):
        x = self.RN(x)
        x = x.view(x.size(0), -1)
        x = self.FCN(x)
        return x

跑10.CNN中的例子，结果如下：

torch.cuda.is_available() True
[1,   300] loss: 0.818
[1,   600] loss: 0.189
[1,   900] loss: 0.127
Accuracy on test set: 97 % [9725/10000]
[2,   300] loss: 0.092
[2,   600] loss: 0.089
[2,   900] loss: 0.074
Accuracy on test set: 98 % [9803/10000]
[3,   300] loss: 0.067
[3,   600] loss: 0.060
[3,   900] loss: 0.054
Accuracy on test set: 98 % [9860/10000]
[4,   300] loss: 0.046
[4,   600] loss: 0.050
[4,   900] loss: 0.045
Accuracy on test set: 98 % [9858/10000]
[5,   300] loss: 0.039
[5,   600] loss: 0.041
[5,   900] loss: 0.039
Accuracy on test set: 98 % [9881/10000]
[6,   300] loss: 0.034
[6,   600] loss: 0.033
[6,   900] loss: 0.032
Accuracy on test set: 98 % [9878/10000]
[7,   300] loss: 0.031
[7,   600] loss: 0.025
[7,   900] loss: 0.029
Accuracy on test set: 98 % [9876/10000]
[8,   300] loss: 0.023
[8,   600] loss: 0.026
[8,   900] loss: 0.025
Accuracy on test set: 98 % [9886/10000]
[9,   300] loss: 0.023
[9,   600] loss: 0.022
[9,   900] loss: 0.023
Accuracy on test set: 98 % [9873/10000]
[10,   300] loss: 0.018
[10,   600] loss: 0.019
[10,   900] loss: 0.022
Accuracy on test set: 99 % [9900/10000]
[11,   300] loss: 0.015
[11,   600] loss: 0.018
[11,   900] loss: 0.017
Accuracy on test set: 98 % [9890/10000]
[12,   300] loss: 0.014
[12,   600] loss: 0.015
[12,   900] loss: 0.015
Accuracy on test set: 98 % [9881/10000]
[13,   300] loss: 0.014
[13,   600] loss: 0.012
[13,   900] loss: 0.013
Accuracy on test set: 98 % [9898/10000]
[14,   300] loss: 0.011
[14,   600] loss: 0.012
[14,   900] loss: 0.012
Accuracy on test set: 98 % [9892/10000]
[15,   300] loss: 0.009
[15,   600] loss: 0.009
[15,   900] loss: 0.012
Accuracy on test set: 98 % [9899/10000]
[16,   300] loss: 0.006
[16,   600] loss: 0.009
[16,   900] loss: 0.012
Accuracy on test set: 99 % [9900/10000]
[17,   300] loss: 0.007
[17,   600] loss: 0.010
[17,   900] loss: 0.008
Accuracy on test set: 99 % [9906/10000]
[18,   300] loss: 0.008
[18,   600] loss: 0.008
[18,   900] loss: 0.004
Accuracy on test set: 98 % [9888/10000]
[19,   300] loss: 0.005
[19,   600] loss: 0.007
[19,   900] loss: 0.006
Accuracy on test set: 98 % [9893/10000]
[20,   300] loss: 0.006
[20,   600] loss: 0.006
[20,   900] loss: 0.005
Accuracy on test set: 98 % [9897/10000]

12. 循环神经网络（RNN）基础

理论

用RNN处理带有序列的数据，最典型比如自然语言。

RNN cell：本质上是一个线性层。如下图右边所示，这儿的RNN Cell是同一个线性层。

h0是先验，当没有先验时可以全设置为0。h1-h4分别对应x1-x4的输出，其中 $h_{i-1}和x_i$ 作为输入参与到 $h_i$ 的运算当中。

RNN Cell实现公式为 $tanh(W_{ih}x_t+b_{ih}+W_{hh}h_{t-1}+b_{hh})$ ，可以利用矩阵性质合并为一个计算，见下图上方的红字。

关于维度，如下图所示。

其中seqLen是句子的长度（x1-x3)，inputSize是一个x的特征数量，hiddenSize是输出维度。

代码基础

pytorch实现了RNN，所以我们可以不用手写RNN cell然后实现for循环，而是直接调用即可。

其中num_layers是指h1上方可以追加多层RNN Cell。

设置batch_first来交换seqLen和batchSize的顺序。

举例由hello->ohlol的RNN。

代码（RNNcell）如下所示。

import torch
import torch.nn as nn

input_size = 4
hidden_size = 4
batch_size = 1

idx2char =['e','h','l','o']
x_data = [1,0,2,2,3]
y_data = [3,1,2,3,2]

one_hot_lookup = [
    [1,0,0,0],
    [0,1,0,0],
    [0,0,1,0],
    [0,0,0,1]
]

def load_data():
    x_one_hot =  [one_hot_lookup[x] for x in x_data] #(seq,input_size)
    inputs = torch.tensor(x_one_hot,dtype=torch.float32).view(-1,batch_size,input_size)#(seq,batch_size,input_size)
    labels = torch.tensor(y_data,dtype=torch.long).view(-1,batch_size)
    return inputs, labels

class Net(nn.Module):
    def __init__(self,input_size,hidden_size,batch_size):
        super(Net, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.batch_size = batch_size
        self.rnncell = torch.nn.RNNCell(input_size=self.input_size,hidden_size=self.hidden_size)

    def forward(self,input,hidden):
        return self.rnncell(input,hidden)

    def init_hidden(self):
        return torch.zeros(self.batch_size,self.hidden_size)

def train(model,inputs,labels):
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
    for epoch in range(15):
        loss = 0
        optimizer.zero_grad()
        hidden = model.init_hidden()
        print("Predicted string: ",end='')
        for data in zip(inputs,labels): #(seqsize,batchsize,inputsize) , (seqsize,1)
            input,label = data#(batchsize,inputsize) ,(1)
            hidden = model(input,hidden)
            loss+=criterion(hidden,label)#计算所有seq的loss和
            _,idx = hidden.max(dim=1)
            print(idx2char[idx.item()],end='')
        loss.backward()
        optimizer.step()
        print(', Epoch[{}/15] loss={:4f}'.format(epoch+1,loss.item()))


if __name__ == '__main__':
    inputs, labels = load_data()
    model = Net(input_size=input_size,hidden_size=hidden_size,batch_size=batch_size)
    train(model,inputs,labels)

对应结果如下所示。

Predicted string: oeeeh, Epoch[1/15] loss=6.824417
Predicted string: ohloh, Epoch[2/15] loss=5.685897
Predicted string: ohloh, Epoch[3/15] loss=4.911095
Predicted string: ohlol, Epoch[4/15] loss=4.359684
Predicted string: ohlol, Epoch[5/15] loss=3.984190
Predicted string: ohlol, Epoch[6/15] loss=3.704204
Predicted string: ohlol, Epoch[7/15] loss=3.451805
Predicted string: ohlol, Epoch[8/15] loss=3.252563
Predicted string: ohlol, Epoch[9/15] loss=3.092643
Predicted string: ohlol, Epoch[10/15] loss=2.926537
Predicted string: ohlol, Epoch[11/15] loss=2.780157
Predicted string: ohlol, Epoch[12/15] loss=2.623175
Predicted string: ohlol, Epoch[13/15] loss=2.471442
Predicted string: ohlol, Epoch[14/15] loss=2.374695
Predicted string: ohlol, Epoch[15/15] loss=2.291214

代码（RNN）如下所示。

import torch
import torch.nn as nn

input_size = 4
hidden_size = 4
batch_size = 1
num_layers = 2

idx2char =['e','h','l','o']
x_data = [1,0,2,2,3]
y_data = [3,1,2,3,2]

one_hot_lookup = [
    [1,0,0,0],
    [0,1,0,0],
    [0,0,1,0],
    [0,0,0,1]
]

def load_data(case):
    x_one_hot =  [one_hot_lookup[x] for x in x_data] #(seq,input_size)
    inputs = torch.tensor(x_one_hot,dtype=torch.float32).view(-1,batch_size,input_size)#(seq,batch_size,input_size)
    if case=="RNN":
        labels = torch.tensor(y_data,dtype=torch.long) #(seq*batch_size)
    elif case=="RNNcell":
        labels = torch.tensor(y_data,dtype=torch.long).view(-1,batch_size) #(seq,batch_size)
    else:
        exit(-1)
    return inputs, labels

class RNN(nn.Module):
    def __init__(self,input_size,hidden_size,batch_size,num_layers =1):
        super(RNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.batch_size = batch_size
        self.num_layers = num_layers
        self.RNN = nn.RNN(input_size=self.input_size,hidden_size=self.hidden_size,num_layers=self.num_layers)

    def forward(self,x):
        hidden = torch.zeros(self.num_layers,self.batch_size,self.hidden_size)
        output,_ = self.RNN(x,hidden)
        return output.view(-1,self.hidden_size) #(seqLen*batchsize,hiddenSize),变成矩阵方便求交叉熵

def train_RNN(model,inputs,labels):
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
    for epoch in range(15):
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs,labels)
        loss.backward()
        optimizer.step()

        _,idx = outputs.max(dim=1)
        idx = idx.data.numpy()
        print('predicted string: ',''.join(idx2char[x] for x in idx),end = '')
        print(', Epoch[{}/15] loss={:4f}'.format(epoch + 1, loss.item()))

if __name__ == '__main__':
    inputs,labels = load_data("RNN")
    model = RNN(input_size=input_size,hidden_size=hidden_size,batch_size=batch_size,num_layers=num_layers)
    train_RNN(model,inputs,labels)

对应结果如下所示。

predicted string:  hllll, Epoch[1/15] loss=1.418129
predicted string:  hhllh, Epoch[2/15] loss=1.239050
predicted string:  hhllh, Epoch[3/15] loss=1.109292
predicted string:  hhllh, Epoch[4/15] loss=1.000548
predicted string:  ohloh, Epoch[5/15] loss=0.898822
predicted string:  ohlol, Epoch[6/15] loss=0.809355
predicted string:  ohlol, Epoch[7/15] loss=0.737357
predicted string:  ohlol, Epoch[8/15] loss=0.682224
predicted string:  ohlol, Epoch[9/15] loss=0.636641
predicted string:  ohlol, Epoch[10/15] loss=0.593457
predicted string:  ohlol, Epoch[11/15] loss=0.552632
predicted string:  ohlol, Epoch[12/15] loss=0.517946
predicted string:  ohlol, Epoch[13/15] loss=0.490962
predicted string:  ohlol, Epoch[14/15] loss=0.470139
predicted string:  ohlol, Epoch[15/15] loss=0.453247

embedding

优点：低维，稠密，从数据中学习

代码如下所示：

import torch
import torch.nn as nn

input_size = 4
hidden_size = 8
batch_size = 1
num_layers = 2
num_class = 4
embedding_size = 10
seq_len =5

idx2char =['e','h','l','o']
x_data = [1,0,2,2,3]
y_data = [3,1,2,3,2]
inputs = torch.LongTensor(x_data) #(batch,seq)
labels = torch.LongTensor(y_data)#(batch*seq)


class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.RNN = nn.Sequential(
            nn.Embedding(input_size, embedding_size),#(batch,seq,embeddingsize)
            nn.RNN(input_size=embedding_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True),
        )
        self.fc = nn.Linear(hidden_size, num_class)

    def forward(self, x):
        hidden = torch.zeros(num_layers, x.size(0), hidden_size)
        x,_ = self.RNN(x)#(batch,seq,hiddensize)
        x = self.fc(x)#(batch,seq,num_class)
        return x.view(-1,num_class)#(batch*seq,num_class)

def train(model):
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.05)

    for epoch in range(15):
        optimizer.zero_grad()
        y_pred = model(inputs)
        loss = criterion(y_pred, labels)
        loss.backward()
        optimizer.step()

        _,idx = y_pred.max(1)
        idx = idx.data.numpy()
        print('predicted string: ', ''.join(idx2char[x] for x in idx), end='')
        print(', Epoch[{}/15] loss={:4f}'.format(epoch + 1, loss.item()))


if __name__ == '__main__':
    model = MyNet()
    train(model)

对应结果如下所示：

predicted string:  eeeee, Epoch[1/15] loss=1.680657
predicted string:  hhhhl, Epoch[2/15] loss=1.320530
predicted string:  ohlll, Epoch[3/15] loss=1.074882
predicted string:  ohlll, Epoch[4/15] loss=0.914481
predicted string:  ohlll, Epoch[5/15] loss=0.765491
predicted string:  ohlol, Epoch[6/15] loss=0.626814
predicted string:  ohlol, Epoch[7/15] loss=0.511682
predicted string:  ohlol, Epoch[8/15] loss=0.411737
predicted string:  ohlol, Epoch[9/15] loss=0.329275
predicted string:  ohlol, Epoch[10/15] loss=0.266427
predicted string:  ohlol, Epoch[11/15] loss=0.213944
predicted string:  ohlol, Epoch[12/15] loss=0.169539
predicted string:  ohlol, Epoch[13/15] loss=0.130982
predicted string:  ohlol, Epoch[14/15] loss=0.096756
predicted string:  ohlol, Epoch[15/15] loss=0.067950

13. 循环神经网络（RNN）进阶

举例

以对名字所属国家进行分类为例：

我们构建如下模型。具体来说，我们RNN网络采用的是GRU Layer。

Bi-GPU：正向反向。考虑到正向序列可能学习不到下一个seq的内容，所以从末尾到开头学习一次最后进行concat。这儿输出的hidden是 $[h^f_N,h^b_N]$ 。

pack_padded_sequence：压缩序列，使RNN模块能够跳过填充部分。

代码如下所示：

import gzip
import csv
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader#Dataset是一个抽象类
from torch.nn.utils.rnn import pack_padded_sequence
from tqdm import tqdm
import matplotlib.pyplot as plt
import numpy as np
import time

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("torch.cuda.is_available()",torch.cuda.is_available())


HIDDEN_SIZE = 100
BATCH_SIZE = 256
N_LAYER = 2
N_EPOCHS = 100
N_CHARS = 128  # 字符集字典维度
N_COUNTRY = -1

class NameDataset(Dataset):
    def __init__(self, train_set = True):
        if train_set:
            filename = 'data/names_train.csv.gz'
        else:
            filename = 'data/names_test.csv.gz'
        with gzip.open(filename, 'rt') as f:
            item = csv.reader(f)
            rows = list(item)#(name,country)
        self.names = [row[0] for row in rows]
        self.len = len(self.names)
        self.countries = [row[1] for row in rows]
        self.country_list = list(sorted(set(self.countries)))#set去重，然后排序
        self.country_dict = self.getCountryDict()
        self.country_num = len(self.country_list)

    def __getitem__(self, item):
        return  self.names[item], self.country_dict[self.countries[item]]

    def __len__(self):
        return self.len

    def getCountryDict(self):
        country_dict = {}
        for i, country in enumerate(self.country_list):
            country_dict[country] = i
        return  country_dict

    def id2country(self, id):
        return self.country_list[id]

    def getCountriesNum(self):
        return self.country_num

class RNNClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, output_size,n_layers = 1,bidirectional = True):
        super(RNNClassifier, self).__init__()
        self.hidden_size = hidden_size
        self.n_layers = n_layers
        if bidirectional:
            self.n_bidirectional = 2 #双向GPU
        else:
            self.n_bidirectional = 1
        self.embedding = nn.Embedding(input_size, hidden_size)
        self.GRU = nn.GRU(hidden_size, hidden_size,n_layers, bidirectional=bidirectional)
        self.fc = nn.Linear(hidden_size*self.n_bidirectional, output_size)

    def forward(self,input,seq_lengths):
        input = input.t()#转置,input shape: b*s -> s*b
        batch_size = input.size(1)
        hidden = self._init_hidden(batch_size)

        embedding = self.embedding(input)#(seq,batch,hidden)
        gru_input = pack_padded_sequence(embedding, seq_lengths.cpu())#将所有padding后填充为0的量去掉
        output, hidden = self.GRU(gru_input, hidden)
        if self.n_bidirectional == 1:
            hidden_cat = hidden[-1]
        else:
            hidden_cat = torch.cat((hidden[-1], hidden[-2]), dim=1)
        return self.fc(hidden_cat)

    def _init_hidden(self, batch_size):
        hidden = torch.zeros(self.n_layers * self.n_bidirectional, batch_size, self.hidden_size)
        return hidden.to(device)


def name2list(name):
    arr = [ord(c) for c in name]#读取ASCII码列表
    return arr,len(arr) #返回元组

def make_tensors(names,countries):
    """
    将名字转换为tensor，并按降序排列，方便pack_padded_sequence处理
    """
    sequences_and_lengths = [name2list(name) for name in names]
    name_sequences = [sl[0] for sl in sequences_and_lengths]#列表长度
    seq_lengths = torch.LongTensor([sl[1] for sl in sequences_and_lengths])
    countries = countries.long()

    #make tensor of name, batchsize*seqlen
    seq_tensor = torch.zeros(len(name_sequences), seq_lengths.max()).long()#padding准备，直接做全0张量，然后填充数据，从而不需要补0
    for i,(seq,seq_length) in enumerate(zip(name_sequences, seq_lengths),0):
        seq_tensor[i,:seq_length] = torch.LongTensor(seq)
    #按照序列长度降序排序
    seq_lengths, perm_idx = seq_lengths.sort(dim=0, descending=True)
    seq_tensor = seq_tensor[perm_idx]
    countries = countries[perm_idx]
    return seq_tensor.to(device), seq_lengths.to(device), countries.to(device)


def train_valid(model,train_loader,test_loader):
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    acc_list = []
    for epoch in range(N_EPOCHS):
        optimizer.zero_grad()
        model.train()
        total_loss = 0
        pbar = tqdm(train_loader, desc="Iteration", postfix='train_stage', ncols=100)
        for i,batch_data in enumerate(pbar,1):
            names, countries = batch_data
            inputs, seq_lengths, target = make_tensors(names, countries)#已经到device上了
            optimizer.zero_grad()
            outputs = model(inputs,seq_lengths)
            loss = criterion(outputs, target)
            loss.backward()
            optimizer.step()

            total_loss += loss.item()
            if i % 10 == 0:
                print(f'Epoch{epoch}', end='')
                print(f'[{i * len(inputs)}/{len(trainset)}]', end='')
                print(f'loss={total_loss / (i * len(inputs))}')

        acc = valid(model,test_loader)
        acc_list.append(acc)

    epoch = np.arange(1, len(acc_list) + 1, 1)
    acc_list = np.array(acc_list)
    plt.plot(epoch, acc_list)
    plt.xlabel('Epoch')
    plt.ylabel('Accuracy')
    plt.grid()
    plt.show()

def valid(model,test_loader):
    correct = 0
    total = 0
    print('evaluating trained model ...')
    model.eval()
    pbar = tqdm(test_loader, desc="Iteration", postfix=type)
    with torch.no_grad():
        for i, batch_data in enumerate(pbar, 1):
            names, countries = batch_data
            inputs, seq_lengths, target = make_tensors(names, countries)  # 已经到device上了
            output = model(inputs,seq_lengths)
            pred = output.max(dim=1, keepdim=True)[1]
            correct += pred.eq(target.view_as(pred)).sum().item()
            total+=len(countries)
        percent = '%.2f' % (100 * correct / total)
        print(f'Test set: Accuracy{correct}/{total} {percent}%')
    return correct / total


if __name__ == '__main__':
    trainset = NameDataset(train_set=True)
    trainloader = DataLoader(trainset, batch_size=BATCH_SIZE, shuffle=True)
    testset = NameDataset(train_set=False)
    testloader = DataLoader(testset, batch_size=BATCH_SIZE, shuffle=False)
    N_COUNTRY = trainset.getCountriesNum()

    model = RNNClassifier(input_size=N_CHARS,hidden_size=HIDDEN_SIZE,output_size=N_COUNTRY,n_layers=N_LAYER,bidirectional=True).to(device)
    train_valid(model,trainloader,testloader)