PyTorch 05： PyTorch推理和验证

在训练神经网络之后，你现在可以使用它来进行预测。这个过程通常称为推理过程，这一术语来自统计学。然而，神经网络在面对训练数据时往往表现得太过优异，因而无法泛化到未见过的数据。这称之为过拟合，会影响推理效果。为了在训练中测试过拟合情况，我们会使用非训练集中的数据（称为验证集）衡量效果。在训练期间监测验证效果时，我们使用正则化避免过拟合。在此 notebook 中，我将演示如何在 PyTorch 中防止过拟合。

和之前一样，首先通过 torchvision 加载数据集。这次，我们将使用测试集，你可以通过设置 train=False 获得测试集：

testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=False, transform=transform)

测试集包含和训练集相似的图像。通常，我们会将原始数据集的 10-20% 作为测试和验证集，剩下的用于训练。

import torch
from torchvision import datasets, transforms

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize((0.5,), (0.5,))])
# Download and load the training data
trainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

# Download and load the test data
testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)

现在我将创建一个正常模型，使用 Part 4 notebook 答案中的模型。

from torch import nn, optim
import torch.nn.functional as F

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Linear(64, 10)
        
    def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)
        
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.log_softmax(self.fc4(x), dim=1)
        
        return x

验证的目的是衡量模型在非训练集数据上的效果。效果标准由开发者自己决定。通常用准确率表示，即网络预测正确的类别所占百分比。其他标准包括精确率和召回率以及 top-5 错误率。我们将侧重于准确率。首先，我将使用测试集中的一批数据进行前向传播。

model = Classifier()

images, labels = next(iter(testloader))
# Get the class probabilities
ps = torch.exp(model(images))
# Make sure the shape is appropriate, we should get 10 class probabilities for 64 examples
print(ps.shape)

torch.Size([64, 10])

torch.Size([64, 10])

torch.Size([64, 10])

算出概率后，我们可以使用 ps.topk 方法获得概率最高的类别。它会返回前 $k$ 个值。因为我们只需概率最高的类别，因此可以使用 ps.topk(1)。它会返回 top-$k$ 值和 top-$k$ 索引元组。如果最高值是第 5 个元素，索引将是 4。

top_p, top_class = ps.topk(1, dim=1)
# Look at the most likely classes for the first 10 examples
print(top_class[:10,:])

tensor([[5],
        [5],
        [5],
        [5],
        [5],
        [5],
        [5],
        [5],
        [5],
        [5]])

现在检查预测类别与标签是否匹配。很简单，只需检测 top_class 和 labels 是否相等，但是要注意形状。top_class 是一个二维张量，形状为 (64, 1)，而 labels 是一个一维张量，形状为 (64)。要检测对等性，top_class 和 labels 必须是相同的形状。

如果执行以下运算

equals = top_class == labels

equals 的形状将为 (64, 64)，自己尝试一下。它会将 top_class 中每行的一个元素与 labels 中的每个元素进行比较，并针对每行返回 64 个True/False 布尔值。

equals = top_class == labels.view(*top_class.shape)

现在我们需要计算正确预测的百分比。equals 具有二元值，为 0 或 1。这就是说，如果将所有值相加并除以值的数量，就能获得正确预测的百分比。这个运算和求平均值一样，因此我们可以通过调用 torch.mean 获得准确率。听起来很简单？实际上，如果你用 torch.mean(equals)，就会报错：

RuntimeError: mean is not implemented for type torch.ByteTensor

这是因为 equals 的类型是 torch.ByteTensor，但是 torch.mean 不适合这种类型的张量。因此我们需要将 equals 转换为浮点张量。注意，当我们执行 torch.mean 时，它会返回一个标量张量，要获得实际浮点值，我们需要进行 accuracy.item()。

accuracy = torch.mean(equals.type(torch.FloatTensor))
print(f'Accuracy: {accuracy.item()*100}%')

Accuracy: 7.8125%

准确率：7.8125%

网络未经过训练，因此它做出的是随机猜测，准确率应该约为 10%。现在我们训练网络并包含验证过程，以便衡量网络在测试集上的效果。因为我们在验证过程中不会更新参数，因此可以使用 torch.no_grad() 关闭梯度，从而加快代码运行速度：

# turn off gradients
with torch.no_grad():
    # validation pass here
    for images, labels in testloader:
        ...

**练习：**实现下面的验证循环并在循环之后输出总准确率。你可以复制粘贴上面的代码，但是建议自己编写，这样才能得到锻炼。通常，自己编写肯定比复制粘贴的学习效果要好。你应该可以取得 80% 以上的准确率。

model = Classifier()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

epochs = 30
steps = 0

train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        
        optimizer.zero_grad()
        
        log_ps = model(images)
        loss = criterion(log_ps, labels)
        loss.backward()
        optimizer.step()
        
        running_loss += loss.item()
        
    else:
        test_loss = 0
        accuracy = 0
        
        # Turn off gradients for validation, saves memory and computations
        with torch.no_grad():
            for images, labels in testloader:
                log_ps = model(images)
                test_loss += criterion(log_ps, labels)
                
                ps = torch.exp(log_ps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor))
                
        train_losses.append(running_loss/len(trainloader))
        test_losses.append(test_loss/len(testloader))

        print("Epoch: {}/{}.. ".format(e+1, epochs),
              "Training Loss: {:.3f}.. ".format(running_loss/len(trainloader)),
              "Test Loss: {:.3f}.. ".format(test_loss/len(testloader)),
              "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

Epoch: 1/30..  Training Loss: 0.511..  Test Loss: 0.453..  Test Accuracy: 0.841
Epoch: 2/30..  Training Loss: 0.392..  Test Loss: 0.464..  Test Accuracy: 0.827
Epoch: 3/30..  Training Loss: 0.358..  Test Loss: 0.390..  Test Accuracy: 0.856
Epoch: 4/30..  Training Loss: 0.335..  Test Loss: 0.372..  Test Accuracy: 0.864
Epoch: 5/30..  Training Loss: 0.319..  Test Loss: 0.381..  Test Accuracy: 0.867
Epoch: 6/30..  Training Loss: 0.305..  Test Loss: 0.369..  Test Accuracy: 0.868
Epoch: 7/30..  Training Loss: 0.297..  Test Loss: 0.388..  Test Accuracy: 0.863
Epoch: 8/30..  Training Loss: 0.284..  Test Loss: 0.365..  Test Accuracy: 0.874
Epoch: 9/30..  Training Loss: 0.277..  Test Loss: 0.378..  Test Accuracy: 0.866
Epoch: 10/30..  Training Loss: 0.267..  Test Loss: 0.373..  Test Accuracy: 0.871
Epoch: 11/30..  Training Loss: 0.260..  Test Loss: 0.403..  Test Accuracy: 0.871
Epoch: 12/30..  Training Loss: 0.255..  Test Loss: 0.366..  Test Accuracy: 0.880
Epoch: 13/30..  Training Loss: 0.249..  Test Loss: 0.384..  Test Accuracy: 0.874
Epoch: 14/30..  Training Loss: 0.245..  Test Loss: 0.355..  Test Accuracy: 0.884
Epoch: 15/30..  Training Loss: 0.235..  Test Loss: 0.388..  Test Accuracy: 0.875
Epoch: 16/30..  Training Loss: 0.234..  Test Loss: 0.359..  Test Accuracy: 0.883
Epoch: 17/30..  Training Loss: 0.228..  Test Loss: 0.387..  Test Accuracy: 0.879
Epoch: 18/30..  Training Loss: 0.223..  Test Loss: 0.386..  Test Accuracy: 0.881
Epoch: 19/30..  Training Loss: 0.220..  Test Loss: 0.396..  Test Accuracy: 0.881
Epoch: 20/30..  Training Loss: 0.217..  Test Loss: 0.374..  Test Accuracy: 0.882
Epoch: 21/30..  Training Loss: 0.211..  Test Loss: 0.405..  Test Accuracy: 0.879
Epoch: 22/30..  Training Loss: 0.207..  Test Loss: 0.397..  Test Accuracy: 0.887
Epoch: 23/30..  Training Loss: 0.212..  Test Loss: 0.409..  Test Accuracy: 0.877
Epoch: 24/30..  Training Loss: 0.199..  Test Loss: 0.408..  Test Accuracy: 0.880
Epoch: 25/30..  Training Loss: 0.199..  Test Loss: 0.413..  Test Accuracy: 0.880
Epoch: 26/30..  Training Loss: 0.198..  Test Loss: 0.404..  Test Accuracy: 0.882
Epoch: 27/30..  Training Loss: 0.194..  Test Loss: 0.410..  Test Accuracy: 0.885
Epoch: 28/30..  Training Loss: 0.194..  Test Loss: 0.425..  Test Accuracy: 0.884
Epoch: 29/30..  Training Loss: 0.187..  Test Loss: 0.437..  Test Accuracy: 0.880
Epoch: 30/30..  Training Loss: 0.185..  Test Loss: 0.429..  Test Accuracy: 0.883

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt

plt.plot(train_losses, label='Training loss')
plt.plot(test_losses, label='Validation loss')
plt.legend(frameon=False)

<matplotlib.legend.Legend at 0x7fe078251438>

过拟合

如果我们观察训练过程中的训练和验证损失，就能发现一种叫做过拟合的现象。（如上图所示）

网络能越来越好地学习训练集中的规律，导致训练损失越来越低。但是，它在泛化到训练集之外的数据时开始出现问题，导致验证损失上升。任何深度学习模型的最终目标是对新数据进行预测，因此我们要尽量降低验证损失。一种方法是使用验证损失最低的模型，在此例中是训练周期约为 8-10 次的模型。这种策略称为早停法 (early stopping)。在实践中，你需要在训练时频繁地保存模型，以便之后选择验证损失最低的模型。

最常用的减少过拟合方法（早停法除外）是丢弃，即随机丢弃输入单元。这样就促使网络在权重之间共享信息，使其更能泛化到新数据。在 PyTorch 中添加丢弃层很简单，使用 nn.Dropout 模块即可。

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Linear(64, 10)
        
        # Dropout module with 0.2 drop probability
        self.dropout = nn.Dropout(p=0.2)
        
    def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)
        
        # Now with dropout
        x = self.dropout(F.relu(self.fc1(x)))
        x = self.dropout(F.relu(self.fc2(x)))
        x = self.dropout(F.relu(self.fc3(x)))
        
        # output so no dropout here
        x = F.log_softmax(self.fc4(x), dim=1)
        
        return x

在训练过程中，我们需要使用丢弃防止过拟合，但是在推理过程中，我们需要使用整个网络。因此在验证、测试和使用网络进行预测时，我们需要关闭丢弃功能。你可以使用 model.eval()。它会将模型设为验证模式，使丢弃率变成 0。也可以使用 model.train() ，将模型设为训练模式，重新开启丢弃功能。通常，验证循环的规律将为：关闭梯度，将模型设为评估模式，计算验证损失和指标，然后将模型重新设为训练模式。

# turn off gradients
with torch.no_grad():
    
    # set model to evaluation mode
    model.eval()
    
    # validation pass here
    for images, labels in testloader:
        ...

# set model back to train mode
model.train()

**练习：**向模型中添加丢弃并重新用 Fashion-MNIST 训练模型。看看你能否获得更低的验证损失或更高的准确率。

## TODO: Define your model with dropout added
class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Linear(64, 10)

        # Dropout module with 0.2 drop probability
        self.dropout = nn.Dropout(p=0.2)

    def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        # Now with dropout
        x = self.dropout(F.relu(self.fc1(x)))
        x = self.dropout(F.relu(self.fc2(x)))
        x = self.dropout(F.relu(self.fc3(x)))

        # output so no dropout here
        x = F.log_softmax(self.fc4(x), dim=1)

        return x

## TODO: Train your model with dropout, and monitor the training progress with the validation loss and accuracy
model = Classifier()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

epochs = 30
steps = 0

train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        
        optimizer.zero_grad()
        
        log_ps = model(images)
        loss = criterion(log_ps, labels)
        loss.backward()
        optimizer.step()
        
        running_loss += loss.item()
        
    else:
        test_loss = 0
        accuracy = 0
        
        # Turn off gradients for validation, saves memory and computations
        with torch.no_grad():
            model.eval() #关闭dropout
            for images, labels in testloader:
                log_ps = model(images)
                test_loss += criterion(log_ps, labels)
                
                ps = torch.exp(log_ps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor))
        
        model.train() #开启dropout
        
        train_losses.append(running_loss/len(trainloader))
        test_losses.append(test_loss/len(testloader))

        print("Epoch: {}/{}.. ".format(e+1, epochs),
              "Training Loss: {:.3f}.. ".format(train_losses[-1]),
              "Test Loss: {:.3f}.. ".format(test_losses[-1]),
              "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

Epoch: 1/30..  Training Loss: 0.609..  Test Loss: 0.492..  Test Accuracy: 0.822
Epoch: 2/30..  Training Loss: 0.479..  Test Loss: 0.420..  Test Accuracy: 0.849
Epoch: 3/30..  Training Loss: 0.451..  Test Loss: 0.424..  Test Accuracy: 0.853
Epoch: 4/30..  Training Loss: 0.429..  Test Loss: 0.404..  Test Accuracy: 0.856
Epoch: 5/30..  Training Loss: 0.420..  Test Loss: 0.397..  Test Accuracy: 0.858
Epoch: 6/30..  Training Loss: 0.411..  Test Loss: 0.392..  Test Accuracy: 0.859
Epoch: 7/30..  Training Loss: 0.406..  Test Loss: 0.396..  Test Accuracy: 0.863
Epoch: 8/30..  Training Loss: 0.393..  Test Loss: 0.401..  Test Accuracy: 0.862
Epoch: 9/30..  Training Loss: 0.391..  Test Loss: 0.405..  Test Accuracy: 0.862
Epoch: 10/30..  Training Loss: 0.396..  Test Loss: 0.391..  Test Accuracy: 0.863
Epoch: 11/30..  Training Loss: 0.381..  Test Loss: 0.393..  Test Accuracy: 0.859
Epoch: 12/30..  Training Loss: 0.381..  Test Loss: 0.391..  Test Accuracy: 0.863
Epoch: 13/30..  Training Loss: 0.378..  Test Loss: 0.377..  Test Accuracy: 0.868
Epoch: 14/30..  Training Loss: 0.374..  Test Loss: 0.386..  Test Accuracy: 0.871
Epoch: 15/30..  Training Loss: 0.363..  Test Loss: 0.397..  Test Accuracy: 0.864
Epoch: 16/30..  Training Loss: 0.370..  Test Loss: 0.398..  Test Accuracy: 0.860
Epoch: 17/30..  Training Loss: 0.372..  Test Loss: 0.428..  Test Accuracy: 0.866
Epoch: 18/30..  Training Loss: 0.367..  Test Loss: 0.382..  Test Accuracy: 0.867
Epoch: 19/30..  Training Loss: 0.359..  Test Loss: 0.394..  Test Accuracy: 0.862
Epoch: 20/30..  Training Loss: 0.366..  Test Loss: 0.382..  Test Accuracy: 0.875
Epoch: 21/30..  Training Loss: 0.357..  Test Loss: 0.380..  Test Accuracy: 0.868
Epoch: 22/30..  Training Loss: 0.352..  Test Loss: 0.386..  Test Accuracy: 0.868
Epoch: 23/30..  Training Loss: 0.353..  Test Loss: 0.366..  Test Accuracy: 0.877
Epoch: 24/30..  Training Loss: 0.358..  Test Loss: 0.379..  Test Accuracy: 0.871
Epoch: 25/30..  Training Loss: 0.343..  Test Loss: 0.380..  Test Accuracy: 0.874
Epoch: 26/30..  Training Loss: 0.345..  Test Loss: 0.367..  Test Accuracy: 0.874
Epoch: 27/30..  Training Loss: 0.345..  Test Loss: 0.379..  Test Accuracy: 0.867
Epoch: 28/30..  Training Loss: 0.349..  Test Loss: 0.369..  Test Accuracy: 0.871
Epoch: 29/30..  Training Loss: 0.345..  Test Loss: 0.391..  Test Accuracy: 0.859
Epoch: 30/30..  Training Loss: 0.351..  Test Loss: 0.388..  Test Accuracy: 0.867

plt.plot(train_losses, label='Training loss')
plt.plot(test_losses, label='Validation loss')
plt.legend(frameon=False)

<matplotlib.legend.Legend at 0x7fe078192dd8>

推理

训练好模型后，我们可以用它推理了。之前已经进行过这一步骤，但是现在需要使用 model.eval() 将模型设为推理模式。对于 torch.no_grad()，你需要关闭 autograd。

# Import helper module (should be in the repo)
import helper

# Test out your network!

model.eval()

dataiter = iter(testloader)
images, labels = dataiter.next()
img = images[0]
# Convert 2D image to 1D vector
img = img.view(1, 784)

# Calculate the class probabilities (softmax) for img
with torch.no_grad():
    output = model.forward(img)

ps = torch.exp(output)

# Plot the image and probabilities
helper.view_classify(img.view(1, 28, 28), ps, version='Fashion')

保存和加载模型参数

pytorch需要保持的参数都在,model.state_dict().keys()

print("my model:\n\n",model,'\n')

my model:

 Classifier(
  (fc1): Linear(in_features=784, out_features=256, bias=True)
  (fc2): Linear(in_features=256, out_features=128, bias=True)
  (fc3): Linear(in_features=128, out_features=64, bias=True)
  (fc4): Linear(in_features=64, out_features=10, bias=True)
  (dropout): Dropout(p=0.2, inplace=False)
) 

print("The state dict keys:\n\n",model.state_dict().keys())

The state dict keys:

 odict_keys(['fc1.weight', 'fc1.bias', 'fc2.weight', 'fc2.bias', 'fc3.weight', 'fc3.bias', 'fc4.weight', 'fc4.bias'])

torch.save(model.state_dict(),'checkpoint.pth')

未加载参数的模型(亦没训练，随机初始化)

model_new = Classifier()
# Import helper module (should be in the repo)
import helper

# Test out your network!

model_new.eval()

dataiter = iter(testloader)
images, labels = dataiter.next()
img = images[0]
# Convert 2D image to 1D vector
img = img.view(1, 784)

# Calculate the class probabilities (softmax) for img
with torch.no_grad():
    output = model_new.forward(img)

ps = torch.exp(output)

# Plot the image and probabilities
helper.view_classify(img.view(1, 28, 28), ps, version='Fashion')

加载保存模型参数，再看推理效果

state_dict_new = torch.load('checkpoint.pth')
print(state_dict_new.keys())

odict_keys(['fc1.weight', 'fc1.bias', 'fc2.weight', 'fc2.bias', 'fc3.weight', 'fc3.bias', 'fc4.weight', 'fc4.bias'])

model_new.load_state_dict(state_dict_new)

<All keys matched successfully>

# Import helper module (should be in the repo)
import helper

# Test out your network!

model_new.eval()

dataiter = iter(testloader)
images, labels = dataiter.next()
img = images[0]
# Convert 2D image to 1D vector
img = img.view(1, 784)

# Calculate the class probabilities (softmax) for img
with torch.no_grad():
    output = model_new.forward(img)

ps = torch.exp(output)

# Plot the image and probabilities
helper.view_classify(img.view(1, 28, 28), ps, version='Fashion')

下一步！

在下个部分，我将介绍如何保存训练的模型。我们肯定不希望每次要用到模型的时候都临时训练，而是希望能保存模型，然后在继续训练时或使用模型推理时再加载模型，所以我们需要学会保存和加载模型。