【pytorch】过拟合和欠拟合详解，并以三阶多项式函数绘图举例 （附pytorch.cat的用法）

一、欠拟合、过拟合解释

• 训练误差（training error）指模型在训练数据集上表现出的误差。
• 泛化误差（generalization error）指模型在任意一个测试数据样本上表现出的误差的期望，并常常通过测试数据集上的误差来近似。
• 欠拟合（underfitting）：模型无法得到较低的训练误差的现象。
• 过拟合（overfitting）：模型的训练误差远小于它在测试数据集上的误差的现象。
• 在实践中，我们要尽可能同时应对欠拟合和过拟合。虽然有很多因素可能导致这两种拟合问题，但是最重要的是模型复杂度和训练数据集大小。
• 模型复杂度越高，训练数据集中样本数过少，特别是比模型参数数量（按元素计）更少时，越容易过拟合。模型复杂度越低，越容易欠拟合。

二、绘制三阶多项式函数拟合，过拟合，欠拟合曲线

（1）torch.cat的用法（后面会用到）

• dim=0则按列拼接，dim=1则按行拼接

import torch

a = torch.ones(3, 3)
b = torch.zeros(2, 3)
c = torch.cat((a, b), dim=0)
print(c)

• 结果

tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [0., 0., 0.],
        [0., 0., 0.]])

（2）绘制拟合曲线

1）代码

import matplotlib.pyplot as plt
import numpy as np
import torch
from IPython import display


def use_svg_display():
    """Use svg format to display plot in jupyter"""
    display.set_matplotlib_formats('svg')


def set_figsize(figsize=(3.5, 2.5)):
    use_svg_display()
    # 设置图的尺寸
    plt.rcParams['figure.figsize'] = figsize


def semilogy(x_vals, y_vals, x_label, y_label, x2_vals=None, y2_vals=None,
             legend=None, figsize=(3.5, 2.5)):
    set_figsize(figsize)
    plt.xlabel(x_label)
    plt.ylabel(y_label)
    # 画折线图，semilogy表示y坐标用指数表示，semilogx坐标用指数表示
    plt.semilogy(x_vals, y_vals)
    if x2_vals and y2_vals:
        plt.semilogy(x2_vals, y2_vals, linestyle=':')
        # 添加图例
        plt.legend(legend)
    plt.show()


def fit_and_plot(train_features, test_features, train_labels, test_labels):
    net = torch.nn.Linear(train_features.shape[-1], 1)
    # 通过Linear文档可知，pytorch已经将参数初始化了，所以我们这里就不手动初始化了
    # 10 10 2
    batch_size = min(10, train_labels.shape[0])
    # 100x3 100
    dataset = torch.utils.data.TensorDataset(train_features, train_labels)
    train_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True)

    optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
    train_ls, test_ls = [], []
    for _ in range(num_epochs):
        for X, y in train_iter:
            l = loss(net(X), y.view(-1, 1))
            optimizer.zero_grad()
            l.backward()
            optimizer.step()
        # 100x1
        train_labels = train_labels.view(-1, 1)
        test_labels = test_labels.view(-1, 1)
        train_ls.append(loss(net(train_features), train_labels).item())
        test_ls.append(loss(net(test_features), test_labels).item())
    print('final epoch: train loss', train_ls[-1], 'test loss', test_ls[-1])
    semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',
             range(1, num_epochs + 1), test_ls, ['train', 'test'])
    print('weight:', net.weight.data,
          '\nbias:', net.bias.data)


if __name__ == '__main__':
    n_train, n_test, true_w, true_b = 100, 100, [1.2, -3.4, 5.6], 5
    features = torch.randn((n_train + n_test, 1))
    num_epochs, loss = 100, torch.nn.MSELoss()
    # 200x3
    poly_features = torch.cat((features, torch.pow(features, 2), torch.pow(features, 3)), 1)
    labels = (true_w[0] * poly_features[:, 0] + true_w[1] * poly_features[:, 1]
              + true_w[2] * poly_features[:, 2] + true_b)
    # 200
    labels += torch.tensor(np.random.normal(0, 0.01, size=labels.size()), dtype=torch.float)
    fit_and_plot(poly_features[:n_train, :], poly_features[n_train:, :], labels[:n_train], labels[n_train:])
    fit_and_plot(features[:n_train, :], features[n_train:, :], labels[:n_train], labels[n_train:])
    fit_and_plot(poly_features[0:2, :], poly_features[n_train:, :], labels[0:2], labels[n_train:])

2）结果

final epoch: train loss 9.381109703099355e-05 test loss 0.0001090286677936092
weight: tensor([[ 1.1943, -3.3993,  5.6010]]) 
bias: tensor([4.9978])
final epoch: train loss 145.16329956054688 test loss 173.38833618164062
weight: tensor([[17.9973]]) 
bias: tensor([1.9011])
final epoch: train loss 3.03792667388916 test loss 9.147336959838867
weight: tensor([[ 1.1994, -3.1014,  4.9920]]) 
bias: tensor([2.4698])

• 三阶多项式函数拟合
  

• 欠拟合
  

• 过拟合
  

三、参考

统计学习理论之VC维究竟是什么
动手学深度学习（pytorch版）