学习笔记|Pytorch使用教程23(正则化之weight_decay)

Pytorch 同时被 2 个专栏收录
39 篇文章 9 订阅
78 篇文章 3 订阅

学习笔记|Pytorch使用教程23

本学习笔记主要摘自“深度之眼”,做一个总结,方便查阅。
使用Pytorch版本为1.2

  • 正则化与偏差方差分解
  • pytorch中的L2正则项weight decay

一.正则化与偏差方差分解

Regularization:减小方差的策略
误差可分解为:偏差,方差与噪声之和。即误差=偏差+方差+噪声之和
偏差度量了学习算法的期望预测与真实结果的偏离程度,即刻画了学习算法本身的拟合能力
方差度量了同样大小的训练集的变动所导致的学习性能的变化,即刻画了数据扰动所造成的影响
噪声则表达了在当前任务上任何学习算法所能达到的期望泛化误差的下界
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

  • 损失函数:衡量模型输出与真实标签的差异
    L o s s = f ( y ∧ , y ) Loss= f\left(y^{\wedge}, y\right) Loss=f(y,y)
  • 代价函数(Cost Function) :
    C o s t = 1 N ∑ t N f ( y i ∧ , y i ) Cost=\frac{1}{N} \sum_{t}^{N} f\left(y_{i}^{\wedge}, y_{i}\right) Cost=N1tNf(yi,yi)
  • 目标函数(Objective Function) :
    O b j = C o s t + R e g u l a r i z a t i o n T e r m Obj = Cost + Regularization Term Obj=Cost+RegularizationTerm
  • L1 Regularization Term: ∑ i N ∣ w i ∣ \sum_{i}^{N}\left|w_{i}\right| iNwi
  • L2 Regularization Term: ∑ i N w i 2 \sum_{i}^{N} w_{i}^{2} iNwi2
    (坐标是L1,右边是L2)
    在这里插入图片描述

二.pytorch中的L2正则项weight decay

L2 Regularization = weight decay (权值衰减)
目标函数(Objective Function) :
O b j = C o s t + R e g u l a r i z a t i o n T e r m Obj = Cost + Regularization Term Obj=Cost+RegularizationTerm
O b j = L o s s + λ 2 ∗ ∑ N i w i 2 Obj=Loss+\frac{\lambda }{2}\ast \sum_{N}^{i}w_{i}^{2} Obj=Loss+2λNiwi2
权值衰减推导过程:
w i + 1 = w i − ∂ O b j ∂ w i = w i − ∂ L o s s ∂ w i w i + 1 = w i − ∂ O b j ∂ w i = w i − ( ∂ L o s s ∂ w i + λ ⋆ w i ) = w i ( 1 − λ ) − ∂ log ⁡ s ∂ w i \begin{aligned} w_{i+1}=w_{i}-\frac{\partial O b j}{\partial w_{i}} &=w_{i}-\frac{\partial Loss}{\partial w_{i}} \\ w_{i+1}=w_{i}-\frac{\partial O b j}{\partial w_{i}} &=w_{i}-\left(\frac{\partial Loss}{\partial w_{i}}+\lambda \star w_{i}\right) \\ &=w_{i}(1-\lambda)-\frac{\partial \log s}{\partial w_{i}} \end{aligned} wi+1=wiwiObjwi+1=wiwiObj=wiwiLoss=wi(wiLoss+λwi)=wi(1λ)wilogs

测试代码:

import torch
import torch.nn as nn
import matplotlib.pyplot as plt
from tools.common_tools import set_seed
from torch.utils.tensorboard import SummaryWriter

set_seed(1)  # 设置随机种子
n_hidden = 200
max_iter = 2000
disp_interval = 200
lr_init = 0.01


# ============================ step 1/5 数据 ============================
def gen_data(num_data=10, x_range=(-1, 1)):

    w = 1.5
    train_x = torch.linspace(*x_range, num_data).unsqueeze_(1)
    train_y = w*train_x + torch.normal(0, 0.5, size=train_x.size())
    test_x = torch.linspace(*x_range, num_data).unsqueeze_(1)
    test_y = w*test_x + torch.normal(0, 0.3, size=test_x.size())

    return train_x, train_y, test_x, test_y


train_x, train_y, test_x, test_y = gen_data(x_range=(-1, 1))


# ============================ step 2/5 模型 ============================
class MLP(nn.Module):
    def __init__(self, neural_num):
        super(MLP, self).__init__()
        self.linears = nn.Sequential(
            nn.Linear(1, neural_num),
            nn.ReLU(inplace=True),
            nn.Linear(neural_num, neural_num),
            nn.ReLU(inplace=True),
            nn.Linear(neural_num, neural_num),
            nn.ReLU(inplace=True),
            nn.Linear(neural_num, 1),
        )

    def forward(self, x):
        return self.linears(x)


net_normal = MLP(neural_num=n_hidden)
net_weight_decay = MLP(neural_num=n_hidden)

# ============================ step 3/5 优化器 ============================
optim_normal = torch.optim.SGD(net_normal.parameters(), lr=lr_init, momentum=0.9)
optim_wdecay = torch.optim.SGD(net_weight_decay.parameters(), lr=lr_init, momentum=0.9, weight_decay=1e-2)

# ============================ step 4/5 损失函数 ============================
loss_func = torch.nn.MSELoss()

# ============================ step 5/5 迭代训练 ============================

writer = SummaryWriter(comment='_test_tensorboard', filename_suffix="12345678")
for epoch in range(max_iter):

    # forward
    pred_normal, pred_wdecay = net_normal(train_x), net_weight_decay(train_x)
    loss_normal, loss_wdecay = loss_func(pred_normal, train_y), loss_func(pred_wdecay, train_y)

    optim_normal.zero_grad()
    optim_wdecay.zero_grad()

    loss_normal.backward()
    loss_wdecay.backward()

    optim_normal.step()
    optim_wdecay.step()

    if (epoch+1) % disp_interval == 0:

        # 可视化
        for name, layer in net_normal.named_parameters():
            writer.add_histogram(name + '_grad_normal', layer.grad, epoch)
            writer.add_histogram(name + '_data_normal', layer, epoch)

        for name, layer in net_weight_decay.named_parameters():
            writer.add_histogram(name + '_grad_weight_decay', layer.grad, epoch)
            writer.add_histogram(name + '_data_weight_decay', layer, epoch)

        test_pred_normal, test_pred_wdecay = net_normal(test_x), net_weight_decay(test_x)

        # 绘图
        plt.scatter(train_x.data.numpy(), train_y.data.numpy(), c='blue', s=50, alpha=0.3, label='train')
        plt.scatter(test_x.data.numpy(), test_y.data.numpy(), c='red', s=50, alpha=0.3, label='test')
        plt.plot(test_x.data.numpy(), test_pred_normal.data.numpy(), 'r-', lw=3, label='no weight decay')
        plt.plot(test_x.data.numpy(), test_pred_wdecay.data.numpy(), 'b--', lw=3, label='weight decay')
        plt.text(-0.25, -1.5, 'no weight decay loss={:.6f}'.format(loss_normal.item()), fontdict={'size': 15, 'color': 'red'})
        plt.text(-0.25, -2, 'weight decay loss={:.6f}'.format(loss_wdecay.item()), fontdict={'size': 15, 'color': 'red'})

        plt.ylim((-2.5, 2.5))
        plt.legend(loc='upper left')
        plt.title("Epoch: {}".format(epoch+1))
        plt.show()
        plt.close()

输出:
在这里插入图片描述在这里插入图片描述
在这里插入图片描述在这里插入图片描述
进入tensorboard
没有带L2正则化的,整个权值变化不大。
在这里插入图片描述
带L2正则化的,权值不断缩减。
在这里插入图片描述

在代码optim_wdecay.step()设置断点,并进入(step into)

    def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            loss = closure()

        for group in self.param_groups:
            weight_decay = group['weight_decay']
            momentum = group['momentum']
            dampening = group['dampening']
            nesterov = group['nesterov']

            for p in group['params']:
                if p.grad is None:
                    continue
                d_p = p.grad.data
                if weight_decay != 0:
                    d_p.add_(weight_decay, p.data)
                if momentum != 0:
                    param_state = self.state[p]
                    if 'momentum_buffer' not in param_state:
                        buf = param_state['momentum_buffer'] = torch.clone(d_p).detach()
                    else:
                        buf = param_state['momentum_buffer']
                        buf.mul_(momentum).add_(1 - dampening, d_p)
                    if nesterov:
                        d_p = d_p.add(momentum, buf)
                    else:
                        d_p = buf

                p.data.add_(-group['lr'], d_p)

        return loss

在代码的d_p.add_(weight_decay, p.data)进行权值衰减。其公式是:d_p = d_p + p.data * weight_decay
在代码p.data.add_(-group['lr'], d_p)进行梯度更新。

  • 4
    点赞
  • 0
    评论
  • 8
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

©️2021 CSDN 皮肤主题: 技术黑板 设计师:CSDN官方博客 返回首页
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值