目录
torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda, last_epoch=-1)
torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)
torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1)
torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma, last_epoch=-1)
torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=0, last_epoch=-1)
Pytorch中的学习率调整有两种方式
- 利用lr_scheduler()提供的几种衰减函数
- 直接修改optimizer中的lr参数;
利用lr_scheduler()提供的几种衰减函数
torch.optim.lr_scheduler.LambdaLr
torch.optim.lr_scheduler.StepLR
torch.optim.lr_scheduler.MultiStepLR
torch.optim.lr_scheduler.ExponentialLR
torch.optim.lr_sheduler.CosineAnneaingLR
torch.optim.lr_scheduler.ReduceLROnPlateau
torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda, last_epoch=-1)
lr_lambda 会接收到一个int参数:epoch,然后根据epoch计算出对应的lr。如果设置多个lambda函数的话,会分别作用于Optimizer中的不同的params_group
import numpy as np
lr_list = []
model = net()
LR = 0.01
optimizer = Adam(model.parameters(),lr = LR)
lambda1 = lambda epoch:np.sin(epoch) / epoch
scheduler = lr_scheduler.LambdaLR(optimizer,lr_lambda = lambda1)
for epoch in range(100):
scheduler.step()
lr_list.append(optimizer.state_dict()['param_groups'][0]['lr'])
plt.plot(range(100),lr_list,color = 'r')
torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)
每个一定的epoch,lr会自动乘以gamma
step_size(int)- 学习率下降间隔数,若为 30,则会在 30、 60、 90…个 step 时,将学习率调整为 lr*gamma。
gamma(float)- 学习率调整倍数,默认为 0.1 倍,即下降 10 倍。
last_epoch:表示上一轮是那个,接着上一轮继续开始
import torch
import torch.optim as optim
from torch.optim import lr_scheduler
from torchvision.models import AlexNet
import matplotlib.pyplot as plt
model = AlexNet(num_classes=2)
optimizer = optim.SGD(params=model.parameters(), lr=0.05)
# lr_scheduler.StepLR()
# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.05 if epoch < 30
# lr = 0.005 if 30 <= epoch < 60
# lr = 0.0005 if 60 <= epoch < 90
scheduler = lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
plt.figure()
x = list(range(100))
y = []
for epoch in range(100):
scheduler.step()
lr = scheduler.get_lr()
print(epoch, scheduler.get_lr()[0])
y.append(scheduler.get_lr()[0])
plt.plot(x, y)
0<epoch<30, lr = 0.05
30<=epoch<60, lr = 0.005
60<=epoch<90, lr = 0.0005
torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1)
三段式lr,epoch进入milestones范围内即乘以gamma,离开milestones范围之后再乘以gamma
# ---------------------------------------------------------------
# 可以指定区间
# lr_scheduler.MultiStepLR()
# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.05 if epoch < 30
# lr = 0.005 if 30 <= epoch < 80
# lr = 0.0005 if epoch >= 80
print()
plt.figure()
y.clear()
scheduler = lr_scheduler.MultiStepLR(optimizer, [30, 80], 0.1)
for epoch in range(100):
scheduler.step()
print(epoch, 'lr={:.6f}'.format(scheduler.get_lr()[0]))
y.append(scheduler.get_lr()[0])
plt.plot(x, y)
plt.show()
torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma, last_epoch=-1)
指数衰减,每个epoch中lr都乘以gamma
scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.9)
print()
plt.figure()
y.clear()
for epoch in range(100):
scheduler.step()
print(epoch, 'lr={:.6f}'.format(scheduler.get_lr()[0]))
y.append(scheduler.get_lr()[0])
plt.plot(x, y)
plt.show()
torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=0, last_epoch=-1)
T_max 对应1/2个cos周期所对应的epoch数值
eta_min 为最小的lr值,默认为0
lr_list = []
model = net()
LR = 0.01
optimizer = Adam(model.parameters(),lr = LR)
scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max = 20)
for epoch in range(100):
scheduler.step()
lr_list.append(optimizer.state_dict()['param_groups'][0]['lr'])
plt.plot(range(100),lr_list,color = 'r')
torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=False, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)
在发现loss不再降低或者acc不再提高之后,降低学习率。各参数意义如下:
mode:'min'模式检测metric是否不再减小,'max'模式检测metric是否不再增大;
factor: 触发条件后lr*=factor;
patience:不再减小(或增大)的累计次数;
verbose:触发条件后print;
threshold:只关注超过阈值的显著变化;
threshold_mode:有rel和abs两种阈值计算模式,rel规则:max模式下如果超过best(1+threshold)为显著,min模式下如果低于best(1-threshold)为显著;abs规则:max模式下如果超过best+threshold为显著,min模式下如果低于best-threshold为显著;
cooldown:触发一次条件后,等待一定epoch再进行检测,避免lr下降过速;
min_lr:最小的允许lr;
eps:如果新旧lr之间的差异小与1e-8,则忽略此次更新。
直接修改optimizer中的lr参数
修改optimizer中的lr,方法一
import torch
import matplotlib.pyplot as plt
%matplotlib inline
from torch.optim import *
import torch.nn as nn
class net(nn.Module):
def __init__(self):
super(net,self).__init__()
self.fc = nn.Linear(1,10)
def forward(self,x):
return self.fc(x)
model = net()
LR = 0.01
optimizer = Adam(model.parameters(),lr = LR)
lr_list = []
for epoch in range(100):
if epoch % 5 == 0:
for p in optimizer.param_groups:
p['lr'] *= 0.9
lr_list.append(optimizer.state_dict()['param_groups'][0]['lr'])
plt.plot(range(100),lr_list,color = 'r')
方法二
# 第epoch值进行计算并更新学习率
def adjust_lr(optimizer, epoch, T_0=10, eta_max=0.1, eta_min=0.):
if T_mul == 2:
i = np.log2(epoch / T_0 + 1).astype(np.int)
T_cur = epoch - T_0 * (T_mult ** (i) - 1)
T_i = (T_0 * T_mult ** i)
elif T_mul == 1:
T_cur = epoch % T_0
T_i = T_0
cur_lr = eta_min + 0.5 * (eta_max - eta_min) * (1 + np.cos(np.pi * T_cur / T_i))
for param_group in optimizer.param_groups:
param_group['lr'] = cur_lr
optimizer = optim.SGD(net.parameters(), lr=0.1, momentum=0.9, weight_decay=1e-4, nesterov=True)
for epoch in range(100):
adjust_lr(optimizer, epoch)
train()