pytorch的在学习率的调整上提供了很多自定义的函数可供使用,让我们一起来可视化看看呗
说明:
- 本文使用的pytorch版本为0.4,新版本可能存在参数不匹配问题
准备过程
class model(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3)
def forward(self, x):
pass
net_1 = model()
optimizer = torch.optim.Adam(net_1.parameters(), lr = 1) # 为了可视化方便
LambdaLR
torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda, last_epoch=-1)
参数
lr_lambda::可以传入list或者function
last_epoch:在恢复训练的时候要用到,指当前epoch
说明
看名字就可以联想python内置的lambda函数,可以实现自定义的学习率,例如实现一个这样子的学习率:
l
r
=
i
n
i
t
_
l
r
∗
(
1
−
e
p
o
c
h
n
u
m
_
e
p
o
c
h
s
)
p
o
w
e
r
\mathit{lr} = \mathit{init\_lr} * \Big(1 - \frac{\mathit{epoch}}{\mathit{num\_epochs}}\Big)^{\mathit{power}}
lr=init_lr∗(1−num_epochsepoch)power
def poly_lr_scheduler(epoch, num_epochs=30, power=0.95):
return (1 - epoch/num_epochs)**power
scheduler = LambdaLR(optimizer, lr_lambda=poly_lr_scheduler)
当num_epochs = 30;power = 0.5时:
StepLR
torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)
参数
step_size
gamma
last_epoch
说明
学习率变化公式:
n
e
w
l
r
=
i
n
i
t
i
a
l
l
r
×
γ
e
p
o
c
h
/
/
s
t
e
p
s
i
z
e
new_{lr}=initial_{lr}×\gamma^{epoch//step_size}
newlr=initiallr×γepoch//stepsize
当step_size = 3;gamma = 0.5时:
MultiStepLR
torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1):
参数
milestones
gamma
last_epoch
说明
学习率变化公式:
n
e
w
l
r
=
i
n
i
t
i
a
l
l
r
×
γ
,
i
f
n
o
w
e
p
o
c
h
i
n
m
i
l
e
s
t
o
n
e
s
new_{lr}=initial_{lr}×\gamma, \quad if \, now \, epoch \, in \, milestones
newlr=initiallr×γ,ifnowepochinmilestones
当milestones = [4,8,12,20];gamma = 0.5时:
参考文献:
- https://blog.csdn.net/qyhaill/article/details/103043637
- https://butui.me/post/lamdalr-in-pytorch/
- https://pytorch.org/docs/0.4.0/_modules/torch/optim/lr_scheduler.html#LambdaLR
- https://www.zhihu.com/question/277807090