一般来说,在模型训练的后期,学习率就可以适当调低一些,以获得更好的训练效果。
需要准备记录这些参数:
- epochs_since_improvement:如果一个新的epoch并没有降低loss,则其加一;反之,则清零
- decays_since_improvement:如果触发了一次学习率调整,则其加一;而如果一个新的epoch下loss降低,则清零
- shrink_factor:收缩因子,控制学习率的缩小速度。默认为0.6,decay每加1,则再乘0.6,加快学习率的收缩速度
准备学习率调整函数:
def adjust_learning_rate(optimizer, shrink_factor):
print("\nDECAYING learning rate.")
for param_group in optimizer.param_groups:
param_group['lr'] = param_group['lr'] * shrink_factor
print("The new learning rate is %f\n" % (optimizer.param_groups[0]['lr'],))
其实就是取出优化器的lr参数进行修改。
调用:
if epochs_since_improvement == 10:
break
if epochs_since_improvement > 0 and epochs_since_improvement % 2 == 0:
checkpoint = 'BEST_checkpoint.tar'
checkpoint = torch.load(checkpoint)
model = checkpoint['model']
optimizer = checkpoint['optimizer']
decays_since_improvement += 1
print("\nDecays since last improvement: %d\n" % (decays_since_improvement,))
adjust_learning_rate(optimizer, 0.6 ** decays_since_improvement)