学习率预热（warmup）代码测试

在看霹导的FCN代码，里面有个学习率预热，为了看看学习率是如何变化的，将学习率预热代码跟训练过程结合到一起，把学习率的变换过程打印出来，代码如下。
注意LambdaLR调整学习率的方式为：学习率 = 初始学习率 * 给定lr_lambda函数的值。

torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=f)

from matplotlib import pyplot as plt

warmup = True
initlr = 0.0001
warmup_epochs = 1
num_step = 366  # 一整份数据，按batch存放。VOC2012，batchsize=4
epochs = 30  # epoch总数
warmup_factor = 1e-3
value = []
return_value = 0
for epoch in range(epochs):
    for step in range(num_step):
        x = epoch * num_step + step + 1
        """
           根据step数返回一个学习率倍率因子，
           注意在训练开始之前，pytorch会提前调用一次lr_scheduler.step()方法
           """
        if warmup is True and x <= (warmup_epochs * num_step):
            alpha = float(x) / (warmup_epochs * num_step)
            # warmup过程中lr倍率因子从warmup_factor -> 1
            return_value = warmup_factor * (1 - alpha) + alpha
        else:
            # warmup后lr倍率因子从1 -> 0
            # 参考deeplab_v2: Learning rate policy
            return_value = (1 - (x - warmup_epochs * num_step) / ((epochs - warmup_epochs) * num_step)) ** 0.9
        value.append(return_value * initlr)
    print(return_value * initlr)
    
plt.plot(value, '*')
plt.show()

结果：

0.0001
9.689113204622e-05
9.377113865345655e-05
9.063956551818755e-05
8.749592154663483e-05
8.433967427912744e-05
8.117024453014075e-05
7.798700006942841e-05
7.478924812104509e-05
7.157622639192217e-05
6.834709225327746e-05
6.510090957654228e-05
6.183663255563447e-05
5.855308560613402e-05
5.52489380827954e-05
5.19226720411192e-05
4.857254048910653e-05
4.5196512365661785e-05
4.179219854861471e-05
3.835674999767602e-05
3.4886713635556415e-05
3.137782165113512e-05
2.782467099882336e-05
2.4220211276354852e-05
2.055487337832409e-05
1.6814958278941684e-05
1.2979289851509707e-05
9.01091302851232e-06
4.828828728591015e-06
0.0