CyclicLR pytorch实现

论文：https://arxiv.org/abs/1506.01186
论文中给了keras版本

1、如何寻找最大和最小学习率：
参考https://github.com/davidtvs/pytorch-lr-finder
安装：

#python3.5以上
pip install torch-lr-finder
# for mixed precision training
pip install torch-lr-finder -v --global-option='apex'

# implementation, two versions
# Tweaked version from fastai
from torch_lr_finder import LRFinder

model = ...
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-7, weight_decay=1e-2)
lr_finder = LRFinder(model, optimizer, criterion, device="cuda")
lr_finder.range_test(trainloader, end_lr=100, num_iter=100)
lr_finder.plot() # to inspect the loss-learning rate graph
lr_finder.reset() # to reset the model and optimizer to their initial state

# Leslie Smith's approach
from torch_lr_finder import LRFinder

model = ...
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.1, weight_decay=1e-2)
lr_finder = LRFinder(model, optimizer, criterion, device="cuda")
lr_finder.range_test(trainloader, val_loader=val_loader, end_lr=1, num_iter=100, step_mode="linear")
lr_finder.plot(log_lr=False)
lr_finder.reset()

2、如何使用：
官方版本：据说pytorch1.1以上才能用。注意，lr更新是每个batch更新，不是每个epoch更新

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.01, max_lr=0.1)
data_loader = torch.utils.data.DataLoader(...)
for epoch in range(10):
    for batch in data_loader:
        train_batch(...)
        scheduler.step()

如果pytorch版本不够，可以参考https://pypi.org/project/pytorch-zoo/#cyclicalmomentumoptimizer-base_momentum08-max_momentum09-step_size2000-modetriangular
（未实践）

!pip install pytorch_zoo
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.CyclicMomentum(optimizer)
data_loader = torch.utils.data.DataLoader(...)
for epoch in range(10):
    for batch in data_loader:
        scheduler.batch_step()
        train_batch(...)