2019最牛的梯度优化算法出炉,AdaBound实验对比代码
https://github.com/LiyuanLucasLiu/RAdam
lookahead
73上用的这个:
https://github.com/alphadl/lookahead.pytorch
https://github.com/dseuss/pytorch-lookahead-optimizer/blob/master/optim.py
这个有用法:
base_opt = torch.optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999)) # Any optimizer
lookahead = Lookahead(base_opt, k=5, alpha=0.5) # I