深度学习优化器：《Lookahead Optimizer: k steps forward, 1 step back》


项目地址：

 https://github.com/michaelrzhang/lookahead


pytorch版本：

 https://github.com/michaelrzhang/lookahead/blob/master/lookahead_pytorch.py


论文地址：

 https://arxiv.org/abs/1907.08610


使用方法：（pytorch）

optimizer = # {any optimizer} e.g. torch.optim.Adam
if args.lookahead:
    optimizer = Lookahead(optimizer, la_steps=args.la_steps, la_alpha=args.la_alpha)
  • 1.
  • 2.
  • 3.


We found that evaluation performance is typically better using the slow weights. This can be done in PyTorch with something like this in your eval loop:


if args.lookahead:
    optimizer._backup_and_load_cache()
    val_loss = eval_func(model)
    optimizer._clear_and_load_backup()
  • 1.
  • 2.
  • 3.
  • 4.




@article{zhang2019lookahead,

title={Lookahead Optimizer: k steps forward, 1 step back},

author={Zhang, Michael R and Lucas, James and Hinton, Geoffrey and Ba, Jimmy},

journal={arXiv preprint arXiv:1907.08610},

year={2019}

}
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.