optimizer总结

SGD -> SGDM -> NAG ->AdaGrad -> AdaDelta -> Adam -> Nadam

1、torch.optim.SGD

  参考资料：

    1）[2010]Large-Scale Machine Learning with Stochastic Gradient Descent

    2）Stochastic Gradient Descent Tricks

    3）On the importance of initialization and momentum in deep learning

2、torch.optim.ASGD

  参考资料：

    Acceleration of stochastic approximation by averaging

3、torch.optim.Adagrad

  参考资料：

    Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

4、torch.optim.Adadelta

  参考资料：

    ADADELTA: An Adaptive Learning Rate Method

5、torch.optim.Adam

  参考资料：

    1）Adam: A Method for Stochastic Optimization

    2）Decoupled Weight Decay Regularization

6、torch.optim.Adamax

  参考资料：

    Adam: A Method for Stochastic Optimization

7、torch.optim.SparseAdam

8、torch.optim.AdamW

  参考资料：

    1）Adam: A Method for Stochastic Optimization

    2）Decoupled Weight Decay Regularization

9、torch.optim.Rprop

  参考资料：

    resilient backpropagation algorithm

10、torch.optim.RMSprop

  参考资料：

    1）G. Hinton课程（https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf）

    2）Generating Sequences With Recurrent Neural Networks

11、torch.optim.LBFGS

  参考资料：

    minFunc <https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html>