SGD -> SGDM -> NAG ->AdaGrad -> AdaDelta -> Adam -> Nadam
1、torch.optim.
SGD
参考资料:
1)[2010]Large-Scale Machine Learning with Stochastic Gradient Descent
2)Stochastic Gradient Descent Tricks
3)
On the importance of initialization and momentum in deep learning
2、torch.optim.
ASGD
参考资料:
Acceleration of stochastic approximation by averaging
3、torch.optim.Adagrad
参考资料:
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
4、torch.optim.
Adadelta
参考资料:
ADADELTA: An Adaptive Learning Rate Method
5、torch.optim.
Adam
参考资料:
1)Adam: A Method for Stochastic Optimization
2)Decoupled Weight Decay Regularization
6、torch.optim.
Adamax
参考资料:
Adam: A Method for Stochastic Optimization
7、torch.optim.
SparseAdam
8、torch.optim.
AdamW
参考资料:
1)Adam: A Method for Stochastic Optimization
2)Decoupled Weight Decay Regularization
9、torch.optim.Rprop
参考资料:
resilient backpropagation algorithm
10、torch.optim.
RMSprop
参考资料:
1)G. Hinton课程(
https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)
2)Generating Sequences With Recurrent Neural Networks
11、torch.optim.
LBFGS
参考资料:
minFunc <https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html>