New Optimization

一、Adam与SGDM

Adam：fast training, large generalization gap,
unstable
• SGDM：stable, little generalization gap, better
convergence(?) 

二、Simply combine Adam with SGDM

SWATS [Keskar, et al., arXiv’17]
Begin with Adam(fast), end with SGDM 

Start
training
Adam
Meet some
criteria
Convergence

三、Towards Improving SGDM

Adaptive learning rate algorithms：dynamically
adjust learning rate over time
• SGD-type algorithms：fix learning rate for all
updates… too slow for small learning rates and
bad result for large learning rates

四、Advices

SGDM：Computer vision
Adam
• NLP
• Speech synthesis
• GAN
• Reinforcement learning