1. SGD + Momentum for Oscillation and Plateau Problem

1. SGD method has trouble when navigating areas where the curvature is steeper in one dimension and is flat in another direction.
   • Ends up oscillating around the slopes and maks slow progress

   可以看到，在同一位置上，目标标函数在垂直方向上得斜率得绝对值远大于在水平方向上的斜率。因此，给定学习步长，SGD迭代自变量时会使自变量在竖直方向上的移动幅度过大，甚至越过最优解。这样不断地振荡，会导致向最优解移动的效率变慢。　　
　期望让寻找最优解的曲线能够更加的平滑，在水平方向的速度更快: Momentum(动量)

2. Incorporation of Momentum
   ${\mathbf{v}}_{\mathbf{t}}=\mathbf{\beta }{\mathbf{v}}_{\mathbf{t}-1}+{\mathbf{\eta }}_{\mathbf{t}}{\mathbf{g}}_{\mathbf{t}},$ $\beta \in [0,1);v_0=0$ and $g_t$ is the gradient at current x
   ${\mathbf{x}}_{\mathbf{t}}={\mathbf{x}}_{\mathbf{t}-1}-{\mathbf{v}}_{\mathbf{t}}$

   • if $\beta =0$, it’s the normal SGD method
     
   • $v_t=\beta v_{t-1}+(1-\beta )\frac{{\eta }_t{g}_t}{1-\beta }＝{\mathbf{\beta }}^{\mathbf{t}}{\mathbf{v}}_0+(1-\mathbf{\beta }){\sum }_{\mathbf{i}=0}^{\mathbf{t}-\mathbf{i}}{\mathbf{\beta }}^{\mathbf{i}}\frac{{\mathbf{\eta }}_{\mathbf{t}-\mathbf{i}}{\mathbf{g}}_{\mathbf{t}-\mathbf{i}}}{1-\mathbf{\beta }}$
   • The weight on $\frac{{\eta }_{t-i}{g}_{t-i}}{1-\beta }=(1-\beta ){\beta }^i$ exponential decreases as i increases.
   • $v_t$ is the eponentially weighted moving average of past v’s
   • Let $n=\frac{1}{1-\beta },\beta =1-\frac{1}{n}$

     • $li{m}_{n->\infty }(1-\frac{1}{n}{)}^n=\frac{1}{e}\approx 0.3679$

     • If we treat this number as a small number, then we can ignore all terms including ${\beta }^{\frac{1}{1-\beta }}$ and higher terms when $\beta \to 1.$ $\mathbf{\beta }\in [0.9,1)$.

       For example: if $\beta =0.95,{\beta }^{1-\beta }\approx 20,v_t＝{\beta }^t{v}_0+(1-\beta ){\sum }_{i=0}^{19}{\beta }^i\frac{{\eta }_{t-i}{g}_{t-i}}{1-\beta }$

     • For mementum, the updated term for the x is approximated equal to the exponentially weighted moving average of previous $\frac{1}{1-\beta }$ updated terms (${\eta }_{t-i}{g}_{t-i}$) and then divided by $1-\beta$

3. Momentum

   • Reduce updates along directions that changes gradients frequently
   • Increase updates along directions that gradients are consistent
   • Dampen oscillations

图转自知乎https://zhuanlan.zhihu.com/p/34240246