2.7 Adam

这一篇写得特别详细：
深度学习优化算法解析(Momentum, RMSProp, Adam)
Adam（Adaptive Moment Estimation）
初始化：

$$v_{dW}=0, v_{db}=0, S_{dW}=0, S_{db}=0$$

On iteration t:
compute $dW, db$ using mini batch

$$\begin{align*} &Momentum: \\ &v_{dW}=\beta_1 v_{dW}+(1-\beta_1) dW \quad v_{db}=\beta_1 v_{db}+(1-\beta_1)db\\ &RMSprop: \\ & S_{dW}=\beta_2S_{dW}+(1-\beta_2)dW^2 \quad S_{db}=\beta_2 S_{db}+(1-\beta_2)db^2\\ &\\ & Bias \quad correction:\\ & v_{dW}^{corrected}=\frac{v_{dW}}{1-\beta_1^2} \quad v_{db}^{corrected}=\frac{v_{db}}{1-\beta_1^2}\\ & S_{dW}^{corrected}=\frac{S_{dW}}{1-\beta_2^2} \quad S_{db}^{corrected}=\frac{S_{db}}{1-\beta_2^2}\\ &\\ & Computation:\\ & W=W-\alpha \frac{v_{dW}}{\sqrt{S_{dW}}}\\ & b=b-\alpha \frac{v_{db}}{\sqrt{S_{db}}} \end{align*}$$

When implementing Adam, what people usually do is just use the default value. So, $\beta_1$ and $\beta_2$ as well as $\epsilon$. I don’t think anyone ever really tunes Epsilon. And then, try a range of values of Alpha to see what works best.

So, where does the term ‘Adam’ come from?

Adam stands for Adaptive Moment Estimation. So $\beta_1$ is computing the mean of the derivatives. This is called the first moment. And $\beta_2$ is used to compute exponentially weighted average of the squares and that’s called the second moment. So that gives rise to the name adaptive moment estimation.