Notes for Deep Learning Lessons of Pro. Hung-yi Lee (2)

Today, Knowledge concerning about the optimization of deep learning is written here. What is the meaning of optimaztion? The following ppt shows us the answer.

1. SGD with Momentum (SGDM)

Just as the name shows us, SGDM is invented by combining SGD with Momentum. For SGDM, the process of updating parameters is shown in the following ppt. What we should pay attention to, or in the other words what makes us better understand the meanings of SGDM, is $v^i$ is actually the weighted sum of all the previous gradient and the closer gradient has more influence on current momentum.

What is the adavantage of adding momentum in SGD program? For SGD, it is easy to lead us to local minima point rather than the global minima point. However, adding momentum takes the history information into account, which means, if we explains it in a more vivid way, SGDM offers us the ability to think whether we are standing at the local minima point.

2. Adagrad

It has been introduced in last blog. So, I do not want to explain it again. (I am a little bit lazy, haha).

3. RMSProp

RMSProp makes a little change on the formula of Adagrad. In Adagrad, $v_t$ is the sum of square of the past gradient. However, in RMSProp, $v_t=\alpha v_{t-1}+(1-\alpha )(g_{t-1}{)}^2$. We can changes the value of $\alpha$ to make $v_{t-1}$ have more or less influence on the current gradient. In common situation, we always set $\alpha$ as a large number with the afraid of a too large $g_{t-1}$ making $\frac{\eta }{\sqrt{v_t}}$ too close to zero.

4. Adam

If we ignore some little differences, Adam can be seen as the combination of SGDM and RMSProp. The little change is that we change the form of $m_t$, which can be called as de-biasing. The reason of this change is that the value of $m_t$ is too close to zero at the beginning of updating.