![4a070c7202de323153fbb9bdec48d584.png](https://i-blog.csdnimg.cn/blog_migrate/76068f12e58392736301ea92a6eff9ca.jpeg)
论文 An overview of gradient descent optimization algorithms 的个人总结。
基于梯度下降的优化方法中,最朴素的 SGD 优化器采用的优化策略如下:
![equation?tex=%5Ctheta+%3D+%5Ctheta+-+%5Cmu+%5Ccdot+%5Cnabla+J%28%5Ctheta%29](https://i-blog.csdnimg.cn/blog_migrate/e11faff3947778f8a698e59b066018b4.png)
其他优化方法均基于对学习率
![equation?tex=%5Cmu](https://i-blog.csdnimg.cn/blog_migrate/4faf95e734b2356b46393046586d6ff5.png)
1. Momentum
Keywords:momentum, 全局学习率
如下图图 a 所示,当一个维度比另一个维度下降地明显更加急促时(经常是局部最优点),朴素 SGD 容易存在收敛极慢的问题。
![31dbd5f13fcf1a6a8c79a530db27b099.png](https://i-blog.csdnimg.cn/blog_migrate/cb845be51a9919fac2a41a9325effc31.png)
momentum(动量)的引入可以直观地较好处理这个问题,其在计算当前时刻的更新向量
![equation?tex=v_t](https://i-blog.csdnimg.cn/blog_migrate/652e673b5d65d85c5b3ba46e80ed886e.png)
![equation?tex=v_%7Bt-1%7D](https://i-blog.csdnimg.cn/blog_migrate/4183ab388a0a217f47faf0c2427b43c4.png)
![equation?tex=v_t+%3D+%5Cgamma+v_%7Bt-1%7D+%2B+%5Ceta+%5Cnabla_%5Ctheta+J%28%5Ctheta%29](https://i-blog.csdnimg.cn/blog_migrate/652e673b5d65d85c5b3ba46e80ed886e.png+%3D+%5Cgamma+v_%7Bt-1%7D+%2B+%5Ceta+%5Cnabla_%5Ctheta+J%28%5Ctheta%29)
![equation?tex=%5Cgamma](https://i-blog.csdnimg.cn/blog_migrate/d6176aef248cf28709595e799ff4c7d9.png)
![equation?tex=%5Ctheta+%3D+%5Ctheta+-+v_t](https://i-blog.csdnimg.cn/blog_migrate/5142bb556e9dca19cc19d38c68c41a29.png)
2. Nesterov accelerated gradient
Keywords:momentum, 全局学习率,前瞻性预估
在momentum基础上,Nesterov accelerating 提供了一种颇有前瞻性的参数更新预估。
我们知道,动量将
![equation?tex=%5Cgamma+v_%7Bt-1%7D](https://i-blog.csdnimg.cn/blog_migrate/d6176aef248cf28709595e799ff4c7d9.png+v_%7Bt-1%7D)
![equation?tex=%5Ctheta](https://i-blog.csdnimg.cn/blog_migrate/679f2e1de0bb439ac572e914fed0f16f.png)
![equation?tex=%5Ctheta+-+%5Cgamma+v_%7Bt-1%7D](https://i-blog.csdnimg.cn/blog_migrate/679f2e1de0bb439ac572e914fed0f16f.png+-+%5Cgamma+v_%7Bt-1%7D)
![equation?tex=v_t+%3D+%5Cgamma+v_%7Bt-1%7D+%2B+%5Ceta+%5Cnabla_%5Ctheta+J%28%5Ctheta+-+%5Cgamma+v_%7Bt-1%7D%29+](https://i-blog.csdnimg.cn/blog_migrate/652e673b5d65d85c5b3ba46e80ed886e.png+%3D+%5Cgamma+v_%7Bt-1%7D+%2B+%5Ceta+%5Cnabla_%5Ctheta+J%28%5Ctheta+-+%5Cgamma+v_%7Bt-1%7D%29+)
![equation?tex=%5Cgamma](https://i-blog.csdnimg.cn/blog_migrate/d6176aef248cf28709595e799ff4c7d9.png)
![equation?tex=%5Ctheta+%3D+%5Ctheta+-+v_t](https://i-blog.csdnimg.cn/blog_migrate/5142bb556e9dca19cc19d38c68c41a29.png)
3. Adagrad
Kerwords: 不同参数自适应学习率,总体学习率不断衰减
Adagrad 为每个参数提供自适应的学习率,它可以为频率低的参数每次提供更大更新、为频率高的参数每次提供更小更新。正因为此,它十分适用于处理稀疏数据。
用
![equation?tex=g_%7Bt%2Ci%7D](https://i-blog.csdnimg.cn/blog_migrate/c944552dabaa658c6831eeca0928b523.png)
![equation?tex=t](https://i-blog.csdnimg.cn/blog_migrate/3b1922fd3ef8dd7c883fb35b699da495.png)
![equation?tex=g_%7Bt%2Ci%7D+%3D+%5Cnabla_%7B%5Ctheta_i%7D+J%28%5Ctheta_%7Bt%2Ci%7D%29](https://i-blog.csdnimg.cn/blog_migrate/c944552dabaa658c6831eeca0928b523.png+%3D+%5Cnabla_%7B%5Ctheta_i%7D+J%28%5Ctheta_%7Bt%2Ci%7D%29)
则 SGD 对每个参数的更新为:
Adagrad对每个参数有自适应的学习率,其参数更新过程为:
![equation?tex=%5Cepsilon](https://i-blog.csdnimg.cn/blog_migrate/9d83d5a736cfe878f7d1f9052bd2ee7b.png)
![equation?tex=G_t](https://i-blog.csdnimg.cn/blog_migrate/ab64bc0fc468aaa3b1de07135b3aecb5.png)
![equation?tex=i](https://i-blog.csdnimg.cn/blog_migrate/b1677f9423a31be742701330c190cf05.png)
![equation?tex=t](https://i-blog.csdnimg.cn/blog_migrate/3b1922fd3ef8dd7c883fb35b699da495.png)
Adagrad的最大优势是初始学习率(一般为0.01)设定后不用管,其可以为不同参数自适应地调整学习率。但同时,随着训练进行
![equation?tex=G_t](https://i-blog.csdnimg.cn/blog_migrate/ab64bc0fc468aaa3b1de07135b3aecb5.png)
Adadelta针对性地解决了Adagrad学习率过度衰减的问题。
4. Adadelta
Keywords: 自适应学习率,解决Adagrad学习率衰减过小的问题
Adadelta解决了Adagrad过度激进地衰减学习率的问题,不同于累加过去所有梯度的平方,它限制了累加的过去的梯度到一个固定窗口宽度
![equation?tex=w](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png)
为进一步简化存储
![equation?tex=w](https://i-blog.csdnimg.cn/blog_migrate/1c7a330b82678d6b033d9748f5120af9.png)
![equation?tex=t](https://i-blog.csdnimg.cn/blog_migrate/3b1922fd3ef8dd7c883fb35b699da495.png)
![equation?tex=E%5Bg%5E2%5D_t+%3D+%5Cgamma+E%5Bg%5E2%5D_%7Bt-1%7D+%2B+%281+-+%5Cgamma%29g%5E2_t](https://i-blog.csdnimg.cn/blog_migrate/8054fb577291e85d250a086ec23d497e.png)
![equation?tex=%5Cgamma](https://i-blog.csdnimg.cn/blog_migrate/d6176aef248cf28709595e799ff4c7d9.png)
因此参数更新向量为:
![equation?tex=%5CDelta+%5Ctheta_t+%3D+-+%5Cfrac+%7B%5Ceta%7D+%7B%5Csqrt%7BE%5Bg%5E2%5D_t+%2B+%5Cepsilon%7D%7D+g_t](https://i-blog.csdnimg.cn/blog_migrate/b99f928f8a3131833723569241c31eb2.png)
其分母为均方根(root mean squared, RMS),因此也常写作:
![equation?tex=%5CDelta+%5Ctheta_t+%3D+-+%5Cfrac+%7B%5Ceta%7D+%7BRMS%5Bg%5D_t%7D+g_t](https://i-blog.csdnimg.cn/blog_migrate/6426b35ade346a2488612a1e9807afa8.png)
本方法的作者进一步提出了更新单元不匹配的问题,时刻
![equation?tex=t](https://i-blog.csdnimg.cn/blog_migrate/3b1922fd3ef8dd7c883fb35b699da495.png)
![equation?tex=E%5B%5CDelta+%5Ctheta%5E2%5D_t+%3D+%5Cgamma+E%5B%5CDelta+%5Ctheta%5E2%5D_%7Bt-1%7D+%2B+%281+-+%5Cgamma%29%5CDelta+%5Ctheta%5E2_t](https://i-blog.csdnimg.cn/blog_migrate/daac3c9e3a09c968e9c788ae40a7c334.png)
有:
![equation?tex=RMS%5B%5CDelta+%5Ctheta%5D_t+%3D+%5Csqrt%7BE%5B%5CDelta+%5Ctheta%5E2%5D_t+%2B+%5Cepsilon%7D](https://i-blog.csdnimg.cn/blog_migrate/7e223b30a04e2c1ecd88f77aa90f17fd.png)
由于时刻
![equation?tex=t](https://i-blog.csdnimg.cn/blog_migrate/3b1922fd3ef8dd7c883fb35b699da495.png)
![equation?tex=RMS%5B%5CDelta+%5Ctheta%5D_t](https://i-blog.csdnimg.cn/blog_migrate/63b74ff9bcbdfc809d301ef24088d1e2.png)
![equation?tex=t-1](https://i-blog.csdnimg.cn/blog_migrate/3b1922fd3ef8dd7c883fb35b699da495.png-1)
![equation?tex=%5CDelta+%5Ctheta_t+%3D+-+%5Cfrac%7BRMS%5B%5CDelta+%5Ctheta%5D_%7Bt-1%7D%7D%7BRMS%5Bg%5D_t%7Dg_t](https://i-blog.csdnimg.cn/blog_migrate/17a750ca54fef86017aa8c3acac48ba8.png)
可以看出,Adadelta事实上不需要设定初始学习率。
5. RMSprop
RMSprot 和 Adadelta 都是为了解决Adagrad的学习率消失问题而提出的。有趣的是,二者被独立提出,却殊途共归。RMSprop的参数更新过程为:
![equation?tex=E%5Bg%5E2%5D_t+%3D+0.9+E%5Bg%5E2%5D_%7Bt-1%7D+%2B+0.1+g%5E2_t](https://i-blog.csdnimg.cn/blog_migrate/b57f271a72a21c9cc2c6d1409fa3a085.png)
![equation?tex=%5Ceta](https://i-blog.csdnimg.cn/blog_migrate/52f604df752ab3306ce22892eb63dd0a.png)
6. Adam
Adaptive Moment Estimation(Adam)是另一个为每个参数动态调整学习率的方法。正如其名,它同时估计了梯度的 first moment(mean,
![equation?tex=m_t](https://i-blog.csdnimg.cn/blog_migrate/023af16ff75b43aa0ae1a0f75512caf0.png)
![equation?tex=v_t](https://i-blog.csdnimg.cn/blog_migrate/652e673b5d65d85c5b3ba46e80ed886e.png)
![equation?tex=m_t+%3D+%5Cbeta_1+m_%7Bt-1%7D+%2B+%281+-+%5Cbeta_1%29+g_t](https://i-blog.csdnimg.cn/blog_migrate/023af16ff75b43aa0ae1a0f75512caf0.png+%3D+%5Cbeta_1+m_%7Bt-1%7D+%2B+%281+-+%5Cbeta_1%29+g_t)
![equation?tex=v_t+%3D+%5Cbeta_2+v_%7Bt-1%7D+%2B+%281-+%5Cbeta_2%29g%5E2_t](https://i-blog.csdnimg.cn/blog_migrate/652e673b5d65d85c5b3ba46e80ed886e.png+%3D+%5Cbeta_2+v_%7Bt-1%7D+%2B+%281-+%5Cbeta_2%29g%5E2_t)
![equation?tex=%5Cbeta_1](https://i-blog.csdnimg.cn/blog_migrate/6867ea0e044fa584f9c0928ceb5e0380.png)
![equation?tex=%5Cbeta_2](https://i-blog.csdnimg.cn/blog_migrate/e13651ebd2e1546d27cb1e503b116547.png)
由于
![equation?tex=m_t](https://i-blog.csdnimg.cn/blog_migrate/023af16ff75b43aa0ae1a0f75512caf0.png)
![equation?tex=n_t](https://i-blog.csdnimg.cn/blog_migrate/2c717fedd3c137ab38a3325c9139a847.png)
![equation?tex=%5Cbeta_1](https://i-blog.csdnimg.cn/blog_migrate/6867ea0e044fa584f9c0928ceb5e0380.png)
![equation?tex=%5Cbeta_2](https://i-blog.csdnimg.cn/blog_migrate/e13651ebd2e1546d27cb1e503b116547.png)
![equation?tex=m_t](https://i-blog.csdnimg.cn/blog_migrate/023af16ff75b43aa0ae1a0f75512caf0.png)
![equation?tex=n_t](https://i-blog.csdnimg.cn/blog_migrate/2c717fedd3c137ab38a3325c9139a847.png)
为解决这一问题,moment estimation 做了 bias-correct:
![equation?tex=%5Chat%7Bm%7D_t+%3D+%5Cfrac%7Bm_t%7D%7B1-%5Cbeta%5Et_1%7D+](https://i-blog.csdnimg.cn/blog_migrate/c107a82199700eb83b8b4a4d060ef532.png)
![equation?tex=%5Chat+v_t+%3D+%5Cfrac%7Bv_t%7D%7B1-%5Cbeta%5Et_2%7D](https://i-blog.csdnimg.cn/blog_migrate/15ddb7abf295ebd6d9a277da376de4c9.png)
参数更新过程为:
Finally,how to choose optimizer?
稀疏数据则选择自适应学习率的算法;而且,只需设定初始学习率而不用再调整即很可能实现最好效果。
Adagrad, Adadelta, RMSprop, Adam可以视为一类算法。RMSprop 与 Adadelta本质相同,都是为了解决Adagrad的学习率消失问题。Adam 可视为添加了 bias-correction 和 momentum的 RMSprop。事实上,Adadelta, RMSprop, Adam在相似场景下表现也很相似。bias-correction 帮助 Adam 在优化后期略优于 RMSprop,因为此时梯度已经很稀疏。
目前来看,无脑用 Adam 似乎已经是最佳选择。