简单例子说明优化器

最新推荐文章于 2022-10-22 00:50:12 发布

Jeff-Chow000

最新推荐文章于 2022-10-22 00:50:12 发布

阅读量162

点赞数

文章标签： 1024程序员节深度学习

本文链接：https://blog.csdn.net/u014611178/article/details/109252955

版权

简单例子说明优化器

我们会用下面的例子来说明优化器：

数据集：
$x_1,y_1) = (1,3) \\ (x_2,y_2) = (2,7) \\ (x_3,y_3) = (3,8) \\ (x_4,y_4) = (4,10) \\ (x_5,y_5) = (5,14)$
损失函数：
$\frac12(y-wx)^2\\$
损失函数对变量 $w$ 的梯度：
$\frac{\partial L(x,y;w)}{\partial w} = -x(y-wx)$
学习率： $\eta=0.01$

初始值： $w_0=0.01$

简单优化器

GD（Gradient Descent）

使用全部数据计算梯度。

更新公式：
$w_{t} = w_{t-1} - \eta G_{t-1} = w_{t-1} - \eta \frac1M \sum_{i=1}^M \frac{\partial L(x_i,y_i;w_{t-1})}{\partial w}$
算每个数据的梯度：
$\frac{\partial L(x_1,y_1;w_{0})}{\partial w} = -x_1(y_1-w_0 x_1) = -1 \times (3-1\times1) = -2 \\ \frac{\partial L(x_2,y_2;w_{0})}{\partial w} = -x_1(y_2-w_0 x_2) = -2 \times (7-1\times2) = -10 \\ \frac{\partial L(x_3,y_3;w_{0})}{\partial w} = -x_1(y_3-w_0 x_3) = -3 \times (8-1\times3) = -15 \\ \frac{\partial L(x_4,y_4;w_{0})}{\partial w} = -x_4(y_4-w_0 x_4) = -4 \times (10-1\times4) = -24 \\ \frac{\partial L(x_5,y_5;w_{0})}{\partial w} = -x_5(y_5-w_0 x_5) = -5 \times (14-1\times5) = -45 \\$
计算平均梯度：
$G_0 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{0})}{\partial w} = -\frac{2+10+15+24+45}{5} = -19.2$
迭代：
$w_{1} = w_{0} - \eta G_0 = 1 - 0.01 \times (-19.2) = 1.192$

SGD（Stochastic Gradient Descent）

使用batch_size个数据计算梯度。

更新公式与Gradient Descent的一样。

设置batch_size=2，随机抽两条数据：
$x_2,y_2) = (2,7) \\ (x_3,y_3) = (3,8)$
计算梯度：
$\frac{\partial L(x_2,y_2;w_{0})}{\partial w} = -10 \\ \frac{\partial L(x_3,y_3;w_{0})}{\partial w} = -15 \\$
计算平均梯度：
$G_0 = \frac12 [\frac{\partial L(x_2,y_2;w_{0})}{\partial w} + \frac{\partial L(x_3,y_3;w_{0})}{\partial w}] = -\frac{10+15}{2} = -12.5$
迭代：
$w_{1} = w_{0} - \eta G_{0} = 1 - 0.01 \times (-12.5) = 1.125$

优化器改进策略

优化器的改进思路有2条：

改进梯度
改进学习率

梯度改进策略

Momentum

增添之前更新的反向梯度。

更新公式：
$m_{t} = \mu m_{t-1} + \eta G_{t-1} = \mu m_{t-1} - \eta \frac1M \sum_{i=1}^M \frac{\partial L(x_i,y_i;w_{t-1})}{\partial w} \\ w_{t} = w_{t-1} - m_{t}$
平均梯度（讲Gradient Descent的时候已经算好）：
$G_0 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{0})}{\partial w} = -\frac{2+10+15+24+45}{5} = -19.2$
初始值： $m_0=0$ ， $\mu=0.3$

第一次迭代：
$m_{1} = \mu m_{0} + \eta G_0 = 0.01 \times (-19.2) = -0.192 \\ w_{1} = w_{0} - m_{1} = 1 - (-0.192) = 1.192$
算每个数据的梯度：
$\frac{\partial L(x_1,y_1;w_{1})}{\partial w} = -x_1(y_1-w_1 x_1) = -1 \times (3-1.192\times1) = -1.808 \\ \frac{\partial L(x_2,y_2;w_{1})}{\partial w} = -x_1(y_2-w_1 x_2) = -2 \times (7-1.192\times2) = -9.232 \\ \frac{\partial L(x_3,y_3;w_{1})}{\partial w} = -x_1(y_3-w_1 x_3) = -3 \times (8-1.192\times3) = -13.272 \\ \frac{\partial L(x_4,y_4;w_{1})}{\partial w} = -x_4(y_4-w_1 x_4) = -4 \times (10-1.192\times4) = -20.928 \\ \frac{\partial L(x_5,y_5;w_{1})}{\partial w} = -x_5(y_5-w_1 x_5) = -5 \times (14-1.192\times5) = -40.2 \\$
计算平均梯度：
$G_1 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{1})}{\partial w} = -\frac{1.808+9.232+13.272+20.928+40.2}{5} = -17.088$
第二次迭代：
$m_{2} = \mu m_{1} + \eta G_1 = 0.3 \times (-1.192) + 0.01 \times (-17.088) = -0.52848 \\ w_{2} = w_{1} - m_{2} = 1.192 - (-0.52848) = 1.72048$

Nesterov Momentum

更新公式：
$m_{t} = \mu m_{t-1} + \eta G_{t-1} = w_{t-1} - \eta \frac1M \sum_{i=1}^M \frac{\partial L(x_i,y_i;w_{t-1}-\mu m_{t-1})}{\partial w} \\ w_{t} = w_{t-1} - m_{t}$
初始值： $m_0=0$ ， $\mu=0.3$

那么： $w_{0}-\mu m_{0}=1$

算每个数据的梯度：
$\frac{\partial L(x_1,y_1;w_{0}-\mu m_{0})}{\partial w} = -x_1[y_1 - (w_{0}-\mu m_{0}) x_1] = -1 \times (3-1\times1) = -2 \\ \frac{\partial L(x_2,y_2;w_{0}-\mu m_{0})}{\partial w} = -x_1[y_2 - (w_{0}-\mu m_{0}) x_2] = -2 \times (7-1\times2) = -10 \\ \frac{\partial L(x_3,y_3;w_{0}-\mu m_{0})}{\partial w} = -x_1[y_3 - (w_{0}-\mu m_{0}) x_3] = -3 \times (8-1\times3) = -15 \\ \frac{\partial L(x_4,y_4;w_{0}-\mu m_{0})}{\partial w} = -x_4[y_4 - (w_{0}-\mu m_{0}) x_4] = -4 \times (10-1\times4) = -24 \\ \frac{\partial L(x_5,y_5;w_{0}-\mu m_{0})}{\partial w} = -x_5[y_5 - (w_{0}-\mu m_{0}) x_5] = -5 \times (14-1\times5) = -45 \\$
平均梯度：
$G_0 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{0}-\mu m_{0})}{\partial w} = -\frac{2+10+15+24+45}{5} = -19.2$
第一次迭代：
$m_{1} = \mu m_{0} + \eta G_0 = 0.01 \times (-19.2) = -0.192 \\w_{1} = w_{0} - m_{1} = 1 - (-0.192) = 1.192$
那么： $w_{1}-\mu m_{1}=1.192-0.3\times(-0.192)=1.2496$

算每个数据的梯度：
$\begin{aligned} &\begin{aligned} \frac{\partial L(x_1,y_1;w_{1}-\mu m_{1})}{\partial w} &= -x_1[y_1 - (w_{1}-\mu m_{1}) x_1] \\ &= -1 \times (3 - 1.2496 \times 1) \\ &= -1.7504 \end{aligned} \\ &\begin{aligned} \frac{\partial L(x_2,y_2;w_{1}-\mu m_{1})}{\partial w} &= -x_1[y_2 - (w_{1}-\mu m_{1}) x_2] \\ &= -2 \times (7 - 1.2496 \times 2) \\ &= -9.0016 \end{aligned} \\ &\begin{aligned} \frac{\partial L(x_3,y_3;w_{1}-\mu m_{1})}{\partial w} &= -x_1[y_3 - (w_{1}-\mu m_{1}) x_3] \\ &= -3 \times (8 - 1.2496 \times 3) \\ &= -12.7536 \end{aligned} \\ &\begin{aligned} \frac{\partial L(x_4,y_4;w_{1}-\mu m_{1})}{\partial w} &= -x_4[y_4 - (w_{1}-\mu m_{1}) x_4] \\ &= -4 \times (10 - 1.2496 \times 4) \\ &= -20.0064 \end{aligned} \\ &\begin{aligned} \frac{\partial L(x_5,y_5;w_{1}-\mu m_{1})}{\partial w} &= -x_5[y_5 - (w_{1}-\mu m_{1}) x_5] \\ &= -5 \times (14 - 1.2496 \times 5) \\ &= -38.76 \end{aligned} \end{aligned}$

计算平均梯度：

$G_1 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{1}-\mu m_{1})}{\partial w} = -\frac{1.7504+9.0016+12.7536+20.0064+38.76}{5} = -16.4544$

第二次迭代：

$m_{2} = \mu m_{1} + \eta G_1 = 0.3 \times (-0.192) + 0.01 \times (-16.4544) = -0.222144 \\w_{2} = w_{1} - m_{2} = 1.192 - (-0.222144) = 1.414144$

学习率改进策略

AdaGrad

与Momentum不同，AdaGrad没有改变梯度，着眼于学习率的递减，递减系数是之前所有更新的梯度平⽅和的累加。

更新公式：
$g_t = g_{t-1} + G_{t-1}^2 = g_{t-1} + [\frac1M \sum_{i=1}^M \frac{\partial L(x_i,y_i;w_{t-1})}{\partial w}]^2 \\ w_{t} = w_{t-1} - \frac{\eta}{\sqrt{g_t+\varepsilon}} G_{t-1}$
$\varepsilon$ 是非常小的数（如 $10^{-8}$ ），它是为了防止除零。

初始值： $g_0=0$ ， $\eta = 0.1$

平均梯度（讲Gradient Descent的时候已经算好）：
$G_0 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{0})}{\partial w} = -\frac{2+10+15+24+45}{5} = -19.2$
第一次迭代：
$g_1 = g_{0} + G_0^2 = 0 + (-19.2)^2 = 368.64 \\ w_{1} = w_{0} - \frac{\eta}{\sqrt{g_1+\varepsilon}} G_0 = 1 - \frac{0.1}{\sqrt{368.64}} \times (-19.2) = 1.1$
算每个数据的梯度：
$\frac{\partial L(x_1,y_1;w_{1})}{\partial w} = -x_1(y_1-w_1 x_1) = -1 \times (3-1.1\times1) = -1.9 \\ \frac{\partial L(x_2,y_2;w_{1})}{\partial w} = -x_1(y_2-w_1 x_2) = -2 \times (7-1.1\times2) = -9.6 \\ \frac{\partial L(x_3,y_3;w_{1})}{\partial w} = -x_1(y_3-w_1 x_3) = -3 \times (8-1.1\times3) = -14.1 \\ \frac{\partial L(x_4,y_4;w_{1})}{\partial w} = -x_4(y_4-w_1 x_4) = -4 \times (10-1.1\times4) = -22.4 \\ \frac{\partial L(x_5,y_5;w_{1})}{\partial w} = -x_5(y_5-w_1 x_5) = -5 \times (14-1.1\times5) = -42.5 \\$
计算平均梯度：
$G_1 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{1})}{\partial w} = -\frac{1.9+9.6+14.1+22.4+42.5}{5} = -18.1$
第二次迭代：
$g_2 = g_{1} + G_1^2 = 368.64 + (-18.1)^2 = 696.25 \\ w_{2} = w_{1} - \frac{\eta}{\sqrt{g_2+\varepsilon}} G_1 = 1.1 - \frac{0.1}{\sqrt{696.25}} \times (-18.1) \approx 1.17$
AdaGrad的缺点：
衰减系数累积了所有更新步骤中的梯度，到训练后期衰减非常大。

RMSProp

为了克服AdaGrad的缺点，只考察最近几步中的梯度来决定衰减系数。

更新公式：

$\begin{aligned} g_t &= decay\_rate \cdot g_{t-1} + (1 - decay\_rate) \cdot G_{t-1}^2 \\ &= decay\_rate \cdot g_{t-1} + (1 - decay\_rate) \cdot [\frac1M \sum_{i=1}^M \frac{\partial L(x_i,y_i;w_{t-1})}{\partial w}]^2 \end{aligned} \\ w_{t} = w_{t-1} - \frac{\eta}{\sqrt{g_t+\varepsilon}} G_{t-1}$

为什么说RMSProp考察的是最近的几步梯度来决定衰减系数呢？

先来看下面公式：
$\begin{aligned} &g_1 = decay\_rate \cdot g_{0} + (1 - decay\_rate) \cdot G_0^2 \\ &\begin{aligned} g_2 =& decay\_rate \cdot g_{1} + (1 - decay\_rate) \cdot G_1^2 \\ =& decay\_rate^2 \cdot g_{0} + decay\_rate \cdot (1 - decay\_rate) \cdot G_0^2 \\ &+ (1 - decay\_rate) \cdot G_1^2 \\ \end{aligned} \\ &\begin{aligned} g_3 =& decay\_rate \cdot g_{2} + (1 - decay\_rate) \cdot G_2^2 \\ =& decay\_rate^3 \cdot g_{0} + decay\_rate^2 \cdot (1 - decay\_rate) \cdot G_0^2 \\ &+ decay\_rate \cdot (1 - decay\_rate) \cdot G_1^2 + (1 - decay\_rate) \cdot G_2^2 \end{aligned} \\ &\quad \quad \quad \quad \quad \quad \quad \quad \quad \vdots \\ &\begin{aligned} g_t =& decay\_rate \cdot g_{t-1} + (1 - decay\_rate) \cdot G_{t-1}^2 \\ =& decay\_rate^{t} \cdot g_{0} + decay\_rate^{t-1} \cdot (1 - decay\_rate) \cdot G_{0}^2 \\ &+ decay\_rate^{t-2} \cdot (1 - decay\_rate) \cdot G_{1}^2 + \cdots \\ &+ decay\_rate \cdot (1 - decay\_rate) \cdot G_{t-2}^2 + (1 - decay\_rate) \cdot G_{t-1}^2 \end{aligned} \end{aligned}$
经过t步迭代后，越早计算的梯度平方（ $G_{0}^2,G_{1}^2$ 等），它们的系数就越小。

例：假定 $decay\_rate=0.9,t=100$ ，那么 $G_{0}^2$ 的系数
$decay\_rate^{t-1} \cdot (1 - decay\_rate) = 0.9^{99} \times 0.1 \approx 2.95 \times 10^{-6}$
可以忽略不计。

初始值： $g_0=0$ ， $\eta = 0.1$ ， $decay\_rate = 0.9$

平均梯度（讲Gradient Descent的时候已经算好）：
$G_0 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{0})}{\partial w} = -\frac{2+10+15+24+45}{5} = -19.2$
第一次迭代：
$g_1 = decay\_rate \cdot g_{0} + (1 - decay\_rate) \cdot G_{0}^2 = 0.9 \times 0 + 0.1 \times (-19.2)^2 = 36.864 \\ w_{1} = w_{0} - \frac{\eta}{\sqrt{g_1+\varepsilon}} G_{0} = 1 - \frac{0.1}{\sqrt{36.864}} \times (-19.2) \approx 1.3162$
算每个数据的梯度：
$\frac{\partial L(x_1,y_1;w_{1})}{\partial w} = -x_1(y_1-w_1 x_1) = -1 \times (3-1.3162\times1) = -1.6838 \\ \frac{\partial L(x_2,y_2;w_{1})}{\partial w} = -x_1(y_2-w_1 x_2) = -2 \times (7-1.3162\times2) = -8.7352 \\ \frac{\partial L(x_3,y_3;w_{1})}{\partial w} = -x_1(y_3-w_1 x_3) = -3 \times (8-1.3162\times3) = -12.1542 \\ \frac{\partial L(x_4,y_4;w_{1})}{\partial w} = -x_4(y_4-w_1 x_4) = -4 \times (10-1.3162\times4) = -18.9408 \\ \frac{\partial L(x_5,y_5;w_{1})}{\partial w} = -x_5(y_5-w_1 x_5) = -5 \times (14-1.3162\times5) = -37.095 \\$
计算平均梯度：
$G_1 = \frac15 \sum_{i=1}^5 \frac{\partial L(x_i,y_i;w_{1})}{\partial w} = -\frac{1.6838+8.7352+12.1542+18.9408+37.095}{5} = -15.7218$
第二次迭代：
$g_2 = decay\_rate \cdot g_{1} + (1-decay\_rate) \cdot G_1^2 = 0.9 \times 36.864 + 0.1 \times (-15.7218)^2 \approx 57.8951 \\ w_{2} = w_{1} - \frac{\eta}{\sqrt{g_2+\varepsilon}} G_1 = 1.3162 - \frac{0.1}{\sqrt{57.8951}} \times (-18.1) \approx 1.5541$

集大成者

Adam

综合了Momentum的梯度更新策略和RMSProp的学习率衰减策略。

更新公式：
$m_t = \beta_1 m_{t-1} + (1-\beta_1) G_{t} = \beta_1 m_{t-1} + (1-\beta_1) (\frac1M \sum_{i=1}^M \frac{\partial L(x_i,y_i;w_{t-1})}{\partial w}) \\ g_t = \beta_2 g_{t-1} + (1-\beta_2) G_{t}^2 = \beta_2 g_{t-1} + (1-\beta_2)[\frac1M \sum_{i=1}^M \frac{\partial L(x_i,y_i;w_{t-1})}{\partial w}]^2 \\ \hat{m}_t = \frac{m_t}{1-\beta_1^t}, \hat{s}_t = \frac{s_t}{1-\beta_2^t} \\ w_t = w_{t-1} - \frac{\eta}{\sqrt{\hat{s}_t + \varepsilon}} \cdot \hat{m}_t$
$\beta_1,\beta_2$ 通常分别取0.9和0.999。

$m_t,s_t$ 之所以要除以 $1-\beta_1^t,1-\beta_2^t$ （t是次方）的原因是：

希望更新初期的梯度和学习率衰减的变化可以比较剧烈，这样有利于增大初期下降路径的随机性，从而可能发现之前的优化器不会找到的最佳路径。

Jeff-Chow000

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
简单例子说明优化器

简单例子说明优化器我们会用下面的例子来说明优化器：数据集：(x1,y1)=(1,3)(x2,y2)=(2,7)(x3,y3)=(3,8)(x4,y4)=(4,10)(x5,y5)=(5,14)(x_1,y_1) = (1,3) \\(x_2,y_2) = (2,7) \\(x_3,y_3) = (3,8) \\(x_4,y_4) = (4,10) \\(x_5,y_5) = (5,14)(x1,y1)=(1,3)(x2,y2)=(2,7)(x3,y3)=(3,8)(x4,y4
复制链接

扫一扫