神经网络的优化（1）----指数衰减学习率

最新推荐文章于 2024-06-24 11:38:01 发布

/home/liupc

最新推荐文章于 2024-06-24 11:38:01 发布

阅读量5.4k

点赞数 1

分类专栏： 11 Python/DL/ML

本文链接：https://blog.csdn.net/pengchengliu/article/details/88254234

版权

一、学习率的影响点击此处返回总目录

二、指数衰减学习率

一、学习率的影响

这一节说一下学习率，学习率决定了参数每次更新的幅度。在训练中，参数的更新是遵照下面这个公式的。

下一时刻参数等于当前时刻参数减去学习率乘以损失函数的导数。也就是，参数的更新向着损失函数梯度下降的方向。

我们来举个例子，看一下学习率是怎么影响参数更新的。

损失函数为loss,梯度为2w+2。

参数初始化赋值为5，学习率设为0.2。

第一次更新时，参数为5，按照上面的更新公式，得到2.6；第二次更新后为1.16；第三次更新后为0.296...

我们画出损失函数loss的图像，为右图。优化参数w,就是要找到某个参数w，是的损失函数的梯度最小。从图上我们可以直观的看到，损失函数梯度最小的点在-1这个位置。我们优化w的目的就是为了找到w=-1的这个点。

那么程序是否可以找到这个点呢？我们看一下代码。

#coding:utf-8
import tensorflow as tf

W=tf.Variable(tf.constant(5,dtype=tf.float32)) #设置W的初值为5

loss=tf.square(W+1) #设定损失函数loss
train_step =tf.train.GradientDescentOptimizer(0.2).minimize(loss) #梯度下降，学习率为0.2

with tf.Session() as sess:
init_op = tf.global_variables_initializer()
sess.run(init_op)
for i in range(40):
sess.run(train_step)
w_val = sess.run(W)
loss_val = sess.run(loss)
print("After %d steps,W is %f ,loss is %f" % (i,w_val,loss_val))

运行结果：

After 0 steps,W is 2.600000 ,loss is 12.959999
After 1 steps,W is 1.160000 ,loss is 4.665599
After 2 steps,W is 0.296000 ,loss is 1.679616
After 3 steps,W is -0.222400 ,loss is 0.604662
After 4 steps,W is -0.533440 ,loss is 0.217678
After 5 steps,W is -0.720064 ,loss is 0.078364
After 6 steps,W is -0.832038 ,loss is 0.028211
After 7 steps,W is -0.899223 ,loss is 0.010156
After 8 steps,W is -0.939534 ,loss is 0.003656
After 9 steps,W is -0.963720 ,loss is 0.001316
After 10 steps,W is -0.978232 ,loss is 0.000474
After 11 steps,W is -0.986939 ,loss is 0.000171
After 12 steps,W is -0.992164 ,loss is 0.000061
After 13 steps,W is -0.995298 ,loss is 0.000022
After 14 steps,W is -0.997179 ,loss is 0.000008
After 15 steps,W is -0.998307 ,loss is 0.000003
After 16 steps,W is -0.998984 ,loss is 0.000001
After 17 steps,W is -0.999391 ,loss is 0.000000
After 18 steps,W is -0.999634 ,loss is 0.000000
After 19 steps,W is -0.999781 ,loss is 0.000000
After 20 steps,W is -0.999868 ,loss is 0.000000
After 21 steps,W is -0.999921 ,loss is 0.000000