一、学习率的影响 点击此处返回总目录 二、指数衰减学习率 一、学习率的影响 这一节说一下学习率,学习率决定了参数每次更新的幅度。在训练中,参数的更新是遵照下面这个公式的。 ![](https://i-blog.csdnimg.cn/blog_migrate/f18e9e3ca337b6c681e4767eca86c521.png) 下一时刻参数等于当前时刻参数减去学习率乘以损失函数的导数。也就是,参数的更新向着损失函数梯度下降的方向。 我们来举个例子,看一下学习率是怎么影响参数更新的。 ![](https://i-blog.csdnimg.cn/blog_migrate/e885cc97729d5d386588b519006750b3.png) 损失函数为loss,梯度为2w+2。 参数初始化赋值为5,学习率设为0.2。 第一次更新时,参数为5,按照上面的更新公式,得到2.6;第二次更新后为1.16;第三次更新后为0.296... 我们画出损失函数loss的图像,为右图。优化参数w,就是要找到某个参数w,是的损失函数的梯度最小。从图上我们可以直观的看到,损失函数梯度最小的点在-1这个位置。我们优化w的目的就是为了找到w=-1的这个点。 那么程序是否可以找到这个点呢?我们看一下代码。
#coding:utf-8 import tensorflow as tf W=tf.Variable(tf.constant(5,dtype=tf.float32)) #设置W的初值为5 loss=tf.square(W+1) #设定损失函数loss train_step =tf.train.GradientDescentOptimizer(0.2).minimize(loss) #梯度下降,学习率为0.2 with tf.Session() as sess: init_op = tf.global_variables_initializer() sess.run(init_op) for i in range(40): sess.run(train_step) w_val = sess.run(W) loss_val = sess.run(loss) print("After %d steps,W is %f ,loss is %f" % (i,w_val,loss_val)) |
运行结果: After 0 steps,W is 2.600000 ,loss is 12.959999 After 1 steps,W is 1.160000 ,loss is 4.665599 After 2 steps,W is 0.296000 ,loss is 1.679616 After 3 steps,W is -0.222400 ,loss is 0.604662 After 4 steps,W is -0.533440 ,loss is 0.217678 After 5 steps,W is -0.720064 ,loss is 0.078364 After 6 steps,W is -0.832038 ,loss is 0.028211 After 7 steps,W is -0.899223 ,loss is 0.010156 After 8 steps,W is -0.939534 ,loss is 0.003656 After 9 steps,W is -0.963720 ,loss is 0.001316 After 10 steps,W is -0.978232 ,loss is 0.000474 After 11 steps,W is -0.986939 ,loss is 0.000171 After 12 steps,W is -0.992164 ,loss is 0.000061 After 13 steps,W is -0.995298 ,loss is 0.000022 After 14 steps,W is -0.997179 ,loss is 0.000008 After 15 steps,W is -0.998307 ,loss is 0.000003 After 16 steps,W is -0.998984 ,loss is 0.000001 After 17 steps,W is -0.999391 ,loss is 0.000000 After 18 steps,W is -0.999634 ,loss is 0.000000 After 19 steps,W is -0.999781 ,loss is 0.000000 After 20 steps,W is -0.999868 ,loss is 0.000000 After 21 steps,W is -0.999921 ,loss is 0.000000 |