深度学习之TensorFlow（五）学习率

最新推荐文章于 2020-10-29 09:41:02 发布

Siri_only

最新推荐文章于 2020-10-29 09:41:02 发布

阅读量260

点赞数

文章标签： tensorflow 深度学习神经网络 deep learning 人工智能

本文链接：https://blog.csdn.net/qq_45416295/article/details/104581695

版权

文章目录

- 学习率

学习率

学习率（learning_rate）：每次参数更新的幅度
$w_{n+1}=w_n-learnin\_grate*\nabla$
$w_{n_+1}$ 是更新后的参数， $w_n$ 是当前参数， $\nabla$ 是损失函数的梯度(导数)
例如损失函数 $loss=(w+1)^2$ ,梯度 $\nabla=\frac{\partial loss}{\partial w}=2w+2$ ，然后 $w_{新值}=w_{旧值}-learnin\_grate*\nabla$ ，当 $2 w + 2 = 0$ 时，w=-1,是最理想值。
测试代码：

#coding:utf-8
import tensorflow as tf

# 定义得优化参数
w = tf.Variable(tf.constant(5,dtype=tf.float32))
# 定义损失函数
loss = tf.square(w+1)
# 定义反向传播方法
train_step = tf.train.GradientDescentOptimizer(0.2).minimize(loss)
# 生成会话
with tf.Session() as sess:
    init_op = tf.global_variables_initializer()
    sess.run(init_op)
    for i in range(40):
        sess.run(train_step)
        w_val = sess.run(w)
        loss_val = sess.run(loss)
        print("After %2s steps :w is %f, loss is %f." % (i,w_val,loss_val))

运行结果：

C:\ProgramData\Anaconda3\envs\tensorflow\python.exe C:/Users/Administrator/PycharmProjects/tensorflow/TensorFlow_notebook/tf_004.py
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2020-02-29 21:16:13.683208: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
After  0 steps :w is 2.600000, loss is 12.959999.
After  1 steps :w is 1.160000, loss is 4.665599.
After  2 steps :w is 0.296000, loss is 1.679616.
After  3 steps :w is -0.222400, loss is 0.604662.
After  4 steps :w is -0.533440, loss is 0.217678.
After  5 steps :w is -0.720064, loss is 0.078364.
After  6 steps :w is -0.832038, loss is 0.028211.
After  7 steps :w is -0.899223, loss is 0.010156.
After  8 steps :w is -0.939534, loss is 0.003656.
After  9 steps :w is -0.963720, loss is 0.001316.
After 10 steps :w is -0.978232, loss is 0.000474.
After 11 steps :w is -0.986939, loss is 0.000171.
After 12 steps :w is -0.992164, loss is 0.000061.
After 13 steps :w is -0.995298, loss is 0.000022.
After 14 steps :w is -0.997179, loss is 0.000008.
After 15 steps :w is -0.998307, loss is 0.000003.
After 16 steps :w is -0.998984, loss is 0.000001.
After 17 steps :w is -0.999391, loss is 0.000000.
After 18 steps :w is -0.999634, loss is 0.000000.
After 19 steps :w is -0.999781, loss is 0.000000.
After 20 steps :w is -0.999868, loss is 0.000000.
After 21 steps :w is -0.999921, loss is 0.000000.
After 22 steps :w is -0.999953, loss is 0.000000.
After 23 steps :w is -0.999972, loss is 0.000000.
After 24 steps :w is -0.999983, loss is 0.000000.
After 25 steps :w is -0.999990, loss is 0.000000.
After 26 steps :w is -0.999994, loss is 0.000000.
After 27 steps :w is -0.999996, loss is 0.000000.
After 28 steps :w is -0.999998, loss is 0.000000.
After 29 steps :w is -0.999999, loss is 0.000000.
After 30 steps :w is -0.999999, loss is 0.000000.
After 31 steps :w is -1.000000, loss is 0.000000.
After 32 steps :w is -1.000000, loss is 0.000000.
After 33 steps :w is -1.000000, loss is 0.000000.
After 34 steps :w is -1.000000, loss is 0.000000.
After 35 steps :w is -1.000000, loss is 0.000000.
After 36 steps :w is -1.000000, loss is 0.000000.
After 37 steps :w is -1.000000, loss is 0.000000.
After 38 steps :w is -1.000000, loss is 0.000000.
After 39 steps :w is -1.000000, loss is 0.000000.

Process finished with exit code 0

学习率设置的度：当学习率过大振荡不收敛，学习率过小收敛速度慢。
指数衰减学习率：learning_rate=LEARNING_RATE_BASE*LEARNING_RATE_DECAY
learning_rate：学习率基数；LEARNING_RATE_BASE：学习率初始值；LEARNING_RATE_DECAY：学习率衰减率（0,1）。
BATCH_SIZE:学习的轮数。
LEARNING_RATE_STEP(多少轮更新一次学习率)=总样本数/batch_size。

# 计数器，标记为不可训练
global_step = tf.Variable(0,trainable=False)
learning_rate = tf.train.exponential_decay(LEARNING_RATE_SIZE,gloabal_step,LEARNING_RATE_STEP,LEARNING_RATE_RECAY,staircase=True)

给出测试代码：

# coding:utf-8
import tensorflow as tf
"""指数衰减学习率"""
# 损失函数 loss=（w+1）^2，w初值是常数10，反向传播求最优w,求最小loss对应的w值

# 最初学习率
LEARNING_RATE_BASE = 0.1
# 衰减学习率
LEARNING_RATE_DECAY = 0.99
# 多少轮更新一次学习率
LEARNING_RATE_STEP = 1

# 计数器，记录运行BATCH_SIZE轮数
global_step = tf.Variable(0,trainable=False)
# 定义指数下降学习率
learning_rate = tf.train.exponential_decay(LEARNING_RATE_BASE,global_step,LEARNING_RATE_STEP,LEARNING_RATE_DECAY,staircase=True)
# 待优化参数
w = tf.Variable(tf.constant(5,dtype=tf.float32))
# 损失函数
loss = tf.square(w+1)
# 反向传播方法
train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss,global_step=global_step)

with tf.Session() as sess:
    init_op = tf.global_variables_initializer()
    sess.run(init_op)
    for i in range(40):
        sess.run(train_step)
        learning_rate_val=sess.run(learning_rate)
        global_step_val = sess.run(global_step)
        w_val = sess.run(w)
        loss_val = sess.run(loss)
        print("After %2s steps :global_step is %9f,w is %f,learning_rate is %f,loss is %8f." % (i,global_step_val,w_val,learning_rate_val,loss_val))

运行结果：


```C:\ProgramData\Anaconda3\envs\tensorflow\python.exe C:/Users/Administrator/PycharmProjects/tensorflow/TensorFlow_notebook/tf_005.py
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2020-02-29 21:56:40.752849: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
After  0 steps :global_step is  1.000000,w is 3.800000,learning_rate is 0.099000,loss is 23.040001.
After  1 steps :global_step is  2.000000,w is 2.849600,learning_rate is 0.098010,loss is 14.819419.
After  2 steps :global_step is  3.000000,w is 2.095001,learning_rate is 0.097030,loss is 9.579033.
After  3 steps :global_step is  4.000000,w is 1.494386,learning_rate is 0.096060,loss is 6.221961.
After  4 steps :global_step is  5.000000,w is 1.015167,learning_rate is 0.095099,loss is 4.060896.
After  5 steps :global_step is  6.000000,w is 0.631886,learning_rate is 0.094148,loss is 2.663051.
After  6 steps :global_step is  7.000000,w is 0.324608,learning_rate is 0.093207,loss is 1.754587.
After  7 steps :global_step is  8.000000,w is 0.077684,learning_rate is 0.092274,loss is 1.161403.
After  8 steps :global_step is  9.000000,w is -0.121202,learning_rate is 0.091352,loss is 0.772287.
After  9 steps :global_step is 10.000000,w is -0.281761,learning_rate is 0.090438,loss is 0.515867.
After 10 steps :global_step is 11.000000,w is -0.411674,learning_rate is 0.089534,loss is 0.346128.
After 11 steps :global_step is 12.000000,w is -0.517024,learning_rate is 0.088638,loss is 0.233266.
After 12 steps :global_step is 13.000000,w is -0.602644,learning_rate is 0.087752,loss is 0.157891.
After 13 steps :global_step is 14.000000,w is -0.672382,learning_rate is 0.086875,loss is 0.107334.
After 14 steps :global_step is 15.000000,w is -0.729305,learning_rate is 0.086006,loss is 0.073276.
After 15 steps :global_step is 16.000000,w is -0.775868,learning_rate is 0.085146,loss is 0.050235.
After 16 steps :global_step is 17.000000,w is -0.814036,learning_rate is 0.084294,loss is 0.034583.
After 17 steps :global_step is 18.000000,w is -0.845387,learning_rate is 0.083451,loss is 0.023905.
After 18 steps :global_step is 19.000000,w is -0.871193,learning_rate is 0.082617,loss is 0.016591.
After 19 steps :global_step is 20.000000,w is -0.892476,learning_rate is 0.081791,loss is 0.011561.
After 20 steps :global_step is 21.000000,w is -0.910065,learning_rate is 0.080973,loss is 0.008088.
After 21 steps :global_step is 22.000000,w is -0.924629,learning_rate is 0.080163,loss is 0.005681.
After 22 steps :global_step is 23.000000,w is -0.936713,learning_rate is 0.079361,loss is 0.004005.
After 23 steps :global_step is 24.000000,w is -0.946758,learning_rate is 0.078568,loss is 0.002835.
After 24 steps :global_step is 25.000000,w is -0.955125,learning_rate is 0.077782,loss is 0.002014.
After 25 steps :global_step is 26.000000,w is -0.962106,learning_rate is 0.077004,loss is 0.001436.
After 26 steps :global_step is 27.000000,w is -0.967942,learning_rate is 0.076234,loss is 0.001028.
After 27 steps :global_step is 28.000000,w is -0.972830,learning_rate is 0.075472,loss is 0.000738.
After 28 steps :global_step is 29.000000,w is -0.976931,learning_rate is 0.074717,loss is 0.000532.
After 29 steps :global_step is 30.000000,w is -0.980378,learning_rate is 0.073970,loss is 0.000385.
After 30 steps :global_step is 31.000000,w is -0.983281,learning_rate is 0.073230,loss is 0.000280.
After 31 steps :global_step is 32.000000,w is -0.985730,learning_rate is 0.072498,loss is 0.000204.
After 32 steps :global_step is 33.000000,w is -0.987799,learning_rate is 0.071773,loss is 0.000149.
After 33 steps :global_step is 34.000000,w is -0.989550,learning_rate is 0.071055,loss is 0.000109.
After 34 steps :global_step is 35.000000,w is -0.991035,learning_rate is 0.070345,loss is 0.000080.
After 35 steps :global_step is 36.000000,w is -0.992297,learning_rate is 0.069641,loss is 0.000059.
After 36 steps :global_step is 37.000000,w is -0.993369,learning_rate is 0.068945,loss is 0.000044.
After 37 steps :global_step is 38.000000,w is -0.994284,learning_rate is 0.068255,loss is 0.000033.
After 38 steps :global_step is 39.000000,w is -0.995064,learning_rate is 0.067573,loss is 0.000024.
After 39 steps :global_step is 40.000000,w is -0.995731,learning_rate is 0.066897,loss is 0.000018.

Process finished with exit code 0

Siri_only

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
深度学习之TensorFlow（五）学习率

文章目录学习率学习率学习率（learning_rate）：每次参数更新的幅度wn+1=wn−learnin_grate∗∇w_{n+1}=w_n-learnin\_grate*\nablawn+1=wn−learnin_grate∗∇wn+1w_{n_+1}wn+1是更新后的参数，wnw_nwn是当前参数，∇\nabla∇是损失函数的梯度(导数)例如损失函数loss=(w+1...
复制链接

扫一扫