tensorflow(yunyang) 》》learn_rate

tensorflow(yunyang) 》》 learn_rate

原代码:

with tf.name_scope('learn_rate'):
    self.global_step = tf.Variable(1.0, dtype=tf.float64, trainable=False, name='global_step')
    # self.warmup_periods = 2
    # self.steps_per_period = len(self.trainset)
    warmup_steps = tf.constant(self.warmup_periods * self.steps_per_period,
                                        dtype=tf.float64, name='warmup_steps')
    # 20,30                            
    train_steps = tf.constant( (self.first_stage_epochs + self.second_stage_epochs)* 
                                self.steps_per_period,
                                dtype=tf.float64, name='train_steps')
    # global_step = 1.0                             
    self.learn_rate = tf.cond(
    pred=self.global_step < warmup_steps,
    # learn_rate_init = 1e-4
    true_fn=lambda: self.global_step / warmup_steps * self.learn_rate_init,
    # learn_rate_end = 1e-6
    false_fn=lambda: self.learn_rate_end + 0.5 * (self.learn_rate_init - self.learn_rate_end) *
                                    (1 + tf.cos(
                                        (self.global_step - warmup_steps) / (train_steps -                                                     warmup_steps)
                                         * np.pi)))

显示代码：

import numpy as np
from matplotlib import pyplot as plt

def learning(global_step):
    steps_per_period = 1000  # 数据集尺寸
    warmup_periods = 2 # 参数 用于设置控制学习率下降
    first_step = 20 # 两步训练第一步
    second_step = 30 # 两步训练第二步
    learn_rate_init = 1e-4 # 初始学习率
    learn_rate_end = 1e-6 # 最低学习率

    warmup_steps =  warmup_periods * steps_per_period
    train_steps = (first_step + second_step)*steps_per_period
    learn_rate = np.where(
    global_step < warmup_steps,
    global_step / warmup_steps * learn_rate_init,
    learn_rate_end + 0.5 * (learn_rate_init-learn_rate_end)*(1+np.cos(
    (global_step -warmup_steps) / (train_steps - warmup_steps) * np.pi
    ))
    )
    return learn_rate
global_step = 1
learn_rate = []
for i in range(50): # epoch: 50
    for batch in range(1000): # len(dataset): 1000
        learn_rate.append(learning(global_step = global_step))
        global_step += 1

plt.figure(num=1)
plt.plot([i for i in range(1,global_step)],learn_rate)
plt.show()

结果显示:

叙述:学习率在warmup_periods*length下从非常低的学习率开始上升,
1. 低的学习率为了防止nan
2. 升高是为了使得模型梯度步长大点
3. 先稳定模型, 在提高学习率, 再缓慢降低.

附: [CSDN] cycle策略: 实践中的学习率设定应该是先增再降https://blog.csdn.net/qq_38906523/article/details/80590314
[GitHub] https://github.com/sgugger/Deep-Learning/blob/master/Cyclical%20LR%20and%20momentums.ipynb