自定义指数衰减学习率函数

Exponentially decayed learning rate

自定义函数如下，相比于tf.train.exponential_decay，添加了lowbound学习率

$\alpha =\max ({\alpha }_0{\beta }^{\lfloor \frac{t}{{\Delta t}^I}\rfloor },{\alpha }_e)$

• 学习率$\alpha$,
• 初始学习率${\alpha }_0$,
• 衰减率$\beta$,
• 学习步数$t$,
• 衰减步长$\Delta t$,
• 阶梯下降指示量$I$,$I=1$表示学习率阶梯下降，$I=0$无阶梯下降
• 最小学习率${\alpha }_e$

仿真：

def exponentially_decay_lr(lr_ini,lr_lb,decay_factor,learning_steps, decay_steps,staircase=1):
    import math
    if staircase:
        med_steps = decay_steps
    else:
        med_steps = 1    
    lr_decayed = lr_ini*decay_factor**(math.floor(learning_steps/med_steps))
    lr = max(lr_decayed,lr_lb)
    return lr
lr_ini = 1
lr_lb = 0.1
decay_factor = 0.97
decay_steps = 10
num_learning_steps = 1000
Lr_all=[]
for step in range(num_learning_steps):
    lr = exponentially_decay_lr(lr_ini,lr_lb,decay_factor=decay_factor,learning_steps=step, decay_steps=decay_steps,staircase=1)
    Lr_all.append(lr)
    
plt.figure(dpi=100)
plt.plot(Lr_all,label=r'$\psi(x)$')

plt.legend(loc='center right')
# plt.xticks(x)
plt.xlabel('steps')
plt.ylabel('lr value')
plt.grid(True)
plt.show()