Exponentially decayed learning rate
自定义函数如下,相比于tf.train.exponential_decay,添加了lowbound学习率
α = max ( α 0 β ⌊ t Δ t I ⌋ , α e ) \alpha = \max(\alpha_0 \beta^{\left \lfloor \frac{t}{{\Delta t}^I}\right \rfloor},\alpha_e) α=max(α0β⌊ΔtIt⌋,αe)
- 学习率 α \alpha α,
- 初始学习率 α 0 \alpha_0 α0,
- 衰减率 β \beta β,
- 学习步数 t t t,
- 衰减步长 Δ t \Delta t Δt,
- 阶梯下降指示量 I I I, I = 1 I=1 I=1表示学习率阶梯下降, I = 0 I=0 I=0无阶梯下降
- 最小学习率 α e \alpha_e αe
仿真:
def exponentially_decay_lr(lr_ini,lr_lb,decay_factor,learning_steps, decay_steps,staircase=1):
import math
if staircase:
med_steps = decay_steps
else:
med_steps = 1
lr_decayed = lr_ini*decay_factor**(math.floor(learning_steps/med_steps))
lr = max(lr_decayed,lr_lb)
return lr
lr_ini = 1
lr_lb = 0.1
decay_factor = 0.97
decay_steps = 10
num_learning_steps = 1000
Lr_all=[]
for step in range(num_learning_steps):
lr = exponentially_decay_lr(lr_ini,lr_lb,decay_factor=decay_factor,learning_steps=step, decay_steps=decay_steps,staircase=1)
Lr_all.append(lr)
plt.figure(dpi=100)
plt.plot(Lr_all,label=r'$\psi(x)$')
plt.legend(loc='center right')
# plt.xticks(x)
plt.xlabel('steps')
plt.ylabel('lr value')
plt.grid(True)
plt.show()