learning rate四种改变方式

Fixed

learning rate固定不变

base_lr = 0.01
lr_policy = "fixed"

Step

learning rate在每迭代stepsize次后减少gamma倍。$lr = lr \times gamma$

base_lr = 0.01
lr_policy = "step"
gamma = 0.1
stepsize= 10000

Polynomial

learning rate呈多项式曲线下降。$LR(t)=base\_lr\times(\frac{t}{T})^{power}$

base_lr = 0.01
lr_policy = "poly"
power = 0.5

Inv

learning rate随迭代次数增加而下降。$LR(t)=base\_lr\times(1+gamma\times iter)^{power}$

base_lr = 0.01
lr_policy = "Inv"
gamma = 0.0001
power = 0.75