进入第三周学习后,开始使用完整案例进行训练测试。
与自学的最大区别是有小伙伴一起学习会效率更好。
先看结果
本次作业是使用衰减学习率,通过多次调参数。
查找 API得知有提供多个api可选择点击查看
学习率调度器¶
当我们使用诸如梯度下降法等方式来训练模型时,一般会兼顾训练速度和损失(loss)来选择相对合适的学习率。但若在训练过程中一直使用一个学习率,训练集的损失下降到一定程度后便不再继续下降,而是在一定范围内震荡。其震荡原理如下图所示,即当损失函数收敛到局部极小值附近时,会由于学习率过大导致更新步幅过大,每步参数更新会反复越过极小值而出现震荡。
这个做法是为了更快下降到稳定值,减小震荡
本次选择了使用piecewise_decay这个api进行测试。
代码节选如下:
def trainadj(model):
countindex=0
boundaries = [40, 80]
values = [0.5, 0.1,0.05]
with fluid.dygraph.guard():
print('start training ... ')
model.train()
epoch_num = 2
# 定义优化器
opt = fluid.optimizer.Momentum(
learning_rate=fluid.layers.piecewise_decay(boundaries=boundaries, values=values),
momentum=0.9,
parameter_list=model.parameters(),
regularization=fluid.regularizer.L2Decay(1e-4)
)
# 定义数据读取器,训练数据读取器和验证数据读取器
train_loader = data_loader(DATADIR, batch_size=10, mode='train')
valid_loader = valid_data_loader(DATADIR2, CSVFILE)
for epoch in range(epoch_num):
for batch_id, data in enumerate(train_loader()):
x_data, y_data = data
img = fluid.dygraph.to_variable(x_data)
label = fluid.dygraph.to_variable(y_data)
# 运行模型前向计算,得到预测值
logits = model(img)
# 进行loss计算
loss = fluid.layers.sigmoid_cross_entropy_with_logits(logits, label)
avg_loss = fluid.layers.mean(loss)
#if batch_id % 10 == 0:
print("{} #:epoch: {}, batch_id: {}, loss is: {}".format(opt.current_step_lr(),epoch, batch_id, avg_loss.numpy()))
#lossindex.append(countindex)
lossdataadj.append(avg_loss.numpy())
countindex=countindex+1
# 反向传播,更新权重,清除梯度
avg_loss.backward()
opt.minimize(avg_loss)
model.clear_gradients()
model.eval()
accuracies = []
losses = []
for batch_id, data in enumerate(valid_loader()):
x_data, y_data = data
img = fluid.dygraph.to_variable(x_data)
label = fluid.dygraph.to_variable(y_data)
# 运行模型前向计算,得到预测值
logits = model(img)
# 二分类,sigmoid计算后的结果以0.5为阈值分两个类别
# 计算sigmoid后的预测概率,进行loss计算
pred = fluid.layers.sigmoid(logits)
loss = fluid.layers.sigmoid_cross_entropy_with_logits(logits, label)
# 计算预测概率小于0.5的类别
pred2 = pred * (-1.0) + 1.0
# 得到两个类别的预测概率,并沿第一个维度级联
pred = fluid.layers.concat([pred2, pred], axis=1)
acc = fluid.layers.accuracy(pred, fluid.layers.cast(label, dtype='int64'))
accuracies.append(acc.numpy())
losses.append(loss.numpy())
print("[validation] accuracy/loss: {}/{}".format(np.mean(accuracies), np.mean(losses)))
model.train()
# save params of model
fluid.save_dygraph(model.state_dict(), 'palm')
# save optimizer state
fluid.save_dygraph(opt.state_dict(), 'palm')
关键修改部分:
原代码优化器部分:
# 定义优化器
opt = fluid.optimizer.Momentum(learning_rate=0.001, momentum=0.9, parameter_list=model.parameters())
修改为
#参数部分
boundaries = [40, 80]
values = [0.5, 0.1,0.05]
# 定义优化器
opt = fluid.optimizer.Momentum(
learning_rate=fluid.layers.piecewise_decay(boundaries=boundaries, values=values),
momentum=0.9,
parameter_list=model.parameters(),
regularization=fluid.regularizer.L2Decay(1e-4)
)
运行后可观察到loss开始时变动很大,后期就平稳下降到,小范围调整
0.5 #:epoch: 0, batch_id: 0, loss is: [0.7019217] 0.5 #:epoch: 0, batch_id: 1, loss is: [12.024451] 0.5 #:epoch: 0, batch_id: 2, loss is: [157.16641] 0.5 #:epoch: 0, batch_id: 3, loss is: [95.9813] 0.5 #:epoch: 0, batch_id: 4, loss is: [4.2797537] 0.5 #:epoch: 0, batch_id: 5, loss is: [7.3096175] 0.5 #:epoch: 0, batch_id: 6, loss is: [4.384579] 0.5 #:epoch: 0, batch_id: 7, loss is: [0.4799278] 0.5 #:epoch: 0, batch_id: 8, loss is: [7.04195] 0.5 #:epoch: 0, batch_id: 9, loss is: [2.5763474] 0.5 #:epoch: 0, batch_id: 10, loss is: [0.71913904] 0.5 #:epoch: 0, batch_id: 11, loss is: [0.6301126] 0.5 #:epoch: 0, batch_id: 12, loss is: [13.080458] 0.5 #:epoch: 0, batch_id: 13, loss is: [0.5763507] 0.5 #:epoch: 0, batch_id: 14, loss is: [0.4996148] 0.5 #:epoch: 0, batch_id: 15, loss is: [0.26528448] 0.5 #:epoch: 0, batch_id: 16, loss is: [19.885382] 0.5 #:epoch: 0, batch_id: 17, loss is: [1.20226] 0.5 #:epoch: 0, batch_id: 18, loss is: [8.281421] 0.5 #:epoch: 0, batch_id: 19, loss is: [10.232064] 0.5 #:epoch: 0, batch_id: 20, loss is: [1.1040335] 0.5 #:epoch: 0, batch_id: 21, loss is: [13.693789] 0.5 #:epoch: 0, batch_id: 22, loss is: [0.7570822] 0.5 #:epoch: 0, batch_id: 23, loss is: [22.65204] 0.5 #:epoch: 0, batch_id: 24, loss is: [0.73202336] 0.5 #:epoch: 0, batch_id: 25, loss is: [0.5780096] 0.5 #:epoch: 0, batch_id: 26, loss is: [11.400303] 0.5 #:epoch: 0, batch_id: 27, loss is: [9.84152] 0.5 #:epoch: 0, batch_id: 28, loss is: [0.60879284] 0.5 #:epoch: 0, batch_id: 29, loss is: [1.6898936] 0.5 #:epoch: 0, batch_id: 30, loss is: [2.4156432] 0.5 #:epoch: 0, batch_id: 31, loss is: [1.0774876] 0.5 #:epoch: 0, batch_id: 32, loss is: [0.34068987] 0.5 #:epoch: 0, batch_id: 33, loss is: [1.7986888] 0.5 #:epoch: 0, batch_id: 34, loss is: [0.7363323] 0.5 #:epoch: 0, batch_id: 35, loss is: [0.6592141] 0.5 #:epoch: 0, batch_id: 36, loss is: [0.42114687] 0.5 #:epoch: 0, batch_id: 37, loss is: [0.74656934] 0.5 #:epoch: 0, batch_id: 38, loss is: [0.7371367] 0.5 #:epoch: 0, batch_id: 39, loss is: [2.3921227]
后期可对
boundaries = [40, 80]
values = [0.5, 0.1,0.05]
这两部分变量进行修改高度,再观察效果即可。
当然,以上结果并不是最好的情况,上述只是记录本次调试参数过程。
最后,百度paddlepaddle学习课程对入门了解与使用带来了很大帮助,值得推荐!