论文链接https://arxiv.org/pdf/1908.05620.pdf。
给出了一种很好看的可视化loss landscape的方法。还可以进一步将训练过程在land scape上标识出来。这个是我在MLP上可视化的训练过程(动态的图是额外的操作,和论文方法无关)。
生成land scape和训练过程曲线是两个分开的步骤。
生成2D land scape
直接放伪代码。加扰动的dir_x和dir_y生成是有讲究的,并非直接的随机向量。对于卷积层,要以filter为单位做normalization,让其2-norm和原模型对应filter的2-norm相等。
inputs, labels # a large batch of data used to evaluate loss
model # a trained model
alpha_range # the range of x-axis
beta_range # the range of y-axis
dir_x # the first direction to change original model weights
dir_y # the second direction to change original model weights
loss_land_scape = [0 * len(alpha_range)] * len(beta_range)
ori_weight = flatten(model.weight)
for x, alpha in enumerate(alpha_range):
for y, beta in enumerate(beta_range):
new_weight = ori_weight + alpha * dir_x + beta * dir_y
model.load(new_weight)
loss = evaluate(model(inputs), labels)
loss_land_scape[x][y] = loss
draw(loss_land_scape)
生成训练过程曲线
model_ckpt_list # record the parameters along the training process, of shape (ckpt_number, parameter_number)
pca = PCA(n_components=2)
path_2d = pca.fit_transform(model_ckpt_list) # shape: (ckpt_number, 2)
参考:
https://github.com/logancyang/loss-landscape-animhttps://github.com/logancyang/loss-landscape-anim