Visualizing the Loss Landscape of Neural Nets论文笔记

论文链接https://arxiv.org/pdf/1908.05620.pdf。

给出了一种很好看的可视化loss landscape的方法。还可以进一步将训练过程在land scape上标识出来。这个是我在MLP上可视化的训练过程（动态的图是额外的操作，和论文方法无关）。

 生成land scape和训练过程曲线是两个分开的步骤。

 生成2D land scape

直接放伪代码。加扰动的dir_x和dir_y生成是有讲究的，并非直接的随机向量。对于卷积层，要以filter为单位做normalization，让其2-norm和原模型对应filter的2-norm相等。

inputs, labels  # a large batch of data used to evaluate loss
model  # a trained model
alpha_range  # the range of x-axis
beta_range  # the range of y-axis
dir_x  # the first direction to change original model weights
dir_y  # the second direction to change original model weights

loss_land_scape = [0 * len(alpha_range)] * len(beta_range)
ori_weight = flatten(model.weight)
for x, alpha in enumerate(alpha_range):
    for y, beta in enumerate(beta_range):
        new_weight = ori_weight + alpha * dir_x + beta * dir_y
        model.load(new_weight)
        loss = evaluate(model(inputs), labels)
        loss_land_scape[x][y] = loss

draw(loss_land_scape)

生成训练过程曲线

model_ckpt_list # record the parameters along the training process, of shape (ckpt_number, parameter_number)

pca = PCA(n_components=2)
path_2d = pca.fit_transform(model_ckpt_list) # shape: (ckpt_number, 2)

参考：

[论文阅读]损失函数可视化及其对神经网络的指导作用 - 知乎论文在 Visualizing the Loss Landscape of Neural Nets，在SGD在两层神经网络上是怎么收敛的？看到的，在写[论文阅读]How Does Batch Normalization Help Optimizati的过程中，实在写不下去了，觉得还是先补补这…https://zhuanlan.zhihu.com/p/52314278

https://github.com/logancyang/loss-landscape-animhttps://github.com/logancyang/loss-landscape-anim

https://towardsdatascience.com/from-animation-to-intuition-visualizing-optimization-trajectory-in-neural-nets-726e43a08d85https://towardsdatascience.com/from-animation-to-intuition-visualizing-optimization-trajectory-in-neural-nets-726e43a08d85