-
学习收获
-
实际训练过程中,神经网络参数很多,很难陷入局部最小,多数情况都是鞍点。
-
train和test的loss曲线存在一个mismatch。
-
Batch大小的文章:
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (https://arxiv.org/abs/1904.00962)
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes (https://arxiv.org/abs/1711.04325)
- Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes Well (https://arxiv.org/abs/2001.02312)
- Large Batch Training of Convolutional Networks (https://arxiv.org/abs/1708.03888)
- Accurate, large minibatch sgd: Training imagenet in 1 hour (https://arxiv.org/abs/1706.02677)
- warm up的文章:
RAdam:https://arxiv.org/abs/1908.03265
-
存在的问题
- 为什么small batch更可能达到flat minima?
- 如何调节优化器的超参数?
... ...