论文记录：Visualizing and Understanding the Effectiveness of BERT

BERT pre-training and fine-tune 与 train from scratch的对比

结论：

• Pre-training Gets a Good Initial Point Across Downstream Tasks

  1. Pre-training Leads to Wider Optima
  2. Pre-training Eases Optimization on Downstream Tasks
  3. Pre-training-then-fine-tuning is Robust to Overfitting

• Pre-training Helps to Generalize Better

  1. Wide and Flat Optima Lead to Better Generalization
  2. Consistency Between Training Loss Surface and Generalization Error Surface

• Lower Layers of BERT are More Invariant and Transferable