【深度学习随笔】神经网络中去掉残差连接的工作

刚刚有个同学问我：“深层神经网络如果去掉一部分残差，到底还能不能正常训练呀？”这个问题着实很好，我也没思考过，也没尝试过，然后试着去Google Scholar检索了一下关键词“without shorcut”，于是看到了以下的文章。让我比较惊奇的是，这是个很多人研究的方向，并且近年来不少文章发表在AI三大会。其中包含：1）残差的融合提高效率（重参数化）；2）去除一部分残差提高效率；3）用更好的架构替代传统残差等多个方面，还挺值得总结一下的。

• [CVPR 2023] Localized Shortcut Removal [paper]
• [ICLR 2023] Deep transformers without shortcuts: Modifying self-attention for faithful signal propagation [paper]
• [ICLR 2022] Deep learning without shortcuts: Shaping the kernel with tailored rectifiers [paper]
• [NeurIPS 2022] Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers [paper]
• [ICML 2022] Monitoring Shortcut Learning using Mutual Information [paper]
• [CVPR 2021] RepVGG: Making VGG-style ConvNets Great Again [paper]
  • 注：Xiaohan Ding的结构重参数化系列都有残差分支和卷积分支的融合
• [ICML 2020] Automatic Shortcut Removal for Self-Supervised Representation Learning [paper]
• [NeurIPS 2020] Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts [paper]
• [arXiv 2019] RMNet: Equivalently Removing Residual Connection from Networks [paper]
• [arXiv 2017] DiracNets: Training Very Deep Neural Networks Without Skip-Connections [paper]
  • 注：非常早期的一篇经典论文，将残差融合进权重