【DL】神奇的BN: Batch Nomalization

最新推荐文章于 2024-03-07 14:05:51 发布

北境の守卫

最新推荐文章于 2024-03-07 14:05:51 发布

阅读量459

点赞数

分类专栏： DL 文章标签： DL batch normalization

本文链接：https://blog.csdn.net/baishuo8/article/details/88715996

版权

Batch Normalization（BN）通过规范化层输入解决内部协变量转移问题，提高训练速度，允许使用更高学习率。它涉及计算每批数据的均值和标准差，调整激活函数避免线性区域，通常配合学习到的尺度参数。BN在许多实验中表现出色，但在某些情况下可能导致性能下降，可能因任务简单或限制梯度范围。在深度学习实践中，BN的效果依赖于具体问题。

摘要由CSDN通过智能技术生成

Backto DL Index

BN 论文源自这里.

Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs.
Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout.

目的就是为了加速训练，防止训练跑偏。防止形如 $0.9^{30} \approx 0.04$