本文探讨了深度学习中卷积神经网络（CNN）的数据依赖初始化方法，包括减弱激活、全局缩放等策略。通过特定算法，确保不同通道经过相同变换，输出通道遵循相同分布，从而稳定网络训练。此外，提出希望每一层权重以相似速率变化，以维持全局尺度的平衡。参考了J. Donahue和T. Koller的相关研究。
Krhenbhl, Philipp, et al. “Data-dependent initializations of convolutional neural networks.” arXiv preprint arXiv:1511.06856 (2015). [Citations: 10].
1 Attenuating Activations
[Idea] Initialize W ’s such that activations have unit variance.
[Algorithm] See Alg. 1
[Analysis] This algorithm works for nonlinearities (such as ReLU and pooling).
• The different channels will undergo the same transformation.
• Then the output channels will follow the same distribution if the input channels do.
• The changed variance will be fixed by the next layer.
2 Global Scaling
[Motivation] The output follow the same distribution per layer. But what about the global
最低0.47元/天 解锁文章
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
