Krhenbhl, Philipp, et al. “Data-dependent initializations of convolutional neural networks.” arXiv preprint arXiv:1511.06856 (2015). [Citations: 10].
1 Attenuating Activations
[Idea] Initialize W ’s such that activations have unit variance.
[Algorithm] See Alg. 1
[Analysis] This algorithm works for nonlinearities (such as ReLU and pooling).
• The different channels will undergo the same transformation.
• Then the output channels will follow the same distribution if the input channels do.
• The changed variance will be fixed by the next layer.
2 Global Scaling
[Motivation] The output follow the same distribution per layer. But what about the global