本文探讨了深度学习中权重初始化的重要性，指出全零初始化的弊端，并介绍了Xavier初始化、ReLU调整的初始化、正交矩阵初始化、数据驱动的初始化以及批量归一化等方法。这些方法旨在保持层间神经元的方差，解决使用ReLU等非线性时的梯度问题，加速深度网络训练。
Training a CNN is hard because
• Large number of parameters requires heavy computation.
• The learning objective is non-convex, and has many poor local minima.
• Deep network has vanishing/exploding gradients problem.
• Need large amount of training data.
As the matter of to handle vanishing/exploding gradients, methods mainly include
• Careful set the learning rate.
• Design better CNN architecture, activation functions, etc.
• Careful initialization of weights.
• Tuning the data distribution.
• Large number of parameters requires heavy computation.
• The learning objective is non-convex, and has many poor local minima.
• Deep network has vanishing/exploding gradients problem.
• Need large amount of training data.
As the matter of to handle vanishing/exploding gradients, methods mainly include
• Careful set the learning rate.
• Design better CNN architecture, activation functions, etc.
• Careful initialization of weights.
• Tuning the data distribution.
We will focus on the last two topics to handle vanishing/exploding gradients problem in this section.
Weight initialization is very important in deep learning. I think one of the reasons that early networks did not work as well is that people did not care about it too much.
Initializing all the weights to 0 is a bad idea since all the neurons learn the same
最低0.47元/天 解锁文章
254
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
