Single Shot MultiBox Detector(MXNet)源码阅读笔记(2)
初始化的方法Xavier
We initialize the parameters for all the newly added convolutional layers with the “xavier” method.
论文中所述,我们对新增加的卷积层使用“xavier”的方法进行初始化。
Xavier Initialization很简单,其实就是
Var(Wi)=2nin+nout
nin -输入神经元的数量
nout -输出神经元的数量
则
scale=magnitudeVar(Wi)−−−−−−−−−−√
具体解释见 andy’sblog
原理出处见论文 Understanding the difficulty of training deep feedforward neural networks
MXnet中代码 initializer.py部分
class Xavier(Initializer):
"""Initialize the weight with Xavier or similar initialization scheme.
Parameters
----------
rnd_type: str, optional
Use ```gaussian``` or ```uniform``` to init
factor_type: str, optional
Use ```avg```, ```in```, or ```out``` to init
magnitude: float, optional
scale of random number range
"""
def __init__(self, rnd_type="uniform", factor_type="avg", magnitude=3):
super(Xavier, self).__init__(rnd_type=rnd_type, factor_type=factor_type,magnitude=magnitude)
self.rnd_type = rnd_type
self.factor_type = factor_type
self.magnitude = float(magnitude)
def _init_weight(self, _, arr):
shape = arr.shape
hw_scale = 1.
if len(shape) > 2:
hw_scale = np.prod(shape[2:])
fan_in, fan_out = shape[1] * hw_scale, shape[0] * hw_scale
factor = 1.
if self.factor_type == "avg":
factor = (fan_in + fan_out) / 2.0
# 计算方差
elif self.factor_type == "in":
factor = fan_in
elif self.factor_type == "out":
factor = fan_out
else:
raise ValueError("Incorrect factor type")
scale = np.sqrt(self.magnitude / factor)
# 计算范围
if self.rnd_type == "uniform":
random.uniform(-scale, scale, out=arr)
elif self.rnd_type == "gaussian":
random.normal(0, scale, out=arr)
else:
raise ValueError("Unknown random type")