简单粗暴PyTorch之权值初始化方法与重要性

一、梯度消失与爆炸

不恰当的权值初始化，可能带来梯度消失与爆炸。

$$\begin{array}{rl}{\mathrm{H}}_2=& {\mathrm{H}}_1\ast {\mathrm{W}}_2\\ \Delta {\mathrm{W}}_2& =\frac{\partial \mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}{\partial {\mathrm{W}}_2}=\frac{\partial \mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}{\partial \mathrm{o}\mathrm{u}\mathrm{t}}\ast \frac{\partial \mathrm{o}\mathrm{u}\mathrm{t}}{\partial {\mathrm{H}}_2}\ast \frac{\partial {\mathrm{H}}_2}{\partial {\mathrm{w}}_2}\\ & =\frac{\partial \mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}{\partial \mathrm{o}\mathrm{u}\mathrm{t}}\ast \frac{\partial \mathrm{o}\mathrm{u}\mathrm{t}}{\partial {\mathrm{H}}_2}\ast {\mathrm{H}}_1\end{array}$$
H1趋向于0，W2梯度消失
H1趋向于无穷，W2梯度梯度爆炸

避免梯度消失、梯度爆炸，就要控制输出层的输出值的范围，不能太大或者太小

1.1 梯度爆炸

1、0均值，1标准差

class MLP(nn.Module):
    def __init__(self, neural_num, layers):
        super(MLP, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(neural_num, neural_num, bias=False) for i in range(layers)])
        self.neural_num = neural_num

    def forward(self, x):
        for (i, linear) in enumerate(self.linears):
            x = linear(x)
            # x = torch.tanh(x)
            # x = torch.relu(x)
            # 打印output为无穷时所在
            print("layer:{}, std:{}".format(i, x.std()))
            if torch.isnan(x.std()):
                print("output is nan in {} layers".format(i))
                break
        return x
    # 参数初始化
    def initialize(self):