构建完网络后, 往往需要初始化权重;其实也可以不需手动初始化,在声明网络时,pytorch有默认的初始化方式,如:
import torch
x = torch.Tensor(2,2)
print(x)
输出为:
tensor([[-2.0363e+09, 4.5914e-41],
[-2.8577e+17, 5.6612e-43]])
上面定义了一个tensor,但没初始化,而pytorch会自动对其进行初始化,只是初始化的参数无规律且相差甚远。如果对于一个没有预训练的网络,有可能难以使网路收敛。因此,在声明网络后,有必要手动对网络进行权重初始化。而一般采用正态分布的方式初始化权重。
在pytorch中,常用的有两种权重初始化的方法:
方法一:
先定义网络,然后加载权重。
import torch,nn as nn
class NET(nn.Module): # 声明网络
'''
定义网络层
'''
net = NET() # 定义网络
def weight_init(m): #初始化权重
if isinstance(m, nn.Conv3d):
n = m.kernel_size[0] * m.kernel_size[1] * m.kernel_size[2] * m.out_channels
m.weight.data.normal_(0, math.sqrt(2.0 / n))
m.bias.data.zero_()
elif isinstance(m, nn.BatchNorm3d):
m.weight.data.fill_(1)
m.bias.data.zero_()
elif isinstance(m, nn.Linear):
m.weight.data.normal_(0, 0.02)
m.bias.data.zero_()
net.apply(weight_init) # 加载权重
方法二:
再声明网络时初始化并加载权重
import torch,nn as nn
class NET(nn.Modele):
def __init__(self):
super(NET, self).__init__()
'''
定义网络层
'''
# 初始化权重
for m in self.modules():
if isinstance(m, nn.Conv3d):
nn.init.kaiming_normal_(m.weight.data)
elif isinstance(m, nn.BatchNorm3d):
m.weight.data.fill_(1)
m.bias.data.zero_()
elif isinstance(m, nn.Linear):
m.weight.data.normal_(0, 0.02)
m.bias.data.zero_()