1.代码中如何进行参数初始化
参数初始化可以在网络结构设计时进行,例如
class AlexNet(nn.Module):
def __init__(self):
super(AlexNet,self).__init__()
self.conv1 = nn.Conv2d(3, 64, 5)
# nn.init.xavier_uniform(self.conv1.weight)
self.pool1 = nn.MaxPool2d(3, 2)
self.conv2 = nn.Conv2d(64, 64, 5)
# nn.init.xavier_uniform(self.conv2.weight)
self.pool2 = nn.MaxPool2d(3, 2)
self.fc1 = nn.Linear(1024, 384)
self.fc2 = nn.Linear(384, 192)
self.fc3 = nn.Linear(192, 10)
但是网络层数太多时这样做明显效率太低,所以可以单独写一个参数初始化的函数,例如:
def weights_init(m):
if isinstance(m, nn.Conv2d):
nn.init.xavier_uniform_(m.weight)
elif isinstance(m,nn.Linear):
nn.init.normal_(m.weight)
net = AlexNet()
net.apply(weights_init)
1)有些旧的格式现在都不推荐使用了,请按照pytorch官网最新格式进行书写,例如:
正确:torch.nn.init.
xavier_uniform_
错误:torch.nn.init.
xavier_uniform
2)pytorch中集成了许多初始化方法,具体可到官网查阅
TORCH.NN.INIT:https://pytorch.org/docs/stable/nn.html#torch-nn-init
博客里的总结:https://blog.csdn.net/faner1994/article/details/78211533
3)自己编写初始化方法
def weight_init(m):
# 使用isinstance来判断m属于什么类型
if isinstance(m, nn.Conv2d):
n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
m.weight.data.normal_(0, math.sqrt(2. / n))
elif isinstance(m, nn.BatchNorm2d):
# m中的weight,bias其实都是Variable,为了能学习参数以及后向传播
m.weight.data.fill_(1)
m.bias.data.zero_()
2.参数初始化的意义
https://blog.csdn.net/mzpmzk/article/details/79839047