为节省时间,特此记录有关pytorch-权重初始化和L2_weight设置的东西。
初始化:
def init_weights(m: nn.Module):
for name, param in m.named_parameters():
if 'weight' in name:
nn.init.normal_(param.data, mean=0, std=0.01)
else:
nn.init.constant_(param.data, 0)
model.apply(init_weights)
l2-weight正则化项:有的只对权重进行正则化,不对偏执进行正则化。当时查了很久,觉得靠谱的版本特此记录:
参考:
https://discuss.pytorch.org/t/l2-regularization-with-only-weight-parameters/56853/2
https://blog.csdn.net/loseinvain/article/details/81708474#commentsedit
weight_p, bias_p = [], []
for name, p in model.named_parameters():
if 'bias' in name:
bias_p += [p]
else:
weight_p += [p]
optimizer = optim.Adam(
[{'params': bias_p, 'weight_decay': 0},
{'params': weight_p, 'weight_decay': 1e-2}],
lr=TRAIN_LR,
betas=(0.5, 0.999)
)