python 正则化_Python学习笔记：正则化,pytorch

1、L2正则化(权重衰减)

2、Dropout

相当于训练了多个网络，测试时综合了多个网络的结果，使得网络泛化能力更强

需要注意的一点是测试时需要对网络输出乘以(1-失活概率P)，可以理解为假设训练时有30个神经元失活，因此训练时的输出为70个神经元的加权和，而测试时所有的100个神经元均会参与运算，那么数据的输出尺度发生了变化，因此需要乘以一项来抵消数据输出尺度的变化。

3、Batch Normalization

将一批数据的特征分布标准化为0均值1方差

引入gama和beta是为了提高模型的capacity ，可以让网络自己判断数据是否需要标准化，比如如何γ=标准差，β=均值，则相当于没有做任何处理。

分母修正项：为了避免分母为0产生错误

import torch

import numpy as np

import torch.nn as nn

from tools.common_tools import set_seed

set_seed(1) # 设置随机种子

# ======================================== nn.BatchNorm1d

# flag = 1

flag = 0

if flag:

batch_size = 3

num_features = 5

momentum = 0.3

features_shape = (1)

feature_map = torch.ones(features_shape) # 1D

feature_maps = torch.stack([feature_map*(i+1) for i in range(num_features)], dim=0) # 2D

feature_maps_bs = torch.stack([feature_maps for i in range(batch_size)], dim=0) # 3D

print("input data:\n{} shape is {}".format(feature_maps_bs, feature_maps_bs.shape))

bn = nn.BatchNorm1d(num_features=num_features, momentum=momentum)

running_mean, running_var = 0, 1

for i in range(2):

outputs = bn(feature_maps_bs)

print("\niteration:{}, running mean: {} ".format(i, bn.running_mean))

print("iteration:{}, running var:{} ".format(i, bn.running_var))

mean_t, var_t = 2, 0

running_mean = (1 - momentum) * running_mean + momentum * mean_t

running_var = (1 - momentum) * running_var + momentum * var_t

print("iteration:{}, 第二个特征的running mean: {} ".format(i, running_mean))

print("iteration:{}, 第二个特征的running var:{}".format(i, running_var))

# ======================================== nn.BatchNorm2d

flag = 1

# flag = 0

if flag:

batch_size = 3

num_features = 6

momentum = 0.3

features_shape = (2, 2)

feature_map = torch.ones(features_shape) # 2D

feature_maps = torch.stack([feature_map*(i+1) for i in range(num_features)], dim=0) # 3D

feature_maps_bs = torch.stack([feature_maps for i in range(batch_size)], dim=0) # 4D

print("input data:\n{} shape is {}".format(feature_maps_bs, feature_maps_bs.shape))

bn = nn.BatchNorm2d(num_features=num_features, momentum=momentum)

running_mean, running_var = 0, 1

for i in range(2):

outputs = bn(feature_maps_bs)

print("\niter:{}, running_mean.shape: {}".format(i, bn.running_mean.shape))

print("iter:{}, running_var.shape: {}".format(i, bn.running_var.shape))

print("iter:{}, weight.shape: {}".format(i, bn.weight.shape))

print("iter:{}, bias.shape: {}".format(i, bn.bias.shape))

# ======================================== nn.BatchNorm3d

# flag = 1

flag = 0

if flag:

batch_size = 3

num_features = 4

momentum = 0.3

features_shape = (2, 2, 3)

feature = torch.ones(features_shape) # 3D

feature_map = torch.stack([feature * (i + 1) for i in range(num_features)], dim=0) # 4D

feature_maps = torch.stack([feature_map for i in range(batch_size)], dim=0) # 5D

print("input data:\n{} shape is {}".format(feature_maps, feature_maps.shape))

bn = nn.BatchNorm3d(num_features=num_features, momentum=momentum)

running_mean, running_var = 0, 1

for i in range(2):

outputs = bn(feature_maps)

print("\niter:{}, running_mean.shape: {}".format(i, bn.running_mean.shape))

print("iter:{}, running_var.shape: {}".format(i, bn.running_var.shape))

print("iter:{}, weight.shape: {}".format(i, bn.weight.shape))

print("iter:{}, bias.shape: {}".format(i, bn.bias.shape))