训练深度神经网络的一些提示和技巧

Ambition_LAO

于 2024-07-26 10:34:33 发布

阅读量455

点赞数 10

文章标签：神经网络深度学习

本文链接：https://blog.csdn.net/GDHBFTGGG/article/details/140709022

版权

训练深度神经网络是一个复杂的过程，需要大量的知识和经验才能获得最佳模型。以下是一些我在训练深度神经网络时学到的提示和技巧，可能对您的研究有所帮助，并可以加快网络架构或参数搜索的速度。

1. 始终在网络中使用标准化层

在网络中使用标准化层非常重要。如果您使用大批量（例如10个或更多）训练网络，请使用BatchNormalization层。否则，如果您使用小批量（例如1个）进行训练，请改用InstanceNormalization层。如果增加批大小，BatchNormalization会提高性能，而小批量情况下InstanceNormalization会略微提高性能。或者您也可以尝试GroupNormalization。

以下是PyTorch中不同归一化层的示例代码：

批标准化（Batch Normalization）

import torch import torch.nn as nn

class ModelWithBatchNorm(nn.Module): def __init__(self): super(ModelWithBatchNorm, self).__init__() self.fc1 = nn.Linear(10, 20) self.bn1 = nn.BatchNorm1d(20) self.fc2 = nn.Linear(20, 10)

def forward(self, x): x = self.fc1(x) x = self.bn1(x) x = torch.relu(x) x = self.fc2(x) return x

model_with_batchnorm = ModelWithBatchNorm()

实例归一化（Instance Normalization）

import torch import torch.nn as nn

class ModelWithInstanceNorm(nn.Module): def __init__(self): super(ModelWithInstanceNorm, self).__init__() self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1) self.in1 = nn.InstanceNorm2d(16) self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1)

def forward(self, x): x = self.conv1(x) x = self.in1(x) x = torch.relu(x) x = self.conv2(x) return x

model_with_instancenorm = ModelWithInstanceNorm()

组规范化（Group Normalization）

import torch import torch.nn as nn

class ModelWithGroupNorm(nn.Module): def __init__(self): super(ModelWithGroupNorm, self).__init__() self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1) self.gn1 = nn.GroupNorm(num_groups=4, num_channels=16) self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1)

def forward(self, x): x = self.conv1(x) x = self.gn1(x) x = torch.relu(x) x = self.conv2(x) return x

model_with_groupnorm = ModelWithGroupNorm()

2. 特征串联后使用 SpatialDropout

如果有两个或多个卷积层对同一输入操作，请在特征串联后使用SpatialDropout。由于这些卷积层对相同的输入进行操作，输出特征可能是相关的，因此SpatialDropout会删除这些相关特征并防止过拟合。

SpatialDropout代码：GitHub

以下是PyTorch框架下的实现代码：

import torch import torch.nn as nn import torch.nn.functional as F

class SpatialDropout(nn.Module): def __init__(self, dropout_ratio): super(SpatialDropout, self).__init__() if not 0.0 <= dropout_ratio < 1.0: raise ValueError('dropout_ratio must be in the range [0, 1)') self.dropout_ratio = dropout_ratio

def forward(self, x): if self.training: batch_size, channels, height, width = x.size() mask = torch.ones(batch_size, channels, height, width, device=x.device) mask = F.dropout2d(mask, p=self.dropout_ratio, training=True) x = x * mask return x

def spatial_dropout(x, ratio=0.1): if ratio < 0 or ratio >= 1: raise ValueError('dropout_ratio must be in the range [0, 1)') return SpatialDropout(ratio)(x)

if __name__ == '__main__': class CNN(nn.Module): def __init__(self): super(CNN, self).__init__() self.conv1 = nn.Conv2d(1, 32, kernel_size=3) self.dropout = SpatialDropout(dropout_ratio=0.5)

def forward(self, x): x = self.conv1(x) x = F.relu(x) x = self.dropout(x) return x

model = CNN() model.train() input_data = torch.randn(1, 1, 28, 28) output = model(input_data) model.eval() output_eval = spatial_dropout(input_data) print("训练时输出的形状:", output.shape) print("评估时输出的形状:", output_eval.shape)

3. 多尺度特征池化模块

为了捕获对象周围的上下文信息，使用多尺度特征池化模块可以进一步帮助提高准确性。这种思想已成功应用于语义分割或前景分割。

空间金字塔池化（SPP）是一个消除网络固定大小约束的池化层，即CNN不需要固定大小的输入图像。以下是PyTorch中的实现代码：GitHub

import math
import torch
from torch import nn

def spatial_pyramid_pool(self, previous_conv, num_sample, previous_conv_size, out_pool_size):
for i in range(len(out_pool_size)):
h_wid = int(math.ceil(previous_conv_size[0] / out_pool_size[i]))
w_wid = int(math.ceil(previous_conv_size[1] / out_pool_size[i]))
h_pad = (h_wid * out_pool_size[i] - previous_conv_size[0] + 1) / 2
w_pad = (w_wid * out_pool_size[i] - previous_conv_size[1] + 1) / 2
maxpool = nn.MaxPool2d((h_wid, w_wid), stride=(h_wid, w_wid), padding=(h_pad, w_pad))
x = maxpool(previous_conv)
if (i == 0):
spp = x.view(num_sample, -1)
else:
spp = torch.cat((spp, x.view(num_sample, -1)), 1)
return spp

class YourModel(nn.Module):
def __init__(self):
super(YourModel, self).__init__()
# 定义其他层

def forward(self, x):
spp_output = self.spatial_pyramid_pool(x, num_sample, previous_conv_size, out_pool_size)
return spp_output

def spatial_pyramid_pool(self, previous_conv, num_sample, previous_conv_size, out_pool_size):
return spatial_pyramid_pool(self, previous_conv, num_sample, previous_conv_size, out_pool_size)

model = YourModel()
batch_size = 32
channels = 3
height = 224
width = 224
x = torch.randn(batch_size, channels, height, width)
output = model(x)

4. 选择一个正确的优化器

流行的自适应优化器有Adam、Adagrad、Adadelta或RMSprop等。SGD+momentum也被广泛应用。自适应优化器可以加快收敛，但可能陷入局部最小值并提供较差的泛化能力。SGD+momentum可以找到全局最小值，但依赖于稳健的初始化，并且可能比其他自适应优化器需要更长的时间才能收敛。建议使用SGD+momentum，因为它通常能达到更好的最优状态。