PyTorch 1.x 常用API

1. 简介

1.1 ResNet模型

1.2 torch.nn.Module

• 所有神经网络模型的基类
• 示例

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

1.2.1 torch.nn.Module.cpu()

• 功能
  • 将所有模型参数和缓存数据移动到 CPU

1.2.2 torch.nn.Module.cuda

• 定义

cuda(device: Union[int, torch.device, None] = None) 

• 功能
  • 将所有模型参数和缓存数据移动到 GPU

1.2.3 torch.nn.Module.eval()

• 功能
  • 将模块设置为评估模式

1.2.4 torch.nn.Module.state_dict

• 定义：

state_dict(destination=None, prefix='', keep_vars=False)

• 功能：
  • 返回整个模型状态的字典

1.2.5 torch.nn.Module.load_state_dict

• 定义：

load_state_dict(state_dict: Dict[str, torch.Tensor], strict: bool = True)

• 功能：
  • 将参数和缓冲区数据从state_dict复制到此模型中

1.3 torch.no_grad

• 功能：
  • 禁用梯度计算的上下文管理器
  • 当确定不会调用Tensor.backward（）时，禁用梯度计算对于推断很有用
  • 它将减少原本需要require_grad = True的计算和内存消耗
• 示例

import torch
x = torch.tensor([1.0], requires_grad=True)
with torch.no_grad():
    y = x * 2
y.requires_grad

2. 卷积层(Convolution Layers)API

2.1 nn.Conv1d

• 功能：在由多个输入平面组成的输入信号上应用一维卷积 (1D Convolution)
• 用途：
  • 只对宽度进行卷积，对高度不卷积。通常，输入大小为word_embedding_dim * max_length，其中，word_embedding_dim为词向量的维度，max_length为句子的最大长度。卷积核窗口在句子长度的方向上滑动，进行卷积操作。
• 定义

class torch.nn.Conv1d(in_channels: int, 
                      out_channels: int, 
                      kernel_size: Union[T, Tuple[T]], 
                      stride: Union[T, Tuple[T]] = 1, 
                      padding: Union[T, Tuple[T]] = 0, 
                      dilation: Union[T, Tuple[T]] = 1, 
                      groups: int = 1, 
                      bias: bool = True, 
                      padding_mode: str = 'zeros')

• 参数说明

参数	说明
in_channels	输入信号的通道数 （在文本应用中，即为词向量的维度）
out_channels	卷积输出的通道数，即filter的个数
kernel_size	卷积核的尺寸，卷积核的大小为(k,)，第二个维度是由in_channels来决定的，所以实际上卷积大小为kernel_size*in_channels
stride	卷积步长
padding	输入的每一条边补充0的层数
dilation	卷积核元素之间的间距
groups	从输入通道到输出通道的阻塞连接数
bias	如果bias=True，添加偏置

• 工作原理
  • 输入Size: $(N,C_{in},L)$
  • 输出Size: $(N,C_{out},L_{out})$
  • N：表示batch size
  • C：表示通道数
  • L：表示信号序列的长度
  • ⋆：valid 1D cross-correlation operator

• 输入输出shape
  

• 示例

m = nn.Conv1d(16, 33, 3, stride=2)
input = torch.randn(20, 16, 50)
output = m(input)
print(output.shape)

# ==
# torch.Size([20, 33, 24])

2.2 nn.Conv2d

• 功能：在由多个输入平面组成的输入信号上应用二维卷积 (2D Convolution)
• 定义

class torch.nn.Conv2d(in_channels: int, 
                      out_channels: int, 
                      kernel_size: Union[T, Tuple[T, T]], 
                      stride: Union[T, Tuple[T, T]] = 1, 
                      padding: Union[T, Tuple[T, T]] = 0, 
                      dilation: Union[T, Tuple[T, T]] = 1, 
                      groups: int = 1, 
                      bias: bool = True, 
                      padding_mode: str = 'zeros')

• 工作原理

  • 输入Size: $(N,C_{in},H,W)$
  • 输出Size: $(N,C_{out},H_{out},W_{out})$
  • N：表示有N个样本
  • C：表示通道数
  • H：表示输入平面的高度（以像素为单位）
  • W：表示输入平面的宽度（以像素为单位）
  • ⋆：valid 2D cross-correlation operator
    

• 输入输出shape
  

• 示例

# With square kernels and equal stride
m = nn.Conv2d(16, 33, 3, stride=2)
# non-square kernels and unequal stride and with padding
m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
# non-square kernels and unequal stride and with padding and dilation
m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2), dilation=(3, 1))
input = torch.randn(20, 16, 50, 100)
output = m(input)
print(output.shape)

# ===
# torch.Size([20, 33, 26, 100])

2.3 nn.Conv3d

• 功能：在由多个输入平面组成的输入信号上应用三维卷积 (3D Convolution)
• 定义

class torch.nn.Conv3d(in_channels: 
                      int, out_channels: int, 
                      kernel_size: Union[T, Tuple[T, T, T]], 
                      stride: Union[T, Tuple[T, T, T]] = 1, 
                      padding: Union[T, Tuple[T, T, T]] = 0, 
                      dilation: Union[T, Tuple[T, T, T]] = 1, 
                      groups: int = 1, 
                      bias: bool = True, 
                      padding_mode: str = 'zeros')

• 工作原理

  • 输入Size: $(N,C_{in},D,H,W)$
  • 输出Size: $(N,C_{out},D_{out},H_{out},W_{out})$
  • N：表示有N个样本
  • C：表示通道数
  • H：表示输入平面的高度（以像素为单位）
  • W：表示输入平面的宽度（以像素为单位）
  • ⋆：valid 3D cross-correlation operator
    

• 输入输出shape
  

• 示例

2.4 nn.ConvTranspose1d

• 功能：在由多个输入平面组成的输入图像上应用一维转置卷积运算 (1D Transposed Convolution)
• 定义

classtorch.nn.ConvTranspose1d(in_channels: int, 
                              out_channels: int, 
                              kernel_size: Union[T, Tuple[T]], 
                              stride: Union[T, Tuple[T]] = 1, 
                              padding: Union[T, Tuple[T]] = 0, 
                              output_padding: Union[T, Tuple[T]] = 0, 
                              groups: int = 1, 
                              bias: bool = True, 
                              dilation: Union[T, Tuple[T]] = 1, 
                              padding_mode: str = 'zeros')

• 输入输出shape
  • 输入Size: $(N,C_{in},L_{in})$
  • 输出Size: $(N,C_{out},L_{out})$
  • N：表示batch size
    $$\begin{gathered} L_{out}=(L_{in}-1)\times stride-2\times padding+dilation\times (kernel\_size-1)+ \\ output\_padding+1 \end{gathered}$$
• 示例

2.5 nn.ConvTranspose2d

• 功能：在由多个输入平面组成的输入图像上应用二维转置卷积运算 ( 2D Transposed Convolution)
• 定义

class torch.nn.ConvTranspose2d(in_channels: int, 
                               out_channels: int, 
                               kernel_size: Union[T, Tuple[T, T]], 
                               stride: Union[T, Tuple[T, T]] = 1, 
                               padding: Union[T, Tuple[T, T]] = 0, 
                               output_padding: Union[T, Tuple[T, T]] = 0, 
                               groups: int = 1, 
                               bias: bool = True, 
                               dilation: int = 1, 
                               padding_mode: str = 'zeros')

• 工作原理

  • 该模块可以看作是Conv2d相对于其输入的梯度
  • 它也被称为分数步法卷积(fractionally-strided convolution)或反卷积(deconvolution)（尽管它不是实际的反卷积运算）

• 输入输出shape

  • 输入：$(N,C_{in},H_{in},W_{in})$
  • 输出：$(N,C_{out},H_{out},W_{out})$
    $$\begin{gathered} H_{out}=(H_{in}-1)\times stride[0]-2\times padding[0]+ \\ dilation[0]\times (kernel\_size[0]-1)+output\_padding[0]+1 \end{gathered}$$
    $$\begin{gathered} W_{out}=(W_{in}-1)\times stride[1]-2\times padding[1]+ \\ dilation[1]\times (kernel\_size[1]-1)+output\_padding[1]+1 \end{gathered}$$

• 示例

# With square kernels and equal stride
m = nn.ConvTranspose2d(16, 33, 3, stride=2)
# non-square kernels and unequal stride and with padding
m = nn.ConvTranspose2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
input = torch.randn(20, 16, 50, 100)
output = m(input)
# exact output size can be also specified as an argument
input = torch.randn(1, 16, 12, 12)
downsample = nn.Conv2d(16, 16, 3, stride=2, padding=1)
upsample = nn.ConvTranspose2d(16, 16, 3, stride=2, padding=1)
h = downsample(input)

print(h.shape)
# torch.Size([1, 16, 6, 6])

output = upsample(h, output_size=input.size())

print(output.size())
# torch.Size([1, 16, 12, 12])

2.6 nn.ConvTranspose3d

• 功能：在由多个输入平面组成的输入图像上应用三维转置卷积运算 ( 3D Transposed Convolution)
• 定义

class torch.nn.ConvTranspose3d(in_channels: int, 
                               out_channels: int, 
                               kernel_size: Union[T, Tuple[T, T, T]], 
                               stride: Union[T, Tuple[T, T, T]] = 1, 
                               padding: Union[T, Tuple[T, T, T]] = 0, 
                               output_padding: Union[T, Tuple[T, T, T]] = 0, 
                               groups: int = 1, 
                               bias: bool = True, 
                               dilation: Union[T, Tuple[T, T, T]] = 1,
                                padding_mode: str = 'zeros')

• 输入输出shape

  • 输入：$(N,C_{in},D_{in},H_{in},W_{in})$
  • 输出：$(N,C_{out},D_{out},H_{out},W_{out})$
    $$\begin{gathered} D_{out}=(D_{in}-1)\times stride[0]-2\times padding[0]+dilation[0]\times (kernel\_size[0]-1)+ \\ output\_padding[0]+1 \end{gathered}$$
    $$\begin{gathered} H_{out}=(H_{in}-1)\times stride[1]-2\times padding[1]+dilation[1]\times (kernel\_size[1]-1)+ \\ output\_padding[1]+1 \end{gathered}$$
    $$\begin{gathered} W_{out}=(W_{in}-1)\times stride[2]-2\times padding[2]+dilation[2]\times (kernel\_size[2]-1)+ \\ output\_padding[2]+1 \end{gathered}$$

• 示例

# With square kernels and equal stride
m = nn.ConvTranspose3d(16, 33, 3, stride=2)
# non-square kernels and unequal stride and with padding
m = nn.ConvTranspose3d(16, 33, (3, 5, 2), stride=(2, 1, 1), padding=(0, 4, 2))
input = torch.randn(20, 16, 10, 50, 100)
output = m(input)
print(output.shape)
# ===
# torch.Size([20, 33, 21, 46, 97])

3. 池化层（Pooling Layers） API

3.1 torch.nn.MaxPool2d

• 对每个channel单独进行下采样
• 定义

class torch.nn.MaxPool2d(kernel_size: Union[T, Tuple[T, ...]], 
                         stride: Optional[Union[T, Tuple[T, ...]]] = None, 
                         padding: Union[T, Tuple[T, ...]] = 0, 
                         dilation: Union[T, Tuple[T, ...]] = 1, 
                         return_indices: bool = False, 
                         ceil_mode: bool = False)

• 输入输出shape
  

• 示例代码

# pool of square window of size=3, stride=2
m = nn.MaxPool2d(3, stride=2)
# pool of non-square window
# m = nn.MaxPool2d((3, 2), stride=(2, 1))
input = torch.randn(20, 16, 50, 32)
output = m(input)
print(output.shape)

# ===
# torch.Size([20, 16, 24, 15])

3.2 torch.nn.AvgPool2d

• 定义

class torch.nn.AvgPool2d(kernel_size: Union[T, Tuple[T, T]], 
                         stride: Optional[Union[T, Tuple[T, T]]] = None, 
                         padding: Union[T, Tuple[T, T]] = 0, 
                         ceil_mode: bool = False, 
                         count_include_pad: bool = True, 
                         divisor_override: bool = None)

• 工作原理

  • 输入Size：$(N,C,H,W)$
  • 输出Size：$(N,C,H_{out},W_{out})$
  • Kernel Size：$(kH,kW)$
    

• 输入输出shape
  

• 示例代码

# pool of square window of size=3, stride=2
m = nn.AvgPool2d(3, stride=2)
# pool of non-square window
# m = nn.AvgPool2d((3, 2), stride=(2, 1))
input = torch.randn(20, 16, 50, 32)
output = m(input)
print(output.shape)

# ===
# torch.Size([20, 16, 24, 15])

3.3 torch.nn.AdaptiveMaxPool2d

• 定义

class torch.nn.AdaptiveMaxPool2d(output_size: Union[T, Tuple[T, ...]], 
                                 return_indices: bool = False)

• 参数说明：

  • output_size：输出平面的Size

• 示例

# target output size of 5x7
m = nn.AdaptiveMaxPool2d((5,7))
input = torch.randn(1, 64, 8, 9)
output = m(input)
print(output.shape)

# target output size of 7x7 (square)
m = nn.AdaptiveMaxPool2d(7)
input = torch.randn(1, 64, 10, 9)
output = m(input)
print(output.shape)

# target output size of 10x7
m = nn.AdaptiveMaxPool2d((None, 7))
input = torch.randn(1, 64, 10, 9)
output = m(input)
print(output.shape)

# ===
# torch.Size([1, 64, 5, 7])
# torch.Size([1, 64, 7, 7])
# torch.Size([1, 64, 10, 7])

3.4 torch.nn.AdaptiveAvgPool2d

• 定义

class torch.nn.AdaptiveAvgPool2d(output_size: Union[T, Tuple[T, ...]])

• 示例

# target output size of 5x7
m = nn.AdaptiveAvgPool2d((5,7))
input = torch.randn(1, 64, 8, 9)
output = m(input)
print(output.shape)

# target output size of 7x7 (square)
m = nn.AdaptiveAvgPool2d(7)
input = torch.randn(1, 64, 10, 9)
output = m(input)
print(output.shape)

# target output size of 10x7
m = nn.AdaptiveMaxPool2d((None, 7))
input = torch.randn(1, 64, 10, 9)
output = m(input)
print(output.shape)

# ===
# torch.Size([1, 64, 5, 7])
# torch.Size([1, 64, 7, 7])
# torch.Size([1, 64, 10, 7])

4. 归一化层(Normalization Layers) API

4.1 torch.nn.BatchNorm2d

• 定义

class torch.nn.BatchNorm2d(num_features,  # 特征平面的个数，即channels number
                           eps=1e-05, 
                           momentum=0.1, 
                           affine=True, 
                           track_running_stats=True)

• 参数说明

参数	说明
num_features	即为输入size $(N,C,H,W)$中的$C$
eps	为数值稳定而添加到分母的值；默认值：1e-5
momentum	用于计算running_mean和running_var的值。可对累积移动平均（即简单平均）设置为“None”；默认值：0.1
affine	一个布尔值，当设置为True时，此模块具有可学习的仿射参数； 默认值：True
track_running_stats	1) 一个布尔值，当设置为True时，此模块跟踪运行平均值和方差； 2) 设置为False时，如果运行均值和方差为None，则此模块不跟踪此类统计信息，而是在训练和评估模式下均使用批处理统计信息 3) 默认值：True

• 输入输出shape
• 输入Size： $(N,C,H,W)$
• 输出Size：$(N,C,H,W)$， 即输出与输出相同
• 示例

# With Learnable Parameters
m = nn.BatchNorm2d(100)
# Without Learnable Parameters
# m = nn.BatchNorm2d(100, affine=False)
input = torch.randn(20, 100, 35, 45)
output = m(input)
print(output.shape)

# ===
# torch.Size([20, 100, 35, 45])

5. 激活函数API

5.1 torch.nn.ReLU

• 功能
  $$ReLU(x)=max(0,x)$$
• 定义

torch.nn.ReLU(inplace: bool = False)

• inplace为True, 将会改变输入的数据 ，否则不会改变原输入，只会产生新的输出

5.2 torch.nn.Linear

• 功能：
  • 对输入数据应用线性变换
  • $y=x{A}^T+B$
• 定义

torch.nn.Linear(in_features: int, out_features: int, bias: bool = True)

• 参数
  • in_features ：每个输入样本的尺寸
  • out_features：每个输出样本的尺寸
  • bias：若为False，则不学习bias；否则学习
• 示例代码

m = nn.Linear(20, 30)
input = torch.randn(128, 20)
output = m(input)
print("input.shape=", input.shape)
print("output.shape=", output.shape)
# input.shape= torch.Size([128, 20])
# output.shape= torch.Size([128, 30])

5.3 torch.nn.Bilinear

• 功能：
  • 对输入数据应用双线性变换
  • $y=x_1^{T}A{x}_2+b$