ResNet浅记

对ResNe比较好奇，既简单学习并浅记一下，方便以后查阅。

ResNet是什么

Deep Residual Learning for Image Recognition

ResNet 网络是在 2015年 由微软实验室中的何凯明等几位大神提出，斩获当年ImageNet竞赛中分类任务第一名，目标检测第一名。获得COCO数据集中目标检测第一名，图像分割第一名。

ResNet的亮点

1. 超深的网络结构（超过1000层）。
2. 提出residual（残差结构）模块。
3. 使用Batch Normalization 加速训练（丢弃dropout）。

为什么采用residual

在ResNet提出之前，所有的神经网络都是通过卷积层和池化层的叠加组成的。

人们认为卷积层和池化层的层数越多，获取到的图片特征信息越全，学习效果也就越好。但是在实际的试验中发现，随着卷积层和池化层的叠加，不但没有出现学习效果越来越好的情况，反而两种问题：

1. 梯度消失和梯度爆炸
   梯度消失：若每一层的误差梯度小于1，反向传播时，网络越深，梯度越趋近于0
   梯度爆炸：若每一层的误差梯度大于1，反向传播时，网络越深，梯度越来越大

2. 退化问题
   随着层数的增加，预测效果反而越来越差。如下图所示

Figure 1: Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper network has higher training error, and thus test error.  

• 为了解决梯度消失或梯度爆炸问题，ResNet论文提出通过数据的预处理以及在网络中使用 BN（Batch Normalization）层来解决。
• 为了解决深层网络中的退化问题，可以人为地让神经网络某些层跳过下一层神经元的连接，隔层相连，弱化每层之间的强联系。这种神经网络被称为 残差网络 (ResNets)。ResNet论文提出了 residual结构（残差结构）来减轻退化问题，下图是使用residual结构的卷积网络，可以看到随着网络的不断加深，效果并没有变差，而是变的更好了。（虚线是train error，实线是test error）

Figure 2.Training on CIFAR-10. Dashed lines denote training error, and bold lines denote testing error. Left: plain networks. The error of plain-110 is higher than 60% and not displayed. Right: ResNets.  

residual的计算方式

residual结构使用了一种shortcut的连接方式，也可理解为捷径。让特征矩阵隔层相加，注意F(X)和X形状要相同，所谓相加是特征矩阵相同位置上的数字进行相加。

Figure 3.Residual learning: a building block.  

ResNet中两种不同的residual

Figure 4: A deeper residual function F for ImageNet. Left: a building block (on 56×56 feature maps) as in Fig. 3 for ResNet- 34. Right: a “bottleneck” building block for ResNet-50/101/152...  

1. 左侧残差结构称为 BasicBlock
2. 右侧残差结构称为 Bottleneck
   其中第一层的1× 1的卷积核的作用是对特征矩阵进行降维操作，将特征矩阵的深度由256降为64;
   第三层的1× 1的卷积核是对特征矩阵进行升维操作，将特征矩阵的深度由64升成256。
   降低特征矩阵的深度主要是为了减少参数的个数。
   如果采用BasicBlock,参数的个数应该是：256×256×3×3×2=1179648
   采用Bottleneck，参数的个数是：1×1×256×64+3×3×64×64+1×1×256×64=69632
   先降后升为了主分支上输出的特征矩阵和捷径分支上输出的特征矩阵形状相同，以便进行加法操作。

注：CNN参数个数 = 卷积核尺寸×卷积核深度 × 卷积核组数 = 卷积核尺寸 × 输入特征矩阵深度 × 输出特征矩阵深度
注意：搭建深层次网络时，采用三层的残差结构。

ResNet网络

Figure 5: Example network architectures for ImageNet. Left: the VGG-19 model [41] (19.6 billion FLOPs) as a reference. Middle: a plain network with 34 parameter layers (3.6 billion FLOPs). Right: a residual network with 34 parameter layers (3.6 billion FLOPs). The dotted shortcuts increase dimensions.  

Table 1: Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Downsampling is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2..  

ResNet_18实现——Pytorch

import torch
import torch.nn as nn



class Identity(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x


class Block(nn.Module):
    def __init__(self, in_dim, out_dim, stride):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=in_dim,
                               out_channels=out_dim,
                               kernel_size=3,
                               stride=stride,
                               padding=1,
                               bias=False)
        self.bn1 = nn.BatchNorm2d(num_features=out_dim)
        self.conv2 = nn.Conv2d(in_channels=out_dim,
                               out_channels=out_dim,
                               kernel_size=3,
                               stride=1,
                               padding=1,
                               bias=False)
        self.bn2 = nn.BatchNorm2d(num_features=out_dim)
        self.relu = nn.ReLU()

        if stride == 2 or in_dim != out_dim:
            self.downsample = nn.Sequential(
                nn.Conv2d(in_channels=in_dim,
                          out_channels=out_dim,
                          kernel_size=1,
                          stride=stride),
                nn.BatchNorm2d(num_features=out_dim)
            )
        else:
            self.downsample = Identity()

    def forward(self, x):
        h = x
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.conv2(x)
        x = self.bn2(x)
        indentity = self.downsample(h)
        x = x + indentity # 残差！！！！！！！
        x = self.relu(x)
        return x

class ResNet_18(nn.Module):
    def __init__(self, in_dim=64, num_classes=10):
        super().__init__()
        self.in_dim = in_dim
        # stem layers
        self.conv1 = nn.Conv2d(in_channels=3,
                               out_channels=in_dim,
                               kernel_size=3,
                               stride=1,
                               padding=1,
                               bias=False)
        self.bn1 = nn.BatchNorm2d(num_features=in_dim)
        self.relu = nn.ReLU()

        # blocks
        self.layersl = self._make_layer(dim=64, n_blocks=2, stride=1)
        self.layers2 = self._make_layer(dim=128, n_blocks=2, stride=2)
        self.layers3 = self._make_layer(dim=256, n_blocks=2, stride=2)
        self.layers4 = self._make_layer(dim=512, n_blocks=2, stride=2)

        # head layer
        self.avgpool = nn.AdaptiveAvgPool2d(output_size=1) # 根据实际调整大小，变成我们想要的尺寸
        self.fc = nn.Linear(in_features=512, out_features=num_classes)

    def _make_layer(self, dim, n_blocks, stride):
        layer_list = []
        layer_list.append(Block(self.in_dim, dim, stride=stride))
        self.in_dim = dim
        for i in range(1, n_blocks):
            layer_list.append(Block(self.in_dim, dim, stride=1))
        return nn.Sequential(*layer_list)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.layersl(x)
        x = self.layers2(x)
        x = self.layers3(x)
        x = self.layers4(x)
        x = self.avgpool(x)
        x = x.view(x.size()[0], -1)  # Flatten
        x = self.fc(x)
        return x

def main():
    t = torch.randn([4, 3, 32, 32])
    model = ResNet_18()
    # print(model)
    out = model(t)
    print(out)

if __name__ == "__main__":
    main()