ResNet

【Kaiming He Microsoft Research 】

参考的优秀博文：ResNet笔记

Abstract

背景：Deeper neural networks are more difficult to train.

提出的方法：present a residual learning framework to ease the training of networks

结果：1st place in all five main tracks 【ResNet152 VGG19】

• ImageNet classification
• ImageNet detection
• ImageNet localization
• COCO detection
• COCO segmentation

1. Introduction

1.1 Importance of Depth

因为CNN能够提取low/mid/high-level的特征，网络的层数越多，意味着能够提取到不同level的特征越丰富。并且，越深的网络提取的特征越抽象，越具有语义信息。

Deep networks naturally integrate low/mid/highlevel features [50] and classifiers in an end-to-end multilayer fashion, and the “levels” of features can be enriched by the number of stacked layers (depth).

1.2 Vanishing/Exploding Gradients

• 对于原来的网络，如果简单地增加深度，会导致梯度弥散或梯度爆炸
• 对于该问题的解决方法是正则化初始化和中间的正则化层（Batch Normalization），这样的话可以训练几十层的网络。

This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].

1.3 Degradation Problem

• 退化问题：网络层数增加，但是在训练集上的准确率却饱和甚至下降了。

with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly.

• 退化不是由于过拟合而造成的：增加更多的layers，导致了更高的training error

adding more layers to a suitably deep model leads to higher training error

1.4 Identity Mapping

• 如果深层网络的后面那些层是恒等映射，那么模型就退化为一个浅层网络

  There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model.

1.5 Deep Residual Learning

• 深度残差学习：把网络设计为H(x) = F(x) + x。我们可以转换为学习一个残差函数F(x) = H(x) - x. 只要F(x)=0，就构成了一个恒等映射H(x) = x.

Formally, denoting the desired underlying mapping as H(x), we let the stacked nonlinear layers fit another mapping of F(x) := H(x) - x. The original mapping is recast into F(x)+x.

• 拟合残差肯定更加容易

We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers

• F是求和前网络映射，H是从输入到求和后的网络映射。比如把5映射到5.1，那么引入残差前是F’(5)=5.1，引入残差后是H(5)=5.1, H(5)=F(5)+5, F(5)=0.1。这里的F’和F都表示网络参数映射，引入残差后的映射对输出的变化更敏感。比如s输出从5.1变到5.2，映射F’的输出增加了1/51=2%，而对于残差结构输出从5.1到5.2，映射F是从0.1到0.2，增加了100%。明显后者输出变化对权重的调整作用更大，所以效果更好。残差的思想都是去掉相同的主体部分，从而突出微小的变化，看到残差网络我第一反应就是差分放大器

1.6 Skip connections

• 用残差连接的方式，实现了恒等映射

In our case, the shortcut connections simply perform identity mapping

• 没有增加额外参数和计算量

Identity shortcut connections add neither extra parameter nor computational complexity.

2. Related Work

3. Deep Residual Learning

3.1. Residual Learning

• 正如我们在Introduction中所讨论的，如果可以将添加的层构造为恒等映射，那么较深的模型的训练误差应该不会大于较浅的模型。
• 退化问题表明，求解器在用多个非线性层逼近恒等映射时可能存在困难。
• 利用残差学习重新公式，如果恒等映射是最优的，求解器可以简单地将多个非线性层的权值向零逼近来实现恒等映射。

As we discussed in the introduction, if the added layers can be constructed as identity mappings, a deeper model should have training error no greater than its shallower counterpart. The degradation problem suggests that the solvers might have difficulties in approximating identity mappings by multiple nonlinear layers. With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings.

• 在实际情况中，恒等映射不太可能是最优的，但我们的重新表述可能有助于预先考虑这个问题。
• 如果最优函数更接近恒等映射而不是零映射，那么求解器就应该更容易找到与恒等映射相关的扰动，而不是将函数作为一个新函数来学习。

In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the problem. If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the solver to find the perturbations with reference to an identity mapping, than to learn the function as a new one.

3.2. Identity Mapping by Shortcuts

• a building block defined as ：

• If the dimensions of x and F not be equal:

• The form of the residual function F is flexible :残差连接可以连接多层网络

3.3 Network Architectures

3.3.1 对比的网络

3.3.2 ResNet的其他变体

3.3.3 deeper residual function

4. Experiments

4.1 Plain Networks vs. Residual Networks

• 实验了plain-18和plain-34，展示了退化问题。说明了退化问题不是因为梯度弥散，因为加入了BN。另外也不能简单地增加迭代次数来使其收敛，增加迭代次数仍然会出现退化问题。
• 实验了ResNet-18和ResNet-34不会出现退化问题，ResNet-34明显表现的比ResNet-18和plain-34好，证明了残差学习解决了随网络深度增加带来的退化问题。 而且同等深度的plain-18和ResNet-18，残差网络更容易优化，收敛更快。

4.2 Identity vs. Projection Shortcuts

• 对于同等映射维度不匹配时，匹配维度的两种方法，zero padding是参数free的，投影法会带来参数。
• 作者比较了这两种方法的优劣。实验证明，投影法会比zero padding表现稍好一些。因为zero padding的部分没有参与残差学习。
• 实验表明，将维度匹配或不匹配的同等映射全用投影法会取得更稍好的结果，但是考虑到不增加复杂度和参数free，一般不采用这种方法。

4.3 Deeper Bottleneck Architectures

• 考虑到时间花费，将原来的building block(残差学习结构)改为瓶颈结构。
• 首端和末端的1x1卷积用来削减和恢复维度，相比于原本结构，只有中间3x3成为瓶颈部分。这两种结构的时间复杂度相似。此时投影法映射带来的参数成为不可忽略的部分（以为输入维度的增大），所以要使用zero padding的同等映射。
• 替换原本ResNet的残差学习结构，同时也可以增加结构的数量，网络深度得以增加。生成了ResNet-50，ResNet-101，ResNet-152.
• 随着深度增加，因为解决了退化问题，性能不断提升。