【EfficientNet】《EfficientNet：Rethinking Model Scaling for Convolutional Neural Networks》

ICML-2019

---

---

---

1 Background and Motivation

Scaling up ConvNets is widely used to achieve better accuracy.

常见的如

1）scale up 网络深度（比如 resnet50 to resnet 101），

2）scale up 网络的宽度（resnet50 to wide-resnet）

还有不常见的如

3）scale up 输入的分辨率

三者之间是否有内在联系？三者联调如何实现最大精度的提升呢？

本文作者 first to empirically quantify the relationship among all three dimensions of network width, depth, and resolution，以高效的提升模型精度

2 Related Work

• ConvNet Accuracy
• ConvNet Efficiency——lightweight network
• Model Scaling——width, depth, and resolutions

3 Advantages / Contributions

效仿 MNASNet AutoML 出 EfficientNet-B0，从 width, depth, and resolutions 三个维度 compound scale up EfficientNet-B0 形成不同大小的 EfficientNet-Bx，在 ImageNet 上实现 SOTA 且网络参数很少，跨数据集验证泛化性能也很棒（5/8 SOTA）

4 Compound Model Scaling

核心：

4.1 Problem Formulation

神经网络 $N$ 可以由堆叠的层 $F(X)$ 来表示

• $X_1$ 是 input tensor
• $F_j$ 是 operator（eg conv 和 activation），其中 $j$ 表示 layer $j$

更模块化一点可以表示为

• $F_i^{L_i}$ 表示 layer $F_i$ 在 stage $i$ 中重复了 $L_i$ 次

网络迭代以提升精度的过程可表示为

4.2 Scaling Dimensions

Scaling up any dimension of network width, depth, or resolution improves accuracy, but the accuracy gain diminishes for bigger models.

1）scaling Depth

优势：capture richer and more complex features

缺点：more difficult to train due to the vanishing gradient problem——diminishing accuracy return for very deep ConvNets（一定深度后 ACC 会达到瓶颈）

2）scaling Width

通道数增加了

优势：wider networks tend to be able to capture more fine-grained features and are easier to train

缺点：have difficulties in capturing higher level features

3）scaling Resolution

优点：potentially capture more fine-grained patterns

4.3 Compound Scaling

上图 width 固定，改变 depth 和 resolution 来观测结果，发现同时改 depth 和 resolution 效果最猛

In order to pursue better accuracy and efficiency, it is critical to balance all dimensions of network width, depth, and resolution during ConvNet scaling

基于4.2 和 4.3 小节红色字体的分析，作者提出了如下的 compound scaling 方法

• $\alpha$、$\beta$、$\gamma$ 是通过 small grid search 来获取的

• $\varphi$ 是 a user-specified coefficient that controls how many more resources are available for model scaling

为啥约束 $\alpha$ 时是 $\alpha$，而约束 $\beta$、$\gamma$ 时是 ${\beta }^2$、${\gamma }^2$？

doubling network depth will double FLOPS, but doubling network width or resolution will increase FLOPS by four times

按照作者的 compound scaling 方式，网络的 FLOPS 变成了原来的

倍

5 EfficientNet Architecture

基于 MNASNet 去 AutoML 基础网络 EfficientNet-B0——we optimize FLOPS rather than latency since we are not targeting any specific hardware device.

MBConv 是 mobilenet V2 的 inverted bottleneck

step 1：固定 $\varphi =1$ 去搜最优的 $\alpha$，$\beta$，$\gamma$——we find the best values for EfficientNet-B0 are $\alpha =1.2$，$\beta =1.1$， $\gamma =1.15$

step 2：固定 $\alpha$，$\beta$，$\gamma$，增大 $\varphi$ 来增大网络（EfficientNet-B1~EfficientNet-B7）

6 Experiments

6.1 Datasets

• ImageNet
• CIFAR10
• CIFAR100
• Birdsnap
• Stanford Cars
• Flowers
• FGVC Aircraft
• Oxford-IIIT Pets
• Food-101

6.2 Experimental for ImageNet

1）Scaling Up MobileNets and ResNets

compound scaling 还是比 single scaling 猛哒

2）ImageNet Results for EfficientNet

下面看看速度

可以说是又快又猛

6.3 Transfer Learning Results for EfficientNet

又快又猛

这个图画成不同网络用 compound scaling（点变成线） 就更惊艳啦

5 / 8 SOTA 强强强

7 Conclusion（own）

• width / depth / resolution 单独调的优缺点以及对网络 FLOPS 影响的差异

• width / depth / resolution 组合调更猛，初始的缩放因子 $\alpha$、$\beta$、$\gamma$ 得 grid search 下

• bigger models need more regularization（eg：越大 dropout 系数越高，当然指数据规模不变的情况下）