《每日论文》ImageNet Classification with Deep Convolutional Neural Networks

这篇论文是剖析 CNN 领域的经典之作，也是入门 CNN 的必读论文。作者训练了一个面向数量为 1.2 百万的高分辨率的图像数据集 ImageNet， 图像的种类为 1000 种的深度卷积神经网络。

下载链接：NIPS-2012-imagenet-classification-with-deep-convolutional-neural-networks-Paper.pdf

---

ImageNet Classification with Deep Convolutional Neural Networks

基于深卷积神经网络的图像网络分类

Abstract

摘要

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes.
我们训练了一个大型的深卷积神经网络，将ImageNet LSVRC-2010竞赛中的120万张高分辨率图像分成1000个不同的类别。

On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art.
在测试数据上，我们获得了前1名和前5名的错误率分别为37.5%和17.0%，这比以前的最新技术要好得多。

The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
这个神经网络有6000万个参数和65万个神经元，由5个卷积层组成，其中一些是max pooling层，还有3个全连接层，最后是1000路softmax。

To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation.
为了加快训练速度，我们使用非饱和神经元和一个非常有效的GPU实现卷积运算。

To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective.
为了减少全连接层中的过拟合，我们采用了一种最近发展起来的正则化方法，称为“dropout”，这被证明是非常有效的。

We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
我们还在ILSVRC-2012竞赛中输入了该模型的一个变体，并获得了15.3%的前5名测试错误率，而第二名的测试错误率为26.2%。

1 Introduction

1简介

Current approaches to object recognition make essential use of machine learning methods.
当前的目标识别方法主要利用机器学习方法。

To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting.
为了提高它们的性能，我们可以收集更大的数据集，学习更强大的模型，并使用更好的技术来防止过度拟合。

Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]).
直到最近，标记图像的数据集相对较小，只有数万张图像（例如，NORB[16]、Caltech-101/256[8,9]和CIFAR-10/100[12]）。

Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations.
使用这种大小的数据集可以很好地解决简单的识别任务，特别是如果使用保留标签的转换对它们进行扩充的话。

For example, the current-best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4].
例如，当前MNIST数字识别任务的最佳错误率（<0.3%）接近人因绩效[4]。

But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets.
但是现实环境中的物体表现出相当大的可变性，所以要学会识别它们就必须使用更大的训练集。

And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to collect labeled datasets with millions of images.
事实上，小图像数据集的缺点已经得到广泛的认可（例如，Pinto等人[21]），但直到最近才有可能收集数百万图像的标记数据集。

The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.
新的更大的数据集包括LabelMe[23]，它由几十万个完全分割的图像组成，ImageNet[6]由22000多个类别的1500多万个标记的高分辨率图像组成。

To learn about thousands of objects from millions of images, we need a model with a large learning capacity.
为了从数百万张图片中了解成千上万的物体，我们需要一个具有很大学习能力的模型。

However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have.
然而，对象识别任务的巨大复杂性意味着即使是像ImageNet这样大的数据集也不能指定这个问题，因此我们的模型也应该有大量的先验知识来补偿我们没有的所有数据。

Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26].
卷积神经网络（CNNs）构成了一类这样的模型[16,11,13,18,15,22,26]。

Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies).
它们的容量可以通过改变它们的深度和宽度来控制，它们还可以对图像的性质（即统计数据的平稳性和像素相关性的局部性）做出强有力且基本正确的假设。

Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.
因此，与具有相似层大小的标准前馈神经网络相比，CNNs的连接和参数要少得多，因此它们更容易训练，而理论上最好的性能可能只是稍差一点。

Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images.
尽管CNNs 具有诱人的特性，并且尽管它们的本地体系结构相对高效，但在大规模应用于高分辨率图像时，它们的成本仍然高得令人望而却步。

Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.
幸运的是，当前的GPUs，加上高度优化的2D卷积实现，足够强大，可以方便地训练有趣的大型CNNs，而最近的数据集（如ImageNet）包含了足够多的标记示例来训练这些模型，而不会出现严重的过拟合。

The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets.
本文的具体贡献如下：我们在ILSVRC-2010和ILSVRC-2012竞赛中使用的ImageNet子集训练了迄今为止最大的卷积神经网络之一[2]，并在这些数据集上取得了迄今为止最好的结果。

We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly.
我们编写了一个高度优化的二维卷积GPU实现，以及训练卷积神经网络所固有的所有其他操作，我们会开源的。

Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3.
我们的网络包含了许多新的和不寻常的特性，这些特性提高了它的性能并减少了它的训练时间，这些特性在第3节中有详细说明。

The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4.
我们的网络规模使得过拟合成为一个严重的问题，即使有120万个标记的训练示例，因此我们使用了几种有效的技术来防止过度拟合，这些技术将在第4节中描述。

Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance.
我们的最终网络包含五个卷积层和三个全连接层，这个深度似乎很重要：我们发现去除任何卷积层（每个卷积层包含的模型参数不超过模型参数的1%）都会导致性能低下。

In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate.
最后，网络的大小主要受限于当前gpu上可用的内存量和我们愿意忍受的训练时间。

Our network takes between five and six days to train on two GTX 580 3GB GPUs.
我们的网络需要在两个GTX 580 3GB的GPU上训练五到六天的时间。

All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.
我们所有的实验都表明，我们的结果可以通过等待更快的gpu和更大的数据集变得可用而得到改进。

2 The Dataset

2数据集

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories.
ImageNet是一个由1500多万张标记的高分辨率图像组成的数据集，属于大约22000个类别。

The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool.
这些图片是从网络上收集的，并由人类标签工使用亚马逊的 Mechanical Turk 众包工具进行标记。

Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held.
从2010年开始，作为Pascal视觉对象挑战赛的一部分，一年一度的比赛被称为ImageNet大规模视觉识别挑战赛（ILSVRC）。

ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories.
ILSVRC使用ImageNet的一个子集，在1000个类别中的每个类别中大约有1000个图像。

In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.
总共大约有120万个训练图像、50000个验证图像和150000个测试图像。

ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments.
ILSVRC-2010是ILSVRC的唯一版本，其中测试集标签可用，因此这是我们进行大部分实验的版本。

Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable.
由于我们也在ILSVRC-2012竞赛中输入了我们的模型，因此在第6节中，我们也报告了这个版本的数据集的结果，对于这个版本，测试集标签是不可用的。

On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.
在ImageNet上，通常报告两个错误率：top-1和top-5，其中top-5错误率是测试图像中正确标签不在模型认为最有可能的五个标签中的一部分。

ImageNet consists of variable-resolution images, while our system requires a constant input dimen sionality.
ImageNet由可变分辨率的图像组成，而我们的系统需要一个恒定的输入维度。

Therefore, we down-sampled the images to a fixed resolution of 256 × 256.
因此，我们将图像下采样到256×256的固定分辨率。

Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256×256 patch from the resulting image.
给定一个矩形图像，我们首先重新缩放图像，使短边的长度为256，然后从得到的图像中裁剪出中心256×256的面片。

We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel.
我们没有以任何其他方式预处理图像，只是从每个像素减去训练集中的平均活动量。

So we trained our network on the (centered) raw RGB values of the pixels.
所以我们在像素的原始RGB值上训练我们的网络。

3 The Architecture

3架构

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected.
图2总结了我们网络的体系结构。它包含八个学习层-五个卷积层和三个全连接层。

Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs.
图2：我们的CNN架构图，明确显示了两个GPU之间的职责划分。
One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom.
一个GPU运行图形顶部的层部件，而另一个GPU运行底部的层部件。
The GPUs communicate only at certain layers.
GPU只在某些层进行通信。
The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264–4096–4096–1000.
网络的输入为150，528维，网络剩余层中的神经元数量为253，440–186，624–64，896–64，896–43，264–4096–4096–1000。

Below, we describe some of the novel or unusual features of our network’s architecture.
下面，我们将介绍我们的网络架构的一些新颖或不寻常的特性。

Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.
第3.1-3.4节根据我们对其重要性的估计进行排序，其中最重要的部分排在第一位。

3.1 ReLU Nonlinearity

3.1 ReLU非线性

The standard way to model a neuron’s output f as a function of its input x is with $f(x)=tanh(x)$ or $f(x)=(1+e^{-x}{)}^{-1}$.
将神经元输出f作为输入x的函数建模的标准方法是$f(x)=tanh(x)$或$f(x)=(1+e^{-x}{)}^{-1}$。

In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity $f(x)=max(0,x)$.
在具有梯度下降的训练时间方面，这些饱和非线性比非饱和非线性$f(x)=max(0,x)$慢得多。

Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs).
在Nair和Hinton[20]之后，我们将具有这种非线性的神经元称为整流线性单元（ReLUs）。

Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units.
具有ReLUs的深卷积神经网络的训练速度是tanh单元的等价物的数倍。

This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network.
图1展示了这一点，图1显示了在CIFAR-10数据集上达到25%训练误差所需的迭代次数，对于特定的四层卷积网络。

Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line).
图1：具有ReLUs（实线）的四层卷积神经网络在CIFAR-10上达到25%的训练错误率，比具有tanh神经元的等效网络（虚线）快6倍。
The learning rates for each network were chosen independently to make training as fast as possible.
每个网络的学习率都是独立选择的，以使训练尽可能快。
No regularization of any kind was employed.
没有进行任何形式的正规化。
The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consistently learn several times faster than equivalents with saturating neurons.
这里所展示的效果的大小随网络结构的不同而变化，但是具有ReLUs的网络学习速度始终比饱和神经元网络快几倍。

This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.
这张图表明，如果我们使用传统的饱和神经元模型，我们就无法用这么大的神经网络来进行这项工作。

We are not the first to consider alternatives to traditional neuron models in CNNs.
我们不是第一个在CNNs中考虑替代传统神经元模型的人。

For example, Jarrett et al. [11] claim that the nonlinearity $f(x)=|tanh(x)|$ works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset.
例如，Jarrett等人[11]声称非线性$f(x)=|tanh(x)|$特别适用于在加州理工大学101数据集上进行局部平均池化的对比度归一化类型。

However, on this dataset the primary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs.
然而，在这个数据集上，主要关注的是防止过度拟合，因此他们观察到的效果不同于我们在使用ReLUs时报告的对训练集的加速拟合能力。

Faster learning has a great influence on the performance of large models trained on large datasets.
快速学习对在大数据集上训练的大模型的性能有很大的影响。

3.2 Training on Multiple GPUs

3.2多个GPU训练

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it.
一个GTX 580 GPU只有3GB的内存，这限制了可以在其上训练的网络的最大大小。

It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU.
结果表明，120万个训练实例足以训练太大而不能安装在一个GPU上的网络。

Therefore we spread the net across two GPUs.
因此，我们将网络分布在两个gpu上。

Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory.
目前的GPU特别适合于跨GPU并行化，因为它们能够直接读写彼此的内存，而无需经过主机内存。

The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers.
我们采用的并行化方案基本上把一半的内核（或神经元）放在每个GPU上，还有一个额外的技巧：GPU只在特定的层中通信。

This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2.
这意味着，例如，第3层的内核从第2层的所有内核映射中获取输入。

However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU.
然而，第四层的内核只接受来自第三层的内核映射的输入，这些映射驻留在同一个GPU上。

Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the mount of computation.
选择连接模式是交叉验证的一个问题，但是这允许我们精确地调整通信量，直到它是计算量的一个可接受的部分。

The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cire¸san et al. [5], except that our columns are not independent (see Figure 2).
由此产生的架构与Cire¸san等人[5]采用的“柱状”CNN有点相似，只是我们的列不是独立的（见图2）。

This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU.
与在一个GPU上训练每个卷积层核数为一半的网络相比，该方案使前1位和前5位的错误率分别降低了1.7%和1.2%。

The two-GPU net takes slightly less time to train than the one-GPU net2.
两个GPU网络比一个GPU net2需要的训练时间稍短。

3.3 Local Response Normalization

3.3局部响应规范化

ReLUs have the desirable property that they do not require input normalization to prevent them from saturating.
ReLUs有一个理想的特性，即它们不需要输入规范化来防止饱和。

If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron.
如果至少有一些训练例子对ReLU产生了积极的输入，那么学习就会发生在这个神经元上。

However, we still find that the following local normalization scheme aids generalization.
然而，我们仍然发现下面的局部规范化方案有助于泛化。

Denoting by $a_{x,y}^i$ the activity of a neuron computed by applying kernel i at position (x, y) and then applying the ReLU nonlinearity, the response-normalized activity bi x,y is given by the expression
用$a_{x,y}^i$表示在(x, y)位置应用核$i$计算的神经元活动度，然后应用ReLU非线性度，由表达式给出响应归一化活性$b_{x,y}^i$由以下表达式给出：
$$b_{x,y}^i=a_{x,y}^i/(k+\alpha \sum _{j=max(0,i-n/2)}^{min(N-1,i+n/2)}(a_{x,y}^i{)}^2)$$

where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer.
其中，和在同一空间位置的n个“相邻”内核映射上运行，n是层中的内核总数。

The ordering of the kernel maps is of course arbitrary and determined before training begins.
当然，在训练开始之前，内核映射的顺序是任意的和确定的。

This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels.
这种响应规范化实现了一种受真实神经元类型启发的侧向抑制形式，在使用不同核计算的神经元输出之间形成对大活动的竞争。

The constants k, n, α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 10 4, and β = 0.75.
常数k、n、α和β是超参数，其值由验证集确定；我们使用k=2、n=5、α=104和β=0.75。

We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).
在某些层中应用ReLU非线性之后，我们应用了这种规范化（见第3.5节）。

This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity.
该方案与Jarrett等人[11]的局部对比度标准化方案有一些相似之处，但我们的方案更准确地称之为“亮度归一化”，因为我们没有减去平均活度。

Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively.
响应标准化使我们的前1名和前5名的错误率分别降低了1.4%和1.2%。

We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization3.
我们还验证了该方案在CIFAR-10数据集上的有效性：一个四层CNN在没有标准化的情况下实现了13%的测试错误率，在标准化3下达到了11%。

3.4 Overlapping Pooling

3.4重叠池

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map.
cnn中的汇聚层在同一个核映射图中总结相邻神经元组的输出。

Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]).
传统上，由相邻池单元汇总的邻域并不重叠（例如[17,11,4]）。

To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z × z centered at the location of the pooling unit.
更准确地说，池层可以被认为是由间隔s个像素的池单元组成的网格，每个单元汇总一个以池单元位置为中心的z×z大小的邻域。

If we set s = z, we obtain traditional local pooling as commonly employed in CNNs.
如果我们设置s=z，我们将得到CNNs中常用的传统本地池。

If we set s < z, we obtain overlapping pooling.
如果设s<z，则得到重叠池。

This is what we use throughout our network, with s = 2 and z = 3.
这是我们在整个网络中使用的，s=2和z=3。

This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions.
与产生等效尺寸输出的非重叠方案s=2，z=2相比，前1和前5的错误率分别降低了0.4%和0.3%。

We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.
我们通常在培训期间观察到，具有重叠池的模型更难过度拟合。

3.5 Overall Architecture

3.5总体架构

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully-connected.
现在我们准备描述一下CNN的整体架构。如图2所示，网络包含八层权重；前五层是卷积的，其余三层是全连接层。

The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels.
最后一个全连接层的输出被输入到1000路softmax，它产生1000个类标签的分布。

Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.
我们的网络将多项式logistic回归目标最大化，这相当于在预测分布下使正确标签的对数概率的跨训练平均数最大化。

The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2).
第二、第四和第五卷积层的内核只连接到位于同一GPU上的前一层的内核映射（见图2）。

The kernels of the third convolutional layer are connected to all kernel maps in the second layer.
第三卷积层的核与第二层的所有核映射相连接。

The neurons in the fully-connected layers are connected to all neurons in the previous layer.
完全连接层中的神经元与前一层的所有神经元相连。

Response-normalization layers follow the first and second convolutional layers.
响应规范化层遵循第一和第二卷积层。

Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer.
第3.4节中描述的最大池化层遵循响应规范化层和第五个卷积层。

The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.
每个卷积层和全连通层的输出均采用ReLU非线性。

The first convolutional layer filters the 224×224×3 input image with 96 kernels of size 11×11×3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map).
第一卷积层用96个11×11×3的核对224×224×3输入图像进行滤波，步长为4个像素（这是核映射图中相邻神经元感受野中心之间的距离）。

The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 × 5 × 48.
第二个卷积层将第一个卷积层的（响应规范化和池化）输出作为输入，并用大小为5×5×48的256个核对其进行滤波。

The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers.
第三、第四和第五卷积层彼此连接而没有任何中间的池或规范化层。

The third convolutional layer has 384 kernels of size 3 × 3 × 256 connected to the (normalized, pooled) outputs of the second convolutional layer.
第三卷积层有384个大小为3×3×256的核，连接到第二个卷积层的（标准化的、混合的）输出端。

The fourth convolutional layer has 384 kernels of size 3 × 3 × 192 , and the fifth convolutional layer has 256 kernels of size 3 × 3 × 192.
第四卷积层有384个3×3×192的核，第五个卷积层有256个3×3×192的核。

The fully-connected layers have 4096 neurons each.
完全连接的层每层有4096个神经元。

4 Reducing Overfitting

4减少过拟合

Our neural network architecture has 60 million parameters.
我们的神经网络结构有6000万个参数。

Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting.
虽然ILSVRC的1000个类使得每个训练示例对从图像到标签的映射施加10个位的约束，但这并不足以在不过拟合的情况下学习这么多参数。

Below, we describe the two primary ways in which we combat overfitting.
下面，我们将介绍我们对抗过拟合的两种主要方法。

4.1 Data Augmentation

4.1数据扩充

The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]).
减少图像数据过拟合的最简单和最常见的方法是使用保留标签的变换（例如[25,4,5]）人为地放大数据集。

We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk.
我们采用了两种不同的数据增强形式，这两种方式都允许从原始图像生成经过变换的图像，而无需将转换后的图像存储在磁盘上。

In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images.
在我们的实现中，转换后的图像是在CPU上用Python代码生成的，而GPU正在对前一批图像进行训练。

So these data augmentation schemes are, in effect, computationally free.
因此，这些数据扩充方案实际上是无需计算的。

The first form of data augmentation consists of generating image translations and horizontal reflections.
数据增强的第一种形式包括生成图像平移和水平反射。

We do this by extracting random 224 × 224 patches (and their horizontal reflections) from the 256×256 images and training our network on these extracted patches4.
我们通过从256×256幅图像中随机提取224×224个面片（及其水平反射），并在这些提取的斑块上训练我们的网络4。

This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly inter-dependent.
这使我们的训练集的规模增加了2048倍，尽管由此产生的训练示例当然是高度相互依赖的。

Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks.
如果没有这个方案，我们的网络将遭受严重的过度拟合，这将迫使我们使用更小的网络。

At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.
在测试时，网络通过提取五个224×224个面片（四角面片和中心面片）及其水平反射（因此总共十个面片）进行预测，并将网络的softmax层在这十个面片上所做的预测取平均值。

The second form of data augmentation consists of altering the intensities of the RGB channels in training images.
第二种形式的数据增强包括改变训练图像中RGB通道的强度。

Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set.
具体地说，我们在ImageNet训练集中对RGB像素值集执行PCA。

To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1.
对于每个训练图像，我们添加找到的主成分的倍数，其大小与相应的特征值成比例乘以从平均值为零，标准偏差为0.1的高斯随机变量。

Therefore to each RGB image pixel Ixy = [IR xy, IG xy, IB xy]T we add the following quantity:
因此，对于每个RGB图像像素$I_{xy}=[I_{xy}^R,I_{xy}^G,I_{xy}^B{]}^T$，我们添加以下数量：$$[P_1,P_2,P_3][[{\alpha }_1{\lambda }_1,{\alpha }_2{\lambda }_2,{\alpha }_3{\lambda }_3{]}^T$$

where p_i and λ_i are ith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable.
其中p_i和λ_i分别是RGB像素值的3×3协方差矩阵的第i个特征向量和特征值，α_i是上述随机变量。

Each αi is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn.
对于特定训练图像的所有像素，每个α_i只绘制一次，直到该图像再次用于训练，此时重新绘制该图像。

This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination.
该方案近似地捕捉到了自然图像的一个重要特性，即物体的同一性随光照强度和颜色的变化而变化。

This scheme reduces the top-1 error rate by over 1%.
该方案使 top-1 错误率降低了1%以上。

4.2 Dropout

4.2丢弃信号

Combining the predictions of many different models is a very successful way to reduce test errors[1, 3], but it appears to be too expensive for big neural networks that already take several days to train.
将许多不同模型的预测结合起来是减少测试误差的一种非常成功的方法[1,3]，但对于已经花费数天时间进行训练的大型神经网络来说，这似乎太贵了。

There is, however, a very efficient version of model combination that only costs about a factor of two during training.
然而，有一个非常有效的模型组合版本，在培训期间只需花费大约两倍的成本。

The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5.
最近引入的技术称为“dropout”[10]，包括将每个隐藏神经元的输出设为0，概率为0.5。

The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in back-propagation.
以这种方式“退出”的神经元不参与前向传递，也不参与反向传播。

So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights.
所以每次输入被提出时，神经网络都会对不同的结构进行采样，但所有这些结构都共享权重。

This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons.
这种技术减少了神经元复杂的共适应，因为一个神经元不能依赖于其他特定神经元的存在。

It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons.
因此，它被迫学习更健壮的特征，这些特征与其他神经元的许多不同的随机子集结合起来是有用的。

At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks.
在测试时，我们使用所有的神经元，但将它们的输出乘以0.5，这是一个合理的近似值，取指数型多个丢失网络产生的预测分布的几何平均值。

We use dropout in the first two fully-connected layers of Figure 2.
我们在图2中的前两个完全连接的层中使用dropout。

Without dropout, our network exhibits substantial overfitting.
在没有dropout的情况下，我们的网络表现出严重的过度适应。

Dropout roughly doubles the number of iterations required to converge.
Dropout大约是收敛所需迭代次数的两倍。

5 Details of learning

5学习细节

We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005.
我们使用随机梯度下降训练我们的模型，批量大小为128个例子，动量为0.9，权重衰减为0.0005。

We found that this small amount of weight decay was important for the model to learn.
我们发现这个小的重量衰减对模型的学习很重要。

In other words, weight decay here is not merely a regularizer: it reduces the model’s training error.
换句话说，这里的权重衰减不仅仅是一个正则化器：它减少了模型的训练误差。

The update rule for weight w was
权重w的更新规则是
$$v_{i+1}:=0.9\cdot v_i-0.0005\cdot \epsilon \cdot w_i-\epsilon \cdot {⟨\frac{\partial L}{\partial w}|w_i⟩}_{D_i}$$$$w_{i+1}:=w_i+v_{i+1}$$
where i is the iteration index, v is the momentum variable, ε is the learning rate, and$⟨\frac{\partial L}{\partial w}|w_i⟩$ is the average over the ith batch Di of the derivative of the objective with respect to w, evaluated at wi.
其中i是迭代指数，v是动量变量，ε是学习率，以及$⟨\frac{\partial L}{\partial w}|w_i⟩$是第i批中目标衍生物相对于w的平均值，在w_i处进行评估。

We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01.
我们用标准差为0.01的零均值高斯分布初始化每个层的权重。

We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1.
我们用常数1初始化第二、第四和第五卷积层以及完全连接的隐藏层中的神经元偏差。

This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs.
这种初始化通过向ReLUs提供积极的输入加快了学习的早期阶段。

We initialized the neuron biases in the remaining layers with the constant 0.
我们用常数0初始化剩余层中的神经元偏差。

We used an equal learning rate for all layers, which we adjusted manually throughout training.
我们对所有层使用相同的学习率，在整个培训过程中我们手动调整。

The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate.
我们遵循的启发式方法是，当验证错误率不再随当前学习率提高时，将学习率除以10。

The learning rate was initialized at 0.01 and reduced three times prior to termination.
学习率初始化为0.01，终止前降低三倍。

We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs.
我们通过120万张图像的训练集训练了网络大约90个周期，在两个NVIDIA GTX 580 3GB GPUs上花了5到6天的时间。

6 Results

6结果

Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%.
表1总结了我们在ILSVRC-2010上的结果。我们的网络实现了前1名和前5名测试集错误率分别为37.5%和17.0%。

The best performance achieved during the ILSVRC-2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best published results are 45.7% and 25.7% with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].
在ILSVRC-2010竞赛中取得的最佳性能分别为47.1%和28.2%，采用一种方法平均化六种不同特征训练的稀疏编码模型产生的预测值[2]，从那时起，最好的发布结果分别为45.7%和25.7%，使用一种方法，即使用两种密集采样特征计算出的Fisher向量（fv）训练的两个分类器的预测值的平均值[24]。

We also entered our model in the ILSVRC-2012 competition and report our results in Table 2.
我们还在ILSVRC-2012竞赛中输入了我们的模型，并在表2中报告了我们的结果。

Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried.
由于ILSVRC-2012测试集标签不公开，我们无法报告我们尝试的所有模型的测试错误率。

In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than 0.1%(see Table 2).
在本段的其余部分中，我们交替使用验证和测试错误率，因为根据我们的经验，它们之间的差异不超过0.1%（见表2）。

The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the predictions of five similar CNNs gives an error rate of 16.4%.
本文描述的CNN的前5位错误率为18.2%。对5个相似的cnn进行平均预测，误差率为16.4%。

Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of 16.6%.
训练一个CNN，在最后一个池层上增加一个第六个卷积层，对整个ImageNet Fall 2011发布版（15M图像，22K类别）进行分类，然后在ILSVRC-2012上对其进行“微调”，错误率为16.6%。

Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of 15.3%.
对2011年秋季发布的两个CNN的预测值进行平均，这两个CNN在上述五个CNN中进行了预训练，其预测误差率为15.3%。

The second-best contest entry achieved an error rate of 26.2% with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features [7].
第二名最佳竞赛条目的错误率为26.2%，其方法是将训练在基于不同类型密集采样特征的FVs上的几个分类器的预测平均化[7]。

Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images.
最后，我们还报告了2009年秋季版本的ImageNet的错误率，该版本包含10184个类别和890万个图像。

On this dataset we follow the convention in the literature of using half of the images for training and half for testing.
在这个数据集上，我们遵循文献中的惯例，使用一半的图像进行训练，另一半用于测试。

Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably.
由于没有已建立的测试集，我们的分割必然不同于先前作者使用的分割，但这不会对结果产生明显的影响。

Our top-1 and top-5 error rates on this dataset are 67.4% and 40.9%, attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer.
我们在这个数据集上的前1名和前5名的错误率分别为67.4%和40.9%，通过上述网络实现，但是在最后一个池层上有一个额外的第六个卷积层。

The best published results on this dataset are 78.1% and 60.9% [19].
在这个数据集上发表的最佳结果分别是78.1%和60.9%[19]。

6.1 Qualitative Evaluations

6.1定性评价

Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers.
图3显示了由网络的两个数据连接层学习的卷积核。

The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs.
网络已经学习了各种频率和方向选择性核，以及各种颜色的斑点。

Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5.
请注意这两个gpu所显示的专门化，这是第3.5节中描述的受限连接的结果。

The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific.
gpu1上的内核基本上是颜色不可知的，而gpu2上的内核主要是颜色特定的。

This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).
这种专门化发生在每次运行期间，并且独立于任何特定的随机权重初始化（对gpu进行模化重新编号）。

Figure 3: 96 convolutional kernels of size 11×11×3 learned by the first convolutional layer on the 224×224×3 input images.
图3:224×224×3输入图像上第一卷积层学习的96个11×11×3卷积核。
The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details.
前48个内核在gpu1上学习，而后48个内核在gpu2上学习。详见第6.1节。

In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images.
在图4的左侧面板中，我们通过计算8张测试图像的前5位预测，定性地评估了网络所学的知识。

Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model.
图4：（左）8个ILSVRC-2010测试图像和我们模型认为最有可能的五个标签。
The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5).
正确的标签写在每个图像下面，并且分配给正确标签的概率也用红色条显示（如果它正好在前5位）。
(Right) Five ILSVRC-2010 test images in the first column.
（右）第一列中的五幅ILSVRC-2010测试图像。
The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.
剩下的列显示了六个训练图像，它们在最后一个隐藏层中生成特征向量，与测试图像的特征向量的欧几里德距离最小。

Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net.
请注意，即使是偏离中心的对象，如左上角的斜接体，也可以被网络识别。

Most of the top-5 labels appear reasonable.
前五名中的大多数似乎都是合理的。

For example, only other types of cat are considered plausible labels for the leopard.
例如，只有其他种类的猫才被认为是豹的合理标签。

In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph.
在某些情况下（格栅，樱桃），有真正的含糊不清的预定焦点的照片。

Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer.
探索网络视觉知识的另一种方法是考虑图像在最后4096维隐藏层引起的特征激活。

If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar.
如果两幅图像产生的特征激活向量具有很小的欧几里德分离，我们可以说，较高层次的神经网络认为他们是相似的。

Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure.
图4显示了来自测试集的五个图像和来自训练集的六个图像，这些图像与此度量中的每一个最相似。

Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column.
注意，在像素级别，检索到的训练图像在L2中通常与第一列中的查询图像不太接近。

For example, the retrieved dogs and elephants appear in a variety of poses.
例如，被找回的狗和大象以各种姿势出现。

We present the results for many more test images in the supplementary material.
我们在补充材料中展示了更多测试图像的结果。

Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes.
利用两个4096维实值向量之间的欧几里德距离来计算相似度是低效的，但是可以通过训练一个自动编码器将这些向量压缩成短二进制码来实现。

This should produce a much better image retrieval method than applying autoencoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.
这将产生比对原始像素应用自动编码器更好的图像检索方法[14]，后者不使用图像标签，因此倾向于检索具有相似边缘模式的图像，无论它们在语义上是否相似。

7 Discussion

7讨论

Our results show that a large, deep convolutional neural network is capable of achieving record-breaking results on a highly challenging dataset using purely supervised learning.
我们的结果表明，一个大的，深卷积神经网络能够在一个具有高度挑战性的数据集上使用纯监督学习获得破纪录的结果。

It is notable that our network’s performance degrades if a single convolutional layer is removed.
值得注意的是，如果去掉单个卷积层，我们的网络性能会下降。

For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network.
例如，删除任何中间层都会导致网络的 top-1 性能损失约2%。

So the depth really is important for achieving our results.
所以深度对我们的结果很重要。

To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data.
为了简化我们的实验，我们没有使用任何无监督的预训练，即使我们期望它会有所帮助，特别是如果我们获得足够的计算能力来显著地增加网络的规模，而不获得相应增加的标记数据量。

Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system.
到目前为止，我们的结果已经有所改善，因为我们已经扩大了我们的网络并训练了它更长的时间，但我们仍然有许多数量级的工作要去匹配人类视觉系统的时间-时间路径。

Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.
最后，我们希望在视频序列上使用非常大和很深的卷积网络，其中时间结构提供了非常有用的信息，而这些信息在静态图像中是缺失的或不太明显的。