Epoch,Iterations and Batch_Size

最近复现论文的时候遇到的。原文说要把Alexnet训练800k个iteration，然而这个iteration具体是啥呢？网上下载下来的代码一般只给你指定一个args.Epoch，那么Iteration和batchsize和epoch数又有什么关系呢？
拿ImageNet的训练举例，Imagenet包含120w左右的训练集。一个epoch就是指模型在所有120w个数据上经过一次训练的过程。但实际上考虑到内存的限制，必须把120w的训练集分成小batch进行训练。一般地，设置batchsize为256，那么显然，训练经过大概5000个batch相当于一个epoch。这个5000就是一个epoch所需要的iteration次数。
对于论文所说的256batchsize下的800k个iteration，就是差不多$\frac{800k}{5k}$次，大概160epoch。

We illustrate this issue by an example of ImageNet, which consist of 1200k images for training. A epoch denotes the procedure for the model to process all the images in dataset for one time. However, considering the computing memory, we must exploit the batchsize SGD to train. Therefore, when we set the batchsize to be 256,then it is obvious that it will cost approximately 5000 times for all the batches to go through the whole database for one epoch. This 5000 is the iterations number for one epoch.
So as for the original problems which mentioned 800k iterations with batchsize being 256, the epoch will be about 800k/5k times, i.e. 160 epochs. Problem solved.