使用lmdb对ImageNet数据进行快速读取（Pytorch）

reference：

1. https://github.com/npuichigo/pytorch_lmdb_dataset
2. https://pytorch.org/docs/stable/_modules/torchvision/datasets/folder.html#ImageFolder
3. https://github.com/Lyken17/Efficient-PyTorch
4. https://github.com/xunge/pytorch_lmdb_imagenet

选择lmdb的原因
The default combination datasets.ImageFolder + data.DataLoader is not enough for large scale classification. According to my experience, even I upgrade to Samsung 960 Pro (read 3.5 GB/s, write 2.0 GB/s), whole training pipeline still suffers at disk I/O.

The reason causing is the slow reading of discountiuous small chunks. To optimize, we need to dump small JPEG images into a large binary file. TensorFlow has its own TFRecord and MXNet uses recordIO. Beside these two, there are other options like hdf5, pth, n5, lmdb etc. Here I choose lmdb because

TFRecord is a private protocal which is hard to hack into. RecordIO’s documentation is confusing and do not provide a clean python API.
hdf5 pth n5, though with a straightforward json-like API, require to put the whole file into memory. This is not practicle when you play with large dataset like imagenet.

转换流程

1. 下载代码
   git clone https://github.com/xunge/pytorch_lmdb_imagenet
2. 修改folder2lmdb.py里的train和val路径，然后运行代码

   python folder2lmdb.py
   

3. 修改dataloader，开始训练。在机械硬盘上，相比于pytorch自带的数据读取接口，使用lmdb的data I/O速度提高了大概十倍左右。