LMDB概念
接上一篇,将SSDa caffen框架代码编译通过后y,接下来就可以利用其脚本将标注好的VOC格式数据集转换成LMDB格式,以便开始SSD模型训练。
LMDB是Lightning Memory-Mapped Database的缩写.即闪电般的内存映射数据库。很多文章把它错误写成IMDB。LMDB文件结构简单,里含一个数据文件和一个锁文件。数据文件包含图片数据,访问简单,IO开销很小,所以读写速度很快。锁文件是用来对多线程读写数据进行保护.
VOC数据集准备
创建新的目录来存放自己标注好的数据集
cd xxx/caffe_ssd/data #caffe_ssd是 SSD caffe框架得根目录
sudo mkdir VOCdevkit
sudo mkdir VOCdevkit/VOC201909
从其他已有的数据集里面拷贝来三个文件,它们就是用来将VOC数据集转换成IMDB格式。当然在正式运行它们之前,必须做一些修改,后面会具体提到。
cd xxx/caffe_ssd/data/VOCdevkit/VOC201909
sudo cp ../../VOC0712/create_list.sh ./
sudo cp ../../VOC0712/create_data.sh ./
sudo cp ../../VOC0712/labelmap_voc.prototxt ./
从py-faster-rcnn项目中拷贝来VOC数据集
#当前目录就是VOC201909
sudo cp -a xxx/py-faster-rcnn/data/VOCdevkit2007/VOC2007/JPEGImages ./
sudo cp -a xxx/py-faster-rcnn/data/VOCdevkit2007/VOC2007/Annotations ./
sudo cp -a xxx/py-faster-rcnn/data/VOCdevkit2007/VOC2007/ImageSets ./
修改三个脚本文件
数据集以及脚本工具准备好后,就可以开始进行转换了。但是如前所述,转换前还需要对这三个脚本进行相应的修改。
对于labelmap_voc.prototxt的修改,则是自己数据集的类型名字添加进去,该删除的要删除,最后的类型个数和名称要完全和自己数据集的相等。
遇到的问题
/work/xxx/caffe_ssd/data/VOCdevkit/VOC201909$ sudo sh create_data.sh
Traceback (most recent call last):
File "/work/xxx/caffe_ssd/scripts/create_annoset.py", line 7, in <module>
from caffe.proto import caffe_pb2ImportError: No module named caffe.protoTraceback
(most recent call last): File "/work/xxx/caffe_ssd/scripts/create_annoset.py", line 7, in <module>
from caffe.proto import caffe_pb2ImportError: No module named caffe.proto
解决办法添加下面一行代码:
import sys
//添加这一行!!!
sys.path.insert(0, '/work/xxx/caffe_ssd/python')
from caffe.proto import caffe_pb2
另外一个错误如下,主要是因为标注文件xml里面图片width或height为0或目标标注的xmax或ymax >= width或ymax。
I0918 16:13:14.338071 4228 convert_annoset.cpp:195] Processed 1000 files.
W0918 16:13:19.104089 4228 io.cpp:332] /work/xxx/caffe_ssd/data/VOCdevkit/VOC201909/Annotations/beacon_20190604_000178.xml bounding box exceeds image boundary.
I0918 16:13:25.096086 4228 convert_annoset.cpp:195] Processed 2000 files.
W0918 16:13:27.803169 4228 io.cpp:332] /work/xxx/caffe_ssd/data/VOCdevkit/VOC201909/Annotations/beacon_20190604_000223.xml bounding box exceeds image boundary.
I0918 16:13:36.512428 4228 convert_annoset.cpp:195] Processed 3000 files.
W0918 16:13:40.616504 4228 io.cpp:332] /work/xxx/caffe_ssd/data/VOCdevkit/VOC201909/Annotations/beacon_20190612_000029.xml bounding box exceeds image boundary.
最终的输出结果
/work/xxx/caffe_ssd/data/VOCdevkit/VOC201909$ sudo sh create_list.sh
Create list for VOC201909 trainval...
Create list for VOC201909 test...I0918 15:28:27.385476 3484
get_image_size.cpp:61] A total of 12 images.I0918 15:28:27.461084 3484 get_image_size.cpp:105] Processed 12 files.
/work/xxx/caffe_ssd/data/VOCdevkit/VOC201909$ sudo sh create_data.sh
... ...
/work/xxx/caffe_ssd/data/VOCdevkit/VOC201909/trainval.txt
/work/xxx/caffe_ssd/data/VOCdevkit/VOC201909/lmdb/VOC201909_trainval_lmdbI0918 16:17:09.543891 4281 convert_annoset.cpp:122] A total of 3544 images.I0918 16:17:09.565481 4281 db_lmdb.cpp:35] Opened lmdb
/work/xxx/caffe_ssd/data/VOCdevkit/VOC201909/lmdb/VOC201909_trainval_lmdbI0918
16:17:20.756366 4281 convert_annoset.cpp:195] Processed 1000 files.I0918 16:17:31.489339
4281 convert_annoset.cpp:195] Processed 2000 files.I0918 16:17:42.797315 4281
convert_annoset.cpp:195] Processed 3000 files.I0918 16:17:50.096717 4281
convert_annoset.cpp:201] Processed 3544 files.