Ubuntu20.04 编译 darknet 训练yolov3-tiny 记录

Ubuntu20.4 编译 darknet 训练yolov3-tiny 记录

帮学弟训练配的，编译踩了些坑故记录一下。

配置及环境

• GPU：GTX1050TI
• CUDA/CUDNN：11.2

1. git darknet

git clone https://github.com/pjreddie/darknet

2.修改makefile文件

GPU=1 #如果使用GPU设置为1，CPU设置为0
CUDNN=0  #如果使用CUDNN设置为1，否则为0
OPENCV=0 #如果调用摄像头，还需要设置OPENCV为1，否则为0
OPENMP=0  #如果使用OPENMP设置为1，否则为0
DEBUG=0  #如果使用DEBUG设置为1，否则为0

3.编译

make

坑

1. 尝试CUDNN=1的时候报错[obj/convolutional_layer.o] Error 1，先根据博客编译darknet报错：[obj/convolutional_layer.o] Error 1 或者[obj/convolutional_kernels.o] Error 1尝试修改NVCC和其他一切和cuda有关的路径，但是还是报错[obj/convolutional_kernels.o] Error 1]。

2. 这时候参考这篇博客’obj/convolutional_kernels.o’ failed make: *** [obj/convolutional_kernels.o] Error 1配置etc/profile，但是仍然报同样的错误。

3. 这时候注意到这个错误上面还有一句nvcc fatal : Unsupported gpu architecture ‘compute_30‘，于是参考这篇博客nvcc fatal : Unsupported gpu architecture ‘compute_30‘注释掉-gencode arch=compute_30,code=sm_30后编译通过

训练

文件夹结构

---darknet
    |---mydataset
        |---Annotations
        |---JPEGImages
        |---ImageSets
            |---Main
        |---test.py
    |---voc_label.py

数据集划分/数据集格式转换

运行test.py，在目录ImageSets/Main下生成test.txt,train.txt,trainval.txt,val.txt
再运行voc_label.py，在目录mydataset下生成文件夹labels以及mydataset_train.txt,mydataset_trainval.txt

修改文件

1. 修改data/config下的voc.data,yolov3-tiny.cfg

• voc.data

classes= 1
train  = /home/{path}/darknet/mydataset/mydataset_train.txt
valid  = /home/{path}/darknet/mydataset/mydataset_trainval.txt
names = data/voc.names
backup = backup

• yolov3-tiny.cfg
  搜索’yolo’，一共有两项

[convolutional]
size=1
stride=1
pad=1
filters=18 #(classes+5)*3
activation=linear



[yolo]
mask = 3,4,5
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=1 #1 class
num=6
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1

修改另外一个也是同理

[convolutional]
size=1
stride=1
pad=1
filters=18
activation=linear

[yolo]
mask = 0,1,2
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=1
num=6
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1

训练部分

[net]
# Testing 因为是训练所以要把test的部分给注释掉，
# batch=1
# subdivisions=1
# Training
batch=32 #根据电脑配置决定，配置低就往小调
subdivisions=16 #也是根据配置决定，配置低往大调
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.001
burn_in=1000
max_batches = 10000 #最大迭代次数
policy=steps
steps=8000,9000 #一般来说是最大迭代次数的80%和90%
scales=.1,.1

参数具体含义可以参考这篇博客yolo v3配置文件说明模型配置文件——cfg/yolov3-voc.cfg
2. 修改data/voc.names
修改为类别名称，比如我这里为标签sidewalk

下载权重文件/提取预训练模型

首先确保终端当前目录为darknet

wget https://pjreddie.com/media/files/yolov3-tiny.weights

./darknet partial cfg/yolov3-tiny.cfg yolov3-tiny.weights yolov3-tiny.conv.15 15

调用GPU训练

sudo ./darknet detector train cfg/voc.data cfg/yolov3-tiny.cfg yolov3-tiny.conv.15 -gpus 0

代码自动每隔100轮保存一次权重文件（到900轮），以及保存第1w轮和最后一轮迭代的权重文件。
以最后一轮终端输出的信息为例：

10000: 0.093652, 0.089722 avg, 0.000010 rate, 1.899265 seconds, 320000 images

上面数据只需要关心前三个，依次表示迭代次数、loss、avg loss，一般来说loss越小越好
其他参数详细可以参考这篇博客目标检测：YOLOv3: 训练自己的数据

测试

./darknet detector test cfg/voc.data cfg/yolov3-tiny.cfg backup/yolov3-tiny_final.weights data/sidewalk.jpg

---

代码
test.py

import os
import random

trainval_percent = 0.1
train_percent = 0.9
xmlfilepath = 'Annotations'
txtsavepath = 'ImageSets\Main'
total_xml = os.listdir(xmlfilepath)
num = len(total_xml)
list = range(num)
tv = int(num * trainval_percent)
tr = int(tv * train_percent)
trainval = random.sample(list, tv)
train = random.sample(trainval, tr)

ftrainval = open('ImageSets/Main/trainval.txt', 'w')
ftest = open('ImageSets/Main/test.txt', 'w')
ftrain = open('ImageSets/Main/train.txt', 'w')
fval = open('ImageSets/Main/val.txt', 'w')

for i in list:
    name = total_xml[i][:-4] + '\n'
    if i in trainval:
        ftrainval.write(name)
        if i in train:
            ftest.write(name)
        else:
            fval.write(name)
    else:
        ftrain.write(name)

ftrainval.close()
ftrain.close()
fval.close()
ftest.close()

voc_label.py

import xml.etree.ElementTree as ET
import pickle
import os
from os import listdir, getcwd
from os.path import join
 
#源代码sets=[('2012', 'train'), ('2012', 'val'), ('2007', 'train'), ('2007', 'val'), ('2007', 'test')]
sets=[('mydataset', 'train'),('mydataset', 'trainval')]  # 改成自己建立的myData
 
classes = ["sidewalk"] # 改成自己的类别
 
def convert(size, box):
    dw = 1./(size[0])
    dh = 1./(size[1])
    x = (box[0] + box[1])/2.0 - 1
    y = (box[2] + box[3])/2.0 - 1
    w = box[1] - box[0]
    h = box[3] - box[2]
    x = x*dw
    w = w*dw
    y = y*dh
    h = h*dh
    return (x,y,w,h)
 
def convert_annotation(year, image_id):
    in_file = open('mydataset/Annotations/%s.xml'%(image_id))  # 源代码VOCdevkit/VOC%s/Annotations/%s.xml
    out_file = open('mydataset/labels/%s.txt'%(image_id), 'w')  # 源代码VOCdevkit/VOC%s/labels/%s.txt
    tree=ET.parse(in_file)
    root = tree.getroot()
    size = root.find('size')
    w = int(size.find('width').text)
    h = int(size.find('height').text)
 
    for obj in root.iter('object'):
        difficult = obj.find('difficult').text
        cls = obj.find('name').text
        if cls not in classes or int(difficult)==1:
            continue
        cls_id = classes.index(cls)
        xmlbox = obj.find('bndbox')
        b = (float(xmlbox.find('xmin').text), float(xmlbox.find('xmax').text), float(xmlbox.find('ymin').text), float(xmlbox.find('ymax').text))
        bb = convert((w,h), b)
        out_file.write(str(cls_id) + " " + " ".join([str(a) for a in bb]) + '\n')
 
wd = getcwd()
 
for year, image_set in sets:
    if not os.path.exists('mydataset/labels/'):  # 改成自己建立的myData
        os.makedirs('mydataset/labels/')
    image_ids = open('mydataset/ImageSets/Main/%s.txt'%(image_set)).read().strip().split()
    list_file = open('mydataset/%s_%s.txt'%(year, image_set), 'w')
    for image_id in image_ids:
        list_file.write('%s/mydataset/JPEGImages/%s.jpg\n'%(wd, image_id))
        convert_annotation(year, image_id)
    list_file.close()