基于PP-YOLOE+的水下生物目标检测+部署
项目链接【https://aistudio.baidu.com/aistudio/projectdetail/4647849?contributionType=1】
1 项目背景
水下目标检测旨在对水下场景中的物体进行定位和识别。这项研究由于在海洋学、水下导航等领域的广泛应用而引起了持续的关注。但是，由于复杂的水下环境和光照条件，这仍然是一项艰巨的任务。
基于深度学习的物体检测系统已在各种应用中表现出较好的性能，但在处理水下目标检测方面仍然感到不足，主要有原因是：可用的水下目标检测数据集稀少，实际应用中的水下场景的图像杂乱无章，并且水下环境中的目标物体通常很小，而当前基于深度学习的目标检测器通常无法有效地检测小物体，或者对小目标物体的检测性能较差。同时，在水下场景中，与波长有关的吸收和散射问题大大降低了水下图像的质量，从而导致了可见度损失，弱对比度和颜色变化等问题。
Al+水下勘探是一个新兴领域，目前专门用于水下研究工作的解决方案不多，高质量的数据集更是弥足珍贵。使用 PP-YOLOE+ 来推进水下目标检测的进步，从而使得水下机器人等设备能够更加智能化，提高海底资源勘探等方面的效率。而水下机器人又反哺出高质量的水下目标检测数据集，推动 Al+水下勘探的发展。
2 方案选择
2.1 问题与挑战
深度学习中，数据往往决定了性能的上限，算法只是不断地逼近上限。尽管基于深度学习的方法在标准的目标检测中取得了可喜的性能。水下目标检测仍具有以下几点挑战：
（1）水下场景的实际应用中目标通常很小，含有大量的小目标；
（2）水下数据集和实际应用中的图像通常是模糊的，图像中具有异构的噪声。
2.2 方案选择
因此，针对以上所述的背景和水中目标检测所遇到的挑战，本项目将选用 PP-YOLOE+ 这一基于飞桨云边一体高精度模型PP-YOLOE迭代优化升级的版本。针对性的解决以上问题。
2.3 模型特点介绍
PP-YOLOE+ 具有如下特点：
-
超强性能
-
训练收敛加速
-
下游任务泛化性显著提升
-
高性能部署能力
3 环境配置
3.1 环境准备
PaddlePaddle >= 2.3.2
Python == 3.7
3.2 环境安装
%cd /home/aistudio/work/
# gitee 国内下载比较快
!git clone https://gitee.com/paddlepaddle/PaddleDetection.git -b develop
# github
# !git clone https://github.com/PaddlePaddle/PaddleDetection.git -b develop
# 环境安装
%cd /home/aistudio/work/
# gitee 国内下载比较快
# !git clone https://gitee.com/paddlepaddle/PaddleDetection.git -b develop
%cd PaddleDetection/
!pip install -r requirements.txt > /dev/null
/home/aistudio/work
/home/aistudio/work/PaddleDetection
4 数据集预处理
4.1 数据集介绍
本项目选用数据集来源于水下目标检测算法赛（注：数据由鹏城实验室提供），训练集是5543张 jpg 格式的水下光学图像与对应标注结果构成，其中主要有海参、海胆、扇贝、海星四种目标。
4.1.1 标签类别
| supercategory | id | name |
|---|---|---|
| component | 1 | echinus |
| component | 2 | holothurian |
| component | 3 | scallop |
| component | 4 | starfish |
| component | 5 | waterweeds |
4.1.2 图像分辨率
| 长度 | 宽度 | 图片数量 |
|---|---|---|
| 704 | 576 | 38 |
| 1920 | 1080 | 596 |
| 3840 | 2160 | 1712 |
| 720 | 405 | 3153 |
| 586 | Text | 44 |
4.2 数据集处理
4.2.1 数据集解压
# 1.数据集解压
!unzip data/data172711/fish.zip > /dev/null
!mv ./fish ./work/PaddleDetection/dataset
4.2.2 voc2coco
%cd work/PaddleDetection/
/home/aistudio/work/PaddleDetection
# 2.voc2coco
import argparse
import glob
import json
import os
import os.path as osp
import sys
import shutil
import numpy as np
import PIL.ImageDraw
import xml.dom.minidom as xmldom
import cv2
label_to_num = {}
categories_list = []
labels_list = []
class MyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, np.integer):
return int(obj)
elif isinstance(obj, np.floating):
return float(obj)
elif isinstance(obj, np.ndarray):
return obj.tolist()
else:
return super(MyEncoder, self).default(obj)
def getbbox(self, points):
polygons = points
mask = self.polygons_to_mask([self.height, self.width], polygons)
return self.mask2box(mask)
def images_labelme(data, num):
image = {}
image['height'] = data['imageHeight']
image['width'] = data['imageWidth']
image['id'] = num + 1
image['file_name'] = data['imagePath'].split('/')[-1]
return image
def images_cityscape(num, img_file, w, h):
image = {}
image['height'] = h
image['width'] = w
image['id'] = num + 1
image['file_name'] = img_file
return image
def categories(label, labels_list):
category = {}
category['supercategory'] = 'component'
category['id'] = len(labels_list) + 1
category['name'] = label
return category
def annotations_rectangle(points, label, image_num, object_num, label_to_num):
annotation = {}
seg_points = np.asarray(points).copy()
annotation['segmentation'] = [list(seg_points.flatten())]
annotation['iscrowd'] = 0
annotation['image_id'] = image_num + 1
annotation['bbox'] = list(
map(float, [
points[0][0], points[0][1], points[1][0] - points[0][0], points[1][
1] - points[0][1]
]))
annotation['area'] = annotation['bbox'][2] * annotation['bbox'][3]
annotation['category_id'] = label_to_num[label]
annotation['id'] = object_num + 1
return annotation
def annotations_polygon(height, width, points, label, image_num, object_num,
label_to_num):
annotation = {}
annotation['segmentation'] = [list(np.asarray(points).flatten())]
annotation['iscrowd'] = 0
annotation['image_id'] = image_num + 1
annotation['bbox'] = list(map(float, get_bbox(height, width, points)))
annotation['area'] = annotation['bbox'][2] * annotation['bbox'][3]
annotation['category_id'] = label_to_num[label]
annotation['id'] = object_num + 1
return annotation
def get_bbox(height, width, points):
polygons = points
mask = np.zeros([height, width], dtype=np.uint8)
mask = PIL.Image.fromarray(mask)
xy = list(map(tuple, polygons))
PIL.ImageDraw.Draw(mask).polygon(xy=xy, outline=1, fill=1)
mask = np.array(mask, dtype=bool)
index = np.argwhere(mask == 1)
rows = index[:, 0]
clos = index[:, 1]
left_top_r = np.min(rows)
left_top_c = np.min(clos)
right_bottom_r = np.max(rows)
right_bottom_c = np.max(clos)
return [
left_top_c, left_top_r, right_bottom_c - left_top_c,
right_bottom_r - left_top_r
]
def deal_json(output_dir, input_dir, img_list):
data_coco = {}
images_list = []
annotations_list = []
image_num = -1
object_num = -1
# labels_list =[]
for img_file in img_list:
img = cv2.imread(osp.join('dataset/fish/Images', img_file))
w = img.shape[1]
h = img.shape[0]
img_label = img_file.split('.')[0]
label_file = osp.join('dataset/fish/Annotations', img_label + '.xml')
# print('Generating dataset from:', label_file)
image_num = image_num + 1
xml_file = xmldom.parse(label_file)
eles = xml_file.documentElement
images_list.append(images_cityscape(image_num, img_file, w, h))
for i in range(len(eles.getElementsByTagName('name'))):
label = eles.getElementsByTagName('name')[i].firstChild.data
# if label == 'starfish':
# continue
object_num = object_num + 1
# print(label)
if label not in labels_list:
# print(label)
categories_list.append(categories(label, labels_list))
labels_list.append(label)
label_to_num[label] = len(labels_list)
points = []
xmin = int(eles.getElementsByTagName('xmin')[i].firstChild.data)
ymin = int(eles.getElementsByTagName('ymin')[i].firstChild.data)
xmax = int(eles.getElementsByTagName('xmax')[i].firstChild.data)
ymax = int(eles.getElementsByTagName('ymax')[i].firstChild.data)
# print(xmin,ymin,xmax,ymax)
# if xmin > 2000:
# print(label_file)
points.append([xmin, ymin])
points.append([xmax, ymax])
annotations_list.append(
annotations_rectangle(points, label, image_num,
object_num, label_to_num))
data_coco['images'] = images_list
data_coco['categories'] = categories_list
data_coco['annotations'] = annotations_list
# print(labels_list)
return data_coco
import os.path as osp
import glob
import os
import shutil
train_img = 'dataset/fish/Images'
train_box = 'dataset/fish/Annotations'
# Allocate the dataset.
total_num = len(glob.glob(osp.join(train_img, '*.jpg')))
train_num = int(total_num * 0.8)
os.makedirs('data/cocome' + '/train')
val_num = total_num - train_num
os.makedirs('data/cocome' + '/val')
count = 1
train_list = []
val_list = []
for img_name in os.listdir(train_img):
if count <= train_num:
if osp.exists('data/cocome' + '/train/'):
shutil.copyfile(
osp.join(train_img, img_name),
osp.join('data/cocome' + '/train/', img_name))
train_list.append(img_name)
else:
if count <= train_num + val_num:
if osp.exists('data/cocome' + '/val/'):
shutil.copyfile(
osp.join(train_img, img_name),
osp.join('data/cocome' + '/val/', img_name))
val_list.append(img_name)
count = count + 1
if not os.path.exists('data/cocome' + '/annotations'):
os.makedirs('data/cocome' + '/annotations')
train_data_coco = deal_json(
'data/cocome' + '/train', train_img ,train_list)
train_json_path = osp.join('data/cocome' + '/annotations', 'instance_train.json')
json.dump(
train_data_coco,
open(train_json_path, 'w'),
indent=4,
cls=MyEncoder)
val_data_coco = deal_json('data/cocome' + '/val', train_img, val_list)
val_json_path = osp.join('data/cocome' + '/annotations', 'instance_val.json')
json.dump(val_data_coco, open(val_json_path, 'w'), indent=4, cls=MyEncoder)
4.2.3 将转换后的数据集移动至dataset/fish
# 3.将转换后的数据集移动至dataset/fish
!mv data/cocome/* dataset/fish/
4.2.4 将自己的数据集的路径进行修改和配置
metric: COCO
num_classes: 5
TrainDataset:
!COCODataSet
image_dir: train
anno_path: annotations/instance_train.json
dataset_dir: dataset/fish
data_fields: ['image', 'gt_bbox', 'gt_class', 'is_crowd']
EvalDataset:
!COCODataSet
image_dir: val
anno_path: annotations/instance_val.json
dataset_dir: dataset/fish
TestDataset:
!ImageFolder
anno_path: label_list.txt # also support txt (like VOC's label_list.txt)
dataset_dir: dataset/fish # if set, anno_path will be 'dataset_dir/anno_path'
5 模型训练
使用 PP-YOLOE+ 进行训练:
在./configs/ppyoloe/ppyoloe_plus_crn_s_80e_coco.yml中提供了基于PP-YOLOE+训练该场景的配置，训练脚本如下：
!python tools/train.py -c configs/ppyoloe/ppyoloe_plus_crn_s_80e_coco.yml -o LearningRate.base_lr=0.000125 -o snapshot_epoch=1 -o worker_num=1 --eval
6 模型评估
在训练模型以后，我们可以通过运行评估命令来得到模型的精度，以确认训练的效果。评估可以参考以下命令执行。
这里使用了我们已经训练好的模型。如希望使用自己训练的模型，请对应将weights=后的值更改为对应模型.pdparams文件的存储路径。
# 模型评估
!python tools/eval.py -c configs/ppyoloe/ppyoloe_plus_crn_s_80e_coco.yml -o weights=output/ppyoloe_plus_crn_s_80e_coco/best_model.pdparams
Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly
W1016 22:54:41.591436 22455 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W1016 22:54:41.598824 22455 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[10/16 22:54:43] ppdet.utils.checkpoint INFO: Finish loading model weights: output/ppyoloe_plus_crn_s_80e_coco/best_model.pdparams
[10/16 22:54:44] ppdet.engine INFO: Eval iter: 0
[10/16 22:54:55] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
[10/16 22:54:55] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=0.50s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=3.09s).
Accumulating evaluation results...
DONE (t=0.43s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.278
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.541
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.267
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.237
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.275
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.304
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.160
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.436
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.543
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.514
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.509
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.572
[10/16 22:54:59] ppdet.engine INFO: Total sample number: 200, averge FPS: 19.422192298181745
7 模型预测
这里我们将训练好的模型对着整个数据集的验证集来一番批量预测，这些预测结果会记录在VisualDL中，可以很方便地与原图对比，观察预测效果。
!python tools/infer.py -c configs/ppyoloe/ppyoloe_plus_crn_s_80e_coco.yml -o weights=output/ppyoloe_plus_crn_s_80e_coco/best_model --infer_dir ./dataset/fish/val --use_vdl=True --vdl_log_dir=./output/image --output_dir ./output/results
8 模型导出
.pdparams只包括了模型的参数数据，实际部署还需要执行导出步骤。导出步骤可以参考下面列举的步骤：
注意，这里使用了我们已经训练好的模型。如希望使用自己训练的模型，请对应将weights=后的值更改为对应模型.pdparams文件的存储路径。如果没有指定–output_dir，那么导出的模型将默认存储在 output_inference/ 路径下。
!python tools/export_model.py -c configs/ppyoloe/ppyoloe_plus_crn_s_80e_coco.yml -o weights=output/ppyoloe_plus_crn_s_80e_coco/best_model.pdparams
至此，我们就完成了水底生物目标检测模型的从训练到导出的过程。接下来，看看该模型使用Paddle Inference部署时的具体性能表现。
9 模型速度测试
!python deploy/python/infer.py --model_dir=output_inference/ppyoloe_plus_crn_s_80e_coco --image_file=dataset/fish/val/c000418.jpg --run_mode=paddle --device=gpu
----------- Running Arguments -----------
action_file: None
batch_size: 1
camera_id: -1
combine_method: nms
cpu_threads: 1
device: gpu
enable_mkldnn: False
enable_mkldnn_bfloat16: False
image_dir: None
image_file: dataset/fish/val/c000418.jpg
match_metric: ios
match_threshold: 0.6
model_dir: output_inference/ppyoloe_plus_crn_s_80e_coco
output_dir: output
overlap_ratio: [0.25, 0.25]
random_pad: False
reid_batch_size: 50
reid_model_dir: None
run_benchmark: False
run_mode: paddle
save_images: True
save_mot_txt_per_img: False
save_mot_txts: False
save_results: False
scaled: False
slice_infer: False
slice_size: [640, 640]
threshold: 0.5
tracker_config: None
trt_calib_mode: False
trt_max_shape: 1280
trt_min_shape: 1
trt_opt_shape: 640
use_coco_category: False
use_dark: True
use_gpu: False
video_file: None
window_size: 50
------------------------------------------
----------- Model Configuration -----------
Model Arch: YOLO
Transform Order:
--transform op: Resize
--transform op: NormalizeImage
--transform op: Permute
--------------------------------------------
class_id:0, confidence:0.7396, left_top:[299.31,0.93],right_bottom:[331.32,32.59]
class_id:0, confidence:0.7280, left_top:[293.39,100.21],right_bottom:[332.52,138.38]
class_id:0, confidence:0.6212, left_top:[528.09,-1.20],right_bottom:[566.73,20.30]
class_id:0, confidence:0.5986, left_top:[71.87,-0.63],right_bottom:[111.20,26.91]
class_id:0, confidence:0.5766, left_top:[120.02,101.65],right_bottom:[181.98,152.03]
class_id:0, confidence:0.5554, left_top:[405.12,215.59],right_bottom:[458.40,272.60]
class_id:0, confidence:0.5486, left_top:[476.20,49.54],right_bottom:[511.81,85.39]
class_id:1, confidence:0.6370, left_top:[446.48,281.86],right_bottom:[504.50,342.79]
class_id:1, confidence:0.5986, left_top:[450.84,89.30],right_bottom:[494.58,123.99]
class_id:1, confidence:0.5715, left_top:[510.54,89.15],right_bottom:[570.02,138.44]
class_id:1, confidence:0.5151, left_top:[512.79,94.07],right_bottom:[562.67,133.47]
save result to: output/c000418.jpg
Test iter 0
------------------ Inference Time Info ----------------------
total_time(ms): 1072.8, img_num: 1
average latency time(ms): 1072.80, QPS: 0.932140
preprocess_time(ms): 18.20, inference_time(ms): 1054.50, postprocess_time(ms): 0.10
10 模型部署
本项目用EasyEdge端与边缘AI服务平台对水底生物目标检测模型进行部署。EasyEdge是基于百度飞桨轻量化推理框架Paddle Lite研发的端与边缘AI服务平台，能够帮助深度学习开发者将自建模型快速部署到设备端。只需上传模型，最快2分种即可获得适配终端硬件/芯片的模型。
10.1 上传本地模型
10.2 生成端到端模型
至此，本项目的一个完整流程就结束啦！如果对此感兴趣，不妨动起来呀！
5462
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
