Brief
kitti衡量指标。
代码文件在kitti_dataset.py
中(SECOND)
- 代码 1
detections += net(example)
。。。
result_dict = eval_dataset.dataset.evaluation(detections,
str(result_path_step))
(1)detections 经过了网络的前向传播得到了预测的
box_pred
,cls_pred
和dir_pred
(2)后续的评价网络是研究的重点。
评价函数----evaluation
该函数定义为:
def evaluation(self, detections, output_dir):
对应的gt如下,这是最原始的kitti自带的label
ground_truth_annotations format:
{
bbox: [N, 4], if you fill fake data, MUST HAVE >25 HEIGHT!!!!!!
alpha: [N], you can use -10 to ignore it.
occluded: [N], you can use zero.
truncated: [N], you can use zero.
name: [N]
location: [N, 3] center of 3d box.
dimensions: [N, 3] dim of 3d box.
rotation_y: [N] angle.
}
输入的第一个参数是预测的内容,第二个参数是想要输出的文件夹。gt是一个自带的属性
- 1 convert_detection_to_kitti_annos
gt_annos = [info["annos"] for info in self._kitti_infos]
dt_annos = self.convert_detection_to_kitti_annos(detections)
(1)将预测出来的
detection
转化为kitti_annos。和对应的gt_nos对应.经过转化后,和gt一样含有5个维度
[bbox, location, dimensions, rotation_y, score]
解释:bbox:【xmin,ymin,xmax,ymax】二维边界框
location:3D中心
dimensions:w,h,l
rotation_y:yaw旋转角度
score:预测的置信度
- offical_eval
文件在eval.py
中。
def get_official_eval_result(gt_annos,
dt_annos,
current_classes,
difficultys=[0, 1, 2],
z_axis=1,
z_center=1.0):
(1)作者代码中写了8个类别的eval,如下图,stack之后
min_overlaps
变成 [ 2 , 3 , 8 ] [2,3,8] [2,3,8]
再经过训练类别选取得到:min_overlaps
为 [ 2 , 3 , 4 ] [2,3,4] [2,3,4]
- do_eval_v3
采用这个函数进行计算对应的准确率。
def do_eval_v3(gt_annos,
dt_annos,
current_classes,
min_overlaps,
compute_aos=False,
difficultys=(0, 1, 2),
z_axis=1,
z_center=1.0):
(1)gt_annos和dt_annos一样含有上面的5个元素的dict。min_overlaps在这里是 [ 2 , 3 , 4 ] [2,3,4] [2,3,4]。
如下如:这里的 [ 2 , 3 , 5 ] [2,3,5] [2,3,5]的含义:
(2)但是在输出界面上会有一个第四行,作者后续计算了一个mAPaos
的精度。应该是朝向预测?
if compute_aos:
mAPaos = get_mAP(metrics["bbox"]["orientation"][j, :, i])
detail[class_name][f"aos"] = mAPaos.tolist()
mAPaos = ", ".join(f"{v:.2f}" for v in mAPaos)
result += print_str(f"aos AP:{mAPaos}")
- coco eval
还有一种评价指标:coco_eval
,对应的含义和官方检测是一样的。
result_coco = get_coco_eval_result(
gt_annos,
dt_annos,
self._class_names,
z_axis=z_axis,
z_center=z_center)
最后检测效果
在kitti上进行了多分类,效果如下:
- 采用官方的计算方法:
Evaluation official
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:90.76, 89.45, 88.17
bev AP:89.83, 86.53, 79.62
3d AP:88.02, 77.50, 75.73
aos AP:0.23, 1.26, 2.32
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:90.76, 89.45, 88.17
bev AP:90.79, 90.06, 89.46
3d AP:90.79, 89.98, 89.27
aos AP:0.23, 1.26, 2.32
Cyclist AP(Average Precision)@0.50, 0.50, 0.50:
bbox AP:82.30, 70.63, 69.67
bev AP:79.81, 66.11, 60.95
3d AP:77.80, 60.30, 58.31
aos AP:0.90, 2.06, 1.94
Cyclist AP(Average Precision)@0.50, 0.25, 0.25:
bbox AP:82.30, 70.63, 69.67
bev AP:85.80, 68.40, 66.92
3d AP:85.79, 68.40, 66.91
aos AP:0.90, 2.06, 1.94
Pedestrian AP(Average Precision)@0.50, 0.50, 0.50:
bbox AP:62.27, 60.92, 54.80
bev AP:66.87, 59.97, 54.21
3d AP:60.86, 53.67, 51.13
aos AP:6.38, 6.48, 6.01
Pedestrian AP(Average Precision)@0.50, 0.25, 0.25:
bbox AP:62.27, 60.92, 54.80
bev AP:79.75, 73.70, 71.25
3d AP:79.75, 73.68, 71.25
aos AP:6.38, 6.48, 6.01
Van AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:45.80, 39.60, 34.79
bev AP:45.12, 39.24, 34.52
3d AP:39.13, 34.98, 30.87
aos AP:0.46, 0.24, 0.62
Van AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:45.80, 39.60, 34.79
bev AP:45.57, 40.04, 39.10
3d AP:45.56, 39.87, 39.02
aos AP:0.46, 0.24, 0.62
疑惑,为什么每一个类别含有两个得分,分别对应mod和easy两种
- coco方式
Evaluation coco
Car coco AP@0.50:0.05:0.95:
bbox AP:72.62, 68.94, 67.36
bev AP:70.40, 66.24, 64.21
3d AP:61.57, 56.84, 54.95
aos AP:0.17, 0.93, 1.76
Cyclist coco AP@0.25:0.05:0.70:
bbox AP:79.73, 67.53, 65.18
bev AP:73.43, 58.34, 56.04
3d AP:70.07, 54.83, 52.43
aos AP:0.87, 1.93, 1.79
Pedestrian coco AP@0.25:0.05:0.70:
bbox AP:54.61, 53.78, 50.98
bev AP:59.19, 54.69, 50.82
3d AP:54.40, 50.16, 47.19
aos AP:5.73, 6.01, 5.94
Van coco AP@0.50:0.05:0.95:
bbox AP:33.89, 29.62, 27.60
bev AP:35.59, 30.77, 28.82
3d AP:28.08, 24.45, 22.28
aos AP:0.36, 0.16, 0.54
函数debug
- 在
train.py
中,对所用的eval
数据生成了dict,再经过nms得到了最后的detections
(dict,shape是[num_batch]),函数调用代码如下,
result_dict = eval_dataset.dataset.evaluation(detections,
str(result_path_step))
- 该函数的实现在
kitti_dataset.py
文件中的class KittiDataset
的evaluation
函数。上文也已经说过了。 get_official_eval_result
函数定义了官方的测试函数。该函数主要调用函数eval_class_v3
,eval_class_v3
定义如下:
def eval_class_v3(gt_annos,
dt_annos,
current_classes,
difficultys,
metric,
min_overlaps,
compute_aos=False,
z_axis=1,
z_center=1.0,
num_parts=50):
(1)输入参数如下表:
输入参数 | shape | 含义 |
---|---|---|
gt_annos | list,len=eval_num | list中的每一个元素分别表示每一个点云的gt内容。每一个gt内容包括有 {’name ’,truncated ,occluded ,alpha ,bbox ,dimensions ,location ,rotation_y ,score ,index ,group_ids ,difficulty ,num_points_in_gt }这些元素的含义和gt中的label一一对应。 bbox 表示了二维bbox,,dim,location,rotation_y 共同表示三维bbox |
dt_annos | list,len=eval_num | 内容同上,不过是预测出来的 |
current_classes | list,len=num_list(=4) | 只检测{car,cyclist,pedestrain,van}四类 |
difficultys | list,len=3 | 内容是{0,1,2}应该是指的困难级别 |
min_overlaps | [2,3,4] | 其中2表示{easy,modrate(重要待补充)}, 3表示{box,bev,3d},4表示4类 |
metrics | int | eval type. 0: bbox, 1: bev, 2: 3d |
(2)分成75个块(3769//50=75),分别进行iou计算,作者表示这样分块可以加速,应该是并行计算吧。每一块50个.
。。。。内容太多,先放弃了
重点
本篇博客主要是帮助博主自己理解kitti evalute的各个含义,
对于官方的预测方式,对车的如下,每一个class都有两种,具体含义如最后的图:
Car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:90.76, 89.45, 88.17
bev AP:89.83, 86.53, 79.62
3d AP:88.02, 77.50, 75.73
aos AP:0.23, 1.26, 2.32
Car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:90.76, 89.45, 88.17
bev AP:90.79, 90.06, 89.46
3d AP:90.79, 89.98, 89.27
aos AP:0.23, 1.26, 2.32
其实博主之前就是这么理解的,但是很不解的是同一个class上下两次一样的easy
的最小IOU
(=0.7)的时候却有着不一样的AP是什么意思?。最后的aos的含义根据作者的回复是可以忽略的。下面是作者的回复:
Car@0.70, 0.70, 0.70 means evaluate car performance in easy, moderate, hard and use 0.7 (easy), 0.7 (mod), 0.7 (hard) as overlap threshold.
bbox means calculate overlap (intersection of union area) by bbox overlap, bev means bev overlap, 3d means 3d overlap. the aos… you can ignore this.