The evaluation of SSD algorith:calculate the mAP

In the past few weeks, I have successfully identified flames by using the trained SSD model, and the effect looks good. So can we simply think that this model is good enough?

It often requires several evaluation indicators to assess the performance of a model , such as the value of confidence loss and location loss. Today we quote the mAP, which is widely used in the target detection model, to evaluate the model we trained before.

Firstly,Here we introduce the concept of IOU.IOU stands for Intersection over Union.The calculation principle of IOU is very simple: it is the ratio of the area of the intersection and union of the prediction box and the ground truth box.If you want to know the details of IOU function,it is highly recommended to refer to the following blog.
IOU
In the prediction of the model, there are many prediction boxes corresponding to the same real box, so we need to use the IOU indicator to help us filter redundant boxes. When the IOU is less than the set threshold, the box which is thought to predict another real boxes will be retained .

def nms(boxes, scores, overlap=0.5, top_k=200):
    keep = scores.new(scores.size(0)).zero_().long()
    # 保存留下来的box的索引 [num_positive]
    # 函数new(): 构建一个有相同数据类型的tensor

    # 如果输入box为空则返回空Tensor
    if boxes.numel() == 0:
        return keep
    x1 = boxes[:, 0]
    y1 = boxes[:, 1]
    x2 = boxes[:, 2]
    y2 = boxes[:, 3]
    area = torch.mul(x2 - x1, y2 - y1)#并行化计算所有框的面积
    v, idx = scores.sort(0)# 升序排序
    idx = idx[-top_k:]# 前top-k的索引，从小到大
    xx1 = boxes.new()
    yy1 = boxes.new()
    xx2 = boxes.new()
    yy2 = boxes.new()
    w = boxes.new()
    h = boxes.new()

    count = 0
    while idx.numel() > 0:
        i = idx[-1]# 目前最大score对应的索引
        keep[count] = i#存储在keep中
        count += 1
        if idx.size(0) == 1:
            break
        idx = idx[:-1]# 去掉最后一个

        # 剩下boxes的信息存储在xx，yy中
        torch.index_select(x1, 0, idx, out=xx1)
        torch.index_select(y1, 0, idx, out=yy1)
        torch.index_select(x2, 0, idx, out=xx2)
        torch.index_select(y2, 0, idx, out=yy2)

        # 计算当前最大置信框与其他剩余框的交集
        xx1 = torch.clamp(xx1, min=x1[i])
        yy1 = torch.clamp(yy1, min=y1[i])
        xx2 = torch.clamp(xx2, max=x2[i])
        yy2 = torch.clamp(yy2, max=y2[i])
        w.resize_as_(xx2)
        h.resize_as_(yy2)
        w = xx2 - xx1
        h = yy2 - yy1
        w = torch.clamp(w, min=0.0)
        h = torch.clamp(h, min=0.0)
        inter = w*h
        # 计算当前最大置信框与其他剩余框的IOU
        # IoU = i / (area(a) + area(b) - i)
        rem_areas = torch.index_select(area, 0, idx)# 剩余的框的面积
        union = (rem_areas - inter) + area[i]#并集
        IoU = inter/union # 计算iou
        # 选出IoU <= overlap的boxes(注意le函数的使用)
        idx = idx[IoU.le(overlap)]
    return keep, count

Then，we need to introduce other concepts including TP、TN、FP、FN。
T represents True classification.
F represents False classification.
P represents Positive sample.
N represents Negative sample.
Flame detection is a binary classification problem.So the flame is positive sample while the background is negative sample.
TP (True Positives) means that the sample is predicted as positive samples, and the result is true.
TN (True Negatives) means that the sample is predicted as a negative sample, and the result is true.
FP (False Positives) means that the sample is predicted as a positive sample, but the result is false.(in fact, this sample is a negative sample)。
FN (False Negatives) means that the sample is predicted as a negative sample, but the result is false. (in fact, this sample is a positive sample).
If you still can not understand what I say,l will take some example to help you comprehend the concept.

As you can see in the above figure,the red box is predicted as negative sample ,but actually it is positive sample,So this red box is FN.Besides, the green box is predicted as positive sample and the result is true.So the green box is TP. And the blue box is ground truth box.Here we need to utilize the IOU indicator to decide whether the green box is really a positive sample .When the IOU of green box and blue box is less than set threhold,the green box is thought to be a positive box.

The evaluation indicator:Precision and Recall
$$Precision=TP/(TP+FP)$$
$$Recall=TP/(TP+FN)$$

The model needs to set a hyperparameter in the detection process: confidence (ie, prediction score)。Only when the confidence of the sample is higher than the set threhold will the sample be predicted as positive sample
Therefore, this hyperparameter also has a great influence on the calculation of precision and recall. The values of precision and recall will also change with different confidence settings, It is inaccurate to evaluate the model based on a single indicator.So, we introduce AP(average precison) as the evaluation index: by taking different confidence values, the calculated precision and recall are drawn into a curve, and the value of the average accuracy rate is the integral area of the curve relative to the x axis.
The mAP（mean average precision） represents the average value of AP in each category.
Since flame identification is a binary classification problem, the value of mAP is the value of flame AP.

The result :




note:In the calculation of the recall ，the sum of TP and FN is the number of all positive samples, ie the number of ground-truth box.
Through the evaluation of mAP indicators, you can find that the performance of the trained SSD is not good.In the case of high precision, the recall rate is very low.So it needs to be improved in later work.
You can use this project to help you draw mAP curves
https://github.com/Cartucho/mAP