maskrcnn benchmark cocoapi 在自己数据集上的性能测试

使用git 中cocoapi的pycocoEvalDemo.ipynb更改
需要：

1 自己验证集的标注

此处使用labelme制作。

格式如下：

{
    "images": [
        {
            "height": 2304,
            "width": 3456,
            "id": 1,
            "file_name": "s1.jpg"
        },
......
    ],
    "categories": [
        {
            "supercategory": "q",
            "id": 1,
            "name": "q"
        },
        {
            "supercategory": "z",
            "id": 2,
            "name": "z"
        },
        {
            "supercategory": "b",
            "id": 3,
            "name": "b"
        },
        {
            "supercategory": "x",
            "id": 4,
            "name": "x"
        }
    ],
   "annotations": [
    {
        "segmentation": [
            [
                1393.2380952380952,
                323.2380952380952,
                1427.2380952380952,
                383.2380952380952,
                1322.2380952380952,
                504.2380952380952,
                867.2380952380952,
                630.2380952380952,
                748.2380952380952,
                561.2380952380952,
                686.2380952380952,
                421.2380952380952,
                524.2380952380952,
                445.2380952380952,
                522.2380952380952,
                426.2380952380952,
                660.2380952380952,
                397.2380952380952,
                636.2380952380952,
                359.2380952380952,
                1134.2380952380952,
                197.23809523809518,
                1308.2380952380952,
                218.23809523809518,
                1360.2380952380952,
                173.23809523809518,
                1441.2380952380952,
                278.2380952380952
            ]
        ],
        "iscrowd": 0,
        "image_id": 1,
        "area": 228602.5,
        "bbox": [
            522.0,
            173.0,
            919.0,
            457.0
        ],
        "category_id": 1,
        "id": 1
    },
       ......

其中area负责在评估时候界定范围．

2 对测试集利用模型生成结果json

2.1　'bbox’格式保存

这里测试了24张图片并保存了bbox的结果
https://blog.csdn.net/qq_35608277/article/details/88920728
结果为list,其中一个结果格式包含以下几个key：
{“image_id”: 24, “category_id”: 1, “bbox”: [992.66943359375, 570.6240844726562, 1033.0626220703125, 527.0519409179688], “score”: 0.9999897480010986}
bbox此处的格式与labelme中的格式保持一致：矩形框左上角坐标+宽高

2.2 'segm’格式保存

该格式需要将输出的mask转化为rle格式
"segmentation"使用cocoapi.PythonAPI.pycocotools 中的 mask．py中encode函数进行编码，其中必须要求mask数据类型为uint８．保存时转化为str进行保存，默认utf格式．

    def feedback_factors(self,image):
        from cocoapi.PythonAPI.pycocotools import mask as maskUtil
        predictions = self.compute_prediction(image)
        top_predictions = self.select_top_predictions(predictions)

        boxes = top_predictions.bbox.tolist()
        scores = top_predictions.get_field("scores").tolist()
        labels = top_predictions.get_field("labels").tolist()
        masks = top_predictions.get_field("mask").numpy()
        # saving masks in form cocoeval need
        masks_lists=[]
        for mask in (masks):
            
            thresh = np.array(mask[0, :, :, None],dtype='uint8',order='F')
            res=maskUtil.encode(thresh)
            res[0]['counts']=res[0]['counts'].decode()
            masks_lists.append(res[0])

        result = []
        for box in boxes:
            box[2]=box[2]-box[0]
            box[3]=box[3]-box[1]
        result.extend(
        [
            {
                "image_id": 1,
                "category_id": labels[k],
                "bbox": box,
                "score": scores[k],
                "segmentation":masks_lists[k],
            }
            for k, box in enumerate(boxes)
        ]
    )

主程序当中对测试图片进行遍历，因为没有改图片名和写数据集接口，只是简单使用遍历，后面对图片编号进行了更改．

将以下元素 进行了保存．imageid是图片序号，从１开始．category_id此处有三类．
“image_id”
“category_id”
“bbox”
“score”
“segmentation”

# -*- coding: utf-8 -*-

from maskrcnn_benchmark.config import cfg
from predictor import Mydata
import cv2
from maskrcnn_benchmark.data.datasets.coco import *
config_file="/home/dong/MASK_RCNN/maskrcnn-benchmark-master/mask/configs/my_cfg/test_mask_rcnn_R_50_FPN.yaml"
cfg.merge_from_file(config_file)

# prepare object that handles inference plus adds predictions on top of image
mydata_demo = Mydata(
    cfg,
    confidence_threshold=0.7,

    min_image_size=800,
)

out=[]
for i in range(73,93):

    img=cv2.imread('/home/dong/MASK_RCNN/maskrcnn-benchmark-master/mask/datasets/coco/train2017/x{}.jpg'.format(i))

    composite = mydata_demo.feedback_factors(img)
    # change the image id 
    for sample in composite:
        sample['image_id']=i-72
    for sample in composite:    
        out.append(sample)#detection result including 
    print('the number ',i,'image','\n')


import json
print("saving...")
with open('test_with_rle.json', 'w', encoding='utf-8') as file:
    json.dump(out, file, ensure_ascii=False)

3 评估代码

import matplotlib.pyplot as plt
from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval
import numpy as np
import skimage.io as io
import pylab
pylab.rcParams['figure.figsize'] = (10.0, 8.0)
import sys
sys.path.append('/home/x/MASK_RCNN/maskrcnn-benchmark-master/mask/cocoapi/PythonAPI/pycocotools')

annType = ['segm','bbox','keypoints']
annType = annType[0]      #specify type here
prefix = 'person_keypoints' if annType=='keypoints' else 'instances'
print ('Running demo for *%s* results.'%(annType))

#dataDir='/home/x/MASK_RCNN/maskrcnn-benchmark-master/mask/datasets/coco'
# use the valadation labelme file
#si lei 
#annFile = '/home/x/MASK_RCNN/maskrcnn-benchmark-master/mask/datasets/coco/annotations/truth.json'

cocoGt=COCO(annFile)
#initialize COCO detections api
# use the generated results
#si lei 
#resFile = '/home/x/MASK_RCNN/maskrcnn-benchmark-master/mask/cocoapi/results/test_data_with_seg.json'#


cocoDt=cocoGt.loadRes(resFile)

imgIds=sorted(cocoGt.getImgIds())
imgIds=imgIds[0:20]
imgId = imgIds[np.random.randint(20)]

# running box evaluation
cocoEval = COCOeval(cocoGt,cocoDt,annType)
cocoEval.params.imgIds  = imgIds
cocoEval.params.iouType='segm'
#choose the classes need to be evaluated
#cocoEval.params.catIds = [1]
cocoEval.params.maxDets = [1,5,100]
cocoEval.evaluate()
cocoEval.accumulate()
cocoEval.summarize()

进行评估前需要设置参数，部分默认参数如下

        self.imgIds = []
        self.catIds = []
        # np.arange causes trouble.  the data point on arange is slightly larger than the true value
        self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
        self.recThrs = np.linspace(.0, 1.00, np.round((1.00 - .0) / .01) + 1, endpoint=True)
        self.maxDets = [1, 10, 100]
        self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]
        self.areaRngLbl = ['all', 'small', 'medium', 'large']
        self.useCats = 1

cocoEval.params.maxDets = [1,5,100]参数为一张图最多检测的目标数量．分了1，5，100
cocoEval.params.iouType='segm’为计算交并比使用的数据，＇bbox＇为计算bbox的，＇segm＇计算掩膜
cocoEval.params.catIds = [1]可以选择只计算哪一类
cocoEval.params.areaRng为四种不同尺度大小，分别是all small medium large
iouThrs为0.5－0.95
recThrs总共101个，有没有人能解释一下这个？？

out

Running demo for *segm* results.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.00s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.616
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.793
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.746
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.616
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.625
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  5 ] = 0.625
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.625
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.625

文档结构

为了直接在同一个文件夹下引用方便，放的比较散，可以优化

P-R图绘制

需要使用matlab下的cocoapi，只提供了matlab的analyze
cocoEval.analyze()
数据使用相同，ap　ar结果相同．
官方示例如下

总结

总结一下整个流程
１．labelme标注评估集，生成单独的json文件．注意命名规则
２．使用labelme２coco＿with_area.py汇总转换为coco格式（注意：没有面积会在matlab＇segm＇模式下无法计算）
３．利用训练好的模型对评估集进行测试，利用generate_my_result.py输出结果，并保存为json格式，（注意保存的格式，１．图片要对应．２对照项目中的范例对各个要素进行检查，如＇bbox＇是float类型３＇segm＇类型中的counts需要rle格式编码转换，在保存时候bytes类型要转化为str类型并且utf编码）
４．调用my_valuate.py ,在matlab中输出map和pr图