mot_challenge官方评估代码

mot_challenge官方代码使用介绍

评估mot17验证集：每个视频序列的后一半帧作为验证集

下载源码：git clone https://hub.fastgit.org/JonathonLuiten/TrackEval.git

官网上下载data样例，放在TrackEval/下，结构如下：

data
|-- gt
|   |-- bdd100k
|   |-- davis
|   |-- kitti
|   |-- mot_challenge
|   |   |-- MOT15-train
|   |   |-- MOT16-train
|   |   |-- MOT17-train
|   |   |-- MOT20-train
|   |   |-- MOTS-train
|   |   `-- seqmaps
|   |-- tao
|   `-- youtube_vis
`-- trackers
    |-- bdd100k
    |-- davis
    |-- kitti
    |-- mot_challenge
    |   |-- MOT15-train
    |   |   `-- MPNTrack
    |   |       `-- data
    |   |-- MOT16-train
    |   |   `-- MPNTrack
    |   |       `-- data
    |   |-- MOT17-train
    |   |   |-- MPNTrack
    |   |   |   `-- data
    |   |   `-- yourtracker
    |   |       `-- data
    |   |-- MOT20-train
    |   |   `-- MPNTrack
    |   |       `-- data
    |   `-- MOTS-train
    |       `-- track_rcnn
    |           `-- data
    |-- tao
    `-- youtube_vis

评估支持多个数据集，这里只介绍mot_challenge如何使用这个评估。

gt表示每个数据集的标注，mot_challenge中只包含训练集的标注，因为测试集没有标注。

data/gt/mot_challenge/seqmaps里面存储的是每个数据需要测试的视频序列名称，因为mot17包含3种检测器的视频序列，通常我们只使用一个检测器的视频序列，因为gt的标注都一样。这里我使用SDP检测的视频序列，所以当评估mot17验证集时，需要把其他检测器的视频序列名称删除，这样你自己的tracker得到文件就不会报找不到文件错误了。

1. 修改1：更改seqmaps文件下需要评测的视频序列名称

例如：

源文件内容：data/gt/mot_challenge/seqmaps/MOT17-train.txt

name
MOT17-02-DPM
MOT17-04-DPM
MOT17-05-DPM
MOT17-09-DPM
MOT17-10-DPM
MOT17-11-DPM
MOT17-13-DPM
MOT17-02-FRCNN
MOT17-04-FRCNN
MOT17-05-FRCNN
MOT17-09-FRCNN
MOT17-10-FRCNN
MOT17-11-FRCNN
MOT17-13-FRCNN
MOT17-02-SDP
MOT17-04-SDP
MOT17-05-SDP
MOT17-09-SDP
MOT17-10-SDP
MOT17-11-SDP
MOT17-13-SDP

删除后文件内容:

name
MOT17-02-SDP
MOT17-04-SDP
MOT17-05-SDP
MOT17-09-SDP
MOT17-10-SDP
MOT17-11-SDP
MOT17-13-SDP

2. 修改2：生成验证集标签：{gt_folder}/{seq}/gt/gt_val_half.txt

代码：

import os.path as osp
import os
import numpy as np
from tqdm import tqdm
def mkdir_if_missing(d):
    if osp.isfile(d):
        d = osp.dirname(d)
    if not osp.exists(d):
        os.makedirs(d)

data_root = osp.join('../data/gt/mot_challenge') # the root directory of the dataset
gt_folder = osp.join(data_root, 'MOT17-train')
seqs_str = '''MOT17-02-SDP
              MOT17-04-SDP
              MOT17-05-SDP
              MOT17-09-SDP
              MOT17-10-SDP
              MOT17-11-SDP
              MOT17-13-SDP'''
seqs = [seq.strip() for seq in seqs_str.split()]

def gen_gt_val():
    for seq in tqdm(seqs):
        print('start seq {}'.format(seq))
        seq_info = open(osp.join(gt_folder, seq, 'seqinfo.ini')).read()
        seqLength = int(seq_info[seq_info.find('seqLength=') + 10:seq_info.find('\nimWidth')])
        gt_txt = osp.join(gt_folder, seq, 'gt', 'gt.txt')
        gt = np.loadtxt(gt_txt, dtype=np.float64, delimiter=',')
        save_val_gt = osp.join(gt_folder, seq, 'gt', 'gt_val_half.txt')
        val_start = seqLength // 2
        with open(save_val_gt, 'w') as f:
            for obj in gt:
                label_str = '{:d},{:d},{:d},{:d},{:d},{:d},{:d},{:d},{:.6f}\n'.format(
                    int(obj[0]), int(obj[1]), int(obj[2]), int(obj[3]), int(obj[4]), int(obj[5]), int(obj[6]), int(obj[7]), obj[8])
                if obj[0] > val_start:
                    f.write(label_str)

if __name__ == '__main__':
    gen_gt_val()

3. 创建yourtrack/data文件夹保存每个视频跟踪的结果。如果是mot17验证集的结果就放入data/trackers/mot_challenge/MOT17-train/yourtracker/data
4. 运行代码脚本：进入TrackEval/scripts

python run_mot_challenge.py --BENCHMARK MOT17 --TRACKERS_TO_EVAL yourtracker --METRICS CLEAR Identity --USE_PARALLEL True --GT_LOC_FORMAT {gt_folder}/{seq}/gt/gt_val_half.txt

脚本参数含义：

    Eval arguments:
        'USE_PARALLEL': False, # 是否使用并行
        'NUM_PARALLEL_CORES': 8,
        'BREAK_ON_ERROR': True,
        'PRINT_RESULTS': True,
        'PRINT_ONLY_COMBINED': False,
        'PRINT_CONFIG': True,
        'TIME_PROGRESS': True,
        'OUTPUT_SUMMARY': True,
        'OUTPUT_DETAILED': True,
        'PLOT_CURVES': True,
    Dataset arguments:
        'GT_FOLDER': os.path.join(code_path, 'data/gt/mot_challenge/'),  # Location of GT data
        'TRACKERS_FOLDER': os.path.join(code_path, 'data/trackers/mot_challenge/'),  # Trackers location
        'OUTPUT_FOLDER': None,  # Where to save eval results (if None, same as TRACKERS_FOLDER)
        'TRACKERS_TO_EVAL': None,  # Filenames of trackers to eval (if None, all in folder)
        'CLASSES_TO_EVAL': ['pedestrian'],  # Valid: ['pedestrian']
        'BENCHMARK': 'MOT17',  # Valid: 'MOT17', 'MOT16', 'MOT20', 'MOT15'
        'SPLIT_TO_EVAL': 'train',  # Valid: 'train', 'test', 'all'
        'INPUT_AS_ZIP': False,  # Whether tracker input files are zipped
        'PRINT_CONFIG': True,  # Whether to print current config
        'DO_PREPROC': True,  # Whether to perform preprocessing (never done for 2D_MOT_2015)
        'TRACKER_SUB_FOLDER': 'data',  # Tracker files are in TRACKER_FOLDER/tracker_name/TRACKER_SUB_FOLDER
        'OUTPUT_SUB_FOLDER': '',  # Output files are saved in OUTPUT_FOLDER/tracker_name/OUTPUT_SUB_FOLDER
    Metric arguments:
        'METRICS': ['HOTA', 'CLEAR', 'Identity', 'VACE']
# 默认配置，trackeval/datasets/mot_challenge_2d_box.py
        default_config = {
            'GT_FOLDER': os.path.join(code_path, 'data/gt/mot_challenge/'),  # Location of GT data
            'TRACKERS_FOLDER': os.path.join(code_path, 'data/trackers/mot_challenge/'),  # Trackers location
            'OUTPUT_FOLDER': None,  # Where to save eval results (if None, same as TRACKERS_FOLDER)
            'TRACKERS_TO_EVAL': None,  # Filenames of trackers to eval (if None, all in folder)
            'CLASSES_TO_EVAL': ['pedestrian'],  # Valid: ['pedestrian']
            'BENCHMARK': 'MOT17',  # Valid: 'MOT17', 'MOT16', 'MOT20', 'MOT15'
            'SPLIT_TO_EVAL': 'train',  # Valid: 'train', 'test', 'all'
            'INPUT_AS_ZIP': False,  # Whether tracker input files are zipped
            'PRINT_CONFIG': True,  # Whether to print current config
            'DO_PREPROC': True,  # Whether to perform preprocessing (never done for MOT15)
            'TRACKER_SUB_FOLDER': 'data',  # Tracker files are in TRACKER_FOLDER/tracker_name/TRACKER_SUB_FOLDER
            'OUTPUT_SUB_FOLDER': '',  # Output files are saved in OUTPUT_FOLDER/tracker_name/OUTPUT_SUB_FOLDER
            'TRACKER_DISPLAY_NAMES': None,  # Names of trackers to display, if None: TRACKERS_TO_EVAL
            'SEQMAP_FOLDER': None,  # Where seqmaps are found (if None, GT_FOLDER/seqmaps)
            'SEQMAP_FILE': None,  # Directly specify seqmap file (if none use seqmap_folder/benchmark-split_to_eval)
            'SEQ_INFO': None,  # If not None, directly specify sequences to eval and their number of timesteps
            'GT_LOC_FORMAT': '{gt_folder}/{seq}/gt/gt.txt',  # '{gt_folder}/{seq}/gt/gt.txt'
            'SKIP_SPLIT_FOL': False,  # If False, data is in GT_FOLDER/BENCHMARK-SPLIT_TO_EVAL/ and in
                                      # TRACKERS_FOLDER/BENCHMARK-SPLIT_TO_EVAL/tracker/
                                      # If True, then the middle 'benchmark-split' folder is skipped for both.
        }

必须需要修改的默认参数配置：gt位置参数格式

• 对应参数--GT_LOC_FORMAT {gt_folder}/{seq}/gt/gt_val_half.txt

需要添加的参数：

--BENCHMARK MOT17 # 数据集
--TRACKERS_TO_EVAL yourtracker # 你的tracker名称
--METRICS CLEAR Identity # 使用哪些度量标准