如何评价训练好的网络：
首先网络有一个参数是loss值，这反应了你训练好的网络得到的结果和真实值之间的差距，具体的公式后续会补充，不过查看loss曲线随着迭代次数的增多，如何变化，有助于查看训练是否过拟合，是否学习率太小。
训练指令：
./darknet detector train cfg/voc-hand.data cfg/yolov3-voc-hand.cfg darknet53.conv.74 -gpu 2>1 | tee visualization/train_yolov3_hand.log
训练指令解释：
1）在DarkNet主目录下编译完成，则DarkNet框架训练环境搭建完成。可以使用命令：“./darknet”
2）detector为训练文件（底层c程序）
3）train表示该命令为训练命令（当测试时输入测试命令将train改为test）
4）cfg/voc-hand.data:表示配置的数据集的参数文件（注意路径）
5）cfg/yolov3-voc-hand.cfg ：表示配置的网络参数文件（注意路径）
6）darknet53.conv.74：表示训练所需要的预训练模型（注意路径，此处在darknet文件夹下）
7）| tee visualization/train_yolov3_hand.log：表示训练时保存日志文件（保存在DarkNet/visualization目录下）
训练产生的log图示：
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000002, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: 0.898378, Class: 0.999900, Obj: 0.999876, No Obj: 0.000432, .5R: 1.000000, .75R: 1.000000, count: 2
Region 106 Avg IOU: 0.868176, Class: 0.999792, Obj: 0.999554, No Obj: 0.000157, .5R: 1.000000, .75R: 1.000000, count: 2
Region 82 Avg IOU: 0.577505, Class: 0.995153, Obj: 0.583519, No Obj: 0.000268, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000246, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.747150, Class: 0.998767, Obj: 0.996712, No Obj: 0.000056, .5R: 1.000000, .75R: 0.000000, count: 1
3690: 0.026425, 0.025064 avg, 0.000010 rate, 3.401929 seconds, 236160 images
训练产生的log的解释：
1）Region 82（94/106）：表示cfg文件中yolo-layer的索引值，三个值，82/94/106
2）Avg IOU：表示在训练过程中预测的bounding box与标注的bounding box的交并比（两个框的交集/两个框的并集），该值期望越大越好，目标值为1
3）Class：表示标注物体的分类准确率，该值期望越大越好，目标期望值为1
4）obj：表示预测有目标的概率,该值期望越大越好，目标值为1
5）No obj：表示预测没有目标的概率，该值期望越小越好，目标期望值为0
6）.5R：表示以IOU=0.5为阈值时被召回（recall）。recall=检出的正样本/实际的正样本
7）.75R：表示以IOU=0.75为阈值时被召回（recall）
8)count:表示正样本的数量
3690: 0.026425, 0.025064 avg, 0.000010 rate, 3.401929 seconds, 236160 images
1）3690：表示当前训练的迭代次数
2）0.026425：表示训练总的Loss损失
3）0.025064 avg： 表示平均Loss，该值期望越低越好，一般该值低于0.060730 avg就可以终止训练了。
4）0.000100 rate： 代表当前的学习率，是在.cfg文件中定义的。
5）3.401929 seconds： 表示当前批次训练花费的总时间（batch/subdivisions）
6）236160 images：表示到目前所参与训练的总的图片数量
查看训练网络的召回率
更改darknet/example/detector.c
// list *plist = get_paths("data/coco_val_5k.list");
list *plist = get_paths("2007_test.txt"); (2007_test.txt 在 darknet目录下)
289 710 746 RPs/Img: 21.33 IOU: 75.44% Recall:95.17%
290 711 748 RPs/Img: 21.34 IOU: 75.38% Recall:95.05%
291 713 750 RPs/Img: 21.32 IOU: 75.41% Recall:95.07%
292 715 752 RPs/Img: 21.37 IOU: 75.37% Recall:95.08%
293 717 754 RPs/Img: 21.33 IOU: 75.40% Recall:95.09%
294 719 756 RPs/Img: 21.32 IOU: 75.42% Recall:95.11%
295 721 758 RPs/Img: 21.32 IOU: 75.41% Recall:95.12%
296 722 759 RPs/Img: 21.28 IOU: 75.42% Recall:95.13%
297 722 759 RPs/Img: 21.36 IOU: 75.42% Recall:95.13%
298 722 759 RPs/Img: 21.42 IOU: 75.42% Recall:95.13%
299 724 761 RPs/Img: 21.41 IOU: 75.44% Recall:95.14%
300 725 762 RPs/Img: 21.43 IOU: 75.44% Recall:95.14%
301 727 764 RPs/Img: 21.42 IOU: 75.48% Recall:95.16%
302 728 765 RPs/Img: 21.43 IOU: 75.46% Recall:95.16%
303 728 765 RPs/Img: 21.40 IOU: 75.46% Recall:95.16%
304 730 767 RPs/Img: 21.40 IOU: 75.48% Recall:95.18%
305 734 771 RPs/Img: 21.42 IOU: 75.50% Recall:95.20%
306 746 783 RPs/Img: 21.45 IOU: 75.59% Recall:95.27%
307 748 785 RPs/Img: 21.43 IOU: 75.62% Recall:95.29%
308 750 787 RPs/Img: 21.42 IOU: 75.59% Recall:95.30%
309 752 789 RPs/Img: 21.43 IOU: 75.62% Recall:95.31%
310 754 791 RPs/Img: 21.44 IOU: 75.63% Recall:95.32%
具体的解释如下：
Number Correct Total Rps/Img IOU Recall
Number： 表示处理到第几张图片。
Correct： 表示正确的识别除了多少bbox。这个值算出来的步骤是这样的，丢进网络一张图片，网络会预测出很多bbox，每个bbox都有其置信概率，概率大于threshold的bbox与实际的bbox，也就是labels中txt的内容计算IOU，找出IOU最大的bbox，如果这个最大值大于预设的IOU的threshold，那么correct加一。
Total： 表示实际有多少个bbox。
Rps/img： 表示平均每个图片会预测出来多少个bbox。
IOU： 这个是预测出的bbox和实际标注的bbox的交集 除以 他们的并集。显然，这个数值越大，说明预测的结果越好。
Recall： 召回率 意思是检测出物体的个数 除以 标注的所有物体个数。通过代码我们也能看出来就是Correct除以Total的值。
参考链接：https://www.jianshu.com/p/7ae10c8f7d77/
综合计算性能指标：
修改yolov3的darknet文件夹example文件夹中的detector程序（计算ap值、recall值、avg、iou值、precision值、fp、tp以及绘制P-R曲线指标所需的precision和recall值等值）
在detector文件中添加计算map的函数并进行设置。
具体操作步骤：
1. 在detector.c文件中找到以下代码：
if(0==strcmp(argv[2], "test")) test_detector(datacfg, cfg, weights, filename, thresh, hier_thresh, outfile, fullscreen);
else if(0==strcmp(argv[2], "train")) train_detector(datacfg, cfg, weights, gpus, ngpus, clear);
else if(0==strcmp(argv[2], "valid")) validate_detector(datacfg, cfg, weights, outfile);
else if(0==strcmp(argv[2], "valid2")) validate_detector_flip(datacfg, cfg, weights, outfile);
else if(0==strcmp(argv[2], "recall")) validate_detector_recall(datacfg, cfg, weights);
2. 在其后面加上一条代码语句：
else if(0==strcmp(argv[2], "map")) validate_detector_map(datacfg, cfg, weights, thresh);
3. 在detector.c程序中的任意位置添加以下代码：
typedef struct {
box b;
float p;
int class_id;
int image_index;
int truth_flag;
int unique_truth_index;
} box_prob;
int detections_comparator(const void *pa, const void *pb)
{
box_prob a = *(box_prob *)pa;
box_prob b = *(box_prob *)pb;
float diff = a.p - b.p;
if (diff < 0) return 1;
else if (diff > 0) return -1;
return 0;
}
void validate_detector_map(char *datacfg, char *cfgfile, char *weightfile, float thresh_calc_avg_iou)
{
list *options = read_data_cfg(datacfg); //get .data file
char *valid_images = option_find_str(options, "valid", "data/train.txt"); //point to the path of valid images
char *difficult_valid_images = option_find_str(options, "difficult", NULL); //get the path to the 'difficult', if it doesn't exist,replace it with NULL
char *name_list = option_find_str(options, "names", "data/names.list"); // find name of each category
char **names = get_labels(name_list);
//char *mapf = option_find_str(options, "map", 0); // get the 'map', what is the map
//int *map = 0;
//if (mapf) map = read_map(mapf);
FILE* reinforcement_fd = NULL;
network *net = load_network(cfgfile, weightfile, 0);
set_batch_network(net, 1);
//fuse_conv_batchnorm(net);
//calculate_binary_weights(net);
srand(time(0));
list *plist = get_paths(valid_images);
char **paths = (char **)list_to_array(plist);
char **paths_dif = NULL;
if (difficult_valid_images) {
list *plist_dif = get_paths(difficult_valid_images);
paths_dif = (char **)list_to_array(plist_dif);
}
layer l = net->layers[net->n - 1];
int classes = l.classes;
int m = plist->size;
int i = 0;
int t;
const float thresh = .005;
const float nms = .45;
const float iou_thresh = 0.5;
int nthreads = 4;
image *val = calloc(nthreads, sizeof(image));
image *val_resized = calloc(nthreads, sizeof(image));
image *buf = calloc(nthreads, sizeof(image));
image *buf_resized = calloc(nthreads, sizeof(image));
pthread_t *thr = calloc(nthreads, sizeof(pthread_t));
load_args args = {0};
args.w = net->w;
args.h = net->h;
//args.type = IMAGE_DATA;
args.type = LETTERBOX_DATA;
//const float thresh_calc_avg_iou = 0.24;
float avg_iou = 0;
int tp_for_thresh = 0;
int fp_for_thresh = 0;
box_prob *detections = calloc(1, sizeof(box_prob));
int detections_count = 0;
int unique_truth_count = 0;
int *truth_classes_count = calloc(classes, sizeof(int));
for (t = 0; t < nthreads; ++t) {
args.path = paths[i + t];
args.im = &buf[t];
args.resized = &buf_resized[t];
thr[t] = load_data_in_thread(args);
}
time_t start = time(0);
for (i = nthreads; i < m + nthreads; i += nthreads) {
fprintf(stderr, "%d\n", i);
for (t = 0; t < nthreads && i + t - nthreads < m; ++t) {
pthread_join(thr[t], 0);
val[t] = buf[t];
val_resized[t] = buf_resized[t];
}
for (t = 0; t < nthreads && i + t < m; ++t) {
args.path = paths[i + t];
args.im = &buf[t];
args.resized = &buf_resized[t];
thr[t] = load_data_in_thread(args);
}
for (t = 0; t < nthreads && i + t - nthreads < m; ++t) {
const int image_index = i + t - nthreads;
char *path = paths[image_index];
char *id = basecfg(path);
float *X = val_resized[t].data;
network_predict(net, X);
int nboxes = 0;
float hier_thresh = 0;
detection *dets;
if (args.type == LETTERBOX_DATA) {
//int letterbox = 1;
dets = get_network_boxes(net, val[t].w, val[t].h, thresh, hier_thresh, 0, 1, &nboxes);
}
else {
//int letterbox = 0;
dets = get_network_boxes(net, 1, 1, thresh, hier_thresh, 0, 0, &nboxes);
}
//detection *dets = get_network_boxes(&net, val[t].w, val[t].h, thresh, hier_thresh, 0, 1, &nboxes, letterbox); // for letterbox=1
if (nms) do_nms_sort(dets, nboxes, l.classes, nms);
char labelpath[4096];
find_replace(path, "images", "labels", labelpath);
find_replace(labelpath, "JPEGImages", "labels", labelpath);
find_replace(labelpath, ".jpg", ".txt", labelpath);
find_replace(labelpath, ".JPEG", ".txt", labelpath);
int num_labels = 0;
box_label *truth = read_boxes(labelpath, &num_labels);
int i, j;
for (j = 0; j < num_labels; ++j) {
truth_classes_count[truth[j].id]++;
}
// difficult
box_label *truth_dif = NULL;
int num_labels_dif = 0;
if (paths_dif)
{
char *path_dif = paths_dif[image_index];
char labelpath_dif[4096];
//replace_image_to_label(path_dif, labelpath_dif);
find_replace(path_dif, "images", "labels", labelpath_dif);
find_replace(labelpath_dif, "JPEGImages", "labels", labelpath_dif);
find_replace(labelpath_dif, ".jpg", ".txt", labelpath_dif);
find_replace(labelpath_dif, ".JPEG", ".txt", labelpath_dif);
truth_dif = read_boxes(labelpath_dif, &num_labels_dif);
}
const int checkpoint_detections_count = detections_count;
for (i = 0; i < nboxes; ++i) {
int class_id;
for (class_id = 0; class_id < classes; ++class_id) {
float prob = dets[i].prob[class_id];
if (prob > 0) {
detections_count++;
detections = realloc(detections, detections_count * sizeof(box_prob));
detections[detections_count - 1].b = dets[i].bbox;
detections[detections_count - 1].p = prob;
detections[detections_count - 1].image_index = image_index;
detections[detections_count - 1].class_id = class_id;
detections[detections_count - 1].truth_flag = 0;
detections[detections_count - 1].unique_truth_index = -1;
int truth_index = -1;
float max_iou = 0;
for (j = 0; j < num_labels; ++j)
{
box t = { truth[j].x, truth[j].y, truth[j].w, truth[j].h };
//printf(" IoU = %f, prob = %f, class_id = %d, truth[j].id = %d \n",
//box_iou(dets[i].bbox, t), prob, class_id, truth[j].id);
float current_iou = box_iou(dets[i].bbox, t);
if (current_iou > iou_thresh && class_id == truth[j].id) {
if (current_iou > max_iou) {
max_iou = current_iou;
truth_index = unique_truth_count + j;
}
}
}
// best IoU
if (truth_index > -1) {
detections[detections_count - 1].truth_flag = 1;
detections[detections_count - 1].unique_truth_index = truth_index;
}
else {
// if object is difficult then remove detection
for (j = 0; j < num_labels_dif; ++j) {
box t = { truth_dif[j].x, truth_dif[j].y, truth_dif[j].w, truth_dif[j].h };
float current_iou = box_iou(dets[i].bbox, t);
if (current_iou > iou_thresh && class_id == truth_dif[j].id) {
--detections_count;
break;
}
}
}
// calc avg IoU, true-positives, false-positives for required Threshold
if (prob > thresh_calc_avg_iou) {
int z, found = 0;
for (z = checkpoint_detections_count; z < detections_count-1; ++z)
if (detections[z].unique_truth_index == truth_index) {
found = 1; break;
}
if(truth_index > -1 && found == 0) {
avg_iou += max_iou;
++tp_for_thresh;
}
else
fp_for_thresh++;
}
}
}
}
unique_truth_count += num_labels;
//static int previous_errors = 0;
//int total_errors = fp_for_thresh + (unique_truth_count - tp_for_thresh);
//int errors_in_this_image = total_errors - previous_errors;
//previous_errors = total_errors;
//if(reinforcement_fd == NULL) reinforcement_fd = fopen("reinforcement.txt", "wb");
//char buff[1000];
//sprintf(buff, "%s\n", path);
//if(errors_in_this_image > 0) fwrite(buff, sizeof(char), strlen(buff), reinforcement_fd);
free_detections(dets, nboxes);
free(id);
free_image(val[t]);
free_image(val_resized[t]);
}
}
if((tp_for_thresh + fp_for_thresh) > 0)
avg_iou = avg_iou / (tp_for_thresh + fp_for_thresh);
// SORT(detections)
qsort(detections, detections_count, sizeof(box_prob), detections_comparator);
typedef struct {
double precision;
double recall;
int tp, fp, fn;
} pr_t;
// for PR-curve
pr_t **pr = calloc(classes, sizeof(pr_t*));
for (i = 0; i < classes; ++i) {
pr[i] = calloc(detections_count, sizeof(pr_t));
}
printf("detections_count = %d, unique_truth_count = %d \n", detections_count, unique_truth_count);
int *truth_flags = calloc(unique_truth_count, sizeof(int));
int rank;
for (rank = 0; rank < detections_count; ++rank) {
if(rank % 100 == 0)
printf(" rank = %d of ranks = %d \r", rank, detections_count);
if (rank > 0) {
int class_id;
for (class_id = 0; class_id < classes; ++class_id) {
pr[class_id][rank].tp = pr[class_id][rank - 1].tp;
pr[class_id][rank].fp = pr[class_id][rank - 1].fp;
}
}
box_prob d = detections[rank];
// if (detected && isn't detected before)
if (d.truth_flag == 1) {
if (truth_flags[d.unique_truth_index] == 0)
{
truth_flags[d.unique_truth_index] = 1;
pr[d.class_id][rank].tp++; // true-positive
}
}
else {
pr[d.class_id][rank].fp++; // false-positive
}
for (i = 0; i < classes; ++i)
{
const int tp = pr[i][rank].tp;
const int fp = pr[i][rank].fp;
const int fn = truth_classes_count[i] - tp; // false-negative = objects - true-positive
pr[i][rank].fn = fn;
if ((tp + fp) > 0) pr[i][rank].precision = (double)tp / (double)(tp + fp);
else pr[i][rank].precision = 0;
if ((tp + fn) > 0) pr[i][rank].recall = (double)tp / (double)(tp + fn);
else pr[i][rank].recall = 0;
}
}
free(truth_flags);
double mean_average_precision = 0;
for (i = 0; i < classes; ++i) {
double avg_precision = 0;
int point;
for (point = 0; point < 11; ++point) {
double cur_recall = point * 0.1;
double cur_precision = 0;
for (rank = 0; rank < detections_count; ++rank)
{
if (pr[i][rank].recall >= cur_recall) { // > or >=
if (pr[i][rank].precision > cur_precision) {
cur_precision = pr[i][rank].precision;
}
}
}
printf("class_id = %d, point = %d, cur_recall = %.4f, cur_precision = %.4f \n", i, point, cur_recall, cur_precision);
avg_precision += cur_precision;
}
avg_precision = avg_precision / 11; // ??
printf("class_id = %d, name = %s, \t ap = %2.2f %% \n", i, names[i], avg_precision*100);
mean_average_precision += avg_precision;
}
printf("---------------------caculate end!!------------------------\n");
const float cur_precision = (float)tp_for_thresh / ((float)tp_for_thresh + (float)fp_for_thresh);
const float cur_recall = (float)tp_for_thresh / ((float)tp_for_thresh + (float)(unique_truth_count - tp_for_thresh));
const float f1_score = 2.F * cur_precision * cur_recall / (cur_precision + cur_recall);
printf(" for thresh = %1.2f, precision = %1.2f, recall = %1.2f, F1-score = %1.2f \n",
thresh_calc_avg_iou, cur_precision, cur_recall, f1_score);
printf(" for thresh = %0.2f, TP = %d, FP = %d, FN = %d, average IoU = %2.2f %% \n",
thresh_calc_avg_iou, tp_for_thresh, fp_for_thresh, unique_truth_count - tp_for_thresh, avg_iou * 100);
mean_average_precision = mean_average_precision / classes;
printf("\n mean average precision (mAP) = %f, or %2.2f %% \n", mean_average_precision, mean_average_precision*100);
for (i = 0; i < classes; ++i) {
free(pr[i]);
}
free(pr);
free(detections);
free(truth_classes_count);
fprintf(stderr, "Total Detection Time: %f Seconds\n", (double)(time(0) - start));
if (reinforcement_fd != NULL) fclose(reinforcement_fd);
}
4. 重新编译：
make clean
make
5. 终端输入命令：
./darknet detector map cfg/voc-hand.data cfg/yolov3-voc-hand-test.cfg backup/yolov3-voc-hand_final.weights
参考博客：
https://blog.csdn.net/sihaiyinan/article/details/87903923
mAP定义及相关概念:
-
mAP: mean Average Precision, 即各类别AP的平均值
-
AP: PR曲线下面积，后文会详细讲解
-
PR曲线: Precision-Recall曲线
-
Precision: TP / (TP + FP)
-
Recall: TP / (TP + FN)
-
TP: IoU>0.5的检测框数量（同一Ground Truth只计算一次）
-
FP: IoU<=0.5的检测框，或者是检测到同一个GT的多余检测框的数量
-
FN: 没有检测到的GT的数量，IoU=0
-
在VOC2010以前，只需要选取当Recall >= 0, 0.1, 0.2, …, 1共11个点时的Precision最大值，然后AP就是这11个Precision的平均值。
-
在VOC2010及以后，需要针对每一个不同的Recall值（包括0和1），选取其大于等于这些Recall值时的Precision最大值，然后计算PR曲线下面积作为AP值 。
举例计算mAP
有3张图如下，要求算法找出face。蓝色框代表标签label，绿色框代表算法给出的结果pre，旁边的红色小字代表置信度。设定第一张图的检出框叫pre1，第一张的标签框叫label1。第二张、第三张同理。
1.根据IOU计算TP,FP
首先我们计算每张图的pre和label的IOU，根据IOU是否大于0.5来判断该pre是属于TP还是属于FP。显而易见，pre1是TP，pre2是FP，pre3是TP。
2.排序
根据每个pre的置信度进行从高到低排序，这里pre1、pre2、pre3置信度刚好就是从高到低。
3.在不同置信度阈值下获得Precision和Recall
首先，设置阈值为0.9，无视所有小于0.9的pre。那么检测器检出的所有框pre即TP+FP=1，并且pre1是TP，那么Precision=1/1。因为所有的label=3，所以Recall=1/3。这样就得到一组P、R值。
然后，设置阈值为0.8，无视所有小于0.8的pre。那么检测器检出的所有框pre即TP+FP=2，因为pre1是TP，pre2是FP，那么Precision=1/2=0.5。因为所有的label=3，所以Recall=1/3=0.33。这样就又得到一组P、R值。
再然后，设置阈值为0.7，无视所有小于0.7的pre。那么检测器检出的所有框pre即TP+FP=3，因为pre1是TP，pre2是FP，pre3是TP，那么Precision=2/3=0.67。因为所有的label=3，所以Recall=2/3=0.67。这样就又得到一组P、R值。
4.绘制PR曲线并计算AP值
根据上面3组PR值绘制PR曲线如下。然后每个“峰值点”往左画一条线段直到与上一个峰值点的垂直线相交。这样画出来的红色线段与坐标轴围起来的面积就是AP值。
5.计算mAP
AP衡量的是对一个类检测好坏，mAP就是对多个类的检测好坏。就是简单粗暴的把所有类的AP值取平均就好了。比如有两类，类A的AP值是0.5，类B的AP值是0.2，那么mAP=（0.5+0.2）/2=0.35
1917
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
