人脸检测（二）

2.级联CNN人脸检测方法
采用级联网络来进行人脸检测，参考2015年CVPR上的一篇论文A Convolution Neural Network Cascade for Face Detection，它采用了12-net，24-net，48-net级联网络用于人脸检测，12-calibration-net，24-calibration，48-calibration边界校订网络用于更好的定位人脸框。它最小能够检测12x12大小的人脸，相比于单个CNN的人脸检测方法，大大加快了人脸检测的速度，并提高了人脸框的准确度，人脸检测的准确率和召回率也很高，在FDDB数据集上达到了当时最高的分数。

论文下载地址
github开源代码
github开源模型

作者的代码写的非常优美易懂，注释简洁明了，其级联CNN人脸检测的基本思路如下：

（1）12-net：首先用一个输入为12 x 12图像的小网络来训练人脸非人脸二分类器，将最后的全连接层修改成卷积层，这样的全卷积网络即12-full-conv-net就可以接受任意大小的输入图像，输出的特征图表示对应感受野区域属于人脸的概率。在检测时，假设要检测的最小人连尺寸为K x K，例如40 x 40，将待检测的图像缩放到原来的12/K，然后将整幅图像输入到训练好的12 x 12的全卷积网络，得到特征图，设定阈值过滤掉得分低的，这样就可以去除绝大部分不感兴趣区域，保留一定数量的候选框。
（2）12-calibration-net：训练一个输入为12 x 12图像的校订网络，来矫正上一步12-net得到的人脸框边界，它实质上是一个45类分类器，判断当前图像中包含的人脸是左右偏了、上下偏了还是大了小了，即包括：
x方向上-0.17、0、0.17共3种平移
y方向上-0.17、0、0.17共3三种平移
以及0.83、0.91、1.0、1.1、1.21共5种缩放尺度
检测时，将上一步12-net得到的所有人脸候选框作为12-calibration-net的输入，根据其分类结果对候选框位置进行校订。
（3）localNMS：对上述12-calibration-net矫正后的人脸候选框做局部非极大值抑制，过滤掉其中重叠的得分较低的候选框，保留得分更高的人脸候选框。

（4）24-net：训练输入为24 x 24图像的人脸分类器网络。测试时，以上一步localNMS得到的人脸候选框缩放到24 x 24大小，作为24-net网络输入，判定是否属于人脸，设置阈值保留得分较高的候选框。
（5）24-calibration-net：同样训练一个输入图像为24 x 24大小的边界校订分类网络，来矫正上一步24-net保留的人脸候选框的位置，候选框区域图像缩放到24 x 24大小，其它与12-calibration-net一致。
（6）localNMS：将24-calibration-net矫正后的人脸候选框进行局部非极大值抑制，过滤掉重叠的得分较低的候选框，保留得分更高的。

（7）48-net：训练一个更加准确的输入为48 x 48的人脸非人脸分类器。测试时，将上一步localNMS得到的人脸候选框缩放到48 x 48大小，作为48-net输入，保留得分高的人脸候选框。
（8）globalNMS：将48-net得到的所有人脸候选框进行全局非极大值抑制，保留所有的最佳人脸框
（9）48-calibration-net：训练一个输入为48 x 48的边界校订分类网络。测试时，将globalNMS得到的最佳人脸框缩放到48 x 48作为输入进行人脸框边界校订。

因此，我们需要先单独训练人脸非人脸二分类12-net，24-net，48-net网络，以及人脸框边界校订12-calibration-net，24-calibration，48-calibration网络。测试时，用图像金字塔来做多尺度人脸检测，对于任意的输入图像，依次经过12-net(12-full-conv-net) -> 12-calibration-net -> localNMS -> 24-net -> 24-calibration-net -> localNMS -> 48-net -> globalNMS -> 48-calibration-net，得到最终的人脸框作为检测结果。
其中localNMS和globalNMS(Non-maximum Suppression,NMS,非极大值抑制)的区别主要在于前者仅使用了IoU(Intersection over Union)，即交集与并集的比值，而后者还用到了IoM(Intersection over Min-area)，即交集与两者中最小面积的比值，来过滤重叠的候选框。
博主没有自己训练，而是直接用了作者训练好的模型。使用CNN_face_detection-master\face_detection文件夹下的face_cascade_fullconv_single_crop_single_image.py脚本可以对但张图像进行测试，代码如下，其中需要注意的是，我们可以根据需求来设置检测最小的人脸尺寸min_face_size，它与检测速度直接相关，如果我们需要检测的人脸比较大，例如在128x128以上时，检测可以达到实时水平。

import numpy as np
import cv2
import time
import os
#from operator import itemgetter
from load_model_functions import *
from face_detection_functions import *

# ==================  caffe  ======================================
#caffe_root = '/home/anson/caffe-master/'  # this file is expected to be in {caffe_root}/examples
import sys
#sys.path.insert(0, caffe_root + 'python')
import caffe
# ==================  load models  ======================================
net_12c_full_conv, net_12_cal, net_24c, net_24_cal, net_48c, net_48_cal = \
    load_face_models(loadNet=True)

nets = (net_12c_full_conv, net_12_cal, net_24c, net_24_cal, net_48c, net_48_cal)

start_time = time.time()

read_img_name = 'C:/Users/Administrator/Desktop/caffe/matlab/demo/1.jpg'
img = cv2.imread(read_img_name)     # BGR

print img.shape

min_face_size = 48 #最小的人脸检测尺寸，设置的越小能检测到更小人脸的同时，速度下降的很快
stride = 5 #步长，实际并未使用

# caffe_image = np.true_divide(img, 255)      # convert to caffe style (0~1 BGR)
# caffe_image = caffe_image[:, :, (2, 1, 0)]
img_forward = np.array(img, dtype=np.float32)
img_forward -= np.array((104, 117, 123))

rectangles = detect_faces_net(nets, img_forward, min_face_size, stride, True, 2, 0.05)
for rectangle in rectangles:    # draw rectangles
        cv2.rectangle(img, (rectangle[0], rectangle[1]), (rectangle[2], rectangle[3]), (255, 0, 0), 2)

end_time = time.time()
print 'aver_time = ',(end_time-start_time)*1000,'ms'

cv2.imshow('test img', img)
cv2.waitKey(0)
cv2.destroyAllWindows()

上述脚本文件中的detect_faces_net()函数在CNN_face_detection-master\face_detection文件夹下的face_detection_functions.py脚本中实现，face_detection_functions.py是整个级联网络的检测过程脚本，写的非常优雅，思路和注释清晰，本人针对自己需求进行了修改，代码如下：

import numpy as np
import cv2
import time
from operator import itemgetter
# ==================  caffe  ======================================
#caffe_root = '/home/anson/caffe-master/'  # this file is expected to be in {caffe_root}/examples
import sys
#sys.path.insert(0, caffe_root + 'python')
import caffe

def find_initial_scale(net_kind, min_face_size):
    '''
    :param net_kind: what kind of net (12, 24, or 48)
    :param min_face_size: minimum face size
    :return:    returns scale factor
    '''
    return float(min_face_size) / net_kind
def resize_image(img, scale):
    '''
    :param img: original img
    :param scale: scale factor
    :return:    resized image
    '''
    height, width, channels = img.shape
    new_height = int(height / scale)     # resized new height
    new_width = int(width / scale)       # resized new width
    new_dim = (new_width, new_height)
    img_resized = cv2.resize(img, new_dim)      # resized image
    return img_resized
def draw_rectangle(net_kind, img, face):
    '''
    :param net_kind: what kind of net (12, 24, or 48)
    :param img: image to draw on
    :param face: # list of info. in format [x, y, scale]
    :return:    nothing
    '''
    x = face[0]
    y = face[1]
    scale = face[2]
    original_x = int(x * scale)      # corresponding x and y at original image
    original_y = int(y * scale)
    original_x_br = int(x * scale + net_kind * scale)    # bottom right x and y
    original_y_br = int(y * scale + net_kind * scale)
    cv2.rectangle(img, (original_x, original_y), (original_x_br, original_y_br), (255,0,0), 2)
def IoU(rect_1, rect_2):
    '''
    :param rect_1: list in format [x11, y11, x12, y12, confidence, current_scale]
    :param rect_2:  list in format [x21, y21, x22, y22, confidence, current_scale]
    :return:    returns IoU ratio (intersection over union) of two rectangles
    '''
    x11 = rect_1[0]    # first rectangle top left x
    y11 = rect_1[1]    # first rectangle top left y
    x12 = rect_1[2]    # first rectangle bottom right x
    y12 = rect_1[3]    # first rectangle bottom right y
    x21 = rect_2[0]    # second rectangle top left x
    y21 = rect_2[1]    # second rectangle top left y
    x22 = rect_2[2]    # second rectangle bottom right x
    y22 = rect_2[3]    # second rectangle bottom right y
    x_overlap = max(0, min(x12,x22) -max(x11,x21))
    y_overlap = max(0, min(y12,y22) -max(y11,y21))
    intersection = x_overlap * y_overlap
    union = (x12-x11) * (y12-y11) + (x22-x21) * (y22-y21) - intersection
    return float(intersection) / union
def IoM(rect_1, rect_2):
    '''
    :param rect_1: list in format [x11, y11, x12, y12, confidence, current_scale]
    :param rect_2:  list in format [x21, y21, x22, y22, confidence, current_scale]
    :return:    returns IoM ratio (intersection over min-area) of two rectangles
    '''
    x11 = rect_1[0]    # first rectangle top left x
    y11 = rect_1[1]    # first rectangle top left y
    x12 = rect_1[2]    # first rectangle bottom right x
    y12 = rect_1[3]    # first rectangle bottom right y
    x21 = rect_2[0]    # second rectangle top left x
    y21 = rect_2[1]    # second rectangle top left y
    x22 = rect_2[2]    # second rectangle bottom right x
    y22 = rect_2[3]    # second rectangle bottom right y
    x_overlap = max(0, min(x12,x22) -max(x11,x21))
    y_overlap = max(0, min(y12,y22) -max(y11,y21))
    intersection = x_overlap * y_overlap
    rect1_area = (y12 - y11) * (x12 - x11)
    rect2_area = (y22 - y21) * (x22 - x21)
    min_area = min(rect1_area, rect2_area)
    return float(intersection) / min_area
def localNMS(rectangles):
    '''
    :param rectangles:  list of rectangles, which are lists in format [x11, y11, x12, y12, confidence, current_scale],
                        sorted from highest confidence to smallest
    :return:    list of rectangles after local NMS
    '''
    result_rectangles = rectangles[:]  # list to return
    number_of_rects = len(result_rectangles)
    threshold = 0.3     # threshold of IoU of two rectangles
    cur_rect = 0
    while cur_rect < number_of_rects - 1:     # start from first element to second last element
        rects_to_compare = number_of_rects - cur_rect - 1      # elements after current element to compare
        cur_rect_to_compare = cur_rect + 1    # start comparing with element after current
        while rects_to_compare > 0:      # while there is at least one element after current to compare
            if (IoU(result_rectangles[cur_rect], result_rectangles[cur_rect_to_compare]) >= threshold) \
                    and (result_rectangles[cur_rect][