OpenPose 基于OpenCV DNN 的多人姿态估计

原文: OpenPose 基于OpenCV DNN 的多人姿态估计 - AIUAI

OpenPose 可以对图片中单个人体目标的姿态估计，也可以处理图片中多人的姿态估计.

OpenPose 基于OpenCV DNN 的单人姿态估计 - AIUAI

这里主要记录基于 OpenCV 4.x DNN 模块和 OpenPose 开源模型的多人人体姿态估计 的实现.

采用的模型的基于 COCO 数据集训练的人体关键点模型.

1. OpenPose 网络结构

OpenPose 的多人人体姿态估计的模型结构如图：

网络输入为 hxwx3 的图片，并输出包含关键点(Keypoints)的置信图(confidence maps) 和每个关键点对(keypoint pair) 的 Part Affinity Heatmaps 的两个输出数组.

[1] - Stage 0:

采用 VGGNet 的前 10 层，提取输入图片的特征图(feature maps).

[2] - Stage 1:

采用 2-分支的 multi-stage CNN 网络结构：

(1) - 分支一：

网络分支一预测人体关键点位置的 2D Confidence Maps ，如 elbow, knee, etc. 每个 Confidence Map 是一个灰度图(grayscale image)，其最大值的位置坐标即为对应人体某个关键点的概率最高. 如图：

(2) - 分支二：

网络分支二预测 Part Affinities (PAF) 的2D 向量场(L, vector fields)，其表示了两个关键点之间关联度(degree of association). 例如，关键点 Neck 和 Left Shoulder 之间的 Part Affinity, 如下图，属于同一个人体的关键点之间的 Affinity 值比较大.

总体来说，Confidence Maps 用于检测关键点位置；而 Affinity Maps 用于检测关键点之间的有效连接.

2. OpenPose 的 OpenCV DNN 实现

根据检测过程，主要涉及的函数有：getKeyponts() 、 getValidPairs() 和 getPersonwiseKeypoints() .

2.1. 模型加载

#!/usr/bin/python3
#!--*-- coding: utf-8 --*--
import cv2
import time
import numpy as np
import matplotlib.pyplot as plt


num_points = 18

keypointsMapping = ['Nose', 'Neck',
                    'R-Sho', 'R-Elb', 'R-Wr',
                    'L-Sho', 'L-Elb', 'L-Wr',
                    'R-Hip', 'R-Knee', 'R-Ank',
                    'L-Hip', 'L-Knee', 'L-Ank',
                    'R-Eye', 'L-Eye', 'R-Ear', 'L-Ear']

POSE_PAIRS = [[1,2], [1,5], [2,3], [3,4], [5,6], [6,7],
              [1,8], [8,9], [9,10], [1,11], [11,12], [12,13],
              [1,0], [0,14], [14,16], [0,15], [15,17],
              [2,17], [5,16] ]

# index of pafs correspoding to the POSE_PAIRS
# e.g for POSE_PAIR(1,2), the PAFs are located at indices (31,32) of output, Similarly, (1,5) -> (39,40) and so on.
mapIdx = [[31,32], [39,40], [33,34], [35,36], [41,42], [43,44],
          [19,20], [21,22], [23,24], [25,26], [27,28], [29,30],
          [47,48], [49,50], [53,54], [51,52], [55,56],
          [37,38], [45,46]]

colors = [[0,100,255], [0,100,255],   [0,255,255], 
          [0,100,255], [0,255,255],   [0,100,255],
          [0,255,0],   [255,200,100], [255,0,255], 
          [0,255,0],   [255,200,100], [255,0,255],
          [0,0,255],   [255,0,0],     [200,200,0], 
          [255,0,0],   [200,200,0],   [0,0,0]]

# dnn 加载模型
start = time.time()
prototxt = "./models/pose/coco/pose_deploy_linevec.prototxt"
caffemodel = "./models/pose/coco/pose_iter_440000.caffemodel"
net = cv2.dnn.readNetFromCaffe(prototxt, caffemodel)
print("[INFO]Time Taken in Model Loading: {}".format(time.time() - start))

# test img
img_cv2 = cv2.imread("test_image.jpg")
img_width, img_height = img_cv2.shape[1], img_cv2.shape[0]

# 根据长宽比，固定网路输入的 height，计算网络输入的 width.
net_height = 368
net_width = int((net_height/img_height)*img_width)

start = time.time()
in_blob = cv2.dnn.blobFromImage(
    img_cv2, 
    1.0 / 255, 
    (net_width, net_height),
    (0, 0, 0), 
    swapRB=False, 
    crop=False)

net.setInput(in_blob)
output = net.forward()
print("[INFO]Time Taken in Forward pass: {}".format(time.time() - start))

2.2. 关键点检测

2.2.1. getKeyponts( )函数

**getKeyponts( )函数功能：**对 Confidence Map 采用 NMS(Non Maximum Suppression) 来检测关键点.

def getKeypoints(probMap, threshold=0.1):
    mapSmooth = cv2.GaussianBlur(probMap,(3,3),0,0)
    mapMask = np.uint8(mapSmooth>threshold)
    
    keypoints = []
    # 1. 找出对应于关键点的所有区域的轮廓(contours)
    contours, hierarchy = cv2.findContours(
        mapMask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    
    # for each blob find the maxima
    # 对于每个关键点轮廓区域，找到最大值.
    for cnt in contours:
        # 2. 创建关键点的 mask；
        blobMask = np.zeros(mapMask.shape)
        blobMask = cv2.fillConvexPoly(blobMask, cnt, 1)
        # 3. 提取关键点区域的 probMap
        maskedProbMap = mapSmooth * blobMask
        # 4. 提取关键点区域的局部最大值.
        _, maxVal, _, maxLoc = cv2.minMaxLoc(maskedProbMap)
        keypoints.append(maxLoc + (probMap[maxLoc[1], maxLoc[0]],))

    return keypoints

2.2.2. 关键点 mapMask

threshold = 0.1
part = 1
probMap = output[0, part, :,:]
mapSmooth = cv2.GaussianBlur(probMap, (3, 3), 0, 0)
mapMask = np.uint8(mapSmooth > threshold)

plt.subplot(1, 3, 1)
plt.imshow(probMap)
plt.title("probMap")
plt.axis("off")
plt.subplot(1, 3, 2)
plt.imshow(mapSmooth)
plt.title("mapSmooth")
plt.axis("off")
plt.subplot(1, 3, 3)
plt.imshow(mapMask)
plt.title("mapMask")
plt.axis("off")
plt.show()

2.2.3. 关键点坐标值

threshold = 0.1
part = 1
probMap = output[0, part, :,:]
probMap = cv2.resize(probMap, (img_cv2.shape[1], img_cv2.shape[0]))
mapSmooth = cv2.GaussianBlur(probMap, (3, 3), 0, 0)
mapMask = np.uint8(mapSmooth > threshold)

keypoints = []
contours, hierarchy = cv2.findContours(mapMask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

for cnt in contours