3D Human Pose Estimation Using Convolutional Neural Networks with 2D Pose Information(2016)

Keywords: human pose estimation, convolutional neural network, 2D-3D joint optimization

Abstract

We tackle the 3D human pose estimation task with end-to-end learning using CNNs.Relative 3D positions between one joint and the other joints are learned via CNNs.

两个创新点：

（1）we added 2D pose information to estimate a 3D pose from an image by concatenating 2D pose estimation result with the features from an image.
（2）we have found that more accurate 3D poses are obtained by combining information on relative positions with respect to multiple joints,instead of just one root joint.

Introduction

整体介绍Human Pose Estimation——由2D的CNN引入3D的CNN，将CNN的优势扩展到3D——总结【5】【6】【7】CNN网络的缺点，叙述增加2D信息的优势（From 2D pose information,undesirable 3D joint positions which generate unnatural human pose may be discarded）

Framework

we propose a simple yet powerful 3D human pose estimation framework based on the regression of joint positions using CNNs.We introduce two strategies to improve the regression results from the baseline CNNs.

（1）not only the image features but also 2D joint classification results are used as input features for 3D pose estimation——this scheme successfully incorporates the correlation between 2D and 3D poses
（2）rather than estimating relative positions with respect to multiple joints——this scheme effectively reduces the error of the joints that are far from the root joint

Related Work

主要介绍基于CNN的2D和3D Human Pose Estimation（详见原文）。
The method proposed in this paper aims to provide an end-to-end learning framework to estimate 3D structure of a human body from a single image.Similar to 【5】,3D and 2D pose information are jointly learned in a single CNN.Unlike the previous works,we directly propagate the 2D classification results to the 3D pose regressors inside the CNNs.

3D-2D Joint Estimation of Human Body Using CNN

The key idea of our method is to train CNN which performs 3D pose estimation using both image features from the input image and 2D pose information retrieved from the same CNN.In other words,the proposed CNN is trained for both 2D joint clasification and 3D joint regression tasks simultaneously.

Structure of the Baseline CNN

The CNN used in this experiment consists of five convolutional layers, three pooling layers, two parallel sets of two fully connected layers, and loss layers for 2D and 3D pose estimation tasks. The CNN accepts a 225 × 225 sized image as an input. The sizes and the numbers of filters as well as the strides are specified in Figure 1. The filter sizes of convolutional and pooling layers are the same as those of ZFnet 【21】, but we reduced the number of feature maps to make the network smaller.
We divided an input image into $N_g\times N_g$grids and treat each grid as a separate class,which results in $N_g^2$classes per joint.The target probability fot the $ith$ grid $g_i$ of the $jth$ joint is inversely proportional to the distance from the ground truth position.
$${\hat{p}}_j(g_i)=\frac{d^{-1}({\hat{y}}_j,c_i)I(g_i)}{{\sum }_{k=1}^{N_g^2}{d}^{-1}({\hat{y}}_j,c_k)I(g_k)}(1)$$
——$d^{-1}(x,y)$ is the inverse of the Euclidean distance between the point x and y in the 2D pixel space,${\hat{y}}_j$ is the ground truth position of the $jth$ joint in the image,and $c_i$ is the center of the grid $g_i$.
$I(g_i)$ is an indicator function that is equal to 1 if the grid $g_i$ is one of the four nearest neighbors.
$$I(g_i)=\{\begin{array}{l}1ifd({\hat{y}}_j,c_i)<{\omega }_g\\ ootherwise,\end{array}(2)$$
——${\omega }_g$ is the width.Hence,higher probability is assigned to the grid closer to the ground truth joint positon,and ${\hat{p}}_j(g_i)$ is normalized so that the sum og the class probabilities is equal to 1.Finally,the objective of the 2D classification task for the $jth$ join is to minimize the following cross entropy loss function.
$$L_{2D}(j)=-\sum _{i=1}^{N_g^2}{\hat{p}}_j(g_i)log{p}_j(g_i),(3)$$
——$p_j(g_i)$ is the probability that comes from the softmax output of the CNN.
Estimating 3D position of joints is formulated as a regression task.Since the search space is much larger than the 2D case,it is undersirable to solve 3D pose estimation as a classification task.The 3D loss funcion is designed as a square of the Euclidean distance between the prediction and the ground truth.We estimate 3D position of each joint relative to the root node.the loss function for the $jth$ joint when the root node is the $rth$ joint becomes
$$L_{3D}(j,r)=||R_j-({\hat{J}}_j-{\hat{J}}_r)|{|}^2(4)$$
——$R_j$ is the predicted relative 3D position of the $jth$ joint from the root node,${\hat{J}}_j$ is the ground truth 3D position of the $jth$ joint,and ${\hat{J}}_r$ is that of the root node.The overall cost function of the CNN combines (3) and (4) with weights,
$$L_{all}={\lambda }_{2D}\sum _{j=1}^{N_j}{L}_{2D}(j)+{\lambda }_{3D}\sum _{j\ne r}^{N_j}{L}_{3D}(j,r)(5)$$

3D Joint Regression with 2D Classification Features

详见原文。The joint locations in an image are usually a strong cue of guessing 3D pose.To exploit 2D classification result for the 3D pose estimation,we concatenate the outputs of softmax in the 2D classification task with the outputs of the fully connected layers in the 3D loss part.The proposed structure after the last pooling layer is shown in Figure(2).First,the 2D classification result is concatenated($probs2DlayerinFigure2$)and passes the fully connected layer($fcprobs2D$).Then,the feature vectors from 2D and 3D part are concatenated($fc2D-3D$),which is used for 3D pose estimation task.Note that the error from the fc probs 2D layer is not back-propagated to the probs 2D layer to ensure that layers used for the 2D classification are trained only by the 2D loss part.【3】repeatedly uses the 2D classification result as an input by concatenating it with feature maps from CNN.we simply vectorized the softmax result to produce $N_g\times N_g\times N_j$ feature vector rather than convolving the probability map with features in the convolutional layers.

Multiple 3D Pose Regression from Different Root Nodes

介绍基础框架及其缺点，【5】提出一种解决办法，并介绍了【5】的缺点，进而提出本文的方法：we estimate the relative position over multiple joints.（基础框架计算各关节与根关节的相对位置，缺点是距离越远，精度越低。【5】提出计算各个节点与父节点之间的相对位置，缺点是中间节点的误差会累积）。令$N_r$为选择的根节点的数目。实验中令$N_r=6$可以使得大部分关节或者是根节点或者是邻居节点，可视化如图3(b)。6个3D regression losses如图2。整体误差为
$$L_{a}ll={\lambda }_{2D}\sum _{j=1}^{N_j}{L}_{2D}(j)+{\lambda }_{3D}\sum _{r\in R}\sum _{j\ne r}^{N_j}{L}_{3D}(j,r)(6)$$
——$R$ is the set containing the joint indices that are used as root nodes.When the 3D losses share the same fully connected layers,the trained model outputs the same pose estimation results across all joints.To break this symmetry,we put the fully connected layers for each 3D losses($fc2layersinFigure2$)
At the test time,all the pose estimation results are translated so that the mean of each pose bacomes zero.Final prediction is generated by averaging the translated results.In other words,the 3D position of the $jth$ joint $X_j$ is calculated as
$$X_j=\frac{{\sum }_{r\in R}{X}_j^{(r)}}{N_r}(7)$$
——$X_j^{(r)}$ is the predicted 3D position of the $jth$ joint when the $rth$ joint is set to a root node.

Implementation Details

详见原文

Conclusions

We expect that the perfprmance can be further improved by incorporating temporal information to the CNN by applying the concepts of recurrent neural network or 3D convolution[26].Also,efficient aligning method for multiple regression results may boost the accuracy of pose estimation.