[Paper Notes] [CVPR2019] Hand + Object (H+O)

Paper information

H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+o: Unified egocentric recognition of 3d hand-object poses and interactions. In CVPR, pages 4511–4520, 2019.

Introduction

Target

Propose a unified method to recognize 3D hand-object poses, and their interactions from egocentric（单视角） monocular（单眼） color images.

Limitations of previous work

• rely on active depth sensors or multi-camera systems
• do not reason about the action the subject is performing

Contributions

• a unified framework for recognizing 3D hand-object interactions.
• a novel single-shot neural network framework that jointly solves for the 3D articulated（骨架） and rigid（刚体） pose estimation problems within the same architecture.
• a temporal model to merge and propagate information in the temporal domain, explicitly model interactions and infer relations between hand and objects, directly in 3D.

Method

Notation	Meaning
${\text{I}}^t(1\le t\le N)$	input color frames sequence
$N_c$	number of 3D control points
$N_a$	number of actions
$N_o$	number of object classes