[reading notes] OpenFace: an open source facial behavior analysis toolkit

Baltru Aitis T, Robinson P, Morency L P. OpenFace: An open source facial behavior analysis toolkit[C]// IEEE Winter Conference on Applications of Computer Vision. IEEE, 2016:1-10.

homepage,  source code and paper

1. OpenFace toolkit funtions

(1) facial landmark detection

(2) head pose estimation

(3) facial action unit recognition

(4) eye-gaze estimation

2. OpenFace toolkit features

(1) open source tool

(2) real time performence

(3) state-of-the-art result

(4) able to run from a simple webcam

(5) easy integration with other applications and devices through a lightweight messaging system.

3. Aplication scenarios 

(1) for computer vision and machine learning researchers, 

(2) for affective computing community 

(3) for people interested in building interactive applications based on facial behavior analysis

4. OpenFace framework

4.1 Facial landmark detection and tracking

OpenFace uses the recently proposed Conditional Local Neural Fields (CLNF) [8] for facial landmark detection and tracking. CLNF is an instance of a Constrained Local Model (CLM) [16], that uses more advanced patch experts and optimization function. The two main components of CLNF are: 

(a) Point Distribution Model (PDM) which captures landmark shape variations; 

(b) patch experts which capture local appearance variations of each landmark. 

4.2. Head pose estimation

CLNF internally uses a 3D representation of facial landmarks and projects them to the image using orthographic camera projection. This allows us to accurately estimate the head pose once the landmarks are detected by solving the PnP problem.

4.3. Eye gaze estimation

CLNF framework is a general deformable shape registration approach so we use it to detect eye-region landmarks as well. The vector from the 3D eyeball center to the pupil location is our estimated gaze vector.

4.4. Action Unit detection

OpenFace AU intensity and presence detection module is based on a recent state-of-the-art AU recognition framework [7, 59].