Openface (一)论文

OpenFace: an open source facial behavior analysis toolkit

1、Facial landmark detection and tracking：Conditional Local Neural Fields (CLNF) ，a general deformable shape registration approach
[Constrained local neural fields for robust facial landmark detection in the wild. In ICCVW, 2013.]
通过初始化平均脸的位置，然后让每个平均脸上的特征点在其邻域位置上进行搜索匹配来完成人脸点检测。
CLNF is an instance of a Constrained Local Model (CLM) ,that uses more advanced patch experts and optimization function.
[Feature detection and tracking with constrained local models. In BMVC, 2006]
The two main components of CLNF are:
1)Point Distribution Model (PDM) captures landmark shape variations; Trained on two datasets; model has 34 non-rigid and 6 rigid shape parameters.
2)patch experts capture local appearance variations of each landmark;Trained 28 patch experts for 7 views and 4 scales(lower or higher resolution).

2、Head pose estimation：
CLNF internally uses a 3D representation of facial landmarks and projects them to the image using orthographic camera projection.
This allows us to accurately estimate the head pose once the landmarks are detected by solving the PnP problem.(已知的n个空间3D点与图像2D点对应的点对，计算相机位姿、或者物体位姿)
为了准确的头部姿态估计，OpenFace需要提供相机校准参数(焦距和主点)。在没有它们的情况下，OpenFace使用基于图像大小的粗略估计.

3、Eye gaze estimation：
使用CLNF检测眼睑eyelids，虹膜iris和瞳孔pupil

使用眼睛和瞳孔的位置信息计算每只眼睛的眼睛注视向量。
我们从相机原点发射一束射线，通过成像平面的瞳孔中心，计算它与眼球的交点，获得3D摄像机坐标中的瞳孔位置。
从三维眼球中心到瞳孔位置的向量是我们估计的注视向量。

4、 Action Unit detection
[Cross-dataset learning and person-specic normalisation for automatic Action Unit detection]

OpenFace 2.0: Facial Behavior Analysis Toolkit
Facial landmark detection and tracking：
Convolutional Experts Constrained Local Model (CE-CLM) [Convolutional experts constrained local model for facial landmark detection. In CVPRW,2017.]
https://blog.csdn.net/taopanpantao/article/details/77109497

面部标记点检测方法可以分为两大类：
1)基于模型的方法:明确地模拟面部标记点的外观和形状，限制形状的搜索空间并提供一种正则化形式。基于约束局部模型及其各种扩展，找到与图像外观相匹配的面部模型的最佳参数。
2)基于回归的方法:不需要明确的形状模型，在外观上直接执行标记点检测。大多数采用级联回归框架