Openface代码阅读笔记(2)------LandmarkDetector 初始化、关键函数、论文公式与代码的对应关系

Openface代码阅读------LandmarkDetector::DetectLandmarksInVideo的算法公式和程序对应关系.

CLNF in OpenFace   (upgrade version of CLM framework) 

paper: CLNF 

code: OpenFace  

1. init detection according to arguments

step 1.

// 将 main 函数的输入参数 保存到 arguments 

vector<string> arguments = get_arguments(argc, argv);

step 2.

// 从 arguments 解析 与landmark detection相关的parameters，得到 det_parameters 
LandmarkDetector::FaceModelParameters det_parameters(arguments);
det_parameters.model_location = root/model_path
其中：
root 在 构造函数中设置为 path(arguments[0]).parent_path()，即执行exe文件所在的目录。
model_path的初始值在 FaceModelParameters::init() 中设置为 model_location = "model/main_clnf_general.txt";
调用exe时，可以用 -mloc 选项设置 model_path。

step 3.

// 根据 det_parameters 到相应地址读取 modules。
// The modules that are being used for tracking
LandmarkDetector::CLNF face_model(det_parameters.model_location);
默认情况下 det_parameters.model_location = "./model/main_clnf_general.txt"

将 main_clnf_general.txt 涉及的所有txt文件 展开如下：

main_clnf_general.txt 
---LandmarkDetector clnf_general.txt
------PDM pdms/In-the-wild_aligned_PDM_68.txt
---------# The mean values of the components (in mm)  ...
---------# The principal components (eigenvectors) of identity or combined identity and expression model ...
---------# The variances of the components (eigenvalues) of identity or combined identity and expression model ...

------Triangulations tris_68.txt
---------# Number of triangulations
---------# Triangulation 1
---------# Triangulation 2
---------# Triangulation ...
---------# Triangulation 7

------PatchesCCNF patch_experts/ccnf_patches_0.25_general.txt (Binary file)
------PatchesCCNF patch_experts/ccnf_patches_0.35_general.txt (Binary file)
------PatchesCCNF patch_experts/ccnf_patches_0.5_general.txt  (Binary file)

---LandmarkDetector_part model_inner/main_clnf_inner.txt inner 17 0 18 1 19 2 20 3 21 4 22 5 23 6 24 7 25 8 26 9 27 10 28 11 29 12 30 13 31 14 32 15 33 16 34 17 35 18 36 19 37 20 38 21 39 22 40 23 41 24 42 25 43 26 44 27 45 28 46 29 47 30 48 31 49 32 50 33 51 34 52 35 53 36 54 37 55 38 56 39 57 40 58 41 59 42 60 43 61 44 62 45 63 46 64 47 65 48 66 49 67 50
------LandmarkDetector clnf_inner.txt
---------PDM pdms/pdm_51_inner.txt
---------PatchesCCNF patch_experts/ccnf_patches_1.00_inner.txt (Binary file)

---LandmarkDetector_part model_eye/main_clnf_synth_left.txt left_eye_28 36 8 37 10 38 12 39 14 40 16 41 18
------LandmarkDetector clnf_left_synth.txt
---------PDM pdms/pdm_28_l_eye_3D_closed.txt
---------PatchesCCNF patch_experts/left_ccnf_patches_1.00_synth_lid_.txt
---------PatchesCCNF patch_experts/left_ccnf_patches_1.50_synth_lid_.txt

---LandmarkDetector_part model_eye/main_clnf_synth_right.txt right_eye_28 42 8 43 10 44 12 45 14 46 16 47 18
------LandmarkDetector clnf_right_synth.txt
---------PDM pdms/pdm_28_eye_3D_closed.txt
---------PatchesCCNF patch_experts/ccnf_patches_1.00_synth_lid_.txt
---------PatchesCCNF patch_experts/ccnf_patches_1.50_synth_lid_.txt

---FaceDetConversion haarAlign.txt
------3
------4
------6
------1.000000 0.000000 0.000000 0.000000 
------0.000000 1.000000 0.000000 0.000000 
------0.055858 0.229781 0.871248 0.892778 

---DetectionValidator detection_validation/validator_general_68.txt (Binary file)

2. call stack

detection_success = LandmarkDetector::DetectLandmarksInVideo(grayscale_image, face_model, det_parameters);

>	FeatureExtraction.exe!LandmarkDetector::CLNF::Fit(const cv::Mat_<unsigned char> & im, const cv::Mat_<float> & depthImg, const std::vector<int,std::allocator<int> > & window_sizes, const LandmarkDetector::FaceModelParameters & parameters) Line 759	C++
 	FeatureExtraction.exe!LandmarkDetector::CLNF::DetectLandmarks(const cv::Mat_<unsigned char> & image, const cv::Mat_<float> & depth, LandmarkDetector::FaceModelParameters & params) Line 571	C++
 	FeatureExtraction.exe!LandmarkDetector::DetectLandmarksInVideo(const cv::Mat_<unsigned char> & grayscale_image, const cv::Mat_<float> & depth_image, LandmarkDetector::CLNF & clnf_model, LandmarkDetector::FaceModelParameters & params) Line 362	C++
 	FeatureExtraction.exe!LandmarkDetector::DetectLandmarksInVideo(const cv::Mat_<unsigned char> & grayscale_image, LandmarkDetector::CLNF & clnf_model, LandmarkDetector::FaceModelParameters & params) Line 424	C++
 	FeatureExtraction.exe!main(int argc, char * * argv) Line 601	C++

3. Constrained Local Model(CLM) Face Landmark Detection

Our approach uses the Constrained Local Model (CLM) framework.

 There are three main parts to a CLM: 

Part A: a point distribution model (PDM),

           which is parameter p in formula (1). 

          calc PDM according to ground true landmarks in train dataset.

          OpenFace read PDM information from txt files.

本例中从如下文件中读取 PDM。

./model/pdms/In-the-wild_aligned_PDM_68.txt

Part B: patch experts, 

            which is used to calculate xi in formula (1).

            Refer to section 3.1. Local Neural Field patch expert.

LandmarkDetectorModel.cpp line:722

		// The patch expert response computation
		if(scale != window_sizes.size() - 1)
		{
			patch_experts.Response(patch_expert_responses, sim_ref_to_img, sim_img_to_ref, im, depth_img_no_background, pdm, params_global, params_local, window_size, scale);
		}

Part C: the fitting approach, 

            which is used to find optimized p* in formula (1).

            Refer to section 3.2. Non-uniform RLMS.

LandmarkDetectorModel.cpp line:759


// the actual optimisation step
		this->NU_RLMS(params_global, params_local, patch_expert_responses, cv::Vec6d(params_global), params_local.clone(), current_shape, sim_img_to_ref, sim_ref_to_img, window_size, view_id, true, scale, this->landmark_likelihoods, tmp_parameters);