Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Conv

Proposed Methodology

Heatmap Regression

将预测AU intensity vector的问题转换为预测multiple AU heatmaps

Fig.2给出了每一个AU的central location
Q：每一个点都应该由68个landmarks通过某些规则计算得到的吧，文中没有仔细说明

AU数量为$N$，对于第$i$个AU，其location的坐标为${\hat{x}}_i$，以${\hat{x}}_i$为中心作用Gaussian function，生成heatmap $g_i(x)$
$$g_i(x)=\frac{I}{2\pi {\sigma }^2}\exp \left(-\frac{{\Vert x-{\hat{x}}_i\Vert }_2^2}{2{\sigma }^2}\right)(1)$$
其中$I$表示AU intensity，$\sigma$是Gaussian function的标准差

网络输出的predicted heatmap为$h_i(x;w,b)$，$x$表示input facial image，$h$和$w$是网络的参数，于是求MSE作为监督损失
$$L_{MSE}=\underset{w,b}{\min }\sum _{i=1}^N\sum _x{\Vert h_i(x;w,b)-g_i(x)\Vert }_2^2(2)$$
由predicted heatmap转换为predicted label时，只需要取最大值即可（若某个AU intensity为0，那么predicted heatmap必然是全黑的）

SCC: Semantic Correspondence Convolution

Given the co-occurrences of different AU intensities, the semantic representations of feature maps are highly correlated in spatial distributions.

本文提出SCC来model the correlation among feature channels，其思想inspired by the dynamic graph convolutions in geometry modeling
注：Wang Y, Sun Y, Liu Z, et al. Dynamic Graph CNN for Learning on Point Clouds[J]. ACM Transactions on Graphics (TOG), 2019, 38(5): 1-12.

后续的SCC有点复杂，是参考TOG paper的

代码

数据预处理
https://github.com/EvelynFan/FAU/blob/master/functions.py#L41-L44

生成heatmap
https://github.com/EvelynFan/FAU/blob/master/model_graph.py#L110

网络定义graph
https://github.com/EvelynFan/FAU/blob/master/model_graph.py#L121