论文阅读：【Joint AU localisation and intensity estimation through heatmap regression】

论文阅读：【Joint AU localisation and intensity estimation through heatmap regression】

The approach aims to learn a pixel-wise regression function returning a score per AU, which indicates an AU intensity at a given spatial location.

Heatmap regression then generates an image, or channel, per AU. The amplitude and size of the Gaussian is determined by the intensity of the AU.

The Hourglass is made of residual blocks, downsampling the image to a low resolution and then upsampling to restore the image resolution, using skip connections to aggregate information at different scales.

The inference consists of simply performing the max operator on each of the heatmaps.

• both the localisation and the intensity estimation are encoded in the representation of the heatmaps.
• label augmentation: compensates the lack of annotated data, and increases the robustness of the models against appearance variations.
• treat the problem as a regression problem in the continuous range within 0 and 5.

Heatmap generation:

For a given AU a, each heatmap contains two or three Gaussians at each of its K selected centers $(c_x,c_y)$.
The heatmap for Action Unit a is finally given as $H_a(i,j)={\max }_k{G}_k(i,j)$.
$G_k(i+c_x,j+c_y)=\{\begin{array}{rlr}{I}_a{e}^{-\frac{(i^2+j^2)}{2{\sigma }^2}},& & if||i||,||j||\le 6I_a;||i||,||j||\in imagebounds\\ 0,& & otherwise\end{array}$,
// $G_k$: a 64×64 image;
// $I_a$: labeled intensity.

Data augmentation:

• Input augmentation (Landmark perturbation):
  perturb the landmarks using a small noise, then register the images according to the eye and mouth positions.
  // augments the training data to account for errors in the landmark localisation process, and implies a random rigid transformation.
• Label perturbation: $I_k\leftarrow 0.2I_k|z|,z\sim N(0,1)$,
  // $I_k$: the intensity of the k-th AU.
  // only affects the heatmap generation, not the loss function during the backpropagation process.
• Color perturbation:
  consists of applying a random perturbation to the RGB channels to prevent overfitting.

Loss function:

In heatmap regression, the loss is a per-pixel function between the predicted and the target heatmaps.
Use the smooth L1 norm (i.e., the Huber loss).
The total loss is computed as the average of the per-pixel loss per AU.
$L_{i,j}=\{\begin{array}{rlr}0.5(x_{i,j}-y_{i,j}{)}^2,& & if|x_{i,j}-y_{i,j}|<1\\ |x_{i,j}-y_{i,j}|-0.5,& & otherwise\end{array}$
// $x_{i,j}$: the output generated by the network at pixel (i, j);
// $y_{i,j}$: the corresponding ground-truth.