Note: Feature Adaptation Network for Surveillance Face Recognition and Normalization

Target

This paper studies face recognition and normalization in surveillance imagery.

1. What is face normalization?
   Face normalization is a general task of generating an identity-preserved face while removing other non-identity variation including pose, expression, illumination and resolution. Most works of face normalization have focused on specifically removing pose variation, e.g. use affine transformation to keep the eyes horizontal, but side face is always side face. This paper integrate $\text{disentangled feature learning}$ to learn identity and non-identity features to help achieve face normalization for visualization, and identity preserving for face recognition, simultaneously
2. Data is important
   But we can’t always collect paired data to train model. This paper proposed a novel method which is suitable for both paired and unpaired data, and a random scale augmentation strategy.

Feature Adaptation Network

Here we must train 4 CNN models.

• Enc_H, for identity-preserve feature from high-resolution images (fixed)
• Enc_Z, for non-identity feature
• Dec, for generating a face image from features
• Enc_L, for identity-preserve feature from low-resolution images
• Dis, for discriminating

The overview of FAN

FAN consists of two stages: disentangled feature leaning and feature adaptation. Dark green is pre-trained and fixed, light green is trained for feature disentanglement. Orange represent the feature adaptation where a LR identity encoder is learned with all other models (green) fixed.

Disentangled Feature Learning

Enc_H which is trained with HR and LR images using standard softmax loss and m-$L_2$ regularization [2] remains fixed for all later stages. Then to learn non-identity features $z_h=En{c}_Z(x_h)$ by performing adversarial training and image reconstruction.

The disentangled features are combined to generate a face image $x_h^{\prime }=Dec(f_h,z_h)$. As $f_h$ is discriminative for face recognition, the non-identity components will be discarded from $f_h$ in the first step.

In this framework, we hope that if we set $z_h=0$, Dec can generate a normalized face image like

Paired and Unpaired Feature Adaptation

Aim to learn a feature extractor that works well for input faces with various resolutions. So this part is just to train Enc_L (another way for super resolution)
The important thing in this part is the scale augmentation strategy, Random Scale Augmentation (RSA).
Given a HR input $x_h\in {\mathcal{R}}^{N_h\times N_h}$, down-sample the image to a random resolution to obtain $x_l\in {\mathcal{R}}^{K\times K}$, where $K\in [N_l,N_h]$ and $N_l$ is the lowest pixel resolution.
(but I thought this is not a contribution)

Loss

Non-identity Loss

$$L_z=||FC(z_h)-yz|{|}_2^2$$
where $y_z=[\frac{1}{N_D},...,\frac{1}{N_D}]\in {\mathcal{R}}^{N_D}$ and $N_D$ is the total number of identities in the training set.

Pixel Loss

$$L_{dec}=||x_h^{\prime }-x_h|{|}_2^2$$

Identity Loss

$$K_{id}=||En{c}_H(x_h^{\prime })-f_h|{|}_2^2$$

GAN-based discriminator loss

use standard binary cross entropy classification loss

Low Resolution Feature Loss

$$L_{enc}=||En{c}_L(x_l)-En{c}_H(x_h)|{|}_2^2$$

SR Loss

$$L_{en{c}_{d}ec}=||Dec(f_l,En{c}_Z(x_h))-x_h|{|}_2^2$$

My Work about FAN

I don’t have m-$L_2$ loss and also enough GPU for training, so I employ Circle Loss for Enc_H. Then I removed the Dis and Enc_L, try to learn Enc_Z to verify if I can disentangle the non-identity feature. I get the results like

But I can’t normalize a side face!!! In addition, I found that if retain dropout layer in the Enc_H, the decoder can hardly generate a face image.

[1] Yin, Xi, et al. “Fan: Feature adaptation network for s
urveillance face recognition and normalization.” Proceedings of the Asian Conference on Computer Vision. 2020.
[2] Yin, Xi, et al. “Feature transfer learning for face recognition with under-represented data.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.