WS-DAN:Weakly Supervised Data Augmentation Netowrk for Fine-Grained Visual Classification

See Better Before Looking Closer: Weakly Supervised Data Augmentation Netowrk for Fine-Grained Visual Classification
Paper PDF

Abstract

In practice, random data augmentation, such as random image cropping, is low-efficiency and might introduce many uncontrolled background noises. In this paper, they propose Weakly Supervised Data Augmentation Network (WS-DAN) to explore the potential of data augmentation. Specifically, for each training image, we first generate attention maps to represent the object’s discriminative parts by weakly supervised learning. Next, we augment the image guided by these attention maps, including attention cropping and attention dropping. The proposed WS-DAN improves the classification accuracy in two folds. In the first stage, images can be seen better since more discriminative parts’ features will be extracted. In the second stage, attention regions provide accurate location of object, which ensures our model to look at the object closer and further improve the performance.

In summary, the main contributions of this work are:

1. They propose Weakly Supervised Attention Learning to generate attention maps to represent the spatial distribution of discriminative object’s parts, And use BAP module to get the whole object feature by accumulating the object’s part feature.
2. Based on attention maps, they propose attention-guided data augmentation to improve the efficiency of data augmentation, including attention cropping and attention dropping. Attention cropping randomly crops and resizes one of the attention part to enhance the local feature representation. Attention dropping randomly erases one of the attention region out of the image in order to encourage the model to extract the feature from multiple discriminative parts.
3. They utilize attention maps to accurately locate the whole object and enlarge it to further improve the classification accuracy.

Innovation

1. Bilinear Attention Pooling(BAP)
2. Attention Regularization
3. Attention-guided Data Augmentation

Pipeline

The training process can be divided into two parts: Weakly Supervised Attention Learning and Attention-guided Data Augmentation:

Weakly Supervised Attention Learning

Spatial Representation

Attention maps A which is obtained from Feature maps F by convolutional function $f(\cdot )$ in Equ 1 . Each Attention map $A_k$ represents one of the object’s part or visual pattern, such as the head of a bird, the wheel of a car or the wing of an aircraft. Attention maps will be utilized to augment training data.
$$A=f(F)=\bigcup _{k=1}^M{A}_k\text{\hspace{1em}(1)}$$

Bilinear Attention Pooling(BAP)

They propose Bilinear Attention Pooling (BAP) to extract features from these parts are represented by Attention maps. We element-wise multiply feature maps F by each attention map $A_k$ in order to generate M part feature maps $F_k$, as shown in Equ 2
$$F_k=A_k\odot F(k=1,2,...,M)\text{\hspace{1em}(2)}$$

Then, They further extract discriminative local feature by additional feature extraction function $g(\cdot )$ such as Global Average Pooling (GAP), Global Maximum Pooling (GMP) or convolutions, in order to obtain $k_{th}$ attention feature $f_k$.
$$f_k=g(F_k)\text{\hspace{1em}(3)}$$

Object’s feature is represented by part feature matrix P which is stacked by these part features $f_k$.

$$P=\left(\begin{array}{c}g(a_1\odot F)\\ g(a_2\odot F)\\ ...\\ g(a_M\odot F)\end{array}\right)=\left(\begin{array}{c}{f}_1\\ f_2\\ ...\\ f_M\end{array}\right)\text{\hspace{1em}(4)}$$

Attention Regularization

For each fine-grained category, They expect that attention map $A_k$ can represent the same $k_{th}$object’s part. They penalize the variances of features that belong to the same object’s part, which means that part feature $f_k$ will get close to the a global feature center $c_k$ and attention map $A_k$ will be activated in the same $k_{th}$ object’s part. The loss function can be represented by $L_A$ in Equ 5.
$$L_A=\sum _{k=1}^M{\Vert f_k-c_k\Vert }_2^2\text{\hspace{1em}(5)}$$

$c_k$ wil updates by the Equ 6 from initialization zero.
$$c_k\leftarrow c_k+\beta (f_k-c_k)\text{\hspace{1em}(6)}$$

Attention-guided Data Augmentation

Random image cropping, is low-efficiency and a high percentage of them contain many background noises, which might lower the training efficiency, affect the quality of the extracted features and cancel out its benefits. Using Attention as guideline, the crop images may focus more on the target.

Augmentation Map

For each training image, they randomly choose one of its attention map $A_k$ to guide the data augmentation process, and normalize it as $k_{th}$ Augmentation Map $A_k^{\ast }$
$$A_k^{\ast }=\frac{A_k-min(A_k)}{max(A_k)-min(A_k)}\text{\hspace{1em}(7)}$$

Attention Cropping

Crop Mask $C_k$ from $A_k^{\ast }$ by setting element $A_k^{\ast }(i,j)$ which is greater than threshold ${\theta }_c$ to 1, and others to 0, as represented in Equ 8.

$$C_k(i,j)=\{\begin{array}{ll}1,& \text{ if }{A}_k^{\ast }(i,j)>{\theta }_c\\ 0,& \text{otherwise.}\end{array}\text{\hspace{1em}(8)}$$

We then find a bounding box Bk that can cover the whole
selected positive region of $C_k$ and enlarge this region from raw image as the augmented input data.

Attention Dropping

To encourage attention maps represent multiple discriminative object’s parts, they propose attention dropping. Specifically, they obtain attention Drop Mask $D_k$ by setting element $A_k^{\ast }(i,j)$ which is greater than threshold ${\theta }_d$ to 0, and others to 1, as shown in Equ 9

$$D_k(i,j)=\{\begin{array}{ll}1,& \text{ if }{A}_k^{\ast }(i,j)>{\theta }_d\\ 0,& \text{otherwise.}\end{array}\text{\hspace{1em}(9)}$$

Object Localization and Refinement

In the testing process, after the model outputs the coarse-stage classification result and corresponding attention maps for the raw image, we can predict the whole region of the object and enlarge it to predict fine-grained result by the same network model. Object Map $A_m$ that indicates the location of object is calculated by Equ 10.
$$A_m=\frac{1}{M}\sum _{k=1}^M{A}_k\text{\hspace{1em}(10)}$$
The final classification result is averaged by the coarse- grained prediction and fine-grained prediction. The detailed process of Coarse-to-Fine prediction is described as Algorithm below:

Experiments

Ablation:

Comparison with random data augmentation

Comparison with Stage-of-the-Art Methods