Notes for Label Smoothing and Mixup

Notes for popular regularization methods used in deep learning.

Label Smoothing

Label smoothing is a mechanism to regularize the classifier layer by estimating the marginalized effect of label-dropout during training.

Why needs Label Smoothing

In a classification task, our model computes the logit $z_i$ of each label for each training example $x$, then these logits will be normalized by $softmax$ to get the probability of each label k ∈ { 1 , . . . , K } k\in \{1,...,K\} k∈{ 1,...,K}
$$p(k|x)=\frac{\exp (z_k)}{{\sum }_{i=1}^K\exp (z_i)}.$$
Consider using a single ground-truth label $y$ and using cross-entropy loss:
$$loss=-\sum _{i=1}^K{q}_i\log p_i,$$

$$q_i=\{\begin{array}{l}1,if(i=y)\\ 0,if(i\ne y)\end{array}$$

In this case, minizing this cross entropy loss is equivalent to maximizing the log-likelihood of the correct label. This maximum is approached if $z_y\gg z_k$ for all $k\ne y$, that is, if the logit corresponding to the ground-truth label is much greater than all other logits, and the logit for incorrect label will approach $-\infty$, which is difficult for the model to output.

Let ∂ l ∂ z k = p k − q k = 0 {\partial l \over {\partial z_k}}=p_k - q_k=0