Label smoothing and Mixup
Notes for popular regularization methods used in deep learning.
Label Smoothing
Label smoothing is a mechanism to regularize the classifier layer by estimating the marginalized effect of label-dropout during training.
Why needs Label Smoothing
In a classification task, our model computes the logit z i z_i zi of each label for each training example x x x, then these logits will be normalized by s o f t m a x softmax softmax to get the probability of each label k ∈ { 1 , . . . , K } k\in \{1,...,K\} k∈{
1,...,K}
p ( k ∣ x ) = exp ( z k ) ∑ i = 1 K exp ( z i ) . p(k|x) = {
{\exp(z_k)}\over {\sum_{i=1}^K\exp(z_i)}}. p(k∣x)=∑i=1Kexp(zi)exp(zk).
Consider using a single ground-truth label y y y and using cross-entropy loss:
l o s s = − ∑ i = 1 K q i log p i , loss = -\sum_{i=1}^Kq_i\log p_i, loss=−i=1∑Kqilogpi,
q i = { 1 , i f ( i = y ) 0 , i f ( i ≠ y ) q_i = \begin{cases} 1,\ if\ \ (i = y) \\ 0,\ if\ \ (i \ne y ) \end{cases} qi={ 1, if (i=y)0, if (i=y)
In this case, minizing this cross entropy loss is equivalent to maximizing the log-likelihood of the correct label. This maximum is approached if z y ≫ z k z_y \gg z_k zy≫zk for all k ≠ y k\ne y k=y, that is, if the logit corresponding to the ground-truth label is much greater than all other logits, and the logit for incorrect label will approach − ∞ -\infty −∞, which is difficult for the model to output.
Let ∂ l ∂ z k = p k − q k = 0 {\partial l \over {\partial z_k}}=p_k - q_k=0