loss盘点: asl loss (Asymmetric Loss) 代码解析详细版

1. BCE公式部分

可以简单浏览下这篇博客的文章：
https://blog.csdn.net/qq_14845119/article/details/114121003

这是多分类 经典 $BCELoss$ 公式
$$L=-y{L}_{+}-(1-y)L_{-}$$

其中，$L_{+/-}$ 是正负例预测概率的log值，即：

$$\begin{array}{rl}{L}_{+}& =log(\hat{y})\\ L_{-}& =log(1-\hat{y})\\ \hat{y}& =sigmoid(logit)\end{array}$$

实际上由于 $label$ 标签 $y$ 值，是一个 $0/1$ 矩阵，实际上充当了一个掩码 $mask$ 的作用，挑选出 $L_{+}$ 中正例部分 和 $L_{-}$ 中负例部分

假设:

$$y=\left[\begin{array}{cc}0& 0\\ 1& 0\end{array}\right]$$

$$\hat{y}=\left[\begin{array}{cc}0.5& 0.1\\ 0.3& 0.2\end{array}\right]L_{+}=\left[\begin{array}{cc}-0.6931& -2.3026\\ -1.2040& -1.6094\end{array}\right]L_{-}=\left[\begin{array}{cc}-0.6931& -0.1054\\ -0.3567& -0.2231\end{array}\right]$$

所以，$L$ 左下角为$L_{+}$对应的值的相反数，左上角和右上角和右下角为$L_{-}$对应的值的相反数

$$L=\left[\begin{array}{cc}0.6931& 0.1054\\ 1.2040& 0.2231\end{array}\right]$$

代码验证：

x = torch.tensor([0.5, 0.1, 0.3, 0.2]).reshape(2, 2).float()
y = torch.tensor([0, 0, 1, 0]).reshape(2, 2).float()
torch.nn.functional.binary_cross_entropy(x, y, reduction='none')

tensor([[0.6931, 0.1054],
        [1.2040, 0.2231]])

(不要小看这个 mask 代码的操作，一会儿写 asl 代码会用的上)

2. focal loss 公式部分

基本公式依旧是这个:
$$L=-y{L}_{+}-(1-y)L_{-}$$

$L_{+}$ 和 $L_{-}$ 如下：
$$\begin{array}{rl}{L}_{+}& =(1-p{)}^{\gamma }\ast log(p)\\ L_{-}& =p^{\gamma }\ast log(1-p)\\ p& =sigmoid(logit)\end{array}$$

3. asl 公式部分

asl loss 是 focal loss的改进版

$$\begin{array}{rlrl}{L}_{+}& =(1-p{)}^{{\gamma }_{+}}& \ast & log(p)\\ L_{-}& =p_m^{{\gamma }_{-}}& \ast & log(1-p_m)\\ p& =sigmoid(logit)& & \\ p_m& =max(p-m,0)& & \end{array}$$

由于 $p_m$ 仅在 $L_{-}$ 中存在，而$p$一般出现在$L_{+}$中，$(1-p)$一般出现在$L_{-}$中，所以将 $p_m$ 做一些反向操作

先引入一个引理，显然成立，x和y都是函数(或者变量)，二者中大的加上负号，就是二者相反数中小的

$$-max(x,y)==min(-x,-y)$$

所以：
$$\begin{array}{rl}{p}_m& =max(p-m,0)\\ & =-min(m-p,0)\\ -p_m& =min(m-p,0)\\ 1-p_m& =min(m-p,0)+1\\ 1-p_m& =min(m-p+1,1)\\ 1-p_m& =min(m+1-p,1)\\ 1-p_m& =np.clip(m+1-p,max=1)\end{array}$$

这一行咱等会要用到

4. asl 代码

看看 asl loss 的代码，torch代码来自：
https://github.com/Alibaba-MIIL/ASL/blob/main/src/loss_functions/losses.py

• self.gamma_neg 是 ${\gamma }_{-}$
• self.gamma_pos 是 ${\gamma }_{+}$
• self.eps 是用作 log 函数内部，防止溢出

class AsymmetricLossOptimized(nn.Module):
    ''' Notice - optimized version, minimizes memory allocation and gpu uploading,
    favors inplace operations'''

    def __init__(self, gamma_neg=4, gamma_pos=1, clip=0.05, eps=1e-8, disable_torch_grad_focal_loss=False):
        super(AsymmetricLossOptimized, self).__init__()

        self.gamma_neg = gamma_neg
        self.gamma_pos = gamma_pos
        self.clip = clip
        self.disable_torch_grad_focal_loss = disable_torch_grad_focal_loss
        self.eps = eps

        # prevent memory allocation and gpu uploading every iteration, and encourages inplace operations
        self.targets = self.anti_targets = self.xs_pos = self.xs_neg = self.asymmetric_w = self.loss = None

    def forward(self, x, y):
        """"
        Parameters
        ----------
        x: input logits
        y: targets (multi-label binarized vector)
        """

        self.targets = y
        self.anti_targets = 1 - y

        # 分别计算正负例的概率
        self.xs_pos = torch.sigmoid(x)
        self.xs_neg = 1.0 - self.xs_pos

        # 非对称裁剪
        if self.clip is not None and self.clip > 0:
            self.xs_neg.add_(self.clip).clamp_(max=1)  # 给 self.xs_neg 加上 clip 值

        # 先进行基本交叉熵计算
        self.loss = self.targets * torch.log(self.xs_pos.clamp(min=self.eps))
        self.loss.add_(self.anti_targets * torch.log(self.xs_neg.clamp(min=self.eps)))

        # Asymmetric Focusing
        if self.gamma_neg > 0 or self.gamma_pos > 0:
            if self.disable_torch_grad_focal_loss:
                torch.set_grad_enabled(False)
                
            # 以下 4 行相当于做了个并行操作
            self.xs_pos = self.xs_pos * self.targets
            self.xs_neg = self.xs_neg * self.anti_targets
            self.asymmetric_w = torch.pow(1 - self.xs_pos - self.xs_neg,
                                          self.gamma_pos * self.targets + self.gamma_neg * self.anti_targets)
            
            if self.disable_torch_grad_focal_loss:
                torch.set_grad_enabled(True)
            self.loss *= self.asymmetric_w

        return -self.loss.sum()

来咱单独看一下代码:

# 非对称裁剪
if self.clip is not None and self.clip > 0:
	self.xs_neg.add_(self.clip).clamp_(max=1)  # 给 self.xs_neg 加上 clip 值

这两行用于计算：
$$\begin{array}{rl}1-p_m& =np.clip(m+1-p,max=1)\end{array}$$

# 先进行基本交叉熵计算
self.loss = self.targets * torch.log(self.xs_pos.clamp(min=self.eps))
self.loss.add_(self.anti_targets * torch.log(self.xs_neg.clamp(min=self.eps)))

这两行用于计算红框部分：

注意 self.targets 和 self.anti_targets 都相当于掩码 mask 的作用，此处的 self.loss 矩阵的shape是和 self.targets 一样的 shape，不理解可以回忆一下 BCE公式部分 的计算

而前面的 幂 相当于权重，就是代码中的 self.asymmetric_w，也就是此处的：

self.asymmetric_w 是这样计算的，这部分很妙！

self.xs_pos = self.xs_pos * self.targets
self.xs_neg = self.xs_neg * self.anti_targets
self.asymmetric_w = torch.pow(1 - self.xs_pos - self.xs_neg,
                              self.gamma_pos * self.targets + self.gamma_neg * self.anti_targets)

插一句 torch.pow 该函数会将两个shape相同的张量的对应位置做幂运算，看这个例子

>>> x = torch.tensor([1, 2, 3, 4])
>>> y = torch.tensor([2, 2, 3, 1])
>>> torch.pow(x, y)
tensor([ 1,  4, 27,  4])

计算 self.asymmetric_w 时，只需将pow的 $x$ 参数对应位置写成 $(1-p)$ 或者 $p_m$，将pow的 $y$ 参数对应位置写成 ${\gamma }_{-}$ 或者 ${\gamma }_{+}$ 即可，先看简单的，$y$ 参数这里计算:

self.gamma_pos * self.targets + self.gamma_neg * self.anti_targets

也是通过 self.targets 的 mask 操作来进行的，而 $x$ 参数这样计算：

1 - self.xs_pos - self.xs_neg

当计算 $L_{+}$ 时，self.xs_neg==0，$x$ 参数对应位置就是 1 - self.xs_pos 即 (1-p)
当计算 $L_{-}$ 时，self.xs_pos==0，$x$ 参数对应位置就是 1 - self.xs_neg 即 $(1-(1-p_m))=p_m$

通过一个 torch.pow 巧妙的计算了 self.asymmetric_w NICE！

之后二者对应位置相乘即可

self.loss *= self.asymmetric_w

5. asl 代码 Paddle 实现

class AsymmetricLossOptimizedWithLogit(nn.Layer):
    ''' Notice - optimized version, minimizes memory allocation and gpu uploading,
    favors inplace operations'''

    def __init__(self, gamma_neg=4, gamma_pos=1, clip=0.05, eps=1e-5, disable_paddle_grad_focal_loss=False):
        super(AsymmetricLossOptimizedWithLogit, self).__init__()

        self.gamma_neg = gamma_neg
        self.gamma_pos = gamma_pos
        self.clip = clip
        self.disable_paddle_grad_focal_loss = disable_paddle_grad_focal_loss
        self.eps = eps

        self.targets = self.anti_targets = self.xs_pos = self.xs_neg = self.asymmetric_w = self.loss = None

    def forward(self, x, y, weights=None):
        """"
        Parameters
        ----------
        x: input logits
        y: targets (multi-label binarized vector)
        """

        self.targets = y
        self.anti_targets = 1 - y

        # Calculating Probabilities
        self.xs_pos = F.sigmoid(x)
        self.xs_neg = 1.0 - self.xs_pos

        # Asymmetric Clipping
        if self.clip is not None and self.clip > 0:
            # self.xs_neg.add_(self.clip).clip_(max=1)
            self.xs_neg = (self.xs_neg + self.clip).clip_(max=1)

        # Basic CE calculation
        self.loss = self.targets * paddle.log(self.xs_pos.clip(min=self.eps))
        self.loss.add_(self.anti_targets * paddle.log(self.xs_neg.clip(min=self.eps)))

        # Asymmetric Focusing
        if self.gamma_neg > 0 or self.gamma_pos > 0:
            if self.disable_paddle_grad_focal_loss:
                paddle.set_grad_enabled(False)
            self.xs_pos = self.xs_pos * self.targets
            self.xs_neg = self.xs_neg * self.anti_targets
            self.asymmetric_w = paddle.pow(1 - self.xs_pos - self.xs_neg,
                                            (self.gamma_pos * self.targets + \
                                            self.gamma_neg * self.anti_targets).astype("float32"))
            if self.disable_paddle_grad_focal_loss:
                paddle.set_grad_enabled(True)
            self.loss *= self.asymmetric_w
       
        if weights is not None:
            self.loss *= weights
            
        _loss = -self.loss.sum()

        return _loss
    
    
if __name__ == "__main__":
    
    np.random.seed(11070109)
    x = np.random.randn(3, 3)
    x = paddle.to_tensor(x).cast("float32")
    y = (x > 0.5).cast("float32")
    
    loss = AsymmetricLossOptimizedWithLogit()
    out = loss(x, y)