多分类多标签模型的评估方式（定义+numpy代码实现）

一、Multi-Class Multi-Label问题定义

所谓多分类（Multi-Class）是区别于二分类的一个概念，在二分类问题当中，数据的标签只是0，1二值类型，比如“是否”是一只狗，“是否”患病。而多分类则对应于更多的类别，比如判断物体是猫，狗，鸟，兔…判断病人患的是A，B，C，D中的某一种病。值得注意的是，多分类问题中常常只有一个类别是正确的。

什么是多标签（Multi-Label）呢？简单来说，就是一个样本同时具有多个标签，例如一张风景图，里面有天空、猫、狗、鸟、树，如果这些类别都属于当前任务所需要识别的类别之内，那么它就具有多个标签。显然，多标签任务的难度要高的多。

二、评估方式

参考[1] [2]，多分类多标签模型的评估指标通常分为两大类： example-based metrics, label-based metrics。

Example-based Metrics

1. Subset accuracy
   $$subsetacc(h)=\frac{1}{p}\sum _{i=1}^{p}I\left[h\left(x_i\right)=Y_i\right]$$

其中$h(\cdot )$指代一个多标签分类器$h:X\to 2^Y$，$h(x)$返回预测的标签集合，$p$为样本个数。

# gt为真实标签，predict为预测标签
# 形式例如：gt=[[1,0,0,1]], predict=[[1,0,1,1]]
def example_subset_accuracy(gt, predict):
    ex_equal = np.all(np.equal(gt, predict), axis=1).astype("float32")
    return np.mean(ex_equal)

2. Example accuracy
   $${\mathrm{Accuracy}}_{\mathrm{exam}}(h)=\frac{1}{p}\sum _{i=1}^p\frac{\left|Y_i\cap h\left(x_i\right)\right|}{\left|Y_i\cup h\left(x_i\right)\right|}$$

def example_accuracy(gt, predict):
    ex_and = np.sum(np.logical_and(gt, predict), axis=1).astype("float32")
    ex_or = np.sum(np.logical_or(gt, predict), axis=1).astype("float32")
    return np.mean(ex_and / (ex_or+epsilon))

3. Example precision
   $${\text{ Precision }}_{\text{exam }}(h)=\frac{1}{p}\sum _{i=1}^p\frac{\left|Y_i\cap h\left(x_i\right)\right|}{\left|h\left(x_i\right)\right|}$$

def example_precision(gt, predict):
    ex_and = np.sum(np.logical_and(gt, predict), axis=1).astype("float32")
    ex_predict = np.sum(predict, axis=1).astype("float32")
    return np.mean(ex_and / (ex_predict + epsilon))

4. Example recall
   $${\text{ Recall }}_{\text{e}xam}(h)=\frac{1}{p}\sum _{i=1}^p\frac{\left|Y_i\cap h\left(x_i\right)\right|}{\left|Y_i\right|}$$

def example_recall(gt, predict):
    ex_and = np.sum(np.logical_and(gt, predict), axis=1).astype("float32")
    ex_gt = np.sum(gt, axis=1).astype("float32")
    return np.mean(ex_and / (ex_gt + epsilon))

5. Example F1 (带$\beta$)
   $\beta >0$度量查全率(recall)对查准率(precision)的相对重要性，$\beta =1$时退化为标准的F1，$\beta >1$时查全率有更大影响， $\beta <1$时查准率有大更影响。
   $$F_{\text{exam }}^{\beta }(h)=\frac{\left(1+{\beta }^2\right)\cdot {\text{ Precsion }}_{\text{exam }}(h)\cdot {\text{ Recall }}_{\text{exam }}(h)}{{\beta }^2\cdot {\text{ Precision }}_{\text{exam }}(h)+{\text{ Recall }}_{\text{exam }}(h)}$$

def example_f1(gt, predict, beta=1):
    p = example_precision(gt, predict)
    r = example_recall(gt, predict)
    return ((1+beta**2) * p * r) / ((beta**2)*(p + r + epsilon))

Label-based Metrics

在计算label-based metrics之前，需要计算所需的基本统计量

• $TP,TN,FP,FN$的计算
  T P j = ∣ { x i ∣ y j ∈ Y i ∧ y j ∈ h ( x i ) , 1 ≤ i ≤ p } ∣ F P j = ∣ { x i ∣ y j ∉ Y i ∧ y j ∈ h ( x i ) , 1 ≤ i ≤ p } ∣ T N j = ∣ { x i ∣ y j ∉ Y i ∧ y j ∉ h ( x i ) , 1 ≤ i ≤ p } ∣ F N j = ∣ { x i ∣ y j ∈ Y i ∧ y j ∉ h ( x i ) , 1 ≤ i ≤ p } ∣ \begin{array}{l}T P_{j}=\left|\left\{x_{i} \mid y_{j} \in Y_{i} \wedge y_{j} \in h\left(x_{i}\right), 1 \leq i \leq p\right\}\right| \\ F P_{j}=\left|\left\{x_{i} \mid y_{j} \notin Y_{i} \wedge y_{j} \in h\left(x_{i}\right), 1 \leq i \leq p\right\}\right| \\ T N_{j}=\left|\left\{x_{i} \mid y_{j} \notin Y_{i} \wedge y_{j} \notin h\left(x_{i}\right), 1 \leq i \leq p\right\}\right| \\ F N_{j}=\left|\left\{x_{i} \mid y_{j} \in Y_{i} \wedge y_{j} \notin h\left(x_{i}\right), 1 \leq i \leq p\right\}\right|\end{array} TPj​=∣{xi​∣yj​∈Yi​∧yj​∈h(xi​),1≤i≤p}∣FPj​=∣{xi​∣yj​∈/​Yi​∧yj​∈h(xi​),1≤i≤p}∣TNj​=∣{xi​∣yj​∈/​Yi​∧yj​∈/​h(xi​),1≤i≤p}∣FNj​=∣{xi​∣yj​∈Yi​∧yj​∈/​h(xi​),1≤i≤p}∣​

其中$p$代表样本个数，$y_j$代表第$j$个类别的真实标签，$T{P}_j,F{P}_j,T{N}_j,F{N}_j$四类基本参数代表各自类别的二元分类性能，满足$T{P}_j+F{P}_j+T{N}_j+F{N}_j=p$。

def _label_quantity(gt, predict):
    tp = np.sum(np.logical_and(gt, predict), axis=0)
    fp = np.sum(np.logical_and(1-gt, predict), axis=0)
    tn = np.sum(np.logical_and(1-gt, 1-predict), axis=0)
    fn = np.sum(np.logical_and(gt, 1-predict), axis=0)
    return np.stack([tp, fp, tn, fn], axis=0).astype("float")

• Accuracy, Precision, Recall,F1的计算
  $$\begin{array}{c}\text{ Accuracy }\left(T{P}_j,F{P}_j,T{N}_j,F{N}_j\right)=\frac{T{P}_j+T{N}_j}{T{P}_j+F{P}_j+T{N}_j+F{N}_j}\\ \text{ Precision }\left(T{P}_j,F{P}_j,T{N}_j,F{N}_j\right)=\frac{T{P}_j}{T{P}_j+F{P}_j}\\ \mathrm{Recall}\left(T{P}_j,F{P}_j,T{N}_j,F{N}_j\right)=\frac{T{P}_j}{T{P}_j+F{N}_j}\\ F^{\beta }\left(T{P}_j,F{P}_j,T{N}_j,F{N}_j\right)=\frac{\left(1+{\beta }^2\right)\cdot T{P}_j}{\left(1+{\beta }^2\right)T{P}_j+{\beta }^2\cdot F{N}_j+F{P}_j}\end{array}$$

• Marco平均与Micro平均

  $$\begin{gathered} B_{\mathrm{macro}}(h)=\frac{1}{q}\sum _{j=1}^{q}B\left(T{P}_j,F{P}_j,T{N}_j,F{N}_j\right) \\ {B}_{\text{micro }}(h)=B\left(\sum _{j=1}^{q}T{P}_j,\sum _{j=1}^{q}F{P}_j,\sum _{j=1}^{q}T{N}_j,\sum _{j=1}^{q}F{N}_j\right) \end{gathered}$$
  其中，$B\left(T{P}_j,F{P}_j,T{N}_j,F{N}_j\right)$指代一种计算方法 $B\in \left\{\text{ Accuracy, Precision, Recall, }{F}^{\beta }\right\}$。Macro指代对类作平均，Micro指代对样本作平均，$q$为总的类别数。

1. Label accuracy

• Macro

def label_accuracy_macro(gt, predict):
    quantity = _label_quantity(gt, predict)
    tp_tn = np.add(quantity[0], quantity[2])
    tp_fp_tn_fn = np.sum(quantity, axis=0)
    return np.mean(tp_tn / (tp_fp_tn_fn + epsilon))

• Micro

def label_accuracy_micro(gt, predict):
    quantity = _label_quantity(gt, predict)
    sum_tp, sum_fp, sum_tn, sum_fn = np.sum(quantity, axis=1)
    return (sum_tp + sum_tn) / (
            sum_tp + sum_fp + sum_tn + sum_fn + epsilon)

2. Label precision

• Macro

def label_precision_macro(gt, predict):
    quantity = _label_quantity(gt, predict)
    tp = quantity[0]
    tp_fp = np.add(quantity[0], quantity[1])
    return np.mean(tp / (tp_fp + epsilon))

• Micro

def label_precision_micro(gt, predict):
    quantity = _label_quantity(gt, predict)
    sum_tp, sum_fp, sum_tn, sum_fn = np.sum(quantity, axis=1)
    return sum_tp / (sum_tp + sum_fp + epsilon)

3. Label recall

• Macro

def label_recall_macro(gt, predict):
    quantity = _label_quantity(gt, predict)
    tp = quantity[0]
    tp_fn = np.add(quantity[0], quantity[3])
    return np.mean(tp / (tp_fn + epsilon))

• Micro

def label_recall_micro(gt, predict):
    quantity = _label_quantity(gt, predict)
    sum_tp, sum_fp, sum_tn, sum_fn = np.sum(quantity, axis=1)
    return sum_tp / (sum_tp + sum_fn + epsilon)

4. Label F1

• Macro

def label_f1_macro(gt, predict, beta=1):
    quantity = _label_quantity(gt, predict)
    tp = quantity[0]
    fp = quantity[1]
    fn = quantity[3]
    return np.mean((1 + beta**2) * tp / ((1 + beta**2) * tp + beta**2 * fn + fp + epsilon))

• Micro

def label_f1_micro(gt, predict, beta=1):
    quantity = _label_quantity(gt, predict)
    tp = np.sum(quantity[0])
    fp = np.sum(quantity[1])
    fn = np.sum(quantity[3])
    return (1 + beta**2) * tp / ((1 + beta**2) * tp + beta**2 * fn + fp + epsilon)

注：epsilon设置为如1e-8的常数防止zero-division的情况发生。

Reference

[1] M. Zhang and Z. Zhou, “A Review on Multi-Label Learning Algorithms,” in IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 8, pp. 1819-1837, Aug. 2014, doi: 10.1109/TKDE.2013.39.
[2]Wei Long, Yang Yang, Hong-Bin Shen, ImPLoc: a multi-instance deep learning model for the prediction of protein subcellular localization based on immunohistochemistry images, Bioinformatics, Volume 36, Issue 7, 1 April 2020, Pages 2244–2250, https://doi.org/10.1093/bioinformatics/btz909
[3]https://github.com/Outliers1106/Multi-Label-Metrics