《Bayes-Optimal Hierarchical Multi-label Classificaiton》-TKDE

This paper systematically concludes the classical loss functions for hierarchical multi-label classification (HMC), and extends the Hamming loss and Ranking loss to support class hierarchy.
Reading Difficulty：$\star \star$
Creativity: $\star \star$
Comprehensiveness (全面性)：$\star \star \star \star \star$

Symbol System:

Symbol	Meaning
y i ∈ { 0 , 1 } y_i \in \{0,1\} yi​∈{0,1}	The label for class $i$
$\uparrow (i),\downarrow (i),\Uparrow (i),\Downarrow (i),\iff (i)$	The parent, children, ancestors, descentors, and sibilings of node $i$
y i ∈ { 0 , 1 } i \mathbf{y}_{\mathbf{i}} \in \{0,1\}^\mathbf{i} yi​∈{0,1}i	the label vector for classes $\mathbf{i}$
H = { 0 , … , N − 1 } \mathcal{H} = \{0,\dots,N-1\} H={0,…,N−1}	The class hierachy, where $N$ is the number of nodes
$I(x)$	An indicator function output 1 when x is true, 0 otherwise.
$\mathcal{R}$	The conditional risk

Hierarchy Constraints
In HMC, if the label structure is a tree, we have:
$$y_i=1\Rightarrow y_{\uparrow (i)}=1.$$

For the DAG-type HMC with, there are two interpretations:

1. AND-interpretation. We have $y_i=1\Rightarrow y_{\uparrow (i)}=1$

2. OR-interpretation. We have $y_i=1\Rightarrow \exists y_{\uparrow (c)}=1$

Loss functions for Flat and Hierarchical Classification
It is a review.

Zero-one loss:
${\ell }_{0/1}(\hat{\mathbf{y}},\mathbf{y})=I(\hat{\mathbf{y}}\ne \mathbf{y})$

Hamming loss:
${\ell }_{\text{hamming}}(\hat{\mathbf{y}},\mathbf{y})={\sum }_{i\in \mathcal{H}}I({\hat{y}}_i\ne y_i)$

Top-$k$ precision:
$k$ most-confident predicted positive labels for each sample.
$\text{top-k-precision}(\hat{\mathbf{y}},\mathbf{y})=\frac{\text{The number of TP predictions in the top-k labels of }\hat{\mathbf{y}}}{k}$
So the loss is
${\ell }_{\text{top-k}}=1-\text{top-k-precision}$

Ranking loss:
${\ell }_{\text{rank}}={\sum }_{(i,j):y_i>y_j}(I(\widehat{y_i}<{\hat{y}}_j)+\frac{I(\widehat{y_i}={\hat{y}}_j)}{2})$

Hierarchical Multi-class Classificaiton
A review.
Note: Only a single path can be predicted positive.

Cai and Hofmann:
$\ell ={\sum }_{i\in \mathcal{H}}{c}_{i}I({\hat{y}}_i\ne y_i)$
where $c_i$ is the cost for node $i$.

Dekel et al. :
It seems that this loss is complicated.
But this paper treats this loss as similar to the former loss?

Hierarchical multi-label classification

H-Loss:
${\ell }_H=\alpha {\sum }_{i:y_i=1,{\hat{y}}_i=0}{c}_{i}I({\hat{\mathbf{y}}}_{\Uparrow (i)}={\mathbf{y}}_{\Uparrow (i)})+\beta {\sum }_{i:y_i=0,{\hat{y}}_i=1}{c}_{i}I({\hat{\mathbf{y}}}_{\Uparrow (i)}={\mathbf{y}}_{\Uparrow (i)})$
where $\alpha$ and $\beta$ are weight for FN and FP.

Often, misclassifications at upper class level are considered more expensive than those at the lower levels.
Thus, there are a cost assigning approach
$$c_i=\{\begin{array}{rlr}& 1,& \text{ i = 0,}\\ & \frac{c_{\Uparrow (i)}}{n_{\iff (i)}},& \text{ i > 0},\end{array}$$
where $n_{\iff (i)}$ is the number of siblings of $i$ (including $i$).

Matching Loss:
${\ell }_{\text{match}}=\alpha {\sum }_{i:y_i=1}\varphi (i,\hat{\mathbf{y}})+\beta {\sum }_{i:{\hat{y}}_i=1}\varphi (i,\mathbf{y})$.
where
$
\phi(i,\mathbf{y}) = \min_{j:y_j=1} \text{cost}(j\rightarrow i)
$
where $\text{cost}(j\to i)$ is the cost traverse from node j to node i in the hierarchy, maybe path length or weighted path length.

Verspoor et al.: Hierarchical versions of precision, recall and F-score, but these are more expensive.

Condensing (压缩) sort and Selection Algorithm for HMC
It is a review.
It can be used on both tree and DAG hierarchies.

It solves this optimization objective via a greedy algorithm called condensing sort and selection algorithm:
max ⁡ { ψ i } i ∈ H ∑ i ∈ H ψ i y ~ i s . t . ψ i ≤ ψ ↑ ( i ) , ∀ i ∈ H ∖ { 0 } , ψ 0 = 1 , ψ i ∈ { 0 , 1 } , ∑ i = 0 N − 1 ψ i = L \begin{aligned} & \max_{\{\psi_i\}_{i \in \mathcal{H}}} \sum_{i \in \mathcal{H}} \psi_i \widetilde{y}_i \\ s.t. \qquad & \psi_i \leq \psi_{\uparrow(i)}, \forall i \in \mathcal{H}\setminus \{0\},\\ & \psi_0 = 1, \psi_i \in \{0, 1\}, \\ & \sum_{i=0}^{N-1} \psi_i = L \end{aligned} s.t.​{ψi​}i∈H​max​i∈H∑​ψi​y ​i​ψi​≤ψ↑(i)​,∀i∈H∖{0},ψ0​=1,ψi​∈{0,1},i=0∑N−1​ψi​=L​

where ${\psi }_i=1$ indicates that node $i$ is predicted positive in $\hat{\mathbf{y}}$; and 0 otherwise.

When the label hierarchy is a DAG, the first constraint of the above objective has to be replaced to
ψ i ≤ ψ j , ∀ i ∈ H ∖ { 0 } , ∀ j ∈ ⇑ ( i ) . \psi_i \leq \psi_j, \forall i \in \mathcal{H} \setminus \{0\}, \forall j \in \Uparrow(i). ψi​≤ψj​,∀i∈H∖{0},∀j∈⇑(i).

Extending Flatten loss
This paper extends Hamming Loss and Ranking Loss to support hierarchy,

For hierarchical hamming loss:
$${\ell }_{\text{H-hamming}}=\alpha \sum _{i:y_i=1\wedge {\hat{y}}_i=0}{c}_i+\beta \sum _{i:y_i=0\wedge {\hat{y}}_i=1}{c}_i$$

DAG class hierarchy derives
$$c_i=\{\begin{array}{rlr}& 1,& i=0,\\ & \sum _{j\in \Uparrow (i)}\frac{c_j}{n_{\downarrow (j)}},& i>0\end{array}$$
where $n$ is the number of children of node $j$.

There are special cases in origin papaer, but it is easy and not discussed here.

For hierarchical ranking loss:
$${\ell }_{\text{H-rank}}=\sum _{(i,j):y_i>y_j}{c}_{ij}(I({\hat{y}}_i<{\hat{y}}_j)+\frac{1}{2}I({\hat{y}}_i={\hat{y}}_j)),$$

where $c_{ij}=c_i{c}_j$ ensures a high penalty when an upper-level positive label is ranked after a lower-level negative label.

Minimizing the risk
The conditional risks (or simply the risk) $\mathcal{R}(\hat{\mathbf{y}})$ of predicting multilabel $\hat{\mathbf{y}}$ is the expectation of $\ell (\hat{\mathbf{y}},\mathbf{y})$ over all possible $y$’s as ground truth, i.e.,
$$(期望风险极小化)\underset{\hat{\mathbf{y}}\in \Omega }{\mathrm{argmin}}\mathcal{R}(\hat{\mathbf{y}})=\sum _{\mathbf{y}}\ell (\hat{\mathbf{y}},\mathbf{y})P(\mathbf{y}|\mathbf{x}).$$

There are three issues to be addressed:
(1) Estimating $P(\mathbf{y}|\mathbf{x})$.
(2) Computing $\mathcal{R}(\hat{\mathbf{y}})$ without exhaustively searching.
(3) Minimizing $\mathcal{R}(\hat{\mathbf{y}})$.

This paper computes $p_i$ through chain rule, and the risk is transferred into different forms for different losses.
The risk for matching loss:
$${\mathcal{R}}_{\text{match}}(\hat{\mathbf{y}})=\sum _{i:{\hat{y}}_i=0}\varphi (i,\hat{\mathbf{y}})+\sum _{i:{\hat{y}}_i}{q}_i$$

where $q_i={\sum }_{j=0}^{d(i)-1}{\sum }_{l=j+1}^{d(i)}{c}_{{\Uparrow }_l(i)}P({\mathbf{y}}_{{\Uparrow }_{0:j}(i)}=1,y_{{\Uparrow }_{j+1}(i)}=0|\mathbf{x})$, $d(i)$ is the depth of node $i$.${\Uparrow }_j(i)$ is the $i$’s ancestor at depth j, ⇑ 0 : j ( i ) = { ⇑ 0 ( i ) , … , ⇑ j ( i ) } \Uparrow_{0:j}(i) = \{\Uparrow_0(i), \dots, \Uparrow_j(i)\} ⇑0:j​(i)={⇑0​(i),…,⇑j​(i)} is the set of $i$’s ancestors at depths 0 t0 j.

The risk for hierarchical hamming loss:
$${\mathcal{R}}_{\text{H-hamming}}(\hat{\mathbf{y}})=\alpha \sum _{i:{\hat{y}}_i=0}{c}_i{p}_i+\beta \sum _{i:{\hat{y}}_i=1}{c}_i(1-p_i)$$

The risk for hierarchical ranking loss:
$${\mathcal{R}}_{\text{H-rank}}(\hat{\mathbf{y}})=\sum _{0\le i<j\le N-1}{c}_{ij}(p_{i}I({\hat{y}}_i\le {\hat{y}}_j)+p_{j}I({\hat{y}}_i\ge {\hat{y}}_j)+\frac{p_i+p_j}{2}I({\hat{y}}_i={\hat{y}}_j))-C$$

Efficient minimizing the risk:
$$\hat{\mathbf{y}}=\underset{L=1,\dots ,N}{\mathrm{argmin}}\mathcal{R}({\hat{\mathbf{y}}}_{(L)}^{\star }),$$
where
$${\hat{\mathbf{y}}}_{(L)}^{\star }=\underset{\hat{\mathbf{y}}\in \Omega }{\mathrm{argmin}}\mathcal{R}(\hat{\mathbf{y}}):|\text{supp}(\hat{\mathbf{y}})|=L$$
where supp ( f ) : = { x ∈ X ∣ f ( x ) ≠ 0 } \text{supp}(f) := \{x \in X | f(x) \neq 0\} supp(f):={x∈X∣f(x)=0} is the support of $f$.

实际上是比较朴素也比较容易理解的优化目标，通过按照positive label的数量来分别优化， 也就是different $L$.

This paper adopts the CSSAG (压缩排序与选择算法 proposed by Bi.) for tree label hierarchy, which is a greedy strategy.

Conclusions

This paper extends matching loss, hamming loss and ranking loss to support tree-type as well as DAG-type class hierarchies.
This paper seems easy to be understood without much innovations, but organized well with strong comprehensiveness, so it is published on TKDE.