【论文阅读】Self-Paced Boost Learning for Classification

论文下载
bib:

@INPROCEEDINGS{PiLi2016SPBL,
  title		= {Self-Paced Boost Learning for Classification},
  author	= {Te Pi and Xi Li and Zhongfei Zhang and Deyu Meng and Fei Wu and Jun Xiao and Yueting Zhuang},
  booktitle	= {IJCAI},
  year		= {2016},
  pages     = {1932--1938}
}

GitHub

1. 摘要

Effectiveness and robustness are two essential aspects of supervised learning studies.

For effective learning, ensemble methods are developed to build a strong effective model from ensemble of weak models.

For robust learning, self-paced learning (SPL) is proposed to learn in a self-controlled pace from easy samples to complex ones.

Motivated by simultaneously enhancing the learning effectiveness and robustness, we propose a unified framework, Self-Paced Boost Learning (SPBL).

With an adaptive from-easy-to-hard pace in boosting process, SPBL asymptotically guides the model to focus more on the insufficiently learned samples with higher reliability.

Via a max-margin boosting optimization with self-paced sample selection, SPBL is capable of capturing the intrinsic inter-class discriminative patterns while ensuring the reliability of the samples involved in learning.

We formulate SPBL as a fully-corrective optimization for classification.

The experiments on several real-world datasets show the superiority of SPBL in terms of both effectiveness and robustness.

Note:

1. 将Self-paced learning（自步学习，从容易到难的学习）和Boost（集成学习）融合在一起，同时保证有效性与鲁棒性。

2. 算法

问题：多分类问题
y ~ ( x ) = arg max ⁡ r ∈ { 1 , … , C } F r ( x ; Θ ) (1) \widetilde{y}(x) = \argmax_{r \in \{1, \dots, C\} }F_r(x; \Theta) \tag{1} y ​(x)=r∈{1,…,C}argmax​Fr​(x;Θ)(1)

• $\{(x_i,y_i){\}}_{i=1}^n$ 表示带标签的训练数据，其中又$n$个带标签的样本。$x_i\in {\mathbb{R}}^d$ 是第$i$个样本的特征， y i ∈ { 1 , … , C } y_i \in \{1, \dots, C\} yi​∈{1,…,C}表示第个样本的标签。
• $F_r(\cdot ):{\mathbb{R}}^d\to \mathbb{R}$ 表示将样本$x$分类到类别$r$的置信度得分。值得注意的是, 这里相当于将多分类问题转化为了$C$个二分类问题，对应于OvA策略。优点是只用训练类别数目$C$个分类器，缺点是，会出现类别不平衡的问题（A对应类别样本多）。
• 最后的多分类预测则是预测样本对应最大评分的类。在实际操作中，可以理解为softmax操作后对应最大概率的类（threshold）。

boost:
boost是一种集成学习中的一个方法，目的是集成多个弱学习器成为一个强学习器。
F r ( x ; W ) = ∑ j = 1 k w r j h j ( x ) , r ∈ { 1 , … , C } (2) F_r(x;W) = \sum_{j=1}^k w_{rj}h_j(x), r \in \{1, \dots, C\} \tag{2} Fr​(x;W)=j=1∑k​wrj​hj​(x),r∈{1,…,C}(2)

• h j ( x ) : R d → { 0 , 1 } h_j (x) : \mathbb{R}^d \rightarrow \{0, 1\} hj​(x):Rd→{0,1}，表示一个弱二分类器， $w_{rj}$学习器对应权重，是一个学习参数。
• $W=[w_1,\dots ,w_C]\in {\mathbb{R}}^{k\times C}$ with each $w_r=[w_{r1},\dots ,w_{r_k}{]}^{\mathsf{T}}$.

general objective of SPBL:
$$\underset{W,v}{\min }\sum _{i=1}^n{v}_i\sum _{r=1}^{C}L({\rho }_{ir})+\sum _{i=1}^{n}g(v_i;\lambda )+\upsilon R(W)s.t.\forall i,r,{\rho }_{i,r}=H_{i:}{w}_{y_i}-H_{i:}{w}_r;W\ge 0;v\in [0,1{]}^n\text{\hspace{1em}(3)}$$.

• $H\in {\mathbb{R}}^{n\times k}$ with each item $H_{ij}=h_j(x_i)$.
• $H_{i:}{w}_{y_i}=H_{i:}\times w_{y_i},w_{y_i}=[w_{y_{i}1},\dots ,w_{y_{i}k}{]}^{\mathsf{T}}$.

specific formulation:
$$\underset{W,v}{\min }\sum _{i,r}{v}_i\ln (1+\exp (-{\rho }_{ir}))+\sum _{i=1}^{n}g(v_i;\lambda )+\upsilon \Vert W{\Vert }_{2,1}$$
$$\text{s.t.}\forall i,r,{\rho }_{i,r}=H_{i:}{w}_{y_i}-H_{i:}{w}_r;W\ge 0;v\in [0,1{]}^n\text{\hspace{1em}(3)}$$

• $\Vert W{\Vert }_{2,1}\Vert ={\sum }_{j=1}^k\Vert W_{j:}{\Vert }_2$，鼓励矩阵行列都稀疏。
• the logistic loss. 我的理解该损失就是简单的对差值求$\exp$。区别在于现有的是二分类的概率，概率值是由$\text{sigmod}=\frac{1}{1+e^{-x}}$计算的，即$\ln (\text{sigmod})=-\ln (1+\exp (-x))$。

3. 总结

关于优化目标的求解，涉及到了对偶问题（dual problem），实在是懂不了了。