Boosting

5.2 Boosting (Sequential) Learning

Boosting is under the PAC learning model. It involves incrementally building an ensemble by training each new model instance to emphasize the training instances that previous models mis-classified.
 
 

5.21 AdaBoost

AdaBoost = Weak Algorithm + Re-Weighting + Linear Aggregation

5.211 What is AdaBoost?

AdaBoost calls a given weak or base learning algorithm repeatedly in a series of rounds $t=1,2,...,T$ ,which is used for classification, mostly binary in practice.
This algorithm takes as input a training set $(x_1,y_1),(x_2,y_2),...,(x_N,y_N)$, where $x_n$ belongs to some domain or instance space $X$. And each lable $y_n$ is in some lable set $Y$.
One of the main ideas of the algorithm is to maintain a distribution or set of weights over the training set. The weight of this distribution on ${\mathbf{x}}_{\mathbf{n}}$ on round $t$ is denoted $u_n^{(t)}$. Initially ,all weights are set equally,but on each round, the weights of incorrectly classified examples are increased so that the weak learner is forced to focus on the hard examples in the training set.

1. 集成学习，适用于分类问题 (classification)。
2. 有序地训练学习基 (base hypothesis or base learner, which is possibly weak).
3. 当前轮的学习基在训练过程中会更重视上一个学习器中判错的样本。

5.212 Why AdaBoost?

AdaBoost can handle weak hypotheses which output real-valued or confidence-rated predictions. That is, for each instance, the weak hypothesis $g_t$ outputs a prediction $g_t(\bf x_n) $ $in\mathbb{R}$, whose sign is the predicted lable (+1, -1)and whose magnitude $|g_t({\mathbf{x}}_{\mathbf{n}})|$ gives a measure of “confidence” in the prediction.

5.213 How AdaBoost?

Our mission is to find a weak hypothesis $g_t$ for each round and their weights used for the final linear aggregation.
  
Given the learning data set $D:$ {$(x_1,y_1),(x_2,y_2),...,(x_N,y_N)$} and algorithm $A.$

1. Set the initial weight $u^{(1)}=[\frac{1}{N},\frac{1}{N},...,\frac{1}{N}]$
2. $\mathbf{f}\mathbf{o}\mathbf{r}\mathbf{t}=1,2,...,\mathbf{T}$
3.  $g_t\leftarrow u^{(t)}$
4.  ${ϵ}_t=\frac{{\sum }_{n=1}^N{u}_n^{(t)}I[y_n̸=g_t({\mathbf{x}}_{\mathbf{n}})]}{{\sum }_{n=1}^N{u}_n^{(t)}}$
5.  $if{ϵ}_t>0.5$, then break
6.  ${\alpha }_t=\frac{1}{2}\ln (\frac{1-{ϵ}_t}{{ϵ}_t})$
7.  $u^{(t+1)}\leftarrow u^{(t)}$
     $y_n̸=g_t({\mathbf{x}}_{\mathbf{n}}):$ $u^{(t+1)}\leftarrow u^{(t)}\sqrt{\frac{1-{ϵ}_t}{{ϵ}_t}}$
     $y_n=g_t({\mathbf{x}}_{\mathbf{n}}):$ $u^{(t+1)}\leftarrow u^{(t)}/\sqrt{\frac{1-{ϵ}_t}{{ϵ}_t}}$
8. $\mathbf{e}\mathbf{n}\mathbf{d}\mathbf{f}\mathbf{o}\mathbf{r}$
9. $\mathbf{R}\mathbf{e}\mathbf{t}\mathbf{u}\mathbf{r}\mathbf{n}$ $G(\mathbf{x})$$=sign[{\sum }_{t=1}^T{\alpha }_t{g}_t(\mathbf{x})]$