Adaboost的推导

加法模型

$$\begin{gathered} H(x)=\sum _{t=1}^T{\alpha }_t{h}_t(x) \\ 构造强分类器为弱分类器的线性组合 \end{gathered}$$

指数损失函数

$$\sum _{i=1}^{N}L(y_i,H(x_i))=\sum _{i=1}^{N}L(y_i,\sum _{t=1}^T{\alpha }_t{h}_t(x))=\sum _{i=1}^N\exp (-y_i\sum _{t=1}^T{\alpha }_t{h}_t(x_i))$$
y的取值只有{-1,1}, 当H(x)越接近y的时候$-y{\sum }_{t=1}^T{\alpha }_t{h}_t(x)$越大, 故$L(y,H(x))$越小

Adaboost算法流程

算法推导

• 构造一个$f_t(x)$
  $f_t(x)$代表前t个h(x)的线性组合: $f_t(x)=f_{t-1}(x)+{\alpha }_t{h}_t(x)$。其中,$f_1(x)={\alpha }_1{h}_1(x)$
• 计算弱分类器的权重${\alpha }_t$, 样本权重${\overline{w}}_{ti}$
  每一轮迭代的时候计算$\arg {\min }_{{\alpha }_t,h_t}{\sum }_{i=1}^N\exp (-y_i(f_{t-1}(x)+{\alpha }_t{h}_t(x)))$, 计算T-1次(t=(2…T))就能得到能让指数损失函数取得最小的T个$\alpha$与$\overline{w}$
  • 首先计算$\arg {\min }_{h_t}{\sum }_{i=1}^N\exp (-y_i(f_{t-1}(x_i)+{\alpha }_t{h}_t(x_i)))$
    $$\begin{gathered} \Rightarrow \arg \underset{h_t}{\min }\sum _{i=1}^N\exp (-y_i(f_{t-1}(x_i))\ast \exp (y_i{h}_t(x_i)) \\ \Rightarrow \arg \underset{h_t}{\min }\sum _{i=1}^N{\overline{w}}_{ti}I(h_t(x_i)̸=y_i) \\ \{\begin{array}{ll}I(\text{A ̸= B})=1,& \text{A ̸= B}\\ I(\text{A ̸= B})=0,& \text{A = B}\end{array} \end{gathered}$$
  • 定义${\overline{w}}_{ti}=\exp (-y_i{f}_{t-1}(x_i))$为样本权重
  • 这个过程相当于第三行伪代码
• 计算弱分类器权重${\alpha }_t$
  把已知的$h_t(x)$代入计算$\arg {\min }_{{\alpha }_t}{\sum }_{i=1}^N\exp (-y_i(f_{t-1}(x_i)+{\alpha }_t{h}_t(x_i)))$
  $$\begin{gathered} 原式=\sum _{i=1}^N\exp (-y_i(f_{t-1}(x_i)+{\alpha }_t{h}_t(x_i))) \\ =\sum _{i=1}^N{\overline{w}}_{ti}\ast exp(-y_i{\alpha }_t{h}_t(x_i)) \\ 将样本分成y_i=h_t(x_i)的部分和y_i̸=h_t(x_i)的部分 \\ 原式=\sum _{y_i=h_t(x_i)}{\overline{w}}_{ti}\exp (-{\alpha }_t)+\sum _{y_i̸=h_t(x_i)}{\overline{w}}_{ti}\exp ({\alpha }_t) \\ =\sum _{i=1}^N{\overline{w}}_{ti}\exp (-{\alpha }_t)-\sum _{y_i̸=h_t(x_i)}{\overline{w}}_{ti}\exp (-{\alpha }_t)+\sum _{i=1}^N{\overline{w}}_{ti}\exp ({\alpha }_t)\ast I(y_i̸=h_t(x_i)) \\ =\sum _{i=1}^N{\overline{w}}_{ti}\exp (-{\alpha }_t)-\sum _{i=1}^N{\overline{w}}_{ti}\exp (-{\alpha }_t)\ast I(y_i̸=h_t(x_i))+\sum _{i=1}^N{\overline{w}}_{ti}\exp ({\alpha }_t)\ast I(y_i̸=h_t(x_i)) \\ =(e^{{\alpha }_t}-e^{-{\alpha }_t})\sum _{i=1}^N{\overline{w}}_{ti}\ast I(y_i̸=h_t(x_i))+e^{-{\alpha }_t}\sum _{i=1}^N{\overline{w}}_{ti} \\ 对{\alpha }_t求导\Rightarrow \sum _{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t(x_i))(e^{{\alpha }_t}+e^{-{\alpha }_t})-\sum _{i=1}^N{\overline{w}}_{ti}{e}^{-{\alpha }_t} \\ 令上式=0\Rightarrow \sum _{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t(x_i))(e^{{\alpha }_t}+e^{-{\alpha }_t})=\sum _{i=1}^N{\overline{w}}_{ti}{e}^{-{\alpha }_t} \\ \Rightarrow \sum _{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t(x_i))(e^{2{\alpha }_t}+1)=\sum _{i=1}^N{\overline{w}}_{ti} \\ \Rightarrow e^{2{\alpha }_t}=\frac{{\sum }_{i=1}^N{\overline{w}}_{ti}}{{\sum }_{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t{x}_i)}-1 \\ {\alpha }_t=\frac{1}{2}\ln (\frac{{\sum }_{i=1}^N{\overline{w}}_{ti}}{{\sum }_{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t(x_i))}-1) \\ {\alpha }_t=\frac{1}{2}\ln (\frac{1-\frac{{\sum }_{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t(x_i))}{{\sum }_{i=1}^N{\overline{w}}_{ti}}}{\frac{{\sum }_{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t(x_i))}{{\sum }_{i=1}^N{\overline{w}}_{ti}}}) \\ 其中\frac{{\sum }_{i=1}^N{\overline{w}}_{ti}I(y_i̸=h_t(x_i))}{{\sum }_{i=1}^N{\overline{w}}_{ti}}就是错误率,分子是错误的个数\ast 样本权重,分母是所有样本个数\ast 样本权重 \end{gathered}$$
  故, 证明出第6行伪代码 ${\alpha }_t=\frac{1}{2}ln(\frac{1-{ϵ}_t}{{ϵ}_t})$
• 计算${\overline{w}}_{ti}$
  $$\begin{gathered} 已知{\overline{w}}_{ti}=\exp (-y_i{f}_{t-1}(x_i)) \\ 能得到{\overline{w}}_{m+1,i}=\exp (-y_i{f}_t(x_i)) \\ =\exp (-y_i(f_{t-1}(x_i)+{\alpha }_t{h}_t(x_i))) \\ =\exp (-y_i{f}_{t-1}(x_i))\ast \exp (-y_i{\alpha }_t{h}_t(x_i)) \\ 故,{\overline{w}}_{t+1,i}={\overline{w}}_{ti}\ast \exp (-y_i{\alpha }_t{h}_t(x_i)) \\ 与伪代码第7条只差一个规范因子z \end{gathered}$$

参考文献

1. 周志华 - 机器学习
2. 李航 - 统计学习方法