利用积分、概率论等数学工具,对AdaBoost算法进行详细推导
AdaBoost算法的详细推导
在上一篇中,显然,预测器
H
(
x
)
H(\boldsymbol{x})
H(x)比分类器
h
(
x
)
=
s
g
n
(
H
(
x
)
)
h(\boldsymbol{x})=\mathrm{sgn}(H(\boldsymbol{x}))
h(x)=sgn(H(x))有更好的数学性质,故使用前者进行推演。
记
H
t
(
x
)
=
∑
i
=
1
t
α
i
h
i
(
x
)
\begin{align} H_t(\boldsymbol{x})=\sum_{i=1}^t{\alpha }_ih_i(\boldsymbol{x}) \tag{8.14} \end{align}
Ht(x)=i=1∑tαihi(x)(8.14)
则
H
t
(
x
)
=
H
t
−
1
(
x
)
+
α
t
h
t
(
x
)
\begin{align} H_t(\boldsymbol{x})=H_{t-1}(\boldsymbol{x})+{\alpha }_th_t(\boldsymbol{x}) \tag{8.15} \end{align}
Ht(x)=Ht−1(x)+αtht(x)(8.15)
我们常使用“均方误差”(MSE)进行性能度量,常用方法:
- 基于测试集进行度量,即【西瓜书式(2.2)】
- 基于样本空间(全体)进行度量,则是采用数学期望。 当变量为连续时,即为【西瓜书式(2.3)】
然而,【西瓜书式(7.2)】表明,使用损失函数进行性能度量时,采用的是损失的数学期望(期望损失也称为风险Risk,故记为R)。 在AdaBoost中,已知训练集
D
D
D但并不知道数据的分布
D
\mathcal{D}
D,为确定
α
i
\alpha _i
αi,在讨论式(8.15)第2项的损失
ℓ
exp
(
α
t
h
t
(
x
)
∣
D
)
{\ell}_{\exp}({\alpha }_th_t(\boldsymbol{x})\,|\,\mathcal{D})
ℓexp(αtht(x)∣D)时,本来应该考虑条件
(
D
)
(\mathcal{D} )
(D)转而考虑条件
(
D
t
)
(\mathcal{D}_t )
(Dt),即用
ℓ
exp
(
α
t
h
t
(
x
)
∣
D
t
)
{\ell}_{\exp}({\alpha }_th_t(\boldsymbol{x})\,|\,\mathcal{D}_t)
ℓexp(αtht(x)∣Dt)近似
ℓ
exp
(
α
t
h
t
(
x
)
∣
D
)
{\ell}_{\exp}({\alpha }_th_t(\boldsymbol{x})\,|\,\mathcal{D})
ℓexp(αtht(x)∣D)。 也即通过构造分布序列
D
1
,
D
2
,
⋯
\mathcal{D}_1,\mathcal{D}_2,\cdots
D1,D2,⋯
去逼近真实分布
D
\mathcal{D}
D。 这就是求
α
t
{\alpha}_t
αt的情形,见【西瓜书式(8.9)】的推导过程。
【西瓜书式(8.11)】表明:因 ϵ t = P x ∼ D t ( h t ( x ) ≠ f ( x ) ) {\epsilon}_t=P_{\boldsymbol{x}\sim \mathcal{D}_t}(h_t(\boldsymbol{x})\neq f(\boldsymbol{x})) ϵt=Px∼Dt(ht(x)=f(x)),故 α t {\alpha}_t αt通过 ϵ t {\epsilon}_t ϵt而依赖于 h t h_t ht,下面我们看看该参数的特性。
- 由AdaBoost算法第5行知,应选取错误率小于50%的 h t h_t ht,即 α t < 1 2 {\alpha}_t<\frac{1}{2} αt<21,代入【西瓜书式(8.11)】知 α t > 0 {\alpha}_t>0 αt>0。
- 若降低误分类率( ϵ t ↓ {\epsilon}_t\downarrow ϵt↓),则 1 ϵ t ↑ \frac{1}{{\epsilon}_t }\uparrow ϵt1↑,则 1 − ϵ t ϵ t = ( 1 ϵ t − 1 ) ↑ \frac{1-{\epsilon}_t}{{\epsilon}_t}=(\frac{1}{{\epsilon}_t}-1)\uparrow ϵt1−ϵt=(ϵt1−1)↑,由【西瓜书式(8.11)】知 α t ↑ {\alpha}_t\uparrow αt↑,即 ϵ t {\epsilon}_t ϵt越小, α t {\alpha}_t αt越大,也即: h t h_t ht越好相应的权重 α t {\alpha}_t αt越大。
预测器
H
t
H_t
Ht的损失函数:
ℓ
(
H
t
(
x
)
∣
D
)
=
ℓ
(
H
t
−
1
(
x
)
+
α
t
h
t
(
x
)
∣
D
)
\begin{align} \ell (H_t(\boldsymbol{x})\,|\,\mathcal{D} )=\ell (H_{t-1}(\boldsymbol{x})+{\alpha }_th_t(\boldsymbol{x})\,|\,\mathcal{D}) \tag{8.16} \end{align}
ℓ(Ht(x)∣D)=ℓ(Ht−1(x)+αtht(x)∣D)(8.16)
注意:式(8.16)中
D
\mathcal{D}
D的作用范围不只是第二项,而是全部,即
ℓ
(
H
t
−
1
(
x
)
+
α
t
h
t
(
x
)
∣
D
)
=
ℓ
(
[
H
t
−
1
(
x
)
+
α
t
h
t
(
x
)
]
∣
D
)
\ell (H_{t-1}(\boldsymbol{x})+{\alpha }_th_t(\boldsymbol{x})\,|\,\mathcal{D})=\ell ([H_{t-1}(\boldsymbol{x})+{\alpha }_th_t(\boldsymbol{x})]\,|\,\mathcal{D})
ℓ(Ht−1(x)+αtht(x)∣D)=ℓ([Ht−1(x)+αtht(x)]∣D)。
希望通过最小化该损失函数来找到 h t h_t ht(此时, H t − 1 H_{t-1} Ht−1已知,而 α t {\alpha }_t αt依赖于 h t h_t ht, h t h_t ht是待定的二分类器),由于 ℓ ( H t − 1 ( x ) + α t h t ( x ) ∣ D ) \ell (H_{t-1}(\boldsymbol{x})+{\alpha }_th_t(\boldsymbol{x})\,|\,\mathcal{D}) ℓ(Ht−1(x)+αtht(x)∣D)中的 α t h t {\alpha }_th_t αtht不好处理,我们转而求其次,即考虑最小化 ℓ ( H t − 1 ( x ) + h t ( x ) ∣ D ) \ell (H_{t-1}(\boldsymbol{x})+h_t(\boldsymbol{x})\,|\,\mathcal{D}) ℓ(Ht−1(x)+ht(x)∣D),这即得到【西瓜书式(8.12)】,进而推导出【西瓜书式(8.13)】。
公式推导中要用到如下数学知识:
虽然, min x 2 ≠ min a x 2 \min x^2 \neq \min ax^2 minx2=minax2,但 arg min x x 2 = arg min x a x 2 ( a > 0 ) \mathop{\arg\min}\limits_x x^2 =\mathop{\arg\min}\limits_x ax^2\quad (a>0) xargminx2=xargminax2(a>0)
即这两个函数(
x
2
x^2
x2与
a
x
2
ax^2
ax2)虽然最小值不相等,但它们取得最小值的点
x
x
x相同,推广到一般情形,对函数
g
(
x
)
g(x)
g(x)有
arg
min
x
a
g
(
x
)
+
b
=
{
arg
min
x
g
(
x
)
(
a
>
0
)
arg
max
x
g
(
x
)
(
a
<
0
)
\begin{align} \mathop{\arg\min}\limits_x ag(x)+b= \begin{cases} \, \mathop{\arg\min}\limits_x g(x)\quad (a>0) \\ \, \mathop{\arg\max}\limits_x g(x)\quad (a<0) \end{cases} \tag{8.17} \end{align}
xargminag(x)+b=⎩
⎨
⎧xargming(x)(a>0)xargmaxg(x)(a<0)(8.17)
对于最大值情况,易类比相应的结论。
考虑
E
x
∼
D
[
e
−
f
(
x
)
H
t
−
1
(
x
)
(
1
−
f
(
x
)
h
(
x
)
+
1
2
)
]
\begin{align} \mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }[\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}(1-f(\boldsymbol{x})h(\boldsymbol{x})+\frac{1}{2})] \tag{8.18} \end{align}
x∼DE[e−f(x)Ht−1(x)(1−f(x)h(x)+21)](8.18)
E
x
∼
D
\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }
x∼DE将会消去变元
x
\boldsymbol{x}
x(类似于求和号、积分号的消元原理),而
f
,
H
t
−
1
f,H_{t-1}
f,Ht−1是已知的,故式(8.18)中的变元为
h
h
h(泛函数)。
式(8.18)
=
−
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
f
(
x
)
h
(
x
)
+
3
2
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
=
(
−
1
)
g
1
(
h
)
+
b
\begin{align} \text{式(8.18)} & =-\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}f(\boldsymbol{x})h(\boldsymbol{x})+\frac{3}{2}\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}\notag \\ & =(-1)g_1(h)+b \tag{8.19} \end{align}
式(8.18)=−x∼DEe−f(x)Ht−1(x)f(x)h(x)+23x∼DEe−f(x)Ht−1(x)=(−1)g1(h)+b(8.19)
将式(8.17)应用于式(8.19),即为【西瓜书p.176】第2个等号。
在处理类似
∑
i
α
i
x
i
\sum_i {\alpha }_ix_i
∑iαixi时,常常将系数“概率化”(使其和为1),形成加权平均数,即改为处理
∑
i
α
i
x
i
=
(
∑
j
α
j
)
∑
i
(
α
i
∑
j
α
j
)
x
i
=
(
∑
i
α
i
)
∑
i
(
α
i
∑
i
α
i
)
x
i
\begin{align} \sum_i {\alpha }_ix_i & =({\sum_j {\alpha }_j})\sum_i\left(\frac{ {\alpha }_i}{\sum_j {\alpha }_j}\right)x_i\notag \\ & =({\sum_i {\alpha }_i})\sum_i\left(\frac{ {\alpha }_i}{\sum_i {\alpha }_i}\right)x_i \tag{8.20} \end{align}
i∑αixi=(j∑αj)i∑(∑jαjαi)xi=(i∑αi)i∑(∑iαiαi)xi(8.20)
依此思路,将和式改为数学期望,应用到下式,则
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
f
(
x
)
h
(
x
)
=
(
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
)
(
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
(
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
)
f
(
x
)
h
(
x
)
)
=
a
g
2
(
h
)
(
a
>
0
)
\begin{align} & \qquad \mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}f(\boldsymbol{x})h(\boldsymbol{x})\notag \\ & =(\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} } \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})})\left(\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\frac{ \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}{(\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} } \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})})}f(\boldsymbol{x})h(\boldsymbol{x})\right)\notag \\ & =ag_2(h)\qquad (a>0) \tag{8.21} \end{align}
x∼DEe−f(x)Ht−1(x)f(x)h(x)=(x∼DEe−f(x)Ht−1(x))
x∼DE(x∼DEe−f(x)Ht−1(x))e−f(x)Ht−1(x)f(x)h(x)
=ag2(h)(a>0)(8.21)
将式(8.17)应用于式(8.21),即为【西瓜书p.176】第3个等号。
设
x
\boldsymbol{x}
x概率分布密度为
D
(
x
)
\mathcal{D} (\boldsymbol{x})
D(x)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
=
∫
R
n
e
−
f
(
x
)
H
t
−
1
(
x
)
D
(
x
)
d
x
\begin{align} \mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})} & =\int_{{\mathbb{R} }^n}\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}\mathcal{D} (\boldsymbol{x}) \,\mathrm{d}\boldsymbol{x} \tag{8.22a} \end{align}
x∼DEe−f(x)Ht−1(x)=∫Rne−f(x)Ht−1(x)D(x)dx(8.22a)
即
∫
R
n
e
−
f
(
x
)
H
t
−
1
(
x
)
D
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
d
x
=
∫
R
n
e
−
f
(
x
)
H
t
−
1
(
x
)
D
(
x
)
d
x
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
=
1
(由式(8.22a))
\begin{align} & \int_{{\mathbb{R} }^n} \frac{\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}\mathcal{D} (\boldsymbol{x})}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}} \,\mathrm{d}\boldsymbol{x} \tag{8.22} \\ & \ = \frac{\int_{{\mathbb{R} }^n}\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}\mathcal{D} (\boldsymbol{x}) \,\mathrm{d}\boldsymbol{x}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}\notag \\ & \ =1\qquad \text{(由式(8.22a))}\notag \end{align}
∫Rnx∼DEe−f(x)Ht−1(x)e−f(x)Ht−1(x)D(x)dx =x∼DEe−f(x)Ht−1(x)∫Rne−f(x)Ht−1(x)D(x)dx =1(由式(8.22a))(8.22)
积分为1,符合概率分布密度定义,即式(8.22)中的被积函数可视为一个概率分布密度,记为
D
t
(
x
)
\mathcal{D}_t (\boldsymbol{x})
Dt(x),即【西瓜书式(8.15)】,这样就找到了与
t
t
t关联的概率分布密度
D
t
\mathcal{D}_t
Dt,即可形成序列。
下面寻找该序列的递推关系式。 由
D
t
(
x
)
\mathcal{D}_t (\boldsymbol{x})
Dt(x)的定义(式(8.22)中的被积函数),有
D
t
+
1
(
x
)
=
D
(
x
)
e
−
f
(
x
)
H
t
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
(
x
)
\begin{align} \mathcal{D}_{t+1}(\boldsymbol{x}) =\frac{\mathcal{D} (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x})H_{t}(\boldsymbol{x})}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t}(\boldsymbol{x})}} \tag{8.23} \end{align}
Dt+1(x)=x∼DEe−f(x)Ht(x)D(x)e−f(x)Ht(x)(8.23)
分母为常数,分子为
D
(
x
)
e
−
f
(
x
)
H
t
(
x
)
=
D
(
x
)
e
−
f
(
x
)
[
H
t
−
1
(
x
)
+
α
t
h
t
(
x
)
]
(由
H
t
的定义【西瓜书式(8.4)】)
=
D
(
x
)
e
−
f
(
x
)
H
t
−
1
(
x
)
e
−
f
(
x
)
α
t
h
t
(
x
)
=
e
−
f
(
x
)
α
t
h
t
(
x
)
D
(
x
)
e
−
f
(
x
)
H
t
−
1
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
=
e
−
f
(
x
)
α
t
h
t
(
x
)
D
t
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
(由
D
t
的定义)
\begin{align} &\quad \mathcal{D} (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x})H_{t}(\boldsymbol{x})}\notag\\ & =\mathcal{D} (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x})[H_{t-1}(\boldsymbol{x})+{\alpha }_th_t(\boldsymbol{x})]} \quad \text{(由$H_t$的定义【西瓜书式(8.4)】)}\notag \\ & =\mathcal{D} (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}\,\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})}\notag \\ & =\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})}\frac{\mathcal{D} (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}\notag \\ & =\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})}\mathcal{D}_t (\boldsymbol{x})\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}\quad \text{(由$\mathcal{D}_t$的定义)} \tag{8.24} \end{align}
D(x)e−f(x)Ht(x)=D(x)e−f(x)[Ht−1(x)+αtht(x)](由Ht的定义【西瓜书式(8.4)】)=D(x)e−f(x)Ht−1(x)e−f(x)αtht(x)=e−f(x)αtht(x)x∼DEe−f(x)Ht−1(x)D(x)e−f(x)Ht−1(x)x∼DEe−f(x)Ht−1(x)=e−f(x)αtht(x)Dt(x)x∼DEe−f(x)Ht−1(x)(由Dt的定义)(8.24)
把式(8.24)代入式(8.23),得
D
t
+
1
(
x
)
=
e
−
f
(
x
)
α
t
h
t
(
x
)
D
t
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
(
x
)
=
D
t
(
x
)
e
−
f
(
x
)
α
t
h
t
(
x
)
C
t
其中,
C
t
=
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
(
x
)
\begin{align} D_{t+1}(\boldsymbol{x}) & =\frac{\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})}\mathcal{D}_t (\boldsymbol{x})\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t}(\boldsymbol{x})}} \notag \\ & =\mathcal{D}_t (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})}C_t \tag{8.25} \\ \text{其中,}C_t & =\frac{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\,\mathrm{e}^{-f(\boldsymbol{x})H_{t}(\boldsymbol{x})}} \tag{8.26} \end{align}
Dt+1(x)其中,Ct=x∼DEe−f(x)Ht(x)e−f(x)αtht(x)Dt(x)x∼DEe−f(x)Ht−1(x)=Dt(x)e−f(x)αtht(x)Ct=x∼DEe−f(x)Ht(x)x∼DEe−f(x)Ht−1(x)(8.25)(8.26)
由于
E
\mathbb{E}
E起了消元作用,故
C
t
C_t
Ct对
x
\boldsymbol{x}
x而言是常数,式(8.25)即为【西瓜书式(8.19)】。
由于
D
(
x
)
D(\boldsymbol{x})
D(x)是未知,故式(8.26)不好直接应用,对式(8.25)两边求积分,则
∫
R
n
D
t
+
1
(
x
)
d
x
=
∫
R
n
D
t
(
x
)
e
−
f
(
x
)
α
t
h
t
(
x
)
C
t
d
x
1
=
C
t
∫
R
n
D
t
(
x
)
e
−
f
(
x
)
α
t
h
t
(
x
)
d
x
=
C
t
Z
t
其中,
Z
t
=
∫
R
n
D
t
(
x
)
e
−
f
(
x
)
α
t
h
t
(
x
)
d
x
\begin{align} \int_{{\mathbb{R} }^n} D_{t+1}(\boldsymbol{x}) \,\mathrm{d}\boldsymbol{x} & =\int_{{\mathbb{R} }^n} \mathcal{D}_t (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})}C_t \,\mathrm{d}\boldsymbol{x}\notag \\ 1 & =C_t\int_{{\mathbb{R} }^n} \mathcal{D}_t (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})} \,\mathrm{d}\boldsymbol{x}\notag \\ & =C_tZ_t \tag{8.27} \\ \text{其中,}Z_t & =\int_{{\mathbb{R} }^n} \mathcal{D}_t (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})} \,\mathrm{d}\boldsymbol{x} \tag{8.28} \end{align}
∫RnDt+1(x)dx1其中,Zt=∫RnDt(x)e−f(x)αtht(x)Ctdx=Ct∫RnDt(x)e−f(x)αtht(x)dx=CtZt=∫RnDt(x)e−f(x)αtht(x)dx(8.27)(8.28)
由式(8.27),式(8.26)变为
D
t
+
1
(
x
)
=
D
t
(
x
)
e
−
f
(
x
)
α
t
h
t
(
x
)
Z
t
\begin{align} D_{t+1}(\boldsymbol{x}) =\frac{\mathcal{D}_t (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x}){\alpha }_th_t(\boldsymbol{x})}}{Z_t} \tag{8.29} \end{align}
Dt+1(x)=ZtDt(x)e−f(x)αtht(x)(8.29)
其中,
Z
t
Z_t
Zt为式(8.28),在时刻
t
+
1
t+1
t+1,它的表达式中没有未知项。
即找到了递推式(8.29),用于AdaBoost算法【西瓜书图8.3】。
回到【西瓜书式(8.14)】,其的计算涉及到未知的分布
D
\mathcal{D}
D,需要作进一步的变换:
E
x
∼
D
[
e
−
f
(
x
)
H
t
−
1
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
f
(
x
)
h
(
x
)
]
=
∫
R
n
[
e
−
f
(
x
)
H
t
−
1
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
f
(
x
)
h
(
x
)
]
D
(
x
)
d
x
=
∫
R
n
[
D
(
x
)
e
−
f
(
x
)
H
t
−
1
(
x
)
E
x
∼
D
e
−
f
(
x
)
H
t
−
1
(
x
)
]
f
(
x
)
h
(
x
)
d
x
=
∫
R
n
D
t
(
x
)
f
(
x
)
h
(
x
)
d
x
(由
D
t
的定义)
=
E
x
∼
D
t
[
f
(
x
)
h
(
x
)
]
\begin{align} & \mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} }\left[\frac{ \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} } \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}f(\boldsymbol{x})h(\boldsymbol{x})\right]\notag \\ & \ =\int_{{\mathbb{R} }^n} \left[\frac{ \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} } \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}f(\boldsymbol{x})h(\boldsymbol{x})\right] \mathcal{D} (\boldsymbol{x}) \,\mathrm{d}\boldsymbol{x}\notag \\ & \ =\int_{{\mathbb{R} }^n} \left[\frac{ \mathcal{D} (\boldsymbol{x})\mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}{\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D} } \mathrm{e}^{-f(\boldsymbol{x})H_{t-1}(\boldsymbol{x})}}\right]f(\boldsymbol{x})h(\boldsymbol{x}) \,\mathrm{d}\boldsymbol{x}\notag \\ & \ =\int_{{\mathbb{R} }^n} \mathcal{D}_{t}(\boldsymbol{x}) f(\boldsymbol{x})h(\boldsymbol{x})\,\mathrm{d}\boldsymbol{x}\qquad \text{(由$\mathcal{D}_{t}$的定义)}\notag \\ & \ =\mathop{\mathbb{E} }\limits_{\boldsymbol{x}\thicksim \mathcal{D}_t }[f(\boldsymbol{x})h(\boldsymbol{x})] \tag{8.30} \end{align}
x∼DE
x∼DEe−f(x)Ht−1(x)e−f(x)Ht−1(x)f(x)h(x)
=∫Rn
x∼DEe−f(x)Ht−1(x)e−f(x)Ht−1(x)f(x)h(x)
D(x)dx =∫Rn
x∼DEe−f(x)Ht−1(x)D(x)e−f(x)Ht−1(x)
f(x)h(x)dx =∫RnDt(x)f(x)h(x)dx(由Dt的定义) =x∼DtE[f(x)h(x)](8.30)
而这时,
D
t
\mathcal{D}_t
Dt是已知的,式(8.30)代入【西瓜书式(8.14)】就得到了【西瓜书式(8.16)】。 其中
f
(
x
)
h
(
x
)
=
{
1
,
当
f
(
x
)
=
h
(
x
)
时
−
1
,
当
f
(
x
)
≠
h
(
x
)
时
=
I
(
f
(
x
)
=
h
(
x
)
)
−
I
(
f
(
x
)
≠
h
(
x
)
)
(由式(B3)、式(B4))
=
(
1
−
I
(
f
(
x
)
≠
h
(
x
)
)
)
−
I
(
f
(
x
)
≠
h
(
x
)
)
=
【西瓜书式(8.17)】
\begin{align} f(\boldsymbol{x})h(\boldsymbol{x}) & = \begin{cases} \, 1\, ,&\qquad \text{当$f(\boldsymbol{x})=h(\boldsymbol{x})$时}\notag \\ \, -1\, ,&\qquad \text{当$f(\boldsymbol{x})\neq h(\boldsymbol{x})$时}\notag \\ \end{cases} \\ & =\mathbb{I} (f(\boldsymbol{x})=h(\boldsymbol{x}))-\mathbb{I} (f(\boldsymbol{x})\neq h(\boldsymbol{x}))\qquad \text{(由式(B3)、式(B4))}\notag \\ & = (1-\mathbb{I} (f(\boldsymbol{x})\neq h(\boldsymbol{x})))-\mathbb{I} (f(\boldsymbol{x})\neq h(\boldsymbol{x}))\notag \\ & =\text{【西瓜书式(8.17)】}\notag \end{align}
f(x)h(x)={1,−1,当f(x)=h(x)时当f(x)=h(x)时=I(f(x)=h(x))−I(f(x)=h(x))(由式(B3)、式(B4))=(1−I(f(x)=h(x)))−I(f(x)=h(x))=【西瓜书式(8.17)】
其中,参见6、指示函数及应用(将分段函数表达成一个式子的技术):
f
(
x
)
=
a
(
x
)
I
A
(
x
)
+
b
(
x
)
I
A
‾
(
x
)
\begin{align} f(\boldsymbol{x}) & =a(\boldsymbol{x})\mathbb{I}_A (\boldsymbol{x})+b(\boldsymbol{x})\mathbb{I}_{\overline{A}} (\boldsymbol{x}) \tag{B4} \end{align}
f(x)=a(x)IA(x)+b(x)IA(x)(B4)
【西瓜书式(8.17)】代入【西瓜书式(8.16)】并应用式(8.17),即得【西瓜书式(8.18)】,这是表达式非常简洁的基学习器 h t h_t ht。 【西瓜书图8.3】算法中第3行实为【西瓜书式(8.18)】。
本文为原创,您可以:
- 点赞(支持博主)
- 收藏(待以后看)
- 转发(他考研或学习,正需要)
- 评论(或讨论)
- 引用(支持原创)
- 不侵权
上一篇:8.2 AdaBoost算法(三合一:分布演进、集成投票、权重优选)
下一篇:8.4 GBDT算法(逐次估值逼近)