核方法(Kernel method)通常需要选取可函数及其超参数的取值。重采样(如: K-fold cross validation 和 Bootstrap)可以处理这个问题,但是通常很耗时。全局Rademacher复杂度和局部Rademacher复杂度是两个测量假设空间复杂度的量。Rademacher复杂度在多标签学习里已经有一些成果,感兴趣的读者可以自行查阅这方面的文献。
Rademacher 复杂度
它的官方定义为: R n ( H ) = E sup h ∈ H 2 n ∑ i = 1 n σ i l ( h ( X i , Y i ) ) R_{n}(\mathcal{H})=\mathbb{E}\sup_{h\in \mathcal{H}}\frac{2}{n}\sum_{i=1}^{n}\sigma_{i}l(h(X_{i},Y_{i})) Rn(H)=Esuph∈Hn2∑i=1nσil(h(Xi,Yi)),
这就是所谓的Global Rademacher复杂度了, 它的经验版本为:
R n ^ ( H ) = E σ sup h ∈ H 2 n ∑ i = 1 n σ i l ( h ( X i , Y i ) ) \hat{R_{n}}(\mathcal{H})=\mathbb{E}_{\sigma}\sup_{h\in \mathcal{H}}\frac{2}{n}\sum_{i=1}^{n}\sigma_{i}l(h(X_{i},Y_{i})) Rn^(H)=Eσsuph∈Hn2∑i=1nσil(h(Xi,Yi)).
以下是经典结果: 对于损失函数 l : X × Y → [ 0 , 1 ] l:X\times Y \to [0,1] l:X×Y→[0,1],任意的假设 h h h,以下不等式以概率 ≥ 1 − 2 e − x \geq 1-2e^{-x} ≥1−2e−x成立:
L ( h ) ≤ L n ( h ) + R n ^ ( H ) + 3 x 2 n L(h)\leq L_{n}(h)+\hat{R_{n}}(\mathcal{H})+3\sqrt{\frac{x}{2n}} L(h)≤Ln(h)+Rn^(H)+32nx .
而local Rademacher复杂度经典结果为:
令
H
^
r
,
a
,
x
,
n
=
{
h
∣
h
∈
H
,
L
^
n
2
≤
1
α
2
(
3
r
+
x
2
n
)
}
\hat{\mathcal{H}}_{r,a,x,n}=\{h|h\in \mathcal{H},\hat{L}^{2}_{n}\leq \frac{1}{\alpha^2}(3r+\sqrt{\frac{x}{2n}})\}
H^r,a,x,n={h∣h∈H,L^n2≤α21(3r+2nx)},
r
^
n
\hat{r}_{n}
r^n为子根函数
ψ ^ n ( r ) = sup α ∈ ( 0 , 1 ] α R ^ n ( H ^ r , a , x , n ) + 2 x n \hat{\psi}_{n}(r)=\sup_{\alpha \in (0,1]}\alpha \hat{R}_{n}(\hat{\mathcal{H}}_{r,a,x,n})+\sqrt{\frac{2x}{n}} ψ^n(r)=supα∈(0,1]αR^n(H^r,a,x,n)+n2x
的固定点,对于损失函数 l : X × Y → [ 0 , 1 ] l:X\times Y \to [0,1] l:X×Y→[0,1],任意的假设 h h h,以下不等式以概率 ≥ 1 − 3 e − x \geq1-3e^{-x} ≥1−3e−x成立:
L ( h ) ≤ min K > 1 [ K K − 1 L ^ n ( h ) + K r ^ n ] + 2 x 2 n L(h)\leq \min_{K>1}[\frac{K}{K-1}\hat{L}_{n}(h)+K\hat{r}_{n}]+2\sqrt{\frac{x}{2n}} L(h)≤minK>1[K−1KL^n(h)+Kr^n]+22nx .
估算Example
Example 1
H
=
{
h
∣
h
(
X
)
=
<
W
,
ϕ
(
X
)
>
,
∣
∣
W
∣
∣
≤
H
}
\mathcal{H}=\{h| h(X)=<W,\phi(X)>,||W||\leq H\}
H={h∣h(X)=<W,ϕ(X)>,∣∣W∣∣≤H},则
一方面,
R
n
^
(
H
)
≤
H
n
∑
i
=
1
n
λ
i
=
H
n
∑
i
=
1
n
Q
i
,
i
\hat{R_{n}}(\mathcal{H})\leq\frac{H}{n}\sqrt{\sum_{i=1}^{n}\lambda_{i}}=\frac{H}{n}\sqrt{\sum_{i=1}^{n}Q_{i,i}}
Rn^(H)≤nH∑i=1nλi=nH∑i=1nQi,i;
另一方面,
R
n
^
(
H
)
≥
H
2
n
∑
i
=
1
n
λ
i
=
H
2
n
∑
i
=
1
n
Q
i
,
i
\hat{R_{n}}(\mathcal{H})\geq\frac{H}{\sqrt{2}n}\sqrt{\sum_{i=1}^{n}\lambda_{i}}=\frac{H}{\sqrt{2}n}\sqrt{\sum_{i=1}^{n}Q_{i,i}}
Rn^(H)≥2nH∑i=1nλi=2nH∑i=1nQi,i.
而且,当 ∣ ∣ ϕ ( X ) ∣ ∣ ≤ 1 ||\phi(X)||\leq 1 ∣∣ϕ(X)∣∣≤1时, 以下不等式以概率 ≥ 1 − 2 e − x \geq 1-2e^{-x} ≥1−2e−x成立:
L H ( h ) ≤ L T , n ( h ) + H n ∑ i = 1 n λ i + 3 x 2 n L_{H}(h)\leq L_{T,n}(h)+\frac{H}{n}\sqrt{\sum_{i=1}^{n}\lambda_{i}}+3\sqrt{\frac{x}{2n}} LH(h)≤LT,n(h)+nH∑i=1nλi+32nx .
Example 2
H
=
{
h
∣
h
(
X
)
=
<
W
,
ϕ
(
X
)
>
,
∣
∣
W
∣
∣
≤
2
H
,
∣
∣
Φ
W
∣
∣
2
≤
c
2
}
\mathcal{H}=\{h| h(X)=<W,\phi(X)>,||W||\leq 2H, ||\Phi W||^2\leq c_{2}\}
H={h∣h(X)=<W,ϕ(X)>,∣∣W∣∣≤2H,∣∣ΦW∣∣2≤c2},则
一方面,
R
n
^
(
H
)
≤
2
H
∑
i
=
1
n
min
[
c
2
4
,
λ
i
]
\hat{R_{n}}(\mathcal{H})\leq 2H\sqrt{\sum_{i=1}^{n}\min[\frac{c_{2}}{4},\lambda_{i}]}
Rn^(H)≤2H∑i=1nmin[4c2,λi],
另一方面,
R
n
^
(
H
)
≥
c
3
H
∑
i
=
1
n
min
[
c
2
4
,
λ
i
]
\hat{R_{n}}(\mathcal{H})\geq c_{3}H\sqrt{\sum_{i=1}^{n}\min[\frac{c_{2}}{4},\lambda_{i}]}
Rn^(H)≥c3H∑i=1nmin[4c2,λi].