机器学习基石笔记（五）：训练与测试

Lecture 5: Training versus Testing

Recap and Preview

Two Central Questions

  make sure that $E_{out}(g)$ is close enough to $E_{in}(g)$.
  make $E_{in}(g)$mall enough.

Fun Time

Data size: how large do we need?
One way to use the inequality
$$\mathbb{P}\left[|E_{\text{ in }}(g)-E_{\text{ out }}(g)|>ϵ\right]\le \underset{\delta }{\underbrace{2\cdot M\cdot \exp \left(-2{ϵ}^{2}N\right)}}$$
is to pick a tolerable difference ? as well as a tolerable BAD probability δ, and then gather data with size (N) large enough to achieve those tolerance criteria. Let ? = 0.1, δ = 0.05, and M = 100.
What is the data size needed?
1.  215              2. 415  $✓$          3. 615           4. 815


Explanation
$N=\frac{1}{2{ϵ}^2}\ln \frac{2M}{\delta }$
所以$N=414.7\approx 415$

Effective Number of Lines

Uniform Bound

$$B_m:|E_{\text{ in }}\left(h_m\right)-E_{\text{ out }}\left(h_m\right)|>ϵ$$

for most $D$,$E_{\mathrm{i}\mathrm{n}}(h_i)=E_{\mathrm{i}\mathrm{n}}(h_j)$
$E_{\mathrm{o}\mathrm{u}\mathrm{t}}(h_i)\approx E_{\mathrm{o}\mathrm{u}\mathrm{t}}(h_j)$

$\begin{array}{r}{\mathbb{P}}_{\mathcal{D}}[\mathrm{B}\mathrm{A}\mathrm{D}\mathcal{D}for{h}_1]+{\mathbb{P}}_{\mathcal{D}}[\mathrm{B}\mathrm{A}\mathrm{D}\mathcal{D}for{h}_2]+...+{\mathbb{P}}_{\mathcal{D}}[\mathrm{B}\mathrm{A}\mathrm{D}\mathcal{D}for{h}_M](unionbound)\end{array}$ over-estimating

为了合并重叠的部分，我们按照类别将类似的假设分组归类。

Many Lines

$$\mathcal{H}=\left\{\text{ all lines in }{\mathbb{R}}^2\right\}$$

Effective Number of Hypotheses

1点2个类别

2个点4个类别

3个点6个类别

4个点14个类别

$effective(N)<<2^N$,用有效类别数($effective(N)$)替换$M$(infite)

Fun Time

What is the effective number of lines for five inputs $ \in R^2$ ?
1.  14              2. 16           3. 22 $✓$           4. 32


Explanation
总共有32($2^5$)种
5个O:1种，4个O1个X:5种，3个O2个X:共有$C_5^3$10种，只有5种满足条件。
而又O和X等价
所以有效类别数共有2X(1+5+5)=22种。

Growth Function

$effective(N)<<2^N$
$M\Rightarrow effective(N)$

Growth Function for Positive Rays

$$\begin{gathered} h(x)=\mathrm{sign}(x-a) \\ {m}_{\mathcal{H}}(N)=N+1\ll 2^N \end{gathered}$$

Growth Function for Positive Intervals

$$\begin{gathered} h(x)=+1\text{ if }x\in [\ell ,r),-1otherwise \\ {m}_{\mathcal{H}}(N)=\frac{1}{2}{N}^2+\frac{1}{2}N+1\ll 2^N \end{gathered}$$

Growth Function for Convex Sets

$$\begin{gathered} everydichotomycanbeimplemented \\ {m}_{\mathcal{H}}(N)=2^N \end{gathered}$$

$m_{\mathcal{H}}(N)=2^N$ call those $N$ inputs ‘shattered’ by $H$

Fun Time

Consider positive and negative rays as H, which is equivalent to the perceptron hypothesis set in 1D. The hypothesis set is often called ‘decision stump’ to describe the shape of its hypotheses. What is the growth function $m_H(N)$?
1.  N             2. N+1           3. 2N $✓$           4. $2^N$


Explanation
正或负方向的Positive Intervals，再加上全X和全O。
2X(N-1)+2 = 2N

Break Point

$m_{\mathcal{H}}$是多项式($O\left(N^{k-1}\right)$)，我们用它替换$M$

$m_{\mathcal{H}}(k)<2^k$ call k a break point for $H$

if k is a break point, k+1, k+2, k+3,… also break points! k is the minimum break point.

Fun Time

Consider positive and negative rays as H, which is equivalent to the perceptron hypothesis set in 1D. As discussed in an earlier quiz question, the growth function $m_H(N)=2N$. What is the minimum break point for $H$?
1.  1             2. 2          3. 3 $✓$           4. 4


Explanation
$$\begin{gathered} m_H(N)=2N \\ 2\ast 2=4=2^2,2\ast 3=6<2^3 \end{gathered}$$
正负线的"一线曙光"是3.

Summary

讲义总结

$m_{\mathcal{H}}$是多项式($O\left(N^{k-1}\right)$)，我们用它替换$M$，降低假设集的上界。
通过minimum break point，求出多项式的幂级数。

Recap and Preview
  two questions: $E_{out}(g)\approx E_{in}(g)$, and $E_{in}(g)\approx 0$

Effective Number of Lines
  at most 14 through the eye of 4 inputs

Effective Number of Hypotheses
  at most $m_H(N)$ through the eye of $N$ inputs

Break Point
  when $m_H(N)$ becomes ‘non-exponential’

参考文献

《Machine Learning Foundations》(机器学习基石)—— Hsuan-Tien Lin (林轩田)