机器学习基石笔记（六）：泛化理论

Lecture 6: Theory of Generalization

Restriction of Break Point

$$\begin{array}{rl}& m_{\mathcal{H}}(N)\\ \le & \text{ maximum possible }{m}_{\mathcal{H}}(N)\text{ given }k\\ \le & poly(N)\end{array}$$

Fun Time

When minimum break point k = 1, what is the maximum possible $m_{\mathcal{H}}(N)$ when $N=3$？
1.  1 $✓$             2. 2          3. 3           4. 4


Explanation
因为$k=1$，所以没有任何一个点可以和它共存，所以$m_H(N)=1$

Bounding Function: Basic Cases

Bounding Function

bounding function $B(N,k)$:
  maximum possible $m_H(N)$ when break point = k
$$B(N,k)\le poly(N)$$

换言之，$B(N,k)$是$m_H(N)$的上界。

Table of Bounding Function

Fun Time

For the 2D perceptrons, which of the following claim is true?
1 minimum break point k = 2
2 $m_{\mathcal{H}}(4)$= 15
3 $m_{\mathcal{H}}(N)<B(N,k)$ when $N = k = $ minimum break point  $✓$
4 $m_{\mathcal{H}}(N)>B(N,k)$ when $N = k = $ minimum break point


Explanation
minimum break point k = 3
$m_{\mathcal{H}}(4)$= 14
$B(N,k)$是$m_H(N)$的上界
不记得2D感知器的同学，可以回顾Lecture 5: Training versus Testing中的Effective Number of Hypotheses ?

Bounding Function: Inductive Cases

$$B(4,3)=11=2\alpha +\beta$$



$$\begin{gathered} \begin{array}{rl}B(N,k)& =2\alpha +\beta \\ \alpha +\beta & \le B(N-1,k)\\ \alpha & \le B(N-1,k-1)\\ \Rightarrow B(N,k)& \le B(N-1,k)+B(N-1,k-1)\end{array} \\ B(N,k)\le \sum _{i=0}^{k-1}\left(\begin{array}{c}N\\ i\end{array}\right) \end{gathered}$$

$\le$ 实际上是$=$
即
$$\begin{gathered} B(N,k)=B(N-1,k)+B(N-1,k-1) \\ B(N,k)=\sum _{i=0}^{k-1}\left(\begin{array}{c}N\\ i\end{array}\right)=C_N^0+C_N^1+...+C_N^{k-1} \end{gathered}$$

2D perceptrons break point at 4， $m_{\mathcal{H}}(N)\le B(N,4)=\frac{1}{6}{N}^3+\frac{5}{6}N+1=O(N^3)$

Fun Time

For 1D perceptrons (positive and negative rays), we know that $m_H(N)$ = 2N. Let k be the minimum break point. Which of the following is not true?
1 k = 3
2 for some integers $N>0,m_{\mathcal{H}}(N)={\sum }_{i=0}^{k-1}\left(\begin{array}{c}N\\ i\end{array}\right)$
3 for all integers $N>0,m_{\mathcal{H}}(N)={\sum }_{i=0}^{k-1}\left(\begin{array}{c}N\\ i\end{array}\right)$  $✓$
4 for all integers $N>2,m_{\mathcal{H}}(N)<{\sum }_{i=0}^{k-1}\left(\begin{array}{c}N\\ i\end{array}\right)$


Explanation
minimum break point k = 3
$B(N,k)={\sum }_{i=0}^{k-1}\left(\begin{array}{c}N\\ i\end{array}\right)$
$B(N,k)$是$m_H(N)$的上界，当N$\ge$k时，$m_H(N)<B(N,k)$; 当N$<$k时，$m_H(N)=B(N,k)$.


拓展：回顾下Lecture 5: Training versus Testing中的Effective Number of Hypotheses Funtime
求2维感知器中5个点的有效分类数(k=3,N=5 $m_{\mathcal{H}}(N)=?\le \frac{1}{6}{N}^3+\frac{5}{6}N+1$)，N>k，=取不到。
正确答案22<($\frac{125}{6}+\frac{25}{6}+1=25$)，验证成功，回顾题目也挺有趣味的。?

A Pictorial Proof

用$E_{in}^{\prime }$（有限）替换$E_{out}$(无限)，但是这个不等式及$\frac{1}{2}$的系数的出处，我没想明白。

将上界定义为以$m_H(2N)$为基准的。

使用无放回的霍夫丁不等式，结果类似，只是$\nu =E_{\text{ in }},\mu =\frac{E_{\text{ in }}+E_{\text{ in }}^{\prime }}{2}$。

Vapnik-Chervonenkis (VC) bound

$$\begin{array}{rl}& \mathbb{P}\left[\exists h\in \mathcal{H}\text{ s.t. }|E_{\text{ in }}(h)-E_{\text{ out }}(h)|>ϵ\right]\\ & \le 4m_{\mathcal{H}}(2N)\exp \left(-\frac{1}{8}{ϵ}^{2}N\right)\end{array}$$
  $m_H(N)$ can replace M with a few changes

Fun Time

For positive rays, $m_H(N)=N+1$. Plug it into the VC bound for ? = 0.1 and N = 10000. What is VC bound of BAD events?
$$\mathbb{P}\left[\exists h\in \mathcal{H}\text{ s.t. }|E_{\text{ in }}(h)-E_{\text{ out }}(h)|>ϵ\right]\le 4m_{\mathcal{H}}(2N)\exp \left(-\frac{1}{8}{ϵ}^{2}N\right)$$
1 $2.77\times 10^{-87}$
2 $5.54\times 10^{-83}$
3 $2.98\times 10^{-1}$  $✓$
4 $2.29\times 10^{-2}$


Explanation
代入公式计算即可。
0.2981471603789822

Summary

本篇讲义主要讲了Bound Function$B(N,k)$以及VC Bound的含义及推导。

讲义总结

若$m_{\mathcal{H}}(N)$有break point，且$N$足够大，那么$E_{\mathrm{o}\mathrm{u}\mathrm{t}}\approx E_{\mathrm{i}\mathrm{n}}$.

Restriction of Break Point
  break point ‘breaks’ consequent points

Bounding Function: Basic Cases
  $B(N,k)$ bounds $m_H(N)$ with break point k

Bounding Function: Inductive Cases
  $B(N,k)$ is poly(N)

A Pictorial Proof
  $m_H(N)$ can replace M with a few changes

参考文献

《Machine Learning Foundations》(机器学习基石)—— Hsuan-Tien Lin (林轩田)