ML Note 2 - Learning Theory

Bias / Variance Tradeoff

In the diagram below, we modeled the training set with $2$, $3$, and $7$ parameters

As we can see, $h_1$ is an underfit of the training set. No matter how large the training set grows, the model cannot capture the structure of the data. This model is said to have high bias.

On the contrary, $h_6$ is an overfit of the training set. The model is too sensitive to random factors that we don’t want to include in our model. This model is said to have large variance.

ERM

For the purpose of simplicity, consider binary classification with

• labels y ∈ { 0 , 1 } y \in \{0,1\} y∈{0,1}
• training set S = { ( x ( i ) , y ( i ) ) ∣ i = 1 , … , m } S = \{(x^{(i)}, y^{(i)}) | i = 1,\dots,m\} S={(x(i),y(i))∣i=1,…,m}
• new sample $(x,y)$

As one of the PAC (probably approximately correct) assumptions, assume that there exists some distribution $D$ such that
$$(x,y),(x^{(1)},y^{(1)}),\dots ,(x^{(m)},y^{(m)})\stackrel{i.i.d.}{\sim }D$$ For a hypothesis $h$ define
Z = I { h ( x ) ≠ y } Z i = I { h ( x ( i ) ) ≠ y ( i ) } \begin{array}{rcl} Z &=& I\{h(x) \neq y\}\\ Z_i &=& I\{h(x^{(i)}) \neq y^{(i)}\} \end{array} ZZi​​==​I{h(x)​=y}I{h(x(i))​=y(i)}​ Note training error or empirical risk as
$$\widehat{ϵ}(h)=\frac{1}{m}\sum _{i=1}^m{Z}_i$$ and generalization error
$$ϵ(h)=P(Z)$$ Then it is obvious that
$$Z,Z_1,Z_2,\dots ,Z_m\stackrel{i.i.d.}{\sim }Bern(ϵ(h))$$ Think of the problem as we are picking $h$ from a hypothesis class, for instance
H = { h θ ∣ θ ∈ R n + 1 } H = \{h_\theta|\theta\in\mathbb{R}^{n+1}\} H={hθ​∣θ∈Rn+1} and our goal is empirical risk minimization
$$\hat{h}=\arg \underset{h\in H}{\min }\widehat{ϵ}(h)$$ Define the theoretically best hypothesis in $H$
$$h^{\ast }=\arg \underset{h\in H}{\min }ϵ(h)$$

Uniform Convergence

Suppose H = { h 1 , … , h k } H = \{h_1,\dots,h_k\} H={h1​,…,hk​} is a finite set. For any $h_i\in H$ donate
A i = I { ∣ ϵ ( h i ) − ϵ ^ ( h i ) ∣ > γ } A_i = I\{|\epsilon(h_i) - \hat\epsilon(h_i)|>\gamma\} Ai​=I{∣ϵ(hi​)−ϵ^(hi​)∣>γ} Hoeffding inequality gives that
$$P(A_i)\le 2\exp (-2{\gamma }^{2}m)$$ Using the union bound, we have that
$$\begin{array}{rcl}P(\exists h_i\in H,|ϵ(h_i)-\widehat{ϵ}(h_i)|>\gamma )& =& P(\bigcup _{i=1}^k{A}_i)\\ & \le & \sum _{i=1}^{k}P(A_i)\\ & \le & 2k\exp (-2{\gamma }^{2}m)\end{array}$$ Therefore
$$P(\forall h\in H,|ϵ(h_i)-\widehat{ϵ}(h_i)|\le \gamma )\ge 1-2k\exp (-2{\gamma }^{2}m)$$

Error bound

Given $m$ and $\delta >0$, with probability $1-\delta$ we have that
$$\forall h\in H,|ϵ(h)-\widehat{ϵ}(h)|\le \sqrt{\frac{1}{2m}\log \frac{2k}{\delta }}$$ Since
$$\widehat{ϵ}(\hat{h})\le \widehat{ϵ}(h^{\ast })$$ we have that
$$ϵ(\hat{h})\le (\underset{h\in H}{\min }ϵ(h))+2\sqrt{\frac{1}{2m}\log \frac{2k}{\delta }}$$ When we expand $H$ to some super-set $H’\supset H$, the first term $\underset{h\in H}{\min }ϵ(h)$ can only decrease, while the second term $\sqrt{\frac{1}{2m}\log \frac{2k}{\delta }}$ can only increase. This loosely corresponds to the bias-variance tradeoff.

Sample Complexity Bound

Given $\gamma$ and $\delta >0$ in order for
$$ϵ(\hat{h})\le ϵ(h^{\ast })+2\gamma$$ to be true with probability at least $1-\delta$ , it suffices that
$$m\ge \frac{1}{2{\gamma }^2}\log \frac{2k}{\delta }=O(\frac{1}{{\gamma }^2}\log \frac{k}{\delta })$$

VC Dimension

Given a set S = { x ( 1 ) , … , x ( d ) } S = \{x^{(1)}, \dots, x^{(d)}\} S={x(1),…,x(d)}, we say that $H$ shatters $S$ if
∀ { y ( 1 ) , … , y ( d ) } ∈ { 0 , 1 } d ,   ∃ h ∈ H ,   s . t .   ∀ i ∈ { 1 , … , d } ,   h ( x ( i ) ) = y ( i ) \forall \{y^{(1)},\dots,y^{(d)}\} \in \{0,1\}^d,\ \exist h\in H,\\\ s.t.\ \forall i \in \{1,\dots,d\},\ h(x^{(i)}) = y^{(i)} ∀{y(1),…,y(d)}∈{0,1}d, ∃h∈H, s.t. ∀i∈{1,…,d}, h(x(i))=y(i) Define Vapnik-Chervonenkis dimension $VC(H)$ to be the size of the largest set that is shattered by $H$. It can be shown that if $H$ contains all linear classifiers in $n$ dimensional, then $VC(H)=n+1$.

For SVMs using kernel, VC dimension is usually small. If
∃ R ,   s . t . ∀ i ∈ { 1 , … , m } ,   ∣ ∣ x ( i ) ∣ ∣ ≤ R \exist R,\ s.t. \forall i \in \{1,\dots,m\},\ ||x^{(i)}|| \le R ∃R, s.t.∀i∈{1,…,m}, ∣∣x(i)∣∣≤R then
$$VC(H)\le \lceil \frac{R^2}{4{\gamma }^2}\rceil +1$$

---

Let $H$ be given and let $d=VC(H)\ne +\infty$. With probability at least $1-\delta$, we have that
$$\forall h\in H,|\widehat{ϵ}(h)-ϵ(h)|\le O\left(\sqrt{\frac{d}{m}\log \frac{m}{d}+\frac{1}{m}\log \frac{1}{\delta }}\right)$$ Thus
$$ϵ(\hat{h})\le (\underset{h\in H}{\min }ϵ(h))+O\left(\sqrt{\frac{d}{m}\log \frac{m}{d}+\frac{1}{m}\log \frac{1}{\delta }}\right)$$ Moreover

For $ϵ(\hat{h})\le ϵ(h^{\ast })+2\gamma$ to hold with probability at least $1-\delta$, it suffices that $m=O_{\gamma ,\delta }(d)$.

Model Selection

Suppse we have a finite set of models M = { M 1 , … , M d } M = \{M_1, \dots, M_d\} M={M1​,…,Md​} that we are trying to select among. There are several techniques to automatically deal with bias-variance tradeoff.

Cross Validation

In hold-out cross validation or simple cross validation, we
$$\begin{array}{rl}& \text{randomly split }S\text{ into }{S}_{\text{train}}\text{ and }{S}_{\text{cv}}(\text{ say }70\%\text{ to }30\%)\\ & \text{for i in 1...d}\\ & \text{train }{M}_i\text{ on }{S}_{\text{train}}\\ & \text{test }{M}_i\text{ on }{S}_{\text{cv}}\text{ to get }{ϵ}_i\\ & \text{choose }M=\arg \underset{M_i}{\min }{ϵ}_i\\ & (\text{optional})\text{ retrain }M\text{ on }S\end{array}$$ A main disadvantage of this algorithm is the waste of data. To hold out less data each time, we can use k-fold cross validation.
$$\begin{array}{rl}& \text{randomly split }S\text{ into }k\text{ disjoint subsets }{S}_1,\dots ,S_k\\ & \text{for i in 1...d}\\ & \text{for j in 1...k}\\ & \text{train }{M}_i\text{ on }S-S_j\\ & \text{test }{M}_i\text{ to get }{ϵ}_{ij}\\ & {ϵ}_i=\frac{1}{k}\sum _{j=1}^k{ϵ}_{ij}\\ & \text{choose }M=M_{\arg \underset{i}{\min }{ϵ}_i}\\ & (\text{optional})\text{ retrain }M\text{ on }S\end{array}$$ A typical choice for $K$ is $10$. But for problems with really scarce data, we can use $k=m$, which lead to the leave-one-out cross validation.

Feature Selection

Suppose you have a supervised learning problem where $n$ is very large, but you suspect that only a small number of features are relevant to the task. In such a setting, you can apply some heuristic algorithms to rank every feature and choose the top-$k$.

Wrapper Model Feature Selection

In a forward search, use $F$ to record the most relevant features
F : = ϕ repeat  { for i in 1...n  { if  ( i ∈ F )  continue F i : = F ∪ { i } } cross validate over  { M i ∣ M i  depends only on  F i } F : = result } \begin{aligned} & F := \phi\\ & \text{repeat }\{\\ & \qquad\text{for i in 1...n }\{\\ & \qquad\qquad\text{if }(i \in F)\ \text{continue}\\ & \qquad\qquad F_i := F \cup\{i\}\\ & \qquad\}\\ & \qquad\text{cross validate over } \{M_i| M_i \text{ depends only on } F_i\}\\ & \qquad F := \text{result}\\ & \} \end{aligned} ​F:=ϕrepeat {for i in 1...n {if (i∈F) continueFi​:=F∪{i}}cross validate over {Mi​∣Mi​ depends only on Fi​}F:=result}​ Similarly, backward search
F : = { 1 , … , n } repeat  { for i in 1...n  { if  ( i ∉ F )  continue F i : = F − { i } } cross validate over  { M i ∣ M i  depends only on  F i } F : = result } \begin{aligned} & F := \{1,\dots,n\}\\ & \text{repeat }\{\\ & \qquad\text{for i in 1...n }\{\\ & \qquad\qquad\text{if }(i \notin F)\ \text{continue}\\ & \qquad\qquad F_i := F -\{i\}\\ & \qquad\}\\ & \qquad\text{cross validate over } \{M_i| M_i \text{ depends only on } F_i\}\\ & \qquad F := \text{result}\\ & \} \end{aligned} ​F:={1,…,n}repeat {for i in 1...n {if (i∈/​F) continueFi​:=F−{i}}cross validate over {Mi​∣Mi​ depends only on Fi​}F:=result}​ Wrapper feature selection algorithms often work quite well, but can be computationally expensive.

Filter Feature Selection

Compute some simple score $S(i)$ to measure how informative each feature $x_i$ is. Then, simply pick the $k$ features with the largest scores $S(i)$.

It is common to choose mutual information as $S$
$$S(i)=MI(x_i,y)=\sum _{x_i}\sum _{y}p(x_i,y)\log \frac{p(x_i,y)}{p(x_i)p(y)}$$ Note that
$$MI(x_i,y)=KL(p(x_i,y)||p(x_i)p(y))$$

Online Learning

So far, we have been considering batch learning in which we are first given a training set to learn. Online learning, on the contrary, requires the algorithm to make predictions continuously even while it is learning.

In this setting, the algorithm is first given $x^{(i)}$ and is asked to predict $h(x^{(i)})$. Then the true $y^{(i)}$ is revealed for the model and this process repeats. Total online error is the total number of errors made by the algorithm during this process.

Assume that the class labels y ∈ { − 1 , 1 } y \in \{-1,1\} y∈{−1,1}, in perceptron algorithm with parameters $\theta \in {\mathbb{R}}^{n+1}$
$$h(x)=g({\theta }^{T}x)$$ where
$$g(z)=\{\begin{array}{ll}1& \text{if }z\ge 0\\ -1& \text{otherwise}\end{array}$$ Given a training example $(x,y)$, the parameter updates if $h(x)\ne y$
$$\theta :=\theta +yx$$ Suppose that
$$\begin{array}{rl}& \exists D,s.t.\forall i,||x^{(i)}||\le D\\ & \exists u(||u||=1),s.t.\forall i,y^{(i)}\cdot (u^T{x}^{(i)})\ge \gamma \end{array}$$ Block and Novikoff shows that the total number of mistakes that the perceptron algorithm makes is at most $(D/\gamma {)}^2$.