Machine Learning-Chapter 2

Chapter 2 : Linear Regression

1 Statistical Learning Theory

1.1 Supervised Learning

In supervised learning, the goal is to learn the mapping (the rules) between a set of inputs and outputs.

1.2 Problem Definition

Given a set of $n$ examples (data) { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x n , y n ) } \left\{\left(\mathbf{x}_{1}, y_{1}\right),\left(\mathbf{x}_{2}, y_{2}\right), \ldots,\left(\mathbf{x}_{n}, y_{n}\right)\right\} {(x1​,y1​),(x2​,y2​),…,(xn​,yn​)}

Question: find function $f$ such that : $f(\mathbf{x})=\hat{y}$

is a good predictor of $y$ for a future input $\mathbf{x}$ (fitting the data is not enough!)

1.3 Statistical Learning Definition

There is an unknown probability distribution on the product space $Z=X\times Y$, written $\mu (x,y)$

We assume that $X$ is a compact domain in Euclidean space and $Y$ a bounded subset of $\mathbb{R}$. The training set ：

S = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x n , y n ) } S=\left\{\left(\mathbf{x}_{1}, y_{1}\right),\left(\mathbf{x}_{2}, y_{2}\right), \ldots,\left(\mathbf{x}_{n}, y_{n}\right)\right\} S={(x1​,y1​),(x2​,y2​),…,(xn​,yn​)} consists of $n$ samples drawn i.i.d. (Independent and identically distributed) from $\mu$ .

$\mathcal{H}$ is the hypothesis space, a space of functions $f:X\to Y$

A learning algorithm is a map $L:Z^n\to \mathcal{H}$ that looks at $S$ and selects from $\mathcal{H}$ a function $f_S:X\to Y$ such that :

$f_S(\mathbf{x})\approx y$ in a predictive way

Given a function $f,$ a loss function $\ell :Y\times Y\to \mathbb{R},$ we define the expected or true error of $f$ is :

$\mathcal{L}(f)={\mathbb{E}}_{X,Y}[\ell (y,f(x))]={\int }_{X\times Y}\ell (y,f(x))d\mu (x,y)$

which is the expected loss on a new example drawn at random from $\mu$

The empirical error of $f_S$ is:

${\mathcal{L}}_S\left(f_S\right)=\frac{1}{n}{\sum }_{i=1}^n\ell \left(y_i,f_S\left({\mathbf{x}}_i\right)\right)$

A very natural requirement for $f_S$ is distribution independent generalization :

$\forall \mu ,\lim \left|{\mathcal{L}}_S\left(f_S\right)-\mathcal{L}\left(f_S\right)\right|=0$

in probability. In other words, the training error for the solution must converge to the expected error and thus be a proxy for it.

2 Linear Regression

2.1 Introduction

y=ax+b

2.2 Problem Setting

2.2.1 Elements

A set of training data { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x n , y n ) } \left\{\left(\mathbf{x}_{1}, y_{1}\right),\left(\mathbf{x}_{2}, y_{2}\right), \ldots,\left(\mathbf{x}_{n}, y_{n}\right)\right\} {(x1​,y1​),(x2​,y2​),…,(xn​,yn​)}

2.2.2 Assumptions

Function $f$ has linear structure between input $X=\left[\begin{array}{c}{X}_0\\ X_1\\ ⋮\\ X_p\end{array}\right]$ , a random vector $\in {\mathbb{R}}^{p+1}$ in which $X_0=1$ and output $Y$, a random variable $\in \mathbb{R}$. It has the form :

$Y=f(X)={\beta }_0+{\sum }_{j=1}^p{X}_j{\beta }_j=X_0{\beta }_0+{\sum }_{j=1}^p{X}_j{\beta }_j=X^T\beta$

where the $\beta$, are unknown parameters or coefficients, $\beta =\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ ⋮\\ {\beta }_p\end{array}\right]$

The loss function $\ell :Y\times Y\to \mathbb{R}$ has the following form:

$\ell (y,\hat{y})=(y-\hat{y}{)}^2$

The empirical error of $f$ is:

${\mathcal{L}}_S(f)=\frac{1}{n}{\sum }_{i=1}^n\ell \left(y_i,f\left({\mathbf{x}}_i\right)\right)$

$=\frac{1}{n}{\sum }_{i=1}^n{\left(y_i-{\mathbf{x}}_i^T\beta \right)}^2$

2.2.3 Matrix Form

A set of training data { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x n , y n ) } \left\{\left(\mathbf{x}_{1}, y_{1}\right),\left(\mathbf{x}_{2}, y_{2}\right), \ldots,\left(\mathbf{x}_{n}, y_{n}\right)\right\} {(x1​,y1​),(x2​,y2​),…,(xn​,yn​)} can be written as matrix and vector form, the matrix form for input is :

$\mathbf{X}=\left[\begin{array}{ccc}-& {\mathbf{x}}_1^T& -\\ -& {\mathbf{x}}_2^T& -\\ & ⋮& \\ -& {\mathbf{x}}_n^T& -\end{array}\right]$

and the vector form for output is

$\mathbf{y}=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right]$

The empirical error of $f$ can be written as matrix form:

${\mathcal{L}}_S(f)=\frac{1}{n}{\sum }_{i=1}^n\ell \left(y_i,f\left({\mathbf{x}}_i\right)\right)$

$=\frac{1}{n}{\sum }_{i=1}^n{\left(y_i-{\mathbf{x}}_i^T\beta \right)}^2$

$=\frac{1}{n}(\mathbf{y}-\mathbf{X}\beta {)}^T(\mathbf{y}-\mathbf{X}\beta )$

2.2.4 Conclusion

Assuming that $\mathbf{X}$ has full column rank, minimization of the empirical error leads to the estimator of the $f$:

$\hat{y}=\mathbf{X}\hat{\beta }$

where :

$\hat{\beta }={\left({\mathbf{X}}^T\mathbf{X}\right)}^{-1}{\mathbf{X}}^T\mathbf{y}$

2.2.4.1 Proof:

Theorem: ${\alpha }_s(t)=(\vec{y}-\mathbb{X}\vec{\beta }{)}^{\prime }(\vec{y}-\mathbb{X}\vec{p})\cdot \left(\frac{1}{n}\text{ ignore }\right)$

Question : $\mathrm{argmin}\alpha (f)$ .$f\in H$ , $H$: Linear function

Conclusion : $\hat{f}=\mathbb{X}\hat{\vec{\beta }}$ , $\hat{\beta }={\left({\mathbf{X}}^T\mathbf{X}\right)}^{-1}{\mathbf{X}}^T\mathbf{y}$

Pf:

$J_{\vec{\beta }}=\frac{d{\alpha }_s}{d\vec{\beta }}$ , $H_{\vec{\beta }}=\frac{d{J}_B}{d\vec{\beta }}=\frac{d^2{\alpha }_s}{d{\vec{\beta }}^2}$

Let $\vec{a}=\vec{y}-\vec{x}\vec{\beta },\vec{b}=\vec{y}-x\vec{\beta }$

$\frac{d{\alpha }_s}{d\vec{\beta }}=\frac{d{\alpha }_s}{\partial \vec{a}}\frac{\partial \vec{a}}{\partial \vec{\beta }}+\frac{d{\alpha }_c}{\partial \vec{b}}\frac{\partial \vec{b}}{\partial \vec{\beta }}$ = ${\vec{b}}^{\top }(-x)+{\vec{a}}^{\prime }(-x)=-(\vec{y}-x\vec{p}{)}^{\top }x-(\vec{y}-x\vec{p}{)}^{\top }x$ = $-2(\vec{y}-x\vec{p}{)}^{\top }x$

$H_p=\frac{d{J}_{\vec{\beta }}}{\partial vec\beta }=\frac{d\left(2(x\vec{\beta }{)}^{\prime }x\right)}{\partial \vec{\beta }}=\frac{d\left(2{\vec{\beta }}^{\prime }{x}^{\prime }x\right)}{\partial \vec{\beta }}=2{\left(x^{\top }x\right)}^{\top }=2x^{\top }x$

${\alpha }_s(f)$ is a convex function ,and any local minimum of a convex function is also a global minimum :

$-2(\vec{y}-x\vec{\beta }{)}^{\prime }x=0\Rightarrow x^{\top }(\vec{y}-x\vec{\beta })=0\Rightarrow x^{\top }x\vec{\beta }=x^{\top }\vec{y}$

If X is full column rank ,$X^{\top }X$ is positive definite and then is invertible:

$\hat{\beta }={\left({\mathbf{X}}^T\mathbf{X}\right)}^{-1}{\mathbf{X}}^T\mathbf{y}$

2.2.4.2 Extra notes

$n>>p$ ,which means $Sample>>Feature$

f $n<<p$

• $\mathrm{rank}(x)⩽\min (n,p+1)=n$

• $\mathrm{rank}\left(x^{\top }x\right)\le \min \left(\mathrm{rank}\left(x^{\top }\right),rank(x)\right)\le n$

• If $X^{\top }X$ is invertible, $\mathrm{rank}\left(x^{\top }x\right)=p+1$

  Then $p+1 \leq n $ , which means $X^{\top }X$ is in vertible.