ML(1) Linear Regression

Introduction

Linear regression is perhaps the most fundamental algorithm in machine learning. In this setting, given a dataset D = { ( x i , y i ) ∣ x i ∈ R n , y i ∈ R } i = 1 m D=\{(x^i,y^i)|x^i\in \mathbb{R}^n, y^i\in\mathbb{R} \}_{i=1}^m D={ (xi,yi)∣xi∈Rn,yi∈R}i=1m​ (x is feature, y is label) we fit a model of the form $h_{\theta }(x)={\theta }^T\varphi (x)$, where $\theta$ is the parameter vector, $\varphi (x)$ is a transformed vector (for example, $\varphi (x)=[1,x_1,x_2,...,x_1{x}_2,...,x_n{x}_{n-1}]$). That is, the model is linear IN TERMS OF parameters instead of input vector $x$, as feature transformation is allowed.

Our goal is to fit the model $h_{\theta }(x)={\theta }^T\varphi (x)$ as good as possible. That is, after tuning our parameters, given an unseen $x^{\ast }$, we should be able to make $h_{\theta }(x^{\ast })\to y^{\ast }$. In a nutshell, find the BEST $\theta$.

Sometimes, our model might fit the training dataset well, yet failed to generalize to unseen data. This introduces the problem of OVERFITTING. To address this, we could use robust linear regression, ridge regression, lasso regression.

In what follows, I will derive the various linear regression (standard, robust, ridge, lasso) from 2 perspectives (deterministic and probabilistic). Also, generalized linear regression will be discussed.

Deterministic perspective

Intuitively, we could let our cost function to be $J(\theta )=\frac{1}{2}{\sum }_i^m(h_{\theta }(x^i)-y^i{)}^2$, another name for it is residual sum of squares (RSS) or sum of squared errors (SSE). Clearly, J is a convex function.

Then, the (standard) linear regression is formulated as ${\theta }^{\ast }:=\arg {\min }_{\theta }J(\theta )$ [How to solve it? 1. gradient descent algorithm; 2. Analytically set $\partial J/\partial \theta =0$. We have a particular nice solution if $\bar{x}=[1,x],$