Big Data Analytics 笔记 2

1 Linear Model

$$y_i={\theta }_1{x}_{i1}+{\theta }_2{x}_{i2}+...+{\theta }_{n_p}{x}_{i{n}_p}+{\epsilon }_i,i=1,2,...,n_d.$$
or,
$$y=X\theta +\epsilon$$
where $\epsilon$ is noise, $\theta$ is the vector of unkown parameters.

• The linear model is parametric with $n_p$ parameters
• If adding an interception ${\theta }_0$, then the intercept is a column of 1 in the design matrix $X$
• For dimension of design matrix $X:n_d\times n_p$ or $X:n_d\times (n_p+1)$ for adding ${\theta }_0$, $n_d>n_p$
• The traditional assumption of distribution of noise (iid noise, means noise with distribution) $\epsilon$: ${\epsilon }_i\sim \mathcal{N}(0,{\sigma }^2),i=1,2,...,n_d$

1.1 parameter estimation

For parameter estimation of Linear Model: via the cost function or ** maximum likelihood principle**, these two approaches coincide to the same optimal solution of the parameter $\theta$, the difference is: cost function reduces the discrepancy between predictions and data, while MLE maximizes the likelihood of observing data given parameters.

1.2 Cost Function

The goal: reduces the discrepancy between predictions and data
$⟹$ reduce the MSE of predictions

$$\begin{gathered} MSE(\hat{y})=\frac{1}{n_d}||y-\hat{y}|{|}^2=\frac{1}{n_d}\sum _{i=1}^{n_d}(y_i-{\hat{y}}_i{)}^2 \\ =\frac{1}{n_d}||y-X\hat{\theta }|{|}^2=\frac{1}{n_d}\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\hat{\theta }{)}^2 \end{gathered}$$
Let $J(\theta )$ be cost function, find $\hat{\theta }$ of $\theta$ to minimize $J(\theta )$, where
$$J(\theta )=||y-X\theta |{|}^2=\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2$$
so $$\hat{\theta }=\arg \underset{\theta }{\min }J(\theta )$$, thia is the same as ordinary least squares (OLS) estimator, because OLS for $\theta =||\epsilon |{|}^2={\sum }_{i=1}^{n_d}{\epsilon }^2={\sum }_{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2$, so OLS estimator for $\hat{\theta }$:
$$\hat{\theta }=\arg \underset{\theta }{\min }||\epsilon (\theta )|{|}^2=\arg \underset{\theta }{\min }(y_i-x_i^{\mathsf{T}}\theta {)}^2$$

The ordinary least squares (OLS) $\hat{\theta }$:

$$\hat{\theta }=\arg \underset{\theta }{\min }||\epsilon (\theta )|{|}^2=\arg \underset{\theta }{\min }(y_i-x_i^{\mathsf{T}}\theta {)}^2$$

Cost Function:

$$J(\theta )=\sum _{i=1}^{n_d}(y_i-h_{\theta }(x_i){)}^2$$, where $h_{\theta }(x)$ n is called the hypothesis and $h_{\theta }(x_i)=x_i^{\mathsf{T}}\theta$ for linear models. So, the estimator obtained by minimizing the cost function for $h_{\theta }(x_i)=x_i^{\mathsf{T}}\theta$ and the OLS estimator obtained by minimizing the squared noise norm $||\epsilon (\theta )|{|}^2$ coincide

Get $\hat{\theta }$:

$$\{\begin{array}{cc}\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{d}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s}\hat{\theta },J(\theta )\mathrm{i}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{x}& \\ \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{o}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{m}\mathrm{e}\mathrm{s}\hat{\theta },J(\theta )\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{x}& \end{array}$$

1.3 Normal equation to get $\hat{\theta }$

For invertible $(X^{\mathsf{T}}X{)}^{-1}$,
$$\hat{\theta }=(X^{\mathsf{T}}X{)}^{-1}{X}^{\mathsf{T}}y$$
Gauss-Markov assumptions

• $E({\epsilon }_i)=0$
• $Var({\epsilon }_i)={\sigma }^2$
• $Cov({\epsilon }_i,{\epsilon }_j)=0$

Under this assumption:

• $X$ should be non-singular to allow its columns to be linearly independent.
• $E(\hat{\theta })=\theta$
• $Var(\hat{\theta })={\sigma }^2(X^{\mathsf{T}}X{)}^{-1}$
• $\hat{\theta }\sim \mathcal{N}(\theta ,{\sigma }^2(X^{\mathsf{T}}X{)}^{-1}),i=1,2,...,n_d$
  ( under the assumption of ${\epsilon }_i\sim \mathcal{N}(0,{\sigma }^2),i=1,2,...,n_d$)
• The maximum likelihood is
  $$\mathcal{L}(y,X|\theta ,{\sigma }^2)=\prod _{i=1}^{n_d}(2\pi {\sigma }^2{)}^{-n_d/2}exp\left(-\frac{1}{2{\sigma }^2}\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2\right)$$
• The associated MLE:
  $$\hat{\theta }=(X^{\mathsf{T}}X{)}^{-1}{X}^{\mathsf{T}}y$$, coincide with OLS for normal noise, equal to $\underset{\theta }{\mathrm{argmin}}(y_i-x_i^{\mathsf{T}}\theta {)}^2$

Limitations of normal equation with large $n_p$

• computational cost of $(X^{\mathsf{T}}X{)}^{-1}$
• singularity, with large $n_p$, its columns might turn to be highly correlated

1.4 Gradient Descent Process to get $\hat{\theta }$

The Logic: start with $\theta =({\theta }_0,{\theta }_1)$ , then keep to change $({\theta }_0,{\theta }_1)$ to reduce $J({\theta }_0,{\theta }_1)$ until reach $\underset{{\theta }_0,{\theta }_1}{\min }J({\theta }_0,{\theta }_1)$

1.4.1 Algorithm

1. Initialize $({\theta }_1,{\theta }_2,...,{\theta }_n)$
2. $({\tilde{\theta }}_1,{\tilde{\theta }}_2,...,{\tilde{\theta }}_n)=({\theta }_1,{\theta }_2,...,{\theta }_n)$
3. Update ${\theta }_i$ with ${\theta }_i={\tilde{\theta }}_i-a\cdot \frac{\partial }{\partial {\theta }_i}J({\tilde{\theta }}_1,{\tilde{\theta }}_2,...,{\tilde{\theta }}_n)$, where $a$ is the learning rate
4. Repeat 2, 3 until ${\theta }_i$ converges

Gradient descent algorithm for linear regression

cost function for multiple linear regression model:
$$J(\theta )=\frac{1}{2n_d}\sum _{i=1}^{n_d}(x_i^{\mathsf{T}}\theta -y_i{)}^2$$
The updating step of gradient descent for linear regression with the given cost function:
$${\theta }_j^{(k+1)}={\theta }_j^{(k)}-a\frac{1}{n_d}\sum _{i=1}^{n_d}(x_i^{\mathsf{T}}{\theta }^{(k)}-y_i)x_{ij}$$

1.4.2 Learning Rate

The learning rate $a$ used in the algorithm should be > 0, constant.

• Learning rate can affect convergence and speed of convergence:
  Too small learning rate might lead to slow convergence, while too large learning rate might lead to non-convergence or divergence.
• Can be selected using a validation set or cross-validation.

1.4.3 Stoping Convergence

Check the error tolerance close to 0:

• Absolute error tolerance:
  $${\epsilon }_{abs}=|J({\theta }^{(k+1)})-J({\theta }^k)|$$
• Relative error tolerance:
  $${\epsilon }_{rel}=\left|\frac{J({\theta }^{(k+1)})-J({\theta }^k)}{J({\theta }^{(k+1)})}\right|$$

2 Logistic Regression

Output: { 0 , 1 } \{0,1\} {0,1}
Decision boundary: $x_i^{\mathsf{T}}\theta$
Hypothesis: $h_{\theta }(x_i)=g(x_i^{\mathsf{T}}\theta )=\frac{1}{1+exp(-x_i^{\mathsf{T}}\theta )}$

Cost function:
$$J({\theta }_0,{\theta }_1)=\{\begin{array}{cc}-log(h_{({\theta }_0,{\theta }_1)}(x)),ify=1& \\ -log(1-h_{({\theta }_0,{\theta }_1)}(x)),ify=0& \end{array}$$

General form for n samples cost function:
$$J(\theta )=-\frac{1}{n}\sum _{i=1}^n(y_{i}log(h_{\theta }(x_i))+(1-y_i)log(1-h_{\theta }(x_i)))$$

Classification:

• $x_i^{\mathsf{T}}\theta \ge 0⟹h_{\theta }(x_i)\ge 0.5⟹\hat{y}=1$
  $y=1$, then $J(\theta )\to 0$
  $y=0,J(\theta )\to \infty$
• $x_i^{\mathsf{T}}\theta <0⟹h_{\theta }(x_i)<0.5⟹\hat{y}=0$
  $y=1$, then $J(\theta )\to \infty$
  $y=0,J(\theta )\to 0$

3 Avoid Overfitting

higher overfitting $⟹$ higher variance of the predictions
higher underfitting $⟹$ higher bias of the predictions

3.1 Penalizing parameters

$$\sum _{i=1}^{n_d}(y_i-h_{\theta }(x_i){)}^2+\lambda r(\theta )$$, where $\lambda r(\theta )$ is the Penalty, $r(\theta )$ is the parameter penalty, $\lambda$ is the regularization parameter.l

regularization parameter $\lambda$

• $\lambda \to 0$ regularized regression tend to linear regression estimates
• $\lambda \to \infty$ parameters get penalized more, the less the over-fitting to the data
• select $\lambda$ via cross-validation

parameter penalty functions $r(\theta )$
Some typical parameter penalty functions (参考distance)

• $d-variation$ function: $r(\theta )={\left({\sum }_{i=1}^{n_p}|{\theta }_i{|}^d\right)}^{1/d}$
• $L_{1}norm:r(\theta )={\sum }_{i=1}^{n_p}|{\theta }_i|$
• $squared{L}_{2}normr(\theta )={\sum }_{i=1}^{n_p}|{\theta }_i{|}^2$

3.2 Regression model with penalty function

3.2.1 Lasso regression

Lasso regression uses $L_{1}norm$ as penalty function. Lasso regression “zeroes out” coefficients, so it performs variable selection and to some extent parameter shrinkage.

• Cost function
  $$J_L(\theta )=||y-X\theta |{|}^2+\lambda ||\theta |{|}_1=\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2+\lambda \sum _{j=1}^{n_d}|{\theta }_j|$$

• Gradient of Cost function
  $$\frac{\partial J_L(\theta )}{\partial {\theta }_1},...\frac{\partial J_L(\theta )}{\partial {\theta }_{nd}}$$

note:
$$\frac{\partial {\sum }_{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2}{\partial {\theta }_1}=2x_{1i}\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta )$$

• Lasso estimate
  $${\hat{\theta }}_L=\underset{\theta }{\mathrm{argmin}}\left(\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2+\lambda \sum _{j=1}^{n_d}|{\theta }_j|\right)$$
  Can’t be solved, but $J_L(\theta )$ is convex, using the least angle regression algorithm to solve.

3.2.2 Ridge regression

Ridge regression uses $L_{2}norm$ as penalty function. So it does not “zero out” coefficients, i.e. it can not perform variable selection, but rather shrinks parameter values.

• Cost function
  $$J_R(\theta )=||y-X\theta |{|}^2+\lambda ||\theta |{|}_2^2=\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2+\lambda \sum _{j=1}^{n_d}({\theta }_j{)}^2$$

• ridge estimate
  $${\hat{\theta }}_R=(X^{\mathsf{T}}X+\lambda I{)}^{-1}{X}^{\mathsf{T}}y$$
  This is the closed form solution.

The ridge estimate is biased because:

$$\begin{array}{rl}E({\hat{\theta }}_R)& =E\left((X^{\mathsf{T}}X+\lambda I{)}^{-1}{X}^{\mathsf{T}}y\right)\\ & =(X^{\mathsf{T}}X+\lambda I{)}^{-1}{X}^{\mathsf{T}}E(y)\\ & =(X^{\mathsf{T}}X+\lambda I{)}^{-1}{X}^{\mathsf{T}}X\theta \\ & =(X^{\mathsf{T}}X+\lambda I{)}^{-1}((X^{\mathsf{T}}X{)}^{-1}{)}^{-1}\theta \\ & ={\left[(X^{\mathsf{T}}X{)}^{-1}(X^{\mathsf{T}}X+\lambda I)\right]}^{-1}\theta \\ & ={\left[I+\lambda (X^{\mathsf{T}}X{)}^{-1}\right]}^{-1}\theta \end{array}$$
$X^{\mathsf{T}}X$ is positive definite, $\lambda >0$ by definition, so $E({\hat{\theta }}_R)\ne \theta$

Tips:
Although ridge regression does not perform variable selection, it performs grouped selection. So if one variable amongst a group of correlated ones is selected, ridge regression automatically includes the whole group. Ridge regression can resolve near multicollinearity.

3.2.3 Elastic net

The elastic net penalty is a compromise between Lasso and ridge.

• Cost function
  $$J_E(\theta )=||y-X\theta |{|}^2+{\lambda }_1||\theta |{|}_1+{\lambda }_2||\theta |{|}_2^2$$
  if X is an $n_d\times n_p$ design matrix
  $$J_E(\theta )=\sum _{i=1}^{n_d}(y_i-x_i^{\mathsf{T}}\theta {)}^2+{\lambda }_1\sum _{j=1}^{n_p}|{\theta }_j|+{\lambda }_2\sum _{j=1}^{n_p}{\theta }_j^2$$
• elastic net estimate
  $${\hat{\theta }}_E=\underset{\theta }{\mathrm{argmin}}\left(||y-X\theta |{|}^2+{\lambda }_1||\theta |{|}_1+{\lambda }_2||\theta |{|}_2^2\right)$$

no closed form solution for ${\theta }_E$, but $J_E(\theta )$ is convex to have a unique minimum

4 Learning curve

Increase prediction accuracy:

• variable selection: increase or reduce number of covariates
• add polynomial features: { x 2 , x 1 x 2 } \{x^2, x_1x_2\} {x2,x1​x2​}
• regularized regression (Lasso, ridge regression and elastic nets):
• Collect more data

Learning curve:

• A learning curve is a metric of prediction accuracy, like cost function $J(\theta )$
• Or error metric (function of a parameter that affects the metric)