Linear Regression

This article records my note about Linear regression.

The Linear Regression is a single layer neural network with a output.Trough this model,we see theory of gradient descent with intuition.

general model

$$f(x)=w^{T}x+b=w_1{x}_1+w_2{x}_2+...+w_n{x}_n+b$$

Numerical solution

loss function

The $\frac{1}{2}$ is for simplicity. The $m$ is number of samples in a batch.
$$\begin{gathered} l=\frac{1}{m}\sum _i^m\frac{1}{2}(f(x^i)-y^i{)}^2 \\ =\frac{1}{2m}\sum _i^m(w^T{x}^i+b-y^i{)}^2 \end{gathered}$$
When batch is given,$l$ is quadratic function about $w$ and $b$.When we fix $b$,we can get gradient of $w$.You can imagine that a quadratic function may jitter in lowest point with a big learning rate.Thus,we need a good lr by testing.

SGD

Firstly,computing derivation.
$$\begin{gathered} \frac{\partial l}{\partial w}=\frac{1}{2m}\sum _i^m(2(w^T{x}^i+b-y^i)x^i), \\ wherewandxbotharevectors. \\ for{w}_j \\ \frac{\partial l}{\partial w_j}=\frac{1}{2m}\sum _i^m(2(w^T{x}^i+b-y^i)x_j^i) \\ =\frac{1}{m}\sum _i^m(w^T{x}^i+b)x_j^i-\frac{1}{m}\sum _i^m(x_j^i{y}^i) \end{gathered}$$
When batch is given,we use constant $p$ and $q$, $z$ to replace $\frac{1}{m}\sum x_j^i{x}^i$ and $\frac{1}{m}\sum (x^i{y}^i)$,$\frac{1}{m}\sum x_j^i$ respectively:
$$\begin{gathered} \frac{\partial l}{\partial w_i}=w^{T}p+zb-q \\ wherezb-qisconstant \end{gathered}$$
Optimizing parameters:
$$w_i-=\eta \ast \frac{\partial l}{\partial w_i}$$

Analytical solution

We can directly use the least square method to solve this question.
Refer to least square method.
Specific derivation process can refer to Zhou ZH Watermelon Book.
(PS: for n*d matrix $w$ where d>n,r($w^{T}w$)<=r($w^T$)<=n).