l2约束的最小二乘学习法

Least Squares

Least squares regression is a traditional ML algorithm that minimizes the total square error between samples and learning output, i.e. minimize

$$J_{LS}(\theta)=\frac{1}{2}\sum_{i=1}^{n}\left( f_{\theta}(x_{i})-y_{i}\right)^{2}$$
We’d like to get the estimated $\theta$ where $J_{LS}$ is the smallest.
$$\hat{\theta}_{LS}=\arg\min_{\theta}J_{LS}(\theta)$$
Take the linear model as an illustration.
$$f_{\theta}(x)=\sum_{j=1}^{b}\theta_{j}\phi_{j}(x)=\theta^{T}\phi(x)$$
Then
$$J_{LS}(\theta)=\frac{1}{2}\| \Phi\theta-y\|^{2}$$
where

Φ=⎛⎝⎜⎜ϕ1(x1)⋮ϕ1(xn)⋯⋱⋯ϕb(x1)⋮ϕb(xn