Least Squares
Least squares regression is a traditional ML algorithm that minimizes the total square error between samples and learning output, i.e. minimize
JLS(θ)=12∑i=1n(fθ(xi)−yi)2 J L S ( θ ) = 1 2 ∑ i = 1 n ( f θ ( x i ) − y i ) 2
We’d like to get the estimated θ θ where JLS J L S is the smallest.
θ^LS=argminθJLS(θ) θ ^ L S = arg min θ J L S ( θ )
Take the linear model as an illustration.
fθ(x)=∑j=1bθjϕj(x)=θTϕ(x) f θ ( x ) = ∑ j = 1 b θ j ϕ j ( x ) = θ T ϕ ( x )
Then
JLS(θ)=12∥Φθ−y∥2 J L S ( θ ) = 1 2 ‖ Φ θ − y ‖ 2
where
Φ=⎛⎝⎜⎜ϕ1(x1)⋮ϕ1(xn)⋯⋱⋯ϕb(x1)⋮ϕb(xn)