# 最小二乘法(least squares)

S(a,b)(y1(ax1+b))2+(y2(ax2+b))2+(y3(ax3+b))2$S(a, b) ＝ (y1 - (ax1+b)) ^{2} + (y2 - (ax2+b)) ^{2} + (y3 - (ax3+b)) ^{2}$

f(x)=dj=1xjwj+ϵ=xw+ϵ$f(\mathbf x) = \sum_{j=1}^{d} x_j w_j + \epsilon = \mathbf x \mathbf w^\intercal + \epsilon$

argmaxwL(w)=lni=1n1σ2πexp(12(yixiwσ)2)=12σ2i=1n(yixiw)2nlnσ2π

argminwf(w)=ni=1(yixiw)2=yXw22$\text{argmin}_{\mathbf w} f(\mathbf w) = \sum_{i=1}^n(\mathbf y_i - \mathbf x_i \mathbf w^\intercal)^2= {\left\lVert{\mathbf y - \mathbf X \mathbf w^\intercal}\right\rVert}_2^2$

nj=1Xijβj=yi, (i=1,2,,m)$\sum_{j=1}^{n} X_{ij}\beta_j = y_i,\ (i=1, 2, \dots, m)$

Xβ=y$X\beta = y$

S(β)=yXβ2$S(\boldsymbol{\beta}) = \bigl\|\mathbf y - \mathbf X \boldsymbol \beta \bigr\|^2$

β̂ =argminβS(β),$\hat{\boldsymbol{\beta}} = \underset{\boldsymbol{\beta}}{\operatorname{arg\,min}}\,S(\boldsymbol{\beta}),$
β$\beta$求微分得：
β̂ =(XTX)1XTy.$\hat{\boldsymbol{\beta}}= (\mathbf X^{\rm T} \mathbf X )^{-1} \mathbf X^{\rm T} \mathbf y.$

python代码；

import numpy as np
import matplotlib.pyplot as plt
input = np.array([
[1, 6],
[2, 5],
[3, 7],
[4, 10]
])
m = np.shape(input)[0]
X = np.matrix([np.ones(m), input[:,0]]).T
y = np.matrix(input[:,1]).T
#直接代入公式
betaHat = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
print(betaHat)
plt.figure(1)
xx = np.linspace(0, 5, 2)
yy = np.array(betaHat[0] + betaHat[1] * xx)
plt.plot(xx, yy.T, color='b')
plt.scatter(input[:,0], input[:,1], color='r')
plt.show()