多元线性回归的参数估计——最小二乘

模型

$y_i={\beta }_0+{\beta }_1{x}_{1i}+\dots +{\beta }_k{x}_{ki}+{ϵ}_i$

从样本的角度推导

假设样本数据集为 { y 1 , … , y n } \{y_1,\ldots,y_n\} {y1​,…,yn​}，令$\beta =({\beta }_0,{\beta }_1,\dots ,{\beta }_k{)}^{\prime }$，$y=(y_1,\dots ,y_n{)}^{\prime }$，$x_i=(1,x_{1i},\dots ,x_{ki}{)}^{\prime }$，$x=(x_1,\dots ,x_n)$，那么：
$$\begin{array}{rl}\hat{\beta }& =\underset{\beta }{\mathrm{argmin}}\sum _{i=1}^n\frac{1}{2}(y_i-x_i^{\prime }\beta {)}^2\\ & =\underset{\beta }{\mathrm{argmin}}\frac{1}{2}||y-x^{\prime }\beta |{|}^2\\ & =\underset{\beta }{\mathrm{argmin}}\frac{1}{2}(y-x^{\prime }\beta {)}^{\prime }(y-x^{\prime }\beta )\\ & =\underset{\beta }{\mathrm{argmin}}\frac{1}{2}(y^{\prime }y-y^{\prime }{x}^{\prime }\beta -{\beta }^{\prime }xy+{\beta }^{\prime }x{x}^{\prime }\beta )\end{array}$$

对$\frac{1}{2}(y^{\prime }y-y^{\prime }{x}^{\prime }\beta -{\beta }^{\prime }xy+{\beta }^{\prime }x{x}^{\prime }\beta )$关于$\beta$求偏导，有：
$$\begin{array}{rl}\frac{\partial }{\partial \beta }\frac{1}{2}(y^{\prime }y-y^{\prime }{x}^{\prime }\beta -{\beta }^{\prime }xy+{\beta }^{\prime }x{x}^{\prime }\beta )& =\frac{1}{2}(-xy-xy+x{x}^{\prime }\beta +x{x}^{\prime }\beta )\\ & =x{x}^{\prime }\beta -xy=0\end{array}$$

故可得$\hat{\beta }=(x{x}^{\prime }{)}^{-1}xy$

从理论的角度推导

$$\begin{array}{rl}\hat{\beta }& =\underset{\beta }{\mathrm{argmin}}MSE\\ & =\underset{\beta }{\mathrm{argmin}}E(y_i-x_i^{\prime }\beta {)}^2\end{array}$$

对$E(y_i-x_i^{\prime }\beta {)}^2$关于$\beta$求偏导，则有：
$$\begin{array}{rl}\frac{\partial }{\partial \beta }E(y_i-x_i^{\prime }\beta {)}^2& =2E(y_i-x_i^{\prime }\beta )(-x_i)\\ & =2E{x}_i{x}_i^{\prime }\beta -2E{x}_i{y}_i=0\end{array}$$

故有：$\hat{\beta }=(E{x}_i{x}_i^{\prime })E{x}_i{y}_i$