多元线性回归推导 — Multi-Variable Linear Regression Derivation

多元线性回归通式定义如下：
$$\{\begin{array}{l}{\hat{y}}_i=f(x)={\theta }_0+\sum _{j=1}^n{\theta }_j{x}_i^j\\ y_i={\hat{y}}_i+{ϵ}_i\end{array}$$
其中${\theta }_0$为偏置项，$x_i^j$为第$i$条数据的第$j$项特征值，${\theta }_j$为第$j$项特征的权重，${\hat{y}}_i$为第$i$条数据$x_i$的预测值，$y_i$为第$i$条数据$x_i$对应的实际值，${ϵ}_i$为预测值${\hat{y}}_i$与实际值$y_i$之间的残差。

令 $x_i^0=1$，${\hat{y}}_i={\theta }_0+\sum _{j=1}^n{\theta }_j{x}_i^j$可被改写成${\hat{y}}_i={\theta }^T{x}_i$。

假设每条数据$x_i$相互独立且同分布，残差项${ϵ}_i$符合标准正态分布。残差项的概率密度函数如下：
$$p({ϵ}_i)=\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{{ϵ}_i^2}{2{\sigma }^2})=\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(y_i-{\theta }^T{x}_i{)}^2}{2{\sigma }^2})$$

我们的目的是最小化$|{ϵ}_i|$，而$p({ϵ}_i)$在${ϵ}_i$值趋近$\mu =0$时取得最大值，如上图。因此，已知$x_i$和$y_i$，我们可以使用最大似然估计来得到$\theta$：
$$L(\theta )=\prod _{i=1}^{m}p(y_i|x_i;\theta )=\prod _{i=1}^m\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(y_i-{\theta }^T{x}_i{)}^2}{2{\sigma }^2})$$

将$L(\theta )$取对数，得到$\log L(\theta )$，连乘变成了累加，消掉了指数幂，减少了函数的复杂度。并且，$L(\theta )$与$\log L(\theta )$具有相同的变化趋势。因此，求$L(\theta )$取得最大值时$\theta$的取值等同于求$\log L(\theta )$取得最大值时$\theta$的取值。
$$\begin{array}{rl}& \log L(\theta )=\sum _{i=1}^m\log (\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(y_i-{\theta }^T{x}_i{)}^2}{2{\sigma }^2}))\\ & =\sum _{i=1}^m(\log \frac{1}{\sqrt{2\pi }\sigma }-\frac{(y_i-{\theta }^T{x}_i{)}^2}{2{\sigma }^2})\\ & =\sum _{i=1}^m\log \frac{1}{\sqrt{2\pi }\sigma }-\sum _{i=1}^m\frac{(y_i-{\theta }^T{x}_i{)}^2}{2{\sigma }^2}\end{array}$$

由于$\sum _{i=1}^m\log \frac{1}{\sqrt{2\pi }\sigma }$和$2{\sigma }^2$是常量, $\sum _{i=1}^m(y_i-{\theta }^T{x}_i{)}^2$取得最小值时，$\log L(\theta )$取得最大值。

于是，我们得到了最小二乘法函数：
$$J(\theta )=\sum _{i=1}^m(y_i-{\theta }^T{x}_i{)}^2$$

$J(\theta )$在其偏微分取值为零时取得最小值。因此，我们对$J(\theta )$偏微分，求其取值为零时$\theta$的值。
$$J_{\theta }=\frac{\partial J}{\partial \theta }={\left[\begin{array}{cccc}\frac{\partial J}{\partial {\theta }_0}& \frac{\partial J}{\partial {\theta }_1}& \dots & \frac{\partial J}{\partial {\theta }_n}\end{array}\right]}^T=0$$

我们以$\frac{\partial J}{\partial {\theta }_0}$为例：
$$\begin{array}{rl}& \frac{\partial J}{\partial {\theta }_0}=\frac{\partial }{\partial {\theta }_0}\sum _{i=1}^m(y_i-{\theta }^T{x}_i{)}^2\\ & =\frac{\partial }{\partial {\theta }_0}[(y_1-{\theta }^T{x}_1{)}^2+(y_2-{\theta }^T{x}_2{)}^2+\cdots +(y_m-{\theta }^T{x}_m{)}^2]\\ & =\frac{\partial }{\partial {\theta }_0}[(y_1-\sum _{j=1}^n{\theta }_j{x}_1^j{)}^2+(y_2-\sum _{j=1}^n{\theta }_j{x}_2^j{)}^2+\cdots +(y_m-\sum _{j=1}^n{\theta }_j{x}_m^j{)}^2]\\ & =2(y_1-\sum _{j=1}^n{\theta }_j{x}_1^j)(-x_1^0)+2(y_2-\sum _{j=1}^n{\theta }_j{x}_2^j)(-x_2^0)+\cdots +2(y_m-\sum _{j=1}^n{\theta }_j{x}_m^j)(-x_m^0)\\ & =-2\left[\begin{array}{cccc}{y}_1-\sum _{j=1}^n{\theta }_j{x}_1^j& y_2-\sum _{j=1}^n{\theta }_j{x}_2^j& \dots & y_m-\sum _{j=1}^n{\theta }_j{x}_m^j\end{array}\right]{\left[\begin{array}{cccc}{x}_1^0& x_2^0& \dots & x_m^0\end{array}\right]}^T\\ & =-2\left[\begin{array}{cccc}{y}_1-{\theta }^T{x}_1& y_2-{\theta }^T{x}_2& \dots & y_m-{\theta }^T{x}_m\end{array}\right]{\left[\begin{array}{cccc}{x}_1^0& x_2^0& \dots & x_m^0\end{array}\right]}^T\end{array}$$

同理，我们可以求得$\frac{\partial J}{\partial {\theta }_1}$至$\frac{\partial J}{\partial {\theta }_n}$的偏微分：
$$\begin{gathered} \frac{\partial J}{\partial {\theta }_0}=-2\left[\begin{array}{cccc}{y}_1-{\theta }^T{x}_1& y_2-{\theta }^T{x}_2& \dots & y_m-{\theta }^T{x}_m\end{array}\right]{\left[\begin{array}{cccc}{x}_1^0& x_2^0& \dots & x_m^0\end{array}\right]}^T \\ \frac{\partial J}{\partial {\theta }_1}=-2\left[\begin{array}{cccc}{y}_1-{\theta }^T{x}_1& y_2-{\theta }^T{x}_2& \dots & y_m-{\theta }^T{x}_m\end{array}\right]{\left[\begin{array}{cccc}{x}_1^1& x_2^1& \dots & x_m^1\end{array}\right]}^T \\ ⋮ \\ \frac{\partial J}{\partial {\theta }_n}=-2\left[\begin{array}{cccc}{y}_1-{\theta }^T{x}_1& y_2-{\theta }^T{x}_2& \dots & y_m-{\theta }^T{x}_m\end{array}\right]{\left[\begin{array}{cccc}{x}_1^n& x_2^n& \dots & x_m^n\end{array}\right]}^T \end{gathered}$$

将其改写成矩阵的形式：
$$\begin{array}{rl}& J_{\theta }=\frac{\partial J}{\partial \theta }=\left[\begin{array}{c}\frac{\partial J}{\partial {\theta }_0}\\ \frac{\partial J}{\partial {\theta }_1}\\ ⋮\\ \frac{\partial J}{\partial {\theta }_n}\end{array}\right]=-2\left[\begin{array}{cccc}{y}_i-{\theta }^T{x}_1& y_2-{\theta }^T{x}_2& \dots & y_m-{\theta }^T{x}_m\end{array}\right]\left[\begin{array}{cccc}{x}_1^0& x_1^1& \dots & x_1^n\\ x_2^0& x_2^1& \dots & x_2^n\\ ⋮& & & \\ x_m^0& x_m^1& \dots & x_m^n\end{array}\right]\\ & =-2(\left[\begin{array}{cccc}{y}_1& y_2& \dots & y_m\end{array}\right]-\left[\begin{array}{cccc}{\theta }^T{x}_1& {\theta }^T{x}_2& \dots & {\theta }^T{x}_m\end{array}\right])\mathbf{X}\\ & =-2({\mathbf{Y}}^T-{\theta }^T\left[\begin{array}{cccc}{x}_1& x_2& \dots & x_m\end{array}\right])\mathbf{X}\\ & =-2({\mathbf{Y}}^T-{\theta }^T{\mathbf{X}}^T)\mathbf{X}\end{array}$$

令$J_{\theta }=-2({\mathbf{Y}}^T-{\theta }^T{\mathbf{X}}^T)\mathbf{X}=0$，得到：
$$\begin{gathered} {\mathbf{Y}}^T\mathbf{X}={\theta }^T{\mathbf{X}}^T\mathbf{X} \\ {\theta }^T={\mathbf{Y}}^T\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}={\mathbf{Y}}^T\mathbf{X}{\mathbf{X}}^{-1}({\mathbf{X}}^T{)}^{-1}={\mathbf{Y}}^T({\mathbf{X}}^T{)}^{-1}={\mathbf{Y}}^T({\mathbf{X}}^{-1}{)}^T \\ \theta =({\theta }^T{)}^T={\mathbf{X}}^{-1}\mathbf{Y} \end{gathered}$$

因此，当$\theta ={\mathbf{X}}^{-1}\mathbf{Y}$时，${\hat{y}}_i={\theta }^T{x}_i$能最好地描述$\mathbf{X}$与$\mathbf{Y}$之间的线形相关性。

然而，现实中的数据并非都能通过求偏导来求极值。例如：$z=y^2-x^2$。如下图，当$\frac{\partial z}{\partial y}=0;\frac{\partial z}{\partial x}=0$时，$y=0;x=0$（鞍点）却非极值点。因此，实际中多采用梯度下降法或牛顿法来求得目标函数极值的近似解。通过线性回归得到回归参数后，可以通过计算判定系数$R^2$来评估回归函数的拟合优度。