ML Note 1.1 - Regression

连续性随机变量的学习问题称为 regression. 回归问题中最常见的是 linear regression
$$y|x;\theta =h(x)+ϵ$$

其中 $ϵ\sim N(0,{\sigma }^2)$ 称为 error term。应用 GLM 可以得出正态分布对应的¹
$$h(x)={\theta }^{T}x$$

再应用 MLE 可以得到 log likelihood
$$\begin{array}{rcl}l(\theta )& =& \sum _{i=1}^m\log \frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(y^{(i)}-h(x^{(i)}){)}^2}{2{\sigma }^2})\\ & =& -\frac{1}{2{\sigma }^2}\sum _{i=1}^m(y^{(i)}-h(x^{(i)}){)}^2+C\\ & =& -\frac{1}{{\sigma }^2}\cdot \frac{1}{2}\sum _{i=1}^m{ϵ}_i^2+C\end{array}$$

定义 cost function
$$J(\theta )=\frac{1}{2}\sum _{i=1}^m{ϵ}_i^2$$

则最小化代价函数和原问题是等价的。Normal equation² 给出了 closed-form 解
$$\theta =(X^{T}X{)}^{-1}{X}^T\vec{y}$$

如果 $X^{T}X$ 不可逆，可能的原因有

• 训练集中有重复变量
• 样本数量小于参数个数

无论哪种原因，删除一部分变量总是有效的。如果可以确定所有参数都是不相关的，那么可以选择增加样本数量或者使用正则化后的正规方程
$$\theta ={\left(X^{T}X+\lambda {\left[\begin{array}{ccccc}0& 0& 0& \cdots & 0\\ 0& 1& 0& \cdots & 0\\ 0& 0& 1& \cdots & 0\\ & ⋮& & \ddots & ⋮\\ 0& 0& 0& \cdots & 1\end{array}\right]}_{(n+1)\times (n+1)}\right)}^{-1}{X}^T\vec{y}$$

模型检验

为了评价模型的拟合优度，定义
$$\begin{array}{rlcl}\text{SSE}& \text{(the sum of squares due to error)}& =& 2J(\theta )\\ \text{SSR}& \text{(sum of squares of the regression)}& =& (m-1)s_h^2\\ \text{SST}& \text{(total sum of squares)}& =& (m-1)s_y^2\\ r^2& \text{(coefficient of determination)}& =& 1-\text{SSE}/\text{SST}\end{array}$$

可以证明平方和分解公式
$$\text{SST}=\text{SSE}+\text{SSR}$$

通过观察发现 SSE 受到 ${\theta }_0,{\theta }_2,\dots ,{\theta }_n$ 共 $n+1$ 个条件制约³，因此其自由度
$$\begin{array}{rlcl}\text{DFE}& \text{(degrees-of-freedom in the error)}& =& m-n-1\end{array}$$

而 SSR 和 SST 的自由度分别为 $n$ 和 $m-1$⁴。由此定义
$$\begin{array}{rlcl}\text{MSE}& \text{(mean squared error)}& =& \text{SSE}/\text{DFE}\\ \text{RMSE}& \text{(root mean squared error)}& =& \sqrt{\text{MSE}}\\ \text{adj-}{r}^2& \text{(df adjusted }{r}^2)& =& 1-\text{MSE}/s_y^2\end{array}$$

可以检验 $H_0:\vec{\theta }=0$
$$F=\frac{\text{SSR}/n}{\text{MSE}}\sim F(n,\text{DFE})$$

如果检验显著，则应继续对每个参数进行 t 检验 $H_0:{\theta }_i=0$. 因为 MSE 为 ${\sigma }^2$ 的无偏估计，因此记
$${\hat{\sigma }}^2=\text{MSE}$$

由公式
$$Cov(\theta )={\hat{\sigma }}^2{\left(X^{T}X\right)}^{-1}$$

可以得到 ${\sigma }_j^2=Var({\theta }_i)$，易知
$${\hat{\theta }}_j\sim N({\theta }_j,{\sigma }_j^2)$$

构造枢轴量
$$\frac{{\hat{\theta }}_j-{\theta }_j}{{\sigma }_j}\sim t(\text{DFE})$$

可以得到单个自变量对目标变量的拟合程度。关于变量预测的置信区间可以查看多元线性回归的预测

一元线性回归

对于 $X=\left[\begin{array}{cc}x& 1\end{array}\right]$ 可以解得
$$\begin{array}{rcl}\theta & =& {\left[\begin{array}{cc}\sum _{i=1}^m{x}_i^2& \sum _{i=1}^m{x}_i\\ \sum _{i=1}^m{x}_i& m\end{array}\right]}^{-1}\left[\begin{array}{c}{x}^T\\ 1\end{array}\right]y\\ & =& \frac{1}{m\sum _{i=1}^m{x}_i^2-{\left(\sum _{i=1}^m{x}_i\right)}^2}\left[\begin{array}{cc}m& -\sum _{i=1}^m{x}_i\\ -\sum _{i=1}^m{x}_i& \sum _{i=1}^m{x}_i^2\end{array}\right]\left[\begin{array}{c}\sum _{i=1}^m{x}_i{y}_i\\ \sum _{i=1}^m{y}_i\end{array}\right]\end{array}$$

引入记号
$$\begin{array}{rcl}{l}_x& =& (n-1)s_x^2\\ l_{xy}& =& (n-1)s_{xy}\end{array}$$

将等式
$$\begin{array}{rcl}\sum _{i=1}^m{x}_i& =& m\bar{x}\\ \sum _{i=1}^m{y}_i& =& m\bar{y}\\ \sum _{i=1}^m{x}_i^2& =& l_x+m{\bar{x}}^2\\ \sum _{i=1}^m{x}_i{y}_i& =& l_{xy}+m\bar{x}\bar{y}\end{array}$$

带入 $\theta$ 表达式
$$\begin{array}{rcl}\theta & =& \frac{1}{m{l}_x}\left[\begin{array}{cc}m& -m\bar{x}\\ -m\bar{x}& l_x+m{\bar{x}}^2\end{array}\right]\left[\begin{array}{c}{l}_{xy}+m\bar{x}\bar{y}\\ m\bar{y}\end{array}\right]\\ & =& \left[\begin{array}{c}{l}_{xy}/l_x\\ \bar{y}-\bar{x}{\theta }_1\end{array}\right]\end{array}$$

可以证明
$$\begin{array}{rcl}\text{SSR}& =& {\theta }_1^2{l}_x\\ r^2& =& {\rho }_{xy}^2\end{array}$$

LWR

Locally weighted regression 是一种非参数估计方法。与线性回归不同，LWR 使用的代价函数引入了 weight $w$. 一般取
$$w^{(i)}=\exp (-\frac{1}{2}(x^{(i)}-x{)}^T{\Sigma }^{-1}(x^{(i)}-x))$$

其中 $\Sigma$ 称为 bandwidth. 修改后的代价函数为
$$J(\theta )=\sum _{i=1}^m{w}^{(i)}(y^{(i)}-h(x^{(i)}){)}^2$$

相应的，模型检验中定义的各个统计量也需要考虑 $w$ 的影响。

参考

多元线性回归中的 T 检验怎样理解？其 p 值为什么划定在 0.05？

---

1. 指数分布族(Exponential Family)相关公式推导及在变分推断中的应用 ↩︎

2. Define the design matrix
   $$X_{m\times (n+1)}=\left(\begin{array}{c}(x^{(1)}{)}^T\\ (x^{(2)}{)}^T\\ \cdots \\ (x^{(m)}{)}^T\end{array}\right)$$ Let
   $$\vec{y}=\left(\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ \cdots \\ y^{(m)}\end{array}\right)$$ Thus
   $$\begin{array}{rcl}J(\theta )& =& \frac{1}{2}(X\theta -\vec{y}{)}^T(X\theta -\vec{y})\\ & & \\ {\nabla }_{\theta }J& =& \frac{1}{2}{\nabla }_{\theta }({\theta }^T{X}^{T}X\theta -{\theta }^T{X}^T\vec{y}-{\vec{y}}^{T}X\theta +{\vec{y}}^T\vec{y})\end{array}$$ since every term is simply a real number
   $$\begin{array}{rcl}{\nabla }_{\theta }J& =& \frac{1}{2}{\nabla }_{\theta }tr({\theta }^T{X}^{T}X\theta -{\theta }^T{X}^T\vec{y}-{\vec{y}}^{T}X\theta )\\ & & \\ & =& \frac{1}{2}({\nabla }_{\theta }tr({\theta }^T{X}^{T}X\theta )-2{\nabla }_{\theta }tr({\vec{y}}^{T}X\theta ))\end{array}$$ By the properties of matrix derivative
   $$\begin{array}{rcl}{\nabla }_{\theta }tr({\vec{y}}^{T}X\theta )& =& {\nabla }_{({\theta }^T{)}^T}tr({\theta }^T{X}^T\vec{y})\\ & & \\ & =& ({\nabla }_{{\theta }^T}tr({\theta }^T{X}^T\vec{y}){)}^T\\ & & \\ & =& X^T\vec{y}\\ & & \\ {\nabla }_{\theta }tr({\theta }^T{X}^{T}X\theta )& =& {\nabla }_{\theta }tr(\theta {\theta }^T{X}^{T}X)\\ & & \\ & =& {\nabla }_{\theta }tr(\theta I{\theta }^T{X}^{T}X)\\ & & \\ & =& 2X^{T}X\theta \end{array}$$ Therefore
   $$\begin{array}{rcl}{\nabla }_{\theta }J& =& X^{T}X\theta -X^T\vec{y}\end{array}$$ Set ${\nabla }_{\theta }J\equiv 0$
   $$X^{T}X\theta =X^T\vec{y}$$ over. ↩︎

3. 统计| 自由度（degree of freedom） ↩︎

4. 详解方差分析表(ANOVA)(二) —— SST、SSE、SSR和它们的自由度 ↩︎