Machine Learning Notes I: The Standard Linear Model

we derive the standard linear model here from Bayesian prospective of view (MAP), with training set $\mathcal{D} = (\bf{X}, \bf{y}) = \{(\bf{x}_i, y_i) | i = 1,..., n\}$, $\bf{x}$ denotes input vector of dimension $D$ and $y$ denotes a scalar output. $\bf{w}$ the weights vector and $\xi$ Gaussian noise

$$f(\bf{x}) = \bf{x}^T \bf{w} \quad \quad y = f(\bf{x})+\xi \quad \quad \xi \sim \mathcal{N}(0, \sigma_n^2)$$

The likelihood function

$$p(\bf{y} | \bf{X}, \bf{w}) = \prod_{i=1}^n p(y_i | \bf{x}_i, \bf{w}) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi}\sigma_n}\exp(-\frac{y_i - \bf{x}_i^T\bf{w}}{2\sigma_n^2}) \\ \quad \quad \quad \quad \quad = \frac{1}{(2\pi\sigma_n^2)^{n/2}}\exp(-\frac{1}{2\sigma_n^2}|\bf{y}-\bf{X}^T\bf{w}|^2) = \mathcal{N}(\bf{X}^T\bf{w}, \sigma_n^2\bf{I})$$

further specify the prior over the parameter $\bf{w}$, set as zero mean Gaussian with covariance matrix $\Sigma_p$, i.e., $\bf{w} \sim \mathcal{N}(0, \Sigma_p)$, the Bayesian linear model (MAP) is

$$p(\bf{w} | \bf{y}, \bf{X}) = \frac{p(\bf{y} | \bf{X}, \bf{w}) p(\bf{w})}{p(\bf{y}|\bf{X})}\propto p(\bf{y} | \bf{X}, \bf{w}) p(\bf{w}) \\ \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \propto \exp(-\frac{1}{2\sigma_n^2}(\bf{y}-\bf{X}\bf{w})^T(\bf{y}-\bf{X}\bf{w}))\exp(-\frac{1}{2}\bf{w}^T\Sigma_p^{-1}\bf{w})\\ \quad \quad \quad \quad \quad \quad \quad \quad \propto \exp(-\frac{1}{2}(\bf{w}-\bar{\bf{w}})^T(\frac{1}{\sigma_n^2}\bf{X}\bf{X}^T+\Sigma_p^{-1})(\bf{w}-\bar{\bf{w}}))$$
with $\bar{\bf{w}} = \sigma_n^{-2}(\sigma_n^{-2}\bf{X}\bf{X}^T+\Sigma_p^{-1})^{-1}\bf{X}\bf{y}$, then we have,
$$p(\bf{w} | \bf{y}, \bf{X}) \sim \mathcal{N}(\frac{1}{\sigma_n^2}(\sigma_n^{-2}\bf{X}\bf{X}^T+\Sigma_p^{-1})^{-1}\bf{X}\bf{y}, \quad (\sigma_n^{-2}\bf{X}\bf{X}^T+\Sigma_p^{-1})^{-1} )$$

make predictions for $\bf{x} = \bf{x}_*$ as

$$E(f_* | \bf{x}_*, \bf{X}, \bf{y}) = \int f_*(\bf{x}_*,\bf{w}) p(\bf{w}|\bf{X}, \bf{y}) d\bf{w} = \int \bf{x}_*^T\bf{w} p(\bf{w}|\bf{X}, \bf{y}) d\bf{w} = \frac{1}{\sigma_n^2}\bf{x}_*^T(\sigma_n^{-2}\bf{X}\bf{X}^T+\Sigma_p^{-1})^{-1}\bf{X}\bf{y}$$

in fact, $p(f_* | \bf{x}_*, \bf{X}, \bf{y}) \sim \mathcal{N}(\frac{1}{\sigma_n^2}\bf{x}_*^T(\sigma_n^{-2}\bf{X}\bf{X}^T+\Sigma_p^{-1})^{-1}\bf{X}\bf{y}, \quad \bf{x}_*^T(\sigma_n^{-2}\bf{X}\bf{X}^T+\Sigma_p^{-1})^{-1}\bf{x}_* )$

In a non-Bayesian setting, the prior is usually thought as a penalty term (Ridge Regression, L2 regularization) $\frac{1}{2}\bf{w}^T\Sigma_p^{-1}\bf{w}$