模式识别 | PRML Chapter 3 Linear Models for Regression

PRML Chapter 3 Linear Models for Regression

3.1 Linear Basis Function Models

The simplest linear model for regression is the form:

$$y(x,w)=w_0+\sum _{j=1}^{M-1}{w}_j{\varphi }_j(x)$$

and when we define ${\varphi }_0(x)=1$, we will have:

$$y(x,w)=\sum _{j=1}^{M-1}{w}_j{\varphi }_j(x)=w^T\varphi (x)$$

There are many possible choices for the basis functions:

• powers of $x$: ${\varphi }_j(x)=x^j$
• Gaussian: ${\varphi }_j(x)=exp\left\{-\frac{(x-{\mu }_j{)}^2}{2s^2}\right\}$
• sigmoid: ${\varphi }_j(x)=\sigma \left(\frac{x-{\mu }_j}{s}\right)$
• logistic sigmoid: ${\sigma }_a=\frac{1}{1+exp(-a)}$
• tanh: $tanh(a)=2\sigma (2a)-1$

3.1.1 Maximum likelihood and least squares

We assume that the target variable $t$ is given by a deterministic function $y(x,w)$ with additive Gaussian noise:

$$t=y(x,w)+ϵ$$

So, we have

$$p(t|x,w,\beta )=N(t|y(x,w),{\beta }^{-1})$$

In the case of this Gaussian conditional distribution, the conditional mean will be

$$E[t|x]=\int tp(t|x)dt=y(x,w)$$

We assume that the data points are independent, and we can get the likelihood function and log likelihood function.

$$p(t|X,w,\beta )=\prod _{n=1}^{N}N(t_n|w^T\varphi (x_n),{\beta }^{-1})$$

$$lnp(t|w,\beta )=\sum _{n=1}^{N}lnN(t_n|w^T\varphi (x_n),{\beta }^{-1})$$

where the sum-of-squares error is E D ( w ) = 1 2 ∑ n = 1 N { t n − w T ϕ ( x n ) } 2 E_{D}(w) = \frac{1}{2}\sum_{n=1}^{N}\{ t_{n} - w^{T}\phi(x_{n}) \}^{2} ED​(w)=21​∑n=1N​{tn​−wTϕ(xn​)}2.

The MLE solution is:

$$w_{ML}=({\Phi }^T\Phi {)}^{-1}{\Phi }^{T}t$$

this is called normal equations for the least squares problem. $\Phi$ is called the design matrix:

$$\Phi =\left(\begin{array}{rlrlrlr}{\varphi }_0(x_1)& & {\varphi }_1(x_1)& & \dots & & {\varphi }_{M-1}(x_1)\\ {\varphi }_0(x_2)& & {\varphi }_1(x_2)& & \dots & & {\varphi }_{M-1}(x_2)\\ ⋮& & ⋮& & \ddots & & ⋮\\ {\varphi }_0(x_N)& & {\varphi }_1(x_N)& & \dots & & {\varphi }_{M-1}(x_N)\end{array}\right)$$

and the ${\Phi }^{†}\equiv ({\Phi }^T\Phi {)}^{-1}{\Phi }^T$ is known as the Moore-Penrose pseudo-inverse of the matrix $\Phi$.

If we make the bias parameter explicit, then the error function above becomes:

E D ( w ) = 1 2 ∑ n = 1 N { t n − w 0 − ∑ j = 1 M − 1 w j ϕ j ( x n ) } 2 E_{D}(w) = \frac{1}{2}\sum_{n=1}^{N}\{ t_{n} - w_{0} - \sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n}) \}^{2} ED​(w)=21​n=1∑N​{tn​−w0​−j=1∑M−1​wj​ϕj​(xn​)}2

Setting the derivative with respect to $w_0$ equal to zero, we obtain:

$$w_0=\bar{t}-\sum _{j=1}^{M-1}{w}_j{\bar{\varphi }}_j$$

where $\bar{t}=\frac{1}{N}{\sum }_{n=1}^N{t}_n,{\bar{\varphi }}_j=\frac{1}{N}{\sum }_{n=1}^N{\varphi }_j(x_n)$, thus the bias $w_0$ compensates for the difference between the averages of the target values and the weighted sum of the averages of the basis function values.

Also we can get the MLE solution of precision parameter.

1 β M L = 1 N ∑ n = 1 N { t n − w M L T ϕ ( x n ) } 2 \frac{1}{\beta_{ML}} = \frac{1}{N}\sum_{n=1}^{N}\{ t_{n} - w_{ML}^{T}\phi(x_{n}) \}^{2} βML​1​=N1​n=1∑N​{tn​−wMLT​ϕ(xn​)}2

3.1.2 Geometry of least squares

3.1.3 Sequential learning

Using gradient descent algorithm updates the parameter vector:

$$w^{(\tau +1)}=w^{(\tau )}-\eta \nabla E_n$$

For the case of the sum-of-squares error function:

$$w^{(\tau +1)}=w^{(\tau )}+\eta (t_n-w^{(\tau )T}{\varphi }_n){\varphi }_n$$

3.1.4 Regularized least squares

The total error function to be minimized takes the form:

$$E_D(w)+\lambda E_W(w)$$

Solving for $w$ as before, we obtain: $w=(\lambda I+{\Phi }^T\Phi {)}^{-1}{\Phi }^{T}t$
Also we can consider the general regularized error:

1 2 ∑ n = 1 N { t n − w T Φ ( x n ) } 2 + λ 2 ∑ j = 1 M ∣ w j ∣ q \frac{1}{2}\sum_{n=1}^{N}\{ t_{n} - w^{T}\Phi(x_{n}) \}^{2} + \frac{\lambda}{2}\sum_{j=1}^{M}|w_{j}|^{q} 21​n=1∑N​{tn​−wTΦ(xn​)}2+2λ​j=1∑M​∣wj​∣q

3.1.5 Multiple outputs

For multiple target variables, use the same set of basis functions to model all of the components of the target vector:

$$y(x,w)=W^T\varphi (x)$$

Suppose the conditional distribution of the target vector is isotropic Gaussian: $p(t|x,W,\beta )=N(t|W^T\varphi (x),{\beta }^{-1}I)$, we can get the MLE solution:

$$W_{ML}=({\Phi }^T\Phi {)}^{-1}{\Phi }^{T}T$$

3.2 The Bias-Variance Decomposition

The conditional expectation can be written as:

$$h(x)=E[t|x]=\int tp(t|x)dt$$

The expected squared loss in 1.5.5 is

E [ L ] = ∫ { y ( x ) − h ( x ) } 2 p ( x ) d x + ∫ { h ( x ) − t } 2 p ( x , t ) d x d t E[L] = \int \{y(x) - h(x)\}^{2}p(x)dx + \int \{h(x)-t\}^2p(x,t)dxdt E[L]=∫{y(x)−h(x)}2p(x)dx+∫{h(x)−t}2p(x,t)dxdt

The expected squared difference between $y(x,D)$ and the regression function $h(x)$ can be expressed as the sum of two terms:

E D [ { y ( x ; D ) − h ( x ) } 2 ] = { E D [ y ( x ; D ) ] − h ( x ) } 2 + E D [ { y ( x ; D ) − E D [ y ( x ; D ) ] } 2 ] E_{D}[\{ y(x;D) - h(x) \}^{2} ] = \{ E_{D}[y(x;D)] - h(x) \}^{2} + E_{D}[\{y(x;D) - E_{D}[y(x;D)] \}^{2} ] ED​[{y(x;D)−h(x)}2]={ED​[y(x;D)]−h(x)}2+ED​[{y(x;D)−ED​[y(x;D)]}2]

Generalize from the case of a single data point to the entire data set, we have:

$$expectedloss=(bias{)}^2+variance+noise$$

where:

b i a s 2 = ∫ { E D [ y ( x ; D ) ] − h ( x ) } 2 p ( x ) d x bias^{2} = \int \{E_{D}[y(x;D)] - h(x)\}^{2}p(x) dx bias2=∫{ED​[y(x;D)]−h(x)}2p(x)dx

v a r i a n c e = ∫ E D [ { y ( x ; D ) − E D [ y ( x ; D ) ] } 2 ] p ( x ) d x variance = \int E_{D}[\{y(x;D) - E_{D}[y(x;D)]\}^{2}]p(x) dx variance=∫ED​[{y(x;D)−ED​[y(x;D)]}2]p(x)dx

n o i s e = ∫ ∫ { h ( x ) − t } 2 p ( x , t ) d x d t noise = \int\int\{ h(x) - t\}^{2}p(x,t) dxdt noise=∫∫{h(x)−t}2p(x,t)dxdt

3.3 Bayesian Linear Regression

3.3.1 Parameter distribution

First we introduce the conjugate prior:

$$p(w)=N(w|m_0,S_0)$$

It is easy to calculate the posterior distribution:

$$p(w|t)=N(w|m_N,S_N)$$

where $m_N=S_N(S_0^{-1}{m}_0+\beta {\Phi }^{T}t)$ and
$S_N^{-1}=S_0^{-1}+\beta {\Phi }^T\Phi$

For the remainder of this chapter, we consider a zero-mean isotropic Gaussian prior in order to simplify the treatment.

$$p(w|\alpha )=N(w|0,{\alpha }^{-1}I)$$

and the parameters in the posterior will be $m_N=\beta S_N{\Phi }^{T}t$ and $S_N^{-1}=\alpha I+\beta {\Phi }^T\Phi$.

3.3.2 Predictive distribution

The predictive distribution is defined by

$$p(t|\mathbf{t},\alpha ,\beta )=\int p(t|w,\beta )p(w|\mathbf{t},\alpha ,\beta )dw$$

It is the convolution of two Gaussian distributions, and the predictive distributions takes the form:

$$p(t|\mathbf{x},\mathbf{t},\alpha ,\beta )=N(t|m_N^T\varphi (x),{\sigma }_N^2(x))$$

where

$${\sigma }_N^2(x)=\frac{1}{\beta }+\varphi (x{)}^T{S}_N\varphi (x)$$

3.3.3 Equivalent kernel

We can transform the predicted mean:

$$y(x,m_N)=m_N^T\varphi (x)=\beta \varphi (x{)}^T{S}_N{\Phi }^{T}t=\sum _{n=1}^N\beta \varphi (x{)}^T{S}_N\varphi (x_n)t_n$$

Thus the mean of the predictive distribution at a point $x$ is given by a linear combination of the training set target variables $t_n$:

$$y(x,m_N)=\sum _{n=1}^{N}k(x,x_n)t_n$$

where the function $k(x,x^{{}^{\prime }})=\beta \varphi (x{)}^T{S}_N\varphi (x^{{}^{\prime }})$ is known as the smoother matrix or the equivalent kernel.

The equivalent kernel shows the connection with the covariance:

$$cov[y(x),y^{\prime }(x)]=cov[\varphi (x{)}^{T}w,w^T\varphi (x^{\prime })]=\varphi (x{)}^T{S}_N\varphi (x^{\prime })={\beta }^{-1}k(x,x^{\prime })$$

Also, for all values of $x$, it satisfies:

$$\sum _{n=1}^{N}k(x,x_n)=1$$

What’s more, it can be expressed in inner product form:

$$k(x,z)=\psi (x{)}^T\psi (z)$$

where $\psi (x)={\beta }^{1/2}{S}_N^{1/2}\varphi (x)$.

3.4 Bayesian Model Comparison

Suppose we wish to compare a set of L models { M i } \{M_i\} {Mi​}, here a model refers to a probability distribution over the observed data $D$. Given a training set $D$, we then wish to evaluate the posterior distribution:

$$p(M_i|D)\propto p(M_i)p(D|M_i)$$

$p(M_i)p(D\Vert M_i)$ means model evidence and the ratio of model evidences $\frac{p(D\Vert M_i)}{p(D\Vert M_j)}$ for two models is known as a Bayes factor.

The model evidence can be calculated through

$$p(D|M_i)=\int p(D|w,M_i)p(w|M_i)dw$$

Assume that $p(w)=\frac{1}{\Delta w_{prior}}$, we will have:

$$p(D)=\int p(D|w)p(w)dw\simeq p(D|w_{MAP})\frac{\Delta w_{posterior}}{\Delta w_{prior}}$$

take the logarithm:

$$\ln p(D)\simeq \ln p(D|w_{MAP})+\ln (\frac{\Delta w_{posterior}}{\Delta w_{prior}})$$

For model with M parameters, and their $\frac{\Delta w_{posterior}}{\Delta w_{prior}}$ are the same, we will have:

$$\ln p(D)\simeq p(D|w_{MAP})+M\ln (\frac{\Delta w_{posterior}}{\Delta w_{prior}})$$

3.5 The Evidence Approximation

Introduce hyperpriors over $\alpha$ and $\beta$, the predictive distribution is obtained by marginalizing over $w,\alpha ,\beta$.

$$p(t|\mathbf{t})=\int \int \int p(t|w,\beta )p(w|\mathbf{t},\alpha ,\beta )p(\alpha ,\beta |\mathbf{t})dwd\alpha d\beta$$

If the posterior distribution $p(\alpha ,\beta \Vert \text{t})$ is sharply peaked around $\hat{\alpha }$ and $\hat{\beta }$, then the predictive distribution is obtained simply by marginalizing over $w$:

$$p(t|\mathbf{t})\simeq p(t|\mathbf{t},\hat{\alpha },\hat{\beta })=\int p(t|w,\hat{\beta })p(w|\mathbf{t},\hat{\alpha },\hat{\beta }))dw$$

From Bayes’ theorem, the posterior distribution for $\alpha$ and $\beta$ is given by:

$$p(\alpha ,\beta |\mathbf{t})\propto p(\mathbf{t}|\alpha ,\beta )p(\alpha ,\beta )$$

3.5.1 Evaluation of the evidence function

The marginal likelihood function $p(\mathbf{t}\Vert \alpha ,\beta )$ is obtained by integrating over the weight parameters $w$, so that

$$p(\mathbf{t}|\alpha ,\beta )=\int p(\mathbf{t}|w,\beta )p(w|\alpha )dw$$

By completing the square in the exponent and making use of the standard form for the normalization coefficient of a Gaussian, we can get the log of the marginal likelihood in the form:

$$\ln p(\mathbf{t}|\alpha ,\beta )=\frac{M}{2}\ln \alpha +\frac{N}{2}\ln \beta -E(m_N)-\frac{1}{2}\ln |A|-\frac{N}{2}\ln (2\pi )$$

3.5.2 Maximizing the evidence function

Defining the eigenvector equation

$$(\beta {\Phi }^T\Phi )u_i={\lambda }_i{u}_i$$

It then follows that $A$ has eigenvalues $\alpha +{\lambda }_i$. Now consider the derivative of the term involving $ln\Vert A\Vert$ with respect to $\alpha$.

$$\frac{d}{d\alpha }\ln |A|=\frac{d}{d\alpha }\ln \prod _i({\lambda }_i+\alpha )=\frac{d}{d\alpha }\sum _i\ln ({\lambda }_i+\alpha )=\sum _i\frac{1}{{\lambda }_i+\alpha }$$

Let the derivative be zero, we will have:
$$\alpha m_N^T{m}_N=M-\alpha \sum _i\frac{1}{{\lambda }_i+\alpha }=\gamma$$

So it is easy to get that

$$\gamma =\sum _i\frac{{\lambda }_i}{\alpha +{\lambda }_i}and\alpha =\frac{\gamma }{m_N^T{m}_N}$$

As for $\beta$, we can follow the same idea to get

1 β = 1 N − γ ∑ n = 1 N { t n − m N P T ϕ ( x n ) } \frac{1}{\beta} = \frac{1}{N-\gamma}\sum_{n=1}^{N}\{ t_{n}-m_{N}^PT \phi(x_{n})\} β1​=N−γ1​n=1∑N​{tn​−mNP​Tϕ(xn​)}

3.5.3 Effective number of parameters

When the number of data points is far more larger than parameters, it will be easy to compute that:

$$\alpha =\frac{M}{2E_W(m_N)}$$

$$\beta =\frac{N}{2E_D(m_N)}$$

3.6 Limitations of Fixed Basis Functions