模式识别 | PRML Chapter 1 Introduction

PRML Chapter 1 Introduction

1.1 Example：Polynomial Curve Fitting

For a simple regression problem, our goal is to use the training set to predict new value $\hat{t}$ for input variable $\hat{x}$. The simplest way is to fit the curve with polynomial.

$$y(x,w)=w_0+w_{1}x+w_2{x}^2+\cdots +w_M{x}^M=\sum _{j=0}^M{w}_j{x}^j$$

To determine the coefficients, we need to minimize the loss function. A general type of loss function could be written as:

$$E(w)=\frac{1}{2}\sum _{n=1}^N{y(x_n,w)-t_n}^2$$

To avoid the coefficients reaching a huge value, regularization is what we need. By adding the penalty term, we can avoid overfitting:

$$\hat{E}(w)=\frac{1}{2}\sum _{n=1}^N{y(x_n,w)-t_n}^2+\frac{\lambda }{2}||w|{|}^2$$

Where:
$$||w|{|}^2=w^{T}w=w_0^2+w_1^2+\cdots +w_M^2$$

1.2 Probability Theory

Two important rules in probability theory:

• sum rule: $p(X)={\sum }_{Y}p(X,Y)$
• product rule:$p(X,Y)=p(Y|X)p(X)$
  Base on the rules we could derive the Bayes’ theorem:

$$P(Y|X)=\frac{p(X|Y)p(Y)}{p(X)}=\frac{p(X|Y)p(Y)}{{\sum }_{Y}p(X|Y)p(Y)}$$

1.2.1 Probability densities

For continuous variables, we define the probability density over x as:

$$p(x\in (a,b))={\int }_a^{b}p(x)dx$$

Also we have transformative rules between variables:

$$p_y(y)=p_x(g(y))\Vert g^{\prime }(y)\Vert$$

1.2.2 Expectations and covariances

In this chapter we define the expectation, variance and covariance:

$$E[f]=\sum _{x}p(x)f(x)$$

$$var[f]=E[(f(x)-E[f(x){]}^2)]=E[f(x{)}^2]-E[f(x){]}^2$$

$$cov[x,y]=E_{x,y}[x-E[x]y-E[y]]=E_{x,y}xy-E[x]E[y]$$

1.2.3 Bayesian probabilities

Here we use bayesian perspective to give a description of uncertainty:

$$p(w|D)=\frac{p(D|w)p(w)}{p(D)}=\frac{p(D|w)p(w)}{\int p(D|w)p(w)dw}$$

which allow us to value the uncertainty of $w$ after we observe the dataset $D$.

1.2.4 The Gaussian distribution

The Gaussian distribution is defined as:

N ( x ∣ μ , σ 2 ) = 1 ( 2 π σ 2 ) 1 2 e x p { − 1 2 σ 2 ( x − μ ) 2 } N(x | \mu,\sigma^{2}) = \frac{1}{(2\pi\sigma^{2})^{\frac{1}{2}}}exp\{-\frac{1}{2\sigma^{2}}(x-\mu)^{2}\} N(x∣μ,σ2)=(2πσ2)21​1​exp{−2σ21​(x−μ)2}

For independent identically distributed dataset, the likelihood function could be written as:

$$p(x|\mu ,{\sigma }^2)=\prod _{n=1}^{N}N(x_n|\mu ,{\sigma }^2)$$

it can also be written in the log form:

$$lnp(x|\mu ,{\sigma }^2)=-\frac{1}{2{\sigma }^2}\sum _{n=1}^N(x_n-\mu {)}^2-\frac{N}{2}ln{\sigma }^2-\frac{N}{2}ln(2\pi )$$

Thus we can calculate the partial derivative to get the MLE solution:

$${\mu }_{ML}=\frac{1}{N}\sum _{n=1}^N{x}_n$$

$${\sigma }_{ML}^2=\frac{1}{N}\sum _{n=1}N(x_n-{\mu }_{ML}{)}^2$$

The expectation of the estimation is unbiased, but the variance is underestimated.

$$E[{\mu }_{ML}]=\mu$$

$$E[{\sigma }_{ML}^2]=\frac{N-1}{N}{\sigma }^2$$

1.2.5 Curve fitting re-visited

We assume that given the value of $x$, the corresponding value of $t$ has a Guassian distribution with a mean equal to the value $y(x,w)$. Thus we have

$$p(t|x,w,\beta )=N(t|y(x,w,{\beta }^{-1}))$$

The likelihood function and log likelihood function is:

$$p(t|x,w,\beta )=\prod _{n=1}^{N}N(t_n|y(x_n,w),{\beta }^{-1})$$

l n p ( t ∣ x , w , β ) = − β 2 ∑ n = 1 N { y ( x n , w ) − t n 2 } + N 2 l n β − N 2 l n ( 2 π ) ln p(t | x, w, \beta) = -\frac{\beta}{2}\sum_{n=1}^{N}\{y(x_{n},w) - t_{n}^{2}\} + \frac{N}{2}ln\beta - \frac{N}{2}ln(2\pi) lnp(t∣x,w,β)=−2β​n=1∑N​{y(xn​,w)−tn2​}+2N​lnβ−2N​ln(2π)

When we calculate $w_{MLE}$, we can see that maximize the likelihood function is equivalent to minimize the sum-of-square error function.

Similarily, for the precision parameter $\beta$, we have:

1 β M L = 1 N ∑ n = 1 N { y ( x n , w M L ) − t n } 2 \frac{1}{\beta_{ML}}=\frac{1}{N}\sum_{n=1}^N\{y(x_n,w_{ML})-t_n\}^2 βML​1​=N1​n=1∑N​{y(xn​,wML​)−tn​}2

To predict the distribution, we substitute the MLE solution into the function:

$$p(t|x,w_{ML},{\beta }_{ML})=N(t|y(x,w_{ML}),{\beta }_{ML}^{-1})$$

Introduce the prior distribution of $w$ (assume Guassian):

p ( w ∣ α ) = N ( w ∣ 0 , α − 1 I ) = ( α 2 π ) M + 1 2 e x p { − α 2 w T w } p(w | \alpha) = N(w | 0, \alpha^{-1}I) = (\frac{\alpha}{2\pi})^{\frac{M+1}{2}}exp\{-\frac{\alpha}{2}w^{T}w \} p(w∣α)=N(w∣0,α−1I)=(2πα​)2M+1​exp{−2α​wTw}

We can apply bayes’ theorem and maximize the posterior:(MAP techique)

$$p(w|x,t,\alpha ,\beta )\propto p(t|x,w,\beta )p(w|\alpha )$$

and it is equivalent to minimize regularized sum-of-square error function:

β 2 ∑ n = 1 N { y ( x n , w ) − t n } 2 + α 2 w T w \frac{\beta}{2}\sum_{n=1}^{N}\{y(x_{n},w) - t_{n} \}^{2} + \frac{\alpha}{2}w^{T}w 2β​n=1∑N​{y(xn​,w)−tn​}2+2α​wTw

1.2.6 Bayesian curve fitting

The pure Bayesian treatment simply corresponds to a consistent application of the sum and product rules.

$$p(t|x,X,t)=\int p(t|x,w)p(w|X,t)dw$$

1.3 Model Selection

When the training and testing data are limited, we can use cross validation to improve the performance. However, it may be time costly.

1.4 The Curse of Dimensionality

In a high dimension space the data will be really sparse. Take the example of sphere(see what happen when $ϵ\to 0$):

$$V_D(r)=K_D{r}^D$$

$$\frac{V_D(1)-V_D(1-ϵ)}{V_D(1)}=1-(1-ϵ{)}^D$$

1.5 Decision Theory

When we predict the category of a new data point $x$, we can use bayes’ theorem:

$$p(C_k|x)=\frac{p(x|C_k)p(C_k)}{p(x)}$$

Intuitively we would choose the class having the higher posterior probability.

1.5.1 Minimizing the misclassification rate

We should minimize the term:
$$p(mistake)=p(x\in R_1,C_2)+p(x\in R_2,C_1)$$

More generally, for k classes, we can maximize:

$$p(mistake)=\sum _{k=1}^{K}p(x\in R_k,C_k)=\sum _{k=1}^K{\int }_{R_k}p(x,C_k)dx$$

1.5.2 Minimizing the expected loss

The uncertainty of real category could be represented by joint distribution $p(x,C_k)$, so we need to minimize the average loss:

$$E[L]=\sum _k\sum _j{\int }_{R_j}{L}_{kj}p(x,C_k)dx$$

1.5.3 The reject option

Set the threshold value $\theta$ to reject the $x$ whose maximum posterior $p(C_k\Vert x)$ value lower then $\theta$.

1.5.4 Inference and decision

• generative models: model the distribution of inputs and outputs $p(x,C_k)$.
• discriminative models: model the posterior probabilities directly $p(C_k\Vert x)$.

1.5.5 Loss functions for regression

The expected loss is given by:

$$E[L]=\int \int L(t,y(x))p(x,t)dxdt$$

For example when we use the squared loss:

E [ L ] = ∫ ∫ { y ( x ) − t } 2 p ( x , t ) d x d t E[L] = \int\int \{y(x) - t\}^{2}p(x, t) dxdt E[L]=∫∫{y(x)−t}2p(x,t)dxdt

By making $\frac{\partial E[L]}{\partial y(x)}$equals to 0, we can get the regression function, the optimal solution is conditional average:

$$y(x)=\frac{\int tp(x,t)dt}{p(x)}=\int tp(t\Vert x)dt=E_t[t\Vert x]$$

From another point of view, the expected loss function can be written as (where the second part can be seen as noise):

E [ L ] = ∫ { y ( x ) − E [ t ∣ x ] } 2 p ( x ) d x + ∫ v a r [ t ∣ x ] p ( x ) d x E[L] = \int \{y(x) - E[t | x]\}^{2}p(x)dx + \int var[t | x]p(x)dx E[L]=∫{y(x)−E[t∣x]}2p(x)dx+∫var[t∣x]p(x)dx

1.6 Information Theory

The amount of information can be viewed as the ‘degree of surprise’ on learning the value of $x$. Also the amount should be non-negative:

$$h(x)=-lo{g}_{2}p(x)$$

And the entropy can be seen as the average amount of information:

$$H[x]=-\sum _{x}p(x)lo{g}_{2}p(x)$$

1.6.1 Relative entropy and mutual information.

Suppose that we are modeling an unknown distribution $p(x)$ using $q(x)$. Then the average additional amount of information required to specify $x$ as a result of using $q(x)$ instead of true $p(x)$ is given by:

K L ( p ∣ ∣ q ) = − ∫ p ( x ) l n q ( x ) d x − ( − ∫ p ( x ) l n p ( x ) d x ) = − ∫ p ( x ) l n { q ( x ) p ( x ) } d x KL(p || q) = -\int p(x)ln q(x)dx - (-\int p(x)lnp(x)dx) = -\int p(x)ln\{\frac{q(x)}{p(x)}\}dx KL(p∣∣q)=−∫p(x)lnq(x)dx−(−∫p(x)lnp(x)dx)=−∫p(x)ln{p(x)q(x)​}dx

which is call relative entropy or KL divergence.

If we have observed a finite set of training points(from $p(x)$), then the expectation with respect to $p(x)$ can by approximated, so that:

K L ( p ∣ ∣ q ) ≃ 1 N { − l n q ( x n ∣ θ ) + l n p ( x n ) } KL(p || q) \simeq \frac{1}{N}\{-ln q(x_{n} | \theta) + ln p(x_{n}) \} KL(p∣∣q)≃N1​{−lnq(xn​∣θ)+lnp(xn​)}

which shows that minimizing KL divergence is equivalent to maximizing the likelihood function.

We can also use mutual information to judge whether two distributions are closed to independent:

$$I[x,y]=KL(p(x,y)||p(x)p(y))=-\int \int p(x,y)ln(\frac{p(x)p(y)}{p(x,y)})dxdy$$

The relationship between mutual information and conditional entropy:

$$I[x,y]=H[x]-H[x|y]=H[y]-H[y|x]$$