模式识别 | PRML Chapter 6 Kernel Methods

PRML Chapter 6 Kernel Methods

For models which are based on a fixed nonlinear feature space mapping $\varphi (x)$, the kernel function is given by the relation:

$$k(x,x^{{}^{\prime }})=\varphi (x{)}^T\varphi (x^{{}^{\prime }})$$

Forms of kernel functions:

• linear kernels: $k(x,x^{\prime })=x^{T}x$
• stationary kernels: $k(x,x^{\prime })=k(x-x^{\prime })$
• homogeneous kernels: $k(x,x^{\prime })=k(||x-x^{\prime }||)$

6.1 Dual Representations

Consider a linear regression model whose parameters are determined by minimizing a regularized sum-of-squares error funciton given by:

J ( w ) = 1 2 ∑ n = 1 N { w T ϕ ( x n ) − t n } 2 + λ 2 w T w J(w) = \frac{1}{2}\sum_{n=1}^{N}\{ w^{T}\phi(x_{n}) - t_{n} \}^{2} + \frac{\lambda}{2}w^{T}w J(w)=21​n=1∑N​{wTϕ(xn​)−tn​}2+2λ​wTw

Set the gradient of $J(w)$ with respect to $w$ equals to zero, we can get the solution:

w = − 1 λ ∑ n = 1 N { w T ϕ ( x n ) − t n } ϕ ( x n ) = ∑ n = 1 N a n ϕ ( x n ) = Φ T a w = -\frac{1}{\lambda}\sum_{n=1}^N \{ w^{T}\phi(x_{n}) - t_{n} \}\phi(x_n)=\sum_{n=1}^N a_n\phi(x_n)= \Phi^{T}a w=−λ1​n=1∑N​{wTϕ(xn​)−tn​}ϕ(xn​)=n=1∑N​an​ϕ(xn​)=ΦTa

a n = − 1 λ { w T ϕ ( x n ) − t n } a_{n} = -\frac{1}{\lambda}\{ w^{T}\phi(x_{n}) - t_{n} \} an​=−λ1​{wTϕ(xn​)−tn​}

We now define the Gram matrix $K=\Phi {\Phi }^T$, which is an $N\ast N$ symmetric matrix with elements:

$$K_{nm}=\varphi (x_n{)}^T\varphi (x_m)=k(x_n,x_m)$$

In terms of the Gram matrix, the sum-of-squares error function can be written as:

$$J(a)=\frac{1}{2}{a}^T\mathbf{K}\mathbf{K}a-a^T\mathbf{K}\mathbf{t}+\frac{1}{2}{\mathbf{t}}^{\mathbf{T}}\mathbf{t}+\frac{\lambda }{2}{a}^T\mathbf{K}a$$

Setting the gradient $J(a)$ with respect to $a$ to zero, we obtain the following solution:

$$a=(K+\lambda {\mathbf{I}}_N{)}^{-1}\mathbf{t}$$

If we substitute this back into the linear regression model, we obtain the following prediction for a new $x$.

$$y(x)=w^T\varphi (x)=a^T\Phi \varphi (x)=k(x{)}^T(K+\lambda {\mathbf{I}}_N{)}^{-1}\mathbf{t}$$

where we have defined the vector $k(x)$ with elements: $k_n(x)=k(x_n,x)$.

6.2 Constructing Kernels

Here the kernel function is defined for a one-dimensional input space by

$$k(x,x^{\prime })=\varphi (x{)}^T\varphi (x^{\prime })=\sum _{i=1}^M{\varphi }_i(x){\varphi }_i(x^{\prime })$$

A necessary and sufficient condition for a function $k(x,x^{\prime })$ to be a valid kernel is that the Gram matrix $K$ should be positive semidefintie for all possible choices of the set { x n } \{x_n\} {xn​}.

Given valid kernels $k_1(x,x^{\prime })$ and $k_2(x,x^{\prime })$, the following new kernels will also be valid:

$$\{\begin{array}{rlr}k(x,x^{{}^{\prime }})& =& c{k}_1(x,x^{{}^{\prime }})\\ k(x,x^{{}^{\prime }})& =& f(x)k_1(x,x^{{}^{\prime }})f(x^{{}^{\prime }})\\ k(x,x^{{}^{\prime }})& =& q(k_1(x,x^{{}^{\prime }}))\\ k(x,x^{{}^{\prime }})& =& exp(k_1(x,x^{{}^{\prime }}))\\ k(x,x^{{}^{\prime }})& =& k_1(x,x^{{}^{\prime }})+k_2(x,x^{{}^{\prime }})\\ k(x,x^{{}^{\prime }})& =& k_1(x,x^{{}^{\prime }})k_2(x,x^{{}^{\prime }})\\ k(x,x^{{}^{\prime }})& =& k_3(\varphi (x),\varphi (x^{{}^{\prime }}))\\ k(x,x^{{}^{\prime }})& =& x^{T}A{x}^{{}^{\prime }}\\ k(x,x^{{}^{\prime }})& =& k_a(x_a,x_a^{{}^{\prime }})+k_b(x_b,x_b^{{}^{\prime }})\\ k(x,x^{{}^{\prime }})& =& k_a(x_a,x_a^{{}^{\prime }})k_b(x_b,x_b^{{}^{\prime }})\end{array}$$

Another commonly used kernel takes the form:

$$k(x,x^{{}^{\prime }})=exp\left(-\frac{||x-x^{{}^{\prime }}|{|}^2}{2{\sigma }^2}\right)$$

which is called Gaussian kernel.

One powerful approach to the construction of kernels starts from a probabilistic generative model, which allows us to apply generative models in a discriminative setting. Given a generative model $p(x)$ we can define a kernel by:

$$k(x,x^{\prime })=p(x)p(x^{\prime })$$

To extend this class of kernels by considering sums over products of different probability distributions:

$$k(x,x^{\prime })=\sum _{i}p(x|i)p(x^{\prime }|i)p(i)$$

Taking the limit of an infinite sum, we can also consider kernels of the form:

$$k(x,x^{\prime })=\int p(x|z)p(x^{\prime }|z)p(z)dz$$

Also, by extending the mixture representation we can define a kernel function measuring the similarity of two sequences $X$ and $X^{\prime }$:

$$k(X,X^{\prime })=\sum _{Z}p(X|Z)p(X^{\prime }|Z)p(Z)$$

Another models is known as Fisher kernel defined by:

$$k(x,x^{\prime })=g(\theta ,x{)}^T{F}^{-1}g(\theta ,x^{\prime })$$

here the Fisher score is $g(\theta ,x)={\nabla }_{\theta }lnp(x\Vert \theta )$, $F$ is the Fisher information matrix, given by $F=E_x[g(\theta ,x)g(\theta ,x{)}^T]$.

6.3 Radial Basis Function Networks

Radial basis functions have the property that each basis function depends only on the radial distance (typically Euclidean) from a centre ${\mu }_j$, so that

$${\varphi }_j(x)=h(||x-{\mu }_j||)$$

6.3.1 Nadaraya-Watson model

Starting with kernel density estimation. Suppose we have a training set { x n , t n } \{x_n,t_n\} {xn​,tn​} and we use a Parzen density estimator to model the joint distribution $p(x,t)$, so that

$$p(x,t)=\frac{1}{N}\sum _{n=1}^{N}f(x-x_n,t-t_n)$$

We now find an expression for the regression function $y(x)$, corresponding to the conditional average of the target variable conditioned on the input variable, which is given by:

$$\begin{array}{rl}y(x)& =E[t|x]={\int }_{-\infty }^{\infty }tp(t|x)\\ & =\frac{\int tp(x,t)dt}{\int p(x,t)dt}\\ & =\frac{{\sum }_n\int tf(x-x_n,t-t_n)dt}{{\sum }_m\int f(x-x_m,t-t_m)dt}\end{array}$$

We now assume for simplicity that the component density functions have zero mean so that:

$${\int }_{-\infty }^{\infty }f(x,t)tdt=0$$

for all values of $x$, using a simple change of variable, we then obtain:

$$y(x)=k(x,x_n)=\frac{g(x-x_n)t_n}{{\sum }_{m}g(x-x_m)}=\sum _{n}k(x,x_n)t_n$$

and we have defined:

$$g(x)={\int }_{-\infty }^{\infty }f(x,t)dt$$

The result is known as the Nadaraya-Watson model or kernel regression.

6.4 Gaussian Processes

6.4.1 Linear regression revisited

Consider a model defined in terms of a linear combination of fixed basis functions:

$$y(x)=w^T\varphi (x)$$

The prior distribution over $w$:

$$p(w)=N(w|0,{\alpha }^{-1}\mathbf{I})$$

Evaluate the function at specific values of $x$, the vector is given by:

$$\mathbf{y}=\Phi w$$

And the mean and covariance:

$$E[\mathbf{y}]=\Phi E[w]=0$$

$$cov[\mathbf{y}]=E[\mathbf{y}{\mathbf{y}}^T]=\mathbf{K}$$

where $K$ is the Gram matrix with elements:

$$K_{nm}=k(x_n,x_m)=\frac{1}{\alpha }\varphi (x_n{)}^T\varphi (x_m)$$

6.4.2 Gaussian processes for regression

In order to apply Gaussian process models to the problem of regression, we need to take account of the noise on the observed target values:

$$t_n=y_n+{ϵ}_n$$

Consider noise processes that have a Gaussian distribution, so that:

$$p(t_n|y_n)=N(t_n|y_n,{\beta }^{-1})$$

The conditional distribution is given by an isotropic Gaussian of the form:

$$p(\mathbf{t}|\mathbf{y})=N(\mathbf{t}|\mathbf{y},{\beta }^{-1}{\mathbf{I}}_N)$$

From the definition, the marginal distribution $p(y)$ is given by a Gaussian:

$$p(\mathbf{y})=N(\mathbf{y}|0,\mathbf{K})$$

We see that the marginal distribution of $\text{t}$ is given by:

$$p(\mathbf{t})=\int p(\mathbf{t}|\mathbf{y})p(\mathbf{y})d\mathbf{y}=N(\mathbf{t}|0,\mathbf{C})$$

where the covariance matrix $C$ has elements:

$$C(x_n,x_m)=k(x_n,x_m)+{\beta }^{-1}{\delta }_{nm}$$

One widely used kernel function for Gaussian process regression is given by the exponential of a quadratic form, with the addition of constant and linear terms to give:

$$k(x_n,x_m)={\theta }_{0}exp\left\{-\frac{{\theta }_1}{2}||x_n-x_m|{|}^2\right\}+{\theta }_2+{\theta }_3{x}_n^T{x}_m$$

The joint distribution over $t_1,...,t_{N+1}$ will be given by:

$$p(t_{N+1})=N(t_{N+1}|0,C_{N+1})$$

So the conditional distribution $p(t_{N+1}\Vert \text{t})$ is a Gaussian distribution with mean and covariance given by:

$$m(x_{N+1})=K^t{c}_N^{-1}\mathbf{t}$$

$${\sigma }^2(x_{N+1})=c-k^T{C}_N^{-1}\mathbf{k}$$

6.4.3 Learning the hyperparameters

Techniques are based on the evaluation of the likelihood function and maximize the log likelihood.

6.4.4 Automatic relevance determination

The example of a Gaussian process with a two-dimensional input space $x=(x_1,x_2)$, having a kernel function of the form:

k ( x , x ′ ) = θ 0 e x p { − 1 2 ∑ i = 1 2 η i ( x i − x i ′ ) 2 } k(x,x')=\theta_0 exp\{-\frac{1}{2}\sum_{i=1}^2 \eta_i(x_i-x_i')^2\} k(x,x′)=θ0​exp{−21​i=1∑2​ηi​(xi​−xi′​)2}

The ARD framework is easily incorporated into the exponential-quadratic kernel to give the following form of kernel function:

k ( x n , x m ) = θ 0 e x p { − 1 2 ∑ i = 1 D η i ( x n i − x m i ) 2 } + θ 2 + θ 3 ∑ i = 1 D x n i x m i k(x_n,x_m)=\theta_0 exp\{-\frac{1}{2}\sum_{i=1}^D\eta_i(x_{ni}-x_{mi})^2\}+\theta_2+\theta_3\sum_{i=1}^D x_{ni}x_{mi} k(xn​,xm​)=θ0​exp{−21​i=1∑D​ηi​(xni​−xmi​)2}+θ2​+θ3​i=1∑D​xni​xmi​

6.4.5 Gaussian processes for classification

For the two-class problem if we define a Gaussian process over a function $a(x)$ and then transform the function using a logistic sigmoid $y=\sigma (a)$, then we obtain a non-Gaussian stochastic process over functions $y(x)$ where $y\in (0,1)$. The distribution of target variable $t$ is given by the Bernoulli distribution:

$$p(t|a)=\sigma (a{)}^t(1-\sigma (a){)}^{1-t}$$

Our goal is to determine the predictive distribution $p(t_{N+1}\Vert \text{t})$, so we introduce a Gaussian process prior over the vector $a_{N+1}$:

$$p(a_{N+1})=N(a_{N+1}|0,C_{N+1})$$

For numerical reasons it is convenient to introduce a noise-like term governed by a parameter $v$ that ensures the covariance matrix is positive definite. Thus the covariance matrix $C_{N+1}$ has elements given by:

$$C(x_n,x_m)=k(x_n,x_m)+\nu {\delta }_{nm}$$

The required predictive distribution is given by:

$$p(t_{N+1}=1|t_N)=\int p(t_{N+1}=1|a_{N+1})p(a_{N+1}|t_N)d{a}_{N+1}$$

where $p(t_{N+1}=1|a_{N+1})=\sigma (a_{N+1})$

6.4.6 Laplace approximation

In order to evaluate the predictive distribution, we seek a Gaussian approximation to the posterior distribution over $a_{N+1}$:

$$p(a_{N+1}|{\text{t}}_N)=\int p(a_{N+1},a_N|{\text{t}}_N)d{a}_N=\int p(a_{N+1}|a_N)p(a_N|{\text{t}}_N)d{a}_N$$

We also need to determine the parameters $\theta$ of the covariance function. The likelihood funciton is defined by:

$$p({\text{t}}_N|\theta )=\int p({\text{t}}_N|a_N)p(a_N|\theta )d{a}_N$$

6.4.7 Connection to neural networks