模式识别 | PRML Chapter 7 Sparse Kernel Machines

PRML Chapter 7 Sparse Kernel Machines

7.1 Maximum Margin Classifiers

The two-class classification problem using linear models of the form:

$$y(x)=w^T\varphi (x)+b$$

The maximum margin solution is found by solving:

$$\arg \underset{w,b}{\max }\left\{\frac{1}{||w||}\underset{n}{\min }[t_n(w^T\varphi (x_n)+b)]\right\}$$

where we have the constraint: $t_{n}y(x_n)\ge 0$

The direct solution of this optimization problem would be very complex and we can transform the objective function to:

$$\arg \underset{w,b}{\max }\frac{1}{2}||w|{|}^2$$

where we have:

$$t(w^T\varphi (x_n)+b)\ge 1$$

To solve the problem, we introduce Lagrange multipliers $a_n\ge 0$, and the Lagrangian function will be:

L ( w , b , a ) = 1 2 ∣ ∣ w ∣ ∣ 2 − ∑ n = 1 N a n { t n ( w T ϕ ( x n ) − 1 ) } L(w, b, a) = \frac{1}{2}||w||^{2} - \sum_{n=1}^{N}a_{n}\{ t_{n}(w^{T}\phi(x_{n}) - 1) \} L(w,b,a)=21​∣∣w∣∣2−n=1∑N​an​{tn​(wTϕ(xn​)−1)}

Setting the derivatives with respect to $w$ and $b$ equal to zero we will obtain:

$$w=\sum _{n=1}^N{a}_n{t}_n\varphi (x_n)$$

$$0=\sum _{n=1}^N{a}_n{t}_n$$

Eliminating $w$ and $b$ from $L(w,b,a)$ using these conditions then gives the dual representation of the maximum margin problem in which we maximize:

$$\tilde{L}(a)=\sum _{n=1}^N{a}_n-\frac{1}{2}\sum _{n=1}^N\sum _{n=1}^{N}a-n{a}_m{t}_n{t}_{m}k(x_n,x_m)$$

with respect to the constraints:

$$a_n\ge 0,n=1,\dots ,N$$

$$\sum _{n=1}^N{a}_n{t}_n=0$$

Substituting for $w$, using parameters { a n } \{a_n\} {an​} and the kernel function:

$$y(x)=\sum _{n=1}^N{a}_n{t}_{n}k(x,x_n)+b$$

A constrained optimization of this form satisfies the KKT conditions:

$$a_n\ge 0$$

$$t_{n}y(x_n)-1\ge 0$$

a n { t n y ( x n ) − 1 } = 0 a_{n}\{ t_{n}y(x_{n}) - 1 \} = 0 an​{tn​y(xn​)−1}=0

thus for every data point, either $a_n=0$ or $t_{n}y(x_n)=1$. Solving for $b$ to give:

$$b=\frac{1}{N_S}\sum _{n\in S}\left(t_n-\sum _{m\in S}{a}_m{t}_{m}k(x_n,x_m)\right)$$

7.1.1 Overlapping class distributions

When we are going to maximize the margin while softy penalizing points that lie on the wrong sie of the margin boundary, we therefore minimize:

$$C\sum _{n=1}^N{\xi }_n+\frac{1}{2}||w|{|}^2$$

The constraints will be replaced with:

$$t_{n}y(x_n)\ge 1-{\xi }_n,n=1,\dots ,N$$

$${\xi }_n\ge 0$$

The corresponding Lagrangian is given by:

L ( w , b , ξ , a , μ ) = 1 2 ∣ ∣ w ∣ ∣ 2 + C ∑ n = 1 N ξ n − ∑ n = 1 N a n { t n y ( x n ) − 1 + ξ n } − ∑ n = 1 N μ n ξ n L(w, b, \xi, a, \mu) = \frac{1}{2}||w||^{2} + C\sum_{n=1}^{N}\xi_{n}-\sum_{n=1}^{N}a_{n}\{t_{n}y(x_{n}) - 1 + \xi_{n}\} - \sum_{n=1}^{N}\mu_{n}\xi_{n} L(w,b,ξ,a,μ)=21​∣∣w∣∣2+Cn=1∑N​ξn​−n=1∑N​an​{tn​y(xn​)−1+ξn​}−n=1∑N​μn​ξn​

The corresponding set of KKT conditions are given by:

$$a_n\ge 0$$

$$t_{n}y(x_n)-1+{\xi }_n\ge 0$$

$$a_n(t_{n}y(x_n)-1+{\xi }_n)=0$$

$${\mu }_n\ge 0$$

$${\xi }_n\ge 0$$

$${\mu }_n{\xi }_n=0$$

After we eliminate $w$,$b$ and ${\xi }_n$ from the Lagrangian we obtain the dual Lagrangian in the form:

$$\tilde{L}(a)=\sum _{n=1}^N{a}_n-\frac{1}{2}\sum _{n=1}^N\sum _{m=1}^N{a}_n{a}_m{t}_n{t}_{m}k(x_n,x_m)$$

with respect to the subjection:

$$0\le a_n\le C$$

$$\sum _{n=1}^N{a}_n{t}_n=0$$

Similarly, a numerically stable solution is obtained by:

$$b=\frac{1}{N_M}\sum _{n\in M}\left(t_n-\sum _{m\in S}{a}_m{t}_{m}k(x_n,x_m)\right)$$

7.1.2 Relation to logistic regression

The objective function can be written in the form:

$$\sum _{n=1}^N{E}_{SV}(y_n{t}_n)+\lambda ||w|{|}^2$$

Also, we can construct an error function by taking the negative logarithm of the likelihood function that, with a quadratic regularizer, takes the form:

$$\sum _{n=1}^N{E}_{LR}(y_n{t}_n)+\lambda ||w|{|}^2$$

7.1.3 Multiclass SVMs

7.1.4 SVMs for regression

To obtain sparse solutions, we therefore minimizing a regularized error function:

$$C\sum _{n=1}^N{E}_{ϵ}(y(x)-t_n)+\frac{1}{2}||w|{|}^2$$

Introducing slack variables, the error function for support vector regression can then be written as:

$$C\sum _{n=1}^N({\xi }_n+\widehat{{\xi }_n})+\frac{1}{2}||w|{|}^2$$

with the corresponding conditions:

$$y_n-ϵ\le t_n\le y_n+ϵ$$

$${\xi }_n\ge 0,\widehat{{\xi }_n}\ge 0$$

The dual problem involves maximizing:

$$\tilde{L}(a,\hat{a})=-\frac{1}{2}\sum _{n=1}^N\sum _{m=1}^N(a_n-\widehat{a_n})(a_m-\widehat{a_m})k(x_n,x_m)-ϵ\sum _{n=1}^N(a_n+\widehat{a_n})+\sum _{n=1}^N(a_n-\widehat{a_n})t_n$$

with respect to the condition:

$$0\le a_n\le C$$

$$0\le {\hat{a}}_n\le C$$

Solve the dual problem and we can see the predictions for new inputs can be made using:

$$y(x)=\sum _{n=1}^N(a_n-{\hat{a}}_n)k(x,x_n)+b$$

which is again expressed in terms of the Kernel function. The corresponding KKT conditions:

$$a_n(ϵ+{\xi }_n+y_n-t_n)=0$$

$${\hat{a}}_n(ϵ+{\hat{\xi }}_n-y_n+t_n)=0$$

$$(C-a_n){\xi }_n=0$$

$$(C-{\hat{a}}_n){\hat{\xi }}_n=0$$

Solving for $b$ we obtain:

$$b=t_n-ϵ-\sum _{m=1}^N(a_m-{\hat{a}}_m)k(x_n,x_m)$$

7.1.5 Computational learning theory

7.2 Relevance Vector Machines

7.2.1 RVM for regression

The relevance vector machine for regression is a linear model of the form discussed in Chapter 3 but with a modified prior that results in sparse solutions. The condition distribution is:

$$p(t|\mathbf{x},\mathbf{w},\beta )=N(t|y(x),{\beta }^{-1})$$

where $\beta ={\sigma }^{-2}$ is the noise precision and the mean is given by a linear model of the form:

$$y(x)=\sum _{i=1}^M{w}_i{\varphi }_i(x)=w^T\varphi (x)$$

this general expression takes the SVM-like form:

$$y(x)=\sum _{n=1}^N{w}_{n}k(x,x_n)+b$$

Suppose we have N observations, the likelihood function is given by:

$$p(\mathbf{t}|\mathbf{X},w,\beta )=\prod _{n=1}^{N}p(t_n|x_n,w,\beta )$$

The prior distribution over the parameter $w$:

$$p(w|\alpha )=\prod _{i=1}^{M}N(w_i|0,{\alpha }_i^{-1})$$

It is easy to infer that the posterior distribution for $w$ takes the form:

$$p(w|\mathbf{t},\mathbf{X},\alpha ,\beta )=N(w|m,\Sigma )$$

where $m=\beta \Sigma {\mathbf{\Phi }}^{\mathbf{T}}\mathbf{t}$ and $\Sigma =(\mathbf{A}+\beta {\mathbf{\Phi }}^{\mathbf{T}}\mathbf{\Phi }{)}^{-1}$.

We should maximize the marginal likelihood function:

$$p(\mathbf{t}|\mathbf{X},\alpha ,\beta )=\int p(\mathbf{t}|\mathbf{X},w,\beta )p(w|\alpha )dw$$

The log marginal likelihood will be the form:

$$p(\mathbf{t}|\mathbf{X},\alpha ,\beta )=\ln N(\mathbf{t}|0,\mathbf{C})$$

$$C={\beta }^{-1}\mathbf{I}+\mathbf{\Phi }{\mathbf{A}}^{-1}{\mathbf{\Phi }}^{\mathbf{T}}$$

The predictive distribution will be:

$$p(t|x,\mathbf{X},\mathbf{t},{\alpha }^{\ast },{\beta }^{\ast })=N(t|m^T\varphi (x),{\sigma }^2(x))$$

7.2.2 Analysis of sparsity

7.2.3 RVM for classification

We can extend the relevance vector machine framework to classification problems by applying the ARD prior over weights to a probabilistic linear classification model of that in Chapter 4.