模式识别 | PRML Chapter 10 Approximate Inference

PRML Chapter 10 Approximate Inference

10.1 Variational Inference

For observed variable X = { x 1 , . . . , x N } X=\{x_1,...,x_N\} X={x1​,...,xN​} and latent Z = { z 1 , . . . , z N } Z=\{z_1,...,z_N\} Z={z1​,...,zN​}. Our probabilistic model specifies the joint distribution $p(Z\Vert X)$ and our goal is to find an approximation for the posterior distribution $p(Z\Vert X)$ as well as for the model evidence $p(X)$. As in our discussion of EM, we can decompose the log marginal probability using:

$$\ln p(X)=\mathcal{L}(q)+KL(q||p)$$

where

$$\mathcal{L}(q)=\int q(Z)\ln \left\{\frac{p(X,Z)}{q(Z)}\right\}dZ$$

K L ( q ∣ ∣ p ) = − ∫ q ( Z ) ln ⁡ { p ( Z ∣ X ) q ( Z ) } d Z KL(q || p) = -\int q(Z)\ln \left\{p(Z | X)q(Z) \right\} dZ KL(q∣∣p)=−∫q(Z)ln{p(Z∣X)q(Z)}dZ

10.1.1 Factorized distributions

Suppose we partition the elements of $Z$ into disjoint groups and the q factorizes with respect to there groups:

$$q(Z)=\prod _{i=1}^M{q}_i(Z_i)$$

substitute it to the fomula above and denoting $q_j(Z_j)$ by simply $q_j$ we obtain:

$$\mathcal{L}(q)=\int q_j\ln \tilde{p}(X,Z_j)-\int q_j\ln q_{j}d{Z}_j+const$$

where we have defined a new distribution $\tilde{p}(X,Z_j)$ by the relation:

$$\ln \tilde{p}(X,Z_j)=E_{i\ne j}[\ln p(X,Z)]+const$$

$$E_{i\ne j}[\ln p(X,Z)]=\int \ln p(X,Z)\prod _{i\ne j}{q}_{i}d{Z}_i$$

maximizing $\mathcal{L}(q)$ is equivalent to minimizing the Kullback-Leibler divergence, and the minimum occurs when $q_j(Z_j)=\tilde{p}(X,Z_j)$. Thus we obtain the generl expression for the optimal solution $q_j^{\ast }(Z_j)$ given by:

$$\ln q_j^{\ast }(Z_j)=E_{i\ne j}[\ln p(X,Z)]+const$$

10.1.2 Properties of factorized approximations

Two forms of Kullback-Leibler divergence are members of the alpha family of divergences defined by:

$$D_{\alpha }(p||q)=\frac{4}{1-{\alpha }^2}(1-\int p(x{)}^{(1+\alpha )/2}q(x{)}^{(1-\alpha )/2}dx)$$

10.1.3 Example: The univariate Gaussian

10.1.4 Model comparison

We can readily verify the following decomposition based on this variational distribution:

$$lnp(X)={\mathcal{L}}_m-\sum _m\sum _{Z}q(Z|m)q(m)ln(\frac{p(Z,m|X)}{q(Z|m)q(m)})$$

where ${\mathcal{L}}_m$ is a lower bound on $lnp(X)$ and is given by

$${\mathcal{L}}_m=\sum _m\sum _{Z}q(Z|m)q(m)ln(\frac{p(Z,X,m)}{q(Z|m)q(m)})$$

We can maximizing $\mathcal{L}$ with respect to the distribution $q(m)$ using a Lagrange multiplier, with the result:

$$q(m)\propto p(m)exp({\mathcal{L}}_m)$$

10.2 Illustration: Variational Mixture of Gaussians

The conditional distribution of $Z$, given the mixing coefficients $\pi$, in the form:

$$p(Z|\pi )=\prod _{n=1}^N\prod _{k=1}^K{\pi }_k^{z_{nk}}$$

Similarly, the conditional distribution of the observed data vectors, given the latent variables and the component parameters:

$$p(X|Z,\mu ,\Lambda )=\prod _{n=1}^N\prod _{k=1}^K\mathcal{N}(x_n|{\mu }_k,{\Lambda }_k^{-1}{)}^{z_{nk}}$$

Choose a Dirichlet distribution over the mixing coefficients $\pi$:

$$p(\pi )Dir(\pi |{\alpha }_0)C({\alpha }_0)\prod _{k=1}^K{\pi }_k^{{\alpha }_0-1}$$

Introduce an independent Gaussian-Wishart prior governing the mean and precision of each Gaussian component, give by:

$$p(\mu ,\Lambda )=p(\mu |\Lambda )p(\Lambda )=\prod _{k=1}^K\mathcal{N}({\mu }_k|m_0,({\beta }_0\Lambda {)}^{-1})\mathcal{W}({\Lambda }_k|W_0,{\nu }_0)$$

10.2.1 Variational distribution

In order to formulate a variational treatment of this model, we write down the joint distribution of all of the random variables:

$$p(X,Z,\pi ,\mu ,\Lambda )=p(X|Z,\mu ,\Lambda )p(Z|\pi )p(\pi )p(\mu |\Lambda )p(\Lambda )$$

Consider a variational distribution which factorizes between the latent variables and the parameters as that:

$$q(Z,\pi ,\mu ,\Lambda )=q(Z)q(\pi ,\mu ,\Lambda )$$

Let us consider the derivation of the update equation for the factor $q(Z)$. The log of the optimized factor is given by:

$$\ln q^{\ast }(Z)=E_{\pi ,\mu ,\Lambda }[\ln p(X,Z,\pi ,\mu ,\Lambda )]+const$$

Absorb terms do not depend on $Z$ into the constant:

$$\ln q^{\ast }(Z)=E_{\pi }[\ln p(Z|\pi )]+E_{\mu ,\Lambda }[\ln p(X|Z,\mu ,\Lambda )]+const$$

Substituting for the two conditional distributions on the right-hand side and absorb terms independent of $Z$ into the constant, we have:

$$ln{q}^{\ast }(Z)=\sum _{n=1}^N\sum _{k=1}^K{z}_{nk}ln{\rho }_{nk}+const$$

where we have defined:

$$\ln {\rho }_{nk}=E[\ln {\pi }_k]+\frac{1}{2}E[\ln |{\Lambda }_k|]-\frac{D}{2}\ln (2\pi )-\frac{1}{2}{E}_{{\mu }_k,{\Lambda }_k}[(x_n-{\mu }_k{)}^T{\Lambda }_k(x_n-{\mu }_k)]$$

Taking the exponential of both sides we obtain:

$$q^{\ast }(Z)\propto \prod _{n=1}^N\prod _{k=1}^K{\rho }_{nk}^{z_{nk}}$$

Normalized the distribution we obtain:

$$q^{\ast }(Z)=\prod _{n=1}^N\prod _{k=1}^K{r}_{nk}^{z_{nk}}$$

where:

$$r_{nk}=\frac{{\rho }_{nk}}{{\sum }_{j=1}^K{\rho }_{nj}}$$

For the distribution $q^{\ast }(Z)$ we have the standard result:

$$E[z_{nk}]=r_{nk}$$

Let us consider the factor $q(\pi ,\mu ,\Lambda )$ in the variational posterior distribution. We have:

$$\ln q^{\ast }(\pi ,\mu ,\Lambda )=\ln p(\pi )\sum _{k=1}^K\ln p({\mu }_k,{\Lambda }_k)+E_Z[\ln p(Z|\pi )]+\sum _{k=1}^K\sum _{n=1}^{N}E[z_{nk}]\ln \mathcal{N}(x_n|{\mu }_k,{\Lambda }_k^{-1})+const$$

The variational approximation will be:

$$q(\pi ,\mu ,\Lambda )=q(\pi )\prod _{k=1}^{k}q({\mu }_k,{\Lambda }_k)$$

The results are given by:

$$q^{\ast }(\pi )=Dir(\pi |\alpha )$$

where $\alpha$ has components ${\alpha }_k$ given by:

$${\alpha }_k={\alpha }_0+\sum _{n=1}^N{r}_{nk}$$

Using $q^{\ast }({\mu }_k,{\Lambda }_k)q^{\ast }({\mu }_k\Vert {\Lambda }_k)q^{\ast }({\Lambda }_k)$, the posterior distribution is a Gaussian-Wishart distribution and is given by:

$$q^{\ast }({\mu }_k,{\Lambda }_k)=\mathcal{N}({\mu }_k|m_k,({\beta }_k{\Lambda }_k{)}^{-1})\mathcal{W}({\Lambda }_k|W_k,{\nu }_k)$$

where

$${\beta }_k={\beta }_0+N_0$$

$$m_k=\frac{1}{{\beta }_k}({\beta }_0{m}_0+N_k{\hat{x}}_k)$$

$$W_k^{-1}=W_0^{-1}+N_k{S}_k+\frac{{\beta }_0{N}_k}{{\beta }_0+N_k}({\hat{x}}_k-m_0)({\hat{x}}_k-m_0{)}^T$$

$${\nu }_k={\nu }_0+N_k$$

10.2.2 Variational lower bound

For the variational mixture of Gaussians, the lower bound is given by:

$$\mathcal{L}=\sum _{Z}q(Z,\pi ,\mu ,\Lambda )\ln \left\{\frac{p(X,Z,\pi ,\mu ,\Lambda )}{q(Z,\pi ,\mu ,\Lambda )}\right\}d\pi d\mu d\Lambda$$

10.2.3 Predictive density

Wihe a new value $\hat{x}$ the predictive density is given by:

$$p(\hat{x}|X)=\sum _{\hat{Z}}\int \int \int p(\hat{X}|\hat{z},\mu ,\Lambda )p(\hat{z}|\pi )p(\pi ,\mu ,\Lambda |X)d\pi d\mu d\Lambda$$

10.2.4 Determining the number of components

10.2.5 Induced factorizations

10.3 Variational Linear Regression

The joint distribution of all the variables is given by:

$$p(t,w,\alpha )=p(t|w)p(w|\alpha )p(\alpha )$$

where

$$p(t|w)=\prod _{n=1}^{N}N(t_n|w^T{\varphi }_n,{\beta }^{-1})$$

$$p(w|\alpha )=N(w|0,{\alpha }^{-1}I)$$

$$p(\alpha )=Gam(\alpha |a_0,b_0)$$

10.3.1 Variational distribution

Our first goal is to find an approximation to the posterior distribution $p(w,\alpha \Vert \mathbf{t})$. The variational posterior distribution is given by the factorized expression:

$$q(w,\alpha )=q(w)q(\alpha )$$

The result will be:

$$q^{\ast }(\alpha )=Gam(\alpha |a_N,b_N)$$

where

$$a_N=a_0+\frac{M}{2}$$

$$b_N=b_0+\frac{1}{2}E[w^{T}w]$$

and the distribution $q^{\ast }(w)$ is Gaussian:

$$q^{\ast }(w)=N(w|m_N,S_N)$$

where

$$m_N=\beta S_N{\Phi }^{T}t$$

$$S_N=(E[\alpha ]I+\beta {\Phi }^T\Phi {)}^{-1}$$

10.3.2 Predictive distribution

The predictive distribution over $t$, given a new input $x$ is evaluated using the Gaussian variational posterior:

$$p(t|x,\mathbf{t})\simeq N(t|m_N^T\varphi (x),{\sigma }^2(x))$$

where the input-dependent variance is given by

$${\sigma }^2(x)=\frac{1}{\beta }+\varphi (x{)}^T{S}_N\varphi (x)$$

10.3.3 Lower bound

10.4 Exponential Family Distributions

Suppose the joint distribution of observed and latent variables is a member of the exponential family, parameterized by natural parameters $\eta$ so that:

p ( X , Z ∣ η ) = ∏ n = 1 N h ( x n , z n ) g ( η ) e x p { η T u ( x n , z n ) } p(X,Z|\eta)=\prod_{n=1}^N h(x_n,z_n)g(\eta)exp\{\eta^Tu(x_n,z_n)\} p(X,Z∣η)=n=1∏N​h(xn​,zn​)g(η)exp{ηTu(xn​,zn​)}

we shall also use a conjugate prior for $eta$, which can be written as:

p ( η ∣ v 0 ) = f ( v 0 , x 0 ) g ( η ) v 0 e x p { v 0 η T x 0 } p(\eta|v_0)=f(v_0,x_0)g(\eta)^{v_0}exp\{v_0\eta^Tx_0\} p(η∣v0​)=f(v0​,x0​)g(η)v0​exp{v0​ηTx0​}

Now consider a variational distribution that factorizes between the latent variables and the parameters, so that $q(Z,\eta )=q(Z)q(\eta )$. The result will be

q ∗ ( z n ) = h ( x n , z n ) g ( E [ η ] ) e x p { E [ η T ] u ( x n , z n ) } q^*(z_n)=h(x_n,z_n)g(E[\eta])exp\{E[\eta^T]u(x_n,z_n)\} q∗(zn​)=h(xn​,zn​)g(E[η])exp{E[ηT]u(xn​,zn​)}

q ∗ ( η ) = f ( v N , x N ) g ( η ) v N e x p { η T x N } q^*(\eta)=f(v_N,x_N)g(\eta)^{v_N}exp\{\eta^Tx_N\} q∗(η)=f(vN​,xN​)g(η)vN​exp{ηTxN​}

where

$$v_N=v_o+N$$

$$x_N=x_0+\sum _{n=1}^N{E}_{z_n}[u(x_n,z_n)]$$

10.4.1 Variational message passing

The joint distribution corresponding to a directed graph can be written using the decomposition:

$$p(x)=\prod _{i}p(x_i|p{a}_i)$$

Now consider a variational approximation in which the distribution $q(x)$ is assumed to factorize with respect to the $x_i$ so that:

$$q(x)=\prod _i{q}_i(x_i)$$

Substitute the formula above into the general result we will get:

$$ln{q}_j^{\ast }(x_j)=E_{i\ne j}[\sum _{i}lnp(x_i|p{a}_i)]+const$$

10.5 Local Variational Methods

For convex funcitons, we can obtain upper bounds by:

g ( η ) = − min ⁡ x { f ( x ) − η x } = max ⁡ x { η x − f ( x ) } g(\eta) = - \min_{x}\{ f(x) - \eta x \} = \max_{x}\{ \eta x - f(x) \} g(η)=−xmin​{f(x)−ηx}=xmax​{ηx−f(x)}

f ( x ) = max ⁡ η { η x − g ( η ) } f(x) = \max_{\eta}\{ \eta x - g(\eta) \} f(x)=ηmax​{ηx−g(η)}

And for concave functions:

f ( x ) = min ⁡ η { η x − g ( η ) } f(x) = \min_{\eta}\{ \eta x- g(\eta) \} f(x)=ηmin​{ηx−g(η)}

g ( η ) = min ⁡ x { η x − f ( x ) } g(\eta) = \min_{x}\{ \eta x - f(x) \} g(η)=xmin​{ηx−f(x)}

If the function is not convex or concave, then we need the invertible transformations. An example will be logistic sigmoid function:

$$\sigma (x)=\frac{1}{1+e^{-x}}$$

The result with that of the logistic sigmoid will be:

$$\sigma (x)\le exp(\eta x-g(\eta ))$$

$$\sigma (x)\ge \sigma (\xi )exp\left\{\frac{x-\xi }{2}-\lambda (\xi )(x^2-{\xi }^2)\right\}$$

where:

$$\lambda (\xi )=-\frac{1}{2\xi }\left[\sigma (\xi )-\frac{1}{2}\right]$$

We cna see how the bounds can be used, suppose we wish to evaluate an integral of the form:

$$I=\int \sigma (a)p(a)da$$

We can employ the variational bound and we will get:

$$I\ge \int f(a,\xi )p(a)da=F(\xi )$$

10.6 Variational Logistic Regression

10.6.1 Variational posterior distribution

In the variational framework, we seek to maximize a lower bound on the marginal likelihood. For the Bayesian logistic regression model, the marginal likelihood takes the form:

$$p(t)=\int p(t|w)p(w)dw=\int \left[\prod _{n=1}^{N}p(t_n|w)\right]p(w)dw$$

The conditional distribution for $t$ can be written as

$$p(t|w)=e^{at}\sigma (-a)$$

where $a=w^T\varphi$. Using the variational lower bound on the logistic sigmoid function we can get:

$$p(t|w)=e^{at}\sigma (-a)\ge e^{at}\sigma (\xi )exp\left\{-\frac{a+\xi }{2}-\lambda (\xi )(a^2-{\xi }^2)\right\}$$

Using $a=w^T\varphi$, and multiplying by the prior distribution, we obtain the bound on the joint distribution of $t$ and $w$:

$$p(t|w)=p(t|w)p(w)\ge h(w,\xi )p(w)$$

where

h ( w , ξ ) = ∏ n = 1 N σ ( ξ n ) e x p { w T ϕ n t n − ( w T ϕ n + ξ n ) / 2 − λ ( ξ n ) ( [ w T ϕ n ] − ξ n 2 ) } h(w, \xi) = \prod_{n=1}^{N}\sigma(\xi_{n})exp\{w^{T}\phi_{n}t_{n} - (w^{T}\phi_{n} + \xi_{n})/2 - \lambda(\xi_{n})([w^{T}\phi_{n}] - \xi^{2}_{n}) \} h(w,ξ)=n=1∏N​σ(ξn​)exp{wTϕn​tn​−(wTϕn​+ξn​)/2−λ(ξn​)([wTϕn​]−ξn2​)}

The Gaussian variational posterior will be the form:

$$q(w)=\mathcal{N}(w|m_N,S_N)$$

where

$$m_N=S_N\left(S_0^{-1}{m}_0+\sum _{n=1}^N(t_n-\frac{1}{2}){\varphi }_n\right)$$

$$S_N^{-1}=S_0^{-1}+2\sum _{n=1}^N\lambda ({\xi }_n){\varphi }_n{\varphi }_n^T$$

10.6.2 Optimizing the variational parameters

Substitute the inequality above back into the marginal likelihood to give:

$$\ln p(t)=\ln \int p(t|w)p(w)dw\ge \ln \int h(w,\xi )p(w)dw=\mathcal{L}(\xi )$$

There are two approaches to determining the ${\xi }_n$: EM algorithm and integrate over $w$.

10.6.3 Inference of hyperparameters

10.7 Expectation Propagation