模式识别 | PRML Chapter 8 Graphical Models

PRML Chapter 8 Graphical Models

8.1 Bayesian Networks

A specific graph can make probabilistic statements for a broad class of distributions. We can writh the joint distribution in the form:

$$p(a,b,c)=p(c|a,b)p(a,b)=p(c|a,b)p(b|a)p(a)$$

For a graph with K nodes, the joint distribution is given by:

$$p(x)=\prod _{k=1}^{K}p(x_k|p{a}_k)$$

8.1.1 Example: Polynomial regression

The joint distribution can be written as:

$$p(\mathbf{t},w)=p(w)\prod _{n=1}^{N}p(t_n|w)$$

It may be useful to make the parameters and stochastic variables explicit:

$$p(\mathbf{t},w|\mathbf{x},\alpha ,{\sigma }^2)=p(w|\alpha )\prod _{n=1}^{N}p(t_n|w,x_n,{\sigma }^2)$$

To calculate the posterior distribution of $w$, we note that:

$$p(w|\text{T})\propto p(w)\prod _{n=1}^{N}p(t_n|w)$$

To predict $\hat{t}$ for a new input value $\hat{x}$ we write down the joint distribution of all the random variables conditioned on the deterministic parameters:

$$p(\hat{t},\mathbf{t},w|\hat{x},\mathbf{x},\alpha ,{\sigma }^2)=[\prod _{n=1}^{N}p(t_n|x_n,w,{\sigma }^2)]p(w|\alpha )p(\hat{t}|\hat{x},w,{\sigma }^2)$$

The required predictive distribution for $\hat{t}$ is then obtained:

$$p(\hat{t}|\hat{x},\mathbf{x},\mathbf{t},\alpha ,{\sigma }^2)\propto \int p(\hat{t},\mathbf{t},w|\hat{x},\mathbf{x},\alpha ,{\sigma }^2)dw$$

8.1.2 Generative models

8.1.3 Discrete variables

The probability distribution $p(x\Vert \mu )$ for a single discrete variable $x$ having K possible states is given by:

$$p(x|\mu )=\prod _{k=1}^K{\mu }_k^{x_k}$$

Suppose we have two discrete variables and each of them has K states:

$$p(x_1,x_2|\mu )=\prod _{k=1}^K\prod _{l=1}^K{\mu }_{kl}^{x_{1k}{x}_{2l}}$$

We can use parameterized models for the conditional distributions, a more parsimonious form for the conditional distribution can be obtained by using logistic sigmoid function:

$$p(y=1|x_1,...,x_M)=\sigma (w_0+\sum _{i=1}^M{w}_i{x}_i)=\sigma (w^{T}x)$$

8.1.4 Linear-Gaussian models

Suppose a single continuous random variable $x_i$ having a Gaussian distribution:

$$p(x_i|p{a}_i)=N(x_i|\sum _{j\in p{a}_i}{w}_{ij}{x}_j+b_i,v_i)$$

The log of the distribution is the log of the product of conditionals over all nodes:

$$\ln p(x)=\sum _{i=1}^{D}p(x_i|p{a}_i)=-\sum _{i=1}^D\frac{1}{2v_i}{\left(x_i-\sum _{j\in p{a}_i}{w}_{ij}{x}_j-b_i\right)}^2+Const$$

Each variable $x_i$ has a Gaussian distribution and we can get:

$$x_i=\sum _{j\in p{a}_i}{w}_{ij}{x}_j+b_i+\sqrt{v_i}{ϵ}_i$$

where ${ϵ}_i$ satisfies: $E[{ϵ}_i]=0$ and $E[{ϵ}_i{ϵ}_j]=I_{ij}$. We can then get the mean and covariance of the joint distribution:

$$E[x_i]=\sum _{j\in p{a}_i}{w}_{ij}E[x_j]+b_i$$

$$cov[x_i,x_j]=\sum _{k\in p{a}_j}{w}_{jk}cov[x_i,x_k]+I_{ij}{v}_j$$

8.2 Conditional Independence

We say that $a$ is conditionally independent of $b$ given $c$ if we have:

$$p(a|b,c)=p(a|c)$$

We use a shorthand notation for conditional independence in which:

$$a\perp \perp b|c$$

8.2.1 Three example graphs

• Tail-to-Tail Node
• Head-to-Tail Node
• Head-to-Head Node

8.2.2 D-separation

Suppose we condition on $\mu$ and the joint distribution of the observations can be written as:

$$p(D|\mu )=\prod _{n=1}^{N}p(x_n|\mu )$$

The conditional independence properties can also explored by Markov blanked or Markov boundary, a conditional distribution can be expressed in the form:

p ( x i ∣ x { j ≠ i } ) = p ( x 1 , . . . , x D ) ∫ p ( x 1 , . . . , x D ) d x i = ∏ k p ( x k ∣ p a k ) ∫ ∏ k p ( x k ∣ p a k ) d x i p(x_i|x_{\{j\not =i\}})=\frac{p(x_1,...,x_D)}{\int p(x_1,...,x_D)dx_i}=\frac{\prod_k p(x_k|pa_k)}{\int \prod_k p(x_k|pa_k)dx_i} p(xi​∣x{j​=i}​)=∫p(x1​,...,xD​)dxi​p(x1​,...,xD​)​=∫∏k​p(xk​∣pak​)dxi​∏k​p(xk​∣pak​)​

8.3 Markov Random Fields

8.3.1 Conditional independence properties

8.3.2 Factorization properties

The joint distribution can be written as a product of potential functions $ps{i}_C(x_C)$ over the maximal cliques of the graph:

$$p(x)=\frac{1}{Z}\prod _C{\psi }_C(x_C)$$

$Z$ is called the partition function and it acts as a normalization constant and is given by:

$$Z=\sum _x\prod _C{\psi }_C(x_C)$$

The potential functions can be expressed by energy function:

ψ C ( x C ) = e x p { − E ( x C ) } \psi_{C}(x_{C}) = exp\{-E(x_{C})\} ψC​(xC​)=exp{−E(xC​)}

8.3.3 Illustration: Image de-noising

The complete energy function for the model takes the form:

E ( x , y ) = h ∑ i x i − β ∑ { i , j } x i x j − η ∑ i x i y i E(x,y)=h\sum_ix_i-\beta\sum_{\{i,j\}}x_ix_j-\eta\sum_ix_iy_i E(x,y)=hi∑​xi​−β{i,j}∑​xi​xj​−ηi∑​xi​yi​

which defines a joint distribution over $x$ and $y$ given by:

p ( x , y ) = 1 Z e x p { − E ( x , y ) } p(x,y)=\frac{1}{Z}exp\{-E(x,y)\} p(x,y)=Z1​exp{−E(x,y)}

8.3.4 Relation to directed graphs

8.4 Inference in Graphical Models

8.4.1 Inference on a chain

The joint distribution for a graph takes the form:

$$p(x)=\frac{1}{Z}{\psi }_{1,2}(x_1,x_2){\psi }_{x_2,x_3}(x_2,x_3)\dots {\psi }_{N-1,N}(x_{N-1},x_N)$$

where for the marginal distribution for a specific node $x_n$:

$$p(x_n)=\sum _{x_1}...\sum _{x_{n-1}}\sum _{x_{n+1}}...\sum _{x_N}p(x)$$

A more efficient algorithm by exploiting the conditional independece properties of the graphical model, the marginal expression can be expresed as:

$$\begin{array}{rl}p(x_n)=\frac{1}{Z}& \left[\sum _{x_n-1}{\psi }_{n-1,n}(x_{n-1},x_n)\dots \left[\sum _{x_2}{\psi }_{2,3}(x_2,x_3)\left[\sum _{x_1}{\psi }_{1,2}(x_1,x_2)\right]\right]\dots \right]\\ & \left[\sum _{x_{n+1}}{\psi }_{n,n+1}(x_n,x_{n+1})\dots \left[\sum _N{\psi }_{N-1,N}(x_{N-1},x_N)\right]\dots \right]\end{array}$$

We can see that the expression for the marginal $p(x_n)$ decomposes into the product of two factors times the normalization constant:

$$p(x_n)=\frac{1}{Z}{\mu }_{\alpha }(x_n){\mu }_{\beta }(x_n)$$

The message ${\mu }_{\alpha }(x_n)$ and $m{u}_{\beta }(x_n)$ can be evaluated recursively:

$${\mu }_{\alpha }(x_n)=\sum _{x_{n-1}}{\psi }_{n-1,n}(x_{n-1},x_n){\mu }_{\alpha }(x_{n-1})$$

$${\mu }_{\beta (x_n)}=\sum _{x_{n+1}}{\psi }_{n,n+1}(x_n,x_{n+1}){\mu }_{\beta }(x_{n+1})$$

We can calculate the joint distribution of the neighboring nodes:

$$p(x_{n-1},x_n)=\frac{1}{Z}{\mu }_{\alpha }(x_{n-1}){\psi }_{n-1,n}(x_{n-1},x_n){\mu }_{\beta }(x_n)$$

8.4.2 Trees

• Undirected tree
• Directed tree
• Polytree

8.4.3 Factor graphs

Factor graphs introduces additional nodes for the factors themselves in addition to the nodes representing the variables. To achieve this we write the joint distribution over a set of variables in the form of a product of factors:

$$p(x)=\prod _s{f}_s(x_s)$$

8.4.4 The sum-product algorithm

Our goal is to calculate the marginal for variable node $x$ and this marginal is given by the product of incoming messages along all of the links arriving at that node.

If a leaf node is a variable node, then the message taht it sends along its one and only link is given by:

$${\mu }_{x\to f}(x)=1$$

IF the leaf node is a factor node, the message sent should take the form:

$${\mu }_{f\to x}(x)=f(x)$$

We can compute recursively that:

$$\begin{gathered} {\mu }_{f_s\to x}(x)=\sum _{x_1}\cdots \sum _{x_M}{f}_s(x,x_1,\dots ,x_M)\prod _{m\in ne(f_s) \\ x}{\mu }_{x_m\to f_s}(x_m) \end{gathered}$$

$$\begin{gathered} {\mu }_{x_m\to f_s}(x_m)=\sum _{l\in ne(x_m) \\ {f}_s}{\mu }_{f_l\to x_m}(x_m) \end{gathered}$$

The marginal distribution of $p(x_i)$ will be:

$$p(x)=\prod _{s\in ne(x)}{\mu }_{f_s\to x}(x)$$

8.4.5 The max-sum algorithm

Our goal is to find a setting of the variables that has the largest probability and to find the value of that probability, we shall simply write out the max operator in terms of its components:

$$\underset{x}{\max }p(x)=\underset{x_1}{\max }\dots \underset{x_M}{\max }p(x)$$

$$p(x)=\prod _s{f}_s(x_s)$$

The max-sum algorithm in terms of message passing can be:

$${\mu }_{f\to x}(x)=\underset{x_1,\dots ,x_M}{\max }\left[\ln f(x,x_1,\dots ,x_M)+\sum _{m\in ne(f)/x}{\mu }_{x_m\to f}(x_m)\right]$$

$${\mu }_{x\to f}(x)=\sum _{l\in ne(x)/f}{\mu }_{f_l\to x}(x)$$

The initial message sent by the leaf nodes are given by:

$${\mu }_{x\to f}(x)=0$$

$${\mu }_{f\to x}(x)=\ln f(x)$$

while at the root node the maximum probability can then be computed, ay analogy in max-sum algorithm, with using:

$$p^{max}=\underset{x}{\max }\left[\sum _{s\in ne(x)}{\mu }_{f_s\to x}(x)\right]$$

8.4.6 Exact inference in general graphs

8.4.7 Loopy belief propagation

8.4.8 Learning the graph structure

From a Bayesian viewpoint, we would like to compute a posterior distribution over graph structures and to make predictions by averaging with respect to this distribution.

$$p(m|D)\propto p(m)p(D|m)$$