模式识别 | PRML Chapter 2 Probability Distributions

PRML Chapter 2 Probability Distributions

2.1 Binary Variables

• bernoulli distribution: $Bern(x|\mu )={\mu }^x(1-\mu {)}^{1-x}$
• binomial distribution: $Bin(m|N,\mu )=\frac{N!}{(N-m)!m!}{\mu }^m(1-\mu {)}^{N-m}$

2.1.1 The beta distribution

To get the MLE solution in a bayesian perspective, we need a prior distribution. Beta is a common one:

$$Beta(\mu |a,b)=\frac{\Gamma (a+b)}{\Gamma (a)\Gamma (b)}{\mu }^{a-1}(1-\mu {)}^{b-1}$$

Where $E[\mu ]=\frac{a}{a+b}$ and $var[\mu ]=\frac{ab}{(a+b{)}^2(a+b+1)}$. We can get the posterior distribution:

$$p(\mu |m,l,a,b)=\frac{\Gamma (m+a+l+b)}{\Gamma (m+a)\Gamma (l+b)}{\mu }^{m+a-1}(1-\mu {)}^{l+b-1}$$

For prediction, we need to estimate the distribution of $x$ given the condition that the training data has known.

$$p(x=1|D)={\int }_0^{1}p(x=1|\mu )p(\mu |D)d\mu ={\int }_0^1\mu p(\mu |D)d\mu =E[\mu |D]$$

So we have:

$$p(x=1|D)=\frac{m+a}{m+a+l+b}$$

On average, the more data we observe, the uncertainty of the posterior possibility will continuous decreasing.

$$E_{\theta }(\theta )=E_D[E_{\theta }[\theta |D]]$$

$$va{r}_{\theta }[\theta ]=E_D[va{r}_{\theta }[\theta |D]]+va{r}_D[E_{\theta }[\theta |D]]$$

2.2 Multinomial Variables

If we want to discribe a variable more than two states with binary variables, we can use the form like $x=(0,0,\dots ,1,\dots ,0{)}^T$. If we use ${\mu }_k$ to represent the probability of $x_k=1$ and we have N independent dataset, the likelihood function will be:

$$p(D|\mu )=\prod _{k=1}^K{\mu }_k^{m_k},m_k=\sum _n{x}_{nk}$$

Using a Lagrange multiplier and let the partial derivative to find the MLE solution for $\mu$:

$$\sum _{k=1}^K{m}_{k}ln{\mu }_k+\lambda (\sum _{k=1}^K{\mu }_k-1)$$

$${\mu }_k^{ML}=\frac{m_k}{N}$$

Now consider the conditional joint distribution, it is called multinomial distribution (here ${\sum }_{k=1}^K{m}_k=N$):

$$Mult(m_1,m_2,\dots ,m_K|\mu ,N)=\frac{N!}{m_1!m_2!\dots m_K!}$$

2.2.1 The Dirichlet Distribution

The prior distribution of parameter ${\mu }_k$ is Dirichlet distribution:

$$DIr(\mu |\alpha )=\frac{\Gamma ({\alpha }_0)}{\Gamma ({\alpha }_1)\dots \Gamma ({\alpha }_K)}\prod _{k=1}^K{\mu }_k^{{\alpha }_{k-1}}$$

where ${\alpha }_0={\sum }_{k=1}^K{\alpha }_k$ and it is easy to get the posterior distribution:

$$p(\mu |D,\alpha )=Dir(\mu |\alpha +m)$$

2.3 The Gaussian Distribution

The geometric form of Guassian distribution: The exponent is a quadratic form:

$${\Delta }^2=(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )$$

$\Sigma$ can be symmetric and from the eigenvector equation $\Sigma {\mu }_i={\lambda }_i{\mu }_i$, when we choose the eigenvertor to be orthogonal, we will have:

$$\Sigma =\sum _{i=1}^D{\lambda }_i{\mu }_i{\mu }_i^T$$

So the quadratic form can be written as:

$${\Delta }^2=\sum _{i=1}^D\frac{y_i^2}{{\lambda }_i},y_i={\mu }_i^T(x-\mu )$$

Consider the newly defined coordinate system by $y_i$, the form of Guassian distribution will be:

p ( y ) = p ( x ) ∣ J ∣ = ∏ j = 1 D 1 ( 2 π λ j ) 1 2 e x p { − y j 2 2 λ j } ,    J i j = ∂ x i ∂ y j = U i j p(y) = p(x)|J| = \prod_{j=1}^{D}\frac{1}{(2\pi\lambda_{j})^{\frac{1}{2}}}exp\{-\frac{y_{j}^{2}}{2\lambda_{j}}\},\ \ J_{ij} = \frac{\partial x_{i}}{\partial y_{j}} = U_{ij} p(y)=p(x)∣J∣=j=1∏D​(2πλj​)21​1​exp{−2λj​yj2​​},  Jij​=∂yj​∂xi​​=Uij​

2.3.1 Conditional Gaussian distributions

Consider multivariate normal distribution, suppose we have:

$$x=\left(\begin{array}{r}{x}_a\\ x_b\end{array}\right),\mu =\left(\begin{array}{r}{\mu }_a\\ {\mu }_b\end{array}\right),\Sigma =\left(\begin{array}{rl}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right)$$

and we introduce the precision matrix:

$${\Sigma }^{-1}=\Lambda =\left(\begin{array}{rl}{\Lambda }_{aa}& {\Lambda }_{ab}\\ {\Lambda }_{ba}& {\Lambda }_{bb}\end{array}\right)$$

To find an expression for the conditional distribution $p(x_a\Vert x_b)$, we obtain:

$$\begin{gathered} -\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )=-\frac{1}{2}(x_a-{\mu }_a{)}^T{\Lambda }_{aa}(x_a-{\mu }_a)-\frac{1}{2}(x_a-{\mu }_a{)}^T{\Lambda }_{ab}(x_b-{\mu }_b) \\ -\frac{1}{2}(x_b-{\mu }_b{)}^T{\Lambda }_{ba}(x_a-{\mu }_a)-\frac{1}{2}(x_b-{\mu }_b{)}^T{\Lambda }_{bb}(x_b-{\mu }_b) \end{gathered}$$

which is the exponential term of conditional Gaussian. Also, it is easy to know that:
$$-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )=-\frac{1}{2}{x}^T{\Sigma }^{-1}x+x^T{\Sigma }^{-1}\mu +const$$

Apply this method to $p(x_a\Vert x_b)$ and $x_b$ is regarded as a constant. Compare the second order term $-\frac{1}{2}{x}_a^T{\Lambda }_{aa}{x}_a$ and the linear terms $x_a^T({\Lambda }_{aa}{\mu }_a-{\Lambda }_{ab}(x_b-{\mu }_b))$ in $x_a$, we can get the variance and mean:

$${\Sigma }_{a|b}={\Lambda }_{aa}^{-1}$$

$$\begin{gathered} {\mu }_{a|b}={\Sigma }_{a|b}({\Lambda }_{aa}{\mu }_a-{\Lambda }_{ab}(x_b-{\mu }_b)) \\ ={\mu }_a-{\Lambda }_{aa}^{-1}{\Lambda }_{ab}(x_b-{\mu }_b) \end{gathered}$$

2.3.2 Marginal Gaussian distributions

To prove the marginal distribution $p(x_a)=\int p(x_a,x_b)d{x}_b$ is Gaussian, the method is similar to 2.3.1, the results are:

$${\Sigma }_a=({\Lambda }_{aa}-{\Lambda }_{ab}{\Lambda }_{bb}^{-1}{\Lambda }_{ba}{)}^{-1}$$

$${\mu }_a={\Sigma }_a({\Lambda }_{aa}-{\Lambda }_{ab}{\Lambda }_{bb}^{-1}{\Lambda }_{ba}){\mu }_a$$

2.3.3 Bayes’ theorem for Gaussian variables

Suppose we are given a Gaussian marginal distribution $p(x)$ and a Gaussian conditional distribution $p(y\Vert x)$ (linear):
$$p(x)=N(x|\mu ,{\Lambda }^{-1})$$

$$p(y|x)=N(y|Ax+b,L^{-1})$$

we wish to find the marginal distribution $p(y)$ and the conditional distribution $p(x\Vert y)$. Consider the log of the joint distribution:

$$\begin{array}{rl}lnp(z)& =lnp(x)+lnp(y|x)\\ & =-\frac{1}{2}(x-\mu {)}^T\Lambda (x-\mu )-\frac{1}{2}(y-Ax-b{)}^{T}L(y-Ax-b)+C\end{array}$$

By comparing quadratic forms we can get the mean and covariance of the joint distribution:

$$cov[z]=R^{-1}=\left(\begin{array}{rl}{\Lambda }^{-1}& {\Lambda }^{-1}{A}^T\\ A{\Lambda }^{-1}& L^{-1}+A{\Lambda }^{-1}{A}^T\end{array}\right)$$

$$E[z]=\left(\begin{array}{r}\mu \\ A\mu +b\end{array}\right)$$

And for the marginal distribution $p(y)$, it is easy to observe that

$$E[y]=A\mu +b$$

$$cov[y]=L^{-1}+A{\Lambda }^{-1}{A}^T$$

Similarly, for conditional distribution, we have:
E [ x ∣ y ] = ( Λ + A T L A ) − 1 { A T L ( y − b ) + a μ } E[x | y] = (\Lambda + A^{T}LA)^{-1}\{A^{T}L(y-b) + a\mu \} E[x∣y]=(Λ+ATLA)−1{ATL(y−b)+aμ}

$$cov[x|y]=(\Lambda +A^{T}LA{)}^{-1}$$

2.3.4 Maximum likelihood for the Gaussian

This part we only show the result:
$${\mu }_{ML}=\frac{1}{N}\sum _{n=1}^N{x}_n$$

$${\Sigma }_{ML}=\frac{1}{N}\sum _{n=1}^N(x_n-{\mu }_{ML})(x_n-{\mu }_{ML}{)}^T$$

2.3.5 Sequential estimation

Sequential methods allow data points to be processed one at a time and then discarded. Refer to 2.3.4, the contribution from the final data point $x_N$ is:

$$\begin{array}{rl}{\mu }_{ML}^{(N)}& =\frac{1}{N}\sum _{n=1}^N{x}_n\\ & ={\mu }_{ML}^{(N-1)}+\frac{1}{N}(x_N-{\mu }_{ML}^{(N-1)})\end{array}$$

A more general formulation pair of sequential learning is called Robbins-Monro algorithm. For random variables $\theta$ and $z$ governed by joint distribution $p(z,\theta )$. The conditional expenctation of $z$ given $\theta$ is

$$f(\theta )=E[z|\theta ]=\int zp(z|\theta )dz$$

$f(\theta )$ is called regression functions. Our goal is to find ${\theta }^{\ast }$ that $f({\theta }^{\ast })=0$. Suppose we observe values of $z$ and we with to find a corresponding sequential estimation scheme for ${\theta }^{\ast }$. Assume that $
E[(z-f)^{2}][\theta] < \infty %]]>$, the sequence of successive estimation will be:

$${\theta }^{(N)}={\theta }^{(N-1)}-{\alpha }_{N-1}z({\theta }^{(N-1)})$$

2.3.6 Bayesian inference for the Gaussian

Estimate $\mu$ ($\sigma$ known):
The likelihood function of Gaussian distribution:

$$p(x|\mu )=\prod _{n=1}^{N}p(x_n|\mu )=\frac{1}{2\pi {\sigma }^2}exp\left\{-\frac{1}{2{\sigma }^2}\sum _{n=1}^N(x_n-{\mu }^2)\right\}$$

Choose $p(\mu )$ as Gaussian distribution: $p(\mu )=N(\mu |{\mu }_0,{\sigma }_0^2)$, and the posterior distribution is given by:

$$p(\mu |x)\propto p(x|\mu )p(\mu )$$

Consequently, the posterior distribution will be:

$$p(\mu |x)=N(\mu |{\mu }_N,{\sigma }_N^2)$$

where

$${\mu }_N=\frac{{\sigma }^2}{N{\sigma }_0^2+{\sigma }^2}{\mu }_0+\frac{N{\sigma }_0^2}{N{\sigma }_0^2+{\sigma }^2}{\mu }_{ML},{\mu }_{ML}=\frac{1}{N}\sum _{n=1}^N{x}_n$$

$$\frac{1}{{\sigma }_N^2}=\frac{1}{{\sigma }_0^2}+\frac{N}{{\sigma }^2}$$

The Bayesian paradigm leads very naturally to a sequential view of the inference problem.

$$p(\mu |x)\propto \left[p(\mu )\prod _{n=1}^{N}p(x_n|\mu )\right]p(x_N|\mu )$$

Estimate $\sigma$ ($\mu$ known):
The likelihood function for $\lambda$ takes the form:

$$p(x|\lambda )=\prod _{n=1}^{N}N(x_n|\mu ,{\lambda }^{-1})\propto {\lambda }^{\frac{1}{2}}exp\left\{-\frac{\lambda }{2}\sum _{n=1}^N(x_n-\mu {)}^2\right\}$$

The prior distribution that we choose is Gamma distribution:

$$Gam(\lambda |a,,b)=\frac{1}{\Gamma (a)}{b}^a{\lambda }^{(a-1)}exp(-b\lambda )$$

The posterior distribution is given by:

$$p(\lambda |x)\propto {\lambda }^{a_0-1}{\lambda }^{\frac{N}{2}}exp\left\{-b_0\lambda -\frac{\lambda }{2}\sum _{n=1}^N(x_n-\mu {)}^2\right\}$$

Consequently, the posterior distribution will be $Gam(\lambda |a_N,b_N)$, where

$$a_N=a_0+\frac{N}{2}$$

$$b_N=b_0+\frac{1}{2}\sum _{n=1}^N(x_n-\mu {)}^2=b_0+\frac{N}{2}{\sigma }_{ML}^2$$

$\mu$ and $\sigma$ are unknown:

The form of prior distribution we choose is:

$$p(\mu ,\lambda )=N(\mu |{\mu }_0,(\beta \lambda {)}^{-1})Gam(\lambda |a,b)$$

Where ${\mu }_0=\frac{c}{\beta }$, $a=\frac{1+\beta }{2}$ and $b=d-\frac{c^2}{2\beta }$.

2.3.7 Student’s t-distribution

If we have a gaussian distribution $N(x|\mu ,{\tau }^{-1})$ and a Gamma Prior $Gam(\tau |a,b)$. Calculate the integral of $\tau$, we can get the marginal distribution of $x$:

$$p(x|\mu ,a,b)={\int }_0^{\infty }N(x|\mu ,{\tau }^{-1})Gam(\tau |a,b)d\tau$$

Replace the parameters by $v=2a$ and $\lambda =\frac{a}{b}$, we can get the Student’s t-distribution:

$$St(x|\mu ,\lambda ,\nu )=\frac{\Gamma (\frac{\nu }{2}+\frac{1}{2})}{\Gamma (\frac{\nu }{2})}{\left(\frac{\lambda }{\pi \nu }\right)}^{\frac{1}{2}}{\left[1+\frac{\lambda (x-\mu {)}^2}{\nu }\right]}^{-\frac{\nu }{2}-\frac{1}{2}}$$

2.3.8 Periodic variables

2.3.9 Mixtures of Gaussians

Taking linear combinations of basic distributions such as Gaussian, almost any continuous density can be approximated to arbitrary accuracy.

$$p(x)=\sum _{k=1}^K{\pi }_{k}N(x|{\mu }_k,{\Sigma }_k)$$

2.4 The Exponential Family

The distribution of variable $x$ with parameter $\eta$ can be defined as:

p ( x ∣ η ) = h ( x ) g ( η ) e x p { η T u ( x ) } p(x | \eta) = h(x)g(\eta)exp\{\eta^{T}u(x)\} p(x∣η)=h(x)g(η)exp{ηTu(x)}

And the formula satisfies:

g ( η ) ∫ h ( x ) e x p { η T u ( x ) } d x = 1 g(\eta)\int h(x)exp\{\eta^{T}u(x)\} dx = 1 g(η)∫h(x)exp{ηTu(x)}dx=1

2.4.1 Maximum likelihood and sufficient statistics

2.4.2 Conjugate priors

For any member of the exponential family, there exists a conjugate a conjugate prior that can be written in the form:

$$p(\eta |\chi ,\nu )=f(\chi ,\nu )g(\eta {)}^{\nu }exp(\nu {\eta }^T\chi )$$

2.4.3 Noninformative priors

Two simple examples of noninformative priors:

The location parameter: $p(x|\mu )=f(x-\mu )$

The scale parameter: $p(x|\sigma )=\frac{1}{\sigma }f(\frac{x}{\sigma })$

2.5 Nonparametric Methods