预测分析笔记chapter 4

课程笔记：预测分析 2021Spring

参考教材：Murphy, K. P. (2021). Probabilistic Machine Learning: An Introduction. MIT press.

In this class,we’ll cover topics in machine learning from a probabilistic view.

We will also introduce some topics in statistical computing,such as EM,MCMC,varaitional inference,some optimization algorithm.

Chapter 4 Bayesian statistics 贝叶斯统计初步

$$p(\mathbf{\theta }\mid \mathcal{D})=\frac{p(\mathbf{\theta })p(\mathcal{D}\mid \mathbf{\theta })}{p(\mathcal{D})}=\frac{p(\mathbf{\theta })p(\mathcal{D}\mid \mathbf{\theta })}{\int p\left({\mathbf{\theta }}^{\prime }\right)p\left(\mathcal{D}\mid {\mathbf{\theta }}^{\prime }\right)d{\mathbf{\theta }}^{\prime }}$$

$p(\mathcal{D})$: marginal likelihood/normalizing constant.Do not depend on $\theta$.数据的边际分布，与参数值/参数的分布无关。

We will contain the following parts:

• summarize posterior 后验
• posterior computation with conjugate prior
• Bayesian mdel comparision
• approximate posterior

summarizing the posterior 后验

point estimates点估计

eg:posterior mean,posterior median

credible interval区间估计

measure confidence in our paremeter estimates.(small sample size or low quality of data may lead to larger uncertainty)

we use $100(1-\alpha )\%$ credible interval,which contains $1-\alpha$ of the posterior probability mass.
$$P(l\le \theta \le \mu |D)=1-\alpha$$

posterior computation with conjugate prior基于共轭先验计算后验

conjugate prior共轭先验

In this section, we consider a set of (prior, likelihood) pairs for which we can compute the posterior in closed form.

在这一章中我们主要考虑后验分布具有显式解的这一部分。

In particular, we will use priors that are “conjugate” to the likelihood.

We say that a prior $p(\mathbf{\theta })\in \mathcal{F}$ is a conjugate prior for a likelihood function $p(\mathcal{D}\mid \mathbf{\theta })$ if the posterior is in the same parameterized family as the prior, i.e., $p(\mathbf{\theta }\mid \mathcal{D})\in \mathcal{F}$.

“共轭”是指先验分布和后验分布均属于同一分布族。

In other words, $\mathcal{F}$ is closed under Bayesian updating. If the family $\mathcal{F}$ corresponds to the exponential family , then the computations can be performed in closed form.

分布族关于Bayesian更新是封闭的。

The Dirichlet-multinomial model：Y-Cat，先验：狄利克雷分布

• 似然函数：

Let $Y\sim \mathrm{Cat}(\mathbf{\theta })$ be a discrete random variable drawn from a categorical distribution. The likelihood has the form
$$p(\mathcal{D}\mid \mathbf{\theta })=\prod _{n=1}^N\mathrm{Cat}\left(y_n\mid \mathbf{\theta }\right)=\prod _{n=1}^N\prod _{c=1}^C{\theta }_c^{\mathrm{I}\left(y_n=c\right)}=\prod _{c=1}^C{\theta }_c^{N_c}$$
where $N_c={\sum }_n\mathbb{I}\left(y_n=c\right)$：#face c appears

eg:有6面的骰子（C=6，N组数据），$N_c$代表在N个数据中第c面出现的个数

• 先验分布：

The conjugate prior for a categorical distribution is the Dirichlet distribution, which is a multivariate generalization of the beta distribution. This has support over the probability simplex, defined by
$$S_K=\left\{\mathbf{\theta }:0\le {\theta }_k\le 1,\sum _{k=1}^K{\theta }_k=1\right\}$$

The pdf of the Dirichlet is defined as follows:
$$\mathrm{Dir}(\mathbf{\theta }\mid \breve{\mathbf{\alpha }})\triangleq \frac{1}{B(\breve{\mathbf{\alpha }})}\prod _{k=1}^K{\theta }_k^{{\breve{\alpha }}_k-1}\mathbb{I}\left(\mathbf{\theta }\in S_K\right)$$
where $B(\vec{\alpha })$ is the multivariate beta function,
$$B(\breve{\mathbf{\alpha }})\triangleq \frac{{\prod }_{k=1}^K\Gamma \left({\breve{\alpha }}_k\right)}{\Gamma \left({\sum }_{k=1}^K{\breve{\alpha }}_k\right)}$$
$\breve{\mathbf{\alpha }}$:是超参数，是给定的，用来推断参数的后验分布

狄利克雷分布是beta分布的推广，用来模拟（多元？）概率密度的分布，beta是1个概率密度的分布。

• 后验分布：

We can combine the multinomial likelihood and Dirichlet prior to compute the posterior, as follows:
$$\begin{array}{rl}p(\mathbf{\theta }\mid \mathcal{D})& \propto p(\mathcal{D}\mid \mathbf{\theta })\mathrm{Dir}(\mathbf{\theta }\mid \tilde{\mathbf{\alpha }})\\ & =\left[\prod _k{\theta }_k^{N_k}\right]\left[\prod _k{\theta }_k^{{\breve{\alpha }}_k-1}\right]\\ & =\mathrm{Dir}\left(\mathbf{\theta }\mid {\breve{\alpha }}_1+N_1,\dots ,{\breve{\alpha }}_K+N_K\right)\\ & =\mathrm{Dir}(\mathbf{\theta }\mid \hat{\mathbf{\alpha }})\end{array}$$
where ${\hat{\alpha }}_k={\breve{\alpha }}_k+N_k$ are the parameters of the posterior.

The Gaussian-Gaussian model：Y-正态，先验：逆Gamma

只讨论当$\mu$已知时，$\Sigma$的分布。

似然函数：

If $\mu$ is a known constant, the likelihood for ${\sigma }^2$ has the form
$$p\left(\mathcal{D}\mid {\sigma }^2\right)\propto {\left({\sigma }^2\right)}^{-N/2}\exp \left(-\frac{1}{2{\sigma }^2}\sum _{n=1}^N{\left(y_n-\mu \right)}^2\right)$$
先验分布：

where we can no longer ignore the $1/\left({\sigma }^2\right)$ term in front. The standard conjugate prior is the inverse Gamma distribution , given by 逆Gamma分布（=1/Gamma分布）
$$\mathrm{I}\mathrm{G}\left({\sigma }^2\mid \breve{a},\breve{b}\right)=\frac{{\breve{b}}^{\breve{a}}}{\Gamma (\breve{a})}{\left({\sigma }^2\right)}^{-(\breve{a}+1)}\exp \left(-\frac{\breve{b}}{{\sigma }^2}\right)$$

后验分布：
$$\begin{array}{rl}p({\mathbf{\sigma }}^2\mid \mathcal{D})& \propto p(\mathcal{D}\mid {\mathbf{\sigma }}^2)p({\mathbf{\sigma }}^2\mid \breve{\mathbf{\alpha }},\breve{\mathbf{b}})\\ & =({\sigma }^2{)}^{-(\frac{N}{2}+\breve{\mathbf{\alpha }}+1)}exp(-\frac{\breve{\mathbf{b}}+\frac{1}{2}{\sum }_{n=1}^N(y_n-\mu {)}^2}{{\sigma }^2})\end{array}$$
i.e.$IG(\frac{N}{2}+\breve{\mathbf{\alpha }},\breve{\mathbf{b}}+\frac{1}{2}{\sum }_{n=1}^N(y_n-\mu {)}^2)$

Multiplying the likelihood and the prior, we see that the posterior is also IG:
$$\begin{array}{rl}p\left({\sigma }^2\mid \mu ,\mathcal{D}\right)& =\mathrm{I}\mathrm{G}\left({\sigma }^2\mid \hat{a},\hat{b}\right)\\ \hat{a}& =\breve{a}+N/2\\ \hat{b}& =\tilde{b}+\frac{1}{2}\sum _{n=1}^N{\left(y_n-\mu \right)}^2\end{array}$$
Generally，we do not have closed form posterior, so we have to used approximate inference method.但在通常情况下，我们得不到有显式解的后验分布。所以通常使用渐进推断的方法。

Bayesian Model comparison 模型比较

All model are wrong , but some are useful——George Box.

所有模型都是不准确的，但是有些模型是有用的。

we assume we have a set of models $M$.

objective: we want to choose the best model from some set $M$.

选取：在当前数据下，出现概率最大的模型。
$$\hat{m}=\underset{m\in \mathcal{M}}{\mathrm{argmax}}p(m\mid \mathcal{D})$$
where
$$p(m\mid \mathcal{D})=\frac{p(\mathcal{D}\mid m)p(m)}{{\sum }_{m\in \mathcal{M}}p(\mathcal{D}\mid m)p(m)}$$
m:model

D:Data

$p(\mathcal{D}\mid m)$:当选定模型m之后的边际概率密度（相当于$P(\theta \mid D)=\frac{P(\theta )\cdot P(D\mid \theta )}{P(D)}$$中的p(\mathcal{D}$) )

If the prior over models is uniform, $p(m)=1/|\mathcal{M}|,$ then the MAP model is given by

在没有额外信息的情况下，我们认为每种模型的可能性相同
$$\hat{m}=\underset{m\in \mathcal{M}}{\mathrm{argmax}}p(\mathcal{D}\mid m)$$
选取：在选定某模型下，该组数据出现概率最大的模型

The quantity $p(\mathcal{D}\mid m)$ is given by
$$p(\mathcal{D}\mid m)=\int p(\mathcal{D}\mid \mathbf{\theta },m)p(\mathbf{\theta }\mid m)d\mathbf{\theta }$$

$\theta $为模型m中的参数

Bayes model averaging 贝叶斯模型平均

If our goal is to perform prediction, we can get better results if we marginalize out over all models, by computing
$$p(y\mid \mathbf{x},\mathcal{D})=\sum _{m\in \mathcal{M}}p(y\mid \mathbf{x},m)p(m\mid \mathcal{D})$$
D：training data，（x，y）：new data，goal：predict y

如果我们想要让预测效果较好，我们可以将所有模型加权平均。其中权重为 ：在该组数据下某模型出现的概率。

Disadvantage : computationally very expensive. 缺点：计算非常昂贵。

week3

Variational Inference 变分推断

objective: we want to approximate $P(z|x)$. 我们希望取估计在给定数据下，参数的后验分布

In variational inference , we pick a family of distribution over the lactect variable, parameter of interest,with its own variational parameters.

our goal is to find $q$ that minimize the distance between $q$ and $p(z|x)$.

• the measure of distance: KL divergence . written as $KL(q||p)$

  and $q=\underset{q}{\mathrm{argmin}}KL(q\Vert p)$

Some introduction about KL divergence

Entropy 熵

The entropy of a probability distribution can be interpreted as a measure of uncertainty, or lack of predictability.

For example, suppose we observe a sequence of symbols $x_n$ ∼ p generated from distribution $p$. If $p$ has high entropy, it will be hard to predict the value of each osbervation $x_n$.

• The entropy of a discrete random variable $X$ with distribution $p$ over $K$ states is defined by

$$\mathbb{H}(X)\triangleq -\sum _{k=1}^{K}p(X=k){\log }_{2}p(X=k)=-{\mathbb{E}}_X[\log p(X)]$$

The discrete distribution with maximum entropy is the uniform distribution.Hence for a $K$ -ary random variable, the entropy is maximized if $p(x=k)=1/K;$ in this case, $\mathbb{H}(X)={\log }_{2}K$.

Conversely, the distribution with minimum entropy (which is zero) is any delta-function that puts all its mass on one state. Such a distribution has no uncertainty.

• Differential entropy for continuous random variables

If $X$ is a continuous random variable with pdf $p(x)$, we define the differential entropy as
$$h(X)\triangleq -{\int }_{\mathcal{X}}dxp(x)\log p(x)$$
均匀分布的熵

assuming this integral exists. For example, suppose $X\sim U(0,a)$. Then
$$h(X)=-{\int }_0^{a}dx\frac{1}{a}\log \frac{1}{a}=\log a$$
Note that, unlike the discrete case, differential entropy can be negative. This is because pdf’s can be bigger than 1. For example if $X\sim U(0,1/8)$, we have $h(X)={\log }_2(1/8)=-3$.

高斯分布的熵

The entropy of a $d$ -dimensional Gaussian is
$$h(\mathcal{N}(\mathbf{\mu },\mathbf{\Sigma }))=\frac{1}{2}\ln |2\pi e\mathbf{\Sigma }|=\frac{1}{2}\ln \left[(2\pi e{)}^d|\mathbf{\Sigma }|\right]=\frac{d}{2}+\frac{d}{2}\ln (2\pi )+\frac{1}{2}\ln |\mathbf{\Sigma }|$$
In the $1\mathrm{d}$ case, this becomes
$$h\left(\mathcal{N}\left(\mu ,{\sigma }^2\right)\right)=\frac{1}{2}\ln \left[2\pi e{\sigma }^2\right]$$

Cross entropy交叉熵

The cross entropy between distribution $p$ and $q$ is defined by
$$\mathbb{H}(p,q)\triangleq -\sum _{k=1}^K{p}_k\log q_k$$
One can show that the cross entropy is the expected number of bits needed to compress some data samples drawn from distribution $p$ using a code based on distribution $q.$ This can be minimized by setting $q=p.

Relative entropy (KL divergence) KL距离

Given two distributions p and q, it is often useful to define a distance metric to measure how “close” or “similar” they are.

we consider a divergence measure D(p,q) ,which quantifies how far q is from p. and we focus on the Kullback-Leibler divergence.

• KL距离的定义：

For discrete distributions, the KL divergence is defined as follows:
$$\mathbb{K}\mathbb{L}(p\Vert q)\triangleq \sum _{k=1}^K{p}_k\log \frac{p_k}{q_k}$$
This naturally extends to continuous distributions as well:
$$\mathbb{K}\mathbb{L}(p\Vert q)\triangleq \int dxp(x)\log \frac{p(x)}{q(x)}$$
Do not distribute without permission from Kevin P. Murphy and MIT Press.

• Interpretation：

We can rewrite the KL as follows:
$$\mathbb{K}\mathbb{L}(p\Vert q)=\underset{-\mathrm{H}(p)}{\underbrace{\sum _{k=1}^K{p}_k\log p_k}}\underset{\mathbb{H}(p,q)}{\underbrace{-\sum _{k=1}^K{p}_k\log q_k}}$$
We recognize the first term as the negative entropy, and the second term as the cross entropy. Thus we can interpret the KL divergence as the “extra number of bits” you need to pay when compressing data samples from $p$ using the incorrect distribution $q$ as the basis of your coding scheme.

• For example, one can show that the KL divergence between two multivariate Gaussian distributions is given by

$$\begin{array}{l}\mathbb{K}\mathbb{L}\left(\mathcal{N}\left(\mathbf{x}\mid {\mathbf{\mu }}_1,{\mathbf{\Sigma }}_1\right)\Vert \mathcal{N}\left(\mathbf{x}\mid {\mathbf{\mu }}_2,{\mathbf{\Sigma }}_2\right)\right)\\ =\frac{1}{2}\left[\mathrm{tr}\left({\mathbf{\Sigma }}_2^{-1}{\mathbf{\Sigma }}_1\right)+{\left({\mathbf{\mu }}_2-{\mathbf{\mu }}_1\right)}^{\top }{\mathbf{\Sigma }}_2^{-1}\left({\mathbf{\mu }}_2-{\mathbf{\mu }}_1\right)-D+\log \left(\frac{\det \left({\mathbf{\Sigma }}_2\right)}{\det \left({\mathbf{\Sigma }}_1\right)}\right)\right]\end{array}$$
​ In the scalar case, this becomes
$$\mathbb{K}\mathbb{L}\left(\mathcal{N}\left(x\mid {\mu }_1,{\sigma }_1\right)\Vert \mathcal{N}\left(x\mid {\mu }_2,{\sigma }_2\right)\right)=\log \frac{{\sigma }_2}{{\sigma }_1}+\frac{{\sigma }_1^2+{\left({\mu }_1-{\mu }_2\right)}^2}{2{\sigma }_2^2}-\frac{1}{2}$$

• Theorem 6.2.1. (Information inequality) $\mathbb{K}\mathbb{L}(p\Vert q)\ge 0$ with equality iff $p=q$.

  Proof. We now prove the theorem following [CT06, p28]. Let A = { x : p ( x ) > 0 } A=\{x: p(x)>0\} A={x:p(x)>0} be the support of $p(x)$. Using the concavity of the log function and Jensen’s inequality, we have that
  $$\begin{array}{rl}-\mathbb{K}\mathbb{L}(p\Vert q)& =-\sum _{x\in A}p(x)\log \frac{p(x)}{q(x)}=\sum _{x\in A}p(x)\log \frac{q(x)}{p(x)}\\ & \le \log \sum _{x\in A}p(x)\frac{q(x)}{p(x)}-\log \sum _{x\in A}q(x)\\ & \le \log \sum _{x\in \mathcal{X}}q(x)=\log 1=0\end{array}$$

• Two properties of KL divergence:

​ nonnegative and nonsymmetric

非负性和不对称性

Variational inference

$$\mathbb{K}\mathbb{L}(q\Vert p)=E[q(x)\log \frac{q(x)}{p(z|x)}]$$

$q(z)$：a distribution that approximates $p(z|x)$ ,$q(z)$ is unknown.

week 4