指数族分布和变分推断

指数族分布

1. 指数族分布的pdf / pmf可以表示成：

$$p(x|\eta )=h(x)exp(T(x{)}^T\eta -A(\eta ))$$

其中，$、T(x)、h(x)$只是包含$x$的函数, $A(\eta )$是只包含$\eta$的函数。$T(x)$叫做sufficient statistics。$A(\eta )$叫做log-normalizer。在变分推断中，$A(\eta )$起到很重要的作用。
$$\begin{gathered} \frac{\int h(x)exp(T(x{)}^T\eta )dx}{exp(A(\eta ))}=1 \\ A(\eta )=log\int h(x)exp(T(X{)}^T\eta )dx \end{gathered}$$
​

2. 我们学到的很多分布都是指数族分布，比如：

Normal, beta, Poisson, gamma, Bernoulli, chi-squared, geometric, exponential, categorical…

3. 举高斯分布为例子

$$p(x|\theta )=p(x|\mu ,{\sigma }^2)=N(\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi }\sigma }exp-\frac{(x-\mu {)}^2}{2{\sigma }^2}$$

4. 例子：怎样把高斯分布写成指数族分布的形式，就是怎样把均值和方差这两个参数替换成${\eta }_1,{\eta }_2$。

$$\begin{array}{rl}N(x|\mu ，{\sigma }^2)& =(2\pi {\sigma }^2{)}^{-\frac{1}{2}}exp(-\frac{(x-\mu {)}^2}{2{\sigma }^2})\\ & =exp(-\frac{x^2-2x\mu +{\mu }^2}{2{\sigma }^2}-\frac{1}{2}ln(2\pi {\sigma }^2)\\ & =exp(-\frac{1}{2{\sigma }^2}{x}^2+\frac{\mu }{{\sigma }^2}x-\frac{{\mu }^2}{2{\sigma }^2}-\frac{1}{2}ln(2\pi {\sigma }^2))\\ & =exp({\left[\begin{array}{c}x\\ x^2\end{array}\right]}^T\left[\begin{array}{c}\frac{\mu }{{\sigma }^2}\\ -\frac{1}{2{\sigma }^2}\end{array}\right]-\frac{{\mu }^2}{2{\sigma }^2}-\frac{1}{2}ln(2\pi {\sigma }^2))\end{array}$$

这里，我们得到：
$$\begin{array}{r}T(x)=\left[\begin{array}{c}x\\ x^2\end{array}\right]\\ \eta =\left[\begin{array}{c}{\eta }_1\\ {\eta }_2\end{array}\right]=\left[\begin{array}{c}\frac{\mu }{{\sigma }^2}\\ -\frac{1}{2{\sigma }^2}\end{array}\right]\\ \theta =\left[\begin{array}{c}\mu \\ {\sigma }^2\end{array}\right]=\left[\begin{array}{c}\frac{-{\eta }_1}{2{\eta }_2}\\ \frac{-1}{2{\eta }_2}\end{array}\right]\\ A(\eta )=\frac{-{\eta }_1^2}{4{\eta }_2}-\frac{1}{2}ln(-2{\eta }_2)\end{array}$$
所以均值和方差可以表示为：
$$\begin{gathered} {\eta }_2=-\frac{1}{2{\sigma }^2}\Rightarrow {\sigma }^2=-\frac{1}{2{\eta }_2} \\ \mu ={\eta }_1{\sigma }^2={\eta }_1\frac{-1}{2{\eta }_2}=-\frac{{\eta }_1}{2{\eta }_2} \end{gathered}$$

5. 指数族分布有什么好处呢？

• 如果一个条件概率可以写成上面的形式，很多问题的求解变得简单。
• 比如：求解$\underset{\theta }{\mathrm{argmax}}[logp(X|\eta )]$：

$$\begin{array}{rl}\underset{\eta }{\mathrm{argmax}}[logp(X|\eta )]& =\underset{\eta }{\mathrm{argmax}}[log\prod _{i=1}^{N}p(x_i|\eta )]\\ & =\underset{\eta }{\mathrm{argmax}}\sum _{i=1}^N[logh(x_i)+T(x_i{)}^T\eta -A(\eta )]\\ & =\underset{\eta }{\mathrm{argmax}}\sum _{i=1}^{N}T(x_i{)}^T\eta -NA(\eta )\end{array}$$

令上式为$L(\eta )$，则
$$\frac{\partial L(\eta )}{\partial \eta }=\sum _{i=1}^{N}T(x_i)-N{A}^{\prime }(\eta )=0$$
即：
$$A^{\prime }(\eta )=\frac{{\sum }_{i=1}^{N}T(x_i)}{N}$$

6. 共轭：

$$p(\beta |x)\propto p(x|\beta )p(\beta )$$

如果似然函数和先验是共轭的，则后验和先验是同一种分布。

如果似然函数是指数族分布，理论上一定可以找到一个与之共轭的先验分布（也是指数族分布）。

7. 一个结论：$A_l^{\prime }(\beta )=E_{p(x|\beta )}[T(x)]$

证明：
$$\begin{gathered} p(x|\beta )=h(x)exp(T(x{)}^T\beta -A_l(\beta )) \\ \because \int p(x|\beta )dx=1 \\ \begin{array}{rl}\therefore \frac{\partial \int p(x|\beta )dx}{\partial \beta }& =\frac{\partial \int h(x)exp(T(x{)}^T\beta -A_l(\beta ))dx}{\partial \beta }=0\\ & ={\int }_x\frac{\partial [h(x)exp[T(x{)}^T\beta -A_l(\beta )]}{\partial \beta }dx\\ & ={\int }_{x}h(x)exp[T(x{)}^T\beta -A_l(\beta )](T(x)-A_l^{\prime }(\beta ))dx\\ & ={\int }_{x}h(x)exp[T(x{)}^T\beta -A_l(\beta )]T(x)dx-{\int }_{x}h(x)exp[T(x{)}^T\beta -A_l(\beta )]A_l^{\prime }(\beta ))dx\\ & =E_{p(x|\beta )}[T(x)]-A_l^{\prime }(\beta )=0\end{array} \end{gathered}$$

8. 数据集合$X$，隐变量集合$Z$，参数集合$\beta$。

后验概率分布：
$$\begin{array}{rl}p(\beta ,Z|X)& =p(\beta |Z,X)p(Z|X)\\ & =p(Z|\beta ,X)p(\beta |X)\end{array}$$
$p(\beta |Z,X)$和$p(Z|\beta ,X)$，这两个后验分布都是指数族分布。

则：
$$p(\beta |Z,X)=h(\beta )exp(T(\beta {)}^T\eta (Z,X)-A_l(\eta (Z,X)))$$
在做变分推断时，希望用函数$q(\beta |\lambda )$去近似$p(\beta |Z,X)$，即：
$$p(\beta |Z,X)\approx q(\beta |\lambda )=h(\beta )exp(T(\beta {)}^T\lambda -A_g(\lambda ))$$
接下来，就要不断地调整$\lambda$，使得$q(\beta |\lambda )$越来越接近于$p(\beta |Z,X)$，即增大$ELOB$函数。

同样的，对于$p(Z|\beta ,X)$也是如此：
$$\begin{array}{rl}p(Z|\beta ,X)& =h(Z)exp(T(Z{)}^T\eta (\beta ,X)-A_l(\eta (\beta ,X)))\\ & \approx q(Z|\varphi )=h(Z)exp(T(Z{)}^T\varphi -A_g(\varphi ))\end{array}$$
$ELOB$函数如下：
$$L(q)=E_{q(Z,\beta )}[logp(X,Z,\beta )]-E_{q(Z,\beta )}[logq(Z,\beta )]$$
现在，$ELOB$函数可以写成：
$$L(\lambda ,\varphi )=E_{q(Z,\beta )}[logP(X,Z,\beta )]-E_{q(Z,\beta )}[logq(Z,\beta )]$$
目标：找到一个$\lambda$和$\varphi$，使得$ELOB$函数最大化。

方法：

• 先固定一个参数，对另一个参数优化

具体做法：

• 固定$\varphi$，优化$\lambda$

• $$\begin{array}{rl}L(\lambda ,\varphi )& =E_{q(Z,\beta )}[logp(X,Z,\beta )]-E_{q(Z,\beta )}[logq(Z,\beta )]\\ & =E_{q(Z,\beta )}[logp(\beta |X,Z)+logp(Z|X)]-E_{q(Z,\beta )}[logq(\beta )]-E_{q(Z,\beta )}[logq(Z)]\\ & =E_{q(Z,\beta )}[logp(\beta |X,Z)]-E_{q(Z,\beta )}[logq(\beta |\lambda )]\end{array}$$

• 将$p(\beta |Z,X)$和$q(\beta |\lambda )$代入上式

• $$\begin{array}{rl}L(\lambda ,\varphi )& =E_{q(Z,\beta )}[logh(\beta )]+E_{q(Z,\beta )}[T(\beta {)}^T\eta (Z,X)]-E_{q(Z,\beta )}[A_g(\eta (X,Z))]-E_{q(Z,\beta )}[logh(\beta )]-E_{q(Z,\beta )}[(T(\beta {)}^T\lambda )]+E_{q(Z,\beta )}[A_g(\lambda )]\\ & =E_{q(\beta )}[T(\beta {)}^T]\cdot E_{q(Z)}[\eta (Z,X)]-E_{q(Z)}[A_g(\eta (X,Z))]-E_{q(\beta )}[(T(\beta {)}^T\lambda )]+A_g(\lambda )\\ & =A_g^{\prime }(\lambda {)}^T{E}_{q(Z)}[\eta (Z,X)]-\lambda A_g^{\prime }(\lambda {)}^T+A_g(\lambda )\end{array}$$

• 上式对$\lambda$求导

• $$\begin{array}{rl}\frac{\partial L(\lambda ,\varphi )}{\partial \lambda }& =A_g^{\prime \prime }(\lambda {)}^T\cdot E_{q(Z)}[\eta (Z,X)]-A_g^{\prime }(\lambda {)}^T-\lambda A_g^{\prime \prime }(\lambda {)}^T+A_g^{\prime }(\lambda )\\ & =A_g^{\prime \prime }(\lambda {)}^T(E_{q(Z)}[\eta (Z,X)]-\lambda )=0\end{array}$$

• 如果$A_g^{\prime \prime }(\lambda {)}^T̸=0$，则
  $$\lambda =E_{q(Z|\varphi )}[\eta (Z,X)]$$
  同样
  $$\varphi =E_{q(\beta |\lambda )}[\eta (X,\beta )]$$