Exponential family of distributions

Choi H. I. Lecture 4: Exponential family of distributions and generalized linear model (GLM).

定义

定义: 一个分布具有如下形式的密度函数:
$$f_{\theta }(x)=\frac{1}{Z(\theta )}h(x)e^{⟨T(x),\theta ⟩},$$
则该分布属于指数族分布.
其中$x\in {\mathbb{R}}^m$, $T(x)=(T_1(x),T_2(x),\cdots ,T_k(x))\in {\mathbb{R}}^k$, $\theta =({\theta }_1,{\theta }_2,\cdots ,{\theta }_k)$为未知参数, $Z(\theta )=\int h(x)e^{⟨T(x),\theta ⟩}\mathrm{d}x$为配平常数.

若令$C(x)=\log h(x)$, $A(\theta )=\log Z(\theta )$, 则
$$f_{\theta }(x)=\exp (⟨T(x),\theta ⟩-A(\theta )+C(x)).$$
指数族分布还有一种更一般的形式:
$$f_{\theta }(x)=\exp (\frac{⟨T(x),\theta ⟩-A(\theta )}{\varphi }+C(x,\varphi )),$$
更甚者
$$f_{\theta }(x)=\exp (\frac{⟨T(x),\lambda (\theta )⟩-A(\theta )}{\varphi }+C(x,\varphi )),$$
$\varphi$控制分布的形状.

性质

$A(\theta )$

Proposition 1:
$${\nabla }_{\theta }A(\theta )=\int f_{\theta }(x)T(x)\mathrm{d}x=\mathbb{E}[T(X)].$$

proof:

已知:
$$\int f_{\theta }(x)\mathrm{d}x=\int \exp (\frac{⟨T(x),\theta ⟩-A(\theta )}{\varphi }+C(x,\varphi ))\mathrm{d}x=1.$$
两边关于$\theta$求梯度得:
$$\int f_{\theta }(x)\frac{T(x)-{\nabla }_{\theta }A(\theta )}{\varphi }\mathrm{d}x=0\Rightarrow {\nabla }_{\theta }A(\theta )=\mathbb{E}[T(X)].$$

Proposition 2:
$$D_{\theta }^{2}A=(\frac{{\partial }^{2}A}{\partial {\theta }_i\partial {\theta }_j})=\frac{1}{\varphi }\mathrm{C}\mathrm{o}\mathrm{v}(T(X),T(X))=\frac{1}{\varphi }Cov(T(X)).$$

proof:

$$\frac{\partial A}{\partial {\theta }_i}=\int \exp (\frac{⟨T(x),\theta ⟩-A(\theta )}{\varphi }+C(x,\varphi ))T_i(x)\mathrm{d}x.$$

$$\begin{array}{ll}\frac{{\partial }^{2}A}{\partial {\theta }_i\partial {\theta }_j}& =\int f_{\theta }(x)\frac{T_j(x)-\frac{\partial A}{\partial {\theta }_j}}{\varphi }{T}_i(x)\mathrm{d}x\\ & =\frac{1}{\varphi }\int f_{\theta }(x)(T_j(x)-\frac{\partial A}{\partial {\theta }_j})(T_i(x)-\frac{\partial A}{\partial {\theta }_i})\mathrm{d}x\\ & =\mathrm{C}\mathrm{o}\mathrm{v}(T_i(X),T_j(X)).\end{array}$$

Corollary 1: $A(\theta )$关于$\theta$是凸函数.

既然其黑塞矩阵半正定.

极大似然估计

设有 { x i } i = 1 n \{x^i\}_{i=1}^n {xi}i=1n​个样本, 则对数似然函数为
$$l(\theta )=\frac{1}{\theta }[⟨\theta ,\sum _{i=1}^{n}T(x^i)-nA(\theta )]+\sum _{i=1}^{n}C(x^i,\varphi ),$$
因为$A(\theta )$是凸函数, 所以上述存在最小值点, 且
$${\nabla }_{\theta }l(\theta )=\frac{1}{\varphi }[\sum _{i=1}^{n}T(x^i)-n{\nabla }_{\theta }A(\theta )],$$
故该最小值点在
$${\nabla }_{\theta }A(\theta )=\frac{1}{n}\sum _{i=1}^{n}T(x^i),$$
处达到.

最大熵

最大熵原理-科学空间

指数族分布实际上满足最大熵分布, 这是在没有任何偏爱的尺度下的分布.

即
$$\underset{f}{\max }H(f)=-\int f(x)\log f(x)\mathrm{d}x.$$

等价于最小化
$$\underset{f}{\min }\int f(x)\log f(x)\mathrm{d}x.$$

往往, 我们会有一些已知的统计信息, 通常以期望的形式表示:
$$\int f(x)h_i(x)\mathrm{d}x=c_i,i=1,2\cdots ,s.$$
则我们的目标实际上是:
$$\begin{gathered} \underset{f}{\min }\int f(x)\log f(x)\mathrm{d}x \\ \mathrm{s}.\mathrm{t}.\int f(x)h_i(x)\mathrm{d}x=c_i,i=0,2\cdots ,s. \end{gathered}$$
其中$h_0=1,c_0=1$, 即密度函数需满足$\int f(x)\mathrm{d}x=1$.

利用拉格朗日乘数得:
$$J(f,\lambda )=\int f(x)\log f(x)\mathrm{d}x+{\lambda }_0(1-\int f(x)\mathrm{d}x)+\sum _{i=1}^s{\lambda }_i[c_i-\int f(x)h_i(x)\mathrm{d}x].$$

最优条件, $J$关于$f$的变分为0, 即
$$1+\log f(x)-{\lambda }_0-\sum _{i=1}^s{\lambda }_i{h}_i(x)=0.$$
即
$$f(x)=\frac{1}{Z}\exp (\sum _{i=1}^s{\lambda }_i{h}_i(x)).$$

属于指数分布族.

例子

Bernoulli

$$P(x)=p^x(1-p{)}^{1-x}=\exp [x\log \frac{p}{1-p}+\log (1-p)].$$

$$\begin{gathered} \theta =\log \frac{p}{1-p}, \\ T(x)=x, \\ A(\theta )=\log (1+e^{\theta }), \\ h(x)=0. \end{gathered}$$

指数分布

$$p(x)=\lambda \cdot e^{-\lambda x}=\exp [-\lambda x+\log \lambda ],x\ge 0.$$

$$\begin{gathered} \theta =\lambda , \\ T(x)=-x, \\ A(\theta )=\log \frac{1}{\lambda }, \\ h(x)=\mathbb{I}(x\ge 0). \end{gathered}$$

正态分布

$$p(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp [-\frac{(x-\mu {)}^2}{2{\sigma }^2}].$$

$\sigma$视作已知参数:
$$p(x)=\exp [\frac{-\frac{1}{2}{x}^2+x\mu -\frac{1}{2}{\mu }^2}{{\sigma }^2}-\frac{1}{2}\log (2\pi {\sigma }^2)].$$

$$\begin{gathered} \theta =(\mu ,1), \\ T(x)=(x,-\frac{1}{2}{x}^2), \\ \varphi ={\sigma }^2, \\ A(\theta )=\frac{1}{2}{\mu }^2, \\ C(x,\varphi )=\frac{1}{2}\log (2\pi {\sigma }^2). \end{gathered}$$

$\sigma$视作未知参数:
$$p(x)=\exp [-\frac{1}{2{\sigma }^2}{y}^2+\frac{\mu }{{\sigma }^2}x-\frac{1}{2{\sigma }^2}{\mu }^2-\log \sigma -\frac{1}{2}\log 2\pi ].$$

$$\begin{gathered} T(x)=(x,\frac{1}{2}{x}^2), \\ \theta =(\frac{\mu }{{\sigma }^2},-\frac{1}{{\sigma }^2}), \\ A(\theta )=\frac{{\mu }^2}{2{\sigma }^2}+\log \sigma , \\ C(x)=-\frac{1}{2}\log (2\pi ). \end{gathered}$$