Beta分布&二项分布, 狄利克雷分布&多项式分布

共轭分布

  由贝叶斯定理，我们知道
$$p(\theta |x)\propto p(x|\theta )p(\theta )$$
如果$\Theta$的先验分布$p(\theta )$和后验分布$p(\theta |x)$属于同一分布族，那么就称先验分布$p(\theta )$和后验分布$p(\theta |x)$为共轭分布，同时，也称$p(\theta )$为似然函数$p(x|\theta )$的共轭先验分布。

Beta分布是二项分布的共轭先验分布

在$n$次独立重复试验中，每次试验结果只有两种，发生和不发生，发生概率为$p$，$n$次试验中发生的次数$X$服从二项分布$X\sim B(n,p)$：
$$P(X=k)=C_n^k{p}^k(1-p{)}^{n-k}$$
Beta分布$X\sim Be(\alpha ,\beta )$：
$$f(x)=\frac{1}{B(\alpha ,\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1},x\in [0,1],\alpha ,\beta >0$$
$$\frac{1}{B(\alpha ,\beta )}=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )+\Gamma (\beta )},\Gamma (z)={\int }_0^{\infty }{t}^{z-1}{e}^{-t}dt$$
$$\Gamma (z+1)=z\Gamma (z),\Gamma (1)=1$$
Beta分布的期望：
$$\begin{array}{rl}E[X]& =\int x\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )+\Gamma (\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1}dx\\ & =\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )+\Gamma (\beta )}\int x^{\alpha }(1-x{)}^{\beta -1}dx\\ & =\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )+\Gamma (\beta )}\frac{\Gamma (\alpha +1)+\Gamma (\beta )}{\Gamma (\alpha +\beta +1)}\\ & =\frac{\alpha }{\alpha +\beta }\end{array}$$

  假设先验分布$\Theta \sim Be(\alpha ,\beta )$：
$$p(\theta )=\frac{1}{B(\alpha ,\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$$
似然函数$X|\Theta \sim B(n,\theta )$：
$$p(X=k|\Theta =\theta )=C_n^k{\theta }^k(1-\theta {)}^{n-k}$$
则后验概率$\Theta |X=k\sim Be(\alpha +k,\beta +n-k)$：
$$\begin{array}{rl}p(X=k|\Theta =\theta )p(\theta )& =C_n^k{\theta }^k(1-\theta {)}^{n-k}\frac{1}{B(\alpha ,\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}\\ & =C_n^k\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )+\Gamma (\beta )}{\theta }^{\alpha +k-1}(1-\theta {)}^{\beta +n-k-1}\\ & =C{\theta }^{\alpha +k-1}(1-\theta {)}^{\beta +n-k-1}\end{array}$$
$$\begin{array}{rl}p(X=k)& =\int p(X=k|\Theta =\theta )p(\theta )d\theta \\ & =C\int {\theta }^{\alpha +k-1}(1-\theta {)}^{\beta +n-k-1}d\theta \\ & =CB(\alpha +k,\beta +n-k)\end{array}$$
$$p(\theta |X=k)=\frac{p(X=k|\Theta =\theta )p(\theta )}{p(X=k)}=\frac{1}{B(\alpha +k,\beta +n-k)}{\theta }^{\alpha +k-1}(1-\theta {)}^{\beta +n-k-1}$$

狄利克雷分布是多项式分布的共轭先验分布

在$n$次独立重复试验中，每次试验结果有$k$个：$A_1,...,A_k$，每个结果出现的概率为$p_1,...,p_k$，$n$次独立重复试验中每个结果出现的次数$X_1,...,X_k$服从多项式分布$X\sim multi(X_1,...,X_k;p_1,...,p_k)$：
$$P(X_1=n_1,...,X_k=n_k)=\frac{n!}{n_1!...n_k!}\prod _{i=1}^k{p}_i^{n_i}$$
$$\sum _{i=1}^k{p}_i=1,p_i>0$$
狄利克雷分布$X\sim Dir(X_1,...,X_k;{\alpha }_1,...,{\alpha }_k)$：
$$f(x_1,..,x_k)=\frac{1}{B({\alpha }_1,...,{\alpha }_k)}\prod _{i=1}^k{x}_i^{{\alpha }_i-1}$$
$$B({\alpha }_1,...,{\alpha }_k)=\frac{{\prod }_{i=1}^k\Gamma ({\alpha }_i)}{\Gamma ({\sum }_{i=1}^k{\alpha }_i)},\sum _{i=1}^k{x}_i=1,{\alpha }_i>0\forall i$$
狄利克雷分布的期望：
$$\begin{array}{rl}E[X_j]& =\int x_j\frac{1}{B({\alpha }_1,...,{\alpha }_k)}\prod _{i=1}^k{x}_i^{{\alpha }_i-1}d{x}_1...d{x}_k\\ & =\frac{1}{B({\alpha }_1,...,{\alpha }_k)}\int x_j^{{\alpha }_j}d{x}_j\prod _{i\ne j}^k\int x_i^{{\alpha }_i-1}d{x}_i\\ & =\frac{B({\alpha }_1,...,{\alpha }_j+1,...,{\alpha }_k)}{B({\alpha }_1,...,{\alpha }_j,...,{\alpha }_k)}\\ & =\frac{{\alpha }_j}{{\sum }_{i=1}^k{\alpha }_i}\end{array}$$

  假设先验分布${\Theta }_1,...,{\Theta }_k\sim Dir({\alpha }_1,...,{\alpha }_k)$：
$$p({\theta }_1,..,{\theta }_k)=\frac{1}{B({\alpha }_1,...,{\alpha }_k)}\prod _{i=1}^k{\theta }_i^{{\alpha }_i-1}$$
似然函数$X_1,...,X_k|{\Theta }_1,...,{\Theta }_k\sim multi({\theta }_1,..,{\theta }_k)$：
$$p(n_1,...,n_k|{\theta }_1,..,{\theta }_k)=\frac{n!}{n_1!...n_k!}\prod _{i=1}^k{\theta }_i^{n_i}$$
则后验概率${\Theta }_1,...,{\Theta }_k|X_1=n_1,...,X_k=n_k\sim Dir({\alpha }_1+n_1,...,{\alpha }_k+n_k)$：
$$\begin{array}{rl}p(n_1,...,n_k|{\theta }_1,..,{\theta }_k)p({\theta }_1,..,{\theta }_k)& =\frac{n!}{n_1!...n_k!}\prod _{i=1}^k{\theta }_i^{n_i}\frac{{\prod }_{i=1}^k\Gamma ({\alpha }_i)}{\Gamma ({\sum }_{i=1}^k{\alpha }_i)}\prod _{i=1}^k{\theta }_i^{{\alpha }_i-1}\\ & =\frac{n!}{n_1!...n_k!}\frac{{\prod }_{i=1}^k\Gamma ({\alpha }_i)}{\Gamma ({\sum }_{i=1}^k{\alpha }_i)}\prod _{i=1}^k{\theta }_i^{{\alpha }_i+n_i-1}\\ & =C\prod _{i=1}^k{\theta }_i^{{\alpha }_i+n_i-1}\end{array}$$
$$\begin{array}{rl}p(n_1,...,n_k)& =\int p(n_1,...,n_k|{\theta }_1,..,{\theta }_k)p({\theta }_1,..,{\theta }_k)d\theta \\ & =C\int \prod _{i=1}^k{\theta }_i^{{\alpha }_i+n_i-1}d\theta \\ & =CB({\alpha }_1+n_1,...,{\alpha }_k+n_k)\end{array}$$
$$\begin{array}{rl}p({\theta }_1,..,{\theta }_k|n_1,...,n_k)& =\frac{p(n_1,...,n_k|{\theta }_1,..,{\theta }_k)p({\theta }_1,..,{\theta }_k)}{p(n_1,...,n_k)}\\ & =\frac{1}{B({\alpha }_1+n_1,...,{\alpha }_k+n_k)}\prod _{i=1}^k{\theta }_i^{{\alpha }_i+n_i-1}\end{array}$$