机器学习中有关概率论的问题（均匀分布、β分布、多变量正态分布、伯努利分布、和连续分布的KL散度）

1、均匀分布的均值及方差

连续变量x 的均匀分布定义为

$$U(x|a,b)=\{\begin{array}{ll}\frac{1}{b-a},& x\in [a,b],\\ 0,& x\notin [a,b]\end{array}$$
求其均值和方差。

1.1 解答

$$E[x]={\int }_{-\infty }^{\infty }xf(x|a,b)\text{d}x=\frac{1}{b-a}{\int }_a^{b}x\text{d}x=\frac{a+b}{2}$$
$$E[x^2]={\int }_{-\infty }^{\infty }{x}^{2}f(x|a,b)\text{d}x=\frac{1}{b-a}{\int }_a^b{x}^2\text{d}x=\frac{a^2+ab+b^2}{3}$$
$$\mathrm{v}\mathrm{a}\mathrm{r}[x]=E[x^2]-E[x{]}^2=\frac{a^2+ab+b^2}{3}-(\frac{a+b}{2}{)}^2=\frac{(b-a{)}^2}{12}$$

2、β分布证归一性，求期望与方差

贝塔分布是定义在$(0,1)$ 区间上含两个参数的⼀类连续分布，参数为$(\alpha ,\beta )$ 的贝塔概率密度函数为
$$f(x|\alpha ,\beta )=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1},$$
试证

(a)归一性
$${\int }_0^{1}f(x|\alpha ,\beta )\mathrm{d}x=1$$

(b)期望
$$E[x]=\frac{\alpha }{\alpha +\beta }$$

$(c)$ ⽅差
$$\mathrm{v}\mathrm{a}\mathrm{r}[x]=\frac{\alpha \beta }{(\alpha +\beta {)}^2(\alpha +\beta +1)}$$

2.1 解答

(a) 贝塔函数$B(\alpha ,\beta )={\int }_0^1{x}^{\alpha -1}(1-x{)}^{\beta -1}\mathrm{d}x$与伽马函数$\Gamma \left(\alpha \right)={\int }_0^{\infty }{e}^{-x}{x}^{\alpha -1}\mathrm{d}x$的关系为
$$B\left(\alpha +\beta \right)=\frac{\Gamma (\alpha )\Gamma (\beta )}{\Gamma (\alpha +\beta )}$$
由此易证归⼀性。另外，贝塔概率密度函数可表⽰为
$$f(x|\alpha ,\beta )=\frac{1}{B(\alpha ,\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1}$$

(b) 贝塔分布的$n$ 阶矩为
$$\begin{array}{rl}E[x^n]& =\frac{1}{B(\alpha ,\beta )}{\int }_0^1{x}^n{x}^{\alpha -1}(1-x{)}^{\beta -1}\mathrm{d}x=\frac{B(n+\alpha ,\beta )}{B(\alpha ,\beta )}\\ & =\frac{\Gamma (n+\alpha )\Gamma (\beta )}{\Gamma (n+\alpha +\beta )}\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}=\frac{\Gamma (n+\alpha )}{\Gamma (n+\alpha +\beta )}\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )}\end{array}$$
令$n=1,2$，并结合伽马函数的性质$\Gamma (\alpha +1)=\alpha \Gamma (\alpha )$ 可得
$$E[x]=\frac{\Gamma (1+\alpha )}{\Gamma (1+\alpha +\beta )}\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )}=\frac{\alpha }{\alpha +\beta }$$
$$E[x^2]=\frac{\Gamma (2+\alpha )}{\Gamma (2+\alpha +\beta )}\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )}=\frac{\alpha (\alpha +1)}{(\alpha +\beta )(\alpha +\beta +1)}$$

$(c)$ 贝塔分布的⽅差为
$$\mathrm{v}\mathrm{a}\mathrm{r}[x]=E[x^2]-E[x{]}^2=\frac{\alpha (\alpha +1)}{(\alpha +\beta )(\alpha +\beta +1)}-\frac{{\alpha }^2}{(\alpha +\beta {)}^2}=\frac{\alpha \beta }{(\alpha +\beta {)}^2(\alpha +\beta +1)}$$

3、多变量正态分布证协方差的最大似然解及其期望

设$\mathbf{X}\sim N(\mu ,\mathbf{\Sigma }),\mu \in R^D,\mathbf{\Sigma }\in R^{D\ast D}$(多变量正态分布)，即
$$p(x|\mu ,\mathbf{\Sigma })=N(x|\mu ,\mathbf{\Sigma })=\frac{1}{(2\pi {)}^{\frac{D}{2}}{\left|\mathbf{\Sigma }\right|}^{\frac{1}{2}}}exp-\frac{1}{2}(x-\mu {)}^T{\mathbf{\Sigma }}^{-1}(x-\mu )$$
设数据集$D=(x_1,...,x_N)$是独⽴地从$N(\mu ,\mathbf{\Sigma })$ 中抽样⽽得到的。证明

(a) 期望的最⼤似然估计为
$${\mu }_{ML}=\frac{1}{N}\sum _{n=1}^N{x}_n$$

(b) 协⽅差的最⼤似然解为