本文深入探讨了概率论的基础——贝叶斯估计，通过一个具体的例子展示了如何找到最优估计，并证明了相关性的概念如何在重构质量中发挥作用。同时，分析了互信息在噪声存在时的性质，揭示了噪声如何影响两个信号之间的信息共享。
文章目录
1-Bayesian Estimation （P317）
Suppose that x = θ + ν w h e r e ν i s a n N ( 0 , σ ) random variable and θ is the value of a n N ( θ 0 , σ 0 ) random variable θ (Fig. 8-7). Find the bayesian estimate θ o f θ . \begin{aligned}&\text{Suppose that x}=\theta+\nu\mathrm{~where~}\nu\mathrm{~is~an~}N(0,\sigma)\text{ random variable and }\theta\text{ is the value of}\\&\mathrm{an~}N(\theta_0,\sigma_0)\text{ random variable }\theta\text{ (Fig. 8-7). Find the bayesian estimate }\theta\mathrm{~of~}\theta.\end{aligned} Suppose that x=θ+ν where ν is an N(0,σ) random variable and θ is the value ofan N(θ0,σ0) random variable θ (Fig. 8-7). Find the bayesian estimate θ of θ.
条件：
f v ( v ) ∼ e − v 2 / 2 σ 2 f θ ( θ ) ∼ e − ( θ − θ 0 ) 2 / 2 σ 0 2 \mathrm{f}_{\mathbf{v}}(\mathbf{v})\sim e^{-\mathbf{v}^{2}/2\sigma^{2}}\quad\mathrm{f}_{\theta}(\theta)\sim e^{-(\theta-\theta_{0})^{2}/2\sigma_{0}^{2}} fv(v)∼e−v2/2σ2fθ(θ)∼e−(θ−θ0)2/2σ02
证明：
f θ ( θ ∣ x ) ∼ e − ( θ − θ 1 ) 2 / 2 σ 1 2 \mathrm{f}_{\theta}(\theta|\mathrm{x})\sim e^{-(\theta-\theta_{1})^{2}/2\sigma_{1}{}^{2}} fθ(θ∣x)∼e−(θ−θ1)2/2σ12，where 1 σ 1 2 ≡ 1 σ 0 2 + n σ 2 θ 1 ≡ σ 1 2 σ 0 2 θ 0 + n σ 1 2 σ 2 x ˉ \frac{1}{
{\sigma_{1}}^{2}}\equiv\frac{1}{
{\sigma_{0}}^{2}}+\frac{\mathrm{n}}{\sigma^{2}}\quad\theta_{1}\equiv\frac{
{\sigma_{1}}^{2}}{
{\sigma_{0}}^{2}}\:\theta_{0}+\frac{\mathrm{n}{\sigma_{1}}^{2}}{\sigma^{2}}\:\bar{\mathrm{x}} σ121≡σ021+σ2nθ1≡σ02σ12θ0+σ2nσ12xˉ
Proof
似然函数，观测值的条件分布为：
随机变量 v = x − θ v = x - \theta v=x−θ的分布与 ν \nu ν相同， f x ( x ∣ θ ) f_x(x|\theta) fx(x∣θ)可以等价地表示为 f v ( x − θ ) f_v(x-\theta) fv(x−θ)，这表明给定 θ \theta θ时 x x x的概率密度函数与 v v v的概率密度函数相同（ v v v是 x − θ x - \theta x−θ的函数）。
f x ( x ∣ θ ) = f v ( x − θ ) ∼ exp { − ( x − θ ) 2 2 σ 2 } \mathrm{f}_{\mathrm{x}}(\mathrm{x}|\theta)\:=\:\mathrm{f}_{\mathrm{v}}(\mathrm{x}-\theta)\:\sim\:\exp\:\left\{\:-\:\frac{(\mathrm{x}-\theta)^{2}}{2\sigma^{2}}\:\right\} fx(x∣θ)=fv(x−θ)∼exp{
−2σ2(x−θ)2}
观测值的全体（概率相乘，指数相加）
f ( X ∣ θ ) ∼ e x p { − 1 2 σ 2 ∑ ( x i − θ ) 2 } \mathrm{f(X|\theta)\:\sim\:exp\:\left\{\:-\:\frac{1}{2\sigma^{2}}\:\sum\:(x_{i}-\theta)^{2}\:\right\}} f(X∣θ)∼exp{
−2σ21∑(xi−θ)2}
利用 ∑ ( x i − θ ) 2 = ∑ ( x i − x ˉ ) 2 + \sum(x_{\mathrm{i}}-\theta)^{2}=\sum(x_{\mathrm{i}}-\bar{x})^{2}+ ∑(xi−θ)2=∑(xi−xˉ)2+ n ( x i − θ ) 2 (x_{\mathrm{i}}-\theta)^{2} (xi−θ)2，
f ( X ∣ θ ) ∼ exp { − 1 2 σ 2 [ ∑ ( x i − x ˉ ) 2 + n ( x i − θ ) 2 ] } = exp { − 1 2 σ 2 ∑ ( x i − x ˉ ) 2 } × exp { − n 2 σ 2 ( θ − x ˉ ) 2 } \begin{aligned} \mathrm{f}(\mathrm{X}|\theta) & \sim\exp\left\{-\frac{1}{2\sigma^{2}}\left[\sum(x_{i}-\bar{x})^{2}+\mathrm{n}(x_{i}-\theta)^{2}\right]\right\} \\ &=\exp\left\{-\frac{1}{2\sigma^{2}}\sum(x_{i}-\bar{x})^{2}\right\}\times\exp\left\{-\frac{\mathrm{n}}{2\sigma^{2}}(\theta-\bar{x})^{2}\right\} \end{aligned} f(X∣θ)∼exp{
−2σ21[∑(xi−xˉ)2+n(xi−θ)2]}=exp{
−2σ21∑(xi−xˉ)2}×exp{
−2σ2n(θ−xˉ)2}
利用贝叶斯估计 f θ ( θ ∣ X ) = f ( X ∣ θ ) f ( X ) f θ ( θ ) f_\theta(\theta\mid X)=\frac{f(X\mid\theta)}{f(X)}f_\theta(\theta) fθ(θ∣X)=f(X)f(X∣θ)fθ(θ)
f ( θ ∣ X ) ∝ f ( X ∣ θ ) × f θ ( θ ) ∝ exp { − 1 2 σ 2 ∑ ( x i − x ˉ ) 2 } × exp { − n 2 σ 2 ( θ − x ˉ ) 2 } × exp { − ( θ − θ 0 ) 2 2 σ 0 2 } \begin{aligned} \mathrm{f}(\theta|\mathrm{X})& \propto\mathrm{f(X|\theta)}\times\mathrm{f}_{\theta}(\theta) \\ &\propto\exp\left\{-\frac{1}{2\sigma^{2}}\sum(x_{i}-\bar{x})^{2}\right\}\times\exp\left\{-\frac{\mathrm{n}}{2\sigma^{2}}(\theta-\bar{x})^{2}\right\}\times\exp\left\{-\frac{(\theta-\theta_{0})^{2}}{2\sigma_{0}^{2}}\right\} \end{aligned} f(θ∣X)∝f(X∣θ)×fθ(θ)∝exp{
−2σ21∑(xi−xˉ)2}×exp{
−2σ2n(θ−xˉ)2}×exp{
−2σ02(θ−θ0)2}
证毕：
f ( θ ∣ X ) ∼ exp { − 1 2 [ ( θ − θ o ) 2 σ o 2 + n ( x ˉ − θ ) 2 σ 2 ] } \mathrm{f(\theta|X)\sim\exp\left\{-\frac{1}{2}\left[\frac{(\theta-\theta_{o})^{2}}{\sigma_{o}^{2}}+\frac{n(\bar{x}-\theta)^{2}}{\sigma^{2}}\right]\right\}} f(θ∣X)∼exp{
−21[σo2(θ−θo)2+σ2n(xˉ−θ)2]}
接下来，就是凑完全平方，以 θ \theta θ为自变量
我们可以识别出新的均值 θ 1 \theta_{1} θ1 和方差 σ 1 2 \sigma_{1}^{2} σ12。我们注意到指数项中的二次项是 ( θ − θ 1 ) 2 (\theta-\theta_{1})^{2} (θ−θ1)2 的形式，这表明我们需要找到 θ 1 \theta_{1} θ1 使得二次项变为完全平方的形式。
我们需要先证明（需要记住结论）：
以均值替换采样
∑ i = 1 n ( x i − θ ) 2 = ∑ i = 1 n [ ( x i − x ˉ ) 2 + ( x ˉ − θ ) 2 ] \sum_{i=1}^n (x_i - \theta)^2 = \sum_{i=1}^n [(x_i - \bar{x})^2 + (\bar{x} - \theta)^2] ∑i=1n(xi−θ)2=∑i=1n[(xi−xˉ)2+(xˉ−θ)2]，
其中 x ˉ = 1 n ∑ i = 1 n x i \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i xˉ=n1∑i=1nxi
Sub Proof：
展开原始表达式：
∑ i = 1 n ( x i − θ ) 2 \sum_{i=1}^n (x_i - \theta)^2 ∑i=1n(xi−θ)2
关键步骤1：
( x i − θ ) 2 = ( x i − x ˉ + x ˉ − θ ) 2 (x_i - \theta)^2 = (x_i - \bar{x} + \bar{x} - \theta)^2 (xi−θ)2=(xi−xˉ+xˉ−θ)2
( x i − x ˉ + x ˉ − θ ) 2 = ( x i − x ˉ ) 2 + 2 ( x i − x ˉ ) ( x ˉ − θ ) + ( x ˉ − θ ) 2 (x_i - \bar{x} + \bar{x} - \theta)^2 = (x_i - \bar{x})^2 + 2(x_i - \bar{x})(\bar{x} - \theta) + (\bar{x} - \theta)^2 (xi−xˉ+xˉ−θ)2=(xi−xˉ)2+2(xi−xˉ)(xˉ−θ)+(xˉ−θ)2
展开：
∑ i = 1 n ( x i − θ ) 2 = ∑ i = 1 n [ ( x i − x ˉ ) 2 + 2 ( x i − x ˉ ) ( x ˉ − θ ) + ( x ˉ − θ ) 2 ] \sum_{i=1}^n (x_i - \theta)^2 = \sum_{i=1}^n \left[ (x_i - \bar{x})^2 + 2(x_i - \bar{x})(\bar{x} - \theta) + (\bar{x} - \theta)^2 \right] ∑i=1n(xi−θ)2=∑i=1n[(xi−xˉ)2+2(xi−xˉ)(xˉ−θ)+(xˉ−θ)2]
∑ i = 1 n ( x i − θ ) 2 = ∑ i = 1 n ( x i − x ˉ ) 2 + ∑ i = 1 n 2 ( x i − x ˉ ) ( x ˉ − θ ) + ∑ i = 1 n ( x ˉ − θ ) 2 \sum_{i=1}^n (x_i - \theta)^2 = \sum_{i=1}^n (x_i - \bar{x})^2 + \sum_{i=1}^n 2(x_i - \bar{x})(\bar{x} - \theta) + \sum_{i=1}^n (\bar{x} - \theta)^2 ∑i=1n(xi−θ)2=∑i=1n(xi−xˉ)2+∑i=1n2(xi−xˉ)(xˉ−θ)+∑i=1n(xˉ−θ)2
关键步骤2（常数提到求和符号外面，再根据均值的定义）：
∑ i = 1 n 2 ( x i − x ˉ ) ( x ˉ − θ ) = 2 ( x ˉ − θ ) ∑ i = 1 n ( x i − x ˉ ) \sum_{i=1}^n 2(x_i - \bar{x})(\bar{x} - \theta) = 2(\bar{x} - \theta) \sum_{i=1}^n (x_i - \bar{x}) ∑i=1n2(xi−xˉ)(xˉ−θ)=2(xˉ−θ)∑i=1n(xi−xˉ)
2 ( x ˉ − θ ) ∑ i = 1 n ( x i − x ˉ ) = 2 ( x ˉ − θ ) ⋅ 0 = 0 2(\bar{x} - \theta) \sum_{i=1}^n (x_i - \bar{x}) = 2(\bar{x} - \theta) \cdot 0 = 0 2(xˉ−θ)∑i=1n(xi−xˉ)=2(xˉ−θ)⋅0=0
∑ i = 1 n ( x ˉ − θ ) 2
最低0.47元/天 解锁文章
扫一扫
4786
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
