线性高斯模型中互信息与MMSE关系

个人博客：www.qiuyun-blog.cn

Notations:

• Mutual information (MI)
  $$\begin{gathered} I(X;Y)=\int p(x,y)\log \frac{p(x|y)}{p(x)}\text{d}x\text{d}y \\ =\int p(x,y)\log \frac{p(x,y)}{p(x)p(y)}\text{d}x\text{d}y \\ =\int p(x,y)\log \frac{p(y|x)}{p(y)}\text{d}x\text{d}y \\ =I(Y;X) \end{gathered}$$
• integration by parts
  $$\int u(x)v^{\prime }(x)\text{d}x=u(x)v(x){|}_{x=-\infty }^{x=+\infty }-\int u^{\prime }(x)v(x)\text{d}x$$
  where $v^{\prime }(x)$ denotes $\frac{\text{d}v(x)}{\text{d}x}$。

Theorem：Given following linear Gaussian model
$$Y=\sqrt{\gamma }X+UU\sim \mathcal{N}(0,1)$$where $\gamma >0$ refers to signal-noise-rate (SNR). We have
$$\frac{\text{d}I(X;Y)}{\text{d}\gamma }=\frac{1}{2}\text{MMSE}$$ in which
$$\text{MMSE}=\int (x-\hat{x}{)}^{2}p(x,y;\gamma )\text{d}x\text{d}y$$
and $\hat{x}=\int xp(x|y;\gamma )\text{d}x$。

$Proof$：Define
$$p_k(y;\gamma )=\int x^{k}p(y,x;\gamma )\text{d}x={\mathbb{E}}_X\left\{X^{k}p(y|X;\gamma )\right\}$$we have follows conclusions
1.
$$\begin{gathered} \frac{\text{d}{p}_k(y;\gamma )}{\text{d}\gamma }=\frac{1}{2\sqrt{\gamma }}y{p}_{k+1}(y;\gamma )-\frac{1}{2}{p}_{k+2}(y;\gamma ) \\ =-2\sqrt{\gamma } \end{gathered}$$