线性高斯模型中互信息与MMSE关系

版权声明:本文为博主原创文章,遵循 CC 4.0 by-sa 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/u012284960/article/details/89363224

个人博客www.qiuyun-blog.cn

Notations:

  • Mutual information (MI)
    I(X;Y)=p(x,y)logp(xy)p(x)dxdy   =p(x,y)logp(x,y)p(x)p(y)dxdy   =p(x,y)logp(yx)p(y)dxdy =I(Y;X) \qquad \qquad I(X;Y) =\int p(x,y)\log \frac{p(x|y)}{p(x)}\text{d}x\text{d}y\\ \qquad \qquad \qquad \qquad \ \ \ =\int p(x,y)\log \frac{p(x,y)}{p(x)p(y)}\text{d}x\text{d}y\\ \qquad \qquad \qquad \quad \ \ \ =\int p(x,y)\log \frac{p(y|x)}{p(y)}\text{d}x\text{d}y\\ \ =I(Y;X)
  • integration by parts
    u(x)v(x)dx=u(x)v(x)x=x=+u(x)v(x)dx \int u(x)v'(x)\text{d}x= u(x)v(x)|_{x=-\infty}^{x=+\infty}-\int u'(x)v(x)\text{d}x
    where v(x)v'(x) denotes dv(x)dx\frac{\text{d}v(x)}{\text{d}x}

Theorem:Given following linear Gaussian model
Y=γX+UUN(0,1) Y=\sqrt{\gamma}X+U\quad U\sim \mathcal{N}(0,1)where γ>0\gamma>0 refers to signal-noise-rate (SNR). We have
dI(X;Y)dγ=12MMSE \frac{\text{d}I(X;Y)}{\text{d}\gamma}=\frac{1}{2}\text{MMSE} in which
MMSE=(xx^)2p(x,y;γ)dxdy\text{MMSE}=\int (x-\hat{x})^2p(x,y;\gamma)\text{d}x\text{d}y
and x^=xp(xy;γ)dx\hat{x}=\int xp(x|y;\gamma)\text{d}x

ProofProof:Define
pk(y;γ)=xkp(y,x;γ)dx=EX{Xkp(yX;γ)} p_k(y;\gamma)=\int x^k p(y,x;\gamma)\text{d}x=\mathbb{E}_X\left\{X^kp(y|X;\gamma)\right\} we have follows conclusions
1.
dpk(y;γ)dγ=12γypk+1(y;γ)12pk+2(y;γ)=12γddypk+1(y;γ) \frac{\text{d} p_k(y;\gamma)}{\text{d}\gamma} =\frac{1}{2\sqrt{\gamma} }yp_{k+1}(y;\gamma)-\frac{1}{2}p_{k+2}(y;\gamma)\\ =-\frac{1}{2\sqrt{\gamma} }\frac{\text{d} }{\text{d}y}p_{k+1}(y;\gamma)
2.
x^MMSE=xp(xy;γ)dx=p1(y;γ)p0(y;γ) \hat{x}_{\text{MMSE} }=\int xp(x|y;\gamma)\text{d}x=\frac{p_1(y;\gamma)}{p_0(y;\gamma)}

Mutual information
I(X;Y)=p(y,x;γ)logp(yx;γ)p(y;γ)dxdy=p(y,x;γ)logp(yx;γ)dxdyξp(y,x;γ)logp(y;γ)dxdyζ I(X;Y) =\int p(y,x;\gamma)\log \frac{p(y|x;\gamma)}{p(y;\gamma)}\text{d}x\text{d}y\\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad =\underbrace{\int p(y,x;\gamma)\log p(y|x;\gamma)\text{d}x\text{d}y}_{\xi}-\underbrace{\int p(y,x;\gamma)\log p(y;\gamma)\text{d}x\text{d}y}_{\zeta}For that,we calculate ξ\xi and ζ\zeta respectively as follows
ξ=p(yx;γ)p(x)logp(yx;γ)dxdy  =(a)12p(yx;γ)p(x)log2πdxdy12(yγx)2p(yx;γ)p(x)dxdy=12log(2πe)  \xi =\int p(y|x;\gamma)p(x)\log p(y|x;\gamma)\text{d}x\text{d}y\\ \quad \qquad \qquad \qquad \qquad \qquad \qquad \quad \ \ \overset{(a)}=-\frac{1}{2}\int p(y|x;\gamma)p(x)\log 2\pi \text{d}x\text{d}y-\frac{1}{2}(y-\sqrt{\gamma}x)^2p(y|x;\gamma)p(x)\text{d}x\text{d}y\\ =-\frac{1}{2}\log (2\pi e) \quad \qquad \qquad \qquad \quad \ where the fact p(yx;γ)=12πexp[(yγx)22]p(y|x;\gamma)=\frac{1}{\sqrt{2\pi} }\exp \left[-\frac{(y-\sqrt{\gamma}x)^2}{2}\right] is used in (a)(a).
ζ=p(y,x;γ)logp(y;γ)dy=yxp(y,x;γ)dxlogp(y;γ)dy=yp(y;γ)logp(y;γ)dy    \zeta =\int p(y,x;\gamma)\log p(y;\gamma)\text{d}y\\ \quad \qquad =\int_y\int_x p(y,x;\gamma)\text{d}x\log p(y;\gamma)\text{d}y\\ =\int_y p(y;\gamma)\log p(y;\gamma)\text{d}y \ \ \ Computing the partial derivation of I(X;Y)I(X;Y) w.r.t. γ\gamma yields
dI(X;Y)dγ=ddγp0(y;γ)logp0(y;γ)dy  =[logp0(y;γ)+1]dp1(y;γ)dγdy =12γlogp0(y;γ)dp1(y;γ)dydy+12γdp1(y;γ)dydyκ  =(a)12γlogp0(y;γ)dp1(y;γ)dydy =(b)12γp1(y;γ)p0(y;γ)dp0(y;γ)dydy  =(c)12γp1(y;γ)p0(y;γ)[yγp1(y;γ)p0(y;γ)]p0(y;γ)dy \frac{\text{d}I(X;Y)}{\text{d}\gamma} =-\frac{\text{d} }{\text{d}\gamma}p_0(y;\gamma)\log p_0(y;\gamma)\text{d}y \qquad \qquad \\ \qquad \quad \ \ =-\int \left[\log p_0(y;\gamma)+1\right]\frac{\text{d}p_1(y;\gamma)}{\text{d}\gamma}\text{d}y\\ \qquad \qquad \qquad \qquad \qquad \qquad \ =\frac{1}{2\sqrt{\gamma} }\int \log p_0(y;\gamma)\frac{\text{d}p_1(y;\gamma)}{\text{d}y}\text{d}y+\frac{1}{2\sqrt{\gamma} }\underbrace{\int \frac{\text{d}p_1(y;\gamma)}{\text{d}y}\text{d}y}_{\kappa}\\ \qquad \ \ \overset{(a)}{=}\frac{1}{2\sqrt{\gamma} }\int \log p_0(y;\gamma)\frac{\text{d}p_1(y;\gamma)}{\text{d}y}\text{d}y\\ \quad \quad \ \overset{(b)}{=}-\frac{1}{2\sqrt{\gamma} }\int \frac{p_1(y;\gamma)}{p_0(y;\gamma)}\frac{\text{d}p_0(y;\gamma)}{\text{d}y}\text{d}y\\ \qquad \qquad \qquad \qquad \ \ \overset{(c)}{=}\frac{1}{2\sqrt{\gamma} }\int \frac{p_1(y;\gamma)}{p_0(y;\gamma)}\left[y-\sqrt{\gamma}\frac{p_1(y;\gamma)}{p_0(y;\gamma)}\right]p_0(y;\gamma)\text{d}y
where (a)(a) holds thanks to integration by parts,
κ=p1(y;γ)y=y=+=0 \kappa=\left.p_1(y;\gamma)\right|_{y=-\infty}^{y=+\infty}=0
(b)(b) holds also based on integration by parts,
logp0(y;γ)dp1(y;γ)dydy=p1(y;γ)logp0(y;γ)y=y=+p1(y;γ)p0(y;γ)dp0(y;γ)dydy=12γp1(y;γ)p0(y;γ)dp0(y;γ)dydy \int \log p_0(y;\gamma)\frac{\text{d}p_1(y;\gamma)}{\text{d}y}\text{d}y \qquad \qquad \\ \qquad \qquad \qquad=\left.{p_1(y;\gamma)\log p_0(y;\gamma)}\right|_{y=-\infty}^{y=+\infty}-\int \frac{p_1(y;\gamma)}{p_0(y;\gamma)}\frac{\text{d}p_0(y;\gamma)}{\text{d}y}\text{d}y\\ =-\frac{1}{2\sqrt{\gamma} }\int \frac{p_1(y;\gamma)}{p_0(y;\gamma)}\frac{\text{d}p_0(y;\gamma)}{\text{d}y}\text{d}y \qquad \quad and (c)(c) holds by conclusion 1.

Based on the above, we have
dI(X;Y)dγ=12γ(xp(xy;γ)dx)[yγ(xp(xy;γ)dx)]p(y;γ)dy =12γx,yxyp(x,y;γ)dxdy12y(xp(xy;γ)dx)2p(y;γ)dy=12x2p(x,y;γ)dxdy12x^p(x,y;γ)dxdy   =12E{X2X^2}=(d)12MMSE  \frac{\text{d}I(X;Y)}{\text{d}\gamma} =\frac{1}{2\sqrt{\gamma} }\int \left(\int xp(x|y;\gamma)\text{d}x\right)\left[y-\sqrt{\gamma}\left(\int xp(x|y;\gamma)\text{d}x\right)\right]p(y;\gamma)\text{d}y\\ \quad \qquad \ =\frac{1}{2\sqrt{\gamma} }\int_{x,y} xyp(x,y;\gamma)\text{d}x\text{d}y-\frac{1}{2}\int_y\left(\int xp(x|y;\gamma)\text{d}x\right)^2p(y;\gamma)\text{d}y\\ =\frac{1}{2}\int x^2p(x,y;\gamma)\text{d}x\text{d}y-\frac{1}{2}\int \hat{x}p(x,y;\gamma)\text{d}x\text{d}y \quad \qquad \quad \ \ \ \\ =\frac{1}{2}\mathbb{E}\left\{X^2-\hat{X}^2\right\} \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \quad \\ \overset{(d)}{=}\frac{1}{2}\text{MMSE} \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad\qquad \ where the expectation is taken over p(x,y;γ)p(x,y;\gamma). In addition, (d)(d) holds by
(xx^)2p(x,y;γ)dxdy=x2p(x,y;γ)dxdy+x^2p(x,y;γ)dxdy2xx^p(x,y;γ)dxdy   =x2p(x)dx+x^2p(y;γ)dy2x^xp(xy;γ)dxp(y;γ)dy  =x2p(x)dx+x^2p(y;γ)dy2x^2p(y;γ)dy=x2p(x)dxx^2p(y;γ)dy =(x2x^)p(x,y;γ)dxdy  \int (x-\hat{x})^2p(x,y;\gamma)\text{d}x\text{d}y \qquad \qquad \qquad \\ \qquad \qquad \qquad \qquad \qquad \quad=\int x^2p(x,y;\gamma)\text{d}x\text{d}y+\int \hat{x}^2 p(x,y;\gamma)\text{d}x\text{d}y-2\int x\hat{x}p(x,y;\gamma)\text{d}x\text{d}y\\ \qquad \qquad \qquad \qquad \quad \ \ \ =\int x^2p(x)\text{d}x+ \int \hat{x}^2p(y;\gamma)\text{d}y-2\int \hat{x} \int x p(x|y;\gamma)\text{d}x p(y;\gamma)\text{d}y\\ \qquad \quad\ \ =\int x^2p(x)\text{d}x+\int \hat{x}^2p(y;\gamma)\text{d}y-2\int \hat{x}^2p(y;\gamma)\text{d}y\\ =\int x^2p(x)\text{d}x-\int \hat{x}^2p(y;\gamma)\text{d}y \qquad \qquad \ \\ =\int (x^2-\hat{x})p(x,y;\gamma)\text{d}x\text{d}y \qquad \qquad \qquad \ Note that x^=xp(xy;γ)dx\hat{x}=\int xp(x|y;\gamma)\text{d}x is the function of yy.

References

[1] Guo D. Gaussian channels: Information, estimation and multiuser detection[D]. Princeton University, 2004.
[2] Guo D, Shamai S, Verdú S. Mutual information and minimum mean-square error in Gaussian channels[J]. IEEE Transactions on Information Theory, 2005, 51(4): 1261-1282.

展开阅读全文

没有更多推荐了,返回首页