互信息的不变性
有几天没写啥东西了,更一个。
Theorem.1 令 I ( X , Y ) I(X,Y) I(X,Y)为 X X X和 Y Y Y之间的互信息。令 U = F ( X ) , V = G ( Y ) U=F(X),V=G(Y) U=F(X),V=G(Y),其中, U , V , X , Y U,V,X,Y U,V,X,Y都是随机向量, F , V F,V F,V是光滑且可逆的映射。则 I ( U , V ) = I ( X , Y ) I(U,V)=I(X,Y) I(U,V)=I(X,Y).
证明:
变换的雅可比行列式为
J
U
=
∣
∂
U
∂
X
∣
J_U=|\frac{\partial U}{\partial X}|
JU=∣∂X∂U∣和
J
U
=
∣
∂
V
∂
Y
∣
J_U=|\frac{\partial V}{\partial Y}|
JU=∣∂Y∂V∣,则有:
f
U
,
V
(
u
,
v
)
J
U
(
x
)
J
V
(
y
)
=
f
X
,
Y
(
x
,
y
)
d
u
=
J
U
d
x
d
v
=
J
v
d
y
f
U
(
u
)
J
U
(
x
)
=
f
X
(
x
)
f
V
(
v
)
J
V
(
y
)
=
f
Y
(
y
)
f_{U,V}(u,v)J_U(x)J_V(y)=f_{X,Y}(x,y)\\ du=J_U dx \ dv = J_vdy\\ f_U(u)J_U(x)=f_X(x)\\ f_V(v)J_V(y)=f_Y(y)
fU,V(u,v)JU(x)JV(y)=fX,Y(x,y)du=JUdx dv=JvdyfU(u)JU(x)=fX(x)fV(v)JV(y)=fY(y)
I ( U , V ) = ∫ ∫ f U , V ( u , v ) l o g ( f U , V ( u , v ) f U ( u ) f V ( v ) ) d u d v = ∫ ∫ f X , Y ( x , y ) J U J V l o g ( f X , Y ( x , y ) J U J V f X ( x ) f Y ( y ) J U J V ) J U d x J V d y = ∫ ∫ f X , Y ( x , y ) l o g ( f X , Y ( x , y ) f X ( x ) f Y ( y ) ) d x d y = I ( X , Y ) \begin{aligned} I(U,V) &= \int\int f_{U,V}(u,v)log(\frac{f_{U,V}(u,v)}{f_U(u)f_V(v)})dudv\\ &= \int\int\frac{f_{X,Y}(x,y)}{J_UJ_V}log(\frac{\frac{f_{X,Y}(x,y)}{J_UJ_V}}{\frac{f_{X}(x)f_Y(y)}{J_UJ_V}})J_UdxJ_Vdy\\ &=\int\int f_{X,Y}(x,y)log(\frac{f_{X,Y}(x,y)}{f_{X}(x)f_Y(y)})dxdy\\ &=I(X,Y) \end{aligned} I(U,V)=∫∫fU,V(u,v)log(fU(u)fV(v)fU,V(u,v))dudv=∫∫JUJVfX,Y(x,y)log(JUJVfX(x)fY(y)JUJVfX,Y(x,y))JUdxJVdy=∫∫fX,Y(x,y)log(fX(x)fY(y)fX,Y(x,y))dxdy=I(X,Y)
因此,对任意随机变量做满足上述要求的变换,其互信息是不变的。这一点在许多论文中提到,例如著名的神经互信息估计MINE。