信息测度实例（二）：信息熵、联合熵、条件熵、互信息、条件互信息

例二：非均匀分布

假设随机变量$X$有两个分布律：

$X$	$A$	$B$	$C$	$D$
$P$	$\frac{1}{4}$	$\frac{1}{4}$	$\frac{1}{4}$	$\frac{1}{4}$

$X$	$A$	$B$	$C$	$D$
$P$	$\frac{1}{2}$	$\frac{1}{4}$	$\frac{1}{8}$	$\frac{1}{8}$

$Y$表示选择两种分布律的概率：

$Y$	$1$	$2$
$P$	$\frac{1}{2}$	$\frac{1}{2}$

则$X$与$Y$的联合分布律为：

$X$	$A$	$B$	$C$	$D$	$A$	$B$	$C$	$D$
$Y$	$1$	$1$	$1$	$1$	$2$	$2$	$2$	$2$
$P$	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{1}{4}$	$\frac{1}{8}$	$\frac{1}{16}$	$\frac{1}{16}$

$Z$的分布率分别为：

$Z$	$0$	$1$
$P$	$\frac{1}{2}$	$\frac{1}{2}$

$X$、$Y$与$Z$的联合分布律为：

$X$	$A$	$B$	$C$	$D$	$A$	$B$	$C$	$D$	$A$	$B$
$Y$	$1$	$1$	$1$	$1$	$2$	$2$	$2$	$2$	$1$	$1$
$Z$	$0$	$0$	$0$	$0$	$0$	$0$	$0$	$0$	$1$	$1$
$P$	$\frac{1}{16}$	$\frac{1}{16}$	$\frac{1}{16}$	$\frac{1}{16}$	$\frac{1}{8}$	$\frac{1}{16}$	$\frac{1}{32}$	$\frac{1}{32}$	$\frac{1}{4}$	$\frac{1}{4}$

例2.1 信息熵

由信息熵的定义，有：
$$\begin{array}{rl}H\left(X\right)& =-\sum _{x\in X}p\left(x\right)\cdot \log p\left(x\right)\\ & =-\frac{7}{16}\log \frac{7}{16}-\frac{3}{8}\log \frac{3}{8}-\frac{3}{32}\log \frac{3}{32}-\frac{3}{32}\log \frac{3}{32}\\ & =\frac{1}{16}\times \left(61\log 2-7\log 7-9\log 3\right)=1.17332\\ & \\ H\left(Y\right)& =-\sum _{y\in Y}p\left(y\right)\cdot \log p\left(y\right)=-\frac{3}{4}\log \frac{3}{4}-\frac{1}{4}\log \frac{1}{4}\\ & =2\log 2-\frac{3}{4}\log 3=0.562335\\ H\left(Z\right)& =-\sum _{z\in Z}p\left(z\right)\cdot \log p\left(z\right)=-\frac{1}{2}\log \frac{1}{2}-\frac{1}{2}\log \frac{1}{2}\\ & =\log 2=0.693147\end{array}$$

例2.2 联合熵

由概率可知：
$$\begin{gathered} p\left(x=A,y=1\right)=\frac{5}{16}\text{，}p\left(x=A,y=2\right)=\frac{1}{8} \\ p\left(x=B,y=1\right)=\frac{5}{16}\text{，}p\left(x=B,y=2\right)=\frac{1}{16} \\ p\left(x=C,y=1\right)=\frac{1}{16}\text{，}p\left(x=C,y=2\right)=\frac{1}{32} \\ p\left(x=D,y=1\right)=\frac{1}{16}\text{，}p\left(x=D,y=2\right)=\frac{1}{32} \end{gathered}$$则：
$$\begin{array}{rl}H\left(X,Y\right)& =-\sum _{x\in X}\sum _{y\in Y}p\left(x,y\right)\log p\left(x,y\right)\\ & \\ & =-\frac{5}{16}\log \frac{5}{16}-\frac{5}{16}\log \frac{5}{16}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{8}\log \frac{1}{8}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{32}\log \frac{1}{32}-\frac{1}{32}\log \frac{1}{32}\\ & \\ & =\frac{1}{16}\times \left(63\log 2-10\log 5\right)=1.72337\end{array}$$

例2.3 条件熵

计算可得
$$\begin{gathered} p\left(x=A|y=1\right)=\frac{p\left(x=A,y=1\right)}{p\left(y=1\right)}=\frac{\frac{5}{16}}{\frac{3}{4}}=\frac{5}{12}\text{，}p\left(x=A|y=2\right)=\frac{p\left(x=A,y=2\right)}{p\left(y=2\right)}=\frac{\frac{1}{8}}{\frac{1}{4}}=\frac{1}{2} \\ p\left(x=B|y=1\right)=\frac{p\left(x=B,y=1\right)}{p\left(y=1\right)}=\frac{\frac{5}{16}}{\frac{3}{4}}=\frac{5}{12}\text{，}p\left(x=B|y=2\right)=\frac{p\left(x=B,y=2\right)}{p\left(y=2\right)}=\frac{\frac{1}{16}}{\frac{1}{4}}=\frac{1}{4} \\ p\left(x=C|y=1\right)=\frac{p\left(x=C,y=1\right)}{p\left(y=1\right)}=\frac{\frac{1}{16}}{\frac{3}{4}}=\frac{1}{12}\text{，}p\left(x=C|y=2\right)=\frac{p\left(x=C,y=2\right)}{p\left(y=2\right)}=\frac{\frac{1}{32}}{\frac{1}{4}}=\frac{1}{8} \\ p\left(x=D|y=1\right)=\frac{p\left(x=D,y=1\right)}{p\left(y=1\right)}=\frac{\frac{1}{16}}{\frac{3}{4}}=\frac{1}{12}\text{，}p\left(x=D|y=2\right)=\frac{p\left(x=D,y=2\right)}{p\left(y=2\right)}=\frac{\frac{1}{32}}{\frac{1}{4}}=\frac{1}{8} \end{gathered}$$
则
$$\begin{array}{rl}H\left(X|Y=1\right)& =-p\left(x=A|y=1\right)\cdot \log p\left(x=A|y=1\right)-p\left(x=B|y=1\right)\cdot \log p\left(x=B|y=1\right)\\ & -p\left(x=C|y=1\right)\cdot \log p\left(x=C|y=1\right)-p\left(x=D|y=1\right)\cdot \log p\left(x=D|y=1\right)\\ & \\ & =-\frac{5}{12}\log \frac{5}{12}-\frac{5}{12}\log \frac{5}{12}-\frac{1}{12}\log \frac{1}{12}-\frac{1}{12}\log \frac{1}{12}\\ & \\ & =\log 12-\frac{5}{6}\log 5=1.14371\end{array}$$
$$\begin{array}{rl}H\left(X|Y=2\right)& =-p\left(x=A|y=2\right)\cdot \log p\left(x=A|y=2\right)-p\left(x=B|y=2\right)\cdot \log p\left(x=B|y=2\right)\\ & -p\left(x=C|y=2\right)\cdot \log p\left(x=C|y=2\right)-p\left(x=D|y=2\right)\cdot \log p\left(x=D|y=2\right)\\ & \\ & =-\frac{1}{2}\log \frac{1}{2}-\frac{1}{4}\log \frac{1}{4}-\frac{1}{8}\log \frac{1}{8}-\frac{1}{8}\log \frac{1}{8}\\ & \\ & =\frac{7}{4}\log 2=1.21301\end{array}$$
通过以上计算可到条件熵为：
$$\begin{array}{rl}H\left(X|Y\right)& =p\left(y=1\right)\cdot H\left(X|Y=1\right)+p\left(y=2\right)\cdot H\left(X|Y=2\right)\\ & \\ & =\frac{3}{4}\times \left(\log 12-\frac{5}{6}\log 5\right)+\frac{1}{4}\times \left(\frac{7}{4}\log 2\right)\\ & \\ & =\frac{3}{4}\log 12-\frac{5}{8}\log 5+\frac{7}{16}\log 2=1.16103\end{array}$$同理可计算出
$$\begin{gathered} p\left(y=1|x=A\right)=\frac{p\left(y=1,x=A\right)}{p\left(x=A\right)}=\frac{\frac{5}{16}}{\frac{7}{16}}=\frac{5}{7}\text{，}p\left(y=2|x=A\right)=\frac{p\left(y=2,x=A\right)}{p\left(x=A\right)}=\frac{\frac{1}{8}}{\frac{7}{16}}=\frac{2}{7} \\ p\left(y=1|x=B\right)=\frac{p\left(y=1,x=B\right)}{p\left(x=B\right)}=\frac{\frac{5}{16}}{\frac{3}{8}}=\frac{5}{6}\text{，}p\left(y=2|x=B\right)=\frac{p\left(y=2,x=B\right)}{p\left(x=B\right)}=\frac{\frac{1}{16}}{\frac{3}{8}}=\frac{1}{6} \\ p\left(y=1|x=C\right)=\frac{p\left(y=1,x=C\right)}{p\left(x=C\right)}=\frac{\frac{1}{16}}{\frac{3}{32}}=\frac{2}{3}\text{，}p\left(y=2|x=C\right)=\frac{p\left(y=2,x=C\right)}{p\left(x=C\right)}=\frac{\frac{1}{32}}{\frac{3}{32}}=\frac{1}{3} \\ p\left(y=1|x=D\right)=\frac{p\left(y=1,x=D\right)}{p\left(x=D\right)}=\frac{\frac{1}{16}}{\frac{3}{32}}=\frac{2}{3}\text{，}p\left(y=2|x=D\right)=\frac{p\left(y=2,x=D\right)}{p\left(x=D\right)}=\frac{\frac{1}{32}}{\frac{3}{32}}=\frac{1}{3} \end{gathered}$$则
$$\begin{array}{rl}H\left(Y|X=A\right)& =-p\left(y=1|x=A\right)\cdot \log p\left(y=1|x=A\right)-p\left(y=2|x=A\right)\cdot \log p\left(y=2|x=A\right)\\ & \\ & =-\frac{5}{7}\log \frac{5}{7}-\frac{2}{7}\log \frac{2}{7}=\log 7-\frac{5}{7}\log 5-\frac{2}{7}\log 2\\ & \\ & =0.59827\\ H\left(Y|X=B\right)& =-p\left(y=1|x=B\right)\cdot \log p\left(y=1|x=B\right)-p\left(y=2|x=B\right)\cdot \log p\left(y=2|x=B\right)\\ & \\ & =-\frac{5}{6}\log \frac{5}{6}-\frac{1}{6}\log \frac{1}{6}=\log 6-\frac{5}{6}\log 5\\ & \\ & =0.450561\\ H\left(Y|X=C\right)& =-p\left(y=1|x=C\right)\cdot \log p\left(y=1|x=C\right)-p\left(y=2|x=C\right)\cdot \log p\left(y=2|x=C\right)\\ & \\ & =-\frac{2}{3}\log \frac{2}{3}-\frac{1}{3}\log \frac{1}{3}=\log 3-\frac{2}{3}\log 2\\ & \\ & =0.636514\\ H\left(Y|X=D\right)& =-p\left(y=1|x=D\right)\cdot \log p\left(y=1|x=D\right)-p\left(y=2|x=D\right)\cdot \log p\left(y=2|x=D\right)\\ & \\ & =-\frac{2}{3}\log \frac{2}{3}-\frac{1}{3}\log \frac{1}{3}=\log 3-\frac{2}{3}\log 2\\ & \\ & =0.636514\end{array}$$于是

$$\begin{array}{rl}H\left(Y|X\right)& =p\left(x=A\right)\cdot H\left(Y|X=A\right)+p\left(x=B\right)\cdot H\left(Y|X=B\right)+p\left(x=C\right)\cdot H\left(Y|X=C\right)+p\left(x=D\right)\cdot H\left(Y|X=D\right)\\ & \\ & =\frac{7}{16}\times \left(\log 7-\frac{5}{7}\log 5-\frac{2}{7}\log 2\right)+\frac{3}{8}\times \left(\log 6-\frac{5}{6}\log 5\right)+\frac{3}{32}\times \left(\log 3-\frac{2}{3}\log 2\right)+\frac{3}{32}\times \left(\log 3-\frac{2}{3}\log 2\right)\\ & \\ & =\frac{1}{16}\times \left(7\log 7+6\log 6-10\log 5+3\log 3-4\log 2\right)\\ & =0.55005\end{array}$$

例2.4 互信息

$$p\left(x=0,y=0\right)=\frac{1}{3}\text{，}p\left(x=0,y=1\right)=\frac{1}{6}\text{，}p\left(x=1,y=0\right)=\frac{1}{6}\text{，}p\left(x=1,y=1\right)=\frac{1}{3}$$

则$X$和$Y$的互信息为：
$$\begin{array}{rl}I\left(X;Y\right)& =p\left(x=A,y=1\right)\cdot \log \frac{p\left(x=A,y=1\right)}{p\left(x=A\right)\cdot p\left(y=1\right)}+p\left(x=A,y=2\right)\cdot \log \frac{p\left(x=A,y=2\right)}{p\left(x=A\right)\cdot p\left(y=2\right)}\\ & \\ & +p\left(x=B,y=1\right)\cdot \log \frac{p\left(x=B,y=1\right)}{p\left(x=B\right)\cdot p\left(y=1\right)}+p\left(x=B,y=2\right)\cdot \log \frac{p\left(x=B,y=2\right)}{p\left(x=B\right)\cdot p\left(y=2\right)}\\ & \\ & +p\left(x=C,y=1\right)\cdot \log \frac{p\left(x=C,y=1\right)}{p\left(x=C\right)\cdot p\left(y=1\right)}+p\left(x=C,y=2\right)\cdot \log \frac{p\left(x=C,y=2\right)}{p\left(x=C\right)\cdot p\left(y=2\right)}\\ & \\ & +p\left(x=D,y=1\right)\cdot \log \frac{p\left(x=D,y=1\right)}{p\left(x=D\right)\cdot p\left(y=1\right)}+p\left(x=D,y=2\right)\cdot \log \frac{p\left(x=D,y=2\right)}{p\left(x=D\right)\cdot p\left(y=2\right)}\\ & \\ & =\frac{5}{16}\times \log \frac{\frac{5}{16}}{\frac{7}{16}\times \frac{3}{4}}+\frac{1}{8}\times \log \frac{\frac{1}{8}}{\frac{7}{16}\times \frac{1}{4}}+\frac{5}{16}\times \log \frac{\frac{5}{16}}{\frac{3}{8}\times \frac{3}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{8}\times \frac{1}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{32}\times \frac{3}{4}}+\frac{1}{32}\times \log \frac{\frac{1}{32}}{\frac{3}{32}\times \frac{1}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{32}\times \frac{3}{4}}+\frac{1}{32}\times \log \frac{\frac{1}{32}}{\frac{3}{32}\times \frac{1}{4}}\\ & \\ & =\frac{1}{16}\times \left(30\log 2-21\log 3+10\log 5-7\log 7\right)\\ & =0.0122853\end{array}$$

例2.5 条件互信息

$$\begin{gathered} p\left(z=0\right)=\frac{1}{2}，p\left(z=1\right)=\frac{1}{2} \\ p\left(x=A|z=0\right)=\frac{3}{8}\text{，}p\left(x=B|z=0\right)=\frac{1}{4}\text{，}p\left(x=C|z=0\right)=\frac{3}{16}\text{，}p\left(x=D|z=0\right)=\frac{3}{16} \\ p\left(x=A|z=1\right)=\frac{1}{2}\text{，}p\left(x=B|z=1\right)=\frac{1}{2}\text{，}p\left(x=C|z=1\right)=0\text{，}p\left(x=D|z=1\right)=0 \\ p\left(y=1|z=0\right)=\frac{1}{2}\text{，}p\left(y=2|z=0\right)=\frac{1}{2}\text{，}p\left(y=1|z=1\right)=1\text{，}p\left(y=2|z=1\right)=0 \end{gathered}$$
记
$$\begin{gathered} p_{01}=p\left(x=A,y=1|z=0\right)=\frac{1}{8}\text{，}{p}_{02}=p\left(x=B,y=1|z=0\right)=\frac{1}{8}\text{，}{p}_{03}=p\left(x=C,y=1|z=0\right)=\frac{1}{8}\text{，}{p}_{04}=p\left(x=D,y=1|z=0\right)=\frac{1}{8} \\ {p}_{05}=p\left(x=A,y=2|z=0\right)=\frac{1}{4}\text{，}{p}_{06}=p\left(x=B,y=2|z=0\right)=\frac{1}{8}\text{，}{p}_{07}=p\left(x=C,y=2|z=0\right)=\frac{1}{16}\text{，}{p}_{08}=p\left(x=D,y=2|z=0\right)=\frac{1}{16} \\ {p}_{11}=p\left(x=A,y=1|z=1\right)=\frac{1}{2}\text{，}{p}_{12}=p\left(x=B,y=1|z=1\right)=\frac{1}{2} \end{gathered}$$

$$\begin{array}{rl}{P}_{01}& =\frac{p\left(x=A,y=1|z=0\right)}{p\left(x=A|z=0\right)\cdot p\left(y=1|z=0\right)}=\frac{\frac{1}{8}}{\frac{3}{8}\times \frac{1}{2}}=\frac{2}{3}\text{，}{P}_{02}=\frac{p\left(x=B,y=1|z=0\right)}{p\left(x=B|z=0\right)\cdot p\left(y=1|z=0\right)}=\frac{\frac{1}{8}}{\frac{1}{4}\times \frac{1}{2}}=1\\ & \\ P_{03}& =\frac{p\left(x=C,y=1|z=0\right)}{p\left(x=C|z=0\right)\cdot p\left(y=1|z=0\right)}=\frac{\frac{1}{8}}{\frac{3}{16}\times \frac{1}{2}}=\frac{4}{3}\text{，}{P}_{04}=\frac{p\left(x=D,y=1|z=0\right)}{p\left(x=D|z=0\right)\cdot p\left(y=1|z=0\right)}=\frac{\frac{1}{8}}{\frac{3}{16}\times \frac{1}{2}}=\frac{4}{3}\\ & \\ P_{05}& =\frac{p\left(x=A,y=2|z=0\right)}{p\left(x=A|z=0\right)\cdot p\left(y=2|z=0\right)}=\frac{\frac{1}{4}}{\frac{3}{8}\times \frac{1}{2}}=\frac{4}{3}\text{，}{P}_{06}=\frac{p\left(x=B,y=2|z=0\right)}{p\left(x=B|z=0\right)\cdot p\left(y=2|z=0\right)}=\frac{\frac{1}{8}}{\frac{1}{4}\times \frac{1}{2}}=1\\ & \\ P_{07}& =\frac{p\left(x=C,y=2|z=0\right)}{p\left(x=C|z=0\right)\cdot p\left(y=2|z=0\right)}=\frac{\frac{1}{16}}{\frac{3}{16}\times \frac{1}{2}}=\frac{2}{3}\text{，}{P}_{08}=\frac{p\left(x=D,y=2|z=0\right)}{p\left(x=D|z=0\right)\cdot p\left(y=2|z=0\right)}=\frac{\frac{1}{16}}{\frac{3}{16}\times \frac{1}{2}}=\frac{2}{3}\\ & \\ P_{11}& =\frac{p\left(x=A,y=1|z=1\right)}{p\left(x=A|z=1\right)\cdot p\left(y=1|z=1\right)}=\frac{\frac{1}{2}}{\frac{1}{2}\times 1}=1\text{，}{P}_{12}=\frac{p\left(x=B,y=1|z=1\right)}{p\left(x=B|z=1\right)\cdot p\left(y=1|z=1\right)}=\frac{\frac{1}{2}}{\frac{1}{2}\times 1}=1\end{array}$$
故条件互信息为：
$$\begin{array}{rl}I\left(X;Y|Z\right)& =\sum _{z\in Z}p\left(z\right)\sum _{x\in X}\sum _{y\in Y}p\left(x,y|z\right)\cdot \log \frac{p\left(x,y|z\right)}{p\left(x|z\right)p\left(y|z\right)}\\ & \\ & =p\left(z=0\right)\times \left[p_{01}\cdot \log P_{01}+p_{02}\cdot \log P_{02}+p_{03}\cdot \log P_{03}+p_{04}\cdot \log P_{04}+p_{05}\cdot \log P_{05}+p_{06}\cdot \log P_{06}+p_{07}\cdot \log P_{07}+p_{08}\cdot \log P_{08}\right]\\ & \\ & +p\left(z=1\right)\times \left[p_{11}\cdot \log P_{11}+p_{12}\cdot \log P_{12}\right]\\ & \\ & =\frac{1}{2}\times \left[\frac{1}{8}\log \frac{2}{3}+0+\frac{1}{8}\log \frac{4}{3}+\frac{1}{8}\log \frac{4}{3}+\frac{1}{4}\log \frac{4}{3}+0+\frac{1}{16}\log \frac{2}{3}+\frac{1}{16}\log \frac{2}{3}\right]+\frac{1}{2}\times \left[0+0\right]\\ & \\ & =\frac{1}{8}\times \left(5\log 2-3\log 3\right)=0.0212374\end{array}$$