例二:非均匀分布
假设随机变量 X X X有两个分布律:
X X X | A A A | B B B | C C C | D D D |
---|---|---|---|---|
P P P | 1 4 \frac{1}{4} 41 | 1 4 \frac{1}{4} 41 | 1 4 \frac{1}{4} 41 | 1 4 \frac{1}{4} 41 |
X X X | A A A | B B B | C C C | D D D |
---|---|---|---|---|
P P P | 1 2 \frac{1}{2} 21 | 1 4 \frac{1}{4} 41 | 1 8 \frac{1}{8} 81 | 1 8 \frac{1}{8} 81 |
Y Y Y表示选择两种分布律的概率:
Y Y Y | 1 1 1 | 2 2 2 |
---|---|---|
P P P | 1 2 \frac{1}{2} 21 | 1 2 \frac{1}{2} 21 |
则 X X X与 Y Y Y的联合分布律为:
X X X | A A A | B B B | C C C | D D D | A A A | B B B | C C C | D D D |
---|---|---|---|---|---|---|---|---|
Y Y Y | 1 1 1 | 1 1 1 | 1 1 1 | 1 1 1 | 2 2 2 | 2 2 2 | 2 2 2 | 2 2 2 |
P P P | 1 8 \frac{1}{8} 81 | 1 8 \frac{1}{8} 81 | 1 8 \frac{1}{8} 81 | 1 8 \frac{1}{8} 81 | 1 4 \frac{1}{4} 41 | 1 8 \frac{1}{8} 81 | 1 16 \frac{1}{16} 161 | 1 16 \frac{1}{16} 161 |
Z Z Z的分布率分别为:
Z Z Z | 0 0 0 | 1 1 1 |
---|---|---|
P P P | 1 2 \frac{1}{2} 21 | 1 2 \frac{1}{2} 21 |
X X X、 Y Y Y与 Z Z Z的联合分布律为:
X X X | A A A | B B B | C C C | D D D | A A A | B B B | C C C | D D D | A A A | B B B |
---|---|---|---|---|---|---|---|---|---|---|
Y Y Y | 1 1 1 | 1 1 1 | 1 1 1 | 1 1 1 | 2 2 2 | 2 2 2 | 2 2 2 | 2 2 2 | 1 1 1 | 1 1 1 |
Z Z Z | 0 0 0 | 0 0 0 | 0 0 0 | 0 0 0 | 0 0 0 | 0 0 0 | 0 0 0 | 0 0 0 | 1 1 1 | 1 1 1 |
P P P | 1 16 \frac{1}{16} 161 | 1 16 \frac{1}{16} 161 | 1 16 \frac{1}{16} 161 | 1 16 \frac{1}{16} 161 | 1 8 \frac{1}{8} 81 | 1 16 \frac{1}{16} 161 | 1 32 \frac{1}{32} 321 | 1 32 \frac{1}{32} 321 | 1 4 \frac{1}{4} 41 | 1 4 \frac{1}{4} 41 |
例2.1 信息熵
由信息熵的定义,有:
H
(
X
)
=
−
∑
x
∈
X
p
(
x
)
⋅
log
p
(
x
)
=
−
7
16
log
7
16
−
3
8
log
3
8
−
3
32
log
3
32
−
3
32
log
3
32
=
1
16
×
(
61
log
2
−
7
log
7
−
9
log
3
)
=
1.17332
H
(
Y
)
=
−
∑
y
∈
Y
p
(
y
)
⋅
log
p
(
y
)
=
−
3
4
log
3
4
−
1
4
log
1
4
=
2
log
2
−
3
4
log
3
=
0.562335
H
(
Z
)
=
−
∑
z
∈
Z
p
(
z
)
⋅
log
p
(
z
)
=
−
1
2
log
1
2
−
1
2
log
1
2
=
log
2
=
0.693147
\begin{aligned} H\left( X \right) &=-\sum_{x\in X}{p\left( x \right)}\cdot \log p\left( x \right) \\ &=-\frac{7}{16}\log \frac{7}{16}-\frac{3}{8}\log \frac{3}{8}-\frac{3}{32}\log \frac{3}{32}-\frac{3}{32}\log \frac{3}{32} \\ &=\frac{1}{16}\times \left( 61\log 2-7\log 7-9\log 3 \right) =1.17332 \\ \\ H\left( Y \right) &=-\sum_{y\in Y}{p\left( y \right)}\cdot \log p\left( y \right) =-\frac{3}{4}\log \frac{3}{4}-\frac{1}{4}\log \frac{1}{4} \\ &=2\log 2-\frac{3}{4}\log 3=0.562335 \\ H\left( Z \right) &=-\sum_{z\in Z}{p\left( z \right)}\cdot \log p\left( z \right) =-\frac{1}{2}\log \frac{1}{2}-\frac{1}{2}\log \frac{1}{2} \\ &=\log 2=0.693147 \end{aligned}
H(X)H(Y)H(Z)=−x∈X∑p(x)⋅logp(x)=−167log167−83log83−323log323−323log323=161×(61log2−7log7−9log3)=1.17332=−y∈Y∑p(y)⋅logp(y)=−43log43−41log41=2log2−43log3=0.562335=−z∈Z∑p(z)⋅logp(z)=−21log21−21log21=log2=0.693147
例2.2 联合熵
由概率可知:
p
(
x
=
A
,
y
=
1
)
=
5
16
,
p
(
x
=
A
,
y
=
2
)
=
1
8
p
(
x
=
B
,
y
=
1
)
=
5
16
,
p
(
x
=
B
,
y
=
2
)
=
1
16
p
(
x
=
C
,
y
=
1
)
=
1
16
,
p
(
x
=
C
,
y
=
2
)
=
1
32
p
(
x
=
D
,
y
=
1
)
=
1
16
,
p
(
x
=
D
,
y
=
2
)
=
1
32
p\left( x=A,y=1 \right) =\frac{5}{16}\text{,}p\left( x=A,y=2 \right) =\frac{1}{8} \\ \ \ \\ \\p\left( x=B,y=1 \right) =\frac{5}{16}\text{,}p\left( x=B,y=2 \right) =\frac{1}{16} \\ \ \ \\ \\p\left( x=C,y=1 \right) =\frac{1}{16}\text{,}p\left( x=C,y=2 \right) =\frac{1}{32} \\ \ \ \\\\p\left( x=D,y=1 \right) =\frac{1}{16}\text{,}p\left( x=D,y=2 \right) =\frac{1}{32}
p(x=A,y=1)=165,p(x=A,y=2)=81 p(x=B,y=1)=165,p(x=B,y=2)=161 p(x=C,y=1)=161,p(x=C,y=2)=321 p(x=D,y=1)=161,p(x=D,y=2)=321则:
H
(
X
,
Y
)
=
−
∑
x
∈
X
∑
y
∈
Y
p
(
x
,
y
)
log
p
(
x
,
y
)
=
−
5
16
log
5
16
−
5
16
log
5
16
−
1
16
log
1
16
−
1
16
log
1
16
−
1
8
log
1
8
−
1
16
log
1
16
−
1
32
log
1
32
−
1
32
log
1
32
=
1
16
×
(
63
log
2
−
10
log
5
)
=
1.72337
\begin{aligned} H\left( X,Y \right) &=-\sum_{x\in X}{\sum_{y\in Y}{p\left( x,y \right)}\log}p\left( x,y \right) \\ \\ &=-\frac{5}{16}\log \frac{5}{16}-\frac{5}{16}\log \frac{5}{16}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{8}\log \frac{1}{8}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{32}\log \frac{1}{32}-\frac{1}{32}\log \frac{1}{32} \\ \\ &=\frac{1}{16}\times \left( 63\log 2-10\log 5 \right) =1.72337 \end{aligned}
H(X,Y)=−x∈X∑y∈Y∑p(x,y)logp(x,y)=−165log165−165log165−161log161−161log161−81log81−161log161−321log321−321log321=161×(63log2−10log5)=1.72337
例2.3 条件熵
计算可得
p
(
x
=
A
∣
y
=
1
)
=
p
(
x
=
A
,
y
=
1
)
p
(
y
=
1
)
=
5
16
3
4
=
5
12
,
p
(
x
=
A
∣
y
=
2
)
=
p
(
x
=
A
,
y
=
2
)
p
(
y
=
2
)
=
1
8
1
4
=
1
2
p
(
x
=
B
∣
y
=
1
)
=
p
(
x
=
B
,
y
=
1
)
p
(
y
=
1
)
=
5
16
3
4
=
5
12
,
p
(
x
=
B
∣
y
=
2
)
=
p
(
x
=
B
,
y
=
2
)
p
(
y
=
2
)
=
1
16
1
4
=
1
4
p
(
x
=
C
∣
y
=
1
)
=
p
(
x
=
C
,
y
=
1
)
p
(
y
=
1
)
=
1
16
3
4
=
1
12
,
p
(
x
=
C
∣
y
=
2
)
=
p
(
x
=
C
,
y
=
2
)
p
(
y
=
2
)
=
1
32
1
4
=
1
8
p
(
x
=
D
∣
y
=
1
)
=
p
(
x
=
D
,
y
=
1
)
p
(
y
=
1
)
=
1
16
3
4
=
1
12
,
p
(
x
=
D
∣
y
=
2
)
=
p
(
x
=
D
,
y
=
2
)
p
(
y
=
2
)
=
1
32
1
4
=
1
8
p\left( x=A\left| y=1 \right. \right) =\frac{p\left( x=A,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{5}{16}}{\frac{3}{4}}=\frac{5}{12}\text{,}p\left( x=A\left| y=2 \right. \right) =\frac{p\left( x=A,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{8}}{\frac{1}{4}}=\frac{1}{2} \\ \,\,\mathrm{ } \\ p\left( x=B\left| y=1 \right. \right) =\frac{p\left( x=B,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{5}{16}}{\frac{3}{4}}=\frac{5}{12}\text{,}p\left( x=B\left| y=2 \right. \right) =\frac{p\left( x=B,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{16}}{\frac{1}{4}}=\frac{1}{4} \\ \,\,\mathrm{ } \\ p\left( x=C\left| y=1 \right. \right) =\frac{p\left( x=C,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{1}{16}}{\frac{3}{4}}=\frac{1}{12}\text{,}p\left( x=C\left| y=2 \right. \right) =\frac{p\left( x=C,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{32}}{\frac{1}{4}}=\frac{1}{8} \\ \,\,\mathrm{ } \\ p\left( x=D\left| y=1 \right. \right) =\frac{p\left( x=D,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{1}{16}}{\frac{3}{4}}=\frac{1}{12}\text{,}p\left( x=D\left| y=2 \right. \right) =\frac{p\left( x=D,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{32}}{\frac{1}{4}}=\frac{1}{8}
p(x=A∣y=1)=p(y=1)p(x=A,y=1)=43165=125,p(x=A∣y=2)=p(y=2)p(x=A,y=2)=4181=21p(x=B∣y=1)=p(y=1)p(x=B,y=1)=43165=125,p(x=B∣y=2)=p(y=2)p(x=B,y=2)=41161=41p(x=C∣y=1)=p(y=1)p(x=C,y=1)=43161=121,p(x=C∣y=2)=p(y=2)p(x=C,y=2)=41321=81p(x=D∣y=1)=p(y=1)p(x=D,y=1)=43161=121,p(x=D∣y=2)=p(y=2)p(x=D,y=2)=41321=81
则
H
(
X
∣
Y
=
1
)
=
−
p
(
x
=
A
∣
y
=
1
)
⋅
log
p
(
x
=
A
∣
y
=
1
)
−
p
(
x
=
B
∣
y
=
1
)
⋅
log
p
(
x
=
B
∣
y
=
1
)
−
p
(
x
=
C
∣
y
=
1
)
⋅
log
p
(
x
=
C
∣
y
=
1
)
−
p
(
x
=
D
∣
y
=
1
)
⋅
log
p
(
x
=
D
∣
y
=
1
)
=
−
5
12
log
5
12
−
5
12
log
5
12
−
1
12
log
1
12
−
1
12
log
1
12
=
log
12
−
5
6
log
5
=
1.14371
\begin{aligned} H\left( X\left| Y \right. =1 \right) &=-p\left( x=A\left| y=1 \right. \right) \cdot \log p\left( x=A\left| y=1 \right. \right) -p\left( x=B\left| y=1 \right. \right) \cdot \log p\left( x=B\left| y=1 \right. \right) \\ \,\, & \,\,\,\,\,\,-p\left( x=C\left| y=1 \right. \right) \cdot \log p\left( x=C\left| y=1 \right. \right) -p\left( x=D\left| y=1 \right. \right) \cdot \log p\left( x=D\left| y=1 \right. \right) \\ \,\,\mathrm{ } \\ &=-\frac{5}{12}\log \frac{5}{12}-\frac{5}{12}\log \frac{5}{12}-\frac{1}{12}\log \frac{1}{12}-\frac{1}{12}\log \frac{1}{12} \\ \,\,\mathrm{ } \\ &=\log 12-\frac{5}{6}\log 5=1.14371 \end{aligned}
H(X∣Y=1)=−p(x=A∣y=1)⋅logp(x=A∣y=1)−p(x=B∣y=1)⋅logp(x=B∣y=1)−p(x=C∣y=1)⋅logp(x=C∣y=1)−p(x=D∣y=1)⋅logp(x=D∣y=1)=−125log125−125log125−121log121−121log121=log12−65log5=1.14371
H
(
X
∣
Y
=
2
)
=
−
p
(
x
=
A
∣
y
=
2
)
⋅
log
p
(
x
=
A
∣
y
=
2
)
−
p
(
x
=
B
∣
y
=
2
)
⋅
log
p
(
x
=
B
∣
y
=
2
)
−
p
(
x
=
C
∣
y
=
2
)
⋅
log
p
(
x
=
C
∣
y
=
2
)
−
p
(
x
=
D
∣
y
=
2
)
⋅
log
p
(
x
=
D
∣
y
=
2
)
=
−
1
2
log
1
2
−
1
4
log
1
4
−
1
8
log
1
8
−
1
8
log
1
8
=
7
4
log
2
=
1.21301
\begin{aligned} H\left( X\left| Y \right. =2 \right) &=-p\left( x=A\left| y=2 \right. \right) \cdot \log p\left( x=A\left| y=2 \right. \right) -p\left( x=B\left| y=2 \right. \right) \cdot \log p\left( x=B\left| y=2 \right. \right) \\ \,\, & \,\,\,\,\,\,-p\left( x=C\left| y=2 \right. \right) \cdot \log p\left( x=C\left| y=2 \right. \right) -p\left( x=D\left| y=2 \right. \right) \cdot \log p\left( x=D\left| y=2 \right. \right) \\ \,\,\mathrm{ } \\ &=-\frac{1}{2}\log \frac{1}{2}-\frac{1}{4}\log \frac{1}{4}-\frac{1}{8}\log \frac{1}{8}-\frac{1}{8}\log \frac{1}{8} \\ \,\,\mathrm{ } \\ &=\frac{7}{4}\log 2=1.21301 \end{aligned}
H(X∣Y=2)=−p(x=A∣y=2)⋅logp(x=A∣y=2)−p(x=B∣y=2)⋅logp(x=B∣y=2)−p(x=C∣y=2)⋅logp(x=C∣y=2)−p(x=D∣y=2)⋅logp(x=D∣y=2)=−21log21−41log41−81log81−81log81=47log2=1.21301
通过以上计算可到条件熵为:
H
(
X
∣
Y
)
=
p
(
y
=
1
)
⋅
H
(
X
∣
Y
=
1
)
+
p
(
y
=
2
)
⋅
H
(
X
∣
Y
=
2
)
=
3
4
×
(
log
12
−
5
6
log
5
)
+
1
4
×
(
7
4
log
2
)
=
3
4
log
12
−
5
8
log
5
+
7
16
log
2
=
1.16103
\begin{aligned} H\left( X\left| Y \right. \right) &=p\left( y=1 \right) \cdot H\left( X\left| Y=1 \right. \right) +p\left( y=2 \right) \cdot H\left( X\left| Y=2 \right. \right) \\ \ \ \\ &=\frac{3}{4}\times \left( \log 12-\frac{5}{6}\log 5 \right) +\frac{1}{4}\times \left( \frac{7}{4}\log 2 \right) \\ \ \ \\ &=\frac{3}{4}\log 12-\frac{5}{8}\log 5+\frac{7}{16}\log 2=1.16103 \end{aligned}
H(X∣Y) =p(y=1)⋅H(X∣Y=1)+p(y=2)⋅H(X∣Y=2)=43×(log12−65log5)+41×(47log2)=43log12−85log5+167log2=1.16103同理可计算出
p
(
y
=
1
∣
x
=
A
)
=
p
(
y
=
1
,
x
=
A
)
p
(
x
=
A
)
=
5
16
7
16
=
5
7
,
p
(
y
=
2
∣
x
=
A
)
=
p
(
y
=
2
,
x
=
A
)
p
(
x
=
A
)
=
1
8
7
16
=
2
7
p
(
y
=
1
∣
x
=
B
)
=
p
(
y
=
1
,
x
=
B
)
p
(
x
=
B
)
=
5
16
3
8
=
5
6
,
p
(
y
=
2
∣
x
=
B
)
=
p
(
y
=
2
,
x
=
B
)
p
(
x
=
B
)
=
1
16
3
8
=
1
6
p
(
y
=
1
∣
x
=
C
)
=
p
(
y
=
1
,
x
=
C
)
p
(
x
=
C
)
=
1
16
3
32
=
2
3
,
p
(
y
=
2
∣
x
=
C
)
=
p
(
y
=
2
,
x
=
C
)
p
(
x
=
C
)
=
1
32
3
32
=
1
3
p
(
y
=
1
∣
x
=
D
)
=
p
(
y
=
1
,
x
=
D
)
p
(
x
=
D
)
=
1
16
3
32
=
2
3
,
p
(
y
=
2
∣
x
=
D
)
=
p
(
y
=
2
,
x
=
D
)
p
(
x
=
D
)
=
1
32
3
32
=
1
3
p\left( y=1\left| x=A \right. \right) =\frac{p\left( y=1,x=A \right)}{p\left( x=A \right)}=\frac{\frac{5}{16}}{\frac{7}{16}}=\frac{5}{7}\text{,}p\left( y=2\left| x=A \right. \right) =\frac{p\left( y=2,x=A \right)}{p\left( x=A \right)}=\frac{\frac{1}{8}}{\frac{7}{16}}=\frac{2}{7} \\ \,\, \\ p\left( y=1\left| x=B \right. \right) =\frac{p\left( y=1,x=B \right)}{p\left( x=B \right)}=\frac{\frac{5}{16}}{\frac{3}{8}}=\frac{5}{6}\text{,}p\left( y=2\left| x=B \right. \right) =\frac{p\left( y=2,x=B \right)}{p\left( x=B \right)}=\frac{\frac{1}{16}}{\frac{3}{8}}=\frac{1}{6} \\ \,\, \\ p\left( y=1\left| x=C \right. \right) =\frac{p\left( y=1,x=C \right)}{p\left( x=C \right)}=\frac{\frac{1}{16}}{\frac{3}{32}}=\frac{2}{3}\text{,}p\left( y=2\left| x=C \right. \right) =\frac{p\left( y=2,x=C \right)}{p\left( x=C \right)}=\frac{\frac{1}{32}}{\frac{3}{32}}=\frac{1}{3} \\ \,\, \\ p\left( y=1\left| x=D \right. \right) =\frac{p\left( y=1,x=D \right)}{p\left( x=D \right)}=\frac{\frac{1}{16}}{\frac{3}{32}}=\frac{2}{3}\text{,}p\left( y=2\left| x=D \right. \right) =\frac{p\left( y=2,x=D \right)}{p\left( x=D \right)}=\frac{\frac{1}{32}}{\frac{3}{32}}=\frac{1}{3}
p(y=1∣x=A)=p(x=A)p(y=1,x=A)=167165=75,p(y=2∣x=A)=p(x=A)p(y=2,x=A)=16781=72p(y=1∣x=B)=p(x=B)p(y=1,x=B)=83165=65,p(y=2∣x=B)=p(x=B)p(y=2,x=B)=83161=61p(y=1∣x=C)=p(x=C)p(y=1,x=C)=323161=32,p(y=2∣x=C)=p(x=C)p(y=2,x=C)=323321=31p(y=1∣x=D)=p(x=D)p(y=1,x=D)=323161=32,p(y=2∣x=D)=p(x=D)p(y=2,x=D)=323321=31则
H
(
Y
∣
X
=
A
)
=
−
p
(
y
=
1
∣
x
=
A
)
⋅
log
p
(
y
=
1
∣
x
=
A
)
−
p
(
y
=
2
∣
x
=
A
)
⋅
log
p
(
y
=
2
∣
x
=
A
)
=
−
5
7
log
5
7
−
2
7
log
2
7
=
log
7
−
5
7
log
5
−
2
7
log
2
=
0.59827
H
(
Y
∣
X
=
B
)
=
−
p
(
y
=
1
∣
x
=
B
)
⋅
log
p
(
y
=
1
∣
x
=
B
)
−
p
(
y
=
2
∣
x
=
B
)
⋅
log
p
(
y
=
2
∣
x
=
B
)
=
−
5
6
log
5
6
−
1
6
log
1
6
=
log
6
−
5
6
log
5
=
0.450561
H
(
Y
∣
X
=
C
)
=
−
p
(
y
=
1
∣
x
=
C
)
⋅
log
p
(
y
=
1
∣
x
=
C
)
−
p
(
y
=
2
∣
x
=
C
)
⋅
log
p
(
y
=
2
∣
x
=
C
)
=
−
2
3
log
2
3
−
1
3
log
1
3
=
log
3
−
2
3
log
2
=
0.636514
H
(
Y
∣
X
=
D
)
=
−
p
(
y
=
1
∣
x
=
D
)
⋅
log
p
(
y
=
1
∣
x
=
D
)
−
p
(
y
=
2
∣
x
=
D
)
⋅
log
p
(
y
=
2
∣
x
=
D
)
=
−
2
3
log
2
3
−
1
3
log
1
3
=
log
3
−
2
3
log
2
=
0.636514
\begin{aligned} H\left( Y\left| X \right. =A \right) &=-p\left( y=1\left| x=A \right. \right) \cdot \log p\left( y=1\left| x=A \right. \right) -p\left( y=2\left| x=A \right. \right) \cdot \log p\left( y=2\left| x=A \right. \right)\\ \\ &=-\frac{5}{7}\log \frac{5}{7}-\frac{2}{7}\log \frac{2}{7}=\log 7-\frac{5}{7}\log 5-\frac{2}{7}\log 2\\ \\ &=0.59827 \\ H\left( Y\left| X \right. =B \right) &=-p\left( y=1\left| x=B \right. \right) \cdot \log p\left( y=1\left| x=B \right. \right) -p\left( y=2\left| x=B \right. \right) \cdot \log p\left( y=2\left| x=B \right. \right)\\ \\ &=-\frac{5}{6}\log \frac{5}{6}-\frac{1}{6}\log \frac{1}{6}=\log 6-\frac{5}{6}\log 5\\ \\ &=0.450561\\ H\left( Y\left| X \right. =C \right) &=-p\left( y=1\left| x=C \right. \right) \cdot \log p\left( y=1\left| x=C \right. \right) -p\left( y=2\left| x=C \right. \right) \cdot \log p\left( y=2\left| x=C \right. \right)\\ \,\,\\ &=-\frac{2}{3}\log \frac{2}{3}-\frac{1}{3}\log \frac{1}{3}=\log 3-\frac{2}{3}\log 2\\ \\ &=0.636514\\ H\left( Y\left| X \right. =D \right) &=-p\left( y=1\left| x=D \right. \right) \cdot \log p\left( y=1\left| x=D \right. \right) -p\left( y=2\left| x=D \right. \right) \cdot \log p\left( y=2\left| x=D \right. \right)\\ \,\,\\ &=-\frac{2}{3}\log \frac{2}{3}-\frac{1}{3}\log \frac{1}{3}=\log 3-\frac{2}{3}\log 2\\ \\ &=0.636514\\ \end{aligned}
H(Y∣X=A)H(Y∣X=B)H(Y∣X=C)H(Y∣X=D)=−p(y=1∣x=A)⋅logp(y=1∣x=A)−p(y=2∣x=A)⋅logp(y=2∣x=A)=−75log75−72log72=log7−75log5−72log2=0.59827=−p(y=1∣x=B)⋅logp(y=1∣x=B)−p(y=2∣x=B)⋅logp(y=2∣x=B)=−65log65−61log61=log6−65log5=0.450561=−p(y=1∣x=C)⋅logp(y=1∣x=C)−p(y=2∣x=C)⋅logp(y=2∣x=C)=−32log32−31log31=log3−32log2=0.636514=−p(y=1∣x=D)⋅logp(y=1∣x=D)−p(y=2∣x=D)⋅logp(y=2∣x=D)=−32log32−31log31=log3−32log2=0.636514于是
H ( Y ∣ X ) = p ( x = A ) ⋅ H ( Y ∣ X = A ) + p ( x = B ) ⋅ H ( Y ∣ X = B ) + p ( x = C ) ⋅ H ( Y ∣ X = C ) + p ( x = D ) ⋅ H ( Y ∣ X = D ) = 7 16 × ( log 7 − 5 7 log 5 − 2 7 log 2 ) + 3 8 × ( log 6 − 5 6 log 5 ) + 3 32 × ( log 3 − 2 3 log 2 ) + 3 32 × ( log 3 − 2 3 log 2 ) = 1 16 × ( 7 log 7 + 6 log 6 − 10 log 5 + 3 log 3 − 4 log 2 ) = 0.55005 \begin{aligned} H\left( Y\left| X \right. \right) &=p\left( x=A \right) \cdot H\left( Y\left| X=A \right. \right) +p\left( x=B \right) \cdot H\left( Y\left| X=B \right. \right) +p\left( x=C \right) \cdot H\left( Y\left| X=C \right. \right) +p\left( x=D \right) \cdot H\left( Y\left| X=D \right. \right) \\ \,\,\mathrm{ } \\ &=\frac{7}{16}\times \left( \log 7-\frac{5}{7}\log 5-\frac{2}{7}\log 2 \right) +\frac{3}{8}\times \left( \log 6-\frac{5}{6}\log 5 \right) +\frac{3}{32}\times \left( \log 3-\frac{2}{3}\log 2 \right) +\frac{3}{32}\times \left( \log 3-\frac{2}{3}\log 2 \right) \\ \,\,\mathrm{ } \\ &=\frac{1}{16}\times \left( 7\log 7+6\log 6-10\log 5+3\log 3-4\log 2 \right) \\ &=0.55005 \end{aligned} H(Y∣X)=p(x=A)⋅H(Y∣X=A)+p(x=B)⋅H(Y∣X=B)+p(x=C)⋅H(Y∣X=C)+p(x=D)⋅H(Y∣X=D)=167×(log7−75log5−72log2)+83×(log6−65log5)+323×(log3−32log2)+323×(log3−32log2)=161×(7log7+6log6−10log5+3log3−4log2)=0.55005
例2.4 互信息
p ( x = 0 , y = 0 ) = 1 3 , p ( x = 0 , y = 1 ) = 1 6 , p ( x = 1 , y = 0 ) = 1 6 , p ( x = 1 , y = 1 ) = 1 3 p\left( x=0,y=0 \right) =\frac{1}{3}\text{,}p\left( x=0,y=1 \right) =\frac{1}{6}\text{,}p\left( x=1,y=0 \right) =\frac{1}{6}\text{,}p\left( x=1,y=1 \right) =\frac{1}{3} p(x=0,y=0)=31,p(x=0,y=1)=61,p(x=1,y=0)=61,p(x=1,y=1)=31
则
X
X
X和
Y
Y
Y的互信息为:
I
(
X
;
Y
)
=
p
(
x
=
A
,
y
=
1
)
⋅
log
p
(
x
=
A
,
y
=
1
)
p
(
x
=
A
)
⋅
p
(
y
=
1
)
+
p
(
x
=
A
,
y
=
2
)
⋅
log
p
(
x
=
A
,
y
=
2
)
p
(
x
=
A
)
⋅
p
(
y
=
2
)
+
p
(
x
=
B
,
y
=
1
)
⋅
log
p
(
x
=
B
,
y
=
1
)
p
(
x
=
B
)
⋅
p
(
y
=
1
)
+
p
(
x
=
B
,
y
=
2
)
⋅
log
p
(
x
=
B
,
y
=
2
)
p
(
x
=
B
)
⋅
p
(
y
=
2
)
+
p
(
x
=
C
,
y
=
1
)
⋅
log
p
(
x
=
C
,
y
=
1
)
p
(
x
=
C
)
⋅
p
(
y
=
1
)
+
p
(
x
=
C
,
y
=
2
)
⋅
log
p
(
x
=
C
,
y
=
2
)
p
(
x
=
C
)
⋅
p
(
y
=
2
)
+
p
(
x
=
D
,
y
=
1
)
⋅
log
p
(
x
=
D
,
y
=
1
)
p
(
x
=
D
)
⋅
p
(
y
=
1
)
+
p
(
x
=
D
,
y
=
2
)
⋅
log
p
(
x
=
D
,
y
=
2
)
p
(
x
=
D
)
⋅
p
(
y
=
2
)
=
5
16
×
log
5
16
7
16
×
3
4
+
1
8
×
log
1
8
7
16
×
1
4
+
5
16
×
log
5
16
3
8
×
3
4
+
1
16
×
log
1
16
3
8
×
1
4
+
1
16
×
log
1
16
3
32
×
3
4
+
1
32
×
log
1
32
3
32
×
1
4
+
1
16
×
log
1
16
3
32
×
3
4
+
1
32
×
log
1
32
3
32
×
1
4
=
1
16
×
(
30
log
2
−
21
log
3
+
10
log
5
−
7
log
7
)
=
0.0122853
\begin{aligned} I\left( X;Y \right) &=p\left( x=A,y=1 \right) \cdot \log \frac{p\left( x=A,y=1 \right)}{p\left( x=A \right) \cdot p\left( y=1 \right)}+p\left( x=A,y=2 \right) \cdot \log \frac{p\left( x=A,y=2 \right)}{p\left( x=A \right) \cdot p\left( y=2 \right)} \\ \\ &+p\left( x=B,y=1 \right) \cdot \log \frac{p\left( x=B,y=1 \right)}{p\left( x=B \right) \cdot p\left( y=1 \right)}+p\left( x=B,y=2 \right) \cdot \log \frac{p\left( x=B,y=2 \right)}{p\left( x=B \right) \cdot p\left( y=2 \right)} \\ \\ &+p\left( x=C,y=1 \right) \cdot \log \frac{p\left( x=C,y=1 \right)}{p\left( x=C \right) \cdot p\left( y=1 \right)}+p\left( x=C,y=2 \right) \cdot \log \frac{p\left( x=C,y=2 \right)}{p\left( x=C \right) \cdot p\left( y=2 \right)} \\ \\ &+p\left( x=D,y=1 \right) \cdot \log \frac{p\left( x=D,y=1 \right)}{p\left( x=D \right) \cdot p\left( y=1 \right)}+p\left( x=D,y=2 \right) \cdot \log \frac{p\left( x=D,y=2 \right)}{p\left( x=D \right) \cdot p\left( y=2 \right)} \\ \\ &=\frac{5}{16}\times \log \frac{\small{\frac{5}{16}}}{\frac{7}{16}\times \frac{3}{4}}+\frac{1}{8}\times \log \frac{\frac{1}{8}}{\frac{7}{16}\times \frac{1}{4}}+\frac{5}{16}\times \log \frac{\frac{5}{16}}{\frac{3}{8}\times \frac{3}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{8}\times \frac{1}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{32}\times \frac{3}{4}}+\frac{1}{32}\times \log \frac{\frac{1}{32}}{\frac{3}{32}\times \frac{1}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{32}\times \frac{3}{4}}+\frac{1}{32}\times \log \frac{\frac{1}{32}}{\frac{3}{32}\times \frac{1}{4}} \\ \\ &=\frac{1}{16}\times \left( 30\log 2-21\log 3+10\log 5-7\log 7 \right) \\ &=0.0122853 \end{aligned}
I(X;Y)=p(x=A,y=1)⋅logp(x=A)⋅p(y=1)p(x=A,y=1)+p(x=A,y=2)⋅logp(x=A)⋅p(y=2)p(x=A,y=2)+p(x=B,y=1)⋅logp(x=B)⋅p(y=1)p(x=B,y=1)+p(x=B,y=2)⋅logp(x=B)⋅p(y=2)p(x=B,y=2)+p(x=C,y=1)⋅logp(x=C)⋅p(y=1)p(x=C,y=1)+p(x=C,y=2)⋅logp(x=C)⋅p(y=2)p(x=C,y=2)+p(x=D,y=1)⋅logp(x=D)⋅p(y=1)p(x=D,y=1)+p(x=D,y=2)⋅logp(x=D)⋅p(y=2)p(x=D,y=2)=165×log167×43165+81×log167×4181+165×log83×43165+161×log83×41161+161×log323×43161+321×log323×41321+161×log323×43161+321×log323×41321=161×(30log2−21log3+10log5−7log7)=0.0122853
例2.5 条件互信息
p
(
z
=
0
)
=
1
2
,
p
(
z
=
1
)
=
1
2
p
(
x
=
A
∣
z
=
0
)
=
3
8
,
p
(
x
=
B
∣
z
=
0
)
=
1
4
,
p
(
x
=
C
∣
z
=
0
)
=
3
16
,
p
(
x
=
D
∣
z
=
0
)
=
3
16
p
(
x
=
A
∣
z
=
1
)
=
1
2
,
p
(
x
=
B
∣
z
=
1
)
=
1
2
,
p
(
x
=
C
∣
z
=
1
)
=
0
,
p
(
x
=
D
∣
z
=
1
)
=
0
p
(
y
=
1
∣
z
=
0
)
=
1
2
,
p
(
y
=
2
∣
z
=
0
)
=
1
2
,
p
(
y
=
1
∣
z
=
1
)
=
1
,
p
(
y
=
2
∣
z
=
1
)
=
0
p\left( z=0 \right) =\frac{1}{2},p\left( z=1 \right) =\frac{1}{2} \\ \ \ \\ p\left( \left. x=A \right|z=0 \right) =\frac{3}{8}\text{,}p\left( \left. x=B \right|z=0 \right) =\frac{1}{4}\text{,}p\left( \left. x=C \right|z=0 \right) =\frac{3}{16}\text{,}p\left( \left. x=D \right|z=0 \right) =\frac{3}{16} \\ \ \ \\ p\left( \left. x=A \right|z=1 \right) =\frac{1}{2}\text{,}p\left( \left. x=B \right|z=1 \right) =\frac{1}{2}\text{,}p\left( \left. x=C \right|z=1 \right) =0\text{,}p\left( \left. x=D \right|z=1 \right) =0 \\ \ \ \\ p\left( \left. y=1 \right|z=0 \right) =\frac{1}{2}\text{,}p\left( \left. y=2 \right|z=0 \right) =\frac{1}{2}\text{,}p\left( \left. y=1 \right|z=1 \right) =1\text{,}p\left( \left. y=2 \right|z=1 \right) =0
p(z=0)=21,p(z=1)=21 p(x=A∣z=0)=83,p(x=B∣z=0)=41,p(x=C∣z=0)=163,p(x=D∣z=0)=163 p(x=A∣z=1)=21,p(x=B∣z=1)=21,p(x=C∣z=1)=0,p(x=D∣z=1)=0 p(y=1∣z=0)=21,p(y=2∣z=0)=21,p(y=1∣z=1)=1,p(y=2∣z=1)=0
记
p
01
=
p
(
x
=
A
,
y
=
1
∣
z
=
0
)
=
1
8
,
p
02
=
p
(
x
=
B
,
y
=
1
∣
z
=
0
)
=
1
8
,
p
03
=
p
(
x
=
C
,
y
=
1
∣
z
=
0
)
=
1
8
,
p
04
=
p
(
x
=
D
,
y
=
1
∣
z
=
0
)
=
1
8
p
05
=
p
(
x
=
A
,
y
=
2
∣
z
=
0
)
=
1
4
,
p
06
=
p
(
x
=
B
,
y
=
2
∣
z
=
0
)
=
1
8
,
p
07
=
p
(
x
=
C
,
y
=
2
∣
z
=
0
)
=
1
16
,
p
08
=
p
(
x
=
D
,
y
=
2
∣
z
=
0
)
=
1
16
p
11
=
p
(
x
=
A
,
y
=
1
∣
z
=
1
)
=
1
2
,
p
12
=
p
(
x
=
B
,
y
=
1
∣
z
=
1
)
=
1
2
p_{01}=p\left( \left. x=A,y=1 \right|z=0 \right) =\frac{1}{8}\text{,}p_{02}=p\left( \left. x=B,y=1 \right|z=0 \right) =\frac{1}{8}\text{,}p_{03}=p\left( \left. x=C,y=1 \right|z=0 \right) =\frac{1}{8}\text{,}p_{04}=p\left( \left. x=D,y=1 \right|z=0 \right) =\frac{1}{8} \\ \ \ \\ p_{05}=p\left( \left. x=A,y=2 \right|z=0 \right) =\frac{1}{4}\text{,}p_{06}=p\left( \left. x=B,y=2 \right|z=0 \right) =\frac{1}{8}\text{,}p_{07}=p\left( \left. x=C,y=2 \right|z=0 \right) =\frac{1}{16}\text{,}p_{08}=p\left( \left. x=D,y=2 \right|z=0 \right) =\frac{1}{16} \\ \ \ \\ p_{11}=p\left( \left. x=A,y=1 \right|z=1 \right) =\frac{1}{2}\text{,}p_{12}=p\left( \left. x=B,y=1 \right|z=1 \right) =\frac{1}{2}
p01=p(x=A,y=1∣z=0)=81,p02=p(x=B,y=1∣z=0)=81,p03=p(x=C,y=1∣z=0)=81,p04=p(x=D,y=1∣z=0)=81 p05=p(x=A,y=2∣z=0)=41,p06=p(x=B,y=2∣z=0)=81,p07=p(x=C,y=2∣z=0)=161,p08=p(x=D,y=2∣z=0)=161 p11=p(x=A,y=1∣z=1)=21,p12=p(x=B,y=1∣z=1)=21
P
01
=
p
(
x
=
A
,
y
=
1
∣
z
=
0
)
p
(
x
=
A
∣
z
=
0
)
⋅
p
(
y
=
1
∣
z
=
0
)
=
1
8
3
8
×
1
2
=
2
3
,
P
02
=
p
(
x
=
B
,
y
=
1
∣
z
=
0
)
p
(
x
=
B
∣
z
=
0
)
⋅
p
(
y
=
1
∣
z
=
0
)
=
1
8
1
4
×
1
2
=
1
P
03
=
p
(
x
=
C
,
y
=
1
∣
z
=
0
)
p
(
x
=
C
∣
z
=
0
)
⋅
p
(
y
=
1
∣
z
=
0
)
=
1
8
3
16
×
1
2
=
4
3
,
P
04
=
p
(
x
=
D
,
y
=
1
∣
z
=
0
)
p
(
x
=
D
∣
z
=
0
)
⋅
p
(
y
=
1
∣
z
=
0
)
=
1
8
3
16
×
1
2
=
4
3
P
05
=
p
(
x
=
A
,
y
=
2
∣
z
=
0
)
p
(
x
=
A
∣
z
=
0
)
⋅
p
(
y
=
2
∣
z
=
0
)
=
1
4
3
8
×
1
2
=
4
3
,
P
06
=
p
(
x
=
B
,
y
=
2
∣
z
=
0
)
p
(
x
=
B
∣
z
=
0
)
⋅
p
(
y
=
2
∣
z
=
0
)
=
1
8
1
4
×
1
2
=
1
P
07
=
p
(
x
=
C
,
y
=
2
∣
z
=
0
)
p
(
x
=
C
∣
z
=
0
)
⋅
p
(
y
=
2
∣
z
=
0
)
=
1
16
3
16
×
1
2
=
2
3
,
P
08
=
p
(
x
=
D
,
y
=
2
∣
z
=
0
)
p
(
x
=
D
∣
z
=
0
)
⋅
p
(
y
=
2
∣
z
=
0
)
=
1
16
3
16
×
1
2
=
2
3
P
11
=
p
(
x
=
A
,
y
=
1
∣
z
=
1
)
p
(
x
=
A
∣
z
=
1
)
⋅
p
(
y
=
1
∣
z
=
1
)
=
1
2
1
2
×
1
=
1
,
P
12
=
p
(
x
=
B
,
y
=
1
∣
z
=
1
)
p
(
x
=
B
∣
z
=
1
)
⋅
p
(
y
=
1
∣
z
=
1
)
=
1
2
1
2
×
1
=
1
\begin{aligned} P_{01}&=\frac{p\left( x=A,y=1\left| z=0 \right. \right)}{p\left( x=A\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{3}{8}\times \frac{1}{2}}=\frac{2}{3}\text{,}P_{02}=\frac{p\left( x=B,y=1\left| z=0 \right. \right)}{p\left( x=B\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{1}{4}\times \frac{1}{2}}=1 \\ \,\,\mathrm{ } \\ P_{03}&=\frac{p\left( x=C,y=1\left| z=0 \right. \right)}{p\left( x=C\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{3}{16}\times \frac{1}{2}}=\frac{4}{3}\text{,}P_{04}=\frac{p\left( x=D,y=1\left| z=0 \right. \right)}{p\left( x=D\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{3}{16}\times \frac{1}{2}}=\frac{4}{3} \\ \,\,\mathrm{ } \\ P_{05}&=\frac{p\left( x=A,y=2\left| z=0 \right. \right)}{p\left( x=A\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{4}}{\frac{3}{8}\times \frac{1}{2}}=\frac{4}{3}\text{,}P_{06}=\frac{p\left( x=B,y=2\left| z=0 \right. \right)}{p\left( x=B\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{1}{4}\times \frac{1}{2}}=1 \\ \,\,\mathrm{ } \\ P_{07}&=\frac{p\left( x=C,y=2\left| z=0 \right. \right)}{p\left( x=C\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{16}}{\frac{3}{16}\times \frac{1}{2}}=\frac{2}{3}\text{,}P_{08}=\frac{p\left( x=D,y=2\left| z=0 \right. \right)}{p\left( x=D\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{16}}{\frac{3}{16}\times \frac{1}{2}}=\frac{2}{3} \\ \ \ \\ P_{11}&=\frac{p\left( x=A,y=1\left| z=1 \right. \right)}{p\left( x=A\left| z=1 \right. \right) \cdot p\left( y=1\left| z=1 \right. \right)}=\frac{\frac{1}{2}}{\frac{1}{2}\times 1}=1\text{,}P_{12}=\frac{p\left( x=B,y=1\left| z=1 \right. \right)}{p\left( x=B\left| z=1 \right. \right) \cdot p\left( y=1\left| z=1 \right. \right)}=\frac{\frac{1}{2}}{\frac{1}{2}\times 1}=1 \end{aligned}
P01P03P05P07 P11=p(x=A∣z=0)⋅p(y=1∣z=0)p(x=A,y=1∣z=0)=83×2181=32,P02=p(x=B∣z=0)⋅p(y=1∣z=0)p(x=B,y=1∣z=0)=41×2181=1=p(x=C∣z=0)⋅p(y=1∣z=0)p(x=C,y=1∣z=0)=163×2181=34,P04=p(x=D∣z=0)⋅p(y=1∣z=0)p(x=D,y=1∣z=0)=163×2181=34=p(x=A∣z=0)⋅p(y=2∣z=0)p(x=A,y=2∣z=0)=83×2141=34,P06=p(x=B∣z=0)⋅p(y=2∣z=0)p(x=B,y=2∣z=0)=41×2181=1=p(x=C∣z=0)⋅p(y=2∣z=0)p(x=C,y=2∣z=0)=163×21161=32,P08=p(x=D∣z=0)⋅p(y=2∣z=0)p(x=D,y=2∣z=0)=163×21161=32=p(x=A∣z=1)⋅p(y=1∣z=1)p(x=A,y=1∣z=1)=21×121=1,P12=p(x=B∣z=1)⋅p(y=1∣z=1)p(x=B,y=1∣z=1)=21×121=1
故条件互信息为:
I
(
X
;
Y
∣
Z
)
=
∑
z
∈
Z
p
(
z
)
∑
x
∈
X
∑
y
∈
Y
p
(
x
,
y
∣
z
)
⋅
log
p
(
x
,
y
∣
z
)
p
(
x
∣
z
)
p
(
y
∣
z
)
=
p
(
z
=
0
)
×
[
p
01
⋅
log
P
01
+
p
02
⋅
log
P
02
+
p
03
⋅
log
P
03
+
p
04
⋅
log
P
04
+
p
05
⋅
log
P
05
+
p
06
⋅
log
P
06
+
p
07
⋅
log
P
07
+
p
08
⋅
log
P
08
]
+
p
(
z
=
1
)
×
[
p
11
⋅
log
P
11
+
p
12
⋅
log
P
12
]
=
1
2
×
[
1
8
log
2
3
+
0
+
1
8
log
4
3
+
1
8
log
4
3
+
1
4
log
4
3
+
0
+
1
16
log
2
3
+
1
16
log
2
3
]
+
1
2
×
[
0
+
0
]
=
1
8
×
(
5
log
2
−
3
log
3
)
=
0.0212374
\begin{aligned} I\left( \left. X;Y \right|Z \right) &=\sum_{z\in Z}{p\left( z \right)}\sum_{x\in X}{\sum_{y\in Y}{p\left( \left. x,y \right|z \right)}}·\log \frac{p\left( \left. x,y \right|z \right)}{p\left( \left. x \right|z \right) p\left( \left. y \right|z \right)} \\ \,\,\mathrm{ } \\ &=p\left( z=0 \right) \times \left[ p_{01}\cdot \log P_{01}+p_{02}\cdot \log P_{02}+p_{03}\cdot \log P_{03}+p_{04}\cdot \log P_{04}+p_{05}\cdot \log P_{05}+p_{06}\cdot \log P_{06}+p_{07}\cdot \log P_{07}+p_{08}\cdot \log P_{08} \right] \\ \,\,\mathrm{ } \\ &+p\left( z=1 \right) \times \left[ p_{11}\cdot \log P_{11}+p_{12}\cdot \log P_{12} \right] \\ \,\,\mathrm{ } \\ &=\frac{1}{2}\times \left[ \frac{1}{8}\log \frac{2}{3}+0+\frac{1}{8}\log \frac{4}{3}+\frac{1}{8}\log \frac{4}{3}+\frac{1}{4}\log \frac{4}{3}+0+\frac{1}{16}\log \frac{2}{3}+\frac{1}{16}\log \frac{2}{3} \right] +\frac{1}{2}\times \left[ 0+0 \right] \\ \,\,\mathrm{ } \\ &=\frac{1}{8}\times \left( 5\log 2-3\log 3 \right) =0.0212374 \end{aligned}
I(X;Y∣Z)=z∈Z∑p(z)x∈X∑y∈Y∑p(x,y∣z)⋅logp(x∣z)p(y∣z)p(x,y∣z)=p(z=0)×[p01⋅logP01+p02⋅logP02+p03⋅logP03+p04⋅logP04+p05⋅logP05+p06⋅logP06+p07⋅logP07+p08⋅logP08]+p(z=1)×[p11⋅logP11+p12⋅logP12]=21×[81log32+0+81log34+81log34+41log34+0+161log32+161log32]+21×[0+0]=81×(5log2−3log3)=0.0212374