信息科学原理第一章(香农熵，条件熵，相对熵)

@(信息科学原理)

导论

香农熵

信息：$h(x)=-\log p(x)$

$$\begin{split} H(X,Y)= -\sum_{x\in X}P(x) \log(P(x)) \\ =E_{x \sim P}log(P(x)) \end{split}$$

其中$0\log 0=0$，并且定义$\log \frac{1}{e}=1 nats$和 $\log \frac {1}{2}=1 bits$

联合熵

$$\begin{array} \large{H}(X,Y) = -\sum_{x\in X,y\in Y}P(x,y)\log P(x,y)\\ \quad = E_{x\sim P}\log P(x,y) \end{array}$$

互信息

$$\begin{array} \large{I}(X,Y)=\sum_{x\in X,y\in Y}P(x, y) \log \frac {P(x, y)}{P(X)P(Y)}\\=E_{x,y \sim P}\log \frac {P(x, y)}{P(X)P(Y)}\\=D_{KL}\left(P(x,y)\mid \mid P(X)P(Y)\right) \end{array}$$

衡量两个信息的相关性大小的量

条件熵

$$\begin{array} \large{H}(Y|X) = -\sum_{x\in X,y\in Y}P(x,y)\log P(y|x) \\ \quad\quad = -\sum_{x\in X,y\in Y}P(x,y)\log \frac{P(x,y)}{P(x)}\\ \quad\quad =\sum_{x\in X,y\in Y}P(x,y)\log \frac {P(x)}{P(x,y)}\\ \quad\quad=E_{x,y \sim P} log \frac {P(x)}{P(x,y)} \end{array}$$
知道的信息越多，随机事件的不确定性就越小

proof: $H(X,Y)=H(X)+H(Y|X)$ :

$$\begin{array} \large{H}(X,Y) = -\sum_{x\in X,y\in Y}P(x,y)\log P(x,y)\\ \quad\quad = -\sum_{x\in X,y\in Y}P(x,y)\log\left[P(y|x)P(x)\right]\\ \quad\quad = -\sum_{x\in X,y\in Y}P(x,y)\left[\log P(y|x)+\log P(x)\right]\\ \quad\quad = -\sum_{x\in X,y\in Y}P(x,y)\log P(y|x)+[-\sum_{x\in X}P(x)\log P(x)]\\ \quad\quad = H(Y|X)+H(x) \end{array}$$
proof: $H(X,Y|Z)=H(X|Z)+H(Y|X,Z)$
$$\begin{array} \large{H}(X,Y|Z) = -\sum_{x,y,z}P(x,y,z)\log P(x,y|z)\\ \quad\quad = -\sum_{x,y,z}P(x,y,z)\log\left[\frac{P(x,y,z)}{P(z)}\right]\\ \quad\quad = -\sum_{x,y,z}P(x,y,z)\log\left[\frac{P(x,y,z)}{P(x,z)}\frac{P(x,z)}{p(z)}\right]\\ \quad\quad=\left[-\sum_{x,y,z}P(x,y,z)\log\frac{P(x,y,z)}{P(x,z)}\right]+\left[-\sum_{x,y,z}P(x,y,z)\log\frac{P(x,z)}{P(z)}\right]\\ \quad\quad=\left[-\sum_{x,y,z}P(x,y,z)\log\frac{P(x,y,z)}{P(x,z)}\right]+\left[-\sum_{x,z}P(x,z)\log\frac{P(x,z)}{P(z)}\right]\\ \quad\quad=H(Y|X,Z)+H(X|Z) \end{array}$$

相对熵(KL-散度)

$$\begin{array} \large{D}_{KL}(P\mid\mid Q)\\ \quad\quad =\sum_{x\in X}P(x)\log \frac {P(x)}{Q(x)}\\ \quad\quad = E_{x\sim P}\left[\log\frac {P(x)}{Q(x)}\right]\\ \quad\quad = E_{x\sim P}\left[\log P(x)-\log Q(x)\right] \end{array}$$

note:$D_{KL}(P\mid\mid Q) \geq 0$,用于衡量两个分布的相似性

交叉熵

$$\begin{array} \large{H}(P,Q)=H(P) +D_{KL}(P\mid \mid Q)\\ H(P,Q) = -E_{x\sim P}\log Q(x) \end{array}$$

边缘概率，条件概率，联合概率

- 边缘概率就是计算每一边
- 联合概率计算的是$P(X=x,Y=y)=P(y|x)P(x)$

- 条件概率计算的是$P(y|x)=\frac{P(x,y)}{P(x)}$

对于离散的随机变量：

对于连续的随机变量：

example

$$\begin{array} \larget{H}(X)=-\sum_{x\in X} P(x)\log p(x)\\ \quad \quad =\frac{1}{2}\log 2+\frac{1}{4}\log 4+\frac{1}{8}\log 8+\frac{1}{8}\log 8\\ \quad\quad = \frac{7}{4}\log2=\frac{7}{4} bits \end{array}$$
$$\begin{array} \larget{H}(X|Y)=-\sum_{x\in X y\in Y}P(x,y)log\frac {P(x,y)}{P(y)}\\ \quad \quad=\frac{4}{32}\log \frac{1/4}{4/32}+\frac{2}{32}\log \frac{1/4}{2/32}+\frac{2}{32}\log \frac{1/4}{2/32} +\cdot\cdot\cdot\\ \quad\quad = \frac{11}{8}bits \end{array}$$
$$\begin{array} \larget{H}(X,Y)=-\sum_{x\in X y\in Y}P(x,y)logP(x,y)\\ \quad\quad=\frac{27}{8}bits \end{array}$$