Information Measures

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

How to measure information? $\to$ Hartley’s approach [slides 5] $\to$ Problems?

no allowance is made for the fact that

• the $n$ alphabet elements have unequal probability of occurrence
• and that there could be a possible dependence between successive symbols

$\to$ Shannon’s method: associating information with uncertainty using the concept of probability

Entropy

Definition 1 (Entropy):

Let $X$ be a discrete random variable with alphabet $\mathcal{X}$ and probability mass function $p(x)=\mathrm{Pr}(X=x),x\in \mathcal{X}$. The entropy $H(X)$ is defined by
$$H(X)=-\sum _{x\in \mathcal{X}}p(x)\log p(x)=-E\log p(X)\text{\hspace{1em}(1)}$$
Sometimes we will use the notation $H(p)$. (Define $-p(x)\log p(x)=0$ if $p(x)=0$)

Interpretation:

• a measure of the uncertainty of a random variable
• the minimum number of ‘yes/no’ questions needed to determine which of the alphabet symbols $x\in \mathcal{X}$ occurred on average [slides 9,10, Ex 2.1]
• the minimum number of bits to represent the outcome of an experiment (Shannon’s first coding theorem) [slides 12-14]
• Why logarithm? To make it growing linearly with system size and “behaving like information”.

Properties:

• $H(X)\ge 0$ with equality iff $\exists i$ such that $p(x_i)=1$.

• $H(X)$ is a concave function of $p(x)$ (since ${\nabla }^{2}H⪯0$)

• $H(X)\le \log k$ with equality iff $p(x_i)=1/k$ for all $i$ (Lagrangian method)

  Entropy is maximum for a uniform distribution. (Maximum uncertainty)

Definition 2 (Joint Entropy):

Let $(X,Y)$ be a pair of discrete random variable with alphabet $\mathcal{X}$ and $\mathcal{Y}$, respectively, and joint probability mass function $p(x,y)=\mathrm{Pr}(X=x,Y=y),x\in \mathcal{X},y\in \mathcal{Y}$. The joint entropy $H(X,Y)$ is defined by
$$H(X,Y)=-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)\log p(x,y)=-E\log p(X,Y)\text{\hspace{1em}(2)}$$

Definition 3 (Conditional Entropy):

What is the entropy of $Y$ if we know $X=x$?
$$H(Y|X=x)=-\sum _{y\in \mathcal{Y}}p(y|x)\log p(y|x)$$
By averaging over all values $x\in \mathcal{X}$, the average amount of information of $Y$ given foreknowledge of $X$ is found
$$H(Y|X)=\sum _{x\in \mathcal{X}}p(x)H(Y|X=x)=-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)\log p(y|x)=-E\log p(Y|X)\text{\hspace{1em}(3)}$$

The naturalness of the definition of joint entropy and conditional entropy is exhibited by the fact that the entropy of a pair of random variables is the entropy of one plus the conditional entropy of the other.

Theorem 1 (Chain Rule for Entropy):
$$H(X,Y)=H(X)+H(Y|X)=H(Y)+H(X|Y)\text{\hspace{1em}(4)}$$
Proof:
$$H(X)=-\sum _{x\in \mathcal{X}}p(x)\log p(x)=-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)\log p(x)$$

$$\begin{array}{rl}H(X)+H(Y|X)& =-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)[\log p(x)+\log p(y|x)]\\ & =-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)\log p(x,y)\\ & =H(X,Y)\end{array}$$

Remarks:

• $H(X|Y)\ne H(Y|X)$ unless $H(X)=H(Y)$

• Conditioning decreases the entropy (uncertainty): $H(Y|X)\le H(Y)$

  Intuitively, if $X,Y$ are related, then we will know some information of $X$ if we observe $Y$. Therefore, the uncertainty is decreased.

  Mathematically, that’s to prove $-{\sum }_{x\in \mathcal{X}}{\sum }_{y\in \mathcal{Y}}p(x,y)\log p(y|x)\le -{\sum }_{x\in \mathcal{X}}{\sum }_{y\in \mathcal{Y}}p(x,y)\log p(y)$, i.e.,
  $$E_{XY}\log \frac{p(X)p(Y)}{p(X,Y)}\le 0$$
  This is true by applying the first order condition for concave functions (Jensen’s Inequality):
  $$E_{XY}\log \frac{p(X)p(Y)}{p(X,Y)}\le \log \left(E_{XY}\frac{p(X)p(Y)}{p(X,Y)}\right)=\log 1=0$$

• As a consequence, $H(X,Y)\le H(X)+H(Y)$ with equality iff $p(x,y)=p(x)p(y)$

Generalization:
$$H(X_1,\cdots ,X_n)=\sum _{i=1}^{n}H(X_i|X_{i-1},\cdots ,X_1)\le \sum _{i=1}^{n}H(X_i)$$

Proof:
$$\begin{array}{rl}H(X_1,\cdots ,X_n)& =-\sum _{x_1,\cdots ,x_n}p(x_1,\cdots ,x_n)\log p(x_1,\cdots ,x_n)\\ & =-\sum _{x_1,\cdots ,x_n}p(x_1,\cdots ,x_n)\log \prod _{i=1}^{n}p(x_i|x_{i-1},\cdots ,x_1)\\ & =-\sum _{i=1}^n\sum _{x_1,\cdots ,x_n}p(x_1,\cdots ,x_n)\log p(x_i|x_{i-1},\cdots ,x_1)\\ & =-\sum _{i=1}^n\sum _{x_1,\cdots ,x_i}p(x_1,\cdots ,x_i)\log p(x_i|x_{i-1},\cdots ,x_1)\\ & =\sum _{i=1}^{n}H(X_i|X_{i-1},\cdots ,X_1)\end{array}$$

Mutual Information

The mutual information is a measure of the amount of information that one random variable contains about another random variable.

Definition 4 (Mutual Information):

Given a pair of random variables $(X,Y)$ with joint probability mass functions $p(x,y)$ and marginal probability mass functions $p(x)$ and $p(y)$, respectively. The mutual information is defined by
$$I(X;Y)=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)\log \frac{p(x,y)}{p(x)p(y)}\text{\hspace{1em}(5)}$$
Corollary:

• $I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$

  The mutual information is the reduction in the uncertainty of one random variable due to the knowledge of other.

• $I(X;Y)=I(Y;X)$

• $I(X;Y)=H(X)+H(Y)-H(X,Y)$

• $I(X;X)=H(X)$

Properties:

• $I(X;Y)\ge 0$ with equality iff $X$ and $Y$ are independent
• $I(X;Y)\le H(X)$ with equality iff $X$ and $Y$ are completely dependent, i.e., $\forall i\exists j:p(x_i|y_j)=1$
• $I(X;Y)$ is a concave function of $p(x)$ for fixed $p(y|x)$ and a convex function of $p(y|x)$ for fixed $p(x)$. [book P33]

Definition 5 (Conditional Mutual Information):

Given the random variables $(X,Y,Z)$ with joint probability mass functions $p(x,y,z)$ and marginal probability mass functions $p(x)$, $p(y)$ and $p(z)$, respectively. The conditional mutual information $I(X;Y|Z)$ is defined by
$$\begin{array}{rl}I(X;Y|Z)& =\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}\sum _{z\in \mathcal{Z}}p(x,y,z)\log \frac{p(x,y|z)}{p(x|z)p(y|z)}\\ & =H(X|Z)-H(X|Y,Z)\end{array}\text{\hspace{1em}(6)}$$
Interpretation:

• The conditional mutual information is the reduction in uncertainty of $X$ due to knowledge of $Y$ when $Z$ is given.
• After introducing the conditional mutual information, we are able to formulate the chain rule for mutual information.

Theorem 2 (Chain Rule for Mutual Information):

Let $(X_1,\cdots ,X_n,Y)$ be a collection of random variables with joint probability mass function $p(x_1,\cdots ,x_n,y)$. Then
$$I(X_1,\cdots ,X_n;Y)=\sum _{i=1}^{n}I(X_i;Y|X_{i-1},\cdots ,X_1)\text{\hspace{1em}(7)}$$
Proof: Since mutual information can be expressed in terms of entropies, it satisfies a chain rule as well.
$$\begin{array}{l}I\left(X_1,X_2,\dots ,X_n;Y\right)\\ =H\left(X_1,X_2,\dots ,X_n\right)-H\left(X_1,X_2,\dots ,X_n|Y\right)\\ ={\sum }_{i=1}^{n}H\left(X_i|X_{i-1},\dots ,X_1\right)-{\sum }_{i=1}^{n}H\left(X_i|X_{i-1},\dots ,X_1,Y\right)\\ ={\sum }_{i=1}^{n}I\left(X_i;Y|X_1,X_2,\dots ,X_{i-1}\right)\end{array}$$

Entropy Rates

A discrete information source is a source emitting a sequence of symbols from a fixed finite alphabet according to a probability law. It can be modelled by a stochastic process, which is a sequence of random variables characterised by a joint probability function
$$p(x_1,\cdots ,x_n)=\mathrm{Pr}((X_1,\cdots ,X_n)=(x_1,\cdots ,x_n)),\forall n$$
If the random variables are mutually independent, that is $p(x_1,\cdots ,x_n)={\prod }_{i}p(x_i)$, we say that the process is memoryless. An information source having memory can be described by a Markov process.

Definition 6 (Markov Process):

A discrete stochastic process is said to be a Markov process of order $m$, if for $n=1,2,\cdots$
$$\mathrm{Pr}(X_n=x_n|X_{n-1}=x_{n-1},\cdots ,X_1=x_1)=\mathrm{Pr}(X_n=x_n|X_{n-1}=x_{n-1},\cdots ,X_{n-m}=x_{n-m})\text{\hspace{1em}(8)}$$
i.e., a Markov process of order $m$ is a stochastic process where each random variable depends on the $m$ preceding variables and is conditionally independent of all other preceding random variables.

Remarks:

• The conditional probability of a value $x_n$ of the random variable $X_n$, given all preceding values, is equal to
  $$p(x_n|x_{n-1},\cdots ,x_1)=p(x_n|x_{n-1},\cdots ,x_{n-m})$$

• The values $S_i=(x_{n-1},\cdots ,x_{n-m})$ form the conditional part of the conditional probability distribution of $X_n$. $S_i$ is called the state of the Markov chain.

• Besides giving the conditional probability distribution of $X_n$, one can also characterize the Markov process by giving a probability transition matrix containing the transition probabilities for the different states. [slides 40-43, Ex 4.7]

How do we measure the information of a process with memory?

If we have a sequence of $n$ random variables, a natural question to ask is: How does the entropy of the sequence grow with $n$? We define the entropy rate as this rate of growth as follows.

Definition 7 (Entropy Rate):

The entropy of a stochastic process { X i } \{X_i\} {Xi​} is defined by
$$H_{\infty }(X)=\underset{n\to \infty }{\lim }\frac{1}{n}H(X_1,X_2,\cdots ,X_n)\text{\hspace{1em}(9)}$$
when the limit exists.

Remarks:

• Independent identically distributed (i.i.d.) random variables
  $$H_{\infty }(X)=\underset{n\to \infty }{\lim }\frac{1}{n}nH(X_1)=H(X_1)$$
  The entropy rate of the stochastic process { X i } \{X_i\} {Xi​} equals to the entropy of the random variable $X_i$.

• We can also define a related quantity for entropy rate
  $$H_{\infty }^{\prime }(X)=\underset{n\to \infty }{\lim }H(X_n|X_{n-1},\cdots ,X_1)\text{\hspace{1em}(10)}$$
  when the limit exists.

• $H_{\infty }(X)$ is the per symbol entropy of $n$ random variables, while $H_{\infty }^{\prime }(X)$ is the conditional entropy of the last random variable given the past.

• For stationary processes, $H_{\infty }(X)=H_{\infty }^{\prime }(X)$. [book P75]

• For stationary Markov process of order $m$, the entropy is given by
  $$H_{\infty }(X)=H_{\infty }^{\prime }(X)=H(X_n|X_{n-1},\cdots ,X_{n-m})$$