Asymptotic Equipartition Property

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

AEP

In information theory, the analog of the law of large numbers is the asymptotic equipartition property (AEP). It is a direct consequence of the weak law of large numbers.

For independent identically distributed (i.i.d.) random variables $X_1,\cdots ,X_n$, the weak law of the large numbers states that
$$\frac{1}{n}\sum _{i=1}^n{x}_i\to EX\text{in probability}$$
The AEP states that
$$-\frac{1}{n}\log p(x_1,\cdots ,x_n)\to -E\log p(X)=H(X)\text{in probability}$$
To show this rigorously, we are going to introduce some definitions and theorems.

---

Definition 1:

Given a sequence of i.i.d. random variables $X_i$. We say that the sequence $X_1,X_2,\cdots $ converges to a random variable $X$:

1. In probability if for every $ϵ>0$, Pr ⁡ { ∣ X n − X ∣ > ϵ } → 0 \Pr\{|X_n-X|>\epsilon\}\to 0 Pr{∣Xn​−X∣>ϵ}→0
2. In mean square if $E(X_n-X{)}^2\to 0$
3. With probability 1 (also called almost surely) if Pr ⁡ { lim ⁡ n → ∞ X n = X } = 1 \Pr\{\lim_{n\to \infty}X_n=X\}=1 Pr{limn→∞​Xn​=X}=1

Theorem 1 (Weak law of large numbers):

Given a sequence of i.i.d. random variables $X_i$. Then for any $ϵ>0$ and $\delta >0$, there exists an $n_0$ such that for any $n>n_0$
$$\mathrm{Pr}\left(\left|\frac{1}{n}\sum _{i=1}^n{X}_i-EX\right|<ϵ\right)\ge 1-\delta \text{\hspace{1em}(1)}$$
It is a direct result of Chebyshev Inequality.

Theorem 2 (AEP):

If $X_1,X_2,\cdots$ are i.i.d. $\sim p(x)$, i.e., { X N , n ∈ Z } ∼ p ( x n ) \{ X_N, n\in \mathbb Z \}\sim p(x^n) {XN​,n∈Z}∼p(xn), then for any $ϵ>0$ and $\delta >0$, there exists an $n_0$ such that for any $n>n_0$
$$\mathrm{Pr}\left(\left|-\frac{1}{n}\log p(X_1,\cdots ,X_n)-H(X)\right|<ϵ\right)\ge 1-\delta \text{\hspace{1em}(2)}$$
Proof: By the weak law of large numbers,
$$\begin{array}{rl}-\frac{1}{n}\log p(X_1,\cdots ,X_n)& =-\frac{1}{n}\log \prod _{i=1}^{n}p(X_i)=-\frac{1}{n}\sum _{i=1}^n\log p(X_i)\\ & \to -E\log p(X)\text{in probability}\\ & =H(X)\end{array}$$

---

As a consequence, the probability $p(x_1,\cdots ,x_n)$ of almost all sequences will be close to $2^{-nH(X)}$ when $n$ is large.

Typical Set

We can derive the set of all sequences into two sets, the typical set, where the sample entropy is close to the true entropy (will be explained later), and the non-typical set, which contains the other sequences.

---

Definition 2 (Typical set):

The typical set, denoted by $A_{ϵ}^{(n)}$, is defined by
$$A_{ϵ}^{(n)}=\left\{(x_1,\cdots ,x_n):\left|-\frac{1}{n}\log p(x_1,\cdots ,x_n)-H(X)\right|<ϵ\right\}\text{\hspace{1em}(3)}$$
It is the set of sequences $(x_1,\cdots ,x_n)\in {\mathcal{X}}^n$ having the property
$$2^{-n(H(X)-ϵ)}>p(x_1,\cdots ,x_n)>2^{-n(H(X)+ϵ)}\text{\hspace{1em}(4)}$$
As a consequence of the AEP, we can show that the set $A_{ϵ}^{(n)}$ has the following properties:

Theorem 3 (Properties of typical sets):

1. If $\left(x_1,x_2,\dots ,x_n\right)\in A_{ϵ}^{(n)},$ then $H(X)-ϵ\le -\frac{1}{n}\log p\left(x_1,x_2,\dots x_n\right)\le H(X)+ϵ$.
2. $\mathrm{Pr}\left\{A_{ϵ}^{(n)}\right\}>1-ϵ$ for $n$ sufficiently large.
3. $\left|A_{ϵ}^{(n)}\right|\le 2^{n(H(X)+ϵ)},$ where $|A|$ denotes the number of elements in the set $A$.
4. $\left|A_{ϵ}^{(n)}\right|\ge (1-ϵ)2^{n(H(X)-ϵ)}$ for $n$ sufficiently large.

Thus, the typical set has probability nearly $1$, all elements of the typical set are nearly equiprobable with probability close to $2^{-nH}$, and the number of elements in the typical set is nearly $2^{nH}$.

Proof:

• The proof of property $(1)$ is immediate from the definition of $A_{ϵ}^{(n)}$.

• $\mathrm{Pr}\left\{A_{ϵ}^{(n)}\right\}$ means the probability of the event $(X_1,X_2,\cdots ,X_n)\in A_{ϵ}^{(n)}$. The proof of property $(2)$ follows directly from Theorem 2, since the probability of the event $(X_1,X_2,\cdots ,X_n)\in A_{ϵ}^{(n)}$ tends to $1$ as $n\to \infty$. Thus, for any $\delta >0$, there exists an $n_0$ such that for all $n\ge n_0$, we have

$$\begin{array}{rl}\mathrm{Pr}\left\{A_{ϵ}^{(n)}\right\}& =\mathrm{Pr}\left\{(X_1,X_2,\cdots ,X_n)\in A_{ϵ}^{(n)}\right\}\\ & =\mathrm{Pr}\left\{\left|-\frac{1}{n}\log p(x_1,\cdots ,x_n)-H(X)\right|<ϵ\right\}\\ & >1-\delta \end{array}$$

• To prove property $(3)$, we can use Eq. $(4)$ and write
  $$\begin{array}{rl}1& =\sum _{\mathbf{x}\in {\mathcal{X}}^n}p(\mathbf{x})\ge \sum _{\mathbf{x}\in A_{ϵ}^{(n)}}p(\mathbf{x})\ge \sum _{\mathbf{x}\in A_{ϵ}^{(n)}}{2}^{-n(H(X)+ϵ)}\\ & =2^{-n(H(X)+ϵ)}\left|A_{ϵ}^{(n)}\right|\end{array}$$
  Hence $\left|A_{ϵ}^{(n)}\right|\le 2^{n(H(X)+ϵ)}$.

• The property $(4)$ can be derived from property $(2)$ and Eq. $(4)$. For sufficiently large $n$, Pr ⁡ { A ϵ ( n ) } > 1 − ϵ \Pr\{A_{\epsilon}^{(n)}\}>1-\epsilon Pr{Aϵ(n)​}>1−ϵ, so that
  1 − ϵ < Pr ⁡ { A ϵ ( n ) } = Pr ⁡ { ( X 1 , X 2 , ⋯   , X n ) ∈ A ϵ ( n ) } = ∑ x ∈ A ϵ ( n ) p ( x ) ≤ ∑ x ∈ A ϵ ( n ) 2 − n ( H ( X ) − ϵ ) = 2 − n ( H ( X ) − ϵ ) ∣ A ϵ ( n ) ∣ \begin{aligned} 1-\epsilon&<\Pr\{A_{\epsilon}^{(n)}\}=\operatorname{Pr}\left\{(X_1,X_2,\cdots,X_n)\in A_\epsilon^{(n)}\right\}\\ &=\sum_{\mathbf x\in A_{\epsilon}^{(n)}}p(\mathbf x)\le \sum_{\mathbf x\in A_{\epsilon}^{(n)}}2^{-n(H(X)-\epsilon)}=2^{-n(H(X)-\epsilon)}\left|A_\epsilon^{(n)}\right| \end{aligned} 1−ϵ​<Pr{Aϵ(n)​}=Pr{(X1​,X2​,⋯,Xn​)∈Aϵ(n)​}=x∈Aϵ(n)​∑​p(x)≤x∈Aϵ(n)​∑​2−n(H(X)−ϵ)=2−n(H(X)−ϵ)∣∣∣​Aϵ(n)​∣∣∣​​
  Hence $\left|A_{ϵ}^{(n)}\right|\ge (1-ϵ)2^{n(H(X)-ϵ)}$.

---

Roughly speaking, typical sequences are sequences of which the proportion of occurrences of its alphabet symbols is close to the true probability of occurrence:
$$N(x_i)/n\approx p(x_i)$$
Examples: slides 17-22.

Discussion: We have that $\mathrm{Pr}(A_{ϵ}^{(n)})\to 1$ as $n\to \infty$. Does this imply that sequences in $\overline{A_{ϵ}^{(n)}}$ have lower probability as compared to the ones in $A_{ϵ}^{(n)}$?

$\mathrm{Pr}(A_{ϵ}^{(n)})\to 1$ means that $X_1,X_2,\cdots X_n$ is likely to be in the typical set as $n\to \infty$. But a single typical sequence does not necessarily have the highest probability. For example:

Consider a stochastic process consisting of Bernoulli random variables having probabilities $\mathrm{Pr}(X=0)=1/3$ and $\mathrm{Pr}(X=1)=2/3$.

The most likely sequence (length 6): 1 1 1 1 1 1

A typical sequence (length 6): 1 0 1 1 1 0

High-probability Sets

From the definition of $A_{ϵ}^{(n)}$, it is clear that $A_{ϵ}^{(n)}$ is a fairly small set that contains most of the probability. But from the definition, it is not clear whether it is the smallest high-probability set. We first give a definition of the smallest high-probability set:

Definition 3 (The smallest high-probability set):

For each $n=1,2,\cdots$, let $Q_q^{(n)}\subset {\mathcal{X}}^n$ as the smallest high-probability set with
Pr ⁡ { Q q ( n ) } ≥ 1 − q (5) \Pr\{Q_q^{(n)}\}\ge 1-q \tag{5} Pr{Qq(n)​}≥1−q(5)
We can show that, for sufficiently small $q$, the set $A_{ϵ}^{(n)}$ and $Q_q^{(n)}$ have significant intersection:
Pr ⁡ { A ϵ ( n ) ∩ Q q ( n ) } = Pr ⁡ { A ϵ ( n ) } + Pr ⁡ { Q q ( n ) } − Pr ⁡ { A ϵ ( n ) ∪ Q q ( n ) } > 1 − δ + 1 − q − 1 = 1 − δ − q \begin{aligned} \Pr\{A_\epsilon^{(n)} \cap Q_q^{(n)} \} &=\Pr\{A_\epsilon^{(n)}\}+\Pr\{Q_q^{(n)}\}-\Pr\{A_\epsilon^{(n)} \cup Q_q^{(n)}\}\\ &>1-\delta+1-q-1\\ &=1-\delta-q \end{aligned} Pr{Aϵ(n)​∩Qq(n)​}​=Pr{Aϵ(n)​}+Pr{Qq(n)​}−Pr{Aϵ(n)​∪Qq(n)​}>1−δ+1−q−1=1−δ−q​
And we are going to show that $|A_{ϵ}^{(n)}|$ and $|Q_q^{(n)}|$ are about the same.

Theorem 4:

For any $0<q<1$, we have
$$\frac{1}{n}\log |Q_q^{(n)}|>H(X)-{ϵ}^{\prime }\text{\hspace{1em}(6)}$$
where ${ϵ}^{\prime }$ can be made arbitrarily small.

Proof:
1 − δ − q < Pr ⁡ { A ϵ ( n ) ∩ Q q ( n ) } = ∑ x n ∈ A ϵ ( n ) ∩ Q q ( n ) p ( x n ) < ∑ x n ∈ A ϵ ( n ) ∩ Q q ( n ) 2 − n ( H ( X ) − ϵ ) = ∣ A ϵ ( n ) ∩ Q q ( n ) ∣ 2 − n ( H ( X ) − ϵ ) ≤ ∣ Q q ( n ) ∣ 2 − n ( H ( X ) − ϵ ) \begin{aligned} 1-\delta-q&<\Pr\{A_\epsilon^{(n)} \cap Q_q^{(n)} \} =\sum_{x^n\in A_\epsilon^{(n)} \cap Q_q^{(n)}}p(x^n)\\ &<\sum_{x^n\in A_\epsilon^{(n)} \cap Q_q^{(n)}}2^{-n(H(X)-\epsilon)}=|A_\epsilon^{(n)} \cap Q_q^{(n)}|2^{-n(H(X)-\epsilon)}\\ &\le |Q_q^{(n)}|2^{-n(H(X)-\epsilon)} \end{aligned} 1−δ−q​<Pr{Aϵ(n)​∩Qq(n)​}=xn∈Aϵ(n)​∩Qq(n)​∑​p(xn)<xn∈Aϵ(n)​∩Qq(n)​∑​2−n(H(X)−ϵ)=∣Aϵ(n)​∩Qq(n)​∣2−n(H(X)−ϵ)≤∣Qq(n)​∣2−n(H(X)−ϵ)​

Hence, $|Q_q^{(n)}|>(1-\delta -q)2^{n(H(X)-ϵ)}$.

Thus, $Q_q^{(n)}$ must have at least $2^{nH}$ elements. And $A_{ϵ}^{(n)}$ has $2^{n(H\pm ϵ)}$ elements. Therefore, $A_{ϵ}^{(n)}$ is about the same size as the smallest high-probability set.