Data Compression

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

Problem description:

Let $X_1,X_2,\cdots ,X_n$ be independent, identically distributed random variables drawn from the probability mass function $p(x)$. We wish to find short descriptions for such sequences of random variables.

How to solve?

Assigning short descriptions to the most frequent outcomes of the data source, and necessarily longer descriptions to the less frequent outcomes.

Most frequent outcomes? $\to$ Typical sequences $\to$ AEP $\to$ Data compression

Consequences of The AEP: Data Compression

We divide all sequences in ${\mathcal{X}}^n$ into two sets: the typical set $A_{ϵ}^{(n)}$ and its complement non-typical set $\overline{A_{ϵ}^{(n)}}$.

• Order all elements in $A_{ϵ}^{(n)}$ and $\overline{A_{ϵ}^{(n)}}$ and represent each element by an index
• Since $\left|A_{ϵ}^{(n)}\right|\le 2^{n(H(X)+ϵ)}$, indexing the sequences in $A_{ϵ}^{(n)}$ requires no more than $n(H(X)+ϵ)+1$ bits, the extra bit needed in case $n(H(X)+ϵ)$ is not an integer
• Since $\left|\overline{A_{ϵ}^{(n)}}\right|\le |\mathcal{X}{|}^n$, we can index each sequence in $\overline{A_{ϵ}^{(n)}}$ using no more than $n\log |\mathcal{X}|+$ 1 bits, where $|\mathcal{X}|$ is the cardinality (number of the elements) of the source alphabet
• To distinguish between $A_{ϵ}^{(n)}$ and $\overline{A_{ϵ}^{(n)}}$ we need one additional bit

Note the following features of the coding scheme:

• The typical sequences have short descriptions of length $\approx nH(X)$

• We used a brute force method to enumerate the elements in $\overline{A_{ϵ}^{(n)}},$ without taking into account the fact that the number of elements in $\overline{A_{ϵ}^{(n)}}$ is less than the number of elements in ${\mathcal{X}}^n$

• The code is one-to-one and easily decodable; the initial bit acts as a flag bit to indicate the length of the codeword that follows

We use the notation $x^n$ to denote a sequence $x_1,x_2,\dots ,x_n.$ Let $l\left(x^n\right)$ be the length of the codeword corresponding to $x^n.$ If $n$ is sufficiently large so that $\mathrm{Pr}\left\{A_{ϵ}^{(n)}\right\}\ge 1-\delta ,$ the expected length of the codeword is
$$\begin{array}{rl}E\left(l\left(X^n\right)\right)& =\sum _{x^n}p\left(x^n\right)l\left(x^n\right)\\ & =\sum _{x^n\in A_{ϵ}^{(n)}}p\left(x^n\right)l\left(x^n\right)+\sum _{x^n\in \overline{A_{ϵ}^{(n)}}}p\left(x^n\right)l\left(x^n\right)\\ & \le \sum _{x^n\in A_{ϵ}^{(n)}}p\left(x^n\right)(n(H(X)+ϵ)+2)+\sum _{x^n\in \overline{A_{ϵ}^{(n)}}}p\left(x^n\right)(n\log |\mathcal{X}|+2)\\ & =\mathrm{Pr}\left\{A_{ϵ}^{(n)}\right\}(n(H(X)+ϵ)+2)+\mathrm{Pr}\left\{\overline{A_{ϵ}^{(n)}}\right\}(n\log |\mathcal{X}|+2)\\ & \le (n(H(X)+ϵ)+2)+\delta (n\log |\mathcal{X}|+2)\\ & =n[H(X)+ϵ+\frac{2}{n}+\delta (\log |\mathcal{X}|+\frac{2}{n})]\\ & =n[H(X)+{ϵ}^{\prime }]\end{array}$$
where ${ϵ}^{\prime }$ can be made arbitrarily small by an appropriate choice of $n$. Hence we have proved the following theorem.

Theorem 1:

Let $X^n$ be i.i.d. $\sim p(x)$. Let $ϵ>0.$ Then there exists a code that maps sequences $x^n$ of length $n$ into binary strings such that the mapping is one-to-one (and therefore invertible) and
$$E\left[\frac{1}{n}l\left(X^n\right)\right]\le H(X)+ϵ\text{\hspace{1em}(1)}$$
for $n$ sufficiently large.

Thus, we can represent sequences $X^n$ using $nH(X)$ bits on the average.

How can the probability of error be arbitrary small? And what if the string is not binary?

Theorem 2 (Source coding theorem):

Given a discrete memoryless i.i.d. source { X n , n ∈ Z } ∼ p ( x n ) \{X_n,n\in \mathbb Z \}\sim p(x^n) {Xn​,n∈Z}∼p(xn), we can encode source messages of length $n$ into codewords of length $l$ from a code alphabet of size $r$ with arbitrary probability of error $P_e\le \delta$ if and only if
$$r^l\ge 2^{n(H(X)+ϵ)}\text{\hspace{1em}(2)}$$
Proof:

The number of elements in $A_{ϵ}^{(n)}$ is $|A_{ϵ}^{(n)}|\le 2^{n(H(X)+ϵ)}\le r^l$, so that the number of codewords is larger than the number of typical source words, and $P_e\le \mathrm{Pr}(\overline{A_{ϵ}^{(n)}})\le \delta$ (AEP).

Remark:

• Code construction based on typical sets requires long source sequences $(n\to \infty )$.
• For short sequences, this coding recipe leads to an inefficient representation of the information.
• Can we do better than this for small $n$? (see [Shannon Code](# Shannon Code) and [Huffman Code](#Huffman Code))

Source Codes

Information produced by a discrete information source is represented using the alphabet X = { x 1 , ⋯   , x k } \mathcal X=\{x_1,\cdots, x_k\} X={x1​,⋯,xk​}

Definition 1 (Source code):

A source code $C$ for a random variable $X$ is a mapping $C:\mathcal{X}\mapsto \mathcal{C}$, the set of finite length strings of symbols from a $r$-ary alphabet. $C(x)$ denotes the codeword corresponding to $x$ and its length is denoted by $l(x)$.

Definition 2 (Extension):

The extension $C^{\ast }$ of a code $C$ is the mapping from finite-length strings of $\mathcal{X}$ to finite-length strings of $\mathcal{D}$, defined by
$$C\left(x_1{x}_2\cdots x_n\right)=C\left(x_1\right)C\left(x_2\right)\cdots C\left(x_n\right)$$
where $C\left(x_1\right)C\left(x_2\right)\cdots C\left(x_n\right)$ indicates concatenation of the corresponding codewords.

E.g. If $C\left(x_1\right)=00$ and $C\left(x_2\right)=11,$ then $C\left(x_1{x}_2\right)=0011.$

---

Definition 3 (Non-singular):

A non-singular code is a code that uniquely maps each of the source symbols $x\in \mathcal{X}$ into a code word $C(x)$. That is
$$x_i\ne x_j⟹C(x_i)\ne C(x_j)$$
Definition 4 (Uniquely decodable):

A code is uniquely decodable if and only if its $n$-extension is non-singular for all $n$. That is
{ x 1 , ⋯   , x n } 1 ≠ { x 1 , ⋯   , x n } 2 ⟹ [ C ( x 1 ) , ⋯   , C ( x n ) ] 1 ≠ [ C ( x 1 ) , ⋯   , C ( x n ) ] 2 \{x_1,\cdots,x_n\}_1\ne \{x_1,\cdots,x_n\}_2 \Longrightarrow [C(x_1),\cdots,C(x_n)]_1\ne [C(x_1),\cdots,C(x_n)]_2 {x1​,⋯,xn​}1​​={x1​,⋯,xn​}2​⟹[C(x1​),⋯,C(xn​)]1​​=[C(x1​),⋯,C(xn​)]2​
Definition 5 (Prefix/Instantaneous code):

A code is called a prefix or instantaneous code if no codeword is a prefix of any other codeword. It can be decoded without reference to future codewords since the end of a codeword is immediately recognizable.

Remark: A prefix code can be represented by a $r$-ary tree ($r$ is the size of the alphabet), where the codewords are represented by a leaf of the pruned tree.

The branches of the tree represent the symbols of the codeword. For example, the $r$ branches arising from the root node represent the $r$ possible values of the first symbol of the codeword. Then each codeword is represented by a leaf on the tree. The path from the root traces out the symbols of the codeword. The prefix condition on the codewords implies that no codeword is an ancestor of any other codeword on the tree. Hence, each codeword eliminates its descendants as possible codewords.

---

Examples:

N.B. Morse code is non-singular, uniquely decodable, but not instantaneous.

Kraft Inequality

We wish to construct instantaneous codes of minimum expected length to describe a given source. It is clear that we cannot assign short codewords to all source symbols and still be prefix-free. The set of codeword lengths possible for instantaneous codes is limited by the following inequality.

Theorem 3 (Kraft inequality):

For any instantaneous code over an alphabet of size $r$, the codewords lengths $l(x_1),\cdots ,l(x_k)$ must satisfy the inequality
$$\sum _{i=1}^k{r}^{-l(x_i)}\le 1\text{\hspace{1em}(3)}$$
Conversely, given a set of codeword lengths that satisfy this inequality, there exists an instantaneous code with these word lengths.

Proof:

Let $l_{\max }$ be the longest codeword. A codeword at level $l\left(x_i\right)$ has $r^{l_{\max }-l\left(x_i\right)}$ descendants at level $l_{\max }.$ In order to be a prefix code, each of these descendent sets must be disjoint and the total number of nodes in these sets is less than $r^{l_{\max }}$. Hence, summing over all the codewords, we obtain
$$\sum _{i=1}^k{r}^{l_{\max }-l\left(x_i\right)}\le r^{l_{\max }}$$
and thus
$$\sum _{i=1}^k{r}^{-l\left(x_i\right)}\le 1$$
N.B. For any countably infinite set of codewords that form a prefix code, the codeword lengths also satisfy the Kraft inequality, i.e.,
$$\sum _{i=1}^{\infty }{r}^{-l(x_i)}\le 1$$

Optimal Codes

What is the minimum expected length of the prefix code? How do we find the prefix code with minimum expected length?

Bounds on the optimal code length

Consider a constrained optimization problem:
$$\underset{l(x_i)}{\min }\sum _{i=1}^{k}p(x_i)l(x_i)\text{subject to }\sum _{i=1}^k{r}^{-l(x_i)}\le 1\text{\hspace{1em}(4)}$$
Lagrange multiplier technique:
$$\underset{l\left(x_i\right)}{\min }J\left(l\left(x_i\right),\lambda \right)=\underset{l\left(x_i\right)}{\min }\left(\sum _{i=1}^{k}p\left(x_i\right)l\left(x_i\right)+\lambda \left(\sum _{i=1}^k{r}^{-l\left(x_i\right)}-1\right)\right)$$
Hence
$$\begin{array}{l}\frac{\partial J}{\partial l\left(x_i\right)}=p\left(x_i\right)-\lambda r^{-l\left(x_i\right)}\ln r\Rightarrow r^{-l^{\ast }\left(x_i\right)}=\frac{p\left(x_i\right)}{\lambda \ln r}\\ {\sum }_{i=1}^k{r}^{-l^{\ast }\left(x_i\right)}\le 1\Rightarrow \lambda \ln r\ge 1\Rightarrow l^{\ast }\left(x_i\right)\ge -{\log }_{r}p\left(x_i\right)\end{array}$$
The average codelength, $El(X),$ then becomes
$$\sum _{i=1}^{k}p\left(x_i\right)l^{\ast }\left(x_i\right)\ge -\sum _{i=1}^{k}p\left(x_i\right){\log }_{r}p\left(x_i\right)=H(X)$$
As a consequence, we have that for any instantaneous code
$$El(X)\ge H(X)\text{\hspace{1em}(5)}$$
with equality iff $r^{-l^{\ast }\left(x_i\right)}=p\left(x_i\right)$.

In the case that $-\log p\left(x_i\right)$ is not integer, we should choose a set of codeword lengths “close” to the optimal set. Shannon suggested to round it up to the nearest integer
$$-\log p\left(x_i\right)\le l\left(x_i\right)<-\log p\left(x_i\right)+1$$
This choice satisfies Kraft’s inequality and we conclude that the optimal codeword length for a given source distribution satisfies
$$H(X)\le El(X)\le H(X)+1\text{\hspace{1em}(6)}$$

• There is an overhead which at most 1 bit per symbol due to the fact that $\log p\left(x_i\right)$ is not always an integer
• Overhead can be reduced by combining symbols into sequences

Encoding of sequences of length $n$ :
$$H\left(X_1,\dots ,X_n\right)\le El\left(X_1,\dots ,X_n\right)<H\left(X_1,\dots ,X_n\right)+1$$
Define $L_n$ to be the expected codeword length per input symbol, that is
$$L_n=\frac{1}{n}El(X_1,\cdots ,X_n)\text{\hspace{1em}(7)}$$
Assuming symbols are drawn i.i.d. according to $p\left(x^n\right),$ we have that $H\left(X_1,\dots ,X_n\right)=nH(X)$ and we conclude that
$$H(X)\le L_n<H(X)+\frac{1}{n}\text{\hspace{1em}(8)}$$

This equation relates to $(1)$.

For a sequence of symbols that is not necessarily i.i.d., we have the bound
$$\frac{H\left(X_1,\dots ,X_n\right)}{n}\le L_n<\frac{H\left(X_1,\dots ,X_n\right)}{n}+\frac{1}{n}\text{\hspace{1em}(9)}$$
For stationary processes, we have the entropy rate
$$H_{\infty }(X)=\underset{n\to \infty }{\lim }\frac{H(X_1,\cdots ,X_n)}{n}$$
Therefore, $L_n\to H_{\infty }(X)$ for $n\to \infty$, which provides another justification for the definition of entropy rate - it is the expected number of bits per symbol required to describe the process.

---

Example:

---

Do there exist uniquely decodable, non-instantaneous codes that achieve shorter expected codelengths?

We have the following result (by McMillan):

The codeword lengths of any uniquely decodable code must satisfy the Kraft inequality.

This rather surprising result implies that the class of uniquely decodable codes does not offer any further choices for the set of codeword lengths than the class of prefix codes!

Shannon Code

In Shannon coding, the symbols are arranged in order from most probable to least probable, and assigned codewords by taking the first $l_i=\lceil -\log p(x_i)\rceil$ bits from the binary expansions of the cumulative probabilities $F(x_i)=\sum _{j=1}^{i-1}p(x_j)$

Example:

Shannon code is asymptotically optimal as $n\to \infty$. However, the Shannon code may be much worse than the optimal code for finite $n$ for some particular symbol.

For example, let $p(x_1)=0.99$ and $p(x_2)=0.01$. Obviously, an optimal code is $C(x_1)=0$ and $C(x_2)=1$. The Shannon code, though asymptotically optimal, assigns a codeword of length $\lceil \log 100\rceil =7$ to $x_2$. Note that in this case $H(X)=0.08$ and $1<El(x)=1.06<H(X)+1$.

Huffman Code

Theorem 4 (Huffman coding):

Huffman coding is optimal, i.e., if $C^{\ast }$ is a Huffman code and $C^{\prime }$ is any other code, then $El(C^{\ast })\le El(C^{\prime })$.

Remarks:

• The lengths are ordered inversely with the probabilities
• The two longest codewords have the same length
• Two of the longest codewords differ only in the last bit and correspond to the two least likely symbols

How to construct Huffman codes:

• For the binary case, the Huffman code arranges the messages in order of decreasing probability, and joins the two least probable source symbols together, resulting in a new message alphabet with one less symbol.
• The new messages are reordered, after which two symbols are again joined together
• Repeat until the last two symbols added to one
• Every time when adding, assign 0 and 1 to the two added probabilities
• Start with a symbol and go to the last “1”. Combine the 0 and 1 encountered on the route and flip the result.

Example:

N.B. The result is not unique. It can also be constructed as

Observations:

• Huffman coding is not ideal since it is an bottom-up approach that requires the calculation of the probabilities of all source sequences and the construction of the corresponding complete code tree.
• Cannot easily extended to longer block length without having to redo all the calculations

Arithmetic Coding

The Huffman coding procedure described is optimal for encoding a random variable with a known distribution that has to be encoded symbol by symbol. However, due to the fact that the codeword lengths for a Huffman code were restricted to be integral, there could be a loss of up to 1 bit per symbol in coding efficiency. We could alleviate this loss by using blocks of input symbols—however, the complexity of this approach increases exponentially with block length. We now describe a method of encoding without this inefficiency. In arithmetic coding, instead of using a sequence of bits to represent a symbol, we represent it by a subinterval of the unit interval.

N.B. Huffman code is still the optimal in the sense that we encode the whole sequence, not symbol by symbol.

[Encoding and Decoding details: Slides 40-52]