Rate Distortion Theory

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

For a stationary discrete source, the minimum number of bits to represent the source signal with arbitrarily small probability of error is given by the entropy rate $H_{\infty }(X)$.

In many situations, however, it is not necessary to perfectly represent the source signal.

For instance, the description of an arbitrary real number requires an infinite number of bits, so a finite representation of a continuous random variable can never be perfect.

How well can we do? $\to$ Define the “goodness” of a representation of a source $\to$ Define a distortion measure

Given a source distribution and distortion measure,

• What is the minimum expected distortion achievable at a particular bit rate? $D(R)$
• What is the minimum rate description required to achieve a particular distortion? $R(D)$

Quantization

Let $\hat{X}(X)$ denote the representation of the random variable $X$. Using $R$ bits to represent $X$, the function $\hat{X}$ can taken on $2^R$ values.

Problem: Find the optimal set of values for $\hat{X}$ and the regions that are associate with each value $\hat{X}$.

An $L$-level quantizer is characterized by a set of $L+1$ decision levels or decision thresholds $x_0<x_1<\cdots <x_L$ and a set X ^ = { x ^ k , k = 1 , ⋯   , L } \hat X=\{ \hat x_k,k=1,\cdots,L \} X^={ x^k​,k=1,⋯,L} such that $\hat{x}={\hat{x}}_k$ if and only if $x_{k-1}\le x<x_k$, where $x_0=-\infty$ and $x_L=\infty$.

The numbers ${\hat{x}}_k$ are called the reconstruction values or reproduction levels and the intervals $C_k=[x_{k-1},x_k)$ are usually referred to as the decision intervals or quantization cells.

The map $\hat{X}:\mathcal{X}\mapsto \hat{\mathcal{X}}$ is given by
$$\hat{X}(x)={\hat{x}}_k\text{for }x\in {\mathcal{C}}_k,k=1,\cdots ,L$$
is a staircase function by definition.

In order to find an optimal quantizer, that is, to find optimal decision and reproduction levels, we need a rule for quantitively assigning a distortion value to every possible approximation of the source samples.

Definition 1 (distortion measure):

A distortion function or distortion measure is a mapping
$$d:\mathcal{X}\times \hat{\mathcal{X}}\mapsto {\mathbb{R}}^{+}$$
from the set of source alphabet-reproduction alphabet pairs into a set of nonnegative numbers. The distortion $d(x,\hat{x})$ is a measure of the cost representing the symbol $x$ by the symbol $\hat{x}$.

Examples:

• Hamming distortion (Probability of error distortion measure)
  $$d(x,\hat{x})=\{\begin{array}{ll}0& \text{ if }x=\hat{x}\\ 1& \text{ if }x\ne \hat{x}\end{array}$$
  $$Ed(X,\hat{X})=\mathrm{Pr}(x=\hat{x})\cdot 0+\mathrm{Pr}(x\ne \hat{x})\cdot 1=\mathrm{Pr}(x\ne \hat{x})$$

• Squared-error distortion
  $$d(x,\hat{x})=(x-\hat{x}{)}^2$$

Assume a squared-error distortion measure. What are the optimal reproduction levels and optimal quantization cells?

That is, we wish to find the function $\hat{X}(X)$ such that $\hat{X}$ takes on at most $L=2^R$ values and minimized $E(X-\hat{X}{)}^2$.
$$E(X-\hat{X}{)}^2=\sum _{k=1}^L{\int }_{{\mathcal{C}}_k}{\left(x-{\hat{x}}_k\right)}^{2}p(x)dx\text{\hspace{1em}(1)}$$

• If the quantization cells ${\mathcal{C}}_k$ are known:

The optimal reproduction levels are found by
$${\frac{\partial E(X-\hat{X}{)}^2}{\partial {\hat{x}}_k}|}_{{\hat{x}}_k={\hat{x}}_k^{\ast }}=-2{\int }_{x\in C_k}\left(x-{\hat{x}}_k^{\ast }\right)p(x)dx=0$$
so that
$${\hat{x}}_k^{\ast }=\frac{{\int }_{x\in C_k}xp(x)dx}{{\int }_{x\in C_k}p(x)dx}$$
Since
$${\int }_{x\in {\mathcal{C}}_k}p(x)dx=\mathrm{Pr}\left(x\in {\mathcal{C}}_k\right)$$
we have, using Bayes’ rule, that
$$\frac{p(x)}{\mathrm{Pr}\left(x\in {\mathcal{C}}_k\right)}=\frac{p\left(x|x\in {\mathcal{C}}_k\right)}{\mathrm{Pr}\left(x\in {\mathcal{C}}_k|x\right)}$$
so that
$${\hat{x}}_k^{\ast }={\int }_{x\in {\mathcal{C}}_k}x\frac{p(x)}{\mathrm{Pr}\left(x\in {\mathcal{C}}_k\right)}dx={\int }_{x\in {\mathcal{C}}_k}x\frac{p\left(x|x\in {\mathcal{C}}_k\right)}{1}dx=E\left(X|x\in {\mathcal{C}}_k\right)\text{\hspace{1em}(2)}$$
It is the conditional mean or centroid of quantisation cell ${\mathcal{C}}_k$.

• If the reproduction levels ${\hat{x}}_k$ are known:

Given a set { x ^ i } \left\{\hat{x}_{i}\right\} { x^i​} of reconstruction points, the distortion is minimized by mapping a source random variable to the representation ${\hat{x}}_i$ that is closest to it. The partition into regions of $\mathcal{X}$ defined by this mapping is called a Voronoi partition.

• The Voronoi regions are determined by the optimal reproduction points, whereas the optimal reproduction points are obtained given the Voronoi regions. How to solve this problem?

Iterative descent algorithm (Lloyd, 1957$)$ :

1. start with an initial collection of reproduction points
2. optimize the partitions for these levels by using a minimum distortion mapping (nearest neighbour quantization)
3. optimize the set of reproduction levels for the given partition (replace the old values by the centroids of the partition cells)

The alternation is continued until convergence to a local, if not global, optimum.

---

Instead of quantizing a single random variable, let us assume that we are given a set of $n$ i.i.d. random variables $X_1,\dots X_n$ drawn from a Gaussian distribution which we want to represent by $nR$ bits

• we will represent the entire sequence by a single index taking $2^{nR}$ values

• this treatment of entire sequences at once achieves a lower distortion for the same rate than independent quantization of the individual samples

  Apparently, rectangular grid points (arising from independent descriptions) do not fill up the space efficiently:

Definition 2 (dimensionless normalized second moment of inertia):

Let $\nu$ denote the volume of a quantization cell. The dimensionless normalized second moment of inertia $G\left({\mathcal{C}}_k\right)$ of a quantization cell is defined by
G ( C k ) = 1 n ν 1 + 2 / n ∫ C k ∥ x − x ^ k ∥ 2 d x G\left(\mathcal{C}_{k}\right)=\frac{1}{n \nu^{1+2 / n}} \int_{\mathcal{C}_{k}}\left\|x-\hat{x}_{k}\right\|^{2} d x G(Ck​)=