Distributed Source Coding

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

Introduction

• We know how to encode a source $X$. A rate $R\ge H(X)$ is sufficient.

• If there are two sources $(X,Y)$, a rate $R\ge H(X,Y)$ is sufficient.

• But what if the $X$ and $Y$ sources must be described separately for some user who wishes to reconstruct both $X$ and $Y$?

• Clearly, by separately encoding $X$ and $Y$, it is seen that a rate $R=R_x+R_y\ge H(X)+H(Y)$ is sufficient.

• However, in a surprising and fundamental paper by Slepian and Wolf, it is shown that a total rate $R=H(X,Y)$ is sufficient even for separate encoding of correlated sources.

• Intuitively, since $H(X,Y)=H(X)+H(Y|X)$, we can first encode source $X$ at a rate $R_1\ge H(X)$ after which we encode source $Y$, given $X$, at a rate $R_2\ge H(Y|X)$.

  More specifically,

  • Using $nH(X)$ bits we can encode $X^n$ efficiently, so that the decoder can reconstruct $X^n$ with arbitrarily low probability of error
  • Associated with every $x^n$ is a typical “fan” of $y^n$ sequences that are jointly typical with the given $x^n,2^{nH(Y|X)}$ in total
  • The encoder can send the index of the $y^n$ within this typical fan for which he needs $nH(Y|X)$ bits
  • The decoder, also knowing $x^n$, can then construct the typical fan and hence reconstruct $y^n$

The graph of the whole process can be presented as

• But what if the $Y$ encoder does not know which sequence $x^n$ is encoded?

Slepian-Wolf Coding

Let $\left(X_1,Y_1\right),\left(X_2,Y_2\right),\dots$ be a sequence of jointly distributed random variables i.i.d. $\sim p(x,y)$

Definition 1 (Distributed source code):

A $\left(\left(2^{n{R}_1},2^{n{R}_2}\right),n\right)$ distributed source code for the joint sources $(X,Y)$ consists of two encoder maps
$$\begin{array}{l}{f}_1:{\mathcal{X}}^n\to \left\{1,2,\dots ,2^{n{R}_1}\right\}\\ f_2:{\mathcal{Y}}^n\to \left\{1,2,\dots ,2^{n{R}_2}\right\}\end{array}$$
and a decoder map
$$g:\left\{1,2,\dots ,2^{n{R}_1}\right\}\times \left\{1,2,\dots ,2^{n{R}_2}\right\}\to {\mathcal{X}}^n\times {\mathcal{Y}}^n$$
Definition 2 (Probability of error):

The probability of error for a distributed source code is defined as
$$P_{ϵ}^{(n)}=\mathrm{Pr}\left(g\left(f_1\left(X^n\right),f_2\left(Y^n\right)\right)\ne \left(X^n,Y^n\right)\right)$$
Definition 3 (Achievable):

A rate pair $\left(R_1,R_2\right)$ is said to be achievable for a distributed source if there exists a sequence of $\left(\left(2^{n{R}_1},2^{n{R}_2}\right),n\right)$ distributed source codes with probability of error $P_{ϵ}^{(n)}\to 0$. The achievable rate region is the closure of the set of achievable rates.

Theorem 1 (Slepian-Wolf):

For a distributed source coding problem for the source $(X,Y)$ drawn i.i.d. $\sim p(x,y)$, the achievable rate region is given by
$$\begin{array}{rl}{R}_1& \ge H(X|Y)\\ R_2& \ge H(Y|X)\\ R_1+R_2& \ge H(X,Y)\end{array}$$

[Example, Slides 17-18]

Random Binning

It is an encoding and decoding scheme that enables $(R_1,R_2)$ to be achievable when $R_1+R_2=H(X,Y)$ even if the $Y$ encoder does not know which sequence $x^n$ is encoded.

Encoding and Decoding Scheme

Encoding:

• For each sequence $x^n$, draw an index at random from $\left\{1,2,\dots ,2^{nR}\right\}$.
• Sequences $x^n$ having the same index are said to form a bin.

Decoding:

• Given a bin index, we look for a typical $x^n$ sequence in the bin
• If there is one and only one typical $x^n$ in the bin, we declare it to be the estimate ${\hat{x}}^n$; otherwise an error is declared

Error: Given a bin index

• the sequence in the bin is non-typical
• there is more than one typical sequence in the bin

We first prove that under this scheme, if $R\ge H(X)$, the probability of error is arbitrarily small and the code achieves the same result as the code introduced by Shannon (typical set coding).
$$\begin{array}{rl}\mathrm{Pr}\left(g\left(f\left(X^n\right)\right)\ne X^n\right)& =\mathrm{Pr}\left({\bar{A}}_{ϵ}^{(n)}\right)+\sum _{x^n\in A_{ϵ}^{(n)}}p\left(x^n\right)\mathrm{Pr}\left(\exists {\tilde{x}}^n\ne x^n:f\left({\tilde{x}}^n\right)=f\left(x^n\right)\right)\\ & \le ϵ+\sum _{x^n\in A_{ϵ}^{(n)}}p\left(x^n\right)\sum _{{\tilde{x}}^n\in A_{ϵ}^{(n)}}{2}^{-nR}\\ & \le ϵ+2^{n(H(X)+ϵ)}{2}^{-nR}\\ & \le {ϵ}^{\prime }\end{array}$$
if $R>H(X)+ϵ$ and $n$ sufficiently large.

Remarks:

• The binning scheme does not require an explicit characterization of the typical set at the encoder; it is needed only at the decoder
• It is this property that enables this code to continue to work in the case of a distributed source

Outline of Proof: Achievability

• Random code generation: Assign every $x^n\in {\mathcal{X}}^n$ to one of $2^{n{R}_1}$ bins independently according to a uniform distribution on $\left\{1,2,\dots ,2^{n{R}_1}\right\}.$ Similarly, randomly assign every $y^n\in {\mathcal{Y}}^n$ to one of $2^{n{R}_2}$ bins. Reveal the assignments $f_1$ and $f_2$ to both sender and receiver.
• Encoding: Encoder 1 sends the index of the bin to which $x^n$ belongs. Encoder 2 sends the index of the bin to which $y^n$ belongs.
• Decoding: Given the received index pair $(i,j)$, declare $\left({\hat{x}}^n,{\hat{y}}^n\right)=$ $\left(x^n,y^n\right)$ if there is one and only one pair of sequences $\left(x^n,y^n\right)$ such that $f_1\left(x^n\right)=i$ and $f_2\left(y^n\right)=j$ and $\left(x^n,y^n\right)\in A_{ϵ}^{(n)}$. Otherwise, declare an error.

Let $\left(X_i,Y_i\right)\sim p(x,y)$. Define the events
E 0 = { ( x n , y n ) ∉ A ϵ ( n ) } E 1 = { ∃ x ~ n ≠ x n : f 1 ( x ~ n ) = f 1 ( x n )  and  ( x ~ n , y n ) ∈ A ϵ ( n ) } E 2 = { ∃ y ~ n ≠ y n : f 2 ( y ~ n ) = f 2 ( y n )  and  ( x n , y ~ n ) ∈ A ϵ ( n ) } E 3 = { ∃ ( x ~ n , y ~ n ) : x ~ n ≠ x n , y ~ n ≠ y n , f 1 ( x ~ n ) = f 1 ( x n ) f 2 ( y ~ n ) = f 2 ( y n )  and  ( x ~ n , y ~ n ) ∈ A ϵ ( n ) } P e ( n ) = Pr ⁡ ( E 0 ∪ E 1 ∪ E 2 ∪ E 3 ) ≤ Pr ⁡ ( E 0 ) + Pr ⁡ ( E 1 ) + Pr ⁡ ( E 2 ) + Pr ⁡ ( E 3 ) \begin{aligned} E_{0}=&\left\{\left(x^{n}, y^{n}\right) \notin A_{\epsilon}^{(n)}\right\} \\ E_{1}=&\left\{\exists \tilde{x}^{n} \neq x^{n}: f_{1}\left(\tilde{x}^{n}\right)=f_{1}\left(x^{n}\right) \text { and }\left(\tilde{x}^{n}, y^{n}\right) \in A_{\epsilon}^{(n)}\right\} \\ E_{2}=&\left\{\exists \tilde{y}^{n} \neq y^{n}: f_{2}\left(\tilde{y}^{n}\right)=f_{2}\left(y^{n}\right) \text { and }\left(x^{n}, \tilde{y}^{n}\right) \in A_{\epsilon}^{(n)}\right\} \\ E_{3}=&\left\{\exists\left(\tilde{x}^{n}, \tilde{y}^{n}\right): \tilde{x}^{n} \neq x^{n}, \tilde{y}^{n} \neq y^{n}, f_{1}\left(\tilde{x}^{n}\right)=f_{1}\left(x^{n}\right)\right.\\ &\left.f_{2}\left(\tilde{y}^{n}\right)=f_{2}\left(y^{n}\right) \text { and }\left(\tilde{x}^{n}, \tilde{y}^{n}\right) \in A_{\epsilon}^{(n)}\right\} \\ P_{e}^{(n)}=& \operatorname{Pr}\left(E_{0} \cup E_{1} \cup E_{2} \cup E_{3}\right) \\ \leq & \operatorname{Pr}\left(E_{0}\right)+\operatorname{Pr}\left(E_{1}\right)+\operatorname{Pr}\left(E_{2}\right)+\operatorname{Pr}\left(E_{3}\right) \end{aligned} E0​=E1​=E2​=E3​=Pe(n)​=≤​{(xn,yn)∈/​Aϵ(n)​}{∃x~n​=xn:f1​(x~n)=f1​(xn) and (x~n,yn)∈Aϵ(n)​}{∃y~​n​=yn:f2​(y~​n)=f2​(yn) and (xn,y~​n)∈Aϵ(n)​}{∃(x~n,y~​n):x~n​=xn,y~​n​=yn,f1​(x~n)=f1​(xn)f2​(y~​n)=f2​(yn) and (x~n,y~​n)∈Aϵ(n)​}Pr(E0​∪E1​∪E2​∪E3​)Pr(E0​)+Pr(E1​)+Pr(E2​)+Pr(E3​)​

$$\begin{array}{rl}\mathrm{Pr}\left(E_1\right)& =\sum _{\left(x^n,y^n\right)\in A_{ϵ}^{(n)}}p\left(x^n,y^n\right)\mathrm{Pr}\left(\exists {\tilde{x}}^n\ne x^n:f_1\left({\tilde{x}}^n\right)=f_1\left(x^n\right)\text{ and }\left({\tilde{x}}^n,y^n\right)\in A_{ϵ}^{(n)}\right)\\ & \le \sum _{\left(x^n,y^n\right)\in A_{ϵ}^{(n)}}p\left(x^n,y^n\right)\sum _{{\tilde{x}}^n:\left({\tilde{x}}^n,y^n\right)\in A_{ϵ}^{(n)}}{2}^{-n{R}_1}\\ & \le 2^{n(H(X\mid Y)+ϵ)}{2}^{-n{R}_1}\\ & \le {ϵ}^{\prime }\end{array}$$
if $R_1>H(X\mid Y)+ϵ$ and $n$ sufficiently large.

Similarly, we find that for sufficiently large $n$, $\mathrm{Pr}\left(E_2\right)<{ϵ}^{\prime }$ if $R_2>$ $H(Y|X)$ and $\mathrm{Pr}\left(E_3\right)<{ϵ}^{\prime }$ if $R_1+R_2>H(X,Y)$. Since $\mathrm{Pr}\left(E_0\right)<ϵ$, we conclude that the probability of error $P_e^{(n)}\to 0$ as $n\to \infty$

Interpretation of Slepian-Wolf Coding

Consider the corner point of the rate region in Slepian-Wolf encoding, where $R_1=H(X)$ and $R_2=H(Y|X)$.

• Instead of trying to determine the typical fan, the $Y$ encoder randomly assigns to all $y^n$ sequences ($2^{nH(Y)}$ in total) an index at random from $\left\{1,2,\dots ,2^{n{R}_2}\right\}$

• If the number of indices is high enough, then with high probability every element in the typical fan associated with $x^n$ will have a unique index

• For $R_2>H(Y|X)$, this number is exponentially larger than the number of elements in the fan

• The decoder, also knowing $x^n$, can construct the typical fan and the received $Y$ index will uniquely determine the $y^n$ sequence within the $x^n$ fan

Interpretation:

How does the random binning scheme bypass the problem that the $Y$ encoder does not know which sequence $x^n$ is encoded?

Under the random binning scheme the decoder is able to decode $x^n$ since the assignments $f_1$ is revealed to both sender and receiver. Therefore, the number of possible $y^n$ is narrowed down from $|y^n|=2^{nH(Y)}$ to $|y^n|=2^{nH(Y|X)}$. The $Y$ encoder can now focus on discriminate $2^{nH(Y|X)}$ sequences.

We can conclude that the decoder knowing which sequence $x^n$ is encoded is equivalent to the $Y$ encoder knowing which sequence $x^n$ is encoded in effect.