Network Information Theory

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

Introduction-Network IT Features

• Multiple senders and/or multiple receivers. E.g., computer networks, satellite networks…

• New elements in the communication problem. E.g., interference cooperation

• Problem description:

  Given the senders, the receivers, and the channel transition matrix (that describes the effects of the interference and the noise in the network), can the sources be transmitted over the channel?

• The general problem is tough and has not yet been solved $\to$ we focus on interesting special cases

Discrete Memoryless Multiple-Access Channel

A discrete memoryless multiple-access channel consists of

• $m$ input alphabets ${\mathcal{X}}_1,{\mathcal{X}}_2,\cdots ,{\mathcal{X}}_m$
• an output alphabet $\mathcal{Y}$, and
• a probability transition matrix $p(y|x_1,x_2,\cdots ,x_m)$

For simplicity, we will further consider the case $m=2$.

Definition 1 (Code for the MA channel):

A $\left(\left(2^{n{R}_1},2^{n{R}_2}\right),n\right)$ code for the multiple-access channel consists of

• two message sets ${\mathcal{W}}_i=\left\{1,2,\dots ,2^{n{R}_i}\right\},i=1,2$,
• two encoding functions $X_i:{\mathcal{W}}_i\to {\mathcal{X}}_i^n,i=1,2$,
• a decoding function $g:{\mathcal{Y}}^n\to {\mathcal{W}}_1\times {\mathcal{W}}_2$.

Definition 2 (Average probability of error):

The average probability of error for a $\left(\left(2^{n{R}_1},2^{n{R}_2}\right),n\right)$ code is
P e ( n ) = 1 2 n ( R 1 + R 2 ) ∑ ( w 1 , w 2 ) Pr ⁡ { g ( Y n ) ≠ ( w 1 , w 2 ) ∣ ( w 1 , w 2 ) }  sent P_{e}^{(n)}=\frac{1}{2^{n\left(R_{1}+R_{2}\right)}} \sum_{\left(w_{1}, w_{2}\right)} \operatorname{Pr}\left\{g\left(Y^{n}\right) \neq\left(w_{1}, w_{2}\right) |\left(w_{1}, w_{2}\right)\right\}\text{ sent} Pe(n)​=2n(R1​+R2​)1​(w1​,w2​)∑​Pr{g(Yn)​=(w1​,w2​)∣(w1​,w2​)} sent
Definition 3 (Achievable):

A rate pair $\left(R_1,R_2\right)$ is said to be achievable if there exists a sequence of $\left(\left(2^{n{R}_1},2^{n{R}_2}\right),n\right)$ codes with $P_e^{(n)}\to 0$.

Definition 4 (Capacity region):

The capacity region of the MA channel is the closure of the set of achievable $\left(R_1,R_2\right)$ rate pairs.

Theorem 1 (MA channel capacity theorem):

The capacity region of the multiple-access channel
$$\left({\mathcal{X}}_1\times {\mathcal{X}}_2,p\left(y|x_1,x_2\right),\mathcal{Y}\right)$$
is the closure of the convex hull of all $\left(R_1,R_2\right)$ satisfying
$$\begin{array}{l}{R}_1<I\left(X_1;Y|X_2\right)\\ R_2<I\left(X_2;Y|X_1\right)\\ R_1+R_2<I\left(X_1,X_2;Y\right)\end{array}$$
for some product distribution on ${\mathcal{X}}_1\times {\mathcal{X}}_2$.

Achievable region for a particular input distribution: (Note that different input distributions will have the same shape but with different values. The capacity region is the convex hull of all achievable regions of different distributions. See [Exercise](# Exercise))

• $I\left(X_1;Y|X_2\right)$: The mutual information of $X_1$ and $Y$ if we know $X_2$. In application, this means that User 1 has the complete control of User 2.
• $I\left(X_2;Y\right)$: The mutual information of $X_2$ and $Y$. We do not know anything about $X_1$.
• Onion-peeling: Consider decoding as a two-stage process
  • In the first stage, the receiver decodes the second sender, considering the first sender as part of the noise. This decoding will have low probability of error if $R_2<I(X_2;Y)$.
  • After the second sender has been decoded successfully, it can be subtracted out and the first sender can be decoded correctly if $R_1<I(X_1;Y|X_2)$.
• Maximizing $R_1$ if $R_2=0$: $I(X_1;Y|X_2)\le {\max }_{x_2}I(X_1;Y|X_2=x_2)$. Hence, set $X_2$ equal to this $x_2$, and choose $p(x_1)$ to maximize the mutual information between $X_1$ and $Y$ given $X_2=x_2$.

Binary Multiplier Channel

Binary Erasure MA Channel

Note: $R_1=1$ means that $P(X_1=0)=P(X_1=1)=1/2$. Then $P(Y=0|X_2=0)=P(X_1=0)=1/2$, $P(Y=2|X_2=1)=P(X_1=1)=1/2$. From the perspective of User 2, $P(Y=0|X_2=0)=1/2$, $P(Y=1|X_2=1)=1/2$.

Gaussian MA Channel

Received signal at time $i$ is $Y_i=X_{1i}+X_{2i}+Z_i$, where

• $Z_i$ is a sequence of i.i.d. zero-mean Gaussian random variables with variance $N$,

• sender $j$ satisfies a power constraint $P_j$ :

$$\frac{1}{n}\sum _{i=1}^n{x}_{ji}^2\left(w_j\right)\le P_j,w_j\in \left\{1,2,\dots ,2^{n{R}_j}\right\},j=1,2$$

The capacity region is the convex hull of the set of rate pairs $\left(R_1,R_2\right)$ satisfying
$$\begin{array}{rlrl}{R}_1& \le I\left(X_1;Y|X_2\right)& & \le C\left(P_1/N\right)\\ R_2& \le I\left(X_2;Y|X_1\right)& & \le C\left(P_2/N\right)\\ R_1+R_2& \le I\left(X_1,X_2;Y\right)& & \le C\left(\left(P_1+P_2\right)/N\right),\end{array}$$
where $C(x)=\frac{1}{2}\log (1+x)$. The maximizing input distribution is $X_j\sim \mathcal{N}\left(0,P_j\right)$.

Note: From the term $C(\frac{P_2}{P_1+N})$ we can observe that the capacity is affected not only by the noise but also the interference of other sources.

Gaussian FDMA Capacity

Gaussian Naïve TDMA Capacity

Exercise

Find the capacity region for this channel.

$$\begin{array}{rl}I(X_1;Y|X_2)& =I(X_1;Y)=H(X_1)-H(X_1|Y)\\ & =H(X_1)=H(p)\\ I(X_2;Y|X_1)& =I(X_2;Y)=H(X_2)-H(X_2|Y)\\ & =H(X_2)-p(Y=0)H(X_2|Y=0)\\ & =(1-p)H(q_1,q_2,q_3,q_4)\\ I(X_1,X_2;Y)& =H(Y)=H(p,(1-p)q_1,\cdots ,(1-p)q_4)\\ & =I(X_2;Y|X_1)+I(X_1;Y)=H(p)+(1-p)H(q_1,q_2,q_3,q_4)\end{array}$$
Maximized when $q_1=\cdots =q_4,H(\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{4})=2$. (Why maximize? Convex hull. Find the farthest boundary)