Channel Capacity 3: Gaussian Channel

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

Differential Entropy

We now introduce the concept of differential entropy, which is the entropy of a continuous random variable.

Definition 1 (Differential Entropy):

The differential entropy $h(X)$ of a continuous random variable $X$ with density $f(x)$ is defined as
$$h(X)=-{\int }_{S}f(x)\log f(x)dx\text{\hspace{1em}(1)}$$
where $S$ is the support set of $X$ (where $f(x)>0$).

• $h(X)$ is sometimes denoted as $h(f)$, as $H(X)$ is sometimes denoted as $H(p)$.
• $\log$ here denotes ${\log }_2$. Do not forget to change the base.

Examples:

• Uniform Distribution:
  $$h(X)=-{\int }_0^a\frac{1}{a}\log \frac{1}{a}dx=\log a\text{\hspace{1em}(2)}$$

  • $\text{larger }a\to \text{larger uncertainty}\to \text{larger }h(X)$
  • For $0<a<1$, the differential entropy $\log a$ is negative! This is different from $H(X)$, which is always $\ge 0$.
  • However, $2^{h(X)}=2^{\log a}=a$ is always positive.

• Normal Distribution:
  $$\begin{array}{rl}h(f)& =-\int f(x)\log f(x)dx=-\int \frac{f(x)}{\ln 2}\left[\frac{-(x-\mu {)}^2}{2{\sigma }^2}-\ln \sqrt{2\pi {\sigma }^2}\right]dx\\ & =\frac{1}{\ln 2}\left[\frac{E(X^2)}{2{\sigma }^2}+\frac{1}{2}\ln (2\pi {\sigma }^2)\right]=\frac{1}{2}\log (2\pi e{\sigma }^2)\end{array}\text{\hspace{1em}(3)}$$

Definition 2 (Joint Differential Entropy):

The joint differential entropy $h(X)$ of a set $X_1,X_2,\cdots ,X_n$ of random variables with density $f(x_1,x_2,\cdots ,x_n)$ is defined as
$$h(X_1,X_2,\cdots ,X_n)=-\int f(x^n)\log f(x^n)d{x}^n\text{\hspace{1em}(4)}$$
N.B. $x^n$ here is a short notation of $(x_1,x_2,\cdots ,x_n)$

Definition 3 (Conditional Differential Entropy):

If $X,Y$ have a joint density function $f(x,y),$ we can define the conditional differential entropy $h(X|Y)$ as
$$h(X|Y)=-\int f(x,y)\log f(x|y)dxdy=h(X,Y)-h(Y)\text{\hspace{1em}(5)}$$

Definition 4 (Mutual Information):

The mutual information $I(X;Y)$ between two random variables $X$ and $Y$ with joint density $f(x,y)$ is defined as
$$\begin{array}{rl}I(X;Y)& =\iint f(x,y)\log \frac{f(x,y)}{f(x)f(y)}dxdy\\ & =h(X)-h(X|Y)=h(Y)-h(Y|X)\\ & =h(X)+h(Y)-h(X,Y)\end{array}\text{\hspace{1em}(6)}$$
N.B. $I(X;Y)\ge 0$ with equality if and only if $X$ and $Y$ are independent.

Gaussian Channels

The most important continuous alphabet channel is the Gaussian channel. This is a time-discrete channel with output $Y_i$ at time $i$,where $Y_i$ is the sum of the input $X_i$ and the noise $Z_i$. The noise $Z_i$ is drawn i.i.d. from a Gaussian distribution with variance $N$. Thus,
$$Y_i=X_i+Z_i,Z_i\sim \mathcal{N}(0,N)\text{\hspace{1em}(7)}$$
The noise $Z_i$ is assumed to be independent of the signal $X_i$.

The most common limitation on the input is an energy or power constraint. We assume an average power constraint. For any codeword $(x_1,x_2,...,x_n)$ transmitted over the channel, we require that
$$\frac{1}{n}\sum _{i=1}^n{x}_i^2\le P\text{\hspace{1em}(8)}$$
[Example-Binary input, Gaussian noise: Slide 6-7]

---

Gaussian Channel Capacity

Definition 5 (Information Capacity):

The information capacity of the Gaussian channel with power constraint $P$ is
$$C=\underset{f(x):E{X}^2\le P}{\max }I(X;Y)=\frac{1}{2}\log \left(1+\frac{P}{N}\right)\text{\hspace{1em}(9)}$$
where the maximum is achieved when $X\sim \mathcal{N}(0,P)$.

Proof: Expanding $I(X;Y),$ we have
$$\begin{array}{rl}I(X;Y)& =h(Y)-h(Y|X)\\ & =h(Y)-h(X+Z|X)\\ & =h(Y)-h(Z|X)\\ & =h(Y)-h(Z)\end{array}$$
since $Z$ is independent of $X.$ From Eq. $(3)$, $h(Z)=\frac{1}{2}\log 2\pi eN.$ Also,
$$E{Y}^2=E(X+Z{)}^2=E{X}^2+2EXEZ+E{Z}^2=P+N$$
since $X$ and $Z$ are independent and $EZ=0.$ Given $E{Y}^2=P+N,$ the entropy of $Y$ is bounded by $\frac{1}{2}\log 2\pi e(P+N)$ by Theorem 8.6.5 (the normal maximizes the entropy for a given variance) [book 254] . Applying this result to bound the mutual information, we obtain
$$\begin{array}{rl}I(X;Y)& =h(Y)-h(Z)\\ & \le \frac{1}{2}\log 2\pi e(P+N)-\frac{1}{2}\log 2\pi eN\\ & =\frac{1}{2}\log \left(1+\frac{P}{N}\right)\end{array}$$
Next, it will be shown that this capacity is also the supremum of the rates achievable for the channel, i.e., the operational capacity.

---

Definition 6 (Code):

An $(M,n)$ code for the Gaussian channel with power constraint $P$ consists of the following:

1. An index set { 1 , 2 , … , M } \{1,2, \ldots, M\} {1,2,…,M}
2. An encoding function x : { 1 , 2 , … , M } → X n x:\{1,2, \ldots, M\} \rightarrow \mathcal{X}^{n} x:{1,2,…,M}→Xn, yielding codewords $x^n(1),x^n(2),\dots ,x^n(M),$ satisfying the power constraint $P;$ that is, for every codeword
   $$\sum _{i=1}^n{x}_i^2(w)\le nP,w=1,2,\dots ,M$$
3. A decoding function

g : Y n → { 1 , 2 , … , M } g: \mathcal{Y}^{n} \rightarrow\{1,2, \ldots, M\} g:Yn→{1,2,…,M}

N.B. Rate $R=\frac{\log M}{n}$, as is defined in the discrete channel.

Definition 7 (Achievable):

A rate $R$ is said to be achievable for a Gaussian channel with a power constraint $P$ if there exists

• a sequence of $\left(2^{nR},n\right)$ codes
• with codewords satisfying the power constraint
• such that the maximal probability of error ${\lambda }^{(n)}$ tends to zero.

The capacity of the channel is the supremum of the achievable rates.

Theorem 1 (The capacity of a Gaussian channel):

The capacity of a Gaussian channel with power constraint $P$ and noise variance $N$ is
$$C=\frac{1}{2}\log \left(1+\frac{P}{N}\right)\text{ bits per transmission}\text{\hspace{1em}(10)}$$
[Proof: book 266-268]

A plausibility argument:

---

Band-Limited Channel

A common model for communication over a radio network or a telephone line is a bandlimited channel with white noise. This is a continuous time channel. The output of such a channel can be described as the convolution
$$Y(t)=(X(t)+Z(t))\ast h(t)\text{\hspace{1em}(11)}$$
where

• $Y(t)$ is output signal waveform
• $X(t)$ is input signal waveform
• $Z(t)$ is white Gaussian noise waveform
• $h(t)$ is impulse response of an ideal bandpass filter (cuts out all frequencies greater than $W$).

Theorem 2 (The sampling theorem):

A function $f(t)$, which is band-limited to $W$, is completely determined by samples of the function spaced $\frac{1}{2W}$ seconds apart.

[Proof: book 271]

Now we can formulate the problem of communication over a bandlimited channel:

• Bandwidth $W$
• Number of samples per second $2W$
• Signal power $P$
• Noise power $N=N_{0}W$, where $N_0$ is the noise power spectral density

If channel is used over the time interval $[0,T]$, then

• energy per sample is $\frac{PT}{2WT}=\frac{P}{2W}$
• noise variance per sample is $\frac{N_{0}WT}{2WT}=\frac{N_0}{2}$

Using Theorem 1 (Eq. $(10)$), we can obtain the capacity per sample:
$$C=\frac{1}{2}\log \left(\frac{1+\frac{P}{2W}}{\frac{N_0}{2}}\right)=\frac{1}{2}\log \left(1+\frac{P}{N_{0}W}\right)\text{ bits per sample }\text{\hspace{1em}(12)}$$
Since there are $2W$ samples per second, the capacity per second:
$$C=W\log \left(1+\frac{P}{N_{0}W}\right)\text{ bits per second }\text{\hspace{1em}(13)}$$
N.B. If $W\to \infty$, using $\ln (1+x)\sim x(x\to 0)$, then $C=\frac{P\log e}{N_0}\mathrm{b}\mathrm{p}\mathrm{s}$.

Definition 8 (Band Efficiency):

Bandwidth efficiency $\eta$ is defined as the rate $R$ (in $\mathrm{b}\mathrm{i}\mathrm{t}/\mathrm{s}$) divided by the bandwidth $W$ (in $\mathrm{H}\mathrm{z}$):
$$\eta =\frac{R}{W}\mathrm{b}\mathrm{i}\mathrm{t}/\mathrm{s}/\mathrm{H}\mathrm{z}\text{\hspace{1em}(14)}$$
From channel capacity formula it follows that
$$R<C=W\log \left(1+\frac{P}{W{N}_0}\right)=W\log \left(1+\frac{R{E}_b}{W{N}_0}\right)$$
where $E_b$ is the energy per bit. Hence,
$$\eta <\log \left(1+\eta \frac{E_b}{N_0}\right),\text{ i.e., }\frac{E_b}{N_0}>\frac{2^{\eta }-1}{\eta }\text{\hspace{1em}(15)}$$

---

Parallel Gaussian Channels

Problem to be solved:
$$\begin{array}{rlrl}& \text{minimize}& & -\sum _{j=1}^k{C}_j=-\sum _{j=1}^k\frac{1}{2}\log \left(1+\frac{P_j}{N_j}\right)\\ & \text{subject to}& & \sum _{j=1}^k{P}_j\le P\end{array}\text{\hspace{1em}(16)}$$
Using Lagrange multipliers gives the function
$$L(P_1,\cdots ,P_k,\lambda )=-\sum _{j=1}^k\frac{1}{2}\log \left(1+\frac{P_j}{N_j}\right)+\lambda (\sum _{j=1}^k{P}_j-P)$$
KKT conditions:
$$\begin{gathered} \sum _{j=1}^k{P}_j\le P,\lambda \ge 0 \\ {\nabla }_{P_j}L=0⟹P_j=\frac{1}{2\lambda }-N_j \\ \lambda (\sum _{j=1}^k{P}_j-P)=0 \end{gathered}$$
Together with the condition that $P_j$ are nonnegative gives the solution
P j = max ⁡ { 0 , 1 2 λ − N j } ≜ ( ν − N j ) + (17) P_j=\max \{0,\frac{1}{2\lambda}-N_j\}\triangleq(\nu-N_j)^+ \tag{17} Pj​=max{0,2λ1​−Nj​}≜(ν−Nj​)+(17)
where $\nu$ is chosen such that $\sum (\nu -N_j{)}^{+}=P$.

This solution is illustrated graphically in Figure 9.4. The vertical levels indicate the noise levels in the various channels. As the signal power is increased from zero, we allot the power to the channels with the lowest noise. When the available power is increased still further, some of the power is put into noisier channels.

The process by which the power is distributed among the various bins is identical to the way in which water distributes itself in a vessel, hence this process is sometimes referred to as water-filling.

[Example: Slides 23-25]

---

Gaussian Channels with Feedback

The feedback allows the input of the channel to depend on the past values of the output:

• Capacity without feedback:
  $$\underset{\mathrm{tr}\left(K_X\right)\le nP}{\max }\frac{1}{2n}\log \frac{\left|K_X+K_Z\right|}{\left|K_Z\right|}\text{\hspace{1em}(18)}$$
• Capacity with feedback:

$$\underset{\mathrm{tr}\left(K_X\right)\le nP}{\max }\frac{1}{2n}\log \frac{\left|K_{X+Z}\right|}{\left|K_Z\right|}\text{\hspace{1em}(19)}$$
where $K_{\dots }$ is $n\times n$ covariance matrix.

Remarks:

• Memoryless channels: feedback does not increase capacity!

• Channels with memory: feedback does increase capacity!

• Feedback does not improve capacity by more than $\frac{1}{2}$ :
  $$C_{\text{withFB }}\le C_{\text{withoutFB }}+\frac{1}{2}\text{\hspace{1em}(20)}$$

• Feedback does not improve capacity by more than a factor of two:
  $$C_{\text{withFB }}\le 2C_{\text{withoutFB }}\text{\hspace{1em}(21)}$$

• Conclusion: feedback may help, but not much!