Channel Capacity 2: Channel Coding Theorem

Reference:

Elements of Information Theory, 2nd Edition

Slides of EE4560, TUD

Preliminaries

We analyze a communication system as shown in Figure 7.8.

• $W$: message drawn from the index set { 1 , 2 , ⋯   , M } \{1,2,\cdots,M\} { 1,2,⋯,M}
• $X^n=f(W)$: transmitted sequence, consisting of $n$ symbols from the channel input alphabet $\mathcal{X}$
• $Y^n\sim p(y^n|x^n)$: received sequence, consisting of $n$ symbols from the channel output alphabet $\mathcal{Y}$
• $\hat{W}=g(Y^n)$: message from the index set { 1 , 2 , ⋯   , M } \{1,2,\cdots,M\} { 1,2,⋯,M}; decoding error in case $\hat{W}\ne W$

Definition 1 (Code):

An $(M,n)$ code for the channel $(\mathcal{X},p(y|x),\mathcal{Y})$ consists of

• An index set { 1 , 2 , ⋯   , M } \{1,2,\cdots,M\} { 1,2,⋯,M}
• An encoding function f : { 1 , 2 , ⋯   , M } → X n f:\{1,2,\cdots,M\}\to \mathcal X^n f:{ 1,2,⋯,M}→Xn, yielding codewords $x^n(1),\cdots ,x^n(M)$. The set of codewords is called the codebook
• A decoding function g : Y n → { 1 , 2 , ⋯   , M } g:\mathcal Y^n\to \{1,2,\cdots,M\} g:Yn→{ 1,2,⋯,M}

Example [slides 5-7]

Definition 2 (Conditional Probability of Error):

Conditional probability of error given that index $i$ was sent:
$$\begin{array}{rl}{\lambda }_i& =\mathrm{Pr}(g(Y^n)\ne i|X^n=x^n(i))=\sum _{y^n}p(y^n|x^n(i))I(g(y^n)\ne i)\end{array}\text{\hspace{1em}(1)}$$
where $I(\cdot )$ is the indicator function. ($x^n$ and $y^n$ are the possible realizations of $X^n$ and $Y^n$)

Maximal probability of error:
λ ( n ) = max ⁡ i ∈ { 1 , 2 , ⋯   , M } λ i (2) \lambda^{(n)}=\max _{i\in \{1,2,\cdots,M\}} \lambda_i \tag{2} λ(n)=i∈{ 1,2,⋯,M}max​λi​(2)
Average probability of error: