【信息技术】【2014.06】确定性系统中的信息损失：连接信息论、系统论与信号处理

本文为奥地利格拉茨技术大学（作者：Dipl.-Ing. BernhardC. Geiger）的博士论文，共192页。

信息理论中的一个基本定理——数据处理不等式——指出确定性处理不能够增加随机变量或随机过程中包含的信息量。信号处理的任务是对信息的物理表示进行操作，以便目标用户可以轻松访问这些信息。根据数据处理的不等式，可以将其视为在尽可能多地保留相关信息的同时，删除无关信息的任务。

本文定义了有关联和无关联概念下无记忆系统处理随机变量或随机过程时的信息损失。这些定义是信息系统理论的基础，它补充了当前流行的以能量为中心的方法。由此得出的结果被用于分析信号处理中的各种系统：多项式、量化器、整流器、具有或不具有带量化效应的线性滤波器、主成分分析、多速率系统等。

本文的分析不仅讨论这些系统的信息处理能力，而且还讨论基于信息论量化与基于能量测度设计原理（如均方误差）之间存在的差异和相似之处。结果表明，至少在某些情况下，简单的能量设计在信息理论上是合理的。因此，本文提出了两种时间齐次的有限马尔可夫链模型复杂性约简方法。当一种方法以损失（一阶）马尔可夫特性为代价保留完整的模型信息时，另一种方法则在较小的状态空间上生成马尔可夫链，减少了模型信息。最后，本文给出了强集总性的信息论表征，即马尔可夫链的函数是（某阶）马尔可夫的情况。

A fundamental theorem in information theory – the data processinginequality – states that deterministic processing cannot increase the amount ofinformation contained in a random variable or a stochastic process. The task ofsignal processing is to operate on the physical representation of informationsuch that the intended user can access this information with little effort. Inthe light of the data processing inequality, this can be viewed as the task of removing irrelevantinformation, while preserving as much relevant information aspossible. This thesis defines information loss for memoryless systemsprocessing random variables or stochastic processes, both with and without anotion of relevance. These definitions are the basis of aninformation-theoretic systems theory, which complements the currentlyprevailing energy-centered approaches. The results thus developed are used toanalyze various systems in the signal processor’s toolbox: polynomials,quantizers, rectifiers, linear filters with and without quantization effects,principal components analysis, multirate systems, etc. The analysis not only focuseson the information processing capabilities of these systems: It also highlightsdifferences and similarities between design principles based oninformation-theoretic quantities and those based on energetic measures, such asthe mean-squared error. It is shown that, at least in some cases, simpleenergetic design can be justified information-theoretically. As a side result,this thesis presents two approaches to model complexity reduction for time homogeneous,finite Markov chains. While one approach preserves full model information with thecost of losing the (first-order) Markov property, the other approach yields aMarkov chain on a smaller state space with reduced model information. Finally,this thesis presents an information theoretic characterization of strong lumpability, the case wherethe function of a Markov chain is Markov (of some order).

1 引言
2 信息损失
3 信息损失率
4 相关信息损失
5 相关信息损失率
6 讨论
附录 详细证明过程