传递熵（Transfer Entropy，TE）滥觞、2000年顶刊论文方法部分：Measuring Information Transfer

论文基本信息

Measuring Information Transfer这篇论文于2000年发表在物理领域顶刊《PHYSICAL REVIEW LETTERS》。
被引频次：2147（WOS）2021年12月7日14:32:30

---

方法部分原文摘录

Kullback entropy

In order to construct an optimal encoding that uses just as many bits as given by the entropy, it is necessary to know the probability distribution $p(i)$. The excess number of bits that will be coded if a different distribution $q(i)$ is used is given by the Kullback entropy [4] $K_I={\sum }_{i}p(i)\log p(i)/q(i)$. We will later also need the Kullback entropy for conditional probabilities $p(i|j)$ . For a single state $j$ we have $K_j={\sum }_{i}p(i|j)\log p(i|j)/q(i|j)$. Summation over $j$ with respect to $p(j)$ yields

$$K_{I|J}=\sum _{i,j}p(i,j)\log \frac{p(i|j)}{q(i|j)}\text{\hspace{1em}(1)}$$

---

One can incorporate dynamical structure by studying transition probabilities rather than static probabilities. Consider a system that may be approximated by a stationary Markov process of order $k$, that is, the conditional probability to find $I$ in state $i_{n+1}$ at time $n+1$ is independent of the state $i_{n-k}:p(i_{n+1}|i_n,\dots ,i_{n-k+1})=p(i_{n+1}|i_n,\dots ,i_{n-k+1},i_{n-k})$. Henceforth we will use the shorthand notation $i_n^{(k)}=(i_n,\dots ,i_{n-k+1})$ for words of length $k$.

The average number of bits needed to encode one additional state of the system if all previous states are known is given by the entropy rate
$$h_I=-\sum p(i_{n+1},i_n^{(k)})\log p(i_{n+1}|i_n^{(k)})\text{\hspace{1em}(3)}$$

---

transfer entropy

For the study of the dynamics of shared information between processes it is desirable to generalize the entropy rate, rather than Shannon entropy, to more than one system, since the dynamics of the processes is contained in the transition probabilities. The most straightforward way to construct a mutual information rate by generalizing $h_I$ to two processes $(I,J)$ is again by measuring the deviation from independence. The corresponding Kullback entropy is still symmetric under the exchange of $I$ and $J$. It is therefore preferable to measuring the deviation from the generalized Markov property, $p(i_{n+1}|i_n^{(k)})=p(i_{n+1}|i_n^{(k)},j_n^{(l)}).$

In the absence of information flow from $J$ to $I$, the state of $J$ has no influence on the transition probabilities on system $I$. The incorrectness of this assumption can again be quantified by a Kullback entropy (1) by which we define the transfer entropy:
$$T_{J\to I}=\sum p(i_{n+1},i_n^{(k)},j_n^{(l)})\log \frac{p(i_{n+1}|i_n^{(k)},j_n^{(l)})}{p(i_{n+1}|i_n^{(k)})}\text{\hspace{1em}(4)}$$

causality

Figure 2 shows M and T as functions of the coupling strength. Both M and T are able to detect the anisotropy since the information is consistently larger in the positive direction. The lattice undergoes a number of bifurcations when the coupling is changed. Around $ϵ=0.18$, the asymptotic state is of temporal and spatial period two. For this case, the mutual information is found to be 1 bit. This is correct although information is neither produced nor exchanged and reflects the static correlation between the sites.

The transfer entropy finds a zero rate of information transport, as desired. Around this periodic window, the mutual information is nonzero in both directions and the signature of the unidirectional coupling is less pronounced. Around $ϵ=0.82$, the lattice settles to a (spatially inhomogeneous) fixed point state. Here both measures correctly show zero information transfer. The most important finding, however, is that the transfer entropy for the negative direction remains consistent with zero for all couplings, reflecting the causality in the system.