# 理解自信息和信息熵——为什么自信息这样算？

“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.”

$I(x_i)=log\frac{1}{P(x_i)}$

$H(X)=\sigma_xp(x)log\frac{1}{p(x)}$

$H(X)=E[I(x_i)]$

The self-information is a measure of deviation from expectation of random variable in shannons (bits, the unit may vary when used in different context) when sampling a random variable.

Understanding of self-information

Why is self-information defined the way it is?

A Brief Introduction to Shannon’s Information Theory（给出了另外一种推导出信息熵公式的方法，很有趣）

The intuition behind Shannon’s Entropy（给出了信息熵的intuition，有友好的图片解说，但是我没太理解）

Entropy (information theory)

05-25
11-18 2万+

11-29 3315
02-19 6563
06-28 2835
04-02 4286
05-21 1624
05-08 2万+
03-19 163
04-16 4667
06-11
01-06 4735
04-24 2037
05-27 9424
01-28 1515
05-15
07-07 4558
07-10 2万+
09-08 4748