In
It is related to the (historically earliest)
Definition
Let
-
这是连续多于n/2次选取一个事件的概率,[0,1].
The Chernoff bound shows that
-
- 之上的S表示概率.
- 之下的S表示事件发生的总次数.
- 上面的式子就是
-
Indeed, noticing that , we get by the multiplicative form of Chernoff bound (see below or Corollary 13.3 in Sinclair's class notes),
This result admits various generalizations as outlined below. One can encounter many flavours of Chernoff bounds: the original
[edit]A motivating example
The simplest case of Chernoff bounds is used to bound the success probability of majority agreement for n independent, equally likely events.
A simple motivating example is to consider a biased coin. One side (say, Heads), is more likely to come up than the other, but you don't know which and would like to find out. The obvious solution is to flip it many times and then choose the side that comes up the most. But how many times do you have to flip it to be confident that you've chosen correctly?
In our example, let
If the coin is noticeably biased, say coming up on one side 60% of the time (p ) accuracy after 150 flips
. If it is 90% biased, then a mere 10 flips suffices. If the coin is only biased a tiny amount, like most real coins are, the number of necessary flips becomes much larger.
More practically, the Chernoff bound is used in
Notice that if
The first step in the proof of Chernoff bounds
The Chernoff bound for a random variable , is obtained by applying
From
For any
In particular optimizing over t and using independence we obtain,
-
(1)
Similarly,
and so,
[edit]Precise statements and proofs
[edit]Theorem for additive form (absolute error)
The following Theorem is due to
Assume random variables
,
, and
. Then
and
where
is the
, then
[edit]Proof
The proof starts from the general inequality (1) above. . Taking
Now, knowing that ,
, we have
Therefore we can easily compute the infimum, using calculus and some logarithms. Thus,
Setting the last equation to zero and solving, we have
so that .
Thus, .
As , we see that
, so our bound is satisfied on
. Having solved for
, we can plug back into the equations above to find that
We now have our desired result, that
To complete the proof for the symmetric case, we simply define the random variable , apply the same proof, and plug it into our bound.
[edit]Simpler bounds
A simpler bound follows by relaxing the theorem using , which follows from the
. This results in a special case of
, which is stronger for
, is also used.
[edit]Theorem for multiplicative form of Chernoff bound (relative error)
Let random variables
. Then, if we let
, for any
[edit]Proof
According to (1),
The third line above follows because
. This is identical to the calculation above in the proof of the
Rewriting
), we set
. The same result can be obtained by directly replacinga
.[1]
Thus,
If we simply set
, we can substitute and find
This proves the result desired. A similar proof strategy can be used to show that
[edit]Better Chernoff bounds for some special cases
We can obtain stronger bounds using simpler proof techniques for some special cases of symmetric random variables.
Let
-
.
(a) .
Then,
-
,
and therefore also
-
.
(b)
Then,
-
,
-
,
-
,
-
.
[edit]Applications of Chernoff bound
Chernoff bounds have very useful applications in
The set balancing problem arises while designing statistical experiments. Typically while designing a statistical experiment, given the features of each participant in the experiment, we need to know how to divide the participants into 2 disjoint groups such that each feature is roughly as balanced as possible between the two groups. Refer to this
Chernoff bounds are also used to obtain tight bounds for permutation routing problems which reduce
[edit]Matrix Chernoff bound
Rudolf Ahlswede
If
,
where
Notice that the number of samples in the inequality depends logarithmically on . In general, unfortunately, such a dependency is inevitable: take for example a diagonal random sign matrix of dimension
. The operator norm of the sum of
. In order to achieve a fixed bound on the maximum deviation with constant probability, it is easy to see that
The following theorem can be obtained by assuming
[edit]Theorem without the dependency on the dimensions
Let
. Set
-
.
If
where
.