Chapter 1 (Sample Space and Probability): Independence (独立性)

本文为 $Introduction$ $to$ $Probability$ 的读书笔记

Independence

• We have introduced the conditional probability $P(A|B)$ to capture the partial information that event $B$ provides about event $A$. An interesting and important special case arises when the occurrence of $B$ provides no such information and does not alter the probability that $A$ has occurred, i.e.,
  $$P(A|B)=P(A)$$
• When the above equality holds. we say that $A$ is independent of $B$ ($A$ and $B$ are independent events). The equation above is equivalent to
  $$P(A\cap B)=P(A)P(B)$$
  • We adopt this latter relation as the definition of independence because it can be used even when $P(B)=0$, in which case $P(A|B)$ is undefined.
  • If $A$ and $B$ are independent, the occurrence of $B$ does not provide any new information on the probability of $A$ occurring.

Pitfall: A common first thought is that two events are independent if they are disjoint, but in fact the opposite is true: two disjoint events $A$ and $B$ with $P(A)>0$ and $P(B)>0$ are never independent. (e.g. $A$ and $A^C$)

---

性质

• 在 $(A,B),(A,\bar{B}),(\bar{A},B),(\bar{A},\bar{B})$ 这四对事件中，如果有一对独立，则另外三对也独立

Conditional Independence 条件独立

• We noted earlier that the conditional probabilities of events form a legitimate probability law. We can thus talk about independence of various events with respect to this conditional law.

---

• In particular, given an event $C$. the events $A$ and $B$ are called conditionally independent if
  $$P(A\cap B|C)=P(A|C)P(B|C)$$
• To derive an alternative characterization of conditional independence, we write
  
• We now compare the preceding two expressions. and after eliminating the common factor $P(B|C)$, assumed nonzero. we see that conditional independence is the same as the condition
  $$P(A|B\cap C)=P(A|C)$$In words, this relation states that if $C$ is known to have occurred, the additional knowledge that $B$ also occurred does not change the probability of $A$.
• Interestingly, independence of two events $A$ and $B$ with respect to the unconditional probability law. does not imply conditional independence, and vice versa, as illustrated by the next example.

---

Example 1.21.

• There are two coins, a blue and a red one. We choose one of the two at random, each being chosen with probability $1/2$, and proceed with two independent tosses. The coins are biased: with the blue coin, the probability of heads in any given toss is $0.99$, whereas for the red coin it is $0.01$.
• Let $B$ be the event that the blue coin was selected. Let also $H_i$ be the event that the $i$th toss resulted in heads. Given the choice of a coin, the events $H_1$ and $H_2$ are independent. Thus,
  $$P(H_1\cap H_2|B)=P(H_1|B)P(H_2|B)=0.99\cdot 0.99$$
• On the other hand, the events $H_1$ and $H_2$ are not independent. Intuitively, if we are told that the first toss resulted in heads, this leads us to suspect that the blue coin was selected. Mathematically,
  $$P(H_1)=P(B)P(H_1|B)+P(B^C)P(H_1|B^C)=\frac{1}{2}\cdot 0.99+\frac{1}{2}\cdot 0.01=\frac{1}{2}$$
  Similarly, we have $P(H_2)=1/2$. Now notice that
  $$\begin{array}{rl}P(H_1\cap H_2)& =P(B)P(H_1\cap H_2|B)+P(B^C)P(H_1\cap H_2|B^C)\\ & =\frac{1}{2}\cdot 0.99\cdot 0.99+\frac{1}{2}\cdot 0.01\cdot 0.01\approx \frac{1}{2}\end{array}$$
• Thus, the events $H_1$ and $H_2$ are dependent, even though they are conditionally independent given $B$.

Summary

Independence of a Collection of Events

---

• For the case of three events, $A_1$, $A_2$, and $A_3$, independence amounts to satisfying the four conditions
  
  The first three conditions simply assert that any two events are independent, a property known as pairwise independence (两两独立). But the fourth condition is also important and does not follow from the first three. Conversely, the fourth condition does not imply the first three.

---

• The intuition behind the independence of a collection of events is analogous to the case of two events. Independence means that the occurrence or non-occurrence of any number of the events from that collection carries no information on the remaining events or their complements.
• For example, if the events $A_1$, $A_2$, $A_3$, $A_4$ are independent, one obtains relations such as
  $$P(A_1\cup A_2|A_3\cap A_4)=P(A_1\cup A_2)$$($P(A_1\cup A_2|A_3\cap A_4)=P(A_1|A_3\cap A_4)+P(A_2|A_3\cap A_4)+P(A_1\cap A_2|A_3\cap A_4)=P(A_1)+P(A_2)+P(A_1\cap A_2)=P(A_1\cup A_2)$)
  or$$P(A_1\cup A_2^C|A_3^C\cap A_4)=P(A_1\cup A_2^C)$$

Independent Trials and the Binomial Probabilities

独立实验 和 二项概率

• If an experiment involves a sequence of independent but identical stages, we say that we have a sequence of independent trials.

---

• In the special case where there are only two possible results at each stage, we say that we have a sequence of independent Bernoulli trials (伯努利试验序列).
  • We can visualize independent Bernoulli trials by means of a sequential description, as shown below for the case where $n=3$.
  • By multiplying the conditional probabilities along the corresponding path of the tree, we see that any particular outcome that involves $k$ heads and $3-k$ tails has probability $p^k(1-p{)}^{3-k}$ .
  • This formula extends to the case of a general number $n$ of tosses. We obtain that the probability of any particular $n$-long sequence that contains $k$ heads and $n-k$ tails is $p^k(1-p{)}^{n-k}$ . for all $k$ from $0$ to $n$.
    

---

• Let us now consider the probability
  
  
  where we use the notation
  
  The number $\left(\begin{array}{c}n\\ k\end{array}\right)$ (read as "$n$ choose $k$") are known as binominal coefficients (二项式系数), while the probabilities $p(k)$ are known as the binomial probabilities (二项概率).
  
  Note that the binomial probabilities $p(k)$ must add to 1, thus showing the binomial formula
  

Exercises

Problem 30.
A hunter has two hunting dogs. One day, on the trail of some animal, the hunter comes to a place where the road diverges into two paths. He knows that each dog. independent of the other. will choose the correct path with probability $p$. The hunter decides to let each dog choose a path, and if they agree, take that one, and if they disagree, to randomly pick a path. Is his strategy better than just letting one of the two dogs decide on a path?

SOLUTION

• The events that lead to the correct path are:
  
• The above events are disjoint, so we can add the probabilities
  
• Thus, the two strategies are equally effective.

---

Problem 33.
Using a biased coin to make an unbiased decision. Alice and Bob want to choose between the opera and the 1novies by tossing a fair coin. Unfortunately, the only available coin is biased (though the bias is not known exactly). How can they use the biased coin to make a decision so that either option (opera or the movies) is equally likely to be chosen?

SOLUTION

• Flip the coin twice. If the outcome is heads-tails, choose the opera. if the outcome is tails-heads, choose the movies. Otherwise, repeat the process, until a decision can be made.
• Let $A_k$ be the event that a decision was made at the $k$th round. Conditional on the event $A_k$, the two choices are equally likely, and we have
  

---

Problem 41.
Consider a game show with an infinite pool of contestants, where at each round $i$, contestant $i$ obtains a number by spinning a continuously calibrated wheel. The contestant with the smallest number thus far survives. Successive wheel spins are independent and we assume that there are no ties. Let $N$ be the round at which contestant $1$ is eliminated. For any positive integer $n$, find $P(N=n)$.

SOLUTION 1

• For $i\le j$, $A_{i,j}$ is the event that contestant $i$'s number is the smallest of the numbers of contestants $1,...,j$. We have
  
  where for
  $$P(A_{1,n-1})=\frac{1}{n-1}$$
• We claim that$$P(A_{n,n}|A_{1,n-1})=P(A_{n,n})=\frac{1}{n}$$The reason is that by symmetry, we have
  
  while by the total probability theorem,
  
  Hence
  $$P(N=n)=\frac{1}{(n-1)n}$$

SOLUTION 2

• Let us fix a particular choice of $n$. Think of an outcome of the experiment as an ordering of the values of the $n$ contestants, so that there are $n!$ equally likely outcomes. The event { N = n } \{N = n\} {N=n} occurs if and only if the first contestant’s number is smallest among the first $n-1$ contestants, and contestant $n$'s number is the smallest among the first $n$ contestants. This event can occur in $(n-2)!$ different ways, namely, all the possible ways of ordering contestants $2,...,n-1$. Thus, the probability of this event is $(n-2)!/n!=1/(n(n-1))$, in agreement with the previous solution.

也可以设第一个人得到数 $i$，则 $P(N=n)=i^{n-2}(1-i)$，然后将左式在 $[0,1]$ 上对 $i$ 积分，也可以得到相同的答案

---

Problem 42. Gambler’s ruin.
A gambler makes a sequence of independent bets. In each bet, he wins $1 with probability $p$, and loses $1 with probability $1-p$. Initially, the gambler has $k$, and plays until he either accumulates $$n$ or has no money left. What is the probability that the gambler will end up with $n?

SOLUTION

• Let us denote by $A$ the event that he ends up with $n, and by $F$ the event that he wins the first bet. Denote also by $w_k$ the probability of event $A$, if he starts with $k. We apply the total probability theorem to obtain
  
  where $q=1-p$. $P(A|F)=w_{k+1}$ and $P(A|F^C)=w_{k-1}$. Thus, we have $w_k=p{w}_{k+1}+q{w}_{k-1}$, ,which can be written as
  
  where $r=q/p$. We will solve for $w_k$ in terms of $p$ and $q$ using iteration, and the boundary values $w_0=0$ and $w_n=1$.
• We have $w_{k+1}-w_k=r^k(w_1-w_0)$, and since $w_0=0$,
  
  We have
  
  Since $w_n=1$, we can solve for $w_1$ and therefore for $w_k$:
  
  so that
  

---

Problem 46. Laplace’s rule of succession.
Consider $m+1$ boxes with the $k$th box containing $k$ red balls and $m-k$ white balls, where $k$ ranges from $0$ to $m$. We choose a box at random (all boxes are equally likely) and then choose a ball at random from that box, $n$ successive times (the ball drawn is replaced each time, and a new ball is selected independently). Suppose a red ball was drawn each of the $n$ times. What is the probability that if we draw a ball one more time it will be red? Estimate this probability for large $m$.

SOLUTION

• We want to find the conditional probability $P(E|R_n)$， where $E$ is the event of a red ball drawn at time $n+1$, and $R_n$ is the event of a red ball drawn each of the $n$ preceding times.
  $$P(E|R_n)=\frac{P(E\cap R_n)}{P(R_n)}$$
  and by using the total probability theorem, we obtain
  
• For large $m$, we can view $P(R_n)$ as a piecewise constant approximation to an integral:
  
  Similarly,
  $$P(E\cap R_n)=P(R_{n+1})\approx \frac{1}{n+2}$$
  so that
  $$P(E|R_n)\approx \frac{n+1}{n+2}$$Thus, for large $m$, drawing a red ball one more time is almost certain when $n$ is large.

---

Problem 48. The Borel-Cantelli lemma (博雷尔-坎泰利引理).
Consider an infinite sequence of trials. The probability of success at the $i$th trial is some positive number $p_i$. Let $N$ be the event that there is no success, and let $I$ be the event that there is an infinite number of successes

• (a) Assume that the trials are independent and that ${\sum }_{i=1}^{\infty }{p}_i=\infty$. Show that $P(N)=0$ and $P(I)=1.$
• (b) Assume that ${\sum }_{i=1}^{\infty }{p}_i<\infty$. Show that $P(I)=0$.

SOLUTION
(a)

• The event $N$ is a subset of the event that there were no successes in the first $n$ trials, so that
  $$P(N)\le \prod _{i=1}^n(1-p_i)$$Taking logarithms,
  $$logP(N)\le \sum _{i=1}^{n}log(1-p_i)\le \sum _{i=1}^n(-p_i)$$Taking the limit as $n$ tends to infinity, we obtain $logP(N)=-\infty$. or $P(N)=0$.
• Let now $L_n$ be the event that there is a finite number of successes and that the last success occurs at the $n$th trial. We use the already established result $P(N)=0$, and apply it to the sequence of trials after trial $n$, to obtain $P(L_n)=0$. The event $I^C$ (finite number of successes) is the union of the disjoint events $L_n$ , $n\ge 1$. and $N$, so that
  $$P(I^C)=P(N)+\sum _{n=1}^{\infty }P(L_n)=0$$and $P(I)=1$.

(b)

• Let $S_i$ be the event that the $i$th trial is a success. Fix some number $n$ and for every $i>n$, let $F_i$ be the event that the first success after time $n$ occurs at time $i$. Note that $F_i\subset S_i$. Finally, let $A_n$ be the event that there is at least one success after time $n$. Note that $I\subset A_n$. Furthermore, the event $A_n$ is the union of the disjoint events $F_i,i>n$. Therefore,
  
  We take the limit of both sides as $n\to \infty$. Because of the assumption ${\sum }_{i=1}^{\infty }{p}_i<\infty$, the right-hand side converges to zero. This implies that $P(I)=0$.