Chapter 9 (Classical Statistical Inference): Binary Hypothesis Testing

本文为 $Introduction$ $to$ $Probability$ 的读书笔记

Binary Hypothesis Testing

• In this section, we revisit the problem of choosing between two hypotheses, but unlike the Bayesian formulation, we will assume no prior probabilities. We may view this as an inference problem where the parameter $\theta$ takes just two values, but consistent with historical usage, we will forgo the $\theta$-notation and denote the two hypotheses as $H_0$ and $H_1$. In traditional statistical language, hypothesis $H_0$ is often called the null hypothesis and $H_1$ the alternative hypothesis. This indicates that $H_0$ plays the role of a default model, to be proved or disproved on the basis of available data.
• The available observation is a vector $X=(X_1,...,X_n)$ of random variables whose distribution depends on the hypothesis. We want to find a decision rule that maps the realized values $x$ of the observation to one of the two hypotheses.
  

rejection/acceptance region

• Any decision rule can be represented by a partition of the set of all possible values of the observation vector $X=(X_1,...,X_n)$ into two subsets: a set $R$, called the rejection region, and its complement, $R^C$, called the acceptance region. Hypothesis $H_0$ is rejected (declared to be false) when the observed data $X=(X_1,...,X_n)$ happen to fall in the rejection region $R$ and is accepted otherwise. Thus, the choice of a decision rule is equivalent to choosing the rejection region.
  
• For a particular choice of the rejection region $R$, there are two possible
  types of errors:
  • (a) Reject $H_0$ even though $H_0$ is true. This is called a false rejection, and happens with probability
    $$\alpha (R)=P(X\in R;H_0)$$
  • (b) Accept $H_0$ even though $H_0$ is false. This is called a false acceptance, and happens with probability
    $$\beta (R)=P(X\notin R;H_1)$$

Binary Hypothesis Testing

• To motivate a particular form of rejection region, we draw an analogy with Bayesian hypothesis testing: given the observed value $x$ of $X$, declare $H_1$ to be true if
  $$p_X(x;H_1)>p_X(x;H_0)$$This decision rule can be rewritten as follows: define the likelihood ratio (似然比) $L(x)$ by
  $$L(x)=\frac{p_X(x;H_1)}{p_X(x;H_0)}$$and declare $H_1$ to be true if the realized value $x$ of the observation vector $X$ satisfies
  $$L(x)>\xi$$, where the critical value (临界值) $\xi =1$. If $X$ is continuous, the approach is the same, except that the likelihood ratio is defined as a ratio of PDFs:
  $$L(x)=\frac{f_X(x;H_1)}{f_X(x;H_0)}$$
• We are led to consider rejection regions of the form
  R = { x ∣ L ( x ) > ξ } R=\{x|L(x)>\xi\} R={x∣L(x)>ξ}The critical value $\xi$ remains free to be chosen on the basis of other considerations. The special case where $\xi =1$ corresponds to the ML rule.

This is the Bayesian hypothesis testing with a flat prior (均匀先验)

---

Example 9.10.

• We have a six-sided die that we want to test for fairness, and we formulate two hypotheses for the probabilities of the six faces:
  
• The likelihood ratio for a single roll $x$ of the die is
• Sinece the likelihood ratio takes only two distinct values. there are three possibilities to consider for the critical value $\xi$, with three corresponding rejection regions:In fact. for a single roll of the die. the test makes sense only in the case $3/4<\xi <3/2$, since for other values of $\xi$, the decision does not depend on the observation.
• The error probabilities can be calculated from the problem data for each critical value. In particular. the probability of false rejection $P(Reject{H}_0;H_0)$ isand the probability of false acceptance $P(Accept{H}_0;H_1)$ is
  

Likelihood Ratio Test (LRT)

• Note that choosing $\xi$ trades off the probabilities of the two types of errors.
  • Indeed, as $\xi$ increases, the rejection region becomes smaller. As a result, the false rejection probability $\alpha (R)$ decreases, while the false acceptance probability $\beta (R)$ increases.
• Because of this tradeoff, there is no single best way of choosing the critical value. The most popular approach is as follows. (似然比检验)
  Typical choices for $\alpha$ are $\alpha =0.1,\alpha =0.05$, or $\alpha =0.01$, depending on the degree of undesirability of false rejection.

Neyman-Pearson Lemma (内曼-皮尔逊引理)

• We have motivated so far the use of a LRT through an analogy with Bayesian inference. However, we will now provide a stronger justification: for a given false rejection probability, the LRT offers the smallest possible false acceptance probability.
  

---

• For a justification of the Neyman-Pearson Lemma, consider a hypothetical Bayesian decision problem where the prior probabilities of $H_0$ and $H_1$ satisfy
  $$\frac{p_{\Theta }({\theta }_0)}{p_{\Theta }({\theta }_1)}=\xi$$so that
  $$p_{\Theta }({\theta }_0)=\frac{\xi }{1+\xi },p_{\Theta }({\theta }_1)=\frac{1}{1+\xi }$$Then, the threshold used by the MAP rule is equal to $\xi$, and the MAP rule is identical to the LRT rule. ($L(X)=\frac{p_{X|\Theta }(x|{\theta }_1)}{p_{X|\Theta }(x|{\theta }_0)}>\frac{p_{\Theta }({\theta }_0)}{p_{\Theta }({\theta }_1)}=\xi$) The probability of error with the MAP rule is
  $$e_{MAP}=\frac{\xi }{1+\xi }\alpha +\frac{1}{1+\xi }\beta$$and from Section 8.2, we know that it is smaller than or equal to the probability of error of any other Bayesian decision rule. This implies that for any choice of rejection region $R$, we have
  $$e_{MAP}\le \frac{\xi }{1+\xi }P(X\in R;H_0)+\frac{1}{1+\xi }P(X\notin R;H_1)$$Comparing the preceding two relations, we see that if $P(X\in R;H_0)\le \alpha$, we must have $P(X\notin R;H_1)\ge \beta$, and that if $P(X\in R;H_0)<\alpha$, we must have $P(X\notin R;H_1)>\beta$, which is the conclusion of the Neyman-Pearson Lemma.
• The Neyman-Pearson Lemma can be interpreted geometrically as shown in Fig. 9.11.
  

---

Example 9.13. Comparison of Different Rejection Regions.

• We observe two i.i.d. normal random variables $X_1$ and $X_2$, with unit variance. Under $H_0$ their common mean is 0; under $H_1$, their common mean is 2. We fix the false rejection probability to $\alpha =0.05$.
• We first derive the form of the LRT, and then calculate the resulting value of $\beta$. The likelihood ratio is of the form
  L ( x ) = 1 2 π exp ⁡ { − ( ( x 1 − 2 ) 2 + ( x 2 − 2 ) 2 ) / 2 } 1 2 π exp ⁡ { − ( x 1 2 + x 2 2 ) / 2 } = exp ⁡ { 2 ( x 1 + x 2 − 4 ) } L(x)=\frac{\frac{1}{\sqrt{2\pi}}\exp\big\{-\big((x_1-2)^2+(x_2-2)^2\big)/2\big\}}{\frac{1}{\sqrt{2\pi}}\exp\big\{-(x_1^2+x_2^2)/2\big\}}=\exp\{2(x_1+x_2-4)\} L(x)=2π ​1​exp{−(x12​+x22​)/2}2π ​1​exp{−((x1​−2)2+(x2​−2)2)/2}​=exp{2(x1​+x2​−4)}Comparing $L(x)$ to a critical value $\xi$ is equivalent to comparing $x_1+x_2$ to $\gamma =(4+log\xi )/2$. Thus, under the LRT, we decide in favor of $H_1$ if $x_1+x_2>\gamma$ for some particular choice of $\gamma$.
• To determine the exact form of the rejection region, we need to find $\gamma$ so that the false rejection probability $P(X_1+X_2>\gamma ;H_0)$ is equal to $0.05$. We note that under $H_0$, $Z=(X_1+X_2)/\sqrt{2}$ is a standard normal random variable. We have
  $$0.05=P(X_1+X_2>\gamma ;H_0)=P(\frac{X_1+X_2}{\sqrt{2}}>\frac{\gamma }{\sqrt{2}};H_0)=P(Z>\frac{\gamma }{\sqrt{2}})$$From the normal tables, we obtain $P(Z>1.645)=0.05$, so we choose
  $$\gamma =1.645\cdot \sqrt{2}=2.33$$resulting in the rejection region
  R = { ( x 1 , x 2 ) ∣ x 1 + x 2 > 2.33 } R=\{(x_1,x_2)|x_1+x_2>2.33\} R={(x1​,x2​)∣x1​+x2​>2.33}
• To evaluate the performance of this test, we calculate the resulting false acceptance probability.
  $$\begin{gathered} \beta (R)=P(X_1+X_2\le 2.33;H_1)=P(\frac{X_1+X_2-4}{\sqrt{2}}\le -1.18;H_1) \\ =1-\Phi (1.18)=0.12 \end{gathered}$$
• We now compare the performance of the LRT with that resulting from a different rejection region $R^{\prime }$. For example, let us consider a rejection region of the form
  R ′ = { ( x 1 , x 2 ) ∣ max ⁡ { x 1 , x 2 } > ξ } R'=\{(x_1,x_2)|\max\{x_1,x_2\}>\xi\} R′={(x1​,x2​)∣max{x1​,x2​}>ξ}where $\xi$ is chosen so that the false rejection probability is again 0.05. To determine the value of $\xi$, we write
  0.05 = P ( max ⁡ { x 1 , x 2 } > ξ ; H 0 ) = 1 − P ( max ⁡ { x 1 , x 2 } ≤ ξ ; H 0 ) = 1 − P ( X 1 ≤ ξ ; H 0 ) P ( X 2 ≤ ξ ; H 0 ) = 1 − ( P ( Z ≤ ξ ; H 0 ) ) 2 0.05=P(\max\{x_1,x_2\}>\xi;H_0)=1-P(\max\{x_1,x_2\}\leq\xi;H_0) \\=1-P(X_1\leq\xi;H_0)P(X_2\leq\xi;H_0)=1-(P(Z\leq\xi;H_0))^2 0.05=P(max{x1​,x2​}>ξ;H0​)=1−P(max{x1​,x2​}≤ξ;H0​)=1−P(X1​≤ξ;H0​)P(X2​≤ξ;H0​)=1−(P(Z≤ξ;H0​))2where $Z$ is a standard normal. This yields $\Phi (z)\approx 0.975$. Using the normal tables, we conclude that $\xi =1.96$. Let us now calculate the resulting false acceptance probability
  β ( R ′ ) = P ( max ⁡ { x 1 , x 2 } ≤ ξ ; H 1 ) = ( P ( X 1 ≤ 1.96 ) ) 2 = ( P ( Z ≤ − 0.04 ) ) 2 = 0.24 \beta(R')=P(\max\{x_1,x_2\}\leq\xi;H_1)=(P(X_1\leq1.96))^2 \\=(P(Z\leq-0.04))^2=0.24 β(R′)=P(max{x1​,x2​}≤ξ;H1​)=(P(X1​≤1.96))2=(P(Z≤−0.04))2=0.24
• We see that the false acceptance probability $\beta (R)=0.12$ of the LRT is much better than the false acceptance probability $\beta (R^{\prime })=0.24$ of the alternative test.

---

Example 9.14. A Discrete Example.
Consider $n=25$ independent tosses of a coin. Under hypothesis $H_0$ (respectively, $H_1$), the probability of a head at each toss is equal to ${\theta }_0=1/2$ (respectively, ${\theta }_1=2/3$). Let $X$ be the number of heads observed. If we set the false rejection probability to 0.1, what is the rejection region associated with the LRT?

SOLUTION

• We observe that when $X=k$, the likelihood ratio is of the form
  $$L(k)=2^k(\frac{2}{3}{)}^{25}$$Note that $L(k)$ is a monotonically increasing function of $k$. Thus, the rejection condition $L(k)>\xi$ is equivalent to a condition $k>\gamma$, for a suitable value of $\gamma$. We conclude that the LRT is of the form
  $$reject{H}_{0}ifX>\gamma$$
• To guarantee the requirement on the false rejection probability, we need to find the smallest possible value of, for which $P(X>\gamma ;H_0)\le 0.1$, or
  $$\sum _{i=\gamma +1}^{25}\left(\begin{array}{c}25\\ i\end{array}\right)2^{-25}\le 0.1$$By evaluating numerically the right-hand side above for different choices of $\gamma$, we find that the required value is $\gamma =16$. (可以用二分法查找)
• An alternative method for choosing $\gamma$ involves an approximation based on the central limit theorem. Under $H_0$,
  $$Z=\frac{X-n{\theta }_0}{\sqrt{n{\theta }_0(1-{\theta }_0)}}=\frac{X-12.5}{\sqrt{25/4}}$$is approximately a standard normal random variable. Therefore, we need
  $$0.1=P(X\ge \gamma ;H_0)=P(\frac{X-12.5}{\sqrt{25/4}}>\frac{\gamma -12.5}{\sqrt{25/4}};H_0)=P(Z>\frac{2\gamma }{5}-5)$$From the normal tables, we have $\Phi (1.28)=0.9$, and therefore, we should choose $\gamma$ so that $\frac{2\gamma }{5}-5=1.28$, or $\gamma =15.7$. Since $X$ is integer-valued, we find that the LRT should reject $H_0$ whenever $X>15$.