Chapter 4 (Further Topics on Random Variables): Covariance and Correlation (协方差和相关)

本文为 $Introduction$ $to$ $Probability$ 的读书笔记

Covariance

Covariance

• The covariance of two random variables $X$ and $Y$, denoted by $cov(X,Y)$, is defined by
  $$\begin{array}{rl}cov(X,Y)& =E[(X-E[X])(Y-E[Y])]\\ & =E[XY]-E[X]E[Y]\end{array}$$
  • When $cov(X,Y)=0$, we say that $X$ and $Y$ are uncorrelated.
  • Roughly speaking, a positive or negative covariance indicates that the values of $X-E[X]$ and $Y-E[Y]$ obtained in a single experiment “tend” to have the same or the opposite sign, respectively. Thus the sign of the covariance provides an important qualitative indicator of the relationship between $X$ and $Y$.
    

Properties of covariances

• For any random variables $X$, $Y$, and $Z$, and any scalars $a$ and $b$, we have
  $$\begin{gathered} \text{cov}(X,X)=\text{var}(X) \\ \text{cov}(X,aY+b)=a\cdot \text{cov}(X,Y) \\ \text{cov}(X,Y+Z)=\text{cov}(X,Y)+\text{cov}(X,Z) \end{gathered}$$
• Note thate if $X$ and $Y$ are independent, we have $\text{cov}(X,Y)=E[XY]-E[X]E[Y]=0$. Thus, if $X$ and $Y$ are independent, they are also uncorrelated. However, the converse is generally not true.
  • Assume that $X$ and $Y$ satisfy
    $$E[X|Y=y]=E[X],forally$$Then, assuming $X$ and $Y$ are discrete, the total expectation theorem implies that
    $$\begin{array}{rl}E[XY]& =\sum _y{p}_Y(y)E[XY|Y=y]=\sum _{y}y{p}_Y(y)E[X|Y=y]\\ & =\sum _{y}y{p}_Y(y)E[X]=E[X]E[Y]\end{array}$$so $X$ and $Y$ are uncorrelated. The argument for the continuous case is similar.

Example 4.13.

• The pair of random variables $(X,Y)$ takes the values $(1,0),(0,1),(-1,0)$ and $(0,-1)$. each with probability $1/4$. Therefore,
  $$\text{cov}(X,Y)=E[XY]-E[X]E[Y]=0-0=0$$and $X$ and $Y$ are uncorrelated.
• However, $X$ and $Y$ are not independent since, for example, a nonzero value of $X$ fixes the value of $Y$ to zero.

Correlation Coefficient

相关系数

• The correlation coefficient $\rho (X,Y)$ of two random variables $X$ and $Y$ that have nonzero variances is defined as (Assuming that $X$ and $Y$ have positive variances)
  $$\rho (X,Y)=\frac{\text{cov}(X,Y)}{\sqrt{\text{var(X)}\text{var(Y)}}}$$

The simpler notation $\rho$ will also be used when $X$ and $Y$ are clear from the context.

• It may be viewed as a normalized version of the covariance $\text{cov}(X,Y)$, and in fact, it can be shown that $\rho$ ranges from $-1$ to $1$.
  • If $\rho >0$ (or $\rho <0$), then the values of $X-E[X]$ and $Y-E[Y]$ “tend” to have the same (or opposite, respectively) sign. The size of $|\rho |$ normalized measure of the extent to which this is true.
  • In fact, always assuming that $X$ and $Y$ have positive variances, it can be shown that $\rho =1$ (or $\rho =-1$) if and only if there exists a positive (or negative, respectively) constant $c$ such that
    $$Y-E[Y]=c(X-E[X])$$

---

Problem 20. Schwarz inequality 施瓦兹不等式

Show that for any random variables $X$ and $Y$, we have
$$(E[XY]{)}^2\le E[X^2]E[Y^2]$$

SOLUTION

• We may assume that $E[Y^2]\ne 0$; otherwise, we have $Y=0$ with probability 1, and hence $E[XY]=0$. so the inequality holds.
• We have
  $$\begin{array}{rl}0& \le E[(X-\frac{E[XY]}{E[Y^2]}Y{)}^2]\\ & =E[X^2-2\frac{E[XY]}{E[Y^2]}XY+\frac{(E[XY]{)}^2}{(E[Y^2]{)}^2}{Y}^2]\\ & =E[X^2]-2\frac{E[XY]}{E[Y^2]}E[XY]+\frac{(E[XY]{)}^2}{(E[Y^2]{)}^2}E[Y^2]\\ & =E[X^2]-\frac{(E[XY]{)}^2}{E[Y^2]}\end{array}$$i.e., $(E[XY]{)}^2\le E[X^2]E[Y^2]$

---

Problem 21. Correlation coefficient.

Consider the correlation coefficient
$$\rho (X,Y)=\frac{cov(X,Y)}{\sqrt{var(X)var(Y)}}$$of two random variables $X$ and $Y$ that have positive variances. Show that:

• $(a)$ $|\rho (X,Y)|\le 1$. [Hint: Use the Schwarz inequality from the preceding problem.]
• $(b)$ If $Y-E[Y]$ is a positive (or negative) multiple of $X-E[X]$, then $\rho (X,Y)=1$ [or $\rho (X.Y)=-1$, respectively].
• $(c)$ If $\rho (X,Y)=1$ [or $\rho (X,Y)=-1$], then, with probability 1, $Y-E[Y]$ is a positive (or negative. respectively) multiple of $X-E[X]$.

SOLUTION

• $(a)$ Let $\tilde{X}=X-E[X]$ and $\tilde{Y}=Y-E[Y]$. Using the Schwarz inequality, we get
  $$\rho (X,Y{)}^2=\frac{(E[\tilde{X}\tilde{Y}]{)}^2}{E[{\tilde{X}}^2]E[{\tilde{Y}}^2]}\le 1$$and hence $|\rho (X,Y)|\le 1$.
• $(b)$ If $\tilde{Y}=a\tilde{X}$, then
  $$\rho (X,Y)=\frac{E[\tilde{X}a\tilde{X}]}{\sqrt{E[\widetilde{X^2}]E[(a\tilde{X}{)}^2]}}=\frac{a}{|a|}$$
• $(c)$ If $|\rho (X,Y)|=1$, the calculation in the solution of Problem 20 yields
  $$\begin{array}{rl}E[(\tilde{X}-\frac{E[\tilde{X}\tilde{Y}]}{E[{\tilde{Y}}^2]}\tilde{Y}{)}^2]& =E[{\tilde{X}}^2]-\frac{(E[\tilde{X}\tilde{Y}]{)}^2}{E[{\tilde{Y}}^2]}\\ & =E[{\tilde{X}}^2](1-(\rho (X,Y){)}^2)\\ & =0\end{array}$$Thus, with probability 1, the random variable
  $$\tilde{X}-\frac{E[\tilde{X}\tilde{Y}]}{E[{\tilde{Y}}^2]}\tilde{Y}$$is equal to zero. It follows that, with probability 1,
  $$\tilde{X}=\frac{E[\tilde{X}\tilde{Y}]}{E[{\tilde{Y}}^2]}\tilde{Y}=\sqrt{\frac{E[{\tilde{X}}^2]}{E[{\tilde{Y}}^2]}}\rho (X,Y)\tilde{Y}$$i.e., the sign of the constant ratio of $X$ and $Y$ is determined by the sign of $\rho (X,Y)$.

Covariance Matrix (协方差矩阵)

• 设 $X_1,...,X_n$ 为 $n$ 个随机变量, ${\mu }_i=E\left(X_i\right)$，令 ${\sigma }_{ij}=\text{cov}(X_i,X_j)$, 称矩阵
  $$\begin{array}{rl}\mathbf{\Sigma }& =({\sigma }_{ij}{)}_{n\times n}\\ & =\left[\begin{array}{cccc}E\left[\left(X_1-{\mu }_1\right)\left(X_1-{\mu }_1\right)\right]& E\left[\left(X_1-{\mu }_1\right)\left(X_2-{\mu }_2\right)\right]& \cdots & E\left[\left(X_1-{\mu }_1\right)\left(X_n-{\mu }_n\right)\right]\\ E\left[\left(X_2-{\mu }_2\right)\left(X_1-{\mu }_1\right)\right]& E\left[\left(X_2-{\mu }_2\right)\left(X_2-{\mu }_2\right)\right]& \cdots & E\left[\left(X_2-{\mu }_2\right)\left(X_n-{\mu }_n\right)\right]\\ ⋮& ⋮& \ddots & ⋮\\ E\left[\left(X_n-{\mu }_n\right)\left(X_1-{\mu }_1\right)\right]& E\left[\left(X_n-{\mu }_n\right)\left(X_2-{\mu }_2\right)\right]& \cdots & E\left[\left(X_n-{\mu }_n\right)\left(X_n-{\mu }_n\right)\right]\end{array}\right]\end{array}$$为 $X_1,...,X_n$ 的协方差矩阵

---

• 设 $\mathbf{X}=\left[\begin{array}{c}{X}_1\\ ⋮\\ X_n\end{array}\right]\in {\mathbb{R}}^n,\mathbf{\mu }=\left[\begin{array}{c}{\mu }_1\\ ⋮\\ {\mu }_n\end{array}\right]\in {\mathbb{R}}^n$, 则
  $$\mathbf{\Sigma }=\mathbb{E}\left[(\mathbf{X}-\mathbf{\mu })(\mathbf{X}-\mathbf{\mu }{)}^{\top }\right]$$

协方差矩阵是半正定矩阵

$$\begin{array}{rl}{\mathbf{x}}^{\mathrm{T}}\mathbf{\Sigma }\mathbf{x}& ={\mathbf{x}}^{\mathrm{T}}\mathbb{E}\left[(\mathbf{X}-\mathbf{\mu })(\mathbf{X}-\mathbf{\mu }{)}^{\top }\right]\mathbf{x}\\ & =\mathbb{E}\left[{\mathbf{x}}^{\mathrm{T}}(\mathbf{X}-\mathbf{\mu })(\mathbf{X}-\mathbf{\mu }{)}^{\top }\mathbf{x}\right]\\ & =\mathbb{E}\left[((\mathbf{X}-\mathbf{\mu }{)}^{\mathrm{T}}\mathbf{x}{)}^{\mathrm{T}}((\mathbf{X}-\mathbf{\mu }{)}^{\mathrm{T}}\mathbf{x})\right]\\ & =\mathbb{E}\left[{\Vert (\mathbf{X}-\mathbf{\mu }{)}^{\mathrm{T}}\mathbf{x}\Vert }^2\right]\\ & \ge 0\end{array}$$

Variance of the Sum of Random Variables

• If $X_1,X_2,...,X_n$ are random variables with finite variance, we have
  $$var(X_1+X_2)=var(X_1)+var(X_2)+2cov(X_1,X_2)$$and, more generally,
  v a r ( ∑ i = 1 n X i ) = ∑ i = 1 n v a r ( X i ) + ∑ { ( i , j ) ∣ i ≠ j } c o v ( X i , X j ) var(\sum_{i=1}^nX_i)=\sum_{i=1}^nvar(X_i)+\sum_{\{(i,j)|i\neq j\}}cov(X_i,X_j) var(i=1∑n​Xi​)=i=1∑n​var(Xi​)+{(i,j)∣i​=j}∑​cov(Xi​,Xj​)

PROOF

• For brevity, we denote ${\tilde{X}}_i=X_i-E[X_i]$, then
  v a r ( ∑ i = 1 n X i ) = E [ ( ∑ i = 1 n X ~ i ) 2 ] = E [ ∑ i = 1 n ∑ j = 1 n X ~ i X ~ j ] = ∑ i = 1 n ∑ j = 1 n E [ X ~ i X ~ j ] = ∑ i = 1 n E [ X ~ i 2 ] + ∑ { ( i , j ) ∣ i ≠ j } E [ X ~ i X ~ j ] = ∑ i = 1 n v a r ( X i ) + ∑ { ( i , j ) ∣ i ≠ j } c o v ( X i , X j ) \begin{aligned}var(\sum_{i=1}^nX_i)&=E[(\sum_{i=1}^n\tilde X_i)^2] \\&=E[\sum_{i=1}^n\sum_{j=1}^n\tilde X_i\tilde X_j] \\&=\sum_{i=1}^n\sum_{j=1}^nE[\tilde X_i\tilde X_j] \\&=\sum_{i=1}^nE[\tilde X_i^2]+\sum_{\{(i,j)|i\neq j\}}E[\tilde X_i\tilde X_j] \\&=\sum_{i=1}^nvar(X_i)+\sum_{\{(i,j)|i\neq j\}}cov(X_i,X_j) \end{aligned} var(i=1∑n​Xi​)​=E[(i=1∑n​X~i​)2]=E[i=1∑n​j=1∑n​X~i​X~j​]=i=1∑n​j=1∑n​E[X~i​X~j​]=i=1∑n​E[X~i2​]+{(i,j)∣i​=j}∑​E[X~i​X~j​]=i=1∑n​var(Xi​)+{(i,j)∣i​=j}∑​cov(Xi​,Xj​)​

---

Example 4.15.

$n$ people throw their hats in a box and then pick a hat at random. Let us find the variance of $X$, the number of people who pick their own hat.

SOLUTION

• We have
  $$X=X_1+\cdot \cdot \cdot +X_n$$where $X_1$ is the random variable that takes the value $1$ if the $i$th person selects his/her own hat, and takes the value $0$ otherwise. Noting that $X_i$ is Bernoulli with parameter $p=P(X_i=1)=1/n$, we obtain
  $$\begin{array}{rl}E[X_i]& =\frac{1}{n}\\ var(X_i)& =\frac{1}{n}(1-\frac{1}{n})\end{array}$$For $i\ne j$, we have
  $$\begin{array}{rl}cov(X_i,X_j)& =E[X_i{X}_j]-E[X_i]E[X_j]\\ & =P(X_i=1and{X}_j=1)-\frac{1}{n^2}\\ & =P(X_i=1)P(X_j=1|X_i=1)-\frac{1}{n^2}\\ & =\frac{1}{n}\cdot \frac{1}{n-1}-\frac{1}{n^2}\\ & =\frac{1}{n^2(n-1)}\end{array}$$Therefore,
  v a r ( X ) = v a r ( ∑ i = 1 n X i ) = ∑ i = 1 n v a r ( X i ) + ∑ { ( i , j ) ∣ i ≠ j } c o v ( X i , X j ) = n ⋅ 1 n ( 1 − 1 n ) + n ( n − 1 ) ⋅ 1 n 2 ( n − 1 ) = 1 \begin{aligned}var(X)&=var(\sum_{i=1}^nX_i) \\&=\sum_{i=1}^nvar(X_i)+\sum_{\{(i,j)|i\neq j\}}cov(X_i,X_j) \\&=n\cdot \frac{1}{n}(1-\frac{1}{n})+n(n-1)\cdot \frac{1}{n^2(n-1)} \\&=1 \end{aligned} var(X)​=var(i=1∑n​Xi​)=i=1∑n​var(Xi​)+{(i,j)∣i​=j}∑​cov(Xi​,Xj​)=n⋅n1​(1−n1​)+n(n−1)⋅n2(n−1)1​=1​