Covariance and Unbiased Estimate

1. Covariance and Correlation

Let $x_i$ and $x_j$ be two real random variables in a random vector $x=[x_1,\cdots ,x_N{]}^T$.
The mean and varicance of a variable $x_i$ and the covariance and correlation coefficient (normalized correlation) between two variables $x_i$ and $x_j$ are defined below:

• Mean of $x_i:$ ${\mu }_i=E(x_i)$
• Variance of $x_i:$ ${\sigma }_i^2=E[(x_i-{\mu }_i{)}^2]=E(x_i^2)-{\mu }_i^2$
• Covariance of $x_i$ and $x_j:$ ${\sigma }_{ij}^2=E[(x_i-{\mu }_i)(x_j-{\mu }_j)]=E(x_i{x}_j)-{\mu }_i{\mu }_j$
• Correlation coefficient between $x_i$ and $x_j$:$r_{ij}=\frac{{\sigma }_{ij}^2}{\sqrt{{\sigma }_i^2{\sigma }_j^2}}=\frac{{\sigma }_{ij}^2}{{\sigma }_i{\sigma }_j}$

Note that: the correlation coefficient $r_{ij}$ can be considered as the normalized covariance ${\sigma }_{ij}^2$.

To obtain these parameters as expectations of the first and second order functions of the random variables, the joint probability density function $p(x_1,\cdots ,x_N)$ is required.
However, when it is not avaiable, the parameters can still be estimated by averaging the outcomes of a random experiment involving these variables repeated $K$ times:

1.1 examples

Assume the experiment concering $x_i$ and $x_j$ is repeated $K=3$ times with the following outcomes:

so, we can get

We see that $x_i$ and $x_j$ are highly correlated.

---

2. Unbiased estimate

Defination: Let $X$ is overall, $\theta \in \Theta$ is an under-estimated parameter which is inclued in the distribution of $X$, and $X_1,X_2,\cdots ,X_n$ is an sample from $X$. If the expectation of the estimation $\hat{\theta }=\hat{\theta }(X_1,X_2,\cdots ,X_n)$ exists, while the equaction $E(\hat{\theta })=\theta$ holds for any $\theta \in \Theta$. Then $\hat{\theta }$ is called the unbiased estimate of $\theta$.

Example1: Let $\mu$ and ${\sigma }^2$ as the mean and variance of $X$, they are unknown, then the estimator of ${\sigma }^2$
$$\widehat{{\sigma }^2}=\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2$$
is a biased estimator.
Proof: As
$$\widehat{{\sigma }^2}=\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2=\frac{1}{n}\sum _{i=1}^n{X}_i^2-{\bar{X}}^2,$$
$$E(\widehat{{\sigma }^2})=E(\frac{1}{n}\sum _{i=1}^n{X}_i^2)-E({\bar{X}}^2)=\frac{1}{n}\sum _{i=1}^{n}E(X_i^2)-E({\bar{X}}^2),$$
and
$$E(X_i^2)=var(X_i)+[E(X_i){]}^2={\sigma }^2+{\mu }^2,$$
$$E({\bar{X}}^2)=var(\bar{X})+[E(\bar{X}){]}^2=\frac{{\sigma }^2}{n}+{\mu }^2,$$
then,
$E({\hat{\sigma }}^2)={\sigma }^2+{\mu }^2-(\frac{{\sigma }^2}{n}+{\mu }^2)=\frac{n-1}{n}{\sigma }^2\ne {\sigma }^2$.
So, ${\hat{\sigma }}^2$ is a biased estimate. If we use ${\hat{\sigma }}^2$ to estimate the value of ${\sigma }^2$, it will be less than the real vaule. (However, when the sample size $n\to \infty$, ${\lim }_{n\to \infty }[E({\hat{\sigma }}^2)-{\sigma }^2]=0$, so the ${\hat{\sigma }}^2$ is called asymptotically unbiased estimation).
For the Sample variance,
$$S^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X}{)}^2=\frac{n-1}{n}{\hat{\sigma }}^2,$$
$$E(S^2)=\frac{n}{n-1}E({\hat{\sigma }}^2)=\frac{n}{n-1}\cdot \frac{n-1}{n}{\sigma }^2={\sigma }^2.$$
That is to say, the sample variance $S^2$ is a unbiased estimator of ${\sigma }^2$. Thus we usually use $S^2$ as the estimator of ${\sigma }^2$.

---

Reference

【1】Covariance and Correlation
【2】统计建模与R软件. 薛毅，陈立萍著.