Covariance and Correlation

Covariance and Correlation

Demystifying the terms

Covariance indicates the direction of the linear relationship between variables.

Correlation on the other hand measures both the strength and direction of the linear relationship between two variables.

Correlation is a function of the covariance. What sets them apart is the fact that correlation values are standardized whereas, covariance values are not.

Defining the terms mathematically

Covariance

$$\begin{array}{rl}cov(x,y)& =E[(x-{\mu }_x)(y-{\mu }_y)]\\ & =E[xy]-E[x]E[y]\end{array}$$

If we have only a single variable $x$, then

$$\begin{array}{rl}cov(x,x)& =E[(x-{\mu }_x)(x-{\mu }_x)]\\ & =E[(x-{\mu }_x{)}^2]\\ & =var(x)={\sigma }^2(x)={\sigma }_x^2\\ \text{Let }var(x)& :=s^2\text{sampled varaince}\end{array}$$

Expand it, we can get

$$\begin{array}{rl}{s}^2=cov(x,x)& =\frac{{\sum }_{i=1}^N(x_i-\bar{x}{)}^2}{n-1}\\ cov(x,y)& =\frac{{\sum }_{i=1}^N(x_i-\bar{x})(y_i-\bar{y})}{n-1}\end{array}$$

The numerator of the first equation is called sum of squared deviation, and the second is called sum of cross product.

Correlation

$$\begin{array}{rl}corr(x,y)=\frac{cov(x,y)}{s_x{s}_y}& =\frac{E[(x-{\mu }_x)(y-{\mu }_y)]}{s_x{s}_y}\\ & =\frac{E[(x-{\mu }_x)(y-{\mu }_y)]}{{\sigma }_x{\sigma }_y}\end{array}$$

So the values of correlation coefficient rnge from [-1, 1]. The positive sign signifies the direction of the correlation i.e. if one of the variables increases, the other variable is also supposed to increase.

Data-matrix representation of covariance and correlation

$$X=\left[\begin{array}{ccc}{x}_{11}& ...& x_{1n}\\ ...& ...& ...\\ x_{m1}& ...& x_{mn}\end{array}\right]=\left[\begin{array}{ccc}{\mathbf{x}}_1& ...& {\mathbf{x}}_n\end{array}\right]$$

order of $X=m\times n$

We call a row is item / subject and a column variable

Now we can calculate the sample mean of $j$th variable

$${\bar{x}}_j=\frac{1}{m}\sum _{i=1}^m{x}_{ij}$$

similarly, the row-mean is

$${\bar{x}}_i=\frac{1}{n}\sum _{j=1}^n{x}_{ij}$$

We then can define the covariance matrix:

$$\begin{array}{rl}S=\frac{1}{m}\left[\begin{array}{c}{\mathbf{x}}_1-{\bar{\mathbf{x}}}_1\\ ...\\ {\mathbf{x}}_n-{\bar{\mathbf{x}}}_n\end{array}\right]\left[\begin{array}{ccc}{\mathbf{x}}_1-{\bar{\mathbf{x}}}_1& ...& {\mathbf{x}}_n-{\bar{\mathbf{x}}}_n\end{array}\right]& =\left[\begin{array}{ccc}{s}_1^2& ...& s_{1n}^2\\ ...& ...& ...\\ s_{n1}^2& ...& s_n^2\end{array}\right]\\ \text{where }{s}_j^2& =\frac{1}{m}\sum _{i=1}^m(x_{ij}-{\bar{x}}_j{)}^2\text{variance of jth variable}\\ s_{jk}& =\frac{1}{m}\sum _{i=1}^m(x_{ij}-{\bar{x}}_j)(x_{ik}-{\bar{x}}_k)\text{covariance between jth and kth variable}\\ {\bar{\mathbf{x}}}_j& =\frac{1}{m}\sum _{i=1}^m{x}_{ij}\text{mean of jth variable}\end{array}$$

We can see that the covariance matrix is a $n\times n$ symmetric matrix

Then we can define the Correlation matrix

$$\begin{array}{rl}R& =\frac{1}{m}\left[\begin{array}{c}({\mathbf{x}}_1-{\bar{\mathbf{x}}}_1)/s_1\\ ...\\ ({\mathbf{x}}_n-{\bar{\mathbf{x}}}_n)/s_n\end{array}\right]\left[\begin{array}{ccc}({\mathbf{x}}_1-{\bar{\mathbf{x}}}_1)/s_1& ...& ({\mathbf{x}}_n-{\bar{\mathbf{x}}}_n)/s_n\end{array}\right]\\ & =\left[\begin{array}{cccc}1& r_{12}& ...& r_{1n}\\ ...& ...& ...& ...\\ r_{n1}& ...& ...& 1\end{array}\right]\end{array}$$

Covariance versus Correlation

• Covariance has unit from the product of the units of the two variables
  Correlation is dimensionless

• Covariance can take value from $(-\infty ,+\infty )$
  Correlation lies between $[-1,1]$