Multivariate Analysis ch1 (Overview)

This is the note of book “Applied multivariate methods for data analysts”1.

Outlines:

• Data: Response Variable v.s. Experimental Units;
• Type of Methods: Variable-directed techinique v.s. Individual-directed techniques;
• Notations and definitions;
• Estimator Statistics;
• About multivariate computing: outliers, missing values, standardization;

1. Data

Multivariate data is common but complex, thus is very important and the major goal is simplify it.

Two aspects of data:

1. Response variables;
2. The experimental units;

Methods focus on the relationship among the response variables, the relationship among the experimental units and the relationship between the response variables and the experimental units;

2. Type of Methods

2.1 Variable-directed techinique

• PCA (Principle component analysis)
• FA (Factor analysis)
• regression (Logistic regression)
• CCA (Canonical correlation analysis)

These methods mainly operates on the correlation matrix and focus on the column of data matrix: the response variable.

2.2 Individual-directed techniques

• DA (Discriminant analysis)
• CA (Cluster analysis)
• MANOVA (Multivariate analysis of variance)

Remarks:

1. These methods focus on the row of data matrix: the observations or experimental units;
2. Many MA methods require the Independence of experimental units;

3. Notations and definitions

Notation	Explanation
$p$	variables
$N$	sample size
$\mathbf{X}=(x_{rj})_{N \times p},r=1,...,N;j=1,...,p$	data matrix
$\mathbf{x}_r=(x_{r1},...,x_{rp})'$	the $r$’th observation
$r,s,t$	subscript for experimental units
$i,j,k$	subscript for response variables

3.1 Multivariate normal distribution

(Def): $\mathbf{x}=(x_1,...,x_p)'$ follows a multivariate normal distribution if $\forall \mathbf{a}$,

$$\mathbf{a'x}=\sum_{i=1}^p a_ix_i$$
follows a univariate normal distribution.
Denote it by: $\mathbf{x} \sim N_p(\mathbf{\mu},\mathbf{\Sigma})$, the p.d.f is :
$$f_{\mathbf{x}}(\mathbf{x,\mu,\Sigma})=\frac{1}{\sqrt{(2\pi)^p}|\Sigma|^{1/2}}exp\{-\frac{1}{2}\mathbf{[(x-\mu)'\Sigma^{-1}(x-\mu)}]\};$$

3.2 Mathmatical numbers:

• Mean vector: $\mathbf{\mu}=E(\mathbf{x})=(E(x_1),...,E(x_p))'=(\mu_1,...,\mu_p)';$
• Variance-covariance matrix: $\Sigma = cov(\mathbf{X}=E[\mathbf{(x-\mu)(x-\mu)’}])=(\sigma_{ij})_{p \times p}= \left( \begin{array}{cccc} \sigma_{11} & \sigma_{12} & \ldots & \sigma_{1p}\\ \vdots & \vdots & \ddots & \vdots \\ \sigma_{p1} & \sigma_{p2} & \ldots & \sigma_{pp}\\ \end{array} \right)$
  
  • $\sigma_{ii}=E[(\mathbf{x_i-\mu_i})^2]$;
  • $\sigma_{ij}=cov(\mathbf{x_i,x_j})$;
• Correlation matrix;
  $\mathbf{P} = \left( \begin{array}{cccc} 1 & \rho_{12} & \ldots & \rho_{1p}\\ \rho_{21} & 1 & \ldots & \rho_{2p}\\ \vdots & \vdots & \ddots & \vdots \\ \rho_{p1} & \rho_{p2} & \ldots & 1\\ \end{array} \right)$

4. Estimator Statistics;

4.1. Unbiased estimators:

$$\hat{\mathbf{\mu}}= \frac{1}{N}(\sum_{r=1}^N \mathbf{x}_r)=rowMeans(\mathbf{X});$$
$$\hat{\Sigma} = \frac{1}{N-1} [\sum_{r=1}^N \mathbf{(x_r-\hat{\mu})(x_r-\hat{\mu})'}]=(\hat{\sigma}_{ij})_{p \times p};$$
$$\hat{\sigma}_{ij} = \frac{1}{N-1} [\sum_{r=1}^N (x_{ri}-\hat{\mu}_i)(x_{rj}-\hat{\mu}_j)')];$$

4.2. Biased but commonly use estimators:

$$r_{ij} = \hat{\rho}_{ij}= \frac{\hat{\sigma}_{ij}}{\sqrt{\hat{\sigma}_{ii}\hat{\sigma}_{jj}}};$$
$$\mathbf{R}=\hat{\mathbf{P}}= \left( \begin{array}{cccc} 1 & r_{12} & \ldots & r_{1p}\\ r_{21} & 1 & \ldots & r_{2p}\\ \vdots & \vdots & \ddots & \vdots \\ r_{p1} & r_{p2} & \ldots & 1\\ \end{array} \right);$$

5. About multivariate computing: outliers, missing values, standardization;

5.1 outliers

Detect it by plot or PCA;
Dealing with it:: analyze the impact of outliers on the results (with outliers v.s. without outliers);

5.2 missing values

use row means (or  KNN) to replace it;
remove the corresponding row;

5.3 standardization

$$z_{rj}=\frac{x_{rj}-\hat{\mu}_j}{\sqrt{\hat{\sigma}_{jj}}};r=1,..,N;j=1,..,p.$$
Standardization is the default operation in the computer programs.

---

References

---

1. Johnson D E, 约翰逊. Applied multivariate methods for data analysts[M]. Pacific Grove, CA: Duxbury Press, 1998. ↩