Multivariate Gaussian Distribution
Maths
This work will present a linear transformation interpretation on the multivariate Gaussian distribution. The entire derivation stems from univariate Gaussian distributions with mean 0 and variance 1.
Recall that the density function of a univariate Gaussian distribution is given by
Therefore, when
we have
Now, suppose we have n independently identically distribution random variables x1, …, xn conforming to univariate Gaussian distributions with mean 0 and variance 1, then the joint density function is
By designing a random vector,
we can re-write the joint density function as
Remarkably, here is the assumption that will lead us to a general case, that is, there exists an invertible mixing matrix B and a noise vector mu which contributes to a linear transformation from Z to X, i.e.,
Problem is, what is density function of X? Here is the derivation. (This requires some advanced linear algebra and probability theory. The relevant proof is on the horizon.)
Letting
we can obtain
where the noise vector mu is the mean of X, since E[X] = E[BZ] + E[mu] = mu, and the n-by-n matrix SIGMA is a symmetric positive semi-definite covariance matrix of X, since
Moreover, since SIGMA is real symmetric, it can be factorized as
where U is a full rank orthogonal matrix containing the eigenvectors of SIGMA as its columns and LAMBDA is a diagonal matrix containing SIGMA’s eigenvalues which are non-negative (since SIGMA is also positive semi-definite).
Defining
to be the diagonal matrix whose entries are the square roots of the corresponding entries from LAMBDA. Noting that
we have
Therefore, the mixing matrix turns out to be
In addition, by inversing the derivation above, we can substantiate the probabilistic standardization of multivariate Gaussian distribution, i.e.,
Proof sketch:
Noting that,
where C is nonsingular, we have,
Substitute CT(X-mu) for Y,
where
we obtain,
Incidentally, the Jacobian determinant plays an essential part in multivariate integral by substitution, geometrically suggesting the value of volume element of an n-dimensional stretched cube composed of a new basis into which the old one is linearly or nonlinearly transformed in Euclidean space.
POSTSCRIPT
Happy birthday to my dear bro, Ryan! Hopefully, one day, you will become an extraordinary mathematician like Johann Carl Friedrich Gauss.
MAY 24th, 2015