Eigenvalue Decomposition of Symmetric Matrices（convex）

Eigenvalue Decomposition of Symmetric Matrices

Symmetric matrices are square with elements that mirror each other across the diagonal. They can be used to describe for example graphs with undirected, weighted edges between the nodes; or distance matrices (between say cities), and a host of other applications. Symmetric matrices are also important in optimization, as they are closely related to quadratic functions.

A fundamental theorem, the spectral theorem, shows that we can decompose any symmetric matrix as a three-term product of matrices, involving anorthogonal transformation and a diagonal matrix. The theorem has a direct implication for quadratic functions: it allows a to decompose any quadratic function into a weighted sum of squared linear functions involving vectors that are mutually orthogonal. The weights are called the eigenvalues of the symmetric matrix.

The spectral theorem allows in particular to determine when a given quadratic function is ‘‘bowl-shaped’’, that is,convex. The spectral theorem also allows to find directions of maximal variance within a data set. Such directions are useful to visualize high-dimensional data points in two or three dimensions. This is the basis of a visualization method known as principal component analysis (PCA).

From： https://inst.eecs.berkeley.edu/~ee127a/book/login/l_sym_main.html

Spectral theorem

An important result of linear algebra, called the spectral theorem, or symmetric eigenvalue decomposition (SED) theorem, states that for any symmetric matrix, there are exactly  (possibly not distinct) eigenvalues, and they are all real; further, that the associated eigenvectors can be chosen so as to form an orthonormal basis. The result offers a simple way to decompose the symmetric matrix as a product of simple transformations.

Theorem: Symmetric eigenvalue decomposition

We can decompose any symmetric matrix  with the symmetric eigenvalue decomposition (SED) 
 
where the matrix of  is orthogonal (that is, ), and contains the eigenvectors of , while the diagonal matrix  contains the eigenvalues of .

Here is a proof. The SED provides a decomposition of the matrix in simple terms, namely dyads.

We check that in the SED above, the scalars  are the eigenvalues, and ’s are associated eigenvectors, since 
 

The eigenvalue decomposition of a symmetric matrix can be efficiently computed with standard software, in time that grows proportionately to its dimension  as . Here is the matlab syntax, where the first line ensure that matlab knows that the matrix  is exactly symmetric.

From： https://inst.eecs.berkeley.edu/~ee127a/book/login/l_sym_sed.html

               https://inst.eecs.berkeley.edu/~ee127a/book/login/glossary.html