Introduction to Linear Algebra(7) Symmetric Matrices and Quadratic Forms

@[TOC](Introduction to Linear Algebra(7) Symmetric Matrices and Quadratic Forms)

Diagonolization Of Sysmmetric Matrices

If $A$ is symmetric, then any tow eigenvectors from different eigenspaces are orthogonal.
PROOF: $$\lambda v_1\cdot v_2=(\lambda v_1{)}^T{v}_2=(A{v}_1{)}^T{v}_2=(v_1^T{A}^T)v_2=v_1^T(A{v}_2)={\lambda }_2{v}_1\cdot v_2$$ for${\lambda }_1̸={\lambda }_2$, $v_1\cdot v_2=0$
An $n\times n$ matrix $A$ is orthogonally diagonalizable if and only if $A$ is a symmetric matrix.
The Spectral Theorem
An $n\times n$ sysmmetric matrix $A$ has the following properties:
a.A has $n$ real eigenvalues, counting multiplicities.
b. The demension of the eigenspace for each eigenvalues $lambda$ equals the multiplicity of $\lambda$ as a root of the characterustic equation.
c. The eigenspaces are mutually orthogonal, in the sense that eigenvectors corresponding to different eigenvalues are orthogonal.
d. $A$ is orthogonally diagonalizable.
Spectral Decomposition
if $A$ is an sysmmetric matrix. then $A$ could be written as $$A={\lambda }_1{u}_1{u}_1^T+{\lambda }_2{u}_2{u}_2^T+\cdots +{\lambda }_n{u}_n{u}_n^T$$

Quadratic Forms

The Principla Axes Theorem
Let $A$ be an $n\times n$ symmetric matrix. Then there is an orthogonal change of variable, $x=Py$, that transforms the quadratic form $x^{T}Ax$ into a quadratic form $y^{T}Dy$ with no cross-product term.

Classifying Quadratic Forms

A quadratic form $Q$ is:
a.positive definite if $Q(x)>0$ for all $X̸=0$,
b. negative definite if $Q(x)<0$ for all $x̸=0$,
c. indefinite if Q(x) assumes both positive and negative values.
Quadratic Forms and Eigenvalues
Let $A$ be an $n\times n$ symmetric matrix. Then a quadratic form $x^{T}Ax$ is:
a. positive definite if and only if the eigenvalues of $A$ are all positive,
b. negative definite if and only if the eigenvalues of $A$ are all negative,
c.indefinite if and only if $A$ has both positive and negative eigenvalues.
Let $A$ be a symmetric matrix, and define $m$ and $M$ . Then $M$ is the greatest eigenvalue ${\lambda }_1$ of $A$ amd $m$ is the least eigenvalue of $A$. Then value of $x^{T}Ax$ is $M$ when $x$ is a unit eigenvector $u_1$ corresponding to $M$. Then value of $x^{T}Ax$ is $m$ when $x$ is a unit eigenvector corresponding to $m$.
Let $A,{\lambda }_1$, and $u_1$ be as in Theorem 6. Then the maximum value of $x^{T}Ax$ subject to the constraints $$x^{T}x=1,x^T{u}_1=0$$
is the second greatset eigenvalue, ${\lambda }_2$, and this maximum is attained when $x$ is an eigenvector $u_2$ corresponding to ${\lambda }_2$. This theorem is also extended to ${\lambda }_k$

The Singular Value Decomposition

Singular Value Decomposition: a factorization $A=QD{P}^{-1}$ is possible for any $m\times n$ matrix $A$
Decomposition
Let $A$ be an $m\times n$ matrix with rank $r$. Then there exists an $m\times n$ matrix $\Sigma$ as $\Sigma =\left[\begin{array}{cc}D& 0\\ 0& 0\end{array}\right]$ for which the diagonal entries in $D$ are the first $r$ singular values of $A$,${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_r>0$, and there exist an $m\times m$ orthogonal matrix $U$ and an $n\times n$ orthogonal matrix $V$ such that $$A=U\Sigma V^T$$
PROOF：
KaTeX parse error: Expected 'EOF', got '\lambd' at position 1: \̲l̲a̲m̲b̲d̲_i and $v_i$ are the eigenvalues and eigenvectors of $A^{T}A$ seperately, so that { A v 1 , ⋯ &ThinSpace; , A v r } \{Av_1,\cdots,Av_r\} {Av1​,⋯,Avr​} is an orthogonal basis for $ColA$. Normalize each $A{v}_i$ to obtain an orthonormal basis { u 1 , ⋯ &ThinSpace; , u r } \{u_1,\cdots,u_r\} {u1​,⋯,ur​}, where $$u_i=\frac{A{v}_i}{||A{v}_i||}=\frac{A{v}_i}{{\sigma }_i}$$ and $$A{v}_i=\sigma u_i,(\le i\le r)$$
Now extend { u i , ⋯ &ThinSpace; , u r } \{u_i,\cdots,u_r\} {ui​,⋯,ur​} to an orthonormal basis { u 1 , ⋯ &ThinSpace; , u m } \{u_1,\cdots,u_m\} {u1​,⋯,um​} of $R^m$, and let$$U=[u_1{u}_2\cdots u_m]andV=[v_1{v}_2\cdots v_n]$$
By construction, $U$ and $v$ are orthogonal matrices. $$AV=[A{v}_1\cdots A{v}_{r}0\cdots 0]=[\sigma u_1\cdots \sigma u_{r}0\cdots 0]$$
Then$U\Sigma =AV$ Thus KaTeX parse error: Expected 'EOF', got '\SigmaV' at position 19: …\Sigma V^{-1}=U\̲S̲i̲g̲m̲a̲V̲^{T}
The Invertible Matrix Theorem (concluded)
Let $A$ be an $n\times n$ matrix. Then the following statements are each equivalent to the statement that $A$ is an invertible matrix.
u.$(ColA{)}^{\perp }=0.$
v.$(NulA{)}^{\perp }=R^n.$
w. $RowA=R^n.$
x. $A$ has $n$ nonzero sigular values.
Reduced $SVD$ and the Pseudoinverse of $A$
let $r$ = rank$A$ then the $U$ and $V$ could be KaTeX parse error: Expected & or \\ or \cr or \end at end of input: …-r}^T=U_rDV_r^T
This factorization of $A$ is called a reduced singular value decomposition of $A$. The following matrix is called the pseudoinverse of $A$: $$A^{+}=V_r{D}^{-1}{U}_r^T$$

Principal Component Analysis

For simplicity, assume that the matrix $[X_1\cdots X_N]$ is already in mean-deviation form. The goal of principal component analysis is to find an orthogonal $p\times p$ matrix $P=[u_1\cdots u_p]$ that determines a changeof variable, $X=PY$,or $$\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right]=\left[\begin{array}{cccc}{u}_1& u_2& \cdots & u_p\end{array}\right]\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_p\end{array}\right]$$