Introduction to Linear Algebra(6) Orthogonal and Least Squares

Inner Product,Length, and Orthiginality

Two vectors $u$ and $v$ in $R^n$ are orthogonal if $u\cdot v=0$.
Two vectors $u$ and $v$ are orthogonal if and only if $||u+v|{|}^2=||u|{|}^2+||v|{|}^2$.
Let A be an $m\times n$ matrix. The orthogonal complement of the row space of $A$ is the null space of $A$, and teh orthogonal somplement of the column space of $A$ is the null space of $A^{\perp }$:$$(RowA{)}^{\perp }=NulA,(ColA{)}^{\perp }=Nul{A}^{\perp }$$

Orthogonal Sets

A set of vectors KaTeX parse error: Can't use function '\u' in math mode at position 14: \{u_1,\cdots,\̲u̲_p\} in $R^n$ is said to be an orthogonal set if each pair of distinct vectors from theset is orthogonal, that is, if KaTeX parse error: Expected 'EOF', got '\cdotu' at position 4: u_i\̲c̲d̲o̲t̲u̲_j=0 whenever $i̸=j$.
Let { u 1 , ⋯   , u p } \{u_1,\cdots,u_p\} {u1​,⋯,up​} be an orthogonal baisis for a subspace $W$ of $R^n$. For each $y$ in $W$, the weights in the lienar combination$$y=c_1{u}_1+\cdots +c_p{u}_p$$
are given by $$c_j=\frac{y\cdot u_j}{u_j\cdot u_j}(j=1,\cdots ,P)$$

An Orthogonal Projection

KaTeX parse error: Expected '}', got '\cdotu' at position 24: …proj_Ly=\frac{y\̲c̲d̲o̲t̲u̲}{u\cdotu}u

Orthonormal Sets

A set { u 1 , ⋯   , u p } \{u_1,\cdots,u_p\} {u1​,⋯,up​} is an orthonormal set if it is an oorthogonal set of
An $m\times n$ matrix $U$ has orthonormal columns if and only if $U^{T}U=I$.
Let $U$ be an $m\times n$ matrix with orthonormal columns, and let $x$ and $y$ be in $R^n$. Then
a.$||Ux||=||x||$
b.$(Ux)\cdot (uy)=x\cdot y$
c.$(Ux)\cdot (uy)=0$ if and only if $x\cdot y=0$
The best approsimation Theorem
Let $W$ be a subspace of $R^n$, let $y$ be any vector in $R^n$, and let $\hat{y}$ be the orthegonal projection of $y$ onto $W$. Then $\hat{y}$ is the closet point in $W$ to $y$, in the sense that $$||y-\hat{y}||<||y-v||$$
for all $v$ in $W$ distinct from $\hat{y}$.
If { u 1 , ⋯ &ThinSpace; , u p } \{u_1,\cdots,u_p\} {u1​,⋯,up​} is an orthonormal basis for a subspace $W$ of $R^n$, then$$pro{j}_{w}y=(y\cdot u_1)u_1+(y\cdot u_2)u_2+\cdots +(y\cdot u_p)u_p$$
If $U=[u_1{u}_2\cdots u_p],then$ $$pro{j}_{w}y=U{U}^{T}y$$ for all $y$ in $R^n$

The GRAM-SCHMIDT PROCESS

Given a basis $x_1,\cdots ,x_p$ for a nonzero subspace $W$ of $R^n$, define$$v_1=x_1$$ $$x_2=x_2-\frac{x_2\cdot v_1}{v_1\cdot v_1}{v}_1$$ $$v_3=x_3-\frac{x_3\cdot v_1}{v_1\cdot v_1}{v}_1-\frac{x_3\cdot v_2}{v_2\cdot v_2}{v}_2$$ $$⋮$$ $$v_p=x_p-\frac{x_p\cdot v_1}{v_1\cdot v_1}{v}_1\cdots -\frac{x_p\cdot v_{p-1}}{v_{p-1}\cdot v_{p-1}}{v}_{p-1}$$
Then $v_1,\cdots ,v_p$ is an orthogonal basis for $W$. In addition S p a n { v 1 , ⋯ &ThinSpace; , v k } = S p a n { x 1 , ⋯ &ThinSpace; , x k } f o r 1 ≤ k ≤ p Span\{v_1,\cdots,v_k\}=Span\{x_1,\cdots, x_k\} for 1 \le k \le p Span{v1​,⋯,vk​}=Span{x1​,⋯,xk​}for1≤k≤p

QR Factorization of Matrices

If A is an $m\times n$ matrix with linearly independent columns, then $A$ cam ne factored as $A=QR$, where $Q$ is an $m\times n$ matrix whose columns form an orthonormal basis for $ColA$ and $R$ is an $n\times n$ upper triangular invertible matrix with positive entries on its diagonal.

Least-Squares Problems

If $A$ is $m\times n$ and $b$ is in $R^m$, a least-squares solution of $Ax=b$ ia an $\hat{x}$ in $R^n$ such that:$$||b-A\hat{x}||\le ||b-Ax||$$
for all $x$ in $R^n$.
Note that
$A\hat{x}$ is in the $ColA$
Let $A$ be an $m\times n$ matrix. THe following statements are logically equivalent:
a. The equation $Ax=b$ has a unique least-squares solution for each $b$ in $R^m$.
b. The columns of $A$ are linearly independent.
c. The matrix $A^{T}A$ is invertible.
When these statments are true, the least-squares solution $\hat{x}$ is given by $$\hat{x}=(A^{T}A{)}^{-1}{A}^{T}b$$
The set of least-squares solutions of $Ax=b$ coincides with the nonempty set of solutions of the normal equations $A^{T}A=x=A^{T}b$.
Alternative Calculations of Least-Squares Solutions
given an $m\times n$ matrix A with linearly independent columns, let $A=QR$ be a QR factorization of $A$. Then for each b in $R^m$, the equation $Ax=b$ has a unique least-squares solution.given by $$\hat{x}=R^{-1}{Q}^{T}b$$

Least-Squares Lines

For a system: predicted y-value: ${\beta }_0+{\beta }_1{x}_1$ Observed y-value:$y_1$
We can write this system as $$X\beta =$$, where $X=$, $\beta =[{\beta }_0,{\beta }_1]$

Inner Product spaces

An inner product on a vector space $V$ is a function that, to each pair of vectors $u$ and $v$ in $V$, associates a real number <u,v> and satisfies the following axioms, for all $u,v$, and $w$ in $V$ and all scalars $c$:
1.$<u,v>=<v,u>$
2.$<u+v,w>=<u,w>+<v,w>$
3.$<cu,v>=c<u,v>$
4.$<u,u>\ge 0$ and $<u,u>=0$ if and only if $u=0$
A vector space with an inner product is called an inner product space.

Trend Analysis of Data

$\hat{g}=c_0{p}_0+c_1{p}_1+c_2{p}_2+c_3{p}_3$ and $\hat{g}$ is called cubic trend function, and $c_0,\cdots ,c_3$ are the trend coefficients of the data. Thus could use Gram-schmidt process to construct the coefficients $c_0,\cdots ,c_3$

Fourier Series

Any function could be approsimated as closely as desired by a function of the form$$F=a_0/2+a_{1}cost+\cdots +a_{n}cosnt+b_{1}sint+\cdots +b_{n}sinnt$$
the set { 1 , c o s t , c o s 2 t , ⋯ &ThinSpace; , c o s n t , s i n t , s i n 2 t , ⋯ &ThinSpace; , s i n n t } \{1,cost,cos2t,\cdots,cosnt,sint,sin2t,\cdots,sinnt\} {1,cost,cos2t,⋯,cosnt,sint,sin2t,⋯,sinnt} si orthogonal with respect to the inner product$$<f,g>={\int }_0^{2\pi }f(t)g(t)dt$$ Thus $$a_k=\frac{<f,coskt>}{<coskt,coskt>},b_k=\frac{<f,sinkt>}{<sinkt,sinkt>},k\ge 1.$$