Chapter 6 (Orthogonality and Least Squares): The Gram–Schmidt process (格拉姆-施密特算法), QR Factorization

本文为《Linear algebra and its applications》的读书笔记

The Gram–Schmidt process

• The Gram–Schmidt process is a simple algorithm for producing an orthogonal or orthonormal basis for any nonzero subspace of ${\mathbb{R}}^n$.

---

EXAMPLE 1

Let W = S p a n { x 1 , x 2 } W = Span\{\boldsymbol x_1, \boldsymbol x_2\} W=Span{x1​,x2​}, where ${\mathbf{x}}_1=\left[\begin{array}{c}3\\ 6\\ 0\end{array}\right]$ and ${\mathbf{x}}_2=\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]$. Construct an orthogonal basis { v 1 , v 2 } \{\boldsymbol v_1, \boldsymbol v_2\} {v1​,v2​} for $W$.

SOLUTION

• The component of ${\mathbf{x}}_2$ orthogonal to ${\mathbf{x}}_1$ is ${\mathbf{x}}_2-\mathbf{p}$, which is in $W$. Let ${\mathbf{v}}_1={\mathbf{x}}_1$ and
  

---

• The next example fully illustrates the Gram–Schmidt process. Study it carefully.

EXAMPLE 2

Let ${\mathbf{x}}_1=\left[\begin{array}{c}1\\ 1\\ 1\\ 1\end{array}\right]$, ${\mathbf{x}}_2=\left[\begin{array}{c}0\\ 1\\ 1\\ 1\end{array}\right]$, and ${\mathbf{x}}_3=\left[\begin{array}{c}0\\ 0\\ 1\\ 1\end{array}\right]$. Then { x 1 , x 2 , x 3 } \{\boldsymbol x_1, \boldsymbol x_2,\boldsymbol x_3\} {x1​,x2​,x3​} is linearly independent and thus is a basis for a subspace $W$ of ${\mathbb{R}}^4$. Construct an orthogonal basis for $W$.

SOLUTION

• $Step1.$ Let ${\mathbf{v}}_1={\mathbf{x}}_1$ and W 1 = S p a n { x 1 } = S p a n { v 1 } W_1 = Span\{\boldsymbol x_1\}= Span\{\boldsymbol v_1\} W1​=Span{x1​}=Span{v1​}.
• $Step2.$ Let ${\mathbf{v}}_2$ be the vector produced by subtracting from ${\mathbf{x}}_2$ its projection onto the subspace $W_1$. That is, let
  Then { v 1 , v 2 } \{\boldsymbol v_1,\boldsymbol v_2\} {v1​,v2​} is an orthogonal basis for the subspace $W_2$ spanned by ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$.
• $Step{2}^{\prime }(optional).$ If appropriate, scale ${\mathbf{v}}_2$ to simplify later computations:
  
• $Step3.$ Let ${\mathbf{v}}_3$ be the vector produced by subtracting from ${\mathbf{x}}_3$ its projection onto the subspace $W_2$. Use the orthogonal basis { v 1 , v 2 ′ } \{\boldsymbol v_1,\boldsymbol v_2'\} {v1​,v2′​} to compute this projection onto $W_2$:
  

---

• The proof of the next theorem shows that this strategy really works. Scaling of vectors is not mentioned because that is used only to simplify hand calculations.

• Theorem 11 shows that any nonzero subspace $W$ of ${\mathbb{R}}^n$ has an orthogonal basis.

PROOF

• For $1\le k\le p$, let W k = S p a n { x 1 , . . . , x k } W_k = Span \{\boldsymbol x_1,..., \boldsymbol x_k\} Wk​=Span{x1​,...,xk​}.
• Set ${\mathbf{v}}_1={\mathbf{x}}_1$, so that S p a n { v 1 } = S p a n { x 1 } Span \{v_1\}= Span \{x_1\} Span{v1​}=Span{x1​}.
• Suppose, for some $k<p$, we have constructed ${\mathbf{v}}_1,...,{\mathbf{v}}_k$ so that { v 1 , . . . , v k } \{\boldsymbol v_1,..., \boldsymbol v_k\} {v1​,...,vk​} an orthogonal basis for $W_k$. Define
  $${\mathbf{v}}_{k+1}={\mathbf{x}}_{k+1}-pro{j}_{W_k}{\mathbf{x}}_{k+1}(2)$$By the Orthogonal Decomposition Theorem, ${\mathbf{v}}_{k+1}$ is orthogonal to $W_k$.
  • Note that $pro{j}_{W_k}{\mathbf{x}}_{k+1}$ is in $W_k$ and hence also in $W_{k+1}$. Since ${\mathbf{x}}_{k+1}$ is in $W_{k+1}$, so is ${\mathbf{v}}_{k+1}$.
  • Furthermore, ${\mathbf{v}}_{k+1}\ne 0$ because ${\mathbf{x}}_{k+1}$ is not in $W_k$.
• Hence { v 1 , . . . , v k + 1 } \{\boldsymbol v_1,..., \boldsymbol v_{k+1}\} {v1​,...,vk+1​} is an orthogonal set of nonzero vectors in $W_{k+1}$. This set is an orthogonal basis for $W_{k+1}$. Hence W k + 1 = S p a n { v 1 , . . . , v k + 1 } W_{k+1}= Span \{\boldsymbol v_1,..., \boldsymbol v_{k+1}\} Wk+1​=Span{v1​,...,vk+1​}. When $k+1=p$, the process stops.

Orthonormal Bases

• An orthonormal basis is constructed easily from an orthogonal basis { v 1 , . . . , v p } \{\boldsymbol v_1,..., \boldsymbol v_p\} {v1​,...,vp​}: simply normalize (i.e., “scale”) all the ${\mathbf{v}}_k$.

Tip: When working problems by hand, this is easier than normalizing each ${\mathbf{v}}_k$ as soon as it is found (because it avoids unnecessary writing of square roots).

QR Factorization of Matrices

• If an $m\times n$ matrix $A$ has linearly independent columns { x 1 , . . . , x n } \{\boldsymbol x_1,..., \boldsymbol x_{n}\} {x1​,...,xn​}, then applying the Gram–Schmidt process (with normalizations) to { x 1 , . . . , x n } \{\boldsymbol x_1,..., \boldsymbol x_{n}\} {x1​,...,xn​} amounts to $factoring$ $A$.
  • This factorization is widely used in computer algorithms for various computations, such as solving equations (discussed in Section 6.5) and finding eigenvalues (mentioned in the exercises for Section 5.2).

---

PROOF

• The columns of $A$ form a basis { x 1 , . . . , x n } \{\boldsymbol x_1,..., \boldsymbol x_{n}\} {x1​,...,xn​} for $ColA$. Construct an orthonormal basis { u 1 , . . . , u n } \{\boldsymbol u_1,..., \boldsymbol u_{n}\} {u1​,...,un​} for $W=ColA$. This basis may be constructed by the Gram–Schmidt process or some other means. Let
  For $k=1,...,n$, ${\mathbf{x}}_k$ is in S p a n { x 1 , . . . , x k } = S p a n { u 1 , . . . , u k } Span \{\boldsymbol x_1,...,\boldsymbol x_k\}= Span\{\boldsymbol u_1,...,\boldsymbol u_k\} Span{x1​,...,xk​}=Span{u1​,...,uk​}. So there are constants, $r_{1k},...,r_{kk}$, such that
  We may assume that $r_{kk}\ge 0$. (If $r_{kk}<0$, multiply both $r_{kk}$ and ${\mathbf{u}}_k$ by $-1$.) This shows that ${\mathbf{x}}_k$ is a linear combination of the columns of $Q$ using as weights the entries in the vector
  That is, ${\mathbf{x}}_k=Q{\mathbf{r}}_k$ for $k=1,...,n$. Let $R=[{\mathbf{r}}_1...{\mathbf{r}}_n]$. Then
  
• The fact that $R$ is invertible follows easily from the fact that the columns of $A$ are linearly independent. Since $R$ is clearly upper triangular, its nonnegative diagonal entries must be positive.

---

EXERCISES

Suppose $A=QR$, where $Q$ is $m\times n$ and $R$ is $n\times n$. Show that if the columns of $A$ are linearly independent, then $R$ must be invertible.

SOLUTION

• [Hint: Study the equation $R\mathbf{x}=0$ and use the fact that $A=QR$.]

---

• $Q$ can be constructed by Gram–Schmidt process and $R$ can be calculated by $R=Q^{T}A$

${}^1$ See $Fundamentals$ $of$ $Matrix$ $Computations$, by $David$ $S.$ $Watkins$ (New York: John Wiley & Sons, 1991), pp. 167–180.

---

EXERCISES 23

Suppose $A=QR$ is a $QR$ factorization of an $m\times n$ matrix $A$ (with linearly independent columns). Partition $A$ as $[A_1{A}_2]$, where $A_1$ has $p$ columns. Show how to obtain a QR factorization of $A_1$ .

SOLUTION
$$\begin{gathered} \because A=QR \\ \therefore \left[\begin{array}{cc}{A}_{1_{m\times p}}& A_{2_{m\times (n-p)}}\end{array}\right]=\left[\begin{array}{cc}{Q}_{1_{m\times p}}& Q_{2_{m\times (n-p)}}\end{array}\right]\left[\begin{array}{cc}{R}_{11_{p\times p}}& R_{12_{p\times (n-p)}}\\ 0_{(n-p)\times p}& R_{22_{(n-p)\times (n-p)}}\end{array}\right] \\ \therefore A_{1_{m\times p}}=Q_{1_{m\times p}}{R}_{11_{p\times p}} \end{gathered}$$