Chapter 6 (Orthogonality and Least Squares): Orthogonal projections (正交投影)

本文为《Linear algebra and its applications》的读书笔记

Orthogonal projections

• Given a vector $\mathbf{y}$ and a subspace $W$ in ${\mathbb{R}}^n$, there is a vector $\hat{\mathbf{y}}$ in $W$ such that
  • (1) $\hat{\mathbf{y}}$ is the unique vector in $W$ for which $\mathbf{y}-\hat{\mathbf{y}}$ is orthogonal to $W$
  • (2) $\hat{\mathbf{y}}$ is the unique vector in $W$ closest to $\mathbf{y}$
    

---

EXAMPLE 1

Let { u 1 , . . . , u 5 } \{\boldsymbol u_1,...,\boldsymbol u_5\} {u1​,...,u5​} be an orthogonal basis for ${\mathbb{R}}^5$ and let
Consider the subspace W = S p a n { u 1 , u 2 } W = Span \{\boldsymbol u_1,\boldsymbol u_2\} W=Span{u1​,u2​}, and write $\mathbf{y}$ as the sum of a vector ${\mathbf{z}}_1$ in $W$ and a vector ${\mathbf{z}}_2$ in $W^{\perp }$.

SOLUTION

---

• The next theorem shows that the decomposition $\mathbf{y}={\mathbf{z}}_1+{\mathbf{z}}_2$ in Example 1 can be computed without having an orthogonal basis for ${\mathbb{R}}^n$. It is enough to have an orthogonal basis only for $W$ .

• The vector $\hat{\mathbf{y}}$ in (1) is called the orthogonal projection of $\mathbf{y}$ onto $W$ and often is written as $pro{j}_W\mathbf{y}$. When $W$ is a one-dimensional subspace, the formula for $\hat{\mathbf{y}}$ matches the formula given in Section 6.2.
  

PROOF

We may assume that $W$ is not the zero subspace, for otherwise $W^{\perp }={\mathbb{R}}^n$ and (1) is simply $\mathbf{y}=0+\mathbf{y}$. The next section will show that any nonzero subspace of ${\mathbb{R}}^n$ has an orthogonal basis.

• Let { u 1 , . . . , u p } \{\boldsymbol u_1,...,\boldsymbol u_p\} {u1​,...,up​} be any orthogonal basis for $W$ , and define $\hat{\mathbf{y}}$ by (2). Let $\mathbf{z}=\mathbf{y}-\hat{\mathbf{y}}$, then
  Thus $\mathbf{z}$ is orthogonal to ${\mathbf{u}}_1$. Similarly, $\mathbf{z}$ is orthogonal to each ${\mathbf{u}}_j$ in the basis for $W$. Hence $\mathbf{z}$ is orthogonal to every vector in $W$ . That is, $\mathbf{z}$ is in $W^{\perp }$.
• To show that the decomposition in (1) is unique, suppose $\mathbf{y}$ can also be written as $\mathbf{y}={\hat{\mathbf{y}}}_1+{\mathbf{z}}_1$, with ${\hat{\mathbf{y}}}_1$ in $W$ and ${\mathbf{z}}_1$ in $W^{\perp }$. Then $\hat{\mathbf{y}}+\mathbf{z}={\hat{\mathbf{y}}}_1+{\mathbf{z}}_1$, and so
  $$\hat{\mathbf{y}}-{\hat{\mathbf{y}}}_1={\mathbf{z}}_1-\mathbf{z}$$This equality shows that the vector $\mathbf{v}=\hat{\mathbf{y}}-{\hat{\mathbf{y}}}_1$ is in $W$ and in $W^{\perp }$. Hence $\mathbf{v}\cdot \mathbf{v}=0$, which shows that $\mathbf{v}=0$. This proves that $\hat{\mathbf{y}}={\hat{\mathbf{y}}}_1$ and also ${\mathbf{z}}_1=\mathbf{z}$.

---

EXERCISE

Suppose that { u 1 , u 2 } \{\boldsymbol u_1, \boldsymbol u_2\} {u1​,u2​} is an orthogonal set of nonzero vectors in ${\mathbb{R}}^3$. How would you find an orthogonal basis of ${\mathbb{R}}^3$ that contains ${\mathbf{u}}_1$ and ${\mathbf{u}}_2$?

SOLUTION

• First, find a vector $\mathbf{v}$ in ${\mathbb{R}}^3$ that is not in the subspace $W$ spanned by ${\mathbf{u}}_1$ and ${\mathbf{u}}_2$. Let ${\mathbf{u}}_3=\mathbf{v}-pro{j}_W\mathbf{v}$, then { u 1 , u 2 , u 3 } \{\boldsymbol u_1, \boldsymbol u_2, \boldsymbol u_3\} {u1​,u2​,u3​} is an orthogonal basis.

---

EXERCISE 23

Let $A$ be an $m\times n$ matrix. Prove that every vector $\mathbf{x}$ in ${\mathbb{R}}^n$ can be written in the form $\mathbf{x}=\mathbf{p}+\mathbf{u}$, where $\mathbf{p}$ is in $RowA$ and $\mathbf{u}$ is in $NulA$. Also, show that if the equation $A\mathbf{x}=\mathbf{b}$ is consistent, then there is a unique $\mathbf{p}$ in $RowA$ such that $A\mathbf{p}=\mathbf{b}$.

SOLUTION

• By the Orthogonal Decomposition Theorem, each $\mathbf{x}$ in ${\mathbb{R}}^n$ can be written uniquely as $\mathbf{x}=\mathbf{p}+\mathbf{u}$, with $\mathbf{p}$ in $RowA$ and $\mathbf{u}$ in $(RowA{)}^{\perp }=NulA$.
• Next, suppose that $A\mathbf{x}=\mathbf{b}$ is consistent. Let $\mathbf{x}$ be a solution, and write $\mathbf{x}=\mathbf{p}+\mathbf{u}$, as above. Then $A\mathbf{p}=A(\mathbf{x}–\mathbf{u})=A\mathbf{x}–A\mathbf{u}=\mathbf{b}–0=\mathbf{b}$. So the equation $A\mathbf{x}=\mathbf{b}$ has at least one solution $\mathbf{p}$ in $RowA$.
• Finally, suppose that $\mathbf{p}$ and ${\mathbf{p}}_1$ are both in $RowA$ and satisfy $A\mathbf{x}=\mathbf{b}$. Then $\mathbf{p}–{\mathbf{p}}_1$ is in $NulA$ because $$A(\mathbf{p}–{\mathbf{p}}_1)=A\mathbf{p}–A{\mathbf{p}}_1=\mathbf{b}–\mathbf{b}=0$$The equations $\mathbf{p}={\mathbf{p}}_1+(\mathbf{p}–{\mathbf{p}}_1)$ and $\mathbf{p}=\mathbf{p}+0$ both decompose $\mathbf{p}$ as the sum of a vector in $RowA$ and a vector in $(RowA{)}^T$. By the uniqueness of the orthogonal decomposition (Theorem 8), ${\mathbf{p}}_1=\mathbf{p}$, so $\mathbf{p}$ is unique.

A Geometric Interpretation of the Orthogonal Projection

• When $W$ is a one-dimensional subspace, the formula (2) for $pro{j}_W\mathbf{y}$ contains just one term. Thus, when $dimW>1$, each term in (2) is itself an orthogonal projection of $\mathbf{y}$ onto a one-dimensional subspace spanned by one of the $\mathbf{u}$’s in the basis for $W$ . Figure 3 illustrates this when $W$ is a subspace of ${\mathbb{R}}^3$ spanned by ${\mathbf{u}}_1$ and ${\mathbf{u}}_2$.
  

Properties of Orthogonal Projections

• This fact also follows from the next theorem.

---

最佳逼近定理

• The vector $\mathbf{y}$ in Theorem 9 is called the best approximation to $\mathbf{y}$ by elements of $W$($W$ 中元素对 $\mathbf{y}$ 的最佳逼近).
  • Later sections in the text will examine problems where a given $\mathbf{y}$ must be replaced, or approximated, by a vector $\mathbf{v}$ in some fixed subspace $W$ . The distance $\Vert \mathbf{y}-\mathbf{v}\Vert$, can be regarded as the “error” of using $\mathbf{v}$ in place of $\mathbf{y}$. Theorem 9 says that this error is minimized when $\mathbf{v}=\hat{\mathbf{y}}$.
• Inequality (3) leads to a new proof that $\hat{\mathbf{y}}$ does not depend on the particular orthogonal basis used to compute it.

PROOF

• Take $\mathbf{v}$ in $W$ distinct from $\hat{\mathbf{y}}$. See Figure 4. Then $\mathbf{y}-\hat{\mathbf{y}}$ is orthogonal to $\hat{\mathbf{y}}-\mathbf{v}$ (which is in $W$). Since
  the Pythagorean Theorem(勾股定理) gives
  Now $\Vert \hat{\mathbf{y}}-\mathbf{v}\Vert >0$ , and so inequality (3) follows immediately.
  

---

• The final theorem in this section shows how formula (2) for $pro{j}_W\mathbf{y}$ is simplified when the basis for $W$ is an orthonormal set.

• Suppose $U$ is an $n\times p$ matrix with orthonormal columns, and let $W$ be the column space of $U$. Then
  

---

EXAMPLE

Let $W$ be a subspace of ${\mathbb{R}}^n$. Let $\mathbf{x}$ and $\mathbf{y}$ be vectors in ${\mathbb{R}}^n$ and let $\mathbf{z}=\mathbf{x}+\mathbf{y}$. If $\mathbf{u}$ is the projection of $\mathbf{x}$ onto $W$ and $\mathbf{v}$ is the projection of $\mathbf{y}$ onto $W$ , show that $\mathbf{u}+\mathbf{v}$ is the projection of $\mathbf{z}$ onto $W$ .

SOLUTION

• Let $U$ be a matrix whose columns consist of an orthonormal basis for $W$ . Then
  $$\begin{array}{rl}pro{j}_W\mathbf{z}& =U{U}^T\mathbf{z}\\ & =U{U}^T(\mathbf{x}+\mathbf{y})\\ & =U{U}^T\mathbf{x}+U{U}^T\mathbf{y}\\ & =pro{j}_W\mathbf{x}+pro{j}_W\mathbf{y}\\ & =\mathbf{u}+\mathbf{v}\end{array}$$