Chapter 6 (Orthogonality and Least Squares): Inner product, length,and orthogonality (内积、长度和正交性)

本文为《Linear algebra and its applications》的读书笔记

The Inner Product

• The number ${\mathbf{u}}^T\mathbf{v}$ is called the inner product of $\mathbf{u}$ and $\mathbf{v}$, and often it is written as $\mathbf{u}\cdot \mathbf{v}$. This inner product is also referred to as a dot product (点积).
  

---

• Properties ($b$) and ($c$) can be combined several times to produce the following useful rule:
  

The Length of a Vector

• For any scalar $c$, the length of $c\mathbf{v}$ is $|c|$ times the length of $\mathbf{v}$. That is,
  
• A vector whose length is $1$ is called a unit vector. If we divide a nonzero vector $\mathbf{v}$ by its length—that is, multiply by $1/\Vert \mathbf{v}\Vert$—we obtain a unit vector $\mathbf{u}$. This process is sometimes called normalizing (单位化) $\mathbf{v}$.

Distance in ${\mathbb{R}}^n$

• Recall that if $a$ and $b$ are real numbers, the distance on the number line between $a$ and $b$ is the number $|a-b|$. This definition of distance in $\mathbb{R}$ has a direct analogue in ${\mathbb{R}}^n$.

---

• In ${\mathbb{R}}^2$ and ${\mathbb{R}}^3$, this definition of distance coincides with the usual formulas for the Euclidean distance between two points.

Orthogonal Vectors

正交向量

• The rest of this chapter depends on the fact that the concept of perpendicular lines in ordinary Euclidean geometry has an analogue in ${\mathbb{R}}^n$.
• Consider ${\mathbb{R}}^2$ or ${\mathbb{R}}^3$ and two lines through the origin determined by vectors $\mathbf{u}$ and $\mathbf{v}$. The two lines shown in Figure 5 are geometrically perpendicular if and only if the distance from $\mathbf{u}$ to $\mathbf{v}$ is the same as the distance from $\mathbf{u}$ to $-\mathbf{v}$.
  The two squared distances are equal if and only if $\mathbf{u}\cdot \mathbf{v}=0$.

---

• The following definition generalizes to ${\mathbb{R}}^n$ this notion of perpendicularity (or $\mathbf{o}\mathbf{r}\mathbf{t}\mathbf{h}\mathbf{o}\mathbf{g}\mathbf{o}\mathbf{n}\mathbf{a}\mathbf{l}\mathbf{i}\mathbf{t}\mathbf{y}$).
  

Observe that the zero vector is orthogonal to every vector in ${\mathbb{R}}^n$.

---

毕达哥拉斯定理 / 勾股定理

Orthogonal Complements

正交补

• If a vector $\mathbf{z}$ is orthogonal to every vector in a subspace $\mathbb{W}$ of ${\mathbb{R}}^n$, then $\mathbf{z}$ is said to be orthogonal to $\mathbb{W}$ . The set of all vectors $\mathbf{z}$ that are orthogonal to $\mathbb{W}$ is called the orthogonal complement of $\mathbb{W}$ and is denoted by ${\mathbb{W}}^{\perp }$ (and read as “$\mathbb{W}$ perpendicular” or simply “$\mathbb{W}$ perp”).

$(RowA{)}^{\perp }=NulA;(ColA{)}^{\perp }=Nul{A}^T$

In fact, Theorem 3 is sometimes called the Fundamental Theorem of Linear Algebra.

PROOF

• The row–column rule for computing $A\mathbf{x}$ shows that if $\mathbf{x}$ is in $NulA$, then $\mathbf{x}$ is orthogonal to each row of $A$. Since the rows of $A$ span the row space, $\mathbf{x}$ is orthogonal to $RowA$. Conversely, if $\mathbf{x}$ is orthogonal to $RowA$, then $A\mathbf{x}=0$.
• This proves the first statement of the theorem. Since this statement is true for any matrix, it is true for $A^T$ . That is, the orthogonal complement of the row space of $A^T$ is the null space of $A^T$ . This proves the second statement, because $Row{A}^T=ColA$.
  

$dimW+dim{W}^{\perp }=n$

EXAMPLE

Let $W$ be a subspace of ${\mathbb{R}}^n$. Thus $W^{\perp }$ is also a subspace of ${\mathbb{R}}^n$. Prove that $dimW+dim{W}^{\perp }=n$.

SOLUTION

• If W ≠ { 0 } W \neq \{\boldsymbol 0\} W​={0}, let { b 1 , . . . , b p } \{\boldsymbol b_1,..., \boldsymbol b_p\} {b1​,...,bp​} be a basis for $W$. Let $A$ be the $p\times n$ matrix having rows ${\mathbf{b}}_1^T,...,{\mathbf{b}}_p^T$. It follows that $W$ is the row space of $A$. Theorem 3 implies that $W^{\perp }=(RowA{)}^{\perp }=NulA$ and hence $dim{W}^{\perp }=dimNulA$. Thus, $dimW+dim{W}^{\perp }=dimRowA+dimNulA=n$.
• If W = { 0 } W =\{\boldsymbol 0\} W={0}, then $W^{\perp }={\mathbb{R}}^n$, and the result follows.

---

EXERCISES

Explain why an equation $A\mathbf{x}=\mathbf{b}$ has a solution if and only if $\mathbf{b}$ is orthogonal to all solutions of the equation $A^T\mathbf{x}=0$.

SOLUTION

• If $A\mathbf{x}=\mathbf{b}$ has a solution, then ${\mathbf{x}}^T{A}^T={\mathbf{b}}^T.$ Let ${\mathbf{x}}^{\prime }$ be any solution of $A^T\mathbf{x}=0$, then ${\mathbf{x}}^T{A}^T{\mathbf{x}}^{\prime }={\mathbf{b}}^T{\mathbf{x}}^{\prime }=0,\mathbf{b}\cdot {\mathbf{x}}^{\prime }=0$. Thus $\mathbf{b}$ is orthogonal to all solutions of the equation $A^T\mathbf{x}=0$.
• If $\mathbf{b}$ is orthogonal to all solutions of the equation $A^T\mathbf{x}=0$, then $\mathbf{b}\in (Nul{A}^T{)}^{\perp }=ColA$. Thus $A\mathbf{x}=\mathbf{b}$ has a solution.

Angles in ${\mathbb{R}}^2$ and ${\mathbb{R}}^3$

• If $\mathbf{u}$ and $\mathbf{v}$ are nonzero vectors in either ${\mathbb{R}}^2$ or ${\mathbb{R}}^3$, then there is a nice connection between their inner product and the angle $\theta$ between the two line segments from the origin to the points identified with $\mathbf{u}$ and $\mathbf{v}$. The formula is
  
• To verify this formula for vectors in ${\mathbb{R}}^2$, consider the triangle shown in Figure 9.
  
  By the law of cosines (余弦定理),
  which can be rearranged to produce
  The verification for ${\mathbb{R}}^3$ is similar. When $n>3$, this formula may be used to define the angle between two vectors in ${\mathbb{R}}^n$.
• In statistics, for instance, the value of $cos\theta$ for suitable vectors $\mathbf{u}$ and $\mathbf{v}$ is what statisticians call a correlation coefficient (相关系数).