Chapter 6 (Orthogonality and Least Squares): Inner product spaces (内积空间)

本文为《Linear algebra and its applications》的读书笔记

Inner product spaces

• Notions of length, distance, and orthogonality are often important in applications involving a vector space. For ${\mathbb{R}}^n$, these concepts were based on the properties of the inner product. For other spaces, we need analogues of the inner product with the same properties.

内积空间的定义

$<\mathbf{v},0>=<0,\mathbf{v}>=0$
PROOF
$<\mathbf{v},0>=<\mathbf{v},00>=0<\mathbf{v},0>=0$

常见的内积空间

加权最小二乘

EXAMPLE 1

• Let $\mathbf{u}=(u_1,u_2)$ and $\mathbf{v}=(v_1,v_2)$, $\mathbf{u},\mathbf{v}$ are in ${\mathbb{R}}^n$. It can be shown that $$<\mathbf{u},\mathbf{v}>=4u_1{v}_1+5u_2{v}_2(1)$$defines an inner product.
  • Inner products similar to (1) can be defined on ${\mathbb{R}}^n$. They arise naturally in connection with “weighted least-squares” problems, in which weights are assigned to the various entries in the sum for the inner product in such a way that more importance is given to the more reliable measurements.

多项式的内积空间

From now on, when an inner product space involves polynomials or other functions, we will write the functions in the familiar way, rather than use the boldface type for vectors. Nevertheless, it is important to remember that each function is a vector when it is treated as an element of a vector space.

EXAMPLE 2

• Let $t_0,...,t_n$ be distinct real numbers. For $p$ and $q$ in ${\mathbb{P}}^n$ (a vector space), define
  Inner product Axioms 1–3 are readily checked.
• For Axiom 4, note that
  Also, $<0,0>=0$. (The boldface zero here denotes the zero polynomial, the zero vector in ${\mathbb{P}}^n$.) If $<p,p>=0$, then $p$ must vanish at $n+1$ points: $t_0,...,t_n$. This is possible only if $p$ is the zero polynomial, because the degree of $p$ is less than $n+1$. Thus (2) defines an inner product on ${\mathbb{P}}^n$.

这个例子很重要，下面的几个例子都是围绕多项式的内积空间展开的

An Inner Product for $C[a,b]$

• Probably the most widely used inner product space for applications is the vector space $C[a,b]$ of all continuous functions on an interval $a\le t\le b$, with an inner product that we will describe.

---

• We begin by considering a polynomial $p$ and any integer $n$ larger than or equal to the degree of $p$. Then $p$ is in ${\mathbb{P}}^n$, and we may compute a “length” for $p$ using the inner product of Example 2 involving evaluation at $n+1$ points in $[a,b]$. However, this length of $p$ captures the behavior at only those $n+1$ points.
• We could use a much larger $n$, with many more points for the “evaluation” inner product. See Figure 4.
  
• Let us partition $[a,b]$ into $n+1$ subintervals of length $\Delta t=(b-a)/(n+1)$, and let $t_0,...,t_n$ be arbitrary points in these subintervals.
  If $n$ is large, the inner product on ${\mathbb{P}}^n$ determined by $t_0,...,t_n$ will tend to give a large value to $<p,p>$, so we scale it down and divide by $n+1$. Observe that $1/(n+1)=\Delta t/(b-a)$, and define
  
• Now, let $n$ increase without bound. Since polynomials $p$ and $q$ are continuous functions, the expression in brackets is a Riemann sum (黎曼和) that approaches a definite integral, and we are led to consider the $average$ $value$ of $p(t)q(t)$ on the interval $[a,b]$:
  This quantity is defined for polynomials of any degree (in fact, for all continuous functions), and it has all the properties of an inner product. The scale factor $1/(b-a)$ is inessential and is often omitted for simplicity.

---

• The inner product discussed above provides a more sophisticated approach to the least-squares curve fitting.

Lengths, Distances, and Orthogonality

• Let $V$ be an inner product space, with the inner product denoted by $<\mathbf{u},\mathbf{v}>$. Just as in ${\mathbb{R}}^n$, we define the length, or norm (范数), of a vector $\mathbf{v}$ to be the scalar
  $$\Vert \mathbf{v}\Vert =\sqrt{<\mathbf{v},\mathbf{v}>}$$
• A unit vector is one whose length is 1.
• The distance between $\mathbf{u}$ and $\mathbf{v}$ is $\Vert \mathbf{u}-\mathbf{v}\Vert$.
• Vectors $\mathbf{u}$ and $\mathbf{v}$ are orthogonal if $<\mathbf{u},\mathbf{v}>=0$.

The Gram–Schmidt Process

• The existence of orthogonal bases for finite-dimensional subspaces of an inner product space can be established by the Gram–Schmidt process, just as in ${\mathbb{R}}^n$.
  • The orthogonal projection of a vector onto a subspace $W$ with an orthogonal basis can be constructed as usual. The projection does not depend on the choice of orthogonal basis, and it has the properties described in the Orthogonal Decomposition Theorem and the Best Approximation Theorem.

---

EXAMPLE 5

Let $V$ be ${\mathbb{P}}^4$ with the inner product in Example 2, involving evaluation of polynomials at $-2,-1,0,1$, and $2$, and view ${\mathbb{P}}^2$ as a subspace of $V$. Produce an orthogonal basis for ${\mathbb{P}}^2$ by applying the Gram–Schmidt process to the polynomials $1,t,$ and $t^2$.

SOLUTION

• The inner product depends only on the values of a polynomial at $-2,-1,0,1,2$. so we list the values of each polynomial as a vector in ${\mathbb{R}}^5$, underneath the name of the polynomial:
  
• The inner product of two polynomials in $V$ equals the (standard) inner product of their corresponding vectors in ${\mathbb{R}}^5$. Observe that $t$ is orthogonal to the constant function $1$. So take $p_0(t)=1$ and $p_1(t)=t$ . For $p_2$, use the vectors in ${\mathbb{R}}^5$ to compute the projection of $t^2$ onto S p a n { p 0 , p 1 } Span\{p_0,p_1\} Span{p0​,p1​}:
  The orthogonal projection of $t^2$ onto S p a n { p 0 , p 1 } Span\{p_0,p_1\} Span{p0​,p1​} is $\frac{10}{5}{p}_0+0p_1$. Thus
  
• An orthogonal basis for the subspace ${\mathbb{P}}^2$ of $V$ is:
  

Best Approximation in Inner Product Spaces

• A common problem in applied mathematics involves a vector space $V$ whose elements are functions. The problem is to approximate a function $f$ in $V$ by a function $g$ from a specified subspace $W$ of $V$ . The “closeness” of the approximation of $f$ depends on the way $\Vert f-g\Vert$ is defined. We will consider only the case in which the distance between $f$ and $g$ is determined by an inner product. In this case, the best approximation to $f$ by functions in $W$ is the orthogonal projection of $f$ onto the subspace $W$.

---

EXAMPLE 6

Let $V$ be ${\mathbb{P}}^4$ with the inner product in Example 5, and let $p_0,p_1,$ and $p_2$ be the orthogonal basis found in Example 5 for the subspace ${\mathbb{P}}^2$. Find the best approximation to $p(t)=5-\frac{1}{2}{t}^4$ by polynomials in ${\mathbb{P}}^2$.

SOLUTION

• The values of $p_0,p_1$, and $p_2$ at the numbers $-2,-1,0,1$, and $2$ are listed in ${\mathbb{R}}^5$ vectors in (3) above. The corresponding values for $p$ are $-3,9/2,5,9/2,$ and $-3$. Compute
  Then the best approximation in $V$ to $p$ by polynomials in ${\mathbb{P}}^2$ is
  
• This polynomial is the closest to $p$ of all polynomials in ${\mathbb{P}}^2$, when the distance between polynomials is measured only at $-2,-1,0,1$, and $2$. See Figure 1.
• The polynomials $p_0,p_1$, and $p_2$ in Examples 5 and 6 belong to a class of polynomials that are referred to in statistics as $orthogonal$ $polynomials$.

The Linearity of an Orthogonal Projection

• In any inner product space, the mapping $\mathbf{y}\mapsto \frac{<\mathbf{y},\mathbf{u}>}{<\mathbf{u},\mathbf{u}>}\mathbf{u}$ is linear, for any nonzero $\mathbf{u}$.
  
• Similarly, if ${\mathbf{u}}_1,...,{\mathbf{u}}_p$ are any nonzero vectors, then the mapping
  is a linear transformation. Thus, if { u 1 , … , u p } \{\boldsymbol u_1, …, \boldsymbol u_p\} {u1​,…,up​} is an orthogonal basis for a subspace $W$, then the mapping $\mathbf{y}\mapsto pro{j}_W\mathbf{y}$ is a linear transformation.

Two Inequalities

两个不等式

• Given a vector $\mathbf{v}$ in an inner product space $V$ and given a finite-dimensional subspace $W$ , we may apply the Pythagorean Theorem to the orthogonal decomposition of $\mathbf{v}$ with respect to $W$ and obtain
  See Figure 2. In particular, this shows that the norm of the projection of $\mathbf{v}$ onto $W$ does not exceed the norm of $\mathbf{v}$ itself. This simple observation leads to the following important inequality.

柯西-施瓦茨不等式

PROOF

• If $\mathbf{u}=0$, then both sides of (4) are zero, and hence the inequality is true in this case.
• If $\mathbf{u}\ne 0$, let $W$ be the subspace spanned by $\mathbf{u}$. Recall that $\Vert c\mathbf{u}\Vert =|c|\Vert \mathbf{u}\Vert$ for any scalar $c$. Thus
  Since $\Vert pro{j}_W\mathbf{v}\Vert \le \Vert \mathbf{v}\Vert$, $(4)$ is proved.

PROOF