双语矩阵论课程笔记（3）—— 【chapter 2】 Inner Product Spaces

• vector space/Linear space 中，向量之间的运算只有加法和数（统称线性运算），但是，如果以解析几何中的三维空间 ${\mathbb{R}}^3$ 作为线性空间的一个模型，我们会发现 ${\mathbb{R}}^3$ 诸如向量长度、向量夹角等度量概念无法在 Linear space 中得到反映。为此，有必要在一般的 Linear space 中引进 inner product 內积运算，这样就导出了 Inner product spaces 內积空间的概念

---

1. Inner product spaces

1.1 inner product (內积)

• Definition：Let $V$ a vector space over filed $P$, $\alpha ,\beta \in V$. If there is an operation $(\cdot ,\cdot )$ from $V$ to $P$ satisfying the following rules

  1. 对称性：$(\alpha ,\beta )=(\beta ,\alpha )$
  2. 可加性：$(\alpha +\beta ,\gamma )=(\alpha ,\gamma )+(\beta ,\gamma )$
  3. 齐次性：$(k\alpha ,\beta )=k(\alpha ,\beta )$
  4. 非负性：$(\alpha ,\alpha )\ge 0$，$\alpha =0\iff (\alpha ,\alpha )=0$

  then $(\alpha ,\beta )$ is the inner product of $\alpha$ and $\beta$

• 用 $V$ 的一组基来表示 $\alpha ,\beta$, 利用上述性质把连加的内积拆开，有
  
  其中 $y^H$ 是向量 $y$ 的共轭转置。可见矩阵 $A$ 和使用的基 ${\epsilon }_i$ 是一一对应的，当基选用标准正交基时，$A$ 矩阵最简单，为单位阵，详见第 2 节

1.2 inner product space (內积空间)

• 內积空间其实就是向量空间/线性空间的基础上加了一个內积运算
• Definition：Let $V$ a vector space over filed $P$ with inner product operation, then $V$ is called an inner product space on $P$
• Special case
  1. if $P=R$, $V$ is called Euclid space（欧氏空间）
  2. if $P=C$, $V$ is called Unitary space（酉空间）
• In inner product space, the following properties hold
  1. $(\alpha ,\beta +\gamma )=(\alpha ,\beta )+(\alpha ,\gamma )$
  2. $(\alpha ,k\beta )=\bar{k}(\alpha ,\beta )$（从第二个元提常数需要变成共轭，$\bar{k}$ 是 $k$ 的共轭）
  3. $(\alpha ,0\text{0}0)=(0\text{0}0,\alpha )=0$

1.3 More Notion in inner product space

1.3.1 Length of vector

• Definition：Suppose that $V$ is an inner product space, The length of vector $\alpha \in V$ is
  $$\Vert \alpha \Vert =\sqrt{(\alpha ,\alpha )}$$
  The vector of length 1 is called a unit vector

• For any $\alpha ,\beta \in V$, the length $\Vert \alpha \Vert$ satisfies the following properties

  1. $\Vert \alpha \Vert \ge 0$ and $\alpha =0\iff \Vert \alpha \Vert =0$
  2. $\Vert k\alpha \Vert =|k|\Vert \alpha \Vert$ for any $k\in P$
  3. $\Vert \alpha +\beta {\Vert }^2+\Vert \alpha -\beta {\Vert }^2=2(\Vert \alpha {\Vert }^2+\Vert \beta {\Vert }^2)$ for all $\alpha ,\beta \in V$
  4. $|(\alpha ,\beta )|\le \Vert \alpha \Vert \Vert \beta \Vert$（Cauchy Inequality 柯西不等式）
  5. $\begin{array}{r}\Vert \alpha +\beta \Vert \le \Vert \alpha \Vert +\Vert \beta \Vert \\ \Vert \alpha -\beta \Vert \le \Vert \alpha \Vert +\Vert \beta \Vert \end{array}$ （三角不等式）

• 上面第 4 条 Cauchy Inequality 是一个重要的不等式

  1. 证明，构造一个向量 $\alpha +x\beta$，其中 $x=-\frac{(\alpha ,\beta )}{(\beta ,\beta )}$，利用自己和自己做内积的非负性，有
     $$\begin{array}{rl}& \begin{array}{rl}\because (\alpha +k\beta ,\alpha +k\beta )& =(\alpha ,\alpha )+x^2(\beta ,\beta )+2x(\alpha ,\beta )\\ & =(\alpha ,\alpha )+\frac{(\alpha ,\beta {)}^2}{(\beta ,\beta )}-2\frac{(\alpha ,\beta {)}^2}{(\beta ,\beta )}\ge 0\end{array}\\ & \therefore (\alpha ,\alpha )(\beta ,\beta )\ge (\alpha ,\beta {)}^2\\ & \therefore \Vert \alpha {\Vert }^2\Vert \beta {\Vert }^2\ge (\alpha ,\beta {)}^2\\ & \therefore \Vert \alpha \Vert \Vert \beta \Vert \ge |(\alpha ,\beta )|\end{array}$$
  2. 在不同的内积空间中，柯西不等式有不同的表现形式
     

• uniformization归一化 / unitization单位化：For any $\alpha \ne 0$, $\frac{\alpha }{\Vert \alpha \Vert }$ is a unit vector

1.3.2 Distance of vectors

• Definition：if $V$ is an inner product space, the distance of $\alpha ,\beta \in V$ is defined as
  $$d(\alpha ,\beta )=\Vert \alpha -\beta \Vert$$
• the distance $d(\alpha ,\beta )$ satisfies the following rules
  1. 非负性：$d(\alpha ,\beta )\ge 0$ and $d(\alpha ,\beta )=0\iff \alpha =\beta$
  2. 对称性：$d(\alpha ,\beta )=d(\beta ,\alpha )$
  3. 三角不等式：$d(\alpha ,\gamma )\le d(\alpha ,\beta )+d(\beta ,\gamma )$

1.3.3 Angle of vectors

• Definition：Suppose that $V$ is an inner product space, $\alpha ,\beta \in V,\alpha ,\beta \ne 0\text{0}0$, the aggle of $\alpha$ and $\beta$ is defined as
  $$⟨\alpha ,\beta ⟩=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{c}\mathrm{o}\mathrm{s}\frac{(\alpha ,\beta )}{\Vert \alpha \Vert \Vert \beta \Vert },0\le ⟨\alpha ,\beta ⟩\le \pi$$
• Especially if $(\alpha ,\beta )=0$, we call that $\alpha$ is orthogonal to $\beta$, denote as $$(\alpha ,\beta )=0\iff \alpha \perp \beta$$
  Two orthogonal vectors satisfy the Pythagorean Law (勾股定理)
  $$\alpha \perp \beta \Rightarrow \Vert \alpha +\beta {\Vert }^2=\Vert \alpha -\beta {\Vert }^2=\Vert \alpha {\Vert }^2+\Vert \beta {\Vert }^2$$

1.4 Example

1.4.1 Example 1

• There are vectors $x=(x_1,x_2,...,x_n{)}^T,y=(y_1,y_2,...,y_n{)}^T$, define the inner product operation as
  $$\begin{array}{rl}& (x,y)=y^{T}x=\sum _{i=1}^n{x}_i{y}_{i}x,y\in R^n\\ & (x,y)=y^{H}x=\sum _{i=1}^n{x}_i\overline{y_i}x,y\in C^n\end{array}$$
  then $R^n$ and $C^n$ are inner product spaces

1.4.2 Example 2

• Vector space $C[a,b]$ is an inner product space with the inner product operation defined as
  $$(f,g)={\int }_a^{b}f(x)g(x)dx$$
• Proof:
  1. $(f,g)={\int }_a^{b}f(x)g(x)dx={\int }_a^{b}g(x)f(x)dx=(g,f)$
  2. $(kf,g)={\int }_a^{b}kf(x)g(x)dx=k{\int }_a^{b}f(x)g(x)dx=k(f,g)$
  3. $(f+g,h)={\int }_a^b(f(x)+g(x))h(x)dx={\int }_a^{b}f(x)h(x)dx+{\int }_a^{b}g(x)h(x)dx=(f,h)+(g,h)$
  4. $(f,f)={\int }_a^b(f(x){)}^{2}dx\ge =0$, and ${\int }_a^b(f(x){)}^{2}dx=0\iff f(x)\equiv 0$

1.4.3 Example 3

• Consider the vector space $C[a,b]$, the convolution operation
  $$(f,g)={\int }_a^{b}f(x)g(a+b-x)dx$$
  is a inner product operation or not? Why?
• Answer：No
  

2. Orthonormal basis

2.1 Orthonormal basis（标准正交基）

2.1.1 Definition

1. Suppose that ${\alpha }_1,{\alpha }_2,...,{\alpha }_m\ne 0\text{0}0$ are nonzero vectors in inner product space $V$. If $$({\alpha }_i,{\alpha }_j)=0,i\ne j$$ then ${\alpha }_1,{\alpha }_2,...,{\alpha }_m$ are orthogonal (正交的). Especially, if
   $$({\alpha }_i,{\alpha }_j)={\delta }_{ij}=\{\begin{array}{ll}0& i\ne j\\ 1& i=j\end{array}$$
   the vector group ${\alpha }_1,{\alpha }_2,...,{\alpha }_m$ are called orthonormal (标准正交的)
2. In $n$-dimensional inner product space $V$, $n$ orthogonal vectors form a basis of $V$ which is named Orthogonal Basis (正交基). If the basis is orthogonal and unit, it is called Orthonormal Basis (标准正交基)
3. Suppose that $V_1,V_2$ are two subspaces of inner product space $V$
   1. Given $\alpha \in V$, if $\forall \beta \in V_1$ satisfies $(\alpha ,\beta )=0$, $\alpha$ is said to be orthogonal to $V_1$, denoted by $\alpha \perp V_1$
      向量正交于空间，指此向量与空间中任意向量都正交
   2. For each $\alpha \in V_1$ and $\beta \in V_2$, if $(\alpha ,\beta )=0$, $V_1$ is said to be orthogonal to $V_2$, denoted by $V_1\perp V_2$
      一空间正交于另一空间，指一空间中任意向量与另一空间中任意向量都正交（只要两个空间的某组基正交）
4. If two subspace $V_1,V_2$ of inner product space $V$ are orthogonal (i.e. $V_1\perp V_2$), then $V_1+V_2$ is named orthogonal sum (正交和), denoted by $V_1\oplus V_2$.（正交和一定是直和，见下面 Theorem 3）
5. Suppose that $V_1$ is a subspace of inner product space $V$, the subspace of $V$ including all the vectors orthogonal to $V_1$ is called the orthogonal complement (正交补) of $V_1$, denoted by
   V 1 ⊥ = { α ∈ V ∣ α ⊥ V 1 } V_1^\perp = \{\alpha\in V|\alpha\perp V_1\} V1⊥​={α∈V∣α⊥V1​}
   $V_1^{\perp }$ 是 $V$ 的 subspace，要证明这一点，可以先证明其包含 $0\text{0}0$ 所以非空，再论证其中元素满足加法和数乘的封闭性
6. Suppose that $V_1$ is a subspace of an inner product space $V$. If any given $\alpha \in V$ can be decomposed as $$\alpha ={\alpha }_1+{\alpha }_2$$
   where ${\alpha }_1\in V_1$ and ${\alpha }_2\perp V_1$, then ${\alpha }_1$ is called the orthogonal projection (正交投影) of $\alpha$ on $V_1$
   由于 $V=V_1\oplus V_1^{\perp }$ (下面定理 4)，任何一个向量都可以分解为相互垂直的两部分，2.2 节的标准正交化过程，其实就是在不停地减去正交投影部分，保留与之前的空间正交的部分
7. Assume that $V_1$ is a subspace of an inner product space $V$. Given $\alpha \in V$, if there exist a vector $a_1\in V_1$ such that
   $$d(\alpha ,V_1):=\Vert \alpha -{\alpha }_1\Vert =\underset{\beta \in V_1}{\min }\Vert \alpha -\beta \Vert$$
   then ${\alpha }_1$ is named the Optimal Approximation (最优近似) of $\alpha$ over $V_1$
   这里定义了向量到空间的距离为 “该向量到空间中所有向量距离的最小值”
   
   如上所述，subspace $V_1$ 中距离 $\alpha$ 最近的向量是 $\alpha$ 在 $V_1$ 中的 orthogonal projection 正交投影

2.1.2 Theorem

1. If the vectors ${\alpha }_1,{\alpha }_2,...,{\alpha }_m$ in inner product space are orthogonal , then ${\alpha }_1,{\alpha }_2,...,{\alpha }_m$ are linearly independent
   一组正交向量一定线性无关
   
2. Suppose that $V$ is an $n$-dimensional inner product space. The orthonormal basis of $V$ surely exist, and arbitrary orthonormal vector group in $V$ can be expanded to be an orthonormal basis of $V$
   n维内积空间中，标准正交基一定存在，且任意一组标准正交向量可以扩展为一组标准正交基
   1. 随便给一组基，可以用下面的标准正交化过程变为一组标准正交基
   2. 给出一组标准正交向量，由于正交，它们一定线性无关，把 $V$ 中不能被它们线性表出的向量都加入这组向量，这样就得到了 $V$ 的一组基，再用标准正交化过程处理，即得一组标准正交基
3. Assume that $V_1$ and $V_2$ are twp subspace of inner product space $V$, if $V_1\perp V_2$ then $V_1+V_2$ is a direct sum
   若內积空间的两个子空间垂直，则两个子空间的和为直和
   使用等价条件 V 1 ⋂ V 2 = { 0 } V_1\bigcap V_2 = \{\pmb{0 }\} V1​⋂V2​={000} 证明，用反证法
   
4. Suppose that inner product space $V$ is the finite dimensional, $V_1$ is a subspace of $V$. There exists unique subspace $V_1^{\perp }$, such that $V=V_1\oplus V_1^{\perp }$
   一个內积空间可以拆成某一子空间及其正交补的正交直和
   
   example: Let $\alpha =\left[\begin{array}{cc}1& 0\\ 0& 0\end{array}\right]\in R^{2\times 2}$, $V_1=\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}(\alpha )$, $(A,B)=tr(B^{H}A)$, please show $V_1^{\perp }$ 下面从不同的角度给出两种解法
   
5. (Projection Theorem) If $V_1$ is a finite dimensional subspace of an inner product space $V$, then each $\alpha \in V$ has the unique orthogonal projection on $V_1$
   內积空间中的任意向量，都在某个子空间中有唯一的正交投影（因为內积空间可以分解为任意子空间与其正交补的正交直和，內积中的任意向量也就可以唯一地直和分解到这两个子空间中）

2.2 Orhogonalization Process（标准正交化过程）

• If the vectors ${\alpha }_1,{\alpha }_2,...,{\alpha }_m$ in inner product space $V$ are linearly independent, then there is an orthonormal vector group ${\epsilon }_1,{\epsilon }_2,...,{\epsilon }_m$ which are equivalent (互相可以线性表出) to ${\alpha }_1,{\alpha }_2,...,{\alpha }_m$
  内积空间中一组线性无关向量一定可以标准正交化
  1. 正交化的思想：把每个向量中和它不正交的部分减掉
  2. 标准化的思想：把向量的长度缩短其模倍
• 将线性无关向量组 ${\alpha }_1,{\alpha }_2,...,{\alpha }_m$ 标准正交化为 ${\epsilon }_1,{\epsilon }_2,...,{\epsilon }_m$ 的过程如下
  
• example
  
  注意要严格按照定义计算，比如求 ${\epsilon }_1$ 时，其中算 $\Vert {\beta }_1\Vert =\sqrt{(\beta ,\beta )}$ 要带入题目给出的內积公式，即 $\sqrt{(\beta ,\beta )}=\sqrt{{\int }_{-1}^{1}1dx}=\sqrt{2}$