Chapter 7 (Symmetric Matrices and Quadratic Forms): Quadratic forms (二次型)

Quadratic forms

• A quadratic form on ${\mathbb{R}}^n$ is a function $Q$ defined on ${\mathbb{R}}^n$ whose value at a vector $\mathbf{x}$ in ${\mathbb{R}}^n$ can be computed by an expression of the form $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, where $A$ is an $n\times n$ symmetric matrix. The matrix $A$ is called the matrix of the quadratic form (关于二次型的矩阵).
  • The simplest example of a nonzero quadratic form is $Q(\mathbf{x})={\mathbf{x}}^{T}I\mathbf{x}={\Vert \mathbf{x}\Vert }^2$.

当 $A$ 不为对称矩阵时，可以把它写为 $A=A^S+A^A$ 的形式，其中 $A^S$ 为对称矩阵，$A^A$ 为反对称矩阵。可以证明，$x^{T}Ax=x^T{A}^{S}x$
.
证明：
$$x^T{A}^{A}x=\sum _{i,j}{x}_i{A}_{ij}^A{x}_j=\frac{1}{2}\left(\sum _{i,j}{x}_i{A}_{ij}^A{x}_j-\sum _{i,j}{x}_j{A}_{ji}^A{x}_i\right)$$可以将第 2 项中的 $i,j$ 互换，可得
$$x^T{A}^{A}x=\frac{1}{2}\left(\sum _{i,j}{x}_i{A}_{ij}^A{x}_j-\sum _{i,j}{x}_i{A}_{ij}^A{x}_j\right)=0$$

---

• Examples 1 and 2 show the connection between any symmetric matrix $A$ and the quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$.

EXAMPLE 1

Let $\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$. Compute ${\mathbf{x}}^{T}A\mathbf{x}$ for the following matrices:

SOLUTION

• $a$.
  
  • Note that the quadratic form associated with the diagonal matrix $A$ in has no $x_1{x}_2$ $cross$-$product$ term (交叉乘积).
• $b$. There are two $-2$ entries in $A$. Watch how they enter the calculations.
  
  • The presence of $-4x_1{x}_2$ in the quadratic form is due to the $-2$ entries off the diagonal in the matrix $A$.

EXAMPLE 2

For $\mathbf{x}$ in ${\mathbb{R}}^3$, let $Q(\mathbf{x})=5x_1^2+3x_2^2+2x_3^2-x_1{x}_2+8x_2{x}_3$. Write this quadratic form as ${\mathbf{x}}^{T}A\mathbf{x}$.

SOLUTION

• The coefficients of ${\mathbf{x}}_1^2,{\mathbf{x}}_2^2,{\mathbf{x}}_3^2$ go on the diagonal of $A$. To make $A$ symmetric, the coefficient of $x_i{x}_j$ for $i\ne j$ must be split evenly between the $(i,j)$- and $(j,i)$-entries in $A$. It is readily checked that
  

Change of Variable in a Quadratic Form

二次型的变量代换

通过坐标变换消除交叉乘积项

• In some cases, quadratic forms are easier to use when they have no cross-product terms—that is, when the matrix of the quadratic form is a diagonal matrix.
  • Fortunately, the cross-product term can be eliminated by making a suitable change of variable.

---

• If $\mathbf{x}$ represents a variable vector in ${\mathbb{R}}^n$, then a change of variable is an equation of the form
  where $P$ is an invertible matrix and $\mathbf{y}$ is a new variable vector in ${\mathbb{R}}^n$. Here $\mathbf{y}$ is the coordinate vector of $\mathbf{x}$ relative to the basis of ${\mathbb{R}}^n$ determined by the columns of $P$.
• If the change of variable (1) is made in a quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$, then
  and the new matrix of the quadratic form is $P^{T}AP$. Since $A$ is symmetric, there is an orthogonal matrix $P$ such that $P^{T}AP$ is a diagonal matrix $D$, and the quadratic form in (2) becomes ${\mathbf{y}}^{T}D\mathbf{y}$.
  

主轴定理

• The columns of $P$ in the theorem are called the principal axes (主轴) of the quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$.
• The vector $\mathbf{y}$ is the coordinate vector of $\mathbf{x}$ relative to the orthonormal basis of ${\mathbb{R}}^n$ given by these principal axes.

A Geometric View of Principal Axes

• Suppose $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, where $A$ is an invertible $2\times 2$ symmetric matrix, and let $c$ be a constant. It can be shown that the set of all $\mathbf{x}$ in ${\mathbb{R}}^2$ that satisfy
  either corresponds to an ellipse (or circle), a hyperbola (双曲线), two intersecting lines, or a single point, or contains no points at all.
  • If $A$ is a diagonal matrix, the graph is in standard position, such as in Figure 2.
    
  • If $A$ is not a diagonal matrix, the graph of equation (3) is rotated out of standard position, as in Figure 3. Finding the principal axes (determined by the eigenvectors of $A$) amounts to finding a new coordinate system with respect to which the graph is in standard position. The positive $y_1$-axis in Figure 3(b) is in the direction of the first column of the matrix $P$, and the positive $y_2$-axis is in the direction of the second column of $P$.
    

Classifying Quadratic Forms

二次型的分类

• When $A$ is an $n\times n$ matrix, the quadratic form $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$ is a real-valued function with domain ${\mathbb{R}}^n$. Figure 4 displays the graphs of four quadratic forms with domain ${\mathbb{R}}^2$. For each point $\mathbf{x}=(x_1,x_2)$ in the domain of a quadratic form $Q$, the graph displays the point $(x_1,x_2,z)$ where $z=Q(\mathbf{x})$.
  • Notice that except at $\mathbf{x}=0$, the values of $Q(\mathbf{x})$ are all positive in Figure 4(a) and all negative in Figure 4(d). The horizontal cross-sections (水平截面) of the graphs are ellipses in Figures 4(a) and 4(d) and hyperbolas in Figure 4$(c)$.
    

正定, 负定, 不定, 半正定, 半负定

• The simple $2\times 2$ examples in Figure 4 illustrate the following definitions
  

positive definite (正定的); negative definite (负定的); indefinite (不定的)

• Also,$Q$ is said to be positive semidefinite (半正定的) if $Q(\mathbf{x})\ge 0$ for all $\mathbf{x}$, and to be negative semidefinite if $Q(\mathbf{x})\le 0$ for all $\mathbf{x}$.

二次型与特征值

• 类似地，半正定矩阵的特征值都大于等于 0；半负定矩阵的特征值都小于等于 0

PROOF

• By the Principal Axes Theorem, there exists an orthogonal change of variable $\mathbf{x}=P\mathbf{y}$ such that
  where ${\lambda }_1,...,{\lambda }_n$ are the eigenvalues of $A$. Since $P$ is invertible, there is a one-to-one correspondence between all nonzero $\mathbf{x}$ and all nonzero $\mathbf{y}$. Thus the values of $Q(\mathbf{x})$ for $\mathbf{x}\ne 0$ coincide with the values of the expression on the right side of (4), which is obviously controlled by the signs of the eigenvalues ${\lambda }_1,...,{\lambda }_n$, in the three ways described in the theorem.

二次型与行列式

Another useful way to characterize quadratic forms

• Let $A=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$. If ${\lambda }_1$ and ${\lambda }_2$ are the eigenvalues of $A$, then the characteristic polynomial is $det(A-\lambda I)={\lambda }^2-(a+d)\lambda +ad-bc$. Thus ${\lambda }_1+{\lambda }_2=a+d$ and ${\lambda }_1{\lambda }_2=detA$. The following statements can be easily verified:
  • $a$. $Q$ is positive definite if $detA>0$ and $a>0$.
  • $b$. $Q$ is negative definite if $detA>0$ and $a<0$.
  • $c$. $Q$ is indefinite if $detA<0$.
• The $2\times 2$ case can be generalized to $n\times n$ matrices.

正定矩阵, 负定矩阵, 不定矩阵, 半正/负定矩阵

• The classification of a quadratic form is often carried over to the matrix of the form. Thus a positive definite matrix (正定矩阵) $A$ is a symmetric matrix for which the quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$ is positive definite. Other terms, such as positive semidefinite matrix, are defined analogously.
  • 注意到，正定矩阵特征值均大于 0，因此无 0 特征值 ($A\mathbf{x}=0\mathbf{x}$ 无非零解)，因此必可逆；同理负定矩阵也可逆
  • 正定矩阵的逆矩阵也可逆
    • 假设 $A$ 为正定矩阵，易证 $A^{-1}$ 为对称矩阵，因此有
      $$x^T{A}^{-1}x=x^T{A}^{-1}A{A}^{-1}x=x^T(A^{-1}{)}^{T}A{A}^{-1}x=(A^{-1}x{)}^{T}A(A^{-1}x)$$

---

EXERCISE 27

Let $A$ and $B$ be symmetric $n\times n$ matrices whose eigenvalues are all positive. Show that the eigenvalues of $A+B$ are all positive.

SOLUTION

• [Hint: Consider quadratic forms.]

Gram Matrix $A^{T}A$ (格拉姆矩阵 / 内积矩阵)

• 设 $G=A^{T}A$ 为 $A$ 的格拉姆矩阵，则 $g_{ij}={\mathbf{a}}_i\cdot {\mathbf{a}}_j$ 显然为一个对称矩阵

---

EXERCISE 25

Show that if $B$ is $m\times n$, then $B^{T}B$ is positive semidefinite; and if $B$ is $n\times n$ and invertible, then $B^{T}B$ is positive definite.

SOLUTION

• [Hint: ${\mathbf{x}}^T{B}^{T}B\mathbf{x}$]

---

EXERCISE 26

Show that if an $n\times n$ matrix $A$ is positive definite, then there exists a positive definite matrix $B$ such that $A=B^{T}B$.

SOLUTION

• [Hint: Write $A=PD{P}^T$, with $P^T=P^{–1}$. Produce a diagonal matrix $C$ such that $D=C^{T}C$, and let $B=PC{P}^T$. Show that $B$ works.]

---

EXERCISE

If $A$ is $m\times n$, show that the Gram matrix of any matrix $A$ has the same rank as $A$.

SOLUTION

• [Hint: Section 6.5 Theorem 14] or Exercise 13 in Section 4.6

楚列斯基分解

EXERCISE

Prove that an $n\times n$ matrix $A$ is positive definite if and only if $A$ admits a Cholesky factorization, namely, $A=R^{T}R$ for some invertible upper triangular matrix $R$ whose diagonal entries are all positive. [Hint: Use a QR factorization and Exercise 26.]

SOLUTION

• If $A=R^{T}R$, where $R$ is invertible, then ${\mathbf{x}}^{T}A\mathbf{x}=(R\mathbf{x}{)}^T(R\mathbf{x})\ge 0$ when $\mathbf{x}\ne 0$. Thus $A$ is positive definite.
• Conversely, suppose that $A$ is positive definite. Then by Exercise 26, $A=B^{T}B$ for some positive definite matrix $B$. Since the eigenvalues of $B$ are positive, 0 is not an eigenvalue and so $B$ is invertible. In particular, the columns of $B$ are linearly independent. By Theorem 12 in Section 6.4, $B=QR$ for some $n\times n$ matrix $Q$ with orthonormal columns and some upper triangular matrix $R$ with positive elements on its diagonal. Since $Q$ is square, $Q^{T}Q=I$. So
  $$A=B^{T}B=(QR{)}^T(QR)=R^T{Q}^{T}QR=R^{T}R$$

References

• $Linear$ $algebra$ $and$ $its$ $applications$