Chapter 2 (Matrix Algebra): Matrix factorizations (矩阵因式分解)

本文为《Linear algebra and its applications》的读书笔记

• A $factorization$ of a matrix $A$ is an equation that expresses $A$ as a product of two or more matrices. Whereas matrix multiplication involves a $synthesis$ of data (combining the effects of two or more linear transformations into a single matrix).
• In the language of computer science, the expression of $A$ as a product amounts to a preprocessing of the data in $A$, organizing that data into two or more parts whose structures are more useful in some way, perhaps more accessible for computation(便于计算).

The LU Factorization

LU 分解

What is LU Factorization?

• Any $m\times n$ matrix $A$ admits a factorization $A=LU$, where $L$ is an $m\times m$ permuted unit lower triangular matrix with 1’s on the diagonal and $U$ is an $m\times n$ echelon form of $A$.
  • That is, $L$ is a matrix such that a permutation (rearrangement) of its rows (using row interchanges) will produce a lower triangular matrix with 1’s on the diagonal.
    

注：如果将 $A$ 行变换至 $U$ 的过程中不需要使用 row interchange，那么 $L$ 为单位下三角矩阵；下面在讲到 “Permuted LU Factorizations” 之前都假设将 $A$ 行变换至 $U$ 的过程中不需要使用 row interchange

Why LU Factorization?

• When $A=LU$, the equation $A\mathbf{x}=\mathbf{b}$ can be written as $L(U\mathbf{x})=\mathbf{b}$. Writing $\mathbf{y}$ for $U\mathbf{x}$, we can find $\mathbf{x}$ by solving the pair of equations
  First solve $L\mathbf{y}=\mathbf{b}$ for $\mathbf{y}$; and then solve $U\mathbf{x}=\mathbf{y}$ for $\mathbf{x}$. See Figure 2. Each equation is easy to solve because $L$ and $U$ are triangular.
  
• When $A$ is invertible, MATLAB finds $A^{-1}$ by factoring $A=LU$ (where $L$ may be permuted lower triangular), inverting $L$ and $U$, and then computing $U^{-1}{L}^{-1}$.

---

EXAMPLE 1

• It can be verified that
  Use this LU factorization of $A$ to solve $A\mathbf{x}=\mathbf{b}$, where $\mathbf{b}=\left[\begin{array}{c}-9\\ 5\\ 7\\ 11\end{array}\right]$.

SOLUTION

• The solution of $L\mathbf{y}=\mathbf{b}$ needs only 6 multiplications and 6 additions.
  Then, for $U\mathbf{x}=\mathbf{y}$, the “backward” phase of row reduction requires 4 divisions, 6 multiplications, and 6 additions. (For instance, creating the zeros in column 4 requires 1 division in row 4 and 3 multiplication–addition pairs to add multiples of row 4 to the rows above.)
  
• To find $\mathbf{x}$ requires 28 arithmetic operations, or “flops” (floating point operations), excluding the cost of finding $L$ and $U$. In contrast, row reduction of $[A\mathbf{b}]$ to $[I\mathbf{x}]$ takes 62 operations.

---

Checkpoint:

Section 2.2 shows how to compute $A^{–1}B$ by row reduction. Describe how you could speed up this calculation if you have an LU factorization of $A$ available (and $A$ is invertible).

Answer:

• If $A$ is an invertible $n\times n$ matrix, with an LU factorization $A=LU$, and if $B$ is $n\times p$, then $A^{–1}B$ can be computed by first row reducing $[LB]$ to a matrix $[IY]$ for some $Y$ and then reducing $[UY]$ to $[I{A}^{–1}B]$.
• MATLAB uses this approach to compute $A^{–1}B$ (after first finding $L$ and $U$).

An LU Factorization Algorithm

• At first, assume that $A$ is an $m\times n$ matrix that can be row reduced to echelon form, $withoutrowinterchanges$. (Later, we will treat the general case.)
• In this case, there exist unit lower triangular elementary matrices $E_1,...,E_p$ such that
  $$E_p...E_{1}A=U(3)$$Then
  $$A=(E_p...E_1{)}^{-1}U=LU(4)$$It can be shown that products and inverses of unit lower triangular matrices are also unit lower triangular. Thus $L$ is unit lower triangular.

PROOF
Let $A$ be a lower triangular $n\times n$ matrix with nonzero entries on the diagonal. Show that $A$ is invertible and $A^{-1}$ is lower triangular.
[Hint: Explain why $A$ can be changed into $I$ using only row replacements and scaling. Also, explain why the row operations that reduce $A$ to $I$ change $I$ into a lower triangular matrix.] 也可以用上一节介绍的分块矩阵+数学归纳法来证明

---

• Note that the row operations in equation (3), which reduce $A$ to $U$, also reduce the $L$ in equation (4) to $I$. This observation is the key to constructing $L$.

Step 1 is not always possible, but when it is, the argument above shows that an LU factorization exists.

---

EXAMPLE 2

Find an LU factorization of
$$A=\left[\begin{array}{rrrrr}2& 4& -1& 5& -2\\ -4& -5& 3& -8& 1\\ 2& -5& -4& 1& 8\\ -6& 0& 7& -3& 1\end{array}\right]$$

SOLUTION

• Since $A$ has four rows, $L$ should be $4\times 4$. The first column of $L$ is the first column of $A$ divided by the top pivot entry:
  $$L=\left[\begin{array}{rrrr}1& 0& 0& 0\\ -2& 1& 0& 0\\ 1& & 1& 0\\ -3& & & 1\end{array}\right]$$Compare the first columns of $A$ and $L$. The row operations that create zeros in the first column of $A$ will also create zeros in the first column of $L$. To make this same correspondence of row operations on $A$ hold for the rest of $L$, watch a row reduction of $A$ to an echelon form $U$. That is, highlight the entries in each matrix that are used to determine the sequence of row operations that transform $A$ into $U$.
  The highlighted entries determine the row reduction of $A$ to $U$. At each pivot column, divide the highlighted entries by the pivot and place the result into $L$:
  

Permuted LU Factorizations

置换 LU 分解

• In practical work, row interchanges are nearly always needed, because partial pivoting (部分主元法) is used for high accuracy. (Recall that this procedure selects, among the possible choices for a pivot, an entry in the column having the largest absolute value.)

---

Permuted LU Factorizations

• Any $m\times n$ matrix $A$ admits a factorization $A=LU$, with $U$ in echelon form and $L$ a permuted unit lower triangular matrix.
  • That is, $L$ is a matrix such that a permutation (rearrangement) of its rows (using row interchanges) will produce a lower triangular matrix with 1’s on the diagonal.

---

Permuted LU Factorizations Algorithm

• To handle row interchanges, the construction of $L$ and $U$ depends on first using row replacements to reduce $A$ to a permuted echelon form $V$ and then using row interchanges to reduce $V$ to an echelon form $U$.
  • By watching the reduction of $A$ to $V$, we can easily construct a permuted unit lower triangular matrix $L$ with the property that the sequence of operations changing $A$ into $U$ also changes $L$ into $I$. This property will guarantee that $A=LU$.
• The following algorithm reduces any matrix to a permuted echelon form. In the algorithm when a row is covered, we ignore it in later calculations.
  • (1) Begin with the leftmost nonzero column. Choose any nonzero entry as the pivot. Designate the corresponding row as a pivot row.
  • (2) Use row replacements to create zeros above and below the pivot (in all uncovered rows). Then cover that pivot row.
  • (3) Repeat steps 1 and 2 on the uncovered submatrix, if any, until all nonzero entries are covered.

---

Example 1

• As an example, choose any entry in the first column of the following matrix as the first pivot, and use the pivot to create zeros in the rest of column 1. We choose the (3, 1)-entry.
  Row 3 is the first pivot row. Choose the (2, 2)-entry as the second pivot, and create zeros in the rest of column 2, excluding the first pivot row.
  Cover row 2 and choose the (4, 4)-entry as the pivot. Create zeros in the other rows, excluding the first two pivot rows.
  
• Let $V$ denote this permuted echelon form, and permute the rows of $V$ to create an echelon form. The resulting echelon matrix $U$ is
  
• The last step is to create $L$. Go back and watch the reduction of $A$ to $V$. As each pivot is selected, take the pivot column, and divide the pivot into each entry in the column that is not yet in a pivot row. Place the resulting column into $L$. At the end, fill the holes in $L$ with zeros.
  

---

Example 2

• The next example illustrates what to do when $V$ has one or more rows of zeros. For the reduction of $A$ to $V$, pivots were chosen to have the largest possible magnitude (the choice used for “partial pivoting”).
  
  The first three columns of $L$ come from the three pivot columns above.
  The matrix $L$ needs two more columns. Use columns 1 and 3 of the $5\times 5$ identity matrix to place 1’s in the “nonpivot” rows 1 and 3. Fill in the remaining holes with zeros.
  

A Matrix Factorization in Electrical Engineering

• Matrix factorization is intimately related to the problem of constructing an electrical network with specified properties.

---

• Suppose the box in Figure 3 represents some sort of electric circuit, with an input and output.
  
• Record the input voltage and current by $\left[\begin{array}{c}{v}_1\\ i_1\end{array}\right]$ and record the output voltage and current by $\left[\begin{array}{c}{v}_2\\ i_2\end{array}\right]$. Frequently, the transformation $\left[\begin{array}{c}{v}_1\\ i_1\end{array}\right]\mapsto \left[\begin{array}{c}{v}_2\\ i_2\end{array}\right]$ is linear. That is, there is a matrix $A$, called the $transfer$ $matrix$, such that
  $$\left[\begin{array}{c}{v}_2\\ i_2\end{array}\right]=A\left[\begin{array}{c}{v}_1\\ i_1\end{array}\right]$$
• Figure 4 shows a $laddernetwork$. The left circuit in Figure 4 is called a $seriescircuit(串联电路)$, with resistance $R_1$. The right circuit in Figure 4 is a $shuntcircuit(并联电路)$, with resistance $R_2$.
  
• Using Ohm’s law and Kirchhoff’s laws, one can show that the transfer matrices of the series and shunt circuits, respectively, are
  
• A network transfer matrix summarizes the input–output behavior (the design specifications) of the network without reference to the interior circuits. To physically build a network with specified properties, an engineer first determines if such a network can be constructed (or realized). Then the engineer tries to factor the transfer matrix into matrices corresponding to smaller circuits that perhaps are already manufactured and ready for assembly. In the common case of alternating current (交流电), the entries in the transfer matrix are usually rational complex-valued functions. A standard problem is to find a minimal realization that uses the smallest number of electrical components.

---

EXAMPLE 3

• a. Compute the transfer matrix of the ladder network in Figure 4.
• b. Design a ladder network whose transfer matrix is $\left[\begin{array}{cc}1& -8\\ -.5& 5\end{array}\right]$.

SOLUTION

• a. Let $A_1$ and $A_2$ be the transfer matrices of the series and shunt circuits, respectively. Then the transfer matrix of the ladder network corresponds to composition of linear transformations
  
• b. To factor the matrix $\left[\begin{array}{cc}1& -8\\ -.5& 5\end{array}\right]$ into the product of transfer matrices, as in equation
  (6), look for $R_1$ and $R_2$ in Figure 4 to satisfy
  

The QR Factorization

QR 分解

• Suppose $A=QR$, where $Q$ and $R$ are $n\times n$. $R$ is invertible and upper triangular, and $Q$ has the property that $Q^{T}Q=I$.
  • When $A=QR$, the equation $A\mathbf{x}=\mathbf{b}$ can be written as $Q(R\mathbf{x})=\mathbf{b}$. Then $R\mathbf{x}=Q^T\mathbf{b}$, which is easy to solve. It can be seen that the equation $A\mathbf{x}=\mathbf{b}$ has a unique solution.