Introduction to Linear Algebra, Chapter-2, Solving Linear Equations, Key Notes

Introduction to Linear Algebra, Chapter-2, Solving Linear Equations, Key Notes

本人在阅读MIT数学教授Gilbert Strang所著线性代数教材"Introduction to Linear Algebra(Fifth Edition)"过程中敲下的笔记

我是用的教学视频是BV1uK4y187ep

课后习题答案及其它相关资料可参照math.mit.edu/linearalgebra

2.1 Vectors and Linear Equations

从两种角度看待线性方程组

$$A\overrightarrow{x}=\overrightarrow{b},\text{假设A是3 by 3的矩阵}$$

• 从行的角度看待（row picture）：$A\overrightarrow{x}=\overrightarrow{b}$表示在三维空间中有三个平面交于一点（A的一行定义一个平面）

• 从列的角度看待（column picture）：$A\overrightarrow{x}=\overrightarrow{b}$表示要寻找一个关于$A$的列的线性组合来表示出$\overrightarrow{b}$

两种方法计算$A\overrightarrow{x}$

• $A$的每一行与$\overrightarrow{x}$的点积（dot product）
• $A$的列的线性组合

其它

For $m$ by $n$ matrices, $A\overrightarrow{x}=\overrightarrow{0}$ may have many solutions, those solutions will go into a vector space, the rank of $A$ leads to the dimension of that vector space.

2.2 The Idea of Elimination

这章基本就一句重点：forward elimination and back substitution.

elimination在遇到为0的pivot的时候有可能会失败，如果能通过行交换找到非0的pivot，则消元可以继续，如果找不到非0的pivot，则遇到了permanent breakdown，这时候方程组没有解或者有无数解。

2.3 Elimination Using Matrices

从行和列两个角度看待矩阵与向量相乘（再次强调）

$A\overrightarrow{x}$ is a combination of the columns of $A$;
Components of $A\overrightarrow{x}$ are the dot products with rows of $A$;

Elimination Matrix（消元矩阵）

start with the identity matrix $I$, change one of its zeros to the multiplier $-l$;
for a elimination matrix $E_{ij}$, it has nonzero entry $-l$ in the $(i,j)$ position, so $E_{ij}A$ will subtract $l$ times of row $j$ from row $i$.

交换律和结合律

矩阵乘法满足结合律（associative law），即$A(BC)=(AB)C$
矩阵乘法不满足交换律（commutative law），即$\text{Often }AB\ne BA$

矩阵相乘（列视角）

$$\text{Matrix Multiplication: }AB=A[\overrightarrow{b_1}\overrightarrow{b_2}\overrightarrow{b_3}]=[A\overrightarrow{b_1}A\overrightarrow{b_2}A\overrightarrow{b_3}]$$

The beauty of matrix multiplication is that all three approaches(rows, columns, whole matrices) come out right（其他视角后续小节讲解）

Permutation Matrix（置换矩阵）

Row Exchange Matrix: $P_{ij}$ is the identity matrix with rows $i$ and $j$ reversed. When this “permutation matrix”$P_{ij}$ multiplies a matrix, it exchanges rows $i$ and $j$.

Argumented Matrix（增广矩阵）

We can include $\overrightarrow{b}$ as an extra column of $A$ and follow it through elimination.

$$\text{Argumented Matrix: }[A\overrightarrow{b}]$$

Matrix multiplication works by rows and at the same time by columns:
ROWS: each row of $E$ acts on $[A\overrightarrow{b}]$ to give a row of $[EAE\overrightarrow{b}]$;
COLUMNS: $E$ acts on each column of $[A\overrightarrow{b}]$ to give a column of $[EAE\overrightarrow{b}]$;

2.4 Rules for Matrix Operations

看待矩阵乘法的第一种视角：每一个元素都是一个向量内积

$$\text{The entry in row }i\text{ column }j\text{ of }AB\text{ is (row }i\text{ of }A)\cdot \text{(column }j\text{ of }B)$$

看待矩阵乘法的第二种视角：Each column of AB is a combination of the columns of A

$$\text{matrix A times every column of matrix B: }A[\overrightarrow{b_1}\overrightarrow{b_2}\cdots \overrightarrow{b_p}]=[A\overrightarrow{b_1}A\overrightarrow{b_2}\cdots A\overrightarrow{b_p}]$$

看待矩阵乘法的第三种视角：Each row of AB is a combination of the rows of B

$$\text{every row of A times matrix B: }\left[\begin{array}{c}\overrightarrow{a_1}\\ \overrightarrow{a_2}\\ \cdots \\ \overrightarrow{a_m}\end{array}\right]B=\left[\begin{array}{c}\overrightarrow{a_1}B\\ \overrightarrow{a_2}B\\ \cdots \\ \overrightarrow{a_m}B\end{array}\right]$$

看待矩阵乘法的第四种视角：columns of A multiply rows of B

$$\left[\begin{array}{c}co{l}_{1}co{l}_2\cdots co{l}_n\end{array}\right]\times \left[\begin{array}{c}ro{w}_1\\ ro{w}_2\\ \cdots \\ ro{w}_n\end{array}\right]=co{l}_1\times ro{w}_1+co{l}_2\times ro{w}_2+\cdots +co{l}_n\times ro{w}_n$$

向量的内积和外积(inner product, outer product)

a row times a column is an inner product, or dot product(produce a single number), while a column times a row is an outer product(produce a matrix).

Laws for Matrix Operations

• 矩阵加法满足交换律（commutative law）：$A+B=B+A$

• 矩阵加法满足分配律（distributive law）：$c(A+B)=cA+cB$

• 矩阵加法满足结合律（associative law）：$A+(B+C)=(A+B)+C$

• 矩阵乘法不满足交换律（the commutative law is usually broken）：$AB\ne BA$

• 矩阵乘法满足左分配律（distributive law from left）：$A(B+C)=AB+AC$

• 矩阵乘法满足右分配律（distributive law from right）：$(A+B)C=AC+BC$

• 矩阵乘法满足结合律（associative law）:$A(BC)=(AB)C$

Matrix Power

$$(A^p)(A^q)=A^{p+q},(A^p{)}^q=A^{pq}$$

矩阵分块及分块相乘

if blocks of A can multiply blocks of B, then block multiplication of AB is allowed. Cuts between columns of A match cuts of rows of B.

$$\left[\begin{array}{c}{A}_{11}{A}_{12}\\ A_{21}{A}_{22}\end{array}\right]\left[\begin{array}{c}{B}_{11}\\ B_{21}\end{array}\right]=\left[\begin{array}{c}{A}_{11}{B}_{11}+A_{12}{B}_{21}\\ A_{21}{B}_{11}+A_{22}{B}_{21}\end{array}\right]$$

Block Elimination

$$\left[\begin{array}{cc}I& \overrightarrow{0}\\ -C{A}^{-1}& I\end{array}\right]\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]=\left[\begin{array}{cc}A& B\\ \overrightarrow{0}& D-C{A}^{-1}B\end{array}\right]$$

the entry “$D-C{A}^{-1}B$” is called Schur complement, 舒尔补

2.5 Inverse Matrices

Six Notes About $A^{-1}$

1. the inverse exists if and onlt if the elimination produces n nonzero pivots(row exchange is allowed).
2. the matrix A cannot have two different inverses.
3. if A is invertible, the only solution to $A\overrightarrow{x}=\overrightarrow{b}$ is $\overrightarrow{x}=A^{-1}\overrightarrow{b}$
4. if there exists a nonzero vector $\overrightarrow{x}$ such that $A\overrightarrow{x}=\overrightarrow{0}$, A is not invertible.
5. for a $2\times 2$ matrix, if is invertible onlt if $ad-bc\ne 0$
   $${\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]}^{-1}=\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]$$
6. a diagonal matrix has an inverse provided that no diagonal entries are zero.

Inverse of Matrix Product

if A and B are invertible, then:
$$(AB{)}^{-1}=B^{-1}{A}^{-1}$$

note that the order is reversed.

this works with three or more matrices:
$$(ABC{)}^{-1}=C^{-1}{B}^{-1}{A}^{-1}$$

Gauss-Jordan Elimination

$$A{A}^{-1}=A\left[\begin{array}{ccc}\overrightarrow{x_1}& \overrightarrow{x_2}& \overrightarrow{x_3}\end{array}\right]=\left[\begin{array}{ccc}\overrightarrow{e_1}& \overrightarrow{e_2}& \overrightarrow{e_3}\end{array}\right]=I$$

对于一个$n\times n$的矩阵来说，求解其逆矩阵相当于求解n个n元方程组，我们可以同时列出包含了这n个方程组的增广矩阵：

$$\left[\begin{array}{cccccc}2& -1& 0& 1& 0& 0\\ -1& 2& -1& 0& 1& 0\\ 0& -1& 2& 0& 0& 1\end{array}\right]$$

先进行一轮forward elimination：

$$\left[\begin{array}{cccccc}2& -1& 0& 1& 0& 0\\ 0& \frac{3}{2}& -1& \frac{1}{2}& 1& 0\\ 0& 0& \frac{4}{3}& \frac{1}{3}& \frac{2}{3}& 0\end{array}\right]$$

然后反过来向上消元，将矩阵变为reduced echelon form（最简行阶梯矩阵），这部分由Jordan贡献。

$$\left[\begin{array}{cccccc}2& 0& 0& \frac{3}{2}& 1& \frac{1}{2}\\ 0& \frac{3}{2}& 0& \frac{3}{4}& \frac{3}{2}& \frac{3}{4}\\ 0& 0& \frac{4}{3}& \frac{1}{3}& \frac{2}{3}& 0\end{array}\right]$$

最后，divided each row by its pivots, then we finally get $[I{A}^{-1}]$ in this argumented matrix;

$$\left[\begin{array}{cccccc}1& 0& 0& \frac{3}{4}& \frac{1}{2}& \frac{1}{4}\\ 0& 1& 0& \frac{1}{2}& 1& \frac{1}{2}\\ 0& 0& 1& \frac{1}{4}& \frac{1}{2}& \frac{3}{4}\end{array}\right]$$

the total cost for $A^{-1}$ is $n^3$ multiplications and subtractions.

Diagonally dominant matrices are intertible

对角优势矩阵是可逆的（充分不必要）；
对角优势矩阵，对角线上的每一个元素，都大于这一行上其余元素的绝对值的和；
对角优势矩阵之所以可逆，是因为其一定满足$A\overrightarrow{x}=\overrightarrow{0}$没有非零向量解；

2.6 Elimination = Factorization: $A=LU$

矩阵的LU分解

$$A=LU$$

或者

$$A=LDU$$

其中，L代表下三角矩阵（lower triangular），储存了高斯消元时所有的$l_{ij}$；
D代表对角矩阵（diagonal matrix），储存了所有的pivots（如果愿意写成这种形式的话）；
U代表上三角矩阵（upper triangular），储存了A高斯消元后的结果（LU形式）或者将高斯消元后的矩阵每一行除以其pivot的结果（LDU形式）；

从LU分解的角度看待线性方程组的求解

方程组$A\overrightarrow{x}=\overrightarrow{b}$，如果写成$LU\overrightarrow{x}=\overrightarrow{b}$，相当于解决两个三角形的方程组，先解决$L\overrightarrow{c}=\overrightarrow{b}$（forward substitution），再解决$U\overrightarrow{x}=\overrightarrow{c}$（backward substitution）。

The Cost of Elimination

Elimination on a $n\times n$ square matrix A needs $\frac{1}{3}n(n+\frac{1}{2})(n+1)$ multiplications and $\frac{1}{3}n(n+\frac{1}{2})(n+1)$ subtractions.

To solve each right side of the equation, $n^2$ muliplications and $n^2$ substractions are needed.

对于带宽为$\omega$的带状矩阵（band matrix）来说，消元只需要消耗$n{\omega }^2$次乘法和减法，求解只需要消耗$2n\omega$次乘法和减法。

2.7 Transposes and Permutations

转置矩阵（Transpose）的一些性质

$$\begin{gathered} \text{Sum: }(A+B{)}^T=A^T+B^T \\ \text{Product: }(AB{)}^T=B^T{A}^T \\ \text{Inverse: }(A^{-1}{)}^T=(A^T{)}^{-1} \\ (ABC{)}^T=C^T{B}^T{A}^T \\ \text{if A is invertible, }{A}^T\text{is invertible, too} \end{gathered}$$

使用矩阵转置表示向量的内积和外积

$$\begin{gathered} \text{Inner Product: }{\overrightarrow{x}}^T\overrightarrow{y}\text{ , T is inside} \\ \text{Outer Product: }\overrightarrow{x}{\overrightarrow{y}}^T\text{ , T is outside} \end{gathered}$$

关于转置矩阵更加“数学”的定义

$A^T$ is the matrix that makes the following two inner product equal:
$$(A\overrightarrow{x}{)}^T\overrightarrow{y}={\overrightarrow{x}}^T(A^T\overrightarrow{y})$$
because $(A\overrightarrow{x}{)}^T\overrightarrow{y}={\overrightarrow{x}}^T{A}^T\overrightarrow{y}$, 使用转置矩阵表示的内积。

斜对称矩阵（Symmetric Matrix）

斜对称矩阵的转置等于本身$P^T=P$；

对于任意矩阵，$A{A}^T$的结果是一个斜对称矩阵；

对斜对称矩阵进行高斯消元，计算量可以减半，因为：$P=LDU=LD{L}^T$；

置换矩阵（Permutation Matrix）

DEFINITION: A Permutation Matrix has the rows of an Indetity Matrix I in any order.

$P^{-1}$也是一个置换矩阵。

$P^{-1}=P^T$，置换矩阵的逆矩阵与转置矩阵相同。

带有行交换的矩阵的LU分解

两种形式：
$$\begin{gathered} PA=LU \\ A=LPU \end{gathered}$$
第一种形式更加常用；

课后习题

作业里的截图都缺了一角，问题应该出在使用CloudConvert将SVG图像转化成PNG图像的步骤上，先不管。

2.1

2.2

2.3

2.4

2.5

2.6

2.7