Chapter 2 Solving Linear Equations (Introduction to Linear Algebar written by Dr. Gilber Strang)

2.1 Vectors and Linear Equations

The central problem of linear algebra is to solve a system of linear equations which means that the unknown are only multiplied by numbers–we never see $x$ times $y$.

1. The column picture of $Ax=b:$ a combination of $n$ columns of A produces the vector $b$.
2. The row picture of $Ax=b:$ $m$ equations from $m$ rows give $m$ planes meeting at $x$.

$$\left[\begin{array}{cc}1& -2\\ 3& 2\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}1\\ 11\end{array}\right]$$
The column picture:
$$x\left[\begin{array}{c}1\\ 3\end{array}\right]+y\left[\begin{array}{c}-2\\ 2\end{array}\right]=\left[\begin{array}{c}1\\ 11\end{array}\right]$$
The row picture:
$$\begin{gathered} x-2y=1 \\ 3x+2y=11 \end{gathered}$$

The column picture is done in numpy, when you create new matrix.
np.linalg.inv(A) calculate the inverse of matrix $A$.

>>> import numpy as np
>>> A = np.array([[1,3],[-2,2]])# The column picture
>>> b = np.array([1,11])
>>> b.dot(np.linalg.inv(A))
array([3., 1.])
>>> np.dot(b,np.linalg.inv(A))
array([3., 1.])

Three Equations in Three Unkonwns

$Ax=b$
$$\begin{gathered} x+2y+3z=6 \\ 2x+5y+2z=4 \\ 6x-3y+z=2 \end{gathered}$$

The Row Picture: Two intersected planes make one line which is intersected with the third plane. This process can do the result.
The Column Picture:
$$x\left[\begin{array}{c}1\\ 2\\ 6\end{array}\right]+y\left[\begin{array}{c}2\\ 5\\ -3\end{array}\right]+z\left[\begin{array}{c}3\\ 2\\ 1\end{array}\right]=\left[\begin{array}{c}6\\ 4\\ 2\end{array}\right]$$

The result should be:
$$\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=\left[\begin{array}{c}0\\ 0\\ 2\end{array}\right]$$

>>> import numpy as np
>>> A = np.array([[1,2,6],[2,5,-3],[3,2,1]])
>>> b = np.array([6,4,2])
>>> b.dot(np.linalg.inv(A))
array([0.00000000e+00, 2.77555756e-17, 2.00000000e+00])

The matrix form merge the row picture and column picture.
$$\left[\begin{array}{ccc}1& 2& 3\\ 2& 5& 2\\ 6& -3& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=\left[\begin{array}{c}6\\ 4\\ 2\end{array}\right]$$

identity matrix $\mathbf{I}$: $$\mathbf{I}\mathbf{x}=\mathbf{x}$$
Whatever vector identity matrix multiplies, that vector is not changed.
$$\mathbf{I}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$$

Except for the Identity Matrix, the following matrix are all interesting.

1. Exchange matrix
   $$\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=x\left[\begin{array}{c}0\\ 1\end{array}\right]+y\left[\begin{array}{c}1\\ 0\end{array}\right]=\left[\begin{array}{c}x\ast 0+y\ast 1\\ x\ast 1+y\ast 0\end{array}\right]=\left[\begin{array}{c}y\\ x\end{array}\right]$$
2. Rotate every vector by 90°
   $$\left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=x\left[\begin{array}{c}0\\ -1\end{array}\right]+y\left[\begin{array}{c}1\\ 0\end{array}\right]=\left[\begin{array}{c}x\ast 0+y\ast 1\\ x\ast (-1)+y\ast 0\end{array}\right]=\left[\begin{array}{c}y\\ -x\end{array}\right]$$
3. Rotate every vector by 180°
   $$\left[\begin{array}{cc}0& -1\\ -1& 0\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=x\left[\begin{array}{c}0\\ -1\end{array}\right]+y\left[\begin{array}{c}-1\\ 0\end{array}\right]=\left[\begin{array}{c}x\ast 0+y\ast (-1)\\ x\ast (-1)+y\ast 0\end{array}\right]=\left[\begin{array}{c}-y\\ -x\end{array}\right]$$
4. Rotate every vector through 45°
   $$\left[\begin{array}{cc}\frac{\sqrt{(}2)}{2}& -\frac{\sqrt{(}2)}{2}\\ \frac{\sqrt{(}2)}{2}& \frac{\sqrt{(}2)}{2}\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=x\left[\begin{array}{c}\frac{\sqrt{(}2)}{2}\\ \frac{\sqrt{(}2)}{2}\end{array}\right]+y\left[\begin{array}{c}-\frac{\sqrt{(}2)}{2}\\ \frac{\sqrt{(}2)}{2}\end{array}\right]=\left[\begin{array}{c}x\frac{\sqrt{(}2)}{2}-y\frac{\sqrt{(}2)}{2}\\ x\frac{\sqrt{(}2)}{2}+y\frac{\sqrt{(}2)}{2}\end{array}\right]$$
   All these problems are solved by the column picture.

Row Picture: Each equation in $Ax=b$ gives a line(n = 2) or a plane(n = 3) or a “hyperplane”(n > 3). They intersect at the solution or solutions, if any.

2.2 The Idea of Elimination

The corner entry $a_{11}$ is the first “pivot” and the ratio $a_{21}/a_{11}$ is the first “multiplier.”

Elimination breaks down if zero appears in the pivot. Exchanging two equations may save it.

2.4 Rules for Matrix Operations

$$AB=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\left[\begin{array}{cc}E& F\\ G& H\end{array}\right]=\left[\begin{array}{c}a\\ c\end{array}\right]\left[\begin{array}{cc}E& F\end{array}\right]+\left[\begin{array}{c}b\\ d\end{array}\right]\left[\begin{array}{cc}G& H\end{array}\right]=\left[\begin{array}{cc}aE& aF\\ cE& cF\end{array}\right]+\left[\begin{array}{cc}bG& bH\\ dG& dH\end{array}\right]=\left[\begin{array}{cc}aE+bG& aF+bH\\ cE+dG& cF+dH\end{array}\right]$$

2.5 Inverse Matrices

Calculating $A^{-1}$ by Gauss-Jordan Eliminations

$$A=\left[\begin{array}{ccc}2& -1& 0\\ -1& 2& -1\\ 0& -1& 2\end{array}\right]$$

$$AI=\left[\begin{array}{cccccc}2& -1& 0& 1& 0& 0\\ -1& 2& -1& 0& 1& 0\\ 0& -1& 2& 0& 0& 1\end{array}\right]\to elimination$$
$$\to \left[\begin{array}{cccccc}1& 0& 0& \frac{3}{4}& \frac{1}{2}& \frac{1}{4}\\ 0& 1& 1& \frac{1}{2}& 1& \frac{1}{1}\\ 0& 0& 1& \frac{1}{4}& \frac{1}{2}& \frac{3}{4}\end{array}\right]$$

>>> import numpy as np
>>> import sympy as sm
>>> I = np.identity(3, dtype = int)
>>> A = [[2,-1,0], [-1,2,-1],[0,-1,2]]
>>> aug_matrix = sm.Matrix(np.concatenate((A, I), axis=1))
>>> aug_matrix
Matrix([
[ 2, -1,  0, 1, 0, 0],
[-1,  2, -1, 0, 1, 0],
[ 0, -1,  2, 0, 0, 1]])
>>> # Elimination on the augmented matrix [A, I] to the
>>> # reduced echelon form阶梯形矩阵
>>> R, rref_pivots = aug_matrix.rref()
>>> # Pick X = inverse(A) fromthe last n columns of R
>>> R[:,3:]
Matrix([
[3/4, 1/2, 1/4],
[1/2,   1, 1/2],
[1/4, 1/2, 3/4]])

Diagonally dominant matrices are invertible. The diagonally dominant matrices are needed to be following the formula.
$$|a_{ij}|>\sum _{j\ne i}|a_{ij}|$$
like the following matrix:
$$\left[\begin{array}{ccc}3& 1& 1\\ 1& 3& 1\\ 1& 1& 3\end{array}\right]$$
Some matrices which are not diagonally dominant matrices are also invertable . like the following matrix:
$$\left[\begin{array}{ccc}2& 1& 1\\ 1& 2& 1\\ 1& 1& 3\end{array}\right]$$

The inverse of a triangular difference matrix $A$ is a triangular sum matrix $S$.
$$AI=\left[\begin{array}{cccccc}1& 0& 0& 1& 0& 0\\ -1& 1& 0& 0& 1& 0\\ 0& -1& 1& 0& 0& 1\end{array}\right]\to \left[\begin{array}{cccccc}1& 0& 0& 1& 0& 0\\ 0& 1& 0& 1& 1& 0\\ 0& 0& 1& 1& 1& 1\end{array}\right]=[I{A}^{-1}]$$
$A$ is difference matrix. $A^{-1}$ is sum matrix.

2.6 Elimination $=$ Factorization

>>> from scipy.linalg import pascal, cholesky
>>> # create 4*4 pascal matrix
>>> A = pascal(4)
>>> # Cholesky factorization
>>> L = cholesky(A, lower=True)
>>> U = cholesky(A, lower = False)
>>> # check the the factorization of A, A = LU
>>> L.dot(U) == A
array([[ True,  True,  True,  True],
       [ True,  True,  True,  True],
       [ True,  True,  True,  True],
       [ True,  True,  True,  True]])
>>> A
array([[ 1,  1,  1,  1],
       [ 1,  2,  3,  4],
       [ 1,  3,  6, 10],
       [ 1,  4, 10, 20]], dtype=uint64)
>>> L
array([[1., 0., 0., 0.],
       [1., 1., 0., 0.],
       [1., 2., 1., 0.],
       [1., 3., 3., 1.]])
>>> U
array([[1., 1., 1., 1.],
       [0., 1., 2., 3.],
       [0., 0., 1., 3.],
       [0., 0., 0., 1.]])

2.7 Transpose and Permutations

$$(AB{)}^T=B^T{A}^T$$

$$(A^{-1}{)}^T=(A^T{)}^{-1}$$

A symmetric matrix has $S^T=S$. This means that $S_{ji}=S_{ij}$

For permutation matrix $P$, $P^{-1}=P^T$

How to find the permutation matrix $P$ in the formula $PA=LU$?
Look down the column for the largest pivot.

>>> from scipy.linalg import lu
>>> import numpy as np
>>> A = np.array([[0,1,1], [1,2,1], [2,7,9]])
>>> P, L, U = lu(A)
>>> A
array([[0, 1, 1],
       [1, 2, 1],
       [2, 7, 9]])
>>> P.dot(L.dot(U))
array([[0., 1., 1.],
       [1., 2., 1.],
       [2., 7., 9.]])

至此本书内容最多的一章啃完了！！！