线性代数笔记（网易公开课）

Linear Algebra Handnote(1)

• If $L$ is lower triangular with 1’s on the diagonal, so is $L^{-1}$

• Elimination = Facotization: $A=LU$

• $A^T$ is the matrix that makes these two inner products equal for every $x$ and $y$:
  

  $$(Ax)^Ty=x^T(A^Ty)$$
  Inner product of $Ax$ with $y$ = Inner product of $x$ with $A^Ty$

• DEFINITION: The space $R^n$ consists of all column vectors $v$ with $n$ components

• DEFINITION: A subspace of a vector space is a set of vectors (including 0) that satisfies two requirements: (1) $v+w$ is in the subspace, (2) $cv$ is in the subspace

• The colomn space consists of all linear combinations of the columns. The combinations are all possible vectors $Ax$. They fill the column space $C(A)$

  The system $Ax=b$ is solvable if and only if $b$ is in the column space of $A$

• The nullspace of A consists of all solutions to $Ax=0$. These vectors $x$ are in $R^n$. The nullspace containing all solutions of $Ax=0$ is denoted by $N(A)$

• the nullspace is a subspace of $R^n$, the column space is a subspace of $R^m$
• the nullspace consists of all combinations of the special solutions

• Nullspace(plane) perpendicular to row space(line)

• $Ax=0$ has $r$ pivots and $n-r$ free variables: $n$ columns minus $r$ pivot columns. The nullspace matrix $N$ (contains all special solutions) contains the $n-r$ special solutions. Then $AN=0$

• $Ax=0$ has $r$ independent equations so it has $n-r$ independent solutions.

• $x_{particular}$: the particular solution solves $Ax_p=b$

• $x_{nullspace}$: the $n-r$ special solutions solve $Ax_n=0$

• Complete solution: one $x_p$, many $x_n$: $x=x_p+x_n$

• The four possibilities for linear equations depend on the rank $r$:

  • $r=m$, and $r=n$: Square ane invertible, $Ax=b$ has 1 solution

  • $r=m$, and $r<n$: Short and wide, $Ax=b$ has $\infty$ solutions
  • $r<m$, and $r=n$: Tall and thin, $Ax=b$ has 0 or 1 solutions
  • $r<m$, and $r<n$: Not full rank, $Ax=b$ has 0 or $\infty$ solutions

  • Independent vections (no extra vectors)

  • Spanning a space (enough vectors to produce the rest)
  • Basis for a space (not too many or too few)

  • Dimension of a space (the number of vectors in a basis)

  • Any set of $n$ vectors in $R^m$ must be linearly dependent if $n>m$

  • The columns spans the column space. The rows span the row space

    • The column space / row space of a matrix is the subspace of $R^m$/$R^n$ spanned by the columns/rows.

  • A basis for a vector space is a sequence of vectors with two properties: linear independent and span the space.

    • The basis is not unique. But the combination that produces the vector is unique.
    • The columns of a $n \times n$ invertible matrix are a basis for $R^n$.
    • The pivot columns of A are a basis for its column space.

  • DEFINITION: The dimension of a space is the number of vectors in every basis.

  • The space $Z$ that contains only the zero vector. The dimension of this space is zero. The empty set (containing no vectors) is a basis for Z. We can never allow the zero vector into a basis, because then linear independence is lost.

    Four Fundamental Subspaces
    1. The row space $C(A^T)$, a subspace of $R^n$
    2. The column space $C(A)$, a subspace of $R^m$
    3. The nullspace is $N(A)$, a subspace of $R^n$
    4. The left nullspace $N(A^T)$, a subspace of $R^m$

    1. $A$ has the same row space as $R$. Same dimension $r$ and same basis.
    2. The column space of $A$ has dimension $r$. The number of independent columns equals the number of independent rows.
    3. $A$ has the same nullspace as $R$. Same dimension $n-r$ and same basis.
    4. The left nullspace of $A$ (the nullspace of $A^T$ has dimension $m-r$.

    Fundamental Theorem of Linear Algebra, Part 1

    • The column space and row space both have dimension $r$.

    • The nullspaces have dimensions $n-r$ and $m-r$.

    • Every rank one matrix has the special form $A=uv^T=column \times row.$

    • The nullspace $N(A)$ and the row space $C(A^T)$ are orthogonal subspaces of $R^n$.

    • DEFINITION: The orthogonal complement of a subspace $V$ contains every vector that is perpendicular to $V$.

    Fundamental Theorem of Linear Algebra, Part 2
    * $N(A)$ is the orthogonal complement of the row space $C(A^T)$ (in $R^n$)
    * $N(A^T)$ is the orthogonal complement of the column space $C(A)$ (in $R^m$)

    Projection Onto a Line
    * The projection matrix $P=\frac{aa^T}{a^Ta}$ onto the line through $a$
    * The projection $p=\bar{x}a=\frac{a^Tb}{a^Ta}a$

    Projection Onto a Subspace

    Problem: Find the combination $p=\bar{x_1}a_1+\cdots+\bar{x_n}a_n$ closest to a given vector $b$. The $n$ vectors $a_1, \cdots, a_n$ in $R_m$ span the column space of $A$. Thus the problem is to find the particular combination $p=A\bar{x}$(the projection) that is closest to $b$. When $n=1$, the best choice is $\frac{a^Tb}{a^Ta}$

    • $A^T(b-A\bar{x})=0$, or $A^TA\bar{x}=A^Tb$

    • The symmetric matrix $A^TA$ is $n\times n$. It is inverible if the $a$’s are independent.

    • The solution is $\bar{x}=(A^TA)^{-1}A^Tb$

    • The projection of $b$ onto the subspace $p=A\bar{x}=A(A^TA)^{-1}A^Tb$
    • The projection matrix $P=A(A^TA)^{-1}A^T$

    * $A^TA$ is invertible if and only if $A$ has linearly independent columns*

    Least Squares Approximations

    • When $Ax=b$ has no solution, multiply by $A^T$ and solve $A^TA\bar{x}=A^Tb$

      • The least squares solution $\bar{x}$ minimizes $E=||Ax-b||^2$. This is the sum of squares of the errors in the $m$ equations ($m>n$)

      • The best $\bar{x}$ comes from the normal equations $A^TA\bar{x}=A^Tb$

      • Orthogonal Bases and Gram-Schmidt

        • orthonormal vectors

          • A matrix with orthonormal columns is assigned the special letter $Q$. The matrix $Q$ is easy to work with because $Q^TQ=I$
          • When $Q$ is square, $Q^TQ=I$ means that $Q^T=Q^{-1}$: transpose = inverse.
          • If the columns are only orthogonal (not unit vectors), dot products give a diagonal matrix (not the identity matrix)

        • Every permutation matrix is an orthogonal matrix.

        • If $Q$ has orthonormal columns ($Q^TQ=I$), it leaves lengths unchanged

        • Orthogonal is good

        • Use Gram-Schmidt for the Factorization $A=QR$

        $\left[ {\begin{array}{*{20}{c}} {\rm{a}}&b&c \end{array}} \right] = \left[ {\begin{array}{*{20}{c}} {{q_1}}&{{q_2}}&{{q_3}} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {q_1^Ta}&{q_1^Tb}&{q_1^Tc}\\ {}&{q_2^Tb}&{q_2^Tc}\\ {}&{}&{q_3^Tc} \end{array}} \right]$

        • (Gram-Schmidt) From independent vectors $a_1, \cdots, a_n$, Gram-Schmidt constructs orthonormal vectors $q_1, \cdots, q_n$. The matrces with these columns satisfy $A=QR$. Then $R=Q^TA$ is upper triangular because later $q$’s are orthogonal to earlier $a$’s.

        • Least squares: $R^TR\bar{x}=R^TQ^Tb$ or $R\bar{x}=Q^Tb$ or $\bar{x}=R^{-1}Q^Tb$

        Determinants

        • The determinant is zero when the matrix has no inverse
        • The product of the pivots is the determinant
        • The determinant changes sign when two rows (or two columns) are exchanged
        • Determinants give $A^{-1}$ and $A^{-1}b$ (this formulat is called Cramer’s Rule)
        • When the edge of a box are the rows of $A$, the volume is $|det A|$
        • For $n$ special numbers $\lambda$, called eigenvalues, the determinants of $A-\lambda I$ is zero.

        The properties of the determinant

        1. The determinant of the $n \times n$ identity matrix is 1.
        2. The determinant changes sign when two rows are exchanged
        3. The determinant is a linear function of each row separately (all other rows stay fixed!)
        4. If two rows of $A$ are equal, then $det A = 0$
        5. Subtracting a multiple of one row from another row leave $det A$ unchanged.
           
           • $\left| {\begin{array}{*{20}{c}} {\rm{a}}&b\ {c - la}&{d - lb} \end{array}} \right| = \left| {\begin{array}{*{20}{c}} a&b\\ c&d \end{array}} \right|$
        6. A matrix with a row of zeros has $det A=0$
        7. If $A$ is triangular then $det A=a_{11}a_{22}\cdots a_{nn}=product of diagonal entries$
        8. If $A$ is singular then $det A=0$. If $A$ is invertible then $det A \neq 0$
           
           • Elimination goes from $A$ to $U$.
           • $det A = +-det U = +-(product of the pivots)$
        9. The determinant of $AB$ is $det A times det B$
        10. The transpose $A^T$ has the same determinant as $A$

        Every rule of the rows can apply to columns*

        Cramer’s Rule

        • If $det A$ is not zero, $Ax=b$ is solved by determinants:
          
          • $x_1=\frac{det B_1}{det A}, x_2=\frac{det B_2}{det A}, \cdots, x_n=\frac{det B_n}{det A}$
          • The matrix $B_j$ has the $j$th column of $A$ replaced by the vector $b$

        Cross Product

        • $||u \times v|| = ||u||||v|||\sin \theta|$

        • $|u \cdot v| = ||u||||v|||\cos \theta|$

        • The length of $u \times v$ equals the area of the parallelogram with sides $u$ and $v$

        • It points by the right hand rule (points along your right thumb when the fingers curl from $u$ to $v$

        Eigenvalues and Eigenvectors

        • The basic equation is $Ax=\lambda x$, The number $\lambda$ is an eigenvalue of $A$

          • When $A$ is squared, the eigenvectors stay the same. The eigenvalues are squared.

        • The projection matrix has eigenvalues $\lambda = 1$ and $\lambda = 0$

          • $P$ is singular, so $\lambda=0$ is an eigenvalue
          • Each column of $P$ adds to 1, so $\lambda=1$ is an eigenvalue
          • $P$ is symmetric, so its eigenvectors are perpendicular
        • Permutations have all $|\lambda|=1$

        • The reflection matrix has eigenvalues 1 and -1

        • Solve the eigenvalue problem for an $n \times n$ matrix

          • Compute the determinant of $A-\lambda I$. It is a polynomial in $\lambda$ of degree $n$
          • Find the roots of this polynomial
          • For each eigenvalue $\lambda$, solve $(A-\lambda I)x=0$ to find an eigenvector $x$

        • Bad news: elimination does not preserve the $\lambda$’s

        • Good news: the product of eigenvalues equals the determinant, the sum of the eigenvalues equals the sum of the diagonal entries (trace)

        Diagonalizing a Matrix

        • Suppose the $n \times n$ matrix $A$ has $n$ linearly independent eigenvectors $x_1, \cdots, x_n$. Put them into the columns of an eigenvector matrix $S$. Then $S^{-1}AS$ is the eigenvalue matrix $\Lambda$:

          • ${S^{ - 1}}AS = \Lambda = \left[ {\begin{array}{*{20}{c}} {{\lambda _1}}&{}&{}\ {}& \ddots &{}\ {}&{}&{{\lambda _n}} \end{array}} \right]$

        • There is no connection between invertibility and diagonalizability:

          • Invertibility is concerned with the eigenvalues ($\lambda=0$ or $\lambda \neq 0$)
          • Diagonalizability is concerned with the eigenvectors (too few or enough for $S$)

        Applications to differential equations

        • One equation $\frac{du}{dt}=\lambda u$ has the solution $u(t)=Ce^{\lambda t}$

        • $n$ equations $\frac{d\textbf{u}}{dt}=A\textbf{u}$ starting from the vector $\textbf{u}(0)$ at $t=0$

        • Solve linear constant coefficient equations by exponentials $e^{\lambda t}\textbf{x}$, when $A\textbf{x}=\lambda \textbf{x}$

        Symmetric Matrices

        • A symmetric matrix has only real eigenvalues.

        • The eigenvectors can be chosen orthonormal.

        • (Spectral Theorem) Every symmetric matrix haas the facorization $A=Q\Lambda Q^T$ with real eigenvalues in $\Lambda$ and orthonormal eigenvectors in $S=Q$:

          • Symmetric diagonalization: $A=Q\Lambda Q^{-1}=Q \Lambda Q^T$ with $Q^{-1}=Q^T$

        • （Orthogonal Eigenvectors） Eigenvectors of a real symmetric matrix (when they correspond to different $\lambda$’s) are always perpendicular.

        • product of pivots = determinant = product of eigenvalues

        • Eigenvalues VS. Pivots

        • For symmetric matrices the pivots and the eigenvalues have the same signs:

          • The number of positive eigenvalues of $A=A^T$ equals the number of positive pivots.

        • All symmetric matrices are diagonalizable

        Positive Definite Matrices

        • Symmetric matrices that have positive eigenvalues

        • 2 $\times$ 2 matrices

          • The eigenvalues of $A$ are positive if and only if $a>0$ and $ac-b^2>0$.

        • $x^TAx$ is positive for all nonzero vectors $x$

          • If $A$ and $B$ are symmetric positive definite, so is $A+B$

        • When a symmetric matrix has one of these five properties, it has them all:

          • All $n$ pivots are positive
          • All $n$ upper left determinants are positive
          • All $n$ eigenvalues are positive
          • $x^TAx$ is positive except at $x=0$. This is the energy-based definition
          • $A$ equals $R^TR$ for a matrix $R$ with independent columns

        Positive Semidefinite Matrices

        Similar Matrices

        • DEFINITION: Let $M$ be any invertible matrix. Then $B=M^{-1}AM$ is similar to $A$

        • (No change in $\lambda$’s) Similar matrices $A$ and $M^{-1}AM$ have the same eigenvalues. If $x$ is an eigenvector of $A$, then $M^{-1}x$ is an eigenvector of $B$. But two matrices can have the same repeated $\lambda$, and fail to be similar.

        Jordan Form

        • What is “Jordan Form”?
          
          • For every $A$, we want to choose $M$ so that $M^{-1}AM$ is nearly diagonal as possible

        • $J^T$ is the similar to $J$, the matrix $M$ that produces the similarity happens to be the reverse identity

        • (Jordan form) If $A$ has $s$ independent eigenvectors, it is similar to a matrix $J$ that has $s$ Jordan blocks on its diagonal: Some matrix $M$ puts $A$ into Jordan form.

          • Jordan block: The eigenvalue is on the diagonal with $1$’s just above it. Each block in $J$ has one eigenvalue $\lambda_i$, one eigenvector. and 1’s above the diagonal

        ${{\rm{M}}^{ - 1}}AM = \left[ {\begin{array}{*{20}{c}} {{J_1}}&{}&{}\ {}& \ddots &{}\ {}&{}&{{J_s}} \end{array}} \right] = J$

        ${J_i} = \left[ {\begin{array}{*{20}{c}} {{\lambda _i}}&1&{}&{}\ {}& \ddots &1&{}\ {}&{}& \ddots &1\ {}&{}&{}&{{\lambda _i}} \end{array}} \right]$

        • $A$ is similar to $B$ if they share the same Jordan form $J$ – not otherwise

        Singular Value Decomposition (SVD)

        • Two sets of singular vectors, $u$’s and $v$’s. The $u$’s are eigenvectors of $AA^T$ and the $v$’s are eigenvectors of $A^TA$.

        • • The singular vectos $v_1, \cdots, v_r$ are in the row space of $A$. The outputs $u_1, \cdots, u_r$ are in the column space of $A$. The singular values $\sigma_1, \cdots, \sigma_r$ are all positive numbers, the equatinos $Av_i=\sigma_i u_i$ tell us:

        $A[\begin{array}{*{20}{c}} {{v_1}}& \cdots &{{v_r}} \end{array}] = \left[ {\begin{array}{*{20}{c}} {{u_1}}& \cdots &{{u_r}} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{\sigma _1}}&{}&{}\\ {}& \ddots &{}\\ {}&{}&{{\sigma _r}} \end{array}} \right]$

        • We need $n-r$ more $v$’s and $m-r$ more $u$’s, from the nullspace $N(A)$ and the left nullspace $N(A^T)$. They can be orthonormal bases for those two nullspaces. Include all the $v$’s and $u$’s in $V$ and $U$, so these matrices become square.

        $A[\begin{array}{*{20}{c}} {{v_1}}& \cdots &{{v_r}} \end{array} \cdots {v_n}] = \left[ {\begin{array}{*{20}{c}} {{u_1}}& \cdots &{{u_r} \cdots u{}_m} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{\sigma _1}}&{}&{}&{}\\ {}& \ddots &{}&{}\\ {}&{}&{{\sigma _r}}&{}\\ {}&{}&{}&{} \end{array}} \right]$

        $V$ is now a square orthogonal matrix, with $V^{-1}=V^T$. So $AV=U\Sigma$ can become $A=U\Sigma V^T$. This is the Singular Value Decomposition:
        

        $$A=U \Sigma V^T=u_1 \sigma_1 v_1^T+\cdots +u_r \sigma_r v_r^T$$

        The orthonormal columns of $U$ and $V$ are eigenvectors of $AA^T$ and $A^TA$