Matrix Theory（矩阵理论）

Matrix Theory（矩阵理论）

（未完待续 2021-09-28）

1. Intro. and Basics

---

1.1. Matrix Definition

• An $m\times n$ matrix is a rectangular array of numbers (or other mathematical objects) with m rows and n columns.
• It is usually denoted as:
  $$A=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right)$$
• Meanwhile, the matrix element of A in the $i$th row and $j$th column is denoted as $a_{ij}$.

1.2. Basic Operations and Calculations

1.2.1. Addition

• Can only be added for the same dimensional matrices:
  $$\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)+\left(\begin{array}{cc}e& f\\ g& h\end{array}\right)=\left(\begin{array}{cc}a+e& b+f\\ c+g& d+h\end{array}\right)$$

1.2.2. Scaling

$$k\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)=\left(\begin{array}{cc}ka& kb\\ kc& kd\end{array}\right)$$

1.2.3. Matrix Multiplication

• Other than the scaling, two matrices can be multiplied only if the number of columns of the left matrix equals the number of rows of the right matrix.

• For two $2\times 2$ matrices:
  $$\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)\left(\begin{array}{cc}e& f\\ g& h\end{array}\right)=\left(\begin{array}{cc}ae+bg& af+bh\\ ce+dg& cf+dh\end{array}\right)$$

• For more general cases: if A is an $m\times n$ matrix while B is an $n\times p$ matrix. Then C = AB is an $m\times p$ matrix. Its $ij$ element can be written as:
  $$c_{ij}=\sum _{k=1}^n{a}_{ik}{b}_{kj}.$$

📣 Hint:
$$[A(BC){]}_{ij}=\sum _{k=1}^n{a}_{ik}[BC{]}_{kj}=\sum _{k=1}^n\sum _{l=1}^p{a}_{ik}{b}_{kl}{c}_{lj}=\sum _{l=1}^p\sum _{k=1}^n{a}_{ik}{b}_{kl}{c}_{lj}=\sum _{l=1}^p[AB{]}_{il}{c}_{lj}=[(AB)C{]}_{ij}$$

1.2.4. Transpose Matrix

• Denoted by $A^T$

• It switches the rows and columns of $A$.

• That is,
  $$\text{if }A=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right),\text{ then }{A}^T=\left(\begin{array}{cccc}{a}_{11}& a_{21}& \cdots & a_{m1}\\ a_{12}& a_{22}& \cdots & a_{m2}\\ ⋮& ⋮& \ddots & ⋮\\ a_{1n}& a_{2n}& \cdots & a_{mn}\end{array}\right)$$

• Or to say: $a_{ij}^T=a_{ji}$.

• Useful properties:

  • $(A^T{)}^T=A$
  • $(A+B{)}^T=A^T+B^T$
  • $(AB{)}^T=B^T{A}^T$

1.2.5. Inner and Outer Products

• Inner Product:
  • Also known as dot product or scalar product.
  • = Matrix product of a row vector times a column vector.
  • For example,
    $$u^{T}v=\left(\begin{array}{ccc}{u}_1& u_2& u_3\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right)=u_1{v}_1+u_2{v}_2+u_3{v}_3$$

✎ Note

• Orthogonal: the inner product of two non-zero vectors is zero $\Rightarrow$ these two vectors are orthogonal.
• Normalized: the norm of a vector = 1.
  $$||u||=(u{u}^T{)}^{\frac{1}{2}}=(u_1^2+u_2^2+u_2^2{)}^{\frac{1}{2}}$$
• Orthonormal: orthogonal + normalized.

• Outer Product:
  • = Matrix product of a column vector times a row vector.
  • For example,
    $$u^{T}v=\left(\begin{array}{c}{u}_1\\ u_2\\ u_3\end{array}\right)\left(\begin{array}{ccc}{v}_1& v_2& v_3\end{array}\right)=\left(\begin{array}{ccc}{u}_1{v}_1& u_1{v}_2& u_1{v}_3\\ u_2{v}_1& u_2{v}_2& u_2{v}_3\\ u_3{v}_1& u_3{v}_2& u_3{v}_3\end{array}\right)$$

📣 Hint!!!

• If $A=\left(\begin{array}{cc}a& d\\ b& e\\ c& f\end{array}\right)$, then $A^{T}A$ is symmetric.
• $Tr(A^{T}A)$ is the sum of the squares of all the elements of $A$.

1.2.6. Inverse Matrix

• Square matrices may have inverses. When a matrix A has an inverse, we say it is invertible.
• Denoted by $A^{-1}$.
• Satisfies: $A{A}^{-1}=A^{-1}A=I$
• Properties:
  • $(AB{)}^{-1}=B^{-1}{A}^{-1}$
  • If A is invertible $\Rightarrow$ $A^T$ is invertible: $(A^T{)}^{-1}=(A^{-1}{)}^T$

📣 Hint: if a matrix is invertible, then its inverse is unique.
$$\begin{gathered} \text{Proof. }\text{Assume there exists }AB=I\text{ and }AC=I\text{ and }B\ne C \\ \text{However, }B=BI=B(AC)=IC=C. \end{gathered}$$

1.3. Special Matrices

1.3.1. Zero Matrix

• Denoted by $0$
• Can be any size and is a matrix consisting of all zero elements.
• Multiplication by a zero matrix results in a zero matrix.
• For example, a $2\times 2$ zero matrix:
  $$0=\left(\begin{array}{cc}0& 0\\ 0& 0\end{array}\right)$$

1.3.2. Identity Matrix

• Denoted by $I$
• $AI=IA=A$, $A$ and $I$ are the same sized square matrices.
• For example, a $2\times 2$ identity matrix:
  $$I=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)$$

1.3.3. Diagonal Matrix

• Nonzero elements only on the diagonal.
• For example, a $2\times 2$ diagonal matrix:
  $$D=\left(\begin{array}{cc}{d}_1& 0\\ 0& d_2\end{array}\right)$$

1.3.4. Banded (Band) Matrix

• Nonzero elements only on diagonal bands.
• For example, a $3\times 3$ diagonal matrix:
  $$B=\left(\begin{array}{ccc}{d}_1& a_1& 0\\ b_1& d_2& a_2\\ 0& b_2& d_3\end{array}\right)$$

1.3.5. Triangular Matrix

• Upper Triangular Matrix:
  $$U=\left(\begin{array}{ccc}a& b& c\\ 0& d& e\\ 0& 0& f\end{array}\right)$$
• Lower Triangular Matrix:
  $$L=\left(\begin{array}{ccc}a& 0& 0\\ b& c& 0\\ d& e& f\end{array}\right)$$

1.3.6. Symmetric Matrices

• $A^T=A$
• For example,
  $$\left(\begin{array}{ccc}a& b& c\\ b& d& e\\ c& e& f\end{array}\right)$$

1.3.7. Skew Symmetric Matrices

• $A^T=-A$
• For example,
  $$\left(\begin{array}{ccc}0& b& c\\ -b& 0& e\\ -c& -e& 0\end{array}\right)$$
• Notice that the diagonal elements of a skew-symmetric matrix must be zero.

📣 Hint!!!

• Any square matrix A can be written as the sum of a symmetric and a skew-symmetric matrix.
  $$\text{Proof. }A=\frac{1}{2}(A+A^T)+\frac{1}{2}(A-A^T)$$
• $A^{T}A$ is symmetric.

1.3.8. Orthogonal Matrices

• Definition:
  • $Q^{-1}=Q^T$. Equivalently, $Q{Q}^T=Q^{T}Q=I$.
  • The columns of Q form an orthonormal set of vectors. The same argument can also be made for the rows of Q.
  • An orthogonal matrix is a matrix that preserves lengths:
    $$||Qx|{|}^2=(Qx{)}^T(Qx)=x^T{Q}^{T}Qx=x^{T}Ix=x^{T}x=||x|{|}^2$$
• Examples of Orthogonal Matrices
  • Rotation Matrices
    A matrix that rotates a vector in space doesn’t change the vector’s length and so should be an orthogonal matrix.

    $$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ z^{\prime }\end{array}\right)=\left(\begin{array}{ccc}cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}x\\ y\\ z\end{array}\right),$$
    it rotates a three-dimensional vector an angle q counterclockwise around the z-axis.

  • Permutation Matxrices
    When multiplying on the left, permutes the rows of a matrix, and when multiplying on the right, permutes the columns.

    $$\left(\begin{array}{ccc}0& 0& 1\\ 1& 0& 0\\ 0& 1& 0\end{array}\right)\left(\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right)=\left(\begin{array}{ccc}g& h& i\\ a& b& c\\ d& e& f\end{array}\right),$$
    it is the row permutaion {3, 1, 2}.

    • A good way to understand and remenber this: $PA=(PI)A$

📣 Hint!!!

• The product of two orthogonal matrices is orthogonal.
• Identity matrix is orthogonal.

参考资料：
[1] Coursera: Matrix Algebra for Engineers