Mathematics for Machine Learning 学习笔记

学习Coursera上Mathematics for Machine Learning Specialization后所做的笔记与整理

第一部分 Linear Algebra 线性代数

1. Vector operations 矢量运算

• $\text{commutative 交换律:}r+s=s+r$
• $2r=r+r$
• $\Vert r{\Vert }^2={\sum }_i{r}_i^2$

1.1 dot or inner product 点积/数量积/内积

点积是一种特殊的内积
$$r\cdot s=\sum _i{r}_i{s}_i$$

• $\text{commutative 交换律:}r\cdot s=s\cdot r$
• $\text{distributive 分配律:}r\cdot (s+t)=r\cdot s+r\cdot t$
• $\text{associative 结合律}r\cdot (as)=a(r\cdot s)$
• $r\cdot r=\Vert r{\Vert }^2$
• $r\cdot s=\Vert r\Vert \Vert s\Vert \cos \theta$

1.2 scalar and vector projection 投影

• scalar projection 投影/标量投影
  例：向量s在向量r上的投影 $\frac{r\cdot s}{\Vert r\Vert }$
• vector projection 矢量投影
  例：向量s在向量r上的投影 $\frac{r\cdot s}{r\cdot r}r$

2. basis 基

A basis is a set of $n$ vectors that:

• are not linear combinations of each other
• span the space
  The Space is then n-dimensional.

在线性空间$V$中，如果存在$n$个元素$a_1,a_2,\dots ,a_n$，满足：

• $a_1,a_2,\dots ,a_n$线性无关；
• $V$中任一元素$a$总可由$a_1,a_2,\dots ,a_n$线性表示。

那么，$a_1,a_2,\dots ,a_n$就称为线性空间$V$的一个基，$n$称为线性空间$V$的维数。只含有一个零元素的线性空间没有基，规定它的维数为0.
维数为$n$的线性空间称为$n$维线性空间，记作$V_n$。
（同济大学线性代数第五版第六章第二节）

3. Matrices 矩阵

由$m\times n$个数$a_{ij}(i=1,2,\cdots ,m;j=1,2,\dots ,n)$排成的$m$行$n$列的数表称为$m$行$n$列矩阵，简称$m\times n$矩阵。记作
$$A=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right)$$
（同济大学线性代数第五版第二章第一节）

矩阵与数相乘

同矩阵间的加法合起来，统称为矩阵的线形运算

• commutative 交换律
  $\lambda A=A\lambda$
• distributive 分配律
  $(\lambda +\mu )A=\lambda A+\mu A$
  $\lambda (A+B)=\lambda A+\lambda B$
• associative 结合律
  $(\lambda \mu )A=\lambda (\mu A)$

矩阵与向量相乘

$$\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\left[\begin{array}{c}e\\ f\end{array}\right]=\left[\begin{array}{c}ae+bf\\ ce+df\end{array}\right]$$

• 向量与矩阵相乘可以理解为: 向量$r$经过矩阵A变换为$r^{\prime }Ar=r^{\prime }$
• $A(nr)=n(Ar)=n{r}^{\prime }$ \quad （$n$是常数）
• 分配律 $A(r+s)=Ar+As$
• Identity 单位矩阵
  $$I=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$$
• clockwise rotation by $\theta$ 顺时针旋转$\theta$角度
  $$\left[\begin{array}{cc}\cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{array}\right]$$
• determinant of 2×2 matrix 行列式
  $$|A|=detA=det\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]=ad-bc$$
• inverse of 2×2 matrix 逆矩阵
  $${\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]}^{-1}=\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]$$

矩阵与矩阵相乘

• multiplying matrices $A$ and $B$ 矩阵相乘
  $$A\in R^{m\times n},B\in R^{n\times l}$$
  $$AB=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \dots & a_{mn}\end{array}\right]\left[\begin{array}{cccc}{b}_{11}& b_{12}& \cdots & b_{1l}\\ b_{21}& b_{22}& \cdots & b_{2l}\\ ⋮& ⋮& \ddots & ⋮\\ b_{n1}& b_{n2}& \dots & b_{nl}\end{array}\right]=\left[\begin{array}{cccc}{c}_{11}& c_{12}& \cdots & c_{1l}\\ c_{21}& c_{22}& \cdots & c_{2l}\\ ⋮& ⋮& \ddots & ⋮\\ c_{m1}& c_{m2}& \dots & c_{ml}\end{array}\right]=C$$
  $$c_{ik}=a{b}_{ik}=\sum _{j=1}^n{a}_{ij}{b}_{jk}$$
  • Einstein summation convention for multiplying matrices $A$ and $B$
    $$AB=C$$
• $A{A}^{-1}=A^{-1}A=I$ (A is square matrix, A is not invertible matrix)
• 不满足交换律
• distributive 分配律
  $A(B+C)=AB+AC$
  $(B+C)A=BA+CA$
• associative 结合律
  $(AB)C=A(BC)$
  $\lambda (AB)=(\lambda A)B=A(\lambda B)$ ($\lambda$ 是常数)

4. change of basis 基变换/坐标变换

Change from an original basis to a new, primed basis. The columns of the transformation matrix $P$ are the new basis vectors in the original coordinate system. So
Let $\alpha (a_1,a_2,\dots ,a_n)$ denote the old basis, $\beta (b_1,b_2,\dots ,b_n)$ denote the new basis. We can get,
$$\beta =\alpha P$$
or,
$${\beta }^T=P^T{\alpha }^T$$
where $r^{\prime }$ is the vector in the new basis, and $r$ is the vector in the original basis.
$$$\alpha r=\beta r^{\prime }=\alpha P{r}^{\prime }$$
$$r^{\prime }=P^{-1}r$$
（参考同济大学线性代数第五版第六章第三节）

正交基

If a matrix $A$ is orthonormal (all the columns are of unit size and orthogonal to each other) then
矩阵$A$是正交矩阵（正交阵）的充分必要条件是$A$等列向量都是单位向量，且两两正交。
$$A^T=A^{-1}$$
即
$$A^T{A}^{-1}=E$$
即
$$\left[\begin{array}{c}{a}_1^T\\ a_2^T\\ \dots \\ a_n^T\end{array}\right](a_1,a_2,\dots ,a_n)=E$$
也即
$$(a_i^T{a}_j)=({\delta }_{ij})$$
相当于$n^2$个关系式
$$a_i^T{a}_j={\delta }_{ij}=\{\begin{array}{ll}1& \text{when }i=j\\ 0& \text{when }i\ne j\end{array}$$
因为$A^T=A^{-1}$,所以上述结论对$A$的行向量亦成立。
（补充阅读：同济大学线性代数第五版第六章第三节）

5. Gram-Schmidt process for constructing an orthonormal basis 用格拉姆-施密特正交化构建正交基

Start with $n$ linearly independent basis vectors v = { v 1 , v 2 , … , v n } v = \{ v_1,v_2,\dots,v_n \} v={ v1​,v2​,…,vn​}. Then
$$e_1=\frac{v_1}{\Vert v_1\Vert }$$
$u_2=v_2-\frac{v_2\cdot e_1}{e_1\cdot e_1}{e}_1=v_2-(v_2\cdot e_1)e_1$, so $e_2=\frac{u_2}{\Vert u_2\Vert }$
… and so on for $u_3$ being the remnant part of $v_3$ not composed of the preceding