矩阵的迹和求导

教程

矩阵微积分(wiki)
机器学习中的线性代数之矩阵求导

笔记

1. 向量对标量求导

$\mathbf{y}=(y_1,y_2,\cdots ,y_n{)}^T$对标量$x$求导：
$$\frac{\partial \mathbf{y}}{\partial x}=(\frac{\partial y_1}{\partial x},\frac{\partial y_2}{\partial x},\cdots ,\frac{\partial y_n}{\partial x}{)}^T$$

2. 标量对向量求导

标量$y$对$\mathbf{x}=(x_1,x_2,\cdots ,x_n{)}^T$求导：
分子布局：
$$\frac{\partial y}{\partial \mathbf{x}}=(\frac{\partial y}{\partial x_1},\frac{\partial y}{\partial x_2},\cdots ,\frac{\partial y}{\partial x_n})$$

3. 向量对向量求导

$\mathbf{y}=(y_1,y_2,\cdots ,y_m{)}^T$对$\mathbf{x}=(x_1,x_2,\cdots ,x_n{)}^T$求导：
分子布局：
$$\frac{\partial \mathbf{y}}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial y_1}{\partial x_1}& \frac{\partial y_1}{\partial x_2}& \cdots & \frac{\partial y_1}{\partial x_n}\\ \frac{\partial y_2}{\partial x_1}& \frac{\partial y_2}{\partial x_2}& \cdots & \frac{\partial y_2}{\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_m}{\partial x_1}& \frac{\partial y_m}{\partial x_2}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right]$$

4. 矩阵对标量求导

矩阵$Y_{m\times n}$对标量$x$求导：
分子布局：
$$\frac{\partial Y}{\partial x}=\left[\begin{array}{cccc}\frac{\partial y_{11}}{\partial x}& \frac{\partial y_{12}}{\partial x}& \cdots & \frac{\partial y_{1n}}{\partial x}\\ \frac{\partial y_{21}}{\partial x}& \frac{\partial y_{22}}{\partial x}& \cdots & \frac{\partial y_{2n}}{\partial x}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_{m1}}{\partial x}& \frac{\partial y_{m2}}{\partial x}& \cdots & \frac{\partial y_{mn}}{\partial x}\end{array}\right]$$

5. 标量对矩阵求导

标量$y$对矩阵$X_{m\times n}$求导：
分子布局：
$$\frac{\partial y}{\partial X}=\left[\begin{array}{cccc}\frac{\partial y}{\partial x_{11}}& \frac{\partial y}{\partial x_{21}}& \cdots & \frac{\partial y}{\partial x_{m1}}\\ \frac{\partial y}{\partial x_{12}}& \frac{\partial y}{\partial x_{22}}& \cdots & \frac{\partial y}{\partial x_{m2}}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y}{\partial x_{1n}}& \frac{\partial y}{\partial x_{2n}}& \cdots & \frac{\partial y}{\partial x_{mn}}\end{array}\right]$$

6. 矩阵对向量求导

矩阵$Y_{m\times n}$对向量$\mathbf{x}=(x_1,x_2,\cdots ,x_n)$求导：
分子布局：
$$\frac{\partial Y}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial y_{11}}{\partial x_1}& \frac{\partial y_{12}}{\partial x_2}& \cdots & \frac{\partial y_{1n}}{\partial x_n}\\ \frac{\partial y_{21}}{\partial x_1}& \frac{\partial y_{22}}{\partial x_2}& \cdots & \frac{\partial y_{2n}}{\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_{m1}}{\partial x_1}& \frac{\partial y_{m2}}{\partial x_2}& \cdots & \frac{\partial y_{mn}}{\partial x_n}\end{array}\right]$$

7. 矩阵的迹

矩阵的迹就是矩阵的对角线元素的和，也是矩阵特征值之和，用tr(A)表示。

1. $\mathbf{t}\mathbf{r}(\mathbf{A})=\mathbf{t}\mathbf{r}({\mathbf{A}}^{{}^{\prime }})$
   证明：矩阵转置不影响对角线的元素，因此迹不变。

2. $\mathbf{t}\mathbf{r}(\mathbf{A}\mathbf{B})=\mathbf{t}\mathbf{r}(\mathbf{B}\mathbf{A})$
   证明：$$\begin{gathered} A_{m\times n},B_{n\times m} \\ tr(AB)={\sum }_{i=1}^n{\sum }_{j=1}^m{\alpha }_{ij}{\beta }_{ji} \\ tr(BA)={\sum }_{i=1}^m{\sum }_{j=1}^n{\beta }_{ij}{\alpha }_{ji}={\sum }_{i=1}^n{\sum }_{j=1}^m{\beta }_{ji}{\alpha }_{ij}=tr(AB) \end{gathered}$$

3. $\mathbf{t}\mathbf{r}(\mathbf{A}\mathbf{B}\mathbf{C})=\mathbf{t}\mathbf{r}(\mathbf{B}\mathbf{C}\mathbf{A})=\mathbf{t}\mathbf{r}(\mathbf{C}\mathbf{A}\mathbf{B})$
   证明：$tr(ABC)=tr(A(BC))=tr(BCA)=tr(B(CA))=tr(CAB)$

4. $\frac{\partial (\mathbf{t}\mathbf{r}(\mathbf{A}\mathbf{B}))}{\partial \mathbf{A}}=\frac{\partial (\mathbf{t}\mathbf{r}(\mathbf{B}\mathbf{A}))}{\partial \mathbf{A}}={\mathbf{B}}^{{}^{\prime }}$
   证明：$\frac{\partial (tr(AB))}{\partial A}=\frac{{\sum }_{i=1}^n{\sum }_{j=1}^m{\alpha }_{ij}{\beta }_{ji}}{\partial a_{ij}}=b_{ji}=B^{{}^{\prime }}$

5. $\frac{\partial (\mathbf{t}\mathbf{r}({\mathbf{A}}^{{}^{\prime }}\mathbf{B}))}{\partial \mathbf{A}}=\frac{\partial (\mathbf{t}\mathbf{r}(\mathbf{B}{\mathbf{A}}^{{}^{\prime }}))}{\partial \mathbf{A}}=\mathbf{B}$
   证明：$$\begin{gathered} A_{m\times n},A_{n\times m}^{{}^{\prime }},B_{m\times n} \\ \frac{\partial (tr(A^{{}^{\prime }}B))}{\partial A}=\frac{{\sum }_{i=1}^n{\sum }_{j=1}^m{\alpha }_{ji}{\beta }_{ji}}{\partial a_{ij}}=\frac{{\sum }_{j=1}^n{\sum }_{i=1}^m{\alpha }_{ij}{\beta }_{ij}}{\partial a_{ij}}{b}_{ij}=B \end{gathered}$$

6. $\frac{\partial (\mathbf{t}\mathbf{r}({\mathbf{A}}^{{}^{\prime }}\mathbf{X}{\mathbf{B}}^{{}^{\prime }}))}{\partial \mathbf{X}}=\frac{\partial (\mathbf{t}\mathbf{r}(\mathbf{B}{\mathbf{X}}^{{}^{\prime }}\mathbf{A}))}{\partial \mathbf{X}}=\mathbf{A}\mathbf{B}$
   证明：$$\begin{gathered} tr(A^{{}^{\prime }}X{B}^{{}^{\prime }})=tr(A^{{}^{\prime }}X{B}^{{}^{\prime }}{)}^{{}^{\prime }}=tr(B{X}^{{}^{\prime }}A)=tr(AB{X}^{{}^{\prime }}) \\ 又\frac{\partial (tr(AB{X}^{{}^{\prime }}))}{\partial X}=AB \\ 因此\frac{\partial (tr(A^{{}^{\prime }}X{B}^{{}^{\prime }}))}{\partial X}=\frac{\partial (tr(B{X}^{{}^{\prime }}A))}{\partial X}=AB \end{gathered}$$

7. $\frac{\partial (\mathbf{t}\mathbf{r}(\mathbf{A}\mathbf{X}\mathbf{B}{\mathbf{X}}^{{}^{\prime }}))}{\partial \mathbf{X}}==\mathbf{A}\mathbf{X}\mathbf{B}+{\mathbf{A}}^{{}^{\prime }}\mathbf{X}{\mathbf{B}}^{{}^{\prime }}$

8. $\frac{\partial (\mathbf{t}\mathbf{r}(\mathbf{A}\mathbf{X}\mathbf{B}\mathbf{X}))}{\partial \mathbf{X}}=={\mathbf{A}}^{{}^{\prime }}{\mathbf{X}}^{{}^{\prime }}{\mathbf{B}}^{{}^{\prime }}+{\mathbf{B}}^{{}^{\prime }}{\mathbf{X}}^{{}^{\prime }}{\mathbf{A}}^{{}^{\prime }}$