矩阵求导——学习笔记

矩阵、向量求导法则，我们这里默认的是分母布局

一、基本定理

1.1 对元素求导

1. 行向量

设 ${\mathbf{y}}^T=[y_1\dots y_n]$ 是行向量，$x$ 是元素，则
$$\frac{\partial {\mathbf{y}}^T}{\partial x}=\left[\begin{array}{lll}\frac{\partial y_1}{\partial x}& \cdots & \frac{\partial y_n}{\partial x}\end{array}\right]$$

2. 列向量

设 $\mathbf{y}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_m\end{array}\right]$是 $m$ 维列向量，$x$ 是元素，则

$$\frac{\partial \mathbf{y}}{\partial x}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x}\\ ⋮\\ \frac{\partial y_m}{\partial x}\end{array}\right]$$

3. 矩阵

设 $Y=\left[\begin{array}{ccc}{y}_{11}& \cdots & y_{1n}\\ ⋮& & ⋮\\ y_{m1}& \cdots & y_{mn}\end{array}\right]$ 是 $m\times n$ 矩阵，$x$ 是元素，则

$$\frac{\partial Y}{\partial x}=\left[\begin{array}{ccc}\frac{\partial y_{11}}{\partial x}& \cdots & \frac{\partial y_{1n}}{\partial x}\\ ⋮& & \\ \frac{\partial y_{m1}}{\partial x}& \cdots & \frac{\partial y_{mn}}{\partial x}\end{array}\right]$$

1.2 对行向量求导

1. 元素
   设 $y$ 是元素，${\mathbf{x}}^T=\left[\begin{array}{lll}{x}_1& \cdots & x_q\end{array}\right]$ 是 $q$ 维行向量，则

$$\frac{\partial y}{\partial {\mathbf{x}}^T}=\left[\begin{array}{lll}\frac{\partial y}{\partial x_1}& \cdots & \frac{\partial y}{\partial x_q}\end{array}\right]$$

2. 列向量

设 $\mathbf{y}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_m\end{array}\right]$是 $m$ 维列向量，${\mathbf{x}}^T=\left[\begin{array}{lll}{x}_1& \cdots & x_q\end{array}\right]$ 是 $q$ 维行向量，则

$$\frac{\partial \mathbf{y}}{\partial {\mathbf{x}}^T}=\left[\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \cdots & \frac{\partial y_1}{\partial x_q}\\ ⋮& & \\ \frac{\partial y_m}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_q}\end{array}\right]$$

3. 行向量

设 ${\mathbf{y}}^T=[y_1\dots y_n]$ 是 $n$ 维行向量，${\mathbf{x}}^T=\left[\begin{array}{lll}{x}_1& \cdots & x_q\end{array}\right]$ 是 $q$ 维行向量，则
$$\frac{\partial {\mathbf{y}}^T}{\partial {\mathbf{x}}^T}=\left[\begin{array}{lll}\frac{\partial {\mathbf{y}}^T}{\partial x_1}& \cdots & \frac{\partial {\mathbf{y}}^T}{\partial x_q}\end{array}\right]$$

4. 矩阵

设 $Y=\left[\begin{array}{ccc}{y}_{11}& \cdots & y_{1n}\\ ⋮& & ⋮\\ y_{m1}& \cdots & y_{mn}\end{array}\right]$ 是 $m\times n$ 矩阵，${\mathbf{x}}^T=\left[\begin{array}{lll}{x}_1& \cdots & x_q\end{array}\right]$ 是 $q$ 维行向量，则

$$\frac{\partial Y}{\partial {\mathbf{x}}^T}=\left[\begin{array}{lll}\frac{\partial Y}{\partial x_1}& \cdots & \frac{\partial Y}{\partial x_q}\end{array}\right]$$

1.3 对列向量求导

1. 元素

设 $y$ 是元素，$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ ⋮\\ x_p\end{array}\right]$ 是 $p$ 维列向量，则
$$\frac{\partial y}{\partial \mathbf{x}}=\left[\begin{array}{c}\frac{\partial y}{\partial x_1}\\ ⋮\\ \frac{\partial y}{\partial x_p}\end{array}\right]$$

2. 行向量

设 ${\mathbf{y}}^T=[y_1\dots y_n]$ 是 $n$ 维行向量，$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ ⋮\\ x_p\end{array}\right]$ 是 $p$ 维列向量，则
$$\frac{\partial {\mathbf{y}}^T}{\partial \mathbf{x}}=\left[\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \cdots & \frac{\partial y_n}{\partial x_1}\\ ⋮& & \\ \frac{\partial y_1}{\partial x_p}& \cdots & \frac{\partial y_n}{\partial x_p}\end{array}\right]$$

3. 列向量

设 $\mathbf{y}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_m\end{array}\right]$是 $m$ 维列向量，$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ ⋮\\ x_p\end{array}\right]$ 是 $p$ 维列向量，则
$$\frac{\partial \mathbf{y}}{\partial \mathbf{x}}=\left[\begin{array}{c}\frac{\partial y_1}{\partial \mathbf{x}}\\ ⋮\\ \frac{\partial y_m}{\partial \mathbf{x}}\end{array}\right]$$

4. 矩阵

设 $Y=\left[\begin{array}{ccc}{y}_{11}& \cdots & y_{1n}\\ ⋮& & ⋮\\ y_{m1}& \cdots & y_{mn}\end{array}\right]$ 是 $m\times n$ 矩阵，$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ ⋮\\ x_p\end{array}\right]$ 是 $p$ 维列向量，则
$$\frac{\partial Y}{\partial \mathbf{x}}=\left[\begin{array}{ccc}\frac{\partial y_{11}}{\partial \mathbf{x}}& \cdots & \frac{\partial y_{1n}}{\partial \mathbf{x}}\\ ⋮& & ⋮\\ \frac{\partial y_{m1}}{\partial \mathbf{x}}& \cdots & \frac{\partial y_{mn}}{\partial \mathbf{x}}\end{array}\right]$$

1.4 对矩阵求导

1. 元素

设 $y$ 是元素，$X=\left[\begin{array}{ccc}{x}_{11}& \cdots & x_{1q}\\ ⋮& & ⋮\\ x_{p1}& \cdots & y_{pq}\end{array}\right]$ 是 $p\times q$ 维矩阵，则

$$\frac{\partial y}{\partial X^T}=\left[\begin{array}{ccc}\frac{\partial y}{\partial x_{11}}& \cdots & \frac{\partial y}{\partial x_{1q}}\\ ⋮& & \\ \frac{\partial y}{\partial x_{p1}}& \cdots & \frac{\partial y}{\partial x_{pq}}\end{array}\right]$$

2. 行向量

设 ${\mathbf{y}}^T=[y_1\dots y_n]$ 是 $n$ 维行向量，
$X=\left[\begin{array}{ccc}{x}_{11}& \cdots & x_{1q}\\ ⋮& & ⋮\\ x_{p1}& \cdots & y_{pq}\end{array}\right]$ 是 $p\times q$ 维矩阵，则

$$\frac{\partial {\mathbf{y}}^T}{\partial X}=\left[\begin{array}{ccc}\frac{\partial {\mathbf{y}}^T}{\partial x_{11}}& \cdots & \frac{\partial {\mathbf{y}}^T}{\partial x_{1q}}\\ ⋮& & \\ \frac{\partial {\mathbf{y}}^T}{\partial x_{p1}}& \cdots & \frac{\partial {\mathbf{y}}^T}{\partial x_{pq}}\end{array}\right]$$

二、例题

这一部分的主要例题是学习了B站UP主 杂谈博士 的视频，大家有空可以去学习一下。

例1 设 $\mathbf{x}={\left({\xi }_1,{\xi }_2,\cdots {\xi }_n\right)}^T$，n 元函数 $f(\mathbf{x})$，求 $\frac{\mathrm{d}f}{\mathrm{d}{\mathbf{x}}^T}$，$\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}}$，和 $\frac{{\mathrm{d}}^{2}f}{\mathrm{d}{\mathbf{x}}^2}$。

解：根据定义有

$$\frac{\mathrm{d}f}{\mathrm{d}{\mathbf{x}}^T}=\left(\frac{\partial f}{\partial {\xi }_1},\frac{\partial f}{\partial {\xi }_2},\cdots ,\frac{\partial f}{\partial {\xi }_n}\right)$$

梯度

$$\nabla f(\mathbf{x})=\frac{\mathbf{d}f}{\mathbf{d}\mathbf{x}}=\left(\begin{array}{c}\frac{\partial f}{\partial {\xi }_1}\\ ⋮\\ \frac{\partial f}{\partial {\xi }_n}\end{array}\right)$$

Hessian阵：

$$\mathbf{H}(\mathbf{x})={\nabla }^{2}f(\mathbf{x})=\frac{{\mathbf{d}}^{2}f}{\mathbf{d}{\mathbf{x}}^2}=\left(\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial {\xi }_1^2}& \frac{{\partial }^{2}f}{\partial {\xi }_1\partial {\xi }_2}& \cdots & \frac{{\partial }^{2}f}{\partial {\xi }_1\partial {\xi }_n}\\ \frac{{\partial }^{2}f}{\partial {\xi }_2\partial {\xi }_1}& \frac{{\partial }^{2}f}{\partial {\xi }_2^2}& \cdots & \frac{{\partial }^{2}f}{\partial {\xi }_2\partial {\xi }_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial {\xi }_n\partial {\xi }_1}& \frac{{\partial }^{2}f}{\partial {\xi }_n\partial {\xi }_2}& \cdots & \frac{{\partial }^{2}f}{\partial {\xi }_n^2}\end{array}\right)$$

---

例2 设 $\mathbf{A}={\left(a_{ij}\right)}_{m\times n}$ 是常矩阵，$\mathbf{X}={\left(x_{ij}\right)}_{n\times m}$ 是矩阵变量，且 $f(X)=\mathrm{tr}(AX)$，求 $\frac{\partial f}{\partial \mathbf{X}}$。

分析：

$$\left(\begin{array}{ccc}{c}_{11}& \cdots & c_{1m}\\ ⋮& \ddots & ⋮\\ c_{m1}& \cdots & c_{mm}\end{array}\right)=\left(\begin{array}{ccc}{a}_{11}& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ a_{m1}& \cdots & a_{mn}\end{array}\right)\left(\begin{array}{ccc}{x}_{11}& \cdots & x_{1m}\\ ⋮& \ddots & ⋮\\ x_{n1}& \cdots & x_{nm}\end{array}\right)$$

解：
由于 $$\mathbf{A}\mathbf{X}={\left(\sum _{k=1}^n{a}_{ik}{x}_{kj}\right)}_{m\times m}$$

所以

$$f(X)=\mathrm{tr}(AX)=\sum _{s=1}^m\left(\sum _{k=1}^n{a}_{sk}{x}_{ks}\right)$$

而

$${\left(\frac{\partial f}{\partial x_{ij}}\right)}_{n\times m}={\left(a_{ji}\right)}_{n\times m}(i=1,2,\cdots ,nj=1,2,\cdots ,m)$$

故

$$\frac{\partial f}{\partial \mathbf{X}}={\left(\frac{\partial f}{\partial x_{ij}}\right)}_{n\times m}={\left(a_{ji}\right)}_{n\times m}={\mathbf{A}}^T$$

---

例3 设 $\mathbf{x}={\left({\xi }_1,{\xi }_2,\cdots {\xi }_n\right)}^T$，$\mathbf{A}={\left(a_{ij}\right)}_{n\times n}$ 是常矩阵，n 元函数 $f(\mathbf{x})={\mathbf{x}}^T\mathbf{A}\mathbf{x}$，求 $\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}}$。

解：因 $$f(\mathbf{x})={\xi }_1\sum _{j=1}^n{a}_{1j}{\xi }_j+\cdots +{\xi }_k\sum _{j=1}^n{a}_{kj}{\xi }_j+\cdots +{\xi }_n\sum _{j=1}^n{a}_{nj}{\xi }_j$$

所以

$$\begin{array}{rl}\frac{\partial f(\mathbf{x})}{\partial {\xi }_k}& ={\xi }_1{a}_{1k}+\dots +{\xi }_{k-1}{a}_{k-1,k}+\left(\sum _{j=1}^n{a}_{kj}{\xi }_j+{\xi }_k{a}_{kk}\right)+{\xi }_{k+1}{a}_{k+1,k}+\dots +{\xi }_n{a}_{nk}\\ & =\sum _{i=1}^n{a}_{ik}{\xi }_i+\sum _{j=1}^n{a}_{kj}{\xi }_j,k=1,2,\cdots ,n\end{array}$$

$$\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}}=\left(\begin{array}{c}\frac{\partial f}{\partial {\xi }_1}\\ \frac{\partial f}{\partial {\xi }_2}\\ ⋮\\ \frac{\partial f}{\partial {\xi }_n}\end{array}\right)=\left(\begin{array}{c}{\sum }_{j=1}^n{a}_{1j}{\xi }_j\\ {\sum }_{j=1}^n{a}_{2j}{\xi }_j\\ ⋮\\ {\sum }_{j=1}^n{a}_{ij}{\xi }_j\end{array}\right)+\left(\begin{array}{c}{\sum }_{i=1}^n{a}_{i1}{\xi }_i\\ {\sum }_{i=1}^n{a}_{i2}{\xi }_i\\ ⋮\\ {\sum }_{i=1}^n{a}_{in}{\xi }_i\end{array}\right)==\mathbf{A}\mathbf{x}+{\mathbf{A}}^T\mathbf{x}=\left(\mathbf{A}+{\mathbf{A}}^T\right)\mathbf{x}$$

特别地，当 $A$ 为对称矩阵时，$\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}}=2Ax$。

---

例4 设$\mathbf{A}\in {\mathbb{R}}^{m\times n}$，$\mathbf{b}\in {\mathbb{R}}^m$，$\mathbf{x}\in {\mathbb{R}}^n$ ，$f(x)=\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2^2$，试求$\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}}$

解：因为

$$\begin{array}{rl}f(\mathbf{x})& =\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2^2=(\mathbf{A}\mathbf{x}-\mathbf{b},\mathbf{A}\mathbf{x}-\mathbf{b})=(\mathbf{A}\mathbf{x}-\mathbf{b}{)}^T(\mathbf{A}\mathbf{x}-\mathbf{b})\\ & =\left({\mathbf{x}}^T{\mathbf{A}}^T-{\mathbf{b}}^T\right)(\mathbf{A}\mathbf{x}-\mathbf{b})\\ & ={\mathbf{x}}^T{\mathbf{A}}^T\mathbf{A}\mathbf{x}-{\mathbf{b}}^T\mathbf{A}\mathbf{x}-{\mathbf{x}}^T{\mathbf{A}}^T\mathbf{b}+{\mathbf{b}}^T\mathbf{b}\\ & =x^T\left(A^{T}A\right)x-2{\left(A^{T}b\right)}^{T}x+b^{T}b\end{array}$$

最后一行我懒得打 \boldsymbol。大家理解一下，doge。此外，二次型的导数之所以可以写成这样，是因为 ${\mathbf{A}}^T\mathbf{A}$ 是一个对称矩阵。

从而

$$\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}}=2{\mathbf{A}}^T\mathbf{A}\mathbf{x}-2{\mathbf{A}}^T\mathbf{b}$$

我们令这个导数等于零，即${\mathbf{A}}^T\mathbf{A}\mathbf{x}={\mathbf{A}}^T\mathbf{b}$ 一定有解的，因为 $\mathrm{tr}({\mathbf{A}}^T\mathbf{A})=\mathrm{tr}({\mathbf{A}}^T\mathbf{b})$。

详细证明我还没学…。