矩阵求导之三：定义篇（中）

4 向量对向量求导

4.1 定义

4.1.1 行向量对列向量求导

也称分母布局，用$\frac{\partial {\mathbf{y}}^T}{\partial \mathbf{x}}$表示。
$m$维行向量${\mathbf{y}}^T=\left[y_1,y_2,\cdots ,y_m\right]$对$n$维列向量$\mathbf{x}={\left[x_1,x_2,\cdots ,x_n\right]}^T$求导，得到的是$n\times m$维矩阵：
$$\frac{\partial {\mathbf{y}}^T}{\partial \mathbf{x}}=\left[\begin{array}{c}\frac{\partial {\mathbf{y}}^T}{\partial x_1}\\ \\ \frac{\partial {\mathbf{y}}^T}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial {\mathbf{y}}^T}{\partial x_n}\end{array}\right]=\left[\begin{array}{cccc}\frac{\partial y_1}{\partial x_1}& \frac{\partial y_2}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_1}\\ & & & \\ \frac{\partial y_1}{\partial x_2}& \frac{\partial y_2}{\partial x_2}& \cdots & \frac{\partial y_m}{\partial x_2}\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ \frac{\partial y_1}{\partial x_n}& \frac{\partial y_2}{\partial x_n}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right]$$
数学上将这种矩阵称之为梯度矩阵

4.1.2 列向量对行向量求导

也称分子布局，用$\frac{\partial \mathbf{y}}{\partial {\mathbf{x}}^T}$表示。
$m$维列向量$\mathbf{y}={\left[y_1,y_2,\cdots ,y_m\right]}^T$对$n$维行向量${\mathbf{x}}^T=\left[x_1,x_2,\cdots ,x_n\right]$求导，得到的是$m\times n$维矩阵：
$$\frac{\partial \mathbf{y}}{\partial {\mathbf{x}}^T}=\left[\begin{array}{c}\frac{\partial y_1}{\partial \mathbf{x}}\\ \\ \frac{\partial y_2}{\partial \mathbf{x}}\\ \\ ⋮\\ \\ \frac{\partial y_m}{\partial \mathbf{x}}\end{array}\right]=\left[\begin{array}{cccc}\frac{\partial y_1}{\partial x_1}& \frac{\partial y_1}{\partial x_2}& \cdots & \frac{\partial y_1}{\partial x_n}\\ & & & \\ \frac{\partial y_2}{\partial x_1}& \frac{\partial y_2}{\partial x_2}& \cdots & \frac{\partial y_2}{\partial x_n}\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ \frac{\partial y_m}{\partial x_1}& \frac{\partial y_m}{\partial x_2}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right]$$
数学上将这种矩阵称之为雅克比 (Jacobian)矩阵。

根据定义可以看出
$$\frac{\partial {\mathbf{y}}^T}{\partial \mathbf{x}}\ne \frac{\partial \mathbf{y}}{\partial {\mathbf{x}}^T},\frac{\partial {\mathbf{y}}^T}{\partial \mathbf{x}}={\left(\frac{\partial \mathbf{y}}{\partial {\mathbf{x}}^T}\right)}^T$$

4.2 运算法则

若$\mathbf{a}\left(\mathbf{x}\right)$和$\mathbf{b}\left(\mathbf{x}\right)$为$m$维列向量函数，$\lambda \left(\mathbf{x}\right)$为数量函数，$\mathbf{x}$为$n$维列向量，则有以下3个运算公式：

4.2.1 加法运算公式

$$\frac{\partial \left({\mathbf{a}}^T\left(\mathbf{x}\right)\pm {\mathbf{b}}^T\left(\mathbf{x}\right)\right)}{\partial \mathbf{x}}=\frac{\partial {\mathbf{a}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\pm \frac{\partial {\mathbf{b}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}$$

4.2.2 数乘运算公式

$$\frac{\partial \left(\lambda \left(\mathbf{x}\right){\mathbf{a}}^T\left(\mathbf{x}\right)\right)}{\partial \mathbf{x}}=\frac{\partial \lambda \left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot {\mathbf{a}}^T\left(\mathbf{x}\right)+\lambda \left(\mathbf{x}\right)\cdot \frac{\partial {\mathbf{a}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}$$

4.2.3 乘法运算公式

$$\frac{\partial \left[{\mathbf{a}}^T\left(\mathbf{x}\right)\cdot \mathbf{b}\left(\mathbf{x}\right)\right]}{\partial \mathbf{x}}=\frac{\partial {\mathbf{a}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{b}\left(\mathbf{x}\right)+\frac{\partial {\mathbf{b}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{a}\left(\mathbf{x}\right)$$
$$\frac{\partial \mathbf{x}}{\partial {\mathbf{x}}^T}=\frac{\partial {\mathbf{x}}^T}{\partial \mathbf{x}}=\mathbf{E}$$

4.3 示例

【例4.1】求证：
$$\frac{\partial \left[{\mathbf{a}}^T\left(\mathbf{x}\right)\cdot \mathbf{b}\left(\mathbf{x}\right)\right]}{\partial \mathbf{x}}=\frac{\partial {\mathbf{a}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{b}\left(\mathbf{x}\right)+\frac{\partial {\mathbf{b}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{a}\left(\mathbf{x}\right)$$
【证】
$$\begin{array}{rl}\frac{\partial \left[{\mathbf{a}}^T\left(\mathbf{x}\right)\cdot \mathbf{b}\left(\mathbf{x}\right)\right]}{\partial \mathbf{x}}& =\left[\begin{array}{c}\frac{\partial {\mathbf{a}}^T\mathbf{b}}{\partial x_1}\\ ⋮\\ \frac{\partial {\mathbf{a}}^T\mathbf{b}}{\partial x_i}\\ ⋮\\ \frac{\partial {\mathbf{a}}^T\mathbf{b}}{\partial x_n}\end{array}\right]=\left[\begin{array}{c}\frac{\partial {\mathbf{a}}^T}{\partial x_1}\cdot \mathbf{b}+{\mathbf{a}}^T\cdot \frac{\partial \mathbf{b}}{\partial x_1}\\ ⋮\\ \frac{\partial {\mathbf{a}}^T}{\partial x_i}\cdot \mathbf{b}+{\mathbf{a}}^T\cdot \frac{\partial \mathbf{b}}{\partial x_i}\\ ⋮\\ \frac{\partial {\mathbf{a}}^T}{\partial x_m}\cdot \mathbf{b}+{\mathbf{a}}^T\cdot \frac{\partial \mathbf{b}}{\partial x_m}\end{array}\right]\\ & \\ & =\left[\begin{array}{c}\frac{\partial {\mathbf{a}}^T}{\partial x_1}\cdot \mathbf{b}+\frac{\partial {\mathbf{b}}^T}{\partial x_1}\cdot \mathbf{a}\\ ⋮\\ \frac{\partial {\mathbf{a}}^T}{\partial x_i}\cdot \mathbf{b}+\frac{\partial {\mathbf{b}}^T}{\partial x_i}\cdot \mathbf{a}\\ ⋮\\ \frac{\partial {\mathbf{a}}^T}{\partial x_m}\cdot \mathbf{b}+\frac{\partial {\mathbf{b}}^T}{\partial x_m}\cdot \mathbf{a}\end{array}\right]\\ & \\ & =\frac{\partial {\mathbf{a}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{b}\left(\mathbf{x}\right)+\frac{\partial {\mathbf{b}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{a}\left(\mathbf{x}\right)\end{array}$$

【例4.2】求
$$\frac{\partial \mathbf{x}}{\partial {\mathbf{x}}^T}$$与$$\frac{\partial {\mathbf{x}}^T}{\partial \mathbf{x}}$$
其中$\mathbf{x}$为$n$维列向量。
【解】
$$\frac{\partial \mathbf{x}}{\partial {\mathbf{x}}^T}=\left[\begin{array}{cccc}\frac{\partial x_1}{\partial x_1}& \frac{\partial x_1}{\partial x_2}& \cdots & \frac{\partial x_1}{\partial x_n}\\ & & & \\ \frac{\partial x_2}{\partial x_1}& \frac{\partial x_2}{\partial x_2}& \cdots & \frac{\partial x_2}{\partial x_n}\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ \frac{\partial x_n}{\partial x_1}& \frac{\partial x_n}{\partial x_2}& \cdots & \frac{\partial x_n}{\partial x_n}\end{array}\right]=\left[\begin{array}{cccc}1& 0& \cdots & 0\\ & & & \\ 0& 1& \cdots & 0\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ 0& 0& \cdots & 1\end{array}\right]=\mathbf{E}$$
$$\frac{\partial {\mathbf{x}}^T}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial x_1}{\partial x_1}& \frac{\partial x_2}{\partial x_1}& \cdots & \frac{\partial x_n}{\partial x_1}\\ & & & \\ \frac{\partial x_1}{\partial x_2}& \frac{\partial x_2}{\partial x_2}& \cdots & \frac{\partial x_n}{\partial x_2}\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ \frac{\partial x_1}{\partial x_n}& \frac{\partial x_2}{\partial x_n}& \cdots & \frac{\partial x_n}{\partial x_n}\end{array}\right]=\left[\begin{array}{cccc}1& 0& \cdots & 0\\ & & & \\ 0& 1& \cdots & 0\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ 0& 0& \cdots & 1\end{array}\right]=\mathbf{E}$$
【例4.3】求
$$\frac{\partial \left({\mathbf{x}}^T\mathbf{A}\right)}{\partial \mathbf{x}}$$
其中$\mathbf{x}$为$n$维列向量，$\mathbf{A}$为$n\times m$维常数阵。
【解】设$$\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{\alpha }}_1& {\mathbf{\alpha }}_2& \cdots & {\mathbf{\alpha }}_m\end{array}\right]$$
其中$${\mathbf{\alpha }}_i={\left[{\begin{array}{cccc}{\alpha }_{i1}& {\alpha }_{i2}& \cdots & \alpha \end{array}}_{in}\right]}^T$$
为$n$维列向量。因此：
$${\mathbf{x}}^T\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{x}}^T{\mathbf{\alpha }}_1& {\mathbf{x}}^T{\mathbf{\alpha }}_2& \cdots & {\mathbf{x}}^T{\mathbf{\alpha }}_m\end{array}\right]$$
根据定义
$$\frac{\partial \left({\mathbf{x}}^T\mathbf{A}\right)}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial \left({\mathbf{x}}^T{\mathbf{\alpha }}_1\right)}{\partial \mathbf{x}}& \frac{\partial \left({\mathbf{x}}^T{\mathbf{\alpha }}_2\right)}{\partial \mathbf{x}}& \cdots & \frac{\partial \left({\mathbf{x}}^T{\mathbf{\alpha }}_m\right)}{\partial \mathbf{x}}\end{array}\right]$$
其中每一个列向量：
$$\frac{\partial \left({\mathbf{x}}^T{\mathbf{\alpha }}_i\right)}{\partial \mathbf{x}}=\frac{\partial {\mathbf{x}}^T}{\partial \mathbf{x}}\cdot {\mathbf{\alpha }}_i+\frac{\partial {{\mathbf{\alpha }}_i}^T}{\partial \mathbf{x}}\cdot \mathbf{x}={\mathbf{\alpha }}_i$$
因此有：
$$\frac{\partial \left({\mathbf{x}}^T\mathbf{A}\right)}{\partial \mathbf{x}}=\left[\begin{array}{cccc}{\mathbf{\alpha }}_1& {\mathbf{\alpha }}_2& \cdots & {\mathbf{\alpha }}_m\end{array}\right]=\mathbf{A}$$
【推论】
若$\mathbf{A}$为$n\times n$方阵，则：
$$\frac{\partial {\mathbf{x}}^T{\mathbf{A}}^T}{\partial \mathbf{x}}={\mathbf{A}}^T$$
【例4.4】求$$\frac{\partial \left(\mathbf{Bx}\right)}{\partial {\mathbf{x}}^T}$$
其中$\mathbf{x}$为$n$维列向量，$\mathbf{B}$为$m\times n$矩阵。
【解】
记${\mathbf{\beta }}_i$为$n$维列向量，则矩阵$\mathbf{B}$写成：
$$\mathbf{B}={\left[{\begin{array}{cccc}{{\mathbf{\beta }}_1}^T& {{\mathbf{\beta }}_2}^T& \cdots & {\mathbf{\beta }}_m\end{array}}^T\right]}^T$$
则：
$$\mathbf{Bx}={\left[{\begin{array}{cccc}{{\mathbf{\beta }}_1}^T\mathbf{x}& {{\mathbf{\beta }}_2}^T\mathbf{x}& \cdots & {\mathbf{\beta }}_m\end{array}}^T\mathbf{x}\right]}^T$$
$$\frac{\partial \left(\mathbf{Bx}\right)}{\partial {\mathbf{x}}^T}={\left[\begin{array}{cccc}\frac{\partial \left({{\mathbf{\beta }}_1}^T\mathbf{x}\right)}{\partial {\mathbf{x}}^T}& \frac{\partial \left({{\mathbf{\beta }}_2}^T\mathbf{x}\right)}{\partial {\mathbf{x}}^T}& \cdots & \frac{\partial \left({{\mathbf{\beta }}_m}^T\mathbf{x}\right)}{\partial {\mathbf{x}}^T}\end{array}\right]}^T$$
其中每一个列向量，
$$\begin{array}{rl}\frac{\partial \left({{\mathbf{\beta }}_i}^T\mathbf{x}\right)}{\partial {\mathbf{x}}^T}& ={\left[\frac{\partial {\left({{\mathbf{\beta }}_i}^T\mathbf{x}\right)}^T}{\partial \mathbf{x}}\right]}^T={\left[\frac{\partial \left({\mathbf{x}}^T{\mathbf{\beta }}_i\right)}{\partial \mathbf{x}}\right]}^T\\ & \\ & \\ & ={\left[\frac{\partial {\mathbf{x}}^T}{\partial \mathbf{x}}\cdot {\mathbf{\beta }}_i+\frac{\partial {{\mathbf{\beta }}_i}^T}{\partial \mathbf{x}}\cdot {\mathbf{x}}^T\right]}^T\\ & \\ & ={{\mathbf{\beta }}_i}^T\end{array}$$
因此有：
$$\frac{\partial \left({{\mathbf{\beta }}_i}^T\mathbf{x}\right)}{\partial {\mathbf{x}}^T}=\left[\begin{array}{c}{{\mathbf{\beta }}_1}^T\\ \\ {{\mathbf{\beta }}_2}^T\\ \\ ⋮\\ \\ {{\mathbf{\beta }}_m}^T\end{array}\right]=\mathbf{B}$$
【例4.5】求二次型${\mathbf{x}}^T\mathbf{Ax}$对$\mathbf{x}$的导数，其中$\mathbf{A}$为对称矩阵。
【解】根据
$$\frac{\partial \left[{\mathbf{a}}^T\left(\mathbf{x}\right)\cdot \mathbf{b}\left(\mathbf{x}\right)\right]}{\partial \mathbf{x}}=\frac{\partial {\mathbf{a}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{b}\left(\mathbf{x}\right)+\frac{\partial {\mathbf{b}}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\cdot \mathbf{a}\left(\mathbf{x}\right)$$
有：
$$\begin{array}{rl}\frac{\partial \left[{\mathbf{x}}^T\mathbf{Ax}\right]}{\partial \mathbf{x}}& =\frac{\partial {\mathbf{x}}^T}{\partial \mathbf{x}}\cdot \left(\mathbf{Ax}\right)+\frac{\partial {\left(\mathbf{Ax}\right)}^T}{\partial \mathbf{x}}\cdot \mathbf{x}\\ & =\mathbf{Ax}+\frac{\partial \left({\mathbf{x}}^T{\mathbf{A}}^T\right)}{\partial \mathbf{x}}\cdot \mathbf{x}\\ & =\mathbf{Ax}+{\mathbf{A}}^T\mathbf{x}=\left(\mathbf{A}+{\mathbf{A}}^T\right)\mathbf{x}\\ & =2\mathbf{Ax}\end{array}$$
即：
$$\frac{\partial \left[{\mathbf{x}}^T\mathbf{Ax}\right]}{\partial \mathbf{x}}=2\mathbf{Ax}$$
又：
$$\frac{\partial {\mathbf{\alpha }}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}={\left[\frac{\partial \mathbf{\alpha }\left(\mathbf{x}\right)}{\partial {\mathbf{x}}^T}\right]}^T,\frac{\partial \mathbf{\alpha }\left(\mathbf{x}\right)}{\partial {\mathbf{x}}^T}={\left[\frac{\partial {\mathbf{\alpha }}^T\left(\mathbf{x}\right)}{\partial \mathbf{x}}\right]}^T$$
故：
$$\frac{\partial \left[{\mathbf{x}}^T\mathbf{Ax}\right]}{\partial {\mathbf{x}}^T}={\left[\frac{\partial {\left[{\mathbf{x}}^T\mathbf{Ax}\right]}^T}{\partial \mathbf{x}}\right]}^T={\left[\frac{\partial \left[{\mathbf{x}}^T\mathbf{Ax}\right]}{\partial \mathbf{x}}\right]}^T=2{\left(\mathbf{Ax}\right)}^T=2{\mathbf{x}}^T\mathbf{A}$$
【例4.6】求函数${\mathbf{\lambda }}^T\mathbf{Ax}$对$\mathbf{x}$的导数。其中${\mathbf{\lambda }}^T$为$1\times n$的行向量，$\mathbf{A}$为$n\times n$的常数矩阵，$\mathbf{x}$为$n$维列向量。
【解】因为${\mathbf{\lambda }}^T\mathbf{Ax}$为标量，其与其转置相等：
$${\mathbf{\lambda }}^T\mathbf{Ax}={\left({\mathbf{\lambda }}^T\mathbf{Ax}\right)}^T={\mathbf{x}}^T\mathbf{A\lambda }$$
于是：
$$\frac{\partial \left({\mathbf{\lambda }}^T\mathbf{Ax}\right)}{\partial \mathbf{x}}=\frac{\partial \left({\mathbf{x}}^T{\mathbf{A}}^T\mathbf{\lambda }\right)}{\partial \mathbf{x}}={\mathbf{A}}^T\mathbf{\lambda }$$

5 向量对矩阵求导

5.1 定义

设矩阵
$${\mathbf{X}}_{m\times n}=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1n}\\ x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ x_{m1}& x_{m2}& \cdots & x_{mn}\end{array}\right]$$
以矩阵$\mathbf{X}$为自变量的$n$维列向量函数：
$$\mathbf{z}\left(\mathbf{X}\right)={\left[\begin{array}{cccc}{z}_1\left(\mathbf{X}\right)& z_2\left(\mathbf{X}\right)& \cdots & z_n\end{array}\left(\mathbf{X}\right)\right]}^T$$
在分子布局下，有：
$$\frac{\partial \mathbf{z}\left(\mathbf{X}\right)}{\partial \mathbf{X}}={\left[\begin{array}{cccc}\frac{\partial z_1\left(\mathbf{X}\right)}{\partial \mathbf{X}}& \frac{\partial z_2\left(\mathbf{X}\right)}{\partial \mathbf{X}}& \cdots & \frac{\partial z_n\left(\mathbf{X}\right)}{\partial \mathbf{X}}\end{array}\right]}^T$$
其中：
$$\frac{\partial z_i\left(\mathbf{X}\right)}{\partial \mathbf{X}}=\left[\begin{array}{cccc}\frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{11}}& \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{12}}& \cdots & \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{1n}}\\ & & & \\ \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{21}}& \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{22}}& \cdots & \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{2n}}\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{m1}}& \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{m2}}& \cdots & \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{mn}}\end{array}\right]$$
在分母布局下，有：
$$\frac{\partial {\mathbf{z}}^T\left(\mathbf{X}\right)}{\partial \mathbf{X}}=\left[\begin{array}{cccc}\frac{\partial z_1\left(\mathbf{X}\right)}{\partial \mathbf{X}}& \frac{\partial z_2\left(\mathbf{X}\right)}{\partial \mathbf{X}}& \cdots & \frac{\partial z_n\left(\mathbf{X}\right)}{\partial \mathbf{X}}\end{array}\right]$$
其中：
$$\frac{\partial z_i\left(\mathbf{X}\right)}{\partial \mathbf{X}}=\left[\begin{array}{cccc}\frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{11}}& \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{12}}& \cdots & \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{1n}}\\ & & & \\ \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{21}}& \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{22}}& \cdots & \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{2n}}\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{m1}}& \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{m2}}& \cdots & \frac{\partial z_i\left(\mathbf{X}\right)}{\partial x_{mn}}\end{array}\right]$$

5.2 形状规则

向量$\mathbf{y}$对矩阵$\mathbf{X}$求导，分为两步：
Step1：向量$\mathbf{y}$的每个元素是标量，先做$\mathbf{y}$的每个元素对矩阵$\mathbf{X}$求导，这里按照标量对矩阵的求导规则进行。
Step2：第一步完成后，将求导结果按$\mathbf{y}$的形状排列。
详细内容请阅读参考文献【1】。

参考文献

[1] 向量对矩阵求导
[2] 向量，标量对向量求导数