矩阵求导笔记

0.符号定义

• 数域：记$\mathbb{F}$为某一数域。
• 标量：记$y$和$x$为标量，相应的$\mathrm{d}y$和$\mathrm{d}x$也为标量，即$x,\mathrm{d}x,y,\mathrm{d}y\in {\mathbb{F}}^1$。
• 向量：记$\vec{y}$和$\vec{x}$分别为$m$和$n$维列向量，相应的$\mathrm{d}\vec{y}$和$\mathrm{d}\vec{x}$也分别为$m$和$n$维列向量。
  即$\vec{x},\mathrm{d}\vec{x},\in {\mathbb{F}}^n$和 $\vec{y},\mathrm{d}\vec{y},\in {\mathbb{F}}^m$。
• 矩阵：记$Y$和$X$为矩阵，相应的$\mathrm{d}y$和$\mathrm{d}x$也为矩阵。
  即$X,\mathrm{d}X,\in {\mathbb{F}}^{r\times s}$和 $Y,\mathrm{d}Y,\in {\mathbb{F}}^{p\times q}$。

其中$\mathrm{d}\vec{x}=(\mathrm{d}{x}_1,\mathrm{d}{x}_2,\cdots ,\mathrm{d}{x}_n{)}^T$, $\mathrm{d}\vec{y}$同理。
$$\mathrm{d}X=\left(\begin{array}{cccc}\mathrm{d}{\vec{x}}_1,& \mathrm{d}{\vec{x}}_2,& \cdots ,& \mathrm{d}{\vec{x}}_s\end{array}\right)={\left(\begin{array}{ccccc}\mathrm{d}{x}_{11}& \mathrm{d}{x}_{12}& \mathrm{d}{x}_{13}& \cdots & \mathrm{d}{x}_{1s}\\ \mathrm{d}{x}_{21}& \mathrm{d}{x}_{22}& \mathrm{d}{x}_{23}& \cdots & \mathrm{d}{x}_{2s}\\ \mathrm{d}{x}_{31}& \mathrm{d}{x}_{32}& \mathrm{d}{x}_{33}& \cdots & \mathrm{d}{x}_{3s}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \mathrm{d}{x}_{r1}& \mathrm{d}{x}_{r2}& \mathrm{d}{x}_{r3}& \cdots & \mathrm{d}{x}_{rs}\end{array}\right)}_{r\times s}$$
矩阵求导类型[1]

标量	向量	矩阵
标量	$\frac{\partial y}{\partial x}$	$\frac{\partial \vec{y}}{\partial x}$
向量	$\frac{\partial y}{\partial \vec{x}}$	$\frac{\partial \vec{y}}{\partial \vec{x}}$
矩阵	$\frac{\partial y}{\partial X}$	$\frac{\partial \vec{y}}{\partial X}$

1.对标量求导

1.1标量对标量求导

为了全文书写上风格统一：
$$\mathrm{d}y=\frac{\partial y}{\partial x}\mathrm{d}x.$$
性质

• SS1(线性)：$\frac{\partial (u+v)}{\partial x}=\frac{\partial u}{\partial x}+\frac{\partial v}{\partial x}$。
• SS2(分部)：$\frac{\partial (uv)}{\partial x}=u\frac{\partial v}{\partial x}+v\frac{\partial u}{\partial x}$。
• SS3(链式)：$\frac{\partial g(u)}{\partial x}=\frac{\partial g(u)}{\partial u}\frac{\partial u}{\partial x}$。

1.2向量对标量求导

$$\frac{\partial \vec{y}}{\partial x}={\left(\begin{array}{c}\frac{\partial y_1}{\partial x}\\ \frac{\partial y_2}{\partial x}\\ \frac{\partial y_3}{\partial x}\\ ⋮\\ \frac{\partial y_n}{\partial x}\end{array}\right)}_{n\times 1}$$

因此有$\mathrm{d}{y}_i=\frac{\partial y_i}{\partial x}\mathrm{d}x,i=1,2,\cdots ,m$。即
$$\mathrm{d}\vec{y}={\left(\begin{array}{c}\frac{\partial y_1}{\partial x}\mathrm{d}x\\ \frac{\partial y_2}{\partial x}\mathrm{d}x\\ \frac{\partial y_3}{\partial x}\mathrm{d}x\\ ⋮\\ \frac{\partial y_n}{\partial x}\mathrm{d}x\end{array}\right)}_{n\times 1}=\frac{\partial \vec{y}}{\partial x}\otimes \mathrm{d}x$$
性质

• VS1(常向量)：对于$\forall \vec{a}\in {\mathbb{F}}^{n\times 1}$的常列向量, $\frac{\partial \vec{a}}{\partial x}={\vec{0}}_{n\times 1}$
• VS2(向量数乘)：对于$\forall \vec{u}(x)\in {\mathbb{F}}^{n\times 1},a\in {\mathbb{F}}^1$， 有$\frac{\partial a\vec{u}}{\partial x}=a\frac{\partial \vec{u}}{\partial x}$。
• VS3(向量矩阵乘)：对于$\forall \vec{u}(x)\in {\mathbb{F}}^{n\times 1},A\in {\mathbb{F}}^{m\times n}$， 有$\frac{\partial A\vec{u}}{\partial x}=A\frac{\partial \vec{u}}{\partial x}$。
• VS4(向量转置)：对于$\forall \vec{u}(x)\in {\mathbb{F}}^{n\times 1}$， 有$\frac{\partial ({\vec{u}}^T)}{\partial x}={\left(\frac{\partial \vec{u}}{\partial x}\right)}^T$。
• VS5(向量加法)： 对于$\forall \vec{u}(x),\vec{v}(x)\in {\mathbb{F}}^{n\times 1}$， 有$\frac{\partial (\vec{u}+\vec{v})}{\partial x}=\frac{\partial \vec{u}}{\partial x}+\frac{\partial \vec{v}}{\partial x}$。
• VS6(链式)： 对于$\forall \vec{u}(x)\in {\mathbb{F}}^{n\times 1},\overrightarrow{g(\stackrel{⃗}{u})}\in {\mathbb{F}}^{m\times 1}$， 有$\frac{\partial \vec{g}}{\partial x}={\left(\frac{\partial \vec{g}}{\partial \vec{u}}\right)}^T\frac{\partial \vec{u}}{\partial x}$。

简单对 VS3(向量矩阵乘) 性质做证明。
$$\begin{array}{rl}\frac{\partial \left(A\vec{u}\right)}{\partial x}=& \frac{\partial }{\partial x}\left(\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\\ ⋮\\ u_n\end{array}\right)\right)\\ =& \frac{\partial }{\partial x}\left(u_1\left(\begin{array}{c}{a}_{11}\\ a_{21}\\ ⋮\\ a_{n1}\end{array}\right)+u_2\left(\begin{array}{c}{a}_{12}\\ a_{22}\\ ⋮\\ a_{n2}\end{array}\right)+\cdots +u_m\left(\begin{array}{c}{a}_{1m}\\ a_{2m}\\ ⋮\\ a_{nm}\end{array}\right)\right)\\ =& \frac{\partial u_1}{\partial x}\left(\begin{array}{c}{a}_{11}\\ a_{21}\\ ⋮\\ a_{n1}\end{array}\right)+\frac{\partial u_2}{\partial x}\left(\begin{array}{c}{a}_{12}\\ a_{22}\\ ⋮\\ a_{n2}\end{array}\right)+\cdots +\frac{\partial u_m}{\partial x}\left(\begin{array}{c}{a}_{1m}\\ a_{2m}\\ ⋮\\ a_{nm}\end{array}\right)\\ =& \left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right)\left(\begin{array}{c}\frac{\partial u_1}{\partial x}\\ \frac{\partial u_2}{\partial x}\\ ⋮\\ \frac{\partial u_n}{\partial x}\end{array}\right)\\ =& A\frac{\partial \vec{u}}{\partial x}\end{array}$$

下面简单证明 VS6(链式) 法则。首先，对于$\frac{\partial \vec{g}}{\partial \vec{u}}$属于向量对向量求导，有
$$\frac{\partial \vec{g}}{\partial \vec{u}}=\left(\begin{array}{ccccc}\frac{\partial g_1}{\partial \vec{u}},& \frac{\partial g_2}{\partial \vec{u}},& \frac{\partial g_3}{\partial \vec{u}},& \cdots ,& \frac{\partial g_m}{\partial \vec{u}}\end{array}\right)={\left(\begin{array}{ccccc}\frac{\partial g_1}{\partial u_1}& \frac{\partial g_2}{\partial u_1}& \frac{\partial g_3}{\partial x_1}& \cdots & \frac{\partial g_m}{\partial u_1}\\ \frac{\partial g_1}{\partial u_2}& \frac{\partial g_2}{\partial u_2}& \frac{\partial g_3}{\partial x_2}& \cdots & \frac{\partial g_m}{\partial u_2}\\ \frac{\partial g_1}{\partial u_3}& \frac{\partial g_2}{\partial u_3}& \frac{\partial g_3}{\partial x_3}& \cdots & \frac{\partial g_m}{\partial u_3}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial g_1}{\partial u_n}& \frac{\partial g_2}{\partial u_n}& \frac{\partial g_3}{\partial u_n}& \cdots & \frac{\partial g_m}{\partial u_n}\end{array}\right)}_{n\times m}$$
$\frac{\partial \vec{u}}{\partial x}$属于向量对标量求导，有
$$\frac{\partial \vec{u}}{\partial x}={\left(\begin{array}{c}\frac{\partial u_1}{\partial x}\\ \frac{\partial u_2}{\partial x}\\ \frac{\partial u_3}{\partial x}\\ ⋮\\ \frac{\partial u_n}{\partial x}\end{array}\right)}_{n\times 1}$$
因此
$$\begin{array}{rl}RHS={\left(\frac{\partial \vec{g}}{\partial \vec{u}}\right)}^T\frac{\partial \vec{u}}{\partial x}=& {\left(\begin{array}{ccccc}\frac{\partial g_1}{\partial u_1}& \frac{\partial g_1}{\partial u_2}& \frac{\partial g_1}{\partial x_3}& \cdots & \frac{\partial g_1}{\partial u_m}\\ \frac{\partial g_2}{\partial u_1}& \frac{\partial g_2}{\partial u_2}& \frac{\partial g_2}{\partial x_3}& \cdots & \frac{\partial g_2}{\partial u_m}\\ \frac{\partial g_3}{\partial u_1}& \frac{\partial g_3}{\partial u_2}& \frac{\partial g_3}{\partial x_3}& \cdots & \frac{\partial g_3}{\partial u_m}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial g_n}{\partial u_1}& \frac{\partial g_n}{\partial u_2}& \frac{\partial g_n}{\partial u_3}& \cdots & \frac{\partial g_m}{\partial u_n}\end{array}\right)}_{m\times n}{\left(\begin{array}{c}\frac{\partial u_1}{\partial x}\\ \frac{\partial u_2}{\partial x}\\ \frac{\partial u_3}{\partial x}\\ ⋮\\ \frac{\partial u_n}{\partial x}\end{array}\right)}_{n\times 1}\\ =& {\left(\begin{array}{c}{\sum }_{i=1}^n\frac{\partial g_1}{\partial u_i}\frac{\partial u_i}{\partial x}\\ {\sum }_{i=1}^n\frac{\partial g_2}{\partial u_i}\frac{\partial u_i}{\partial x}\\ {\sum }_{i=1}^n\frac{\partial g_3}{\partial u_i}\frac{\partial u_i}{\partial x}\\ ⋮\\ {\sum }_{i=1}^n\frac{\partial g_m}{\partial u_i}\frac{\partial u_i}{\partial x}\end{array}\right)}_{m\times 1}\end{array}$$

如果将$\frac{\partial \vec{g}}{\partial x}$看成向量对标量求导，则
$$LHS=\frac{\partial \vec{g}}{\partial x}={\left(\begin{array}{c}\frac{\partial g_1}{\partial x}\\ \frac{\partial g_2}{\partial x}\\ \frac{\partial g_3}{\partial x}\\ ⋮\\ \frac{\partial g_m}{\partial x}\end{array}\right)}_{m\times 1}={\left(\begin{array}{c}{\sum }_{i=1}^n\frac{\partial g_1}{\partial u_i}\frac{\partial u_i}{\partial x}\\ {\sum }_{i=1}^n\frac{\partial g_2}{\partial u_i}\frac{\partial u_i}{\partial x}\\ {\sum }_{i=1}^n\frac{\partial g_3}{\partial u_i}\frac{\partial u_i}{\partial x}\\ ⋮\\ {\sum }_{i=1}^n\frac{\partial g_m}{\partial u_i}\frac{\partial u_i}{\partial x}\end{array}\right)}_{m\times 1}=RHS.$$

1.3矩阵对标量求导

$$\frac{\partial Y}{\partial x}=\left(\begin{array}{cccc}\frac{\partial {\vec{y}}_1}{\partial x},& \frac{\partial {\vec{y}}_2}{\partial x},& \cdots ,& \frac{\partial {\vec{y}}_q}{\partial x}\end{array}\right)={\left(\begin{array}{ccccc}\frac{\partial y_{11}}{\partial x}& \frac{\partial y_{12}}{\partial x}& \frac{\partial y_{13}}{\partial x}& \cdots & \frac{\partial y_{1q}}{\partial x}\\ \frac{\partial y_{21}}{\partial x}& \frac{\partial y_{22}}{\partial x}& \frac{\partial y_{23}}{\partial x}& \cdots & \frac{\partial y_{2q}}{\partial x}\\ \frac{\partial y_{31}}{\partial x}& \frac{\partial y_{32}}{\partial x}& \frac{\partial y_{33}}{\partial x}& \cdots & \frac{\partial y_{3q}}{\partial x}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_{p1}}{\partial x}& \frac{\partial y_{p2}}{\partial x}& \frac{\partial y_{p3}}{\partial x}& \cdots & \frac{\partial y_{pq}}{\partial x}\end{array}\right)}_{p\times q}$$
因此有$\mathrm{d}{y}_{ij}=\frac{\partial y_{ij}}{\partial x}\mathrm{d}x,i=1,2,\cdots ,p,j=1,2,\cdots ,q$。即
$$\mathrm{d}Y=\left(\begin{array}{cccc}\frac{\partial {\vec{y}}_1}{\partial x}\mathrm{d}x,& \frac{\partial {\vec{y}}_2}{\partial x}\mathrm{d}x,& \cdots ,& \frac{\partial {\vec{y}}_q}{\partial x}\mathrm{d}x\end{array}\right)={\left(\begin{array}{ccccc}\frac{\partial y_{11}}{\partial x}\mathrm{d}x& \frac{\partial y_{12}}{\partial x}\mathrm{d}x& \frac{\partial y_{13}}{\partial x}\mathrm{d}x& \cdots & \frac{\partial y_{1q}}{\partial x}\mathrm{d}x\\ \frac{\partial y_{21}}{\partial x}\mathrm{d}x& \frac{\partial y_{22}}{\partial x}\mathrm{d}x& \frac{\partial y_{23}}{\partial x}\mathrm{d}x& \cdots & \frac{\partial y_{2q}}{\partial x}\mathrm{d}x\\ \frac{\partial y_{31}}{\partial x}\mathrm{d}x& \frac{\partial y_{32}}{\partial x}\mathrm{d}x& \frac{\partial y_{33}}{\partial x}\mathrm{d}x& \cdots & \frac{\partial y_{3q}}{\partial x}\mathrm{d}x\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_{p1}}{\partial x}\mathrm{d}x& \frac{\partial y_{p2}}{\partial x}\mathrm{d}x& \frac{\partial y_{p3}}{\partial x}\mathrm{d}x& \cdots & \frac{\partial y_{pq}}{\partial x}\mathrm{d}x\end{array}\right)}_{p\times q}=\frac{\partial Y}{\partial x}\otimes \mathrm{d}x$$
性质

• MS1(矩阵数乘)：对于$\forall U(x)\in {\mathbb{F}}^{m\times n}$, 有$\frac{\partial (aU)}{\partial x}=a\frac{\partial U}{\partial x}$。
• MS2(矩阵乘)：对于$\forall U(x)\in {\mathbb{F}}^{m\times n},A\in {\mathbb{F}}^{r\times m},B\in {\mathbb{F}}^{n\times s}$, 有$\frac{\partial (AUB)}{\partial x}=A\frac{\partial U}{\partial x}B$。
• MS3(线性)：对于$\forall U(x),V(x)\in {\mathbb{F}}^{m\times n}$, 有$\frac{\partial (U+V)}{\partial x}=\frac{\partial U}{\partial x}+\frac{\partial V}{\partial x}$。
• MS4(分部)：对于$\forall U(x)\in {\mathbb{F}}^{m\times n},V(x)\in {\mathbb{F}}^{n\times l}$, 有$\frac{\partial (UV)}{\partial x}=\frac{\partial U}{\partial x}V+U\frac{\partial V}{\partial x}$。

先证MS4(分部)。
为了书写上的方便，记$\frac{\partial Y}{\partial x}={\left(\frac{\partial y_{ij}}{\partial x}\right)}_{p\times q}$。
$$\begin{array}{rl}\frac{\partial (UV)}{\partial x}=& {\left(\sum _{k=1}^n\frac{\partial \left(u_{ik}{v}_{kj}\right)}{\partial x}\right)}_{m\times l}\\ =& {\left(\sum _{k=1}^n\left(\frac{\partial u_{ik}}{\partial x}{v}_{kj}+u_{ik}\frac{\partial v_{kj}}{\partial x}\right)\right)}_{m\times l}\\ =& {\left(\sum _{k=1}^n\frac{\partial u_{ik}}{\partial x}{v}_{kj}\right)}_{m\times l}+{\left(\sum _{k=1}^n{u}_{ik}\frac{\partial v_{kj}}{\partial x}\right)}_{m\times l}\\ =& \frac{\partial U}{\partial x}V+U\frac{\partial V}{\partial x}.\end{array}$$

根据 MS4(分部) 再证 MS2(矩阵乘) 。
$$\begin{array}{rl}\frac{\partial (AUB)}{\partial x}=& \frac{\partial A}{\partial x}UB+A\left(\frac{\partial UB}{\partial x}\right)\\ =& \frac{\partial A}{\partial x}UB+A\left(\frac{\partial U}{\partial x}B+U\frac{\partial B}{\partial x}\right)\\ =& 0UB+A\frac{\partial U}{\partial x}B+AU0\\ =& A\frac{\partial U}{\partial x}B.\end{array}$$

2.对向量求导

2.1标量对向量求导

$$\frac{\partial y}{\partial \vec{x}}={\left(\begin{array}{c}\frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2}\\ \frac{\partial y}{\partial x_3}\\ ⋮\\ \frac{\partial y}{\partial x_n}\end{array}\right)}_{n\times 1}$$
上式俗称梯度。
根据全微分公式:
$$\mathrm{d}y=\sum _{i=1}^n\frac{\partial y}{\partial x_i}\mathrm{d}{x}_i=\left(\begin{array}{ccccc}\frac{\partial y}{\partial x_1},& \frac{\partial y}{\partial x_2},& \frac{\partial y}{\partial x_3},& \cdots ,& \frac{\partial y}{\partial x_n}\end{array}\right)\times \left(\begin{array}{c}\mathrm{d}{x}_1\\ \mathrm{d}{x}_2\\ \mathrm{d}{x}_3\\ ⋮\\ \mathrm{d}{x}_n\end{array}\right)={\left(\frac{\partial y}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}$$
性质

• SV1(数乘) ：对于$\forall u(x),a\in \mathbb{F}$, 有$\frac{\partial (au)}{\partial \vec{x}}=a\frac{\partial u}{\partial \vec{x}}$。
• SV2(线性)：对于$\forall u(x),v(x)\in \mathbb{F}$, 有$\frac{\partial (u+v)}{\partial \vec{x}}=\frac{\partial u}{\partial \vec{x}}+\frac{\partial v}{\partial \vec{x}}$。
• SV3(分部)：对于$\forall u(x),v(x)\in \mathbb{F}$, 有$\frac{\partial (uv)}{\partial x}=\frac{\partial u}{\partial \vec{x}}v+u\frac{\partial v}{\partial \vec{x}}$。
• SV4(链式)：对于$\forall u(x),g(u)\in \mathbb{F}$, 有$\frac{\partial g(u)}{\partial \vec{x}}=\frac{\partial g(u)}{\partial u}\frac{\partial u}{\partial \vec{x}}$。
• SV5：对于$\forall \vec{u}(\vec{x}),\vec{v}(\vec{x})\in {\mathbb{F}}^m$, 有$\frac{\partial ({\vec{u}}^T\vec{v})}{\partial \vec{x}}=\frac{\partial \vec{v}}{\partial \vec{x}}\vec{u}+\frac{\partial \vec{u}}{\partial \vec{x}}\vec{v}$。
• SV6：对于$\forall A\in {\mathbb{F}}^{m\times n},\vec{u}(\vec{x})\in {\mathbb{F}}^m,\vec{v}(\vec{x})\in {\mathbb{F}}^n$，有$\frac{\partial \left({\vec{u}}^{T}A\vec{v}\right)}{\partial \vec{x}}=\frac{\partial \vec{v}}{\partial \vec{x}}{A}^T\vec{u}+\frac{\partial \vec{u}}{\partial \vec{x}}A\vec{v}$。

首先对SV5做简要证明。左右两边都是$n\times 1$向量，只需证每行相等即可。对于第$i$行，
$$\begin{array}{rl}LH{S}_i=& \frac{\partial {\vec{u}}^T\vec{v}}{\partial x_i}=\frac{\partial }{\partial x_i}\left(\sum _{j=1}^m{u}_i{v}_i\right)\\ =& \sum _{j=1}^m\left(\frac{\partial u_i{v}_i}{\partial x_i}\right)\\ =& \sum _{j=1}^m\left(\frac{\partial v_i}{\partial x_i}{u}_i+\frac{\partial u_i}{\partial x_i}{v}_i\right).\end{array}$$

根据向量对向量求导，右边第$i$行为
$$\begin{array}{rl}RH{S}_i=& \left(\begin{array}{ccccc}\frac{\partial v_1}{\partial x_i},& \frac{\partial v_2}{\partial x_i},& \frac{\partial v_3}{\partial x_i},& \cdots ,& \frac{\partial v_m}{\partial x_i}\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\\ u_3\\ ⋮\\ u_m\end{array}\right)+\left(\begin{array}{ccccc}\frac{\partial u_1}{\partial x_i},& \frac{\partial u_2}{\partial x_i},& \frac{\partial u_3}{\partial x_i},& \cdots ,& \frac{\partial u_m}{\partial x_i}\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\\ ⋮\\ v_m\end{array}\right)\\ =& \sum _{j=1}^m\left(\frac{\partial v_i}{\partial x_i}{u}_i\right)+\sum _{j=1}^m\left(\frac{\partial u_i}{\partial x_i}{v}_i\right)\\ =& LH{S}_i.\end{array}$$

关于SV6，证明如下：
$$\frac{\partial \left({\vec{u}}^{T}A\vec{v}\right)}{\partial \vec{x}}\stackrel{SV5}{=}\frac{\partial A\vec{v}}{\partial \vec{x}}\vec{u}+\frac{\partial \vec{u}}{\partial \vec{x}}A\vec{v}\stackrel{VV3}{=}\frac{\partial \vec{v}}{\partial \vec{x}}{A}^T\vec{u}+\frac{\partial \vec{u}}{\partial \vec{x}}A\vec{v}.$$

2.2向量对向量求导

$$\frac{\partial y_i}{\partial \vec{x}}={\left(\begin{array}{c}\frac{\partial y_i}{\partial x_1}\\ \frac{\partial y_i}{\partial x_2}\\ \frac{\partial y_i}{\partial x_3}\\ ⋮\\ \frac{\partial y_i}{\partial x_n}\end{array}\right)}_{n\times 1},i=1,2,\cdots ,m.$$
因此
$$\frac{\partial \vec{y}}{\partial \vec{x}}=\left(\begin{array}{ccccc}\frac{\partial y_1}{\partial \vec{x}},& \frac{\partial y_2}{\partial \vec{x}},& \frac{\partial y_3}{\partial \vec{x}},& \cdots ,& \frac{\partial y_m}{\partial \vec{x}}\end{array}\right)={\left(\begin{array}{ccccc}\frac{\partial y_1}{\partial x_1}& \frac{\partial y_2}{\partial x_1}& \frac{\partial y_3}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2}& \frac{\partial y_2}{\partial x_2}& \frac{\partial y_3}{\partial x_2}& \cdots & \frac{\partial y_m}{\partial x_2}\\ \frac{\partial y_1}{\partial x_3}& \frac{\partial y_2}{\partial x_3}& \frac{\partial y_3}{\partial x_3}& \cdots & \frac{\partial y_m}{\partial x_3}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_1}{\partial x_n}& \frac{\partial y_2}{\partial x_n}& \frac{\partial y_3}{\partial x_n}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right)}_{n\times m}$$
由上面的标量对向量求导，可知$\mathrm{d}{y}_i={\left(\frac{\partial y_i}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}$.因此
$$\mathrm{d}\vec{y}=\left(\begin{array}{c}\mathrm{d}{y}_1\\ \mathrm{d}{y}_2\\ \mathrm{d}{y}_3\\ ⋮\\ \mathrm{d}{y}_m\end{array}\right)=\left[\begin{array}{c}{\left(\frac{\partial y_1}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}\\ {\left(\frac{\partial y_2}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}\\ {\left(\frac{\partial y_3}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}\\ ⋮\\ {\left(\frac{\partial y_m}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}\end{array}\right]=\left[\begin{array}{c}{\left(\frac{\partial y_1}{\partial \vec{x}}\right)}^T\\ {\left(\frac{\partial y_2}{\partial \vec{x}}\right)}^T\\ {\left(\frac{\partial y_3}{\partial \vec{x}}\right)}^T\\ ⋮\\ {\left(\frac{\partial y_m}{\partial \vec{x}}\right)}^T\end{array}\right]\mathrm{d}\vec{x}={\left(\begin{array}{ccccc}\frac{\partial y_1}{\partial x_1}& \frac{\partial y_2}{\partial x_1}& \frac{\partial y_3}{\partial x_1}& \cdots & \frac{\partial y_m}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2}& \frac{\partial y_2}{\partial x_2}& \frac{\partial y_3}{\partial x_2}& \cdots & \frac{\partial y_m}{\partial x_2}\\ \frac{\partial y_1}{\partial x_3}& \frac{\partial y_2}{\partial x_3}& \frac{\partial y_3}{\partial x_3}& \cdots & \frac{\partial y_m}{\partial x_3}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_1}{\partial x_n}& \frac{\partial y_2}{\partial x_n}& \frac{\partial y_3}{\partial x_n}& \cdots & \frac{\partial y_m}{\partial x_n}\end{array}\right)}^T\mathrm{d}\vec{x}={\left(\frac{\partial \vec{y}}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}$$

• VV1(数乘)：对于$\forall \vec{u}(\vec{x})\in {\mathbb{F}}^m,a(\vec{x})\in \mathbb{F}$, 有$\frac{\partial (a\vec{u})}{\partial \vec{x}}=a\frac{\partial \vec{u}}{\partial \vec{x}}+\frac{\partial a}{\partial \vec{x}}{\vec{u}}^T$。
• VV2(线性)：对于$\forall \vec{u}(\vec{x}),\vec{v}(\vec{x})\in {\mathbb{F}}^m$, 有$\frac{\partial (\vec{u}+\vec{v})}{\partial \vec{x}}=\frac{\partial \vec{u}}{\partial \vec{x}}+\frac{\partial \vec{v}}{\partial \vec{x}}$。
• VV3(乘矩阵)：对于$\forall \vec{u}(\vec{x})\in {\mathbb{F}}^m,A\in {\mathbb{F}}^{p\times m}$, 有$\frac{\partial (A\vec{u})}{\partial \vec{x}}=\frac{\partial \vec{u}}{\partial \vec{x}}{A}^T$。
• VV4(链式)：对于$\forall \vec{u}(\vec{x})\in {\mathbb{F}}^p,\vec{g}(\vec{u})\in {\mathbb{F}}^q$, 有$\frac{\partial \vec{g}(\vec{u})}{\partial \vec{x}}=\frac{\partial \vec{u}}{\partial \vec{x}}\frac{\partial \vec{g}}{\partial \vec{u}}$。

证VV1(数乘)，同样为了书写方便记$\frac{\partial \vec{u}}{\vec{x}}={\left(\frac{\partial u_j}{\partial x_i}\right)}_{n\times m}$。
$$\begin{array}{rl}\frac{\partial (a\vec{u})}{\partial \vec{x}}=& {\left(\frac{\partial a{u}_j}{\partial x_i}\right)}_{n\times m}\\ =& {\left(a\frac{\partial u_j}{\partial x_i}+\frac{\partial a}{\partial x_i}{u}_j\right)}_{n\times m}\\ =& {\left(a\frac{\partial u_j}{\partial x_i}\right)}_{n\times m}+{\left(\frac{\partial a}{\partial x_i}{u}_j\right)}_{n\times m}\\ =& a{\left(\frac{\partial u_j}{\partial x_i}\right)}_{n\times m}+\left(\begin{array}{c}\frac{\partial a}{\partial x_1}\\ \frac{\partial a}{\partial x_2}\\ \frac{\partial a}{\partial x_3}\\ ⋮\\ \frac{\partial a}{\partial x_n}\end{array}\right)\left(\begin{array}{cccc}{u}_1,& u_2,& u_3,& \cdots ,u_m\end{array}\right)\\ =& a\frac{\partial \vec{u}}{\partial \vec{x}}+\frac{\partial a}{\partial \vec{x}}{\vec{u}}^T.\end{array}$$
证VV3(乘矩阵)，向量$A\vec{u}$记成${\left({\sum }_{k=1}^m{a}_{jk}{u}_k\right)}_p$，即其第$j$行元素为${\sum }_{k=1}^m{a}_{jk}{u}_k$。
根据向量对向量求导的特点，可以得到
$$\frac{\partial }{\partial \vec{x}}\left(\sum _{k=1}^m{a}_{jk}{u}_k\right)=\left(\begin{array}{c}\frac{\partial }{\partial x_1}\left({\sum }_{k=1}^m{a}_{jk}{u}_k\right)\\ \frac{\partial }{\partial x_2}\left({\sum }_{k=1}^m{a}_{jk}{u}_k\right)\\ \frac{\partial }{\partial x_3}\left({\sum }_{k=1}^m{a}_{jk}{u}_k\right)\\ ⋮\\ \frac{\partial }{\partial x_n}\left({\sum }_{k=1}^m{a}_{jk}{u}_k\right)\end{array}\right)$$
因此LHS可以写为
$$\frac{\partial (A\vec{u})}{\partial \vec{x}}={\left(\frac{\partial }{\partial x_i}\left(\sum _{k=1}^m{a}_{jk}{u}_k\right)\right)}_{n\times p}={\left(\sum _{k=1}^m\left(\frac{\partial u_k}{\partial x_i}{a}_{jk}\right)\right)}_{n\times p}.$$
即$LH{S}_{i,j}={\sum }_{k=1}^m\left(\frac{\partial u_k}{\partial x_i}{a}_{jk}\right)$。
现在考虑$RH{S}_{i,j}$，它是由$\frac{\partial \vec{u}}{\partial \vec{x}}$的第$i$行乘$A^T$的第$j$列得到的。
$$RH{S}_{i,j}=\left(\frac{\partial u_1}{\partial x_i},\frac{\partial u_2}{\partial x_i},\cdots ,\frac{\partial u_m}{\partial x_i}\right)\left(\begin{array}{c}{a}_{j1}\\ a_{j2}\\ ⋮\\ a_{jm}\end{array}\right)=\sum _{k=1}^m\left(\frac{\partial u_k}{\partial x_i}{a}_{jk}\right)=LH{S}_{i,j}.$$

最后证VV4(链式)。
$$\frac{\partial \vec{u}}{\partial \vec{x}}\frac{\partial \vec{g}}{\partial \vec{u}}={\left(\frac{\partial u_j}{\partial x_i}\right)}_{n\times p}{\left(\frac{\partial g_k}{\partial u_j}\right)}_{p\times q}={\left(\sum _{j=1}^p\left(\frac{\partial u_j}{\partial x_i}\frac{\partial g_k}{\partial u_j}\right)\right)}_{n\times q}$$
即$RH{S}_{i,k}={\sum }_{j=1}^p\left(\frac{\partial u_j}{\partial x_i}\frac{\partial g_k}{\partial u_j}\right)$。
而$\frac{\partial \vec{g}(\vec{u})}{\partial \vec{x}}={\left(\frac{\partial g_k}{\partial x_i}\right)}_{n\times q}$，即
$$\begin{array}{rl}LH{S}_{i,k}=& \frac{\partial g_k}{\partial x_i}\\ \stackrel{SS3}{=}& \frac{\partial g_k}{\partial u_1}\frac{\partial u_1}{\partial x_i}+\frac{\partial g_k}{\partial u_2}\frac{\partial u_2}{\partial x_i}+\cdots +\frac{\partial g_k}{\partial u_p}\frac{\partial u_p}{\partial x_i}\\ =& \sum _{j=1}^p\left(\frac{\partial u_j}{\partial x_i}\frac{\partial g_k}{\partial u_j}\right)\\ =& RH{S}_{i,k}.\end{array}$$

2.3矩阵对向量求导

首先将矩阵$Y$按列优先向量化，即
$$vec(Y_{p\times q})=vec\left({\left(\begin{array}{ccccc}{y}_{11}& y_{12}& y_{13}& \cdots & y_{1q}\\ y_{21}& y_{22}& y_{23}& \cdots & y_{2q}\\ y_{31}& y_{32}& y_{33}& \cdots & y_{3q}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ y_{p1}& y_{p2}& y_{p3}& \cdots & y_{pq}\end{array}\right)}_{p\times q}\right)={\left({\vec{y}}_1,{\vec{y}}_2,{\vec{y}}_3,\cdots ,{\vec{y}}_q\right)}^T={\left(\begin{array}{c}{y}_{11}\\ y_{21}\\ ⋮\\ y_{p1}\\ y_{12}\\ y_{22}\\ ⋮\\ y_{p2}\\ ⋮\\ ⋮\\ y_{1q}\\ y_{2q}\\ ⋮\\ y_{pq}\end{array}\right)}_{pq\times 1}.$$
根据向量对向量求导，有
$$\frac{\partial {\vec{y}}_i}{\partial \vec{x}}=\left(\begin{array}{ccccc}\frac{\partial y_{1i}}{\partial \vec{x}},& \frac{\partial y_{2i}}{\partial \vec{x}},& \frac{\partial y_{3i}}{\partial \vec{x}},& \cdots ,& \frac{\partial y_{pi}}{\partial \vec{x}}\end{array}\right)={\left(\begin{array}{ccccc}\frac{\partial y_{1i}}{\partial x_1}& \frac{\partial y_{2i}}{\partial x_1}& \frac{\partial y_{3i}}{\partial x_1}& \cdots & \frac{\partial y_{pi}}{\partial x_1}\\ \frac{\partial y_{1i}}{\partial x_2}& \frac{\partial y_{2i}}{\partial x_2}& \frac{\partial y_{3i}}{\partial x_2}& \cdots & \frac{\partial y_{pi}}{\partial x_2}\\ \frac{\partial y_{1i}}{\partial x_3}& \frac{\partial y_{2i}}{\partial x_3}& \frac{\partial y_{3i}}{\partial x_3}& \cdots & \frac{\partial y_{pi}}{\partial x_3}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y_{1i}}{\partial x_n}& \frac{\partial y_{2i}}{\partial x_n}& \frac{\partial y_{3i}}{\partial x_n}& \cdots & \frac{\partial y_{pi}}{\partial x_n}\end{array}\right)}_{n\times p}$$
因此
$$\begin{array}{rl}\frac{\partial vec(Y)}{\partial \vec{x}}=& \left(\begin{array}{cccccccccc}\frac{\partial y_{11}}{\partial \vec{x}},& \frac{\partial y_{21}}{\partial \vec{x}},& \cdots ,& \frac{\partial y_{p1}}{\partial \vec{x}},& \frac{\partial y_{22}}{\partial \vec{x}},& \cdots ,& \frac{\partial y_{p2}}{\partial \vec{x}},& \cdots ,& \cdots ,& \frac{\partial y_{pq}}{\partial \vec{x}}\end{array}\right)\\ =& {\left(\begin{array}{cccccccccc}\frac{\partial y_{11}}{\partial x_1},& \frac{\partial y_{21}}{\partial x_1},& \cdots ,& \frac{\partial y_{p1}}{\partial x_1},& \frac{\partial y_{22}}{\partial x_1},& \cdots ,& \frac{\partial y_{p2}}{\partial x_1},& \cdots ,& \cdots ,& \frac{\partial y_{pq}}{\partial x_1}\\ \frac{\partial y_{11}}{\partial x_2},& \frac{\partial y_{21}}{\partial x_2},& \cdots ,& \frac{\partial y_{p1}}{\partial x_2},& \frac{\partial y_{22}}{\partial x_2},& \cdots ,& \frac{\partial y_{p2}}{\partial x_2},& \cdots ,& \cdots ,& \frac{\partial y_{pq}}{\partial x_2}\\ \frac{\partial y_{11}}{\partial x_3},& \frac{\partial y_{21}}{\partial x_3},& \cdots ,& \frac{\partial y_{p1}}{\partial x_3},& \frac{\partial y_{22}}{\partial x_3},& \cdots ,& \frac{\partial y_{p2}}{\partial x_3},& \cdots ,& \cdots ,& \frac{\partial y_{pq}}{\partial x_3}\\ ⋮& ⋮& \ddots ,& ⋮& ⋮& \ddots ,& ⋮& \ddots ,& \cdots ,& \frac{\partial y_{pq}}{\partial x_2}\\ \frac{\partial y_{11}}{\partial x_n},& \frac{\partial y_{21}}{\partial x_n},& \cdots ,& \frac{\partial y_{p1}}{\partial x_n},& \frac{\partial y_{22}}{\partial x_n},& \cdots ,& \frac{\partial y_{p2}}{\partial x_n},& \cdots ,& \cdots ,& \frac{\partial y_{pq}}{\partial x_n}\end{array}\right)}_{n\times pq}\end{array}$$
得
$$vec(\mathrm{d}Y)={\left(\frac{\partial vec(Y)}{\partial \vec{x}}\right)}^T\mathrm{d}\vec{x}$$

3.对矩阵求导

3.1标量对矩阵求导

$$\frac{\partial y}{\partial X}=\left(\begin{array}{cccc}\frac{\partial y}{\partial {\vec{x}}_1},& \frac{\partial y}{\partial {\vec{x}}_2},& \cdots ,& \frac{\partial y}{\partial {\vec{x}}_s}\end{array}\right)={\left(\begin{array}{ccccc}\frac{\partial y}{\partial x_{11}}& \frac{\partial y}{\partial x_{12}}& \frac{\partial y}{\partial x_{13}}& \cdots & \frac{\partial y}{\partial x_{1s}}\\ \frac{\partial y}{\partial x_{21}}& \frac{\partial y}{\partial x_{22}}& \frac{\partial y}{\partial x_{23}}& \cdots & \frac{\partial y}{\partial x_{2s}}\\ \frac{\partial y}{\partial x_{31}}& \frac{\partial y}{\partial x_{32}}& \frac{\partial y}{\partial x_{33}}& \cdots & \frac{\partial y}{\partial x_{3s}}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y}{\partial x_{r1}}& \frac{\partial y}{\partial x_{r2}}& \frac{\partial y}{\partial x_{r3}}& \cdots & \frac{\partial y}{\partial x_{rs}}\end{array}\right)}_{r\times s}$$
同样的，由全微分公式有$\mathrm{d}y={\sum }_{i=1}^r{\sum }_{j=1}^s\frac{\partial y}{\partial x_{ij}}\mathrm{d}{x}_{ij}$。
$$\begin{array}{rl}{\left(\frac{\partial y}{\partial X}\right)}^T\mathrm{d}X=& {\left(\begin{array}{ccccc}\frac{\partial y}{\partial x_{11}}& \frac{\partial y}{\partial x_{21}}& \frac{\partial y}{\partial x_{31}}& \cdots & \frac{\partial y}{\partial x_{r1}}\\ \frac{\partial y}{\partial x_{12}}& \frac{\partial y}{\partial x_{22}}& \frac{\partial y}{\partial x_{32}}& \cdots & \frac{\partial y}{\partial x_{r2}}\\ \frac{\partial y}{\partial x_{13}}& \frac{\partial y}{\partial x_{23}}& \frac{\partial y}{\partial x_{33}}& \cdots & \frac{\partial y}{\partial x_{r3}}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \frac{\partial y}{\partial x_{1s}}& \frac{\partial y}{\partial x_{2s}}& \frac{\partial y}{\partial x_{3s}}& \cdots & \frac{\partial y}{\partial x_{rs}}\end{array}\right)}_{s\times r}\times {\left(\begin{array}{ccccc}\mathrm{d}{x}_{11}& \mathrm{d}{x}_{12}& \mathrm{d}{x}_{13}& \cdots & \mathrm{d}{x}_{1s}\\ \mathrm{d}{x}_{21}& \mathrm{d}{x}_{22}& \mathrm{d}{x}_{23}& \cdots & \mathrm{d}{x}_{2s}\\ \mathrm{d}{x}_{31}& \mathrm{d}{x}_{32}& \mathrm{d}{x}_{33}& \cdots & \mathrm{d}{x}_{3s}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \mathrm{d}{x}_{r1}& \mathrm{d}{x}_{r2}& \mathrm{d}{x}_{r3}& \cdots & \mathrm{d}{x}_{rs}\end{array}\right)}_{r\times s}\\ =& {\left(\begin{array}{ccccc}{\sum }_{i=1}^r\frac{\partial y}{\partial x_{i1}}\mathrm{d}{x}_{i1}& \cdots & \cdots & \cdots & \cdots \\ \cdots & {\sum }_{i=1}^r\frac{\partial y}{\partial x_{i2}}\mathrm{d}{x}_{i2}& \cdots & \cdots & \cdots \\ \cdots & \cdots & {\sum }_{i=1}^r\frac{\partial y}{\partial x_{i3}}\mathrm{d}{x}_{i3}& \cdots & \cdots \\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \cdots & \cdots & \cdots & \cdots & {\sum }_{i=1}^r\frac{\partial y}{\partial x_{is}}\mathrm{d}{x}_{is}\end{array}\right)}_{s\times s}\end{array}$$
因此
$$\begin{array}{rlrl}tr\left({\left(\frac{\partial y}{\partial X}\right)}^T\mathrm{d}X\right)=& \sum _{j=1}^s\sum _{i=1}^r\frac{\partial y}{\partial x_{ij}}\mathrm{d}{x}_{ij}=& \sum _{i=1}^r\sum _{j=1}^s\frac{\partial y}{\partial x_{ij}}\mathrm{d}{x}_{ij}=& \mathrm{d}y\end{array}$$

3.2向量对矩阵求导

$$\mathrm{d}\vec{y}=\left(\begin{array}{c}\mathrm{d}{y}_1\\ \mathrm{d}{y}_2\\ ⋮\\ \mathrm{d}{y}_m\end{array}\right)=\left(\begin{array}{c}tr\left({\left(\frac{\partial y_1}{\partial X}\right)}^T\mathrm{d}X\right)\\ tr\left({\left(\frac{\partial y_2}{\partial X}\right)}^T\mathrm{d}X\right)\\ ⋮\\ tr\left({\left(\frac{\partial y_m}{\partial X}\right)}^T\mathrm{d}X\right)\end{array}\right)$$

3.3矩阵对矩阵求导

如果采用向量对矩阵求导，我们有
$$\begin{array}{rl}\mathrm{d}Y=& \left(\begin{array}{cccc}\mathrm{d}{\vec{y}}_1,& \mathrm{d}{\vec{y}}_2,& \cdots ,& \mathrm{d}{\vec{y}}_q\end{array}\right)\\ =& {\left(\begin{array}{ccccc}tr\left({\left(\frac{\partial y_{11}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{12}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{13}}{\partial X}\right)}^T\mathrm{d}X\right)& \cdots & tr\left({\left(\frac{\partial y_{1q}}{\partial X}\right)}^T\mathrm{d}X\right)\\ tr\left({\left(\frac{\partial y_{21}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{22}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{23}}{\partial X}\right)}^T\mathrm{d}X\right)& \cdots & tr\left({\left(\frac{\partial y_{2q}}{\partial X}\right)}^T\mathrm{d}X\right)\\ tr\left({\left(\frac{\partial y_{31}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{32}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{33}}{\partial X}\right)}^T\mathrm{d}X\right)& \cdots & tr\left({\left(\frac{\partial y_{3q}}{\partial X}\right)}^T\mathrm{d}X\right)\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ tr\left({\left(\frac{\partial y_{p1}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{p2}}{\partial X}\right)}^T\mathrm{d}X\right)& tr\left({\left(\frac{\partial y_{p3}}{\partial X}\right)}^T\mathrm{d}X\right)& \cdots & tr\left({\left(\frac{\partial y_{pq}}{\partial X}\right)}^T\mathrm{d}X\right)\end{array}\right)}_{r\times s}\end{array}$$
当然，如果采用将矩阵向量化，则有
$$vec(\mathrm{d}Y)={\left(\frac{\partial vec(Y)}{\partial vec(x)}\right)}^{T}vec(\mathrm{d}X)$$

矩阵微分算子[2]

• D1(线性)：$\mathrm{d}\left(X\pm Y\right)=\mathrm{d}X\pm \mathrm{d}Y$。
• D2(矩阵乘法)：$\mathrm{d}\left(XY\right)=(\mathrm{d}X)Y+X(\mathrm{d}Y)$。
• D3(转置)：$\mathrm{d}(X^T)=(\mathrm{d}X{)}^T$。
• D4(迹)：$\mathrm{d}\left(tr(X)\right)=tr(\mathrm{d}X)$。
• D5(逆)：若$X$可逆，$\mathrm{d}(X^{-1})=(\mathrm{d}X{)}^{-1}$。
• D6(行列式)：$\mathrm{d}|X|=tr\left(X^{adjugate}(\mathrm{d}X)\right)$,若$X$可逆，则$\mathrm{d}|X|=|X|tr\left(X^{-1}\mathrm{d}X\right)$。
• D7(逐元素乘法)：$\mathrm{d}\left(X\odot Y\right)=(\mathrm{d}X)\odot Y+X\odot (\mathrm{d}Y)$。
• D7(逐元素函数)：$\mathrm{d}f(X)=f^{{}^{\prime }}(X)\odot (\mathrm{d}X)$。

4.行列式对矩阵求导

行列式对矩阵求导，同样也属于标量对矩阵求导类型。

• DM1：对于$\forall X\in {\mathbb{F}}^{n\times n},A\in {\mathbb{F}}^{p\times n},B\in {\mathbb{F}}^{n\times q}$, 有$\frac{\partial |AXB|}{\partial X}=|AXB|(X^{-1}{)}^T$。
• DM2：对于$\forall X\in {\mathbb{F}}^{n\times n}$, 有$\frac{\partial ln(|X|)}{\partial X}=(X^{-1}{)}^T$。
• DM3：对于$\forall X(z)\in {\mathbb{F}}^{n\times n},z\in \mathbb{F}$, 有$\frac{\partial ln(|X(z)|)}{\partial z}=tr\left(X^{-1}\frac{\partial X}{\partial z}\right)$。
• DM4：对于$\forall X\in {\mathbb{F}}^{n\times m},A\in {\mathbb{F}}^{n\times n}$, 有$\frac{\partial |X^{T}AX|}{\partial X}=|X^{T}AX|\left(AX{\left(X^{T}AX\right)}^{-1}+A^{T}X{\left(X^T{A}^{T}X\right)}^{-1}\right)$。

5.迹对矩阵求导

迹对矩阵求导，本质上属于标量对矩阵求导类型。
迹的性质

• TR1(标量)：对于$\forall a\in {\mathbb{F}}^1$, 都有$a=tr(a)$。
• TR2(转置)：对于$\forall A\in {\mathbb{F}}^{m\times n}$, 都有$tr(A)=tr(A^T)$。
• TR3(线性)：对于$\forall A,B\in {\mathbb{F}}^{m\times n}$, 都有$tr(A\pm B)=tr(A)\pm tr(B)$。
• TR4(对矩阵乘法交换律)：对于$\forall A\in {\mathbb{F}}^{m\times n},B\in {\mathbb{F}}^{n\times m}$, 都有$tr(AB)=tr(BA)$。
• TR5(对矩阵乘法/逐元素乘法交换律)：对于$\forall A,B,C\in {\mathbb{F}}^{m\times n}$, 有$tr\left(A^T(B\odot C)\right)=tr\left((A\odot B{)}^{T}C\right)$。

常用的函数[2]

$f$	$\mathrm{d}f$	$\frac{\partial f}{\partial X}$
$tr(X)$	$tr(I\mathrm{d}X)$	$I$
$tr(X^T)$	$2tr(X^T\mathrm{d}X)$	$2X$
$tr(X^2)$	$2tr(X\mathrm{d}X)$	$2X^T$
$tr(AX)$	$tr(A\mathrm{d}X)$	$A^T$
$tr(X^{T}AX)$	$tr(X^T(A+A^T)\mathrm{d}X)$	$(A+A^T)X$
$tr(XA{X}^T)$	$tr((A+A^T)X^T\mathrm{d}X)$	$X(A+A^T)$
$tr(XAX)$	$tr((AX+XA)\mathrm{d}X)$	$X^T{A}^T+A^T{X}^T$
$tr(A{X}^{-1})$	$-tr(X^{-1}A{X}^{-1}\mathrm{d}X)$	$-{\left(X^{-1}A{X}^{-1}\right)}^T$
$tr(XAXB)$	$tr((AXB+BXA)\mathrm{d}X)$	${\left(AXB+BXA\right)}^T$
$tr(XA{X}^{T}B)$	$tr((A{X}^{T}B+A^T{X}^T{B}^T)\mathrm{d}X)$	$B^{T}X{A}^T+BXA$

• TM1：对于$\forall X\in {\mathbb{F}}^{n\times n}$,
  有$\frac{\partial tr(X)}{\partial X}=I$。
• TM2：对于$\forall X\in {\mathbb{F}}^{n\times m},A\in {\mathbb{F}}^{m\times n}$,有$\frac{\partial tr(XA)}{\partial X}=\frac{\partial tr(AX)}{\partial X}=A^T$。
• TM3：对于$\forall X\in {\mathbb{F}}^{n\times m},A\in {\mathbb{F}}^{n\times n}$, 有$\frac{\partial tr(X^{T}AX)}{\partial X}=(A+A^T)X$。
• TM4：对于$\forall X,A\in {\mathbb{F}}^{n\times n}$, 有$\frac{\partial tr(X^{-1}A)}{\partial X}=-X^{-1}{A}^T{X}^{-1}$。

这里为了描述方便，用$x_{i_X,j_X}$表示矩阵$X$的第$i$、$j$列元素。

证TM1，
$$\begin{array}{rl}\frac{\partial tr(X)}{\partial X}=& {\left(\frac{\partial tr(X)}{\partial x_{i_X,j_X}}\right)}_{n\times n}\\ =& {\left(\frac{\partial }{\partial x_{i_X,j_X}}\left(\sum _{i=1}^n{x}_{i,i}\right)\right)}_{n\times n}\\ =& {\left(\frac{\partial x_{i_X,i_X}}{\partial x_{i_X,j_X}}\right)}_{n\times n}\\ =& I.\end{array}$$

证TM2，
$$\begin{array}{rl}\frac{\partial tr(XA)}{\partial X}=& {\left(\frac{\partial tr(XA)}{\partial x_{i_X,j_X}}\right)}_{n\times m}\\ =& {\left(\frac{\partial }{\partial x_{i_X,j_X}}\left(\sum _{i=1}^n\sum _{k=1}^m{x}_{i,k}{a}_{k,i}\right)\right)}_{n\times m}\\ =& {\left(\frac{\partial x_{i_X,j_X}{a}_{j_X,i_X}}{\partial x_{i_X,j_X}}\right)}_{n\times m}\\ =& {\left(a_{j_X,i_X}\right)}_{n\times m}\\ =& A^T.\end{array}$$

证TM3，记
$$X={\left(x_{i_{X1},j_{X1}}\right)}_{n\times m},X^T={\left(x_{j_{X1},i_{X1}}\right)}_{m\times n},A={\left(a_{j_A,i_A}\right)}_{n\times n},$$
$$X^{T}A={\left(\sum _{i=1}^n{x}_{j_{X1},i}{a}_{i,j_A}\right)}_{m\times n},X^{T}AX={\left(\sum _{j=1}^n\sum _{i=1}^n{x}_{j_{X1},i}{a}_{i,j}{x}_{j,j_{X2}}\right)}_{m\times m}.$$
$$AX={\left(\sum _{i=1}^n{a}_{i_A,i}{x}_{i,j_X}\right)}_{n\times m},A^{T}X={\left(\sum _{i=1}^n{a}_{i,j_A}{x}_{i,j_X}\right)}_{n\times m},$$
因此$$tr(X^{T}AX)=\sum _{k=1}^m\sum _{j=1}^n\sum _{i=1}^n{x}_{k,i}{a}_{i,j}{x}_{j,k}.$$
$$\begin{array}{rl}\frac{\partial tr(X^{T}AX)}{\partial X}=& {\left(\frac{\partial tr(X^{T}AX)}{\partial x_{i_X,j_X}}\right)}_{n\times m}\\ =& {\left(\frac{\partial }{\partial x_{i_X,j_X}}\left(\sum _{k=1}^m\sum _{j=1}^n\sum _{i=1}^n{x}_{k,i}{a}_{i,j}{x}_{j,k}\right)\right)}_{n\times m}\\ =& {\left(\frac{\partial }{\partial x_{i_X,j_X}}\left(\sum _{j=1}^n{x}_{i_X,j_X}{a}_{j_X,j}{x}_{j,i_X}\right)+\frac{\partial }{\partial x_{i_X,j_X}}\left(\sum _{i=1}^n{x}_{j_X,i}{a}_{i,i_X}{x}_{i_X,j_X}\right)\right)}_{n\times m}\\ =& {\left(\left(\sum _{j=1}^n{a}_{j_X,j}{x}_{j,i_X}\right)+\left(\sum _{i=1}^n{x}_{j_X,i}{a}_{i,i_X}\right)\right)}_{n\times m}\\ =& AX+A^{T}X\\ =& (A+A^T)X.\end{array}$$

6.例题

这里的例题均摘录自[3]。

【例1】，$f={\vec{a}}^{T}X\vec{b},\vec{a}\in {\mathbb{F}}^{m\times 1},X\in {\mathbb{F}}^{m\times n},\vec{b}\in {\mathbb{F}}^{n\times 1}$， 求$\frac{\partial f}{\partial X}$。

【解】 $\frac{\partial f}{\partial X}=\frac{\partial tr\left({\vec{a}}^{T}X\vec{b}\right)}{\partial X}\stackrel{TR4}{=}\frac{\partial tr\left(\vec{b}{\vec{a}}^{T}X\right)}{\partial X}\stackrel{TM2}{=}\vec{a}{\vec{b}}^T$。

【例2】 $f={\vec{a}}^{T}exp(X\vec{b}),\vec{a}\in {\mathbb{F}}^{m\times 1},X\in {\mathbb{F}}^{m\times n},\vec{b}\in {\mathbb{F}}^{n\times 1}$，求$\frac{\partial f}{\partial X}$。

【解】 先采用微分算子操作$\mathrm{d}f={\vec{a}}^T\left(exp(X\vec{b})\odot (\mathrm{d}X\vec{b})\right)$。

两边取迹，然后凑成TM2形式。
$$\begin{array}{rl}\mathrm{d}f=& tr\left({\vec{a}}^T\left(exp(X\vec{b})\odot (\mathrm{d}X\vec{b})\right)\right)\\ \stackrel{TR5}{=}& tr\left({\left(\vec{a}\odot exp(X\vec{b})\right)}^T(\mathrm{d}X\vec{b})\right)\\ \stackrel{TR4}{=}& tr\left(\vec{b}{\left(\vec{a}\odot exp(X\vec{b})\right)}^T\mathrm{d}X\right)\end{array}$$
得到$\frac{\partial f}{\partial X}={\left(\vec{b}{\left(\vec{a}\odot exp(X\vec{b})\right)}^T\right)}^T=\left(\vec{a}\odot exp(X\vec{b})\right){\vec{b}}^T$。

【例3】 $f=tr\left(Y^{T}MY\right),Y=\sigma \left(WX\right)$，求$\frac{\partial f}{\partial X}$。其中$W\in {\mathrm{F}}^{l\times m},X\in {\mathrm{F}}^{m\times n},Y\in {\mathrm{F}}^{l\times n},M\in {\mathrm{F}}^{l\times l}$， $\sigma$是逐元素函数，$f$是标量。

【解】 先求$\frac{\partial f}{\partial Y}$部分，
$$\frac{\partial f}{\partial Y}=\left(M+M^T\right)Y.$$
得到$\mathrm{d}f$与$\mathrm{d}Y$的关系$\mathrm{d}f=tr\left({\frac{\partial f}{\partial Y}}^T\mathrm{d}Y\right)=tr\left(Y^T\left(M+M^T\right)\mathrm{d}Y\right)$。

再求$\mathrm{d}Y$，
$$\begin{array}{rl}\mathrm{d}Y\stackrel{D7}{=}& {\sigma }^{{}^{\prime }}(WX)\odot \mathrm{d}(WX)\\ =& {\sigma }^{{}^{\prime }}(WX)\odot \left(W\mathrm{d}X\right).\end{array}$$
合并得到
$$\begin{array}{rl}\mathrm{d}f& =tr\left(Y^T\left(M+M^T\right){\sigma }^{{}^{\prime }}(WX)\odot \left(W\mathrm{d}X\right)\right)\\ \stackrel{TR5}{=}& tr\left({\left((M+M^T)Y\odot {\sigma }^{{}^{\prime }}(WX)\right)}^{T}W\mathrm{d}X\right).\end{array}$$
得$\frac{\partial f}{\partial X}=W^T\left((M+M^T)Y\odot {\sigma }^{{}^{\prime }}(WX)\right)$。

【例4】 $l=\Vert X\vec{w}-\vec{y}{\Vert }^2,\vec{y}\in {\mathbb{F}}^{m\times 1},X\in {\mathbb{F}}^{m\times n},\vec{w}\in {\mathbb{F}}^{m\times 1}$，求$\vec{w}$的最小二乘估计。

【解】
$$\begin{array}{rl}l=& \Vert X\vec{w}-\vec{y}{\Vert }^2\\ =& {\left(X\vec{w}-\vec{y}\right)}^T\left(X\vec{w}-\vec{y}\right)\\ =& \left({\vec{w}}^T{X}^T-{\vec{y}}^T\right)\left(X\vec{w}-\vec{y}\right)\\ =& {\vec{w}}^T{X}^{T}X\vec{w}-{\vec{w}}^T{X}^T\vec{y}-{\vec{y}}^{T}X\vec{w}+{\vec{y}}^T\vec{y}.\end{array}$$

$$\begin{array}{rl}\frac{\partial l}{\partial \vec{w}}=& \frac{\partial tr(l)}{\partial \vec{w}}\\ =& \frac{\partial }{\partial \vec{w}}tr\left({\vec{w}}^T{X}^{T}X\vec{w}-{\vec{w}}^T{X}^T\vec{y}-{\vec{y}}^{T}X\vec{w}+{\vec{y}}^T\vec{y}\right)\\ =& \frac{\partial }{\partial \vec{w}}tr\left({\vec{w}}^T{X}^{T}X\vec{w}\right)-\frac{\partial }{\partial \vec{w}}tr\left(2{\vec{y}}^{T}X\vec{w}\right)\\ =& 2(X^{T}X)\vec{w}-2X^T\vec{y}\\ =& 0.\end{array}$$
得$\vec{w}=(X^{T}X{)}^{-1}{X}^T\vec{y}$。

【例5】 样本${\vec{x}}_1,\cdots ,{\vec{x}}_N\sim \mathcal{N}\left(\vec{\mu },\Sigma \right)$，
求方差$\Sigma$的极大似然估计。

【解】 对数似然函数为$l=ln|\Sigma |+\frac{1}{N}{\sum }_{i=1}^N{\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)}^T{\Sigma }^{-1}\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right).$

因此
$$\begin{array}{rl}\frac{\partial l}{\partial \Sigma }=& \frac{\partial }{\partial \Sigma }\left(ln|\Sigma |+\frac{1}{N}\sum _{i=1}^N{\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)}^T{\Sigma }^{-1}\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)\right)\\ \stackrel{DM2}{=}& {\left({\Sigma }^{-1}\right)}^T+\frac{\partial }{\partial \Sigma }tr\left(\frac{1}{N}\sum _{i=1}^N{\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)}^T{\Sigma }^{-1}\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)\right)\\ =& {\left({\Sigma }^{-1}\right)}^T+\frac{1}{N}\sum _{i=1}^N\frac{\partial }{\partial \Sigma }tr\left(\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right){\left({\Sigma }^{-1}\right)}^T{\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)}^T\right)\\ =& {\left({\Sigma }^{-1}\right)}^T+\frac{1}{N}\sum _{i=1}^N\frac{\partial }{\partial \Sigma }tr\left({\left({\Sigma }^{-1}\right)}^T{\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)}^T\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)\right)\\ =& {\left({\Sigma }^{-1}\right)}^T-\frac{1}{N}\sum _{i=1}^N\left({\left({\Sigma }^{-1}\right)}^T\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right){\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)}^T{\left({\Sigma }^{-1}\right)}^T\right)\\ =& {\left({\Sigma }^{-1}\right)}^T-{\left({\Sigma }^{-1}\right)}^T\left(\frac{1}{N}\sum _{i=1}^N\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right){\left({\vec{x}}_i-\overline{\stackrel{⃗}{x}}\right)}^T\right){\left({\Sigma }^{-1}\right)}^T\\ =& {\left({\Sigma }^{-1}\right)}^T-{\left({\Sigma }^{-1}\right)}^T{S}^2{\left({\Sigma }^{-1}\right)}^T\\ =& {\left({\Sigma }^{-1}-{\Sigma }^{-1}{S}^2{\Sigma }^{-1}\right)}^T\\ =& 0.\end{array}$$
得到方差估计$\Sigma =S^2$。

【例6】 $l=-{\vec{y}}^{T}logsoftmax(W\vec{x}),\vec{y}\in {\mathbb{F}}^{m\times 1},W\in {\mathbb{F}}^{m\times n},\vec{x}\in {\mathbb{F}}^{n\times 1}$。求$\frac{\partial l}{\partial W}$。其中$\vec{y}$只有一个元素为$1$，其他都是$0$。

【解】 首先，对于$\vec{u}\in {\mathbb{F}}^{n\times 1},c\in {\mathbb{F}}^1$，
有$log(\frac{\vec{u}}{c})=log(\vec{u})-\vec{1}log(c)$。
因此
$$\begin{array}{rl}l=& -{\vec{y}}^{T}logsoftmax(W\vec{x})\\ =& -{\vec{y}}^{T}log\left(\frac{exp(W\vec{x})}{{\vec{1}}^{T}exp(W\vec{x})}\right)\\ =& -{\vec{y}}^T\left(W\vec{x}-\vec{1}log\left({\vec{1}}^{T}exp(W\vec{x})\right)\right)\\ =& -{\vec{y}}^{T}W\vec{x}+log\left({\vec{1}}^{T}exp(W\vec{x})\right).\end{array}$$
第一部分$\frac{\partial }{\partial W}\left(-{\vec{y}}^{T}W\vec{x}\right)=\frac{\partial }{\partial W}tr\left(-\vec{x}{\vec{y}}^{T}W\right)=-\vec{y}{\vec{x}}^T.$
第二部分
$$\begin{array}{rl}\mathrm{d}\left(log\left({\vec{1}}^{T}exp(W\vec{x})\right)\right)=& \mathrm{d}tr\left(log\left({\vec{1}}^{T}exp(W\vec{x})\right)\right)\\ =& \mathrm{d}tr\left(\frac{{\vec{1}}^T\left(exp(W\vec{x})\odot \left(\mathrm{d}W\vec{x}\right)\right)}{{\vec{1}}^{T}exp(W\vec{x})}\right)\\ =& \mathrm{d}tr\left(\frac{\left(\vec{1}\odot exp(W\vec{x}{)}^T\right)\left(\mathrm{d}W\vec{x}\right)}{{\vec{1}}^{T}exp(W\vec{x})}\right)\\ =& \mathrm{d}tr\left(\frac{\vec{x}exp(W\vec{x}{)}^T\left(\mathrm{d}W\right)}{{\vec{1}}^{T}exp(W\vec{x})}\right)\end{array}$$
故得
$$\frac{\partial l}{\partial W}=-\vec{y}{\vec{x}}^T+softmax(W\vec{x}){\vec{x}}^T=\left(softmax(W\vec{x})-\vec{y}\right){\vec{x}}^T.$$

【例7】 有样本$({\vec{x}}_1,{\vec{y}}_1),({\vec{x}}_2,{\vec{y}}_2),\cdots ,({\vec{x}}_N,{\vec{y}}_N)$。${\vec{y}}_i\in {\mathbb{F}}^{m\times 1}$，${\vec{y}}_i$只有一个元素为$1$，其他都是$0$，${\vec{x}}_i\in {\mathbb{F}}^{n\times 1}$，
$W_1\in {\mathbb{F}}^{p\times n}$，$W_2\in {\mathbb{F}}^{m\times p}$，${\vec{b}}_1\in {\mathbb{F}}^{p\times 1}$，
${\vec{b}}_2\in {\mathbb{F}}^{m\times 1}$，${\vec{a}}_{1,i}=W_1{\vec{x}}_i+{\vec{b}}_1$, $\overrightarrow{h_{1,i}}=\sigma ({\vec{a}}_{1,i})$，
${\vec{a}}_{2,i}=W_1{\vec{h}}_{1,i}+{\vec{b}}_2$, 定义损失函数为$l=-{\sum }_{i=1}^N{\vec{y}}_i^T\log softmax({\vec{a}}_{2,i})$.

【解】 先求损失对第2层输出的微分$\frac{\partial l}{\partial {\vec{a}}_{2,i}}=softmax({\vec{a}}_{2,i})-{\vec{y}}_i$。
再求损失对第1层输出、连接第1-2层间的权重的微分。这里由于没有定义对矩阵求导的一些链式法则，因此使用导数与微分的关系。
$$\begin{array}{rl}\mathrm{d}l=& tr\left(\sum _{i=1}^N{\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T\mathrm{d}{\vec{a}}_{2,i}\right)\\ =& \sum _{i=1}^{N}tr\left({\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T\mathrm{d}\left(W_2{\vec{h}}_{1,i}+{\vec{b}}_2\right)\right)\\ =& \sum _{i=1}^{N}tr\left({\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T\mathrm{d}\left(W_2\right){\vec{h}}_{1,i}\right)+\sum _{i=1}^{N}tr\left({\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T{W}_2\mathrm{d}\left({\vec{h}}_{1,i}\right)\right)+\sum _{i=1}^{N}tr\left({\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T\mathrm{d}\left({\vec{b}}_2\right)\right)\\ =& \sum _{i=1}^{N}tr\left({\vec{h}}_{1,i}{\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T\mathrm{d}\left(W_2\right)\right)+\sum _{i=1}^{N}tr\left({\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T{W}_2\mathrm{d}\left({\vec{h}}_{1,i}\right)\right)+\sum _{i=1}^{N}tr\left({\left(\frac{\partial l}{\partial {\vec{a}}_{2,i}}\right)}^T\mathrm{d}\left({\vec{b}}_2\right)\right).\end{array}$$
得到$$\frac{\partial l}{\partial W_2}=\sum _{i=1}^N\frac{\partial l}{\partial {\vec{a}}_{2,i}}{\vec{h}}_{1,i}^T$$.
$$\frac{\partial l}{\partial b_2}=\sum _{i=1}^N\frac{\partial l}{\partial {\vec{a}}_{2,i}}.$$
$$\frac{\partial l}{\partial h_{1,i}}=W_2^T\frac{\partial l}{\partial {\vec{a}}_{2,i}}.$$
再求损失对第1层输入的微分。
$$\frac{\partial l}{\partial {\vec{a}}_{1,i}}=\frac{\partial l}{\partial h_{1,i}}\odot {\sigma }^{{}^{\prime }}({\vec{a}}_{1,i}).$$
最后再求损失对连接输入层到第1层的权重的微分。
$$\begin{array}{rl}\mathrm{d}l=& tf\left(\sum _{i=1}^N{\left(\frac{\partial l}{\partial {\vec{a}}_{1,i}}\right)}^T\mathrm{d}{\vec{a}}_{1,i}\right)\\ =& tf\left(\sum _{i=1}^N{\left(\frac{\partial l}{\partial {\vec{a}}_{1,i}}\right)}^T\mathrm{d}\left(W_1{\vec{x}}_i+{\vec{b}}_i\right)\right)\\ =& tf\left(\sum _{i=1}^N{\left(\frac{\partial l}{\partial {\vec{a}}_{1,i}}\right)}^T\mathrm{d}{W}_1{\vec{x}}_i\right)+tf\left(\sum _{i=1}^N{\left(\frac{\partial l}{\partial {\vec{a}}_{1,i}}\right)}^T\mathrm{d}{\vec{b}}_i\right)\\ =& tf\left(\sum _{i=1}^N{\vec{x}}_i{\left(\frac{\partial l}{\partial {\vec{a}}_{1,i}}\right)}^T\mathrm{d}{W}_1\right)+tf\left(\sum _{i=1}^N{\left(\frac{\partial l}{\partial {\vec{a}}_{1,i}}\right)}^T\mathrm{d}{\vec{b}}_i\right)\end{array}$$
得$$\frac{\partial l}{\partial W_1}=\sum _{i=1}^N\frac{\partial l}{\partial {\vec{a}}_{1,i}}{\vec{x}}_i^T$$.
$$\frac{\partial l}{\partial {\vec{b}}_1}=\sum _{i=1}^N\frac{\partial l}{\partial {\vec{a}}_{1,i}}.$$
【例8】 将上题给成矩阵形式，$X=[{\vec{x}}_1,\cdots ,{\vec{x}}_N]$，
$A_1=[{\vec{a}}_{1,1},\cdots ,{\vec{a}}_{1,N}]=W_{1}X+{\vec{b}}_1{\vec{1}}^T$，
$H_1=[{\vec{h}}_{1,1},\cdots ,{\vec{h}}_{1,N}]=\sigma (A_1)$，
$A_2=[{\vec{a}}_{2,1},\cdots ,{\vec{a}}_{2,N}]=W_2{H}_1+{\vec{b}}_2{\vec{1}}^T$.

【解】 先求损失对第2层输出的微分$\frac{\partial l}{\partial A_2}=[softmax(\overrightarrow{a_{2,1}})-{\vec{y}}_1,\cdots ,softmax(\overrightarrow{a_{2,N}})-{\vec{y}}_N]$。
再求损失对第1层输出、连接第1-2层间的权重的微分。这里由于没有定义对矩阵求导的一些链式法则，因此使用导数与微分的关系。
$$\begin{array}{rl}\mathrm{d}l=& tf\left({\left(\frac{\partial l}{\partial A_2}\right)}^T\mathrm{d}{A}_2\right)\\ =& tf\left({\left(\frac{\partial l}{\partial A_2}\right)}^T\mathrm{d}\left(W_2{H}_1+{\vec{b}}_2{\vec{1}}^T\right)\right)\\ =& tf\left(H_1{\left(\frac{\partial l}{\partial A_2}\right)}^T\mathrm{d}(W_2)\right)+tf\left({\left(\frac{\partial l}{\partial A_2}\right)}^T{W}_2\mathrm{d}(H_1)\right)+tf\left({\left(\frac{\partial l}{\partial A_2}\vec{1}\right)}^T\mathrm{d}{\vec{b}}_2\right).\end{array}$$
得
$$\frac{\partial l}{\partial W_2}=\frac{\partial l}{\partial A_2}{H}_1^T$$.
$$\frac{\partial l}{\partial H_1}=W_2^T\frac{\partial l}{\partial A_2}$$.
$$\frac{\partial l}{\partial {\vec{b}}_2}=\frac{\partial l}{\partial A_2}\vec{1}$$.
再求损失对第1层输入的微分。
$$\frac{\partial l}{\partial A_1}=\frac{\partial l}{\partial H_1}\odot {\sigma }^{{}^{\prime }}(A_1)$$.
再求损失对第1层输出、连接第1-2层间的权重的微分。
$$\begin{array}{rl}\mathrm{d}l=& tf\left({\left(\frac{\partial l}{\partial A_1}\right)}^T\mathrm{d}{A}_1\right)\\ =& tf\left({\left(\frac{\partial l}{\partial A_1}\right)}^T\mathrm{d}\left(W_{1}X+{\vec{b}}_1{\vec{1}}^T\right)\right)\\ =& tf\left({\left(\frac{\partial l}{\partial A_1}\right)}^T(\mathrm{d}{W}_1)X\right)+tf\left({\left(\frac{\partial l}{\partial A_1}\right)}^T{W}_1(\mathrm{d}X)\right)+tf\left({\left(\frac{\partial l}{\partial A_1}\right)}^T(\mathrm{d}{\vec{b}}_1){\vec{1}}^T\right)\\ =& tf\left(X{\left(\frac{\partial l}{\partial A_1}\right)}^T(\mathrm{d}{W}_1)\right)+tf\left({\left(\frac{\partial l}{\partial A_1}\right)}^T{W}_1(\mathrm{d}X)\right)+tf\left({\vec{1}}^T{\left(\frac{\partial l}{\partial A_1}\right)}^T(\mathrm{d}{\vec{b}}_1)\right).\end{array}$$
得$$\frac{\partial l}{\partial W_1}=\frac{\partial l}{\partial A_1}{X}^T.$$
$$\frac{\partial l}{\partial {\vec{b}}_1}=\frac{\partial l}{\partial A_1}\vec{1}.$$

参考文献

[1] KHENG L W. Matrix differentiation,cs5240 theoretical foundations in multimedia.
[2] 张贤达. 矩阵分析与应用[M]. 北京: 清华大学出版社, 2004: 255-285.
[3] 长躯鬼侠. 矩阵求导术（上).