矩阵求导之五：机器学习中的常用公式（上）

在了解了矩阵求导的定义之后，可以借助定义进行一些机器学习中常用公式的推导。约定$\mathbf{x}$为$n$维列向量：
$$\mathbf{x}={\left[{\begin{array}{cccc}{x}_1& x_2& \cdots & x\end{array}}_n\right]}^T$$

结论一

$$\frac{\partial a}{\partial \mathbf{x}}=0$$
【证明】
$$\frac{\partial a}{\partial \mathbf{x}}={\left[\begin{array}{cccc}\frac{\partial a}{\partial x_1}& \frac{\partial a}{\partial x_2}& \cdots & \frac{\partial a}{\partial x_n}\end{array}\right]}^T={\left[\begin{array}{cccc}0& 0& \cdots & 0\end{array}\right]}^T$$

结论二

$$\frac{\partial \left({\mathbf{x}}^T\cdot \mathbf{A}\right)}{\partial \mathbf{x}}=\frac{\partial \left({\mathbf{A}}^T\cdot \mathbf{x}\right)}{\partial \mathbf{x}}=\mathbf{A}$$
【证明】
记
$$\mathbf{A}={\left[\begin{array}{cccc}{{\mathbf{\alpha }}_1}^T& {{\mathbf{\alpha }}_2}^T& \cdots & {{\mathbf{\alpha }}_n}^T\end{array}\right]}^T$$
其中：
$${\mathbf{\alpha }}_i={\left[\begin{array}{cccc}{a}_{i1}& a_{i2}& \cdots & a_{in}\end{array}\right]}^T$$
则有：
$$\begin{array}{rl}\frac{\partial \left({\mathbf{x}}^T\cdot \mathbf{A}\right)}{\partial \mathbf{x}}& =\frac{\partial \left({\mathbf{A}}^T\cdot \mathbf{x}\right)}{\partial \mathbf{x}}\\ & =\frac{\partial \left(x_1\cdot {{\mathbf{\alpha }}_1}^T+x_2\cdot {{\mathbf{\alpha }}_2}^T+\cdots +x_n\cdot {{\mathbf{\alpha }}_n}^T\right)}{\partial \mathbf{x}}\\ & =\left[\begin{array}{c}\frac{\partial \left(x_1\cdot {{\mathbf{\alpha }}_1}^T+x_2\cdot {{\mathbf{\alpha }}_2}^T+\cdots +x_n\cdot {{\mathbf{\alpha }}_n}^T\right)}{\partial x_1}\\ \\ \frac{\partial \left(x_1\cdot {{\mathbf{\alpha }}_1}^T+x_2\cdot {{\mathbf{\alpha }}_2}^T+\cdots +x_n\cdot {{\mathbf{\alpha }}_n}^T\right)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial \left(x_1\cdot {{\mathbf{\alpha }}_1}^T+x_2\cdot {{\mathbf{\alpha }}_2}^T+\cdots +x_n\cdot {{\mathbf{\alpha }}_n}^T\right)}{\partial x_n}\end{array}\right]\\ & =\left[\begin{array}{c}{{\mathbf{\alpha }}_1}^T\\ \\ {{\mathbf{\alpha }}_2}^T\\ \\ ⋮\\ \\ {{\mathbf{\alpha }}_n}^T\end{array}\right]=A\end{array}$$

结论三

$$\frac{\partial {\mathbf{x}}^T\mathbf{x}}{\partial \mathbf{x}}=2\mathbf{x}$$
【证明】设
$$\mathbf{x}={\left[x_1,x_2,\cdots ,x_m\right]}^T$$

$$f\left(\mathbf{x}\right)={\mathbf{x}}^T\mathbf{x}={x_1}^2+{x_2}^2+\cdots +{x_n}^2$$

$$\frac{\partial f}{\partial \mathbf{x}}=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \\ \frac{\partial f}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f}{\partial x_n}\end{array}\right]=\left[\begin{array}{c}2x_1\\ \\ 2x_2\\ \\ ⋮\\ \\ 2x_n\end{array}\right]=2\mathbf{x}$$
即：
$$\frac{\partial {\mathbf{x}}^T\mathbf{x}}{\partial \mathbf{x}}=2\mathbf{x}$$

结论四

$$\frac{\partial \left({\mathbf{x}}^T\mathbf{Ax}\right)}{\partial \mathbf{x}}=\mathbf{Ax}+{\mathbf{A}}^T\mathbf{x}$$
【证明】
$$\begin{array}{rl}{\mathbf{x}}^T\mathbf{Ax}& =\left[\begin{array}{cccc}{x}_1& x_2& \cdots & x_n\end{array}\right]\cdot \left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]\\ & =\left[\begin{array}{cccc}{x}_1{a}_{11}+x_2{a}_{21}+\cdots +x_n{a}_{n1}& x_1{a}_{12}+x_2{a}_{22}+\cdots +x_n{a}_{n2}& \cdots & x_1{a}_{1n}+x_2{a}_{2n}+\cdots +x_n{a}_{nn}\end{array}\right]\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]\\ & =x_1\left(x_1{a}_{11}+x_2{a}_{21}+\cdots +x_n{a}_{n1}\right)+x_2\left(x_1{a}_{12}+x_2{a}_{22}+\cdots +x_n{a}_{n2}\right)+\cdots +x_n\left(x_1{a}_{1n}+x_2{a}_{2n}+\cdots +x_n{a}_{nn}\right)\end{array}$$
记
$$f\left(\mathbf{x}\right)={\mathbf{x}}^T\mathbf{Ax}$$
则：
$$\frac{\partial f\left(\mathbf{x}\right)}{\partial x_1}=\left(x_1{a}_{11}+x_2{a}_{21}+\cdots +x_n{a}_{n1}\right)+\left(x_1{a}_{11}+x_2{a}_{12}+\cdots +x_n{a}_{1\mathbf{n}}\right)$$
$$\begin{array}{rl}\frac{\partial f\left(\mathbf{x}\right)}{\partial \mathbf{x}}& =\left[\begin{array}{c}\frac{\partial f\left(\mathbf{x}\right)}{\partial x_1}\\ \\ \frac{\partial f\left(\mathbf{x}\right)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f\left(\mathbf{x}\right)}{\partial x_n}\end{array}\right]=\left[\begin{array}{c}\left(x_1{a}_{11}+x_2{a}_{21}+\cdots +x_n{a}_{n1}\right)+\left(x_1{a}_{11}+x_2{a}_{12}+\cdots +x_n{a}_{1n}\right)\\ \\ \left(x_1{a}_{12}+x_2{a}_{22}+\cdots +x_n{a}_{n2}\right)+\left(x_1{a}_{21}+x_2{a}_{22}+\cdots +x_n{a}_{2n}\right)\\ \\ ⋮\\ \\ \left(x_1{a}_{1n}+x_2{a}_{2n}+\cdots +x_n{a}_{nn}\right)+\left(x_1{a}_{n1}+x_2{a}_{n2}+\cdots +x_n{a}_{nn}\right)\end{array}\right]\\ & \\ & =\left[\begin{array}{c}{x}_1{a}_{11}+x_2{a}_{21}+\cdots +x_n{a}_{n1}\\ \\ x_1{a}_{12}+x_2{a}_{22}+\cdots +x_n{a}_{n2}\\ \\ ⋮\\ \\ x_1{a}_{1n}+x_2{a}_{2n}+\cdots +x_n{a}_{nn}\end{array}\right]+\left[\begin{array}{c}{x}_1{a}_{11}+x_2{a}_{12}+\cdots +x_n{a}_{1n}\\ \\ x_1{a}_{21}+x_2{a}_{22}+\cdots +x_n{a}_{2n}\\ \\ ⋮\\ \\ x_1{a}_{n1}+x_2{a}_{n2}+\cdots +x_n{a}_{nn}\end{array}\right]\\ & \\ & =\left[\begin{array}{cccc}{a}_{11}& a_{21}& \cdots & a_{n1}\\ a_{12}& a_{22}& \cdots & a_{n2}\\ ⋮& ⋮& \ddots & ⋮\\ a_{1n}& a_{2n}& \cdots & a_{nn}\end{array}\right]\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]+\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]\\ & \\ & ={\mathbf{A}}^T\mathbf{x}+\mathbf{Ax}\end{array}$$

结论五

$$\frac{\partial \left({\mathbf{x}}^T\mathbf{a}\right)}{\partial \mathbf{x}}=\frac{\partial \left({\mathbf{a}}^T\mathbf{x}\right)}{\partial \mathbf{x}}=\mathbf{a}$$
其中$\mathbf{a}$为常数向量：
$$\mathbf{a}={\left[\begin{array}{cccc}{a}_1& a_2& \cdots & a_n\end{array}\right]}^T$$
【证明】
$$\begin{array}{rl}\frac{\partial \left({\mathbf{x}}^T\mathbf{a}\right)}{\partial \mathbf{x}}& =\frac{\partial \left({\mathbf{a}}^T\mathbf{x}\right)}{\partial \mathbf{x}}\\ & \\ & =\frac{\partial \left(x_1{a}_1+x_2{a}_2+\cdots +x_n{a}_n\right)}{\partial \mathbf{x}}\\ & \\ & =\left[\begin{array}{c}\frac{\partial \left(x_1{a}_1+x_2{a}_2+\cdots +x_n{a}_n\right)}{\partial x_1}\\ \\ \frac{\partial \left(x_1{a}_1+x_2{a}_2+\cdots +x_n{a}_n\right)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial \left(x_1{a}_1+x_2{a}_2+\cdots +x_n{a}_n\right)}{\partial x_n}\\ \end{array}\right]\\ & =\left[\begin{array}{c}{a}_1\\ \\ a_2\\ \\ ⋮\\ \\ a_n\end{array}\right]\\ & =\mathbf{a}\end{array}$$

结论六

$$\frac{\partial \left({\mathbf{a}}^T{\mathbf{xx}}^T\mathbf{b}\right)}{\partial \mathbf{x}}={\mathbf{ab}}^T\mathbf{x}+{\mathbf{ba}}^T\mathbf{x}$$
其中$\mathbf{a}$，$\mathbf{b}$为常数向量：
$$\begin{gathered} \mathbf{a}={\left[\begin{array}{cccc}{a}_1& a_2& \cdots & a_n\end{array}\right]}^T \\ \mathbf{b}={\left[\begin{array}{cccc}{b}_1& b_2& \cdots & b_n\end{array}\right]}^T \end{gathered}$$
【证明】
因为${\mathbf{a}}^T\mathbf{x}={\mathbf{x}}^T\mathbf{a},{\mathbf{x}}^T\mathbf{b}={\mathbf{b}}^T\mathbf{x}$，所以有：
$$\frac{\partial \left({\mathbf{a}}^T{\mathbf{xx}}^T\mathbf{b}\right)}{\partial \mathbf{x}}=\frac{\partial \left({\mathbf{x}}^T{\mathbf{ab}}^T\mathbf{x}\right)}{\partial \mathbf{x}}$$
又因为${\mathbf{ab}}^T$是$n\times n$的常数矩阵，由结论四可知：
$$\frac{\partial \left({\mathbf{a}}^T{\mathbf{xx}}^T\mathbf{b}\right)}{\partial \mathbf{x}}=\frac{\partial \left({\mathbf{x}}^T{\mathbf{ab}}^T\mathbf{x}\right)}{\partial \mathbf{x}}={\mathbf{ab}}^T\mathbf{x}+{\mathbf{ba}}^T\mathbf{x}$$

结论七

$$\frac{\partial \left({\mathbf{a}}^T\mathbf{Xb}\right)}{\partial \mathbf{X}}={\mathbf{ab}}^T$$
其中$\mathbf{a}$，$\mathbf{b}$为常数向量：
$$\begin{gathered} \mathbf{a}={\left[\begin{array}{cccc}{a}_1& a_2& \cdots & a_m\end{array}\right]}^T \\ \mathbf{b}={\left[\begin{array}{cccc}{b}_1& b_2& \cdots & b_n\end{array}\right]}^T \end{gathered}$$
【证明】
$$\begin{array}{rl}{\mathbf{a}}^T\mathbf{Xb}& =\left[\begin{array}{cccc}{a}_1& a_2& \cdots & a_m\end{array}\right]\cdot \left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1n}\\ x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ x_{m1}& x_{m2}& \cdots & x_{mn}\end{array}\right]\cdot \left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_n\end{array}\right]\\ & =\left[\begin{array}{cccc}{a}_1{x}_{11}+a_2{x}_{21}+\cdots +a_m{a}_{m1}& a_1{x}_{12}+a_2{x}_{22}+\cdots +a_m{a}_{m2}& \cdots & a_1{x}_{1n}+a_2{x}_{2n}+\cdots +a_m{a}_{mn}\end{array}\right]\cdot \left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_n\end{array}\right]\\ & =b_1\left(a_1{x}_{11}+a_2{x}_{21}+\cdots +a_m{a}_{m1}\right)+b_2\left(a_1{x}_{12}+a_2{x}_{22}+\cdots +a_m{a}_{m2}\right)+\cdots +b_n\left(a_1{x}_{1n}+a_2{x}_{2n}+\cdots +a_m{a}_{mn}\right)\end{array}$$
记
$$f\left(\mathbf{X}\right)={\mathbf{a}}^T\mathbf{Xb}$$
则：
$$\begin{array}{rl}\frac{\partial \left({\mathbf{a}}^T\mathbf{Xb}\right)}{\partial \mathbf{X}}& =\frac{\partial f\left(\mathbf{X}\right)}{\partial \mathbf{X}}\\ & \\ & ={\left[\begin{array}{cccc}\frac{\partial f}{\partial x_{11}}& \frac{\partial f}{\partial x_{12}}& \cdots & \frac{\partial f}{\partial x_{1n}}\\ & & & \\ \frac{\partial f}{\partial x_{21}}& \frac{\partial f}{\partial x_{22}}& \cdots & \frac{\partial f}{\partial x_{2n}}\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ \frac{\partial f}{\partial x_{m1}}& \frac{\partial f}{\partial x_{m2}}& \cdots & \frac{\partial f}{\partial x_{mn}}\end{array}\right]}_{m\times n}\\ & \\ & ={\left[\begin{array}{cccc}{a}_1{b}_1& a_1{b}_2& \cdots & a_1{b}_n\\ & & & \\ a_2{b}_1& a_2{b}_2& \cdots & a_2{b}_n\\ & & & \\ ⋮& ⋮& \ddots & ⋮\\ & & & \\ a_m{b}_1& a_m{b}_2& \cdots & a_m{b}_n\end{array}\right]}_{m\times n}\\ & \\ & =\left[\begin{array}{c}{a}_1\\ \\ a_2\\ \\ ⋮\\ \\ a_m\end{array}\right]\cdot \left[\begin{array}{cccc}{b}_1& b_2& \cdots & b_n\end{array}\right]\\ & \\ & ={\mathbf{ab}}^T\end{array}$$

参考文献

[1] 机器学习中的矩阵求导方法
[2] 矩阵求导公式的数学推导
[3] 矩阵的求导