人工智能基础数学之矩阵求导快速入门

0.参考链接

矩阵求导

1.标量函数和向量函数

$$\{\begin{array}{l}{y}_1=w_1{x}_{11}+w_2{x}_{12}+\cdots +w_n{x}_{1n}\\ ⋮\\ y_m=w_1{x}_{m1}+w_2{x}_{m2}+\cdots +w_n{x}_{mn}\end{array}$$

1.1标量函数：

1.输出为标量的函数为标量函数：
输入，输出都是标量：
$$f(x)=x^2,{\mathbb{R}}_x\to {\mathbb{R}}_{x^2}$$
输入不是标量，输出是标量：
$$f(x)=x_1^2+x_2^2,{\mathbb{R}}_{(x_1,x_2{)}^T}^2\to {\mathbb{R}}_{x_1^2+x_2^2}$$

2.输出为向量的函数为向量函数：
输入是标量，输出是向量：
$$f(x)=\left[\begin{array}{c}{f}_1(x)=x\\ f_2(x)=x^2\end{array}\right],{\mathbb{R}}_x\to {\mathbb{R}}_{(x,x^2{)}^T}^2$$
输入，输出都是向量：
$$f(x)=\left[\begin{array}{cc}{f}_{11}(x)=x& f_{12}(x)=x^2\\ f_{21}(x)=x^3& f_{22}(x)=x^4\end{array}\right],{\mathbb{R}}_x\to {\mathbb{R}}^{2\ast 2}$$

$$f(x)=\left[\begin{array}{cc}{f}_{11}(x)=x_1+x_2& f_{12}(x)=x_1^2+x_2^2\\ f_{21}(x)=x_1^3+x_2^3& f_{22}(x)=x_1^4+x_2^4\end{array}\right],{\mathbb{R}}_{(x_1,x_2{)}^T}^2\to {\mathbb{R}}^{2\ast 2}$$

1.2总结

$$\begin{gathered} x\to 标量，向量，矩阵 \\ f(x)\to 标量，向量，矩阵 \end{gathered}$$
只看标量和向量，那$\frac{df(x)}{dx}$就有四种可能。
矩阵求导的本质：$\frac{dA}{dB}$就是矩阵$A$中的每一个元素对矩阵$B$中的每一个元素求导。

1.3从求导后元素个数的角度

$A_{1\ast 1},B_{1\ast 1}\to {\frac{dA}{dB}}_{1\ast 1}$，
$A_{1\ast p},B_{1\ast n}\to {\frac{dA}{dB}}_{p\ast n}$,
$A_{q\ast p},B_{m\ast n}\to {\frac{dA}{dB}}_{p\ast q\ast m\ast n}$

2.求导法(YX拉伸法)(重点)

$$\{\begin{array}{l}1.标量不变，向量拉伸\\ 2.前面横向拉，后面纵向拉\end{array}$$
以下是三个例子：
例1：$\frac{df(x)}{dx},f(x)=f(x_1,\cdots ,x_n)是标量函数,x=[x_1,x_2,\cdots ,x_n{]}^T是向量$，
$$\frac{df(x)}{dx}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \frac{\partial f(x)}{\partial x_2}\\ ⋮\\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$，可以看出结果是$f(x)$是标量不变，$x$是向量纵向拉伸，实际就是将多元函数的偏导数写在一个列向量中。

例2：$\frac{df(x)}{dx},f(x)=\left[\begin{array}{c}{f}_1(x)\\ f_2(x)\\ ⋮\\ f_n(x)\end{array}\right]是向量函数,x是标量$，$$\frac{df(x)}{dx}=\left[\begin{array}{cccc}\frac{\partial f_1(x)}{\partial x}& \frac{\partial f_2(x)}{\partial x}& \cdots & \frac{\partial f_n(x)}{\partial x}\end{array}\right]$$，符合前面横向拉伸。

例3：$\frac{df(x)}{dx},f(x)=\left[\begin{array}{c}{f}_1(x)\\ f_2(x)\\ ⋮\\ f_n(x)\end{array}\right]是向量函数,[x_1,x_2,\cdots ,x_n{]}^T是向量$，
$$\frac{df(x)}{dx}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \frac{\partial f(x)}{\partial x_2}\\ ⋮\\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$，此时$\partial f(x)$是向量，而$x_1$等是标量，所以$$\frac{df(x)}{dx}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \frac{\partial f(x)}{\partial x_2}\\ ⋮\\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]=\left[\begin{array}{ccccc}\frac{\partial f_1(x)}{\partial x_1}& \frac{\partial f_2(x)}{\partial x_1}& \cdots & \frac{\partial f_n(x)}{\partial x_1}& \\ \frac{\partial f_1(x)}{\partial x_2}& \frac{\partial f_2(x)}{\partial x_2}& \cdots & \frac{\partial f_n(x)}{\partial x_2}& \\ ⋮& ⋮& \cdots & ⋮& \\ \frac{\partial f_1(x)}{\partial x_n}& \frac{\partial f_2(x)}{\partial x_n}& \cdots & \frac{\partial f_n(x)}{\partial x_n}& \end{array}\right]$$，符合先纵向拉伸，再横向拉伸。

3.常见的矩阵求导公式推导

例1：$f(x)=A^{T}X,A={\left[\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right]}_{n\ast 1},X={\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]}_{n\ast 1},求\frac{df(x)}{dx}$
解：$f(x)是标量函数，X是向量，f(x)=A^{T}X={\sum }_{i=1}^n{a}_i{x}_i$
$$\frac{df(x)}{dX}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \frac{\partial f(x)}{\partial x_2}\\ ⋮\\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]=\left[\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right]=A$$，注意到，$f(x)=A^{T}X=X^{T}A（标{量}^T=标量）$，所以$\frac{d{A}^{T}X}{dX}=\frac{d{X}^{T}A}{dX}=A$。

例2，$f(x)=X^{T}AX,X={\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]}_{n\ast 1},A={\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]}_{n\ast n},求\frac{df(x)}{dX}$
解：$f(x)=X^{T}AX是标量函数$，
$$f(x)=\left[\begin{array}{cccc}{x}_1& x_2& \cdots & x_n\end{array}\right]\cdot \left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]$$

$f(x)={\sum }_{i=1}^n{\sum }_{j=1}^n{a}_{ij}{x}_i{x}_j$,

$$\begin{array}{rl}\frac{df(x)}{dX}& =\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \frac{\partial f(x)}{\partial x_2}\\ ⋮\\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]\\ & =\left[\begin{array}{c}{\sum }_{j=1}^n{a}_{1j}{x}_j+{\sum }_{j=1}^n{a}_{i1}{x}_i\\ {\sum }_{j=1}^n{a}_{2j}{x}_j+{\sum }_{i=1}^n{a}_{i2}{x}_i\\ ⋮\\ {\sum }_{j=1}^n{a}_{nj}{x}_j+{\sum }_{i=1}^n{a}_{in}{x}_i\end{array}\right]\\ & =\left[\begin{array}{c}{\sum }_{j=1}^n{a}_{1j}{x}_j\\ {\sum }_{j=1}^n{a}_{2j}{x}_j\\ ⋮\\ {\sum }_{j=1}^n{a}_{nj}{x}_j\end{array}\right]+\left[\begin{array}{c}{\sum }_{j=1}^n{a}_{i1}{x}_i\\ {\sum }_{i=1}^n{a}_{i2}{x}_i\\ ⋮\\ {\sum }_{i=1}^n{a}_{in}{x}_i\end{array}\right]\\ & =\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]\\ & +{\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]}^T\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]\\ & =AX+A^{T}X=(A+A^T)X\end{array}$$，所以$\frac{d{X}^{T}AX}{dX}=(A+A^T)X$，特别地，$(X^{T}AX{)}^T=X^T{A}^{T}X，\frac{d{X}^T{A}^{T}X}{dX}=(A+A^T)X$

4.矩阵求导布局

$$\{\begin{array}{l}分母布局\to YX拉伸\\ 分子布局\to XY拉伸\end{array}$$
区别：总的不变：前面横向拉，后面纵向拉。
YX拉伸（分母布局），Y横向（f(x)），X纵向拉。
XY拉伸（分子布局），相反
通常$(分母布局{)}^T=(分子布局)$
例：$f(x)=X^{T}X,X=[x_1,x_2,\cdots ,x_n{]}^T$
分母布局：
$$\frac{df(x)}{dx}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \frac{\partial f(x)}{\partial x_2}\\ ⋮\\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]=\left[\begin{array}{c}2x_1\\ 2x_2\\ ⋮\\ 2x_n\end{array}\right]=2X$$
分子布局：
$$\frac{df(x)}{dx}=\left[\begin{array}{cccc}\frac{\partial f(x)}{\partial x_1}& \frac{\partial f(x)}{\partial x_2}& \cdots & \frac{\partial f(x)}{\partial x_n}\end{array}\right]=\left[\begin{array}{cccc}2x_1& 2x_2& \cdots & 2x_n\end{array}\right]=2X^T$$

5.矩阵求导的乘法和加法公式

$$U={\left[\begin{array}{c}{u}_1(x)\\ u_2(x)\\ ⋮\\ u_n(x)\end{array}\right]}_{n\ast 1},V={\left[\begin{array}{c}{v}_1(x)\\ v_2(x)\\ ⋮\\ v_n(x)\end{array}\right]}_{n\ast 1},X={\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]}_{n\ast 1}$$
注意到$U^{T}V$是标量
$$\frac{d{U}^{T}V}{dX}=\frac{\partial U}{\partial X}V+\frac{\partial V}{\partial X}U$$

$$\frac{d(U+V)}{dX}=\frac{dU}{dX}+\frac{dV}{dX}$$