复合函数的前向微分与反向自动微分计算

复合函数的前向微分与反向自动微分计算

关于

• 首次发表日期：2024-09-13
• 参考：
  • https://rufflewind.com/2016-12-30/reverse-mode-automatic-differentiation
  • Calculus Early Transcendentals 9e - James Stewart (2020)
  • https://en.wikipedia.org/wiki/Automatic_differentiation
• 水平有限，如有错误，请不吝指出

前向与反向自动微分：数学

先复习一下微积分求导法则

微积分求导法则复习

乘法法则

$$f(x)=u(x)\times v(x)$$

$$\begin{array}{rl}\frac{dy}{dx}& =\frac{du}{dx}\times v+\frac{dv}{dx}\times u\\ f^{\prime }(x)& =u^{\prime }v+v^{\prime }u\end{array}$$

$$\begin{array}{rl}f(x)& =(3x-5)\times (4x+7)\\ u& =3x-5v=4x+7\\ u^{\prime }& =3v^{\prime }=4\\ f^{\prime }(x)& =3(4x+7)+4(3x-5)\\ & =12x+21+12x-20=24x+1\\ & =24x+1\end{array}$$

除法法则

$$f(x)=\frac{u(x)}{v(x)}$$

$$\begin{array}{rl}{f}^{\prime }(x)& =\frac{u^{\prime }v-v^{\prime }u}{v^2}\\ \frac{dy}{dx}& =\frac{\frac{du}{dx}v-\frac{dv}{dx}u}{v^2}\end{array}$$

$$\begin{array}{rl}f(x)& =\frac{3x-5}{4x+7}\\ u& =3x-5v=4x+7\\ u^{\prime }& =3v^{\prime }=4\\ f^{\prime }(x)& =\frac{3(4x+7)-4(3x-5)}{(4x+7{)}^2}\\ & =\frac{12x+21-12x+20}{(4x+7{)}^2}\\ & =\frac{41}{(4x+7{)}^2}\end{array}$$

cos和sin求导

$$\begin{array}{rl}y& =\sin (x)\\ \frac{dy}{dx}& =\cos (x)\end{array}$$

$$\begin{array}{r}y=\cos (x)\\ \frac{dy}{dx}=-\sin (x)\end{array}$$

链式法则(单变量复合函数)

$$y=f(u)u=f(x)$$

$$\frac{dy}{dx}=\frac{dy}{du}\cdot \frac{du}{dx}$$

$$\begin{array}{rl}y& =(2x+4{)}^3\\ y& =u^3\text{ and }u=2x+4\\ \frac{dy}{du}& =3u^2\frac{du}{dx}=2\\ \frac{dy}{dx}& =3u^2\times 2=2\times 3(2x+4{)}^2\\ & =6(2x+4{)}^2\end{array}$$

多变量链式法则（Case 1）

$$\begin{array}{rl}z& =f(x,y)\\ x& =g(t)\\ y& =h(t)\end{array}$$

$$\frac{dz}{dt}=\frac{\partial f}{\partial x}\frac{dx}{dt}+\frac{\partial f}{\partial y}\frac{dy}{dt}$$

多变量链式法则（Case 2）

$$\begin{array}{rl}z& =f(x,y)\\ x& =g(s,t)\\ y& =h(s,t)\end{array}$$

$$\frac{\partial z}{\partial s}=\frac{\partial z}{\partial x}\frac{\partial x}{\partial s}+\frac{\partial z}{\partial y}\frac{\partial y}{\partial s}\frac{\partial z}{\partial t}=\frac{\partial z}{\partial x}\frac{\partial x}{\partial t}+\frac{\partial z}{\partial y}\frac{\partial y}{\partial t}$$

当计算$\frac{\partial z}{\partial s}$时，我们保持（hold）$t$ 固定并计算 $z$ 对 $s$ 的普通导数，即应用多变量链式法则（Case 1）。计算$\frac{\partial z}{\partial t}$时同理。

多变量链式法则（广义版）

$$\begin{array}{rl}u& =f(x_1,x_2,\dots ,x_n)\\ x_k& =g(t_1,t_2,\dots ,t_m)\text{for }1\le k\le n\end{array}$$

$$\begin{array}{rl}& \frac{\partial u}{\partial t_i}=\frac{\partial u}{\partial x_1}\frac{\partial x_1}{\partial t_i}+\frac{\partial u}{\partial x_2}\frac{\partial x_2}{\partial t_i}+\cdots +\frac{\partial u}{\partial x_n}\frac{\partial x_n}{\partial t_i}\end{array}\text{for }1\le i\le m$$

复合函数，偏微分，链式法则，前向和反向自动微分

前向与反向的计算顺序

对于组合函数：

$$\begin{array}{rl}y& =f(g(h(x)))=f\left(g\left(h\left(w_0\right)\right)\right)=f\left(g\left(w_1\right)\right)=f\left(w_2\right)=w_3\\ w_0& =x\\ w_1& =h\left(w_0\right)\\ w_2& =g\left(w_1\right)\\ w_3& =f\left(w_2\right)=y\end{array}$$

链式法则将给出：

$$\begin{array}{rl}\frac{\partial y}{\partial x}& =\frac{\partial y}{\partial w_2}\frac{\partial w_2}{\partial w_1}\frac{\partial w_1}{\partial x}=\frac{\partial f\left(w_2\right)}{\partial w_2}\frac{\partial g\left(w_1\right)}{\partial w_1}\frac{\partial h\left(w_0\right)}{\partial x}\end{array}$$

计算顺序：

• 前向微分计算时 ，先计算$\partial w_1/\partial x$，然后计算$\partial w_2/\partial w_1$，最后计算$\partial y/\partial w_2$
• 反向微分计算时，先计算$\partial y/\partial w_2$，然后计算$\partial w_2/\partial w_1$，最后计算$\partial w_1/\partial x$

前向微分

对于组合函数：

$$\begin{array}{rl}r& =?\\ s& =?\\ t& =?\\ x& =g(r,s,t)\\ y& =h(r,s,t)\\ z& =i(r,s,t)\\ u& =f(x,y,z)\end{array}$$

前向微分计算：

$$\begin{array}{rl}\frac{\partial r}{\partial v}& =?\\ \frac{\partial s}{\partial v}& =?\\ \frac{\partial t}{\partial v}& =?\\ & \\ \frac{\partial x}{\partial v}& =\frac{\partial x}{\partial r}\frac{\partial r}{\partial v}+\frac{\partial x}{\partial s}\frac{\partial s}{\partial v}+\frac{\partial x}{\partial t}\frac{\partial t}{\partial v}\\ \frac{\partial y}{\partial v}& =\frac{\partial y}{\partial r}\frac{\partial r}{\partial v}+\frac{\partial y}{\partial s}\frac{\partial s}{\partial v}+\frac{\partial y}{\partial t}\frac{\partial t}{\partial v}\\ \frac{\partial z}{\partial v}& =\frac{\partial z}{\partial r}\frac{\partial r}{\partial v}+\frac{\partial z}{\partial s}\frac{\partial s}{\partial v}+\frac{\partial z}{\partial t}\frac{\partial t}{\partial v}\\ & \\ \frac{\partial u}{\partial v}& =\frac{\partial u}{\partial x}\frac{\partial x}{\partial v}+\frac{\partial u}{\partial y}\frac{\partial y}{\partial v}+\frac{\partial u}{\partial z}\frac{\partial z}{\partial v}\end{array}$$

当$v=r$，即将$r$作为独立变量并将$s$和$t$固定时，可得

$$\begin{array}{rl}\frac{\partial r}{\partial v}& =1\\ \frac{\partial s}{\partial v}& =0\\ \frac{\partial t}{\partial v}& =0\\ \frac{\partial u}{\partial r}& =\frac{\partial u}{\partial x}\frac{\partial x}{\partial r}+\frac{\partial u}{\partial y}\frac{\partial y}{\partial r}+\frac{\partial u}{\partial z}\frac{\partial z}{\partial r}\end{array}$$

当$v=s$，即将$s$作为独立变量并将$r$和$t$固定时，可得

$$\begin{array}{rl}\frac{\partial r}{\partial v}& =0\\ \frac{\partial s}{\partial v}& =1\\ \frac{\partial t}{\partial v}& =0\\ \frac{\partial u}{\partial s}& =\frac{\partial u}{\partial x}\frac{\partial x}{\partial s}+\frac{\partial u}{\partial y}\frac{\partial y}{\partial s}+\frac{\partial u}{\partial z}\frac{\partial z}{\partial s}\end{array}$$

当$v=t$，即将$t$作为独立变量并将$s$和$r$固定时，可得

$$\begin{array}{rl}\frac{\partial r}{\partial v}& =0\\ \frac{\partial s}{\partial v}& =0\\ \frac{\partial t}{\partial v}& =1\\ \frac{\partial u}{\partial t}& =\frac{\partial u}{\partial x}\frac{\partial x}{\partial t}+\frac{\partial u}{\partial y}\frac{\partial y}{\partial t}+\frac{\partial u}{\partial z}\frac{\partial z}{\partial t}\end{array}$$

反向微分

对于组合函数：

$$\begin{array}{rl}{u}_1& =r(x_1,x_2)\\ u_2& =s(x_1,x_2)\\ y_1& =f(u_1,u_2)\\ y_2& =g(u_1,u_2)\\ y_3& =h(u_1,u_2)\end{array}$$

反向微分计算：

$$\begin{array}{rl}\frac{\partial s}{\partial y_1}& =?\\ \frac{\partial s}{\partial y_2}& =?\\ \frac{\partial s}{\partial y_3}& =?\\ & \\ \frac{\partial s}{\partial u_1}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial u_1}+\frac{\partial s}{\partial y_2}\frac{\partial y_2}{\partial u_1}+\frac{\partial s}{\partial y_3}\frac{\partial y_3}{\partial u_1}\\ \frac{\partial s}{\partial u_2}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial u_2}+\frac{\partial s}{\partial y_2}\frac{\partial y_2}{\partial u_2}+\frac{\partial s}{\partial y_3}\frac{\partial y_3}{\partial u_2}\\ & \\ \frac{\partial s}{\partial x_1}& =\frac{\partial s}{\partial u_1}\frac{\partial u_1}{\partial x_1}+\frac{\partial s}{\partial u_2}\frac{\partial u_2}{\partial x_1}\\ \frac{\partial s}{\partial x_2}& =\frac{\partial s}{\partial u_1}\frac{\partial u_1}{\partial x_x}+\frac{\partial s}{\partial u_2}\frac{\partial u_2}{\partial x_x}\end{array}$$

可以想象有一个函数$s=function(y_1,y_2,y_3)$

当$s=y_1$，即将$y_1$作为独立变量并将$y_2$和$y_3$固定时，可得

$$\begin{array}{rl}\frac{\partial s}{\partial y_1}& =1\\ \frac{\partial s}{\partial y_2}& =0\\ \frac{\partial s}{\partial y_3}& =0\\ & \\ \frac{\partial s}{\partial u_1}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial u_1}\\ \frac{\partial s}{\partial u_2}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial u_2}\\ & \\ \frac{\partial s}{\partial x_1}& =\frac{\partial s}{\partial u_1}\frac{\partial u_1}{\partial x_1}+\frac{\partial s}{\partial u_2}\frac{\partial u_2}{\partial x_1}\\ \frac{\partial s}{\partial x_2}& =\frac{\partial s}{\partial u_1}\frac{\partial u_1}{\partial x_x}+\frac{\partial s}{\partial u_2}\frac{\partial u_2}{\partial x_x}\end{array}$$

以例子说明自动微分的计算

例子

假设有2个输入变量（$x_1$, $x_2$）和2个输出变量（$y_1$, $y_2$）：

$$\begin{array}{rl}{m}_1& =x_1\cdot x_2+\sin (x_1)\\ m_2& =4x_1+2x_2+\cos (x_2)\\ y_1& =m_1+m_2\\ y_2& =m_1\cdot m_2\end{array}\text{\hspace{1em}(1)}$$
即：

$$\begin{array}{rl}{y}_1& =x_1\cdot x_2+\sin (x_1)+4x_1+2x_2+\cos (x_2)\\ y_2& =(x_1+x_2+\sin (x_1))\cdot (4x_1+2x_2+\cos (x_2))\end{array}$$

其中：

$$\begin{array}{rl}\frac{\partial y_1}{\partial x_1}& =x_2+\cos (x_1)+4\\ \frac{\partial y_1}{\partial x_2}& =x_1+2-\sin (x_2)\\ \frac{\partial y_2}{\partial x_1}& =(x_2+\cos (x_1))\cdot m_2+m_1\cdot 4\end{array}$$

接下来，我们将以这个例子说明如何进行前向自动微分和反向自动微分

前向自动微分

我们将用到如下的链式法则：

$$\begin{array}{rl}\frac{\partial w}{\partial t}& =\sum _i\left(\frac{\partial w}{\partial u_i}\cdot \frac{\partial u_i}{\partial t}\right)\\ & =\frac{\partial w}{\partial u_1}\cdot \frac{\partial u_1}{\partial t}+\frac{\partial w}{\partial u_2}\cdot \frac{\partial u_2}{\partial t}+\cdots \end{array}$$

其中：

• $w$表示输出
  • 在例子中，为$y_1$或者$y_2$
• $u_i$表示直接影响$w$的输入变量
  • 在例子中，为$a$和$b$
• $t$表示有待给出的输入变量
  • 在例子中，为$x_1$或者$x_2$其中之一

在计算之前，我们先将公式（1）分解为简单的算子计算：

$$\begin{array}{rl}{x}_1& =?\\ x_2& =?\\ & \\ a& =x_1\cdot x_2\\ b& =\sin (x_1)\\ & \\ c& =4x_1+2x_2\\ d& =\cos (x_2)\\ & \\ m_1& =a+b\\ m_2& =c+d\\ & \\ y_1& =m_1+m_2\\ y_2& =m_1\cdot m_2\end{array}\text{\hspace{1em}(2)}$$

现在我们对有待给出的变量$t$求导：

$$\begin{array}{rl}\frac{\partial x_1}{\partial t}& =?\\ \frac{\partial x_2}{\partial t}& =?\\ & \\ \frac{\partial a}{\partial t}& =x_2\frac{\partial x_1}{\partial t}+x_1\frac{\partial x_2}{\partial t}\\ \frac{\partial b}{\partial t}& =\cos (x_1)\frac{\partial x_1}{\partial t}\\ & \\ \frac{\partial c}{\partial t}& =4\frac{\partial x_1}{\partial t}+2\frac{\partial x_2}{\partial t}\\ \frac{\partial d}{\partial t}& =-\sin (x_2)\frac{\partial x_2}{\partial t}\\ & \\ \frac{\partial m_1}{\partial t}& =\frac{\partial a}{\partial t}+\frac{\partial b}{\partial t}\\ \frac{\partial m_2}{\partial t}& =\frac{\partial c}{\partial t}+\frac{\partial d}{\partial t}\\ & \\ \frac{\partial y_1}{\partial t}& =\frac{\partial m_1}{\partial t}+\frac{\partial m_2}{\partial t}\\ \frac{\partial y_2}{\partial t}& =\frac{\partial m_1}{\partial t}\cdot m_2+\frac{\partial m_2}{\partial t}\cdot m_1\end{array}$$

前面有提到$t$是有待给出的，现在是时候给出了：

• 将$t=x_1$代入以上公式，则$\frac{\partial x_1}{\partial t}=1$而$\frac{\partial x_2}{\partial t}=0$，然后可以计算$\frac{\partial y_1}{\partial x_1}$和$\frac{\partial y_2}{\partial x_1}$

$$\begin{array}{rl}\frac{\partial x_1}{\partial t}& =1\\ \frac{\partial x_2}{\partial t}& =0\\ & \\ \frac{\partial a}{\partial t}& =x_2\frac{\partial x_1}{\partial t}+x_1\frac{\partial x_2}{\partial t}=x_2\\ \frac{\partial b}{\partial t}& =\cos (x_1)\frac{\partial x_1}{\partial t}=\cos (x_1)\\ & \\ \frac{\partial c}{\partial t}& =4\frac{\partial x_1}{\partial t}+2\frac{\partial x_2}{\partial t}=4\\ \frac{\partial d}{\partial t}& =-\sin (x_2)\frac{\partial x_2}{\partial t}=0\\ & \\ \frac{\partial m_1}{\partial t}& =\frac{\partial a}{\partial t}+\frac{\partial b}{\partial t}=x_2+\cos (x_1)\\ \frac{\partial m_2}{\partial t}& =\frac{\partial c}{\partial t}+\frac{\partial d}{\partial t}=4\\ & \\ \frac{\partial y_1}{\partial t}& =\frac{\partial m_1}{\partial t}+\frac{\partial m_2}{\partial t}=x_2+\cos (x_1)+4\\ \frac{\partial y_2}{\partial t}& =\frac{\partial m_1}{\partial t}\cdot m_2+\frac{\partial m_2}{\partial t}\cdot m_1=(x_2+\cos (x_1))\cdot m_2+4\cdot m_1\end{array}$$

• 将$t=x_2$代入以上公式，则$\frac{\partial x_1}{\partial t}=0$而$\frac{\partial x_2}{\partial t}=1$，然后可以计算$\frac{\partial y_1}{\partial x_2}$和$\frac{\partial y_2}{\partial x_2}$

可以推断：

• 当有$n$个输入变量时（本例中有2个），需要计算$n$次上述公式。
• 假设神经网络中的输入是一张1280 x 720的图片，输出是51个浮点数，那么前向微分方法则需要计算921600次。

反向自动微分

我们将用到如下的链式法则：

$$\begin{array}{rl}\frac{\partial s}{\partial u}& =\sum _i\left(\frac{\partial w_i}{\partial u}\cdot \frac{\partial s}{\partial w_i}\right)\\ & =\frac{\partial w_1}{\partial u}\cdot \frac{\partial s}{\partial w_1}+\frac{\partial w_2}{\partial u}\cdot \frac{\partial s}{\partial w_2}+\cdots \end{array}$$

其中：

• $u$ 表示输入变量
• $w_i$ 表示依赖 $u$ 的输出变量
• $s$ 表示有待给出的变量

回顾拆解后的简单算子计算（2）：

$$\begin{array}{rl}{x}_1& =?\\ x_2& =?\\ & \\ a& =x_1\cdot x_2\\ b& =\sin (x_1)\\ & \\ c& =4x_1+2x_2\\ d& =\cos (x_2)\\ & \\ m_1& =a+b\\ m_2& =c+d\\ & \\ y_1& =m_1+m_2\\ y_2& =m_1\cdot m_2\end{array}\text{\hspace{1em}(2)}$$

现在计算反向微分：

$$\begin{array}{rl}\frac{\partial s}{\partial y_1}& =?\\ \frac{\partial s}{\partial y_2}& =?\\ & \\ \frac{\partial s}{\partial m_1}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial m_1}+\frac{\partial s}{\partial y_2}\frac{\partial y_2}{\partial m_1}\\ \frac{\partial s}{\partial m_2}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial m_2}+\frac{\partial s}{\partial y_2}\frac{\partial y_2}{\partial m_2}\\ & \\ \frac{\partial s}{\partial a}& =\frac{\partial s}{\partial m_1}\frac{\partial m_1}{\partial a}\\ \frac{\partial s}{\partial b}& =\frac{\partial s}{\partial m_1}\frac{\partial m_1}{\partial b}\\ \frac{\partial s}{\partial c}& =\frac{\partial s}{\partial m_2}\frac{\partial m_2}{\partial c}\\ \frac{\partial s}{\partial d}& =\frac{\partial s}{\partial m_2}\frac{\partial m_2}{\partial d}\\ & \\ \frac{\partial s}{\partial x_1}& =\frac{\partial s}{\partial a}\frac{\partial a}{\partial x_1}+\frac{\partial s}{\partial b}\frac{\partial b}{\partial x_1}+\frac{\partial s}{\partial c}\frac{\partial c}{\partial x_1}\\ \frac{\partial s}{\partial x_2}& =\frac{\partial s}{\partial a}\frac{\partial a}{\partial x_1}+\frac{\partial s}{\partial c}\frac{\partial c}{\partial x_1}+\frac{\partial s}{\partial d}\frac{\partial d}{\partial x_1}\end{array}$$

当$s=y_1$时：

$$\begin{array}{rl}\frac{\partial s}{\partial y_1}& =1\\ \frac{\partial s}{\partial y_2}& =0\\ & \\ \frac{\partial s}{\partial m_1}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial m_1}+\frac{\partial s}{\partial y_2}\frac{\partial y_2}{\partial m_1}=1\\ \frac{\partial s}{\partial m_2}& =\frac{\partial s}{\partial y_1}\frac{\partial y_1}{\partial m_2}+\frac{\partial s}{\partial y_2}\frac{\partial y_2}{\partial m_2}=1\\ & \\ \frac{\partial s}{\partial a}& =\frac{\partial s}{\partial m_1}\frac{\partial m_1}{\partial a}=1\\ \frac{\partial s}{\partial b}& =\frac{\partial s}{\partial m_1}\frac{\partial m_1}{\partial b}=1\\ \frac{\partial s}{\partial c}& =\frac{\partial s}{\partial m_2}\frac{\partial m_2}{\partial c}=1\\ \frac{\partial s}{\partial d}& =\frac{\partial s}{\partial m_2}\frac{\partial m_2}{\partial d}=1\\ & \\ \frac{\partial s}{\partial x_1}& =\frac{\partial s}{\partial a}\frac{\partial a}{\partial x_1}+\frac{\partial s}{\partial b}\frac{\partial b}{\partial x_1}+\frac{\partial s}{\partial c}\frac{\partial c}{\partial x_1}=1\cdot x_2+1\cdot \cos (x_1)+1\cdot 4=x_2+\cos (x_1)+4\\ \frac{\partial s}{\partial x_2}& =\frac{\partial s}{\partial a}\frac{\partial a}{\partial x_2}+\frac{\partial s}{\partial c}\frac{\partial c}{\partial x_2}+\frac{\partial s}{\partial d}\frac{\partial d}{\partial x_2}=1\cdot x_1+1\cdot 2+1\cdot (-\sin (x_2))=x_1+2-\sin (x_2)\end{array}$$

同理可以计算当$s=y_2$时。

可以推断：

• 当有$n$个输出变量时（本例中有2个），需要计算$n$次上述公式。
• 假设神经网络中的输入是一张1280 x 720的图片，输出是51个浮点数，那么反向微分方法则需要计算51次。