【深度学习】动手学深度学习(PyTorch版)李沐 2.4.3 梯度【公式推导】

CoralRhino

已于 2024-01-10 23:50:06 修改

阅读量1.4k

点赞数 20

文章标签：深度学习人工智能

于 2024-01-10 17:32:37 首次发布

本文链接：https://blog.csdn.net/zhangjiuding/article/details/135500038

版权

2.4.3. 梯度

我们可以连接一个多元函数对其所有变量的偏导数，以得到该函数的梯度（gradient）向量。具体而言，设函数 $f:\mathbb{R}^{n}\to\mathbb{R}$ 的输入是一个 $n$ 维向量 $\vec x=\begin{bmatrix} x_1\\x_2\\···\\x_n\end{bmatrix}$ ，输出是一个标量。函数 $f(\vec x)$ 相对于 $\vec x$ 的梯度是一个包含 $n$ 个偏导数的向量：
$\nabla_{\vec x} f(\vec x) = \begin{bmatrix}\frac{\partial f(\vec x)}{\partial x_1}\\\frac{\partial f(\vec x)}{\partial x_2}\\···\\ \frac{\partial f(\vec x)}{\partial x_n}\end{bmatrix}$
其中 $\nabla_{\vec x} f(\vec x)$ 通常在没有歧义时被 $\nabla f(\vec x)$ 取代。

假设 $\vec x$ 为 $n$ 维向量，在微分多元函数时经常使用以下规则:

一、对于所有 $\in \mathbb{R^{m\times n}}$ ，都有 $\nabla_{\vec x} A\vec x = A^\top$ ；

证明：设 $A_{(m,n)}$ = $\begin{bmatrix} a_{1,1}&a_{1,2}&···&a_{1,n} \\ a_{2,1}&a_{2,2}&···&a_{2,n} \\ ··· & ··· & ··· & ··· \\ a_{m,1} & a_{m,2} &···&a_{m,n} \end{bmatrix}$ ，
则 $A\vec x_{(m,1)}$ = $\begin{bmatrix} a_{1,1}x_1+a_{1,2}x_2+···+a_{1,n}x_n \\ a_{2,1}x_1+a_{2,2}x_2+···+a_{2,n}x_n \\ ··· \\ a_{m,1}x_1+a_{m,2}x_2+···+a_{m,n}x_n \end{bmatrix}$ ,
$\nabla_{\vec x}A\vec x$ = $\begin{bmatrix}\frac{\partial A\vec x}{\partial x_1}\\\frac{\partial A\vec x}{\partial x_2}\\···\\ \frac{\partial A\vec x}{\partial x_n}\end{bmatrix}$
= $\begin{bmatrix}\frac{\partial a_{1,1}x_1+a_{1,2}x_2+···+a_{1,n}x_n}{\partial x_1}& \frac{\partial a_{2,1}x_1+a_{2,2}x_2+···+a_{2,n}x_n}{\partial x_1}&···&\frac{\partial a_{m,1}x_1+a_{m,2}x_2+···+a_{m,n}x_n}{\partial x_1}\\ \frac{\partial a_{1,1}x_1+a_{1,2}x_2+···+a_{1,n}x_n}{\partial x_2}& \frac{\partial a_{2,1}x_1+a_{2,2}x_2+···+a_{2,n}x_n}{\partial x_2}&···&\frac{\partial a_{m,1}x_1+a_{m,2}x_2+···+a_{m,n}x_n}{\partial x_2}\\ ···&···&···&···\\ \frac{\partial a_{1,1}x_1+a_{1,2}x_2+···+a_{1,n}x_n}{\partial x_n}& \frac{\partial a_{2,1}x_1+a_{2,2}x_2+···+a_{2,n}x_n}{\partial x_n}&···&\frac{\partial a_{m,1}x_1+a_{m,2}x_2+···+a_{m,n}x_n}{\partial x_n}\end{bmatrix}$
= $\begin{bmatrix} a_{1,1} & a_{2,1} & ··· & a_{m,1}\\ a_{1,2} & a_{2,2} & ··· & a_{m,2} \\ ···&···&···&··· \\ a_{1,n}&a_{2,n}&···&a_{m,n} \end{bmatrix}$ = $A^\top$

二、对于所有 $\in \mathbb{R^{n\times m}}$ ，都有 $\nabla_{\vec x} \vec x^\top A = A$ ；

证明：设 $A_{(n,m)}$ = $\begin{bmatrix} a_{1,1}&a_{1,2}&···&a_{1,m} \\ a_{2,1}&a_{2,2}&···&a_{2,m} \\ ··· & ··· & ··· & ··· \\ a_{n,1} & a_{n,2} &···&a_{n,m} \end{bmatrix}$ ，
则 $\vec x^\top A$ =
$\begin{bmatrix} a_{1,1}x_1+a_{2,1}x_2+···+a_{n,1}x_n & a_{1,2}x_1+a_{2,2}x_2+···+a_{n,2}x_n & ···&a_{1,m}x_1+a_{2,m}x_2+···+a_{n,m}x_n \end{bmatrix}$ ,
$\nabla_{\vec x}\vec x^\top A$ = $\begin{bmatrix}\frac{\partial \vec x^\top A}{\partial x_1}\\\frac{\partial \vec x^\top A}{\partial x_2}\\···\\ \frac{\partial \vec x^\top A}{\partial x_n}\end{bmatrix}$
= $\begin{bmatrix}\frac{\partial a_{1,1}x_1+a_{2,1}x_2+···+a_{n,1}x_n}{\partial x_1}& \frac{\partial a_{1,2}x_1+a_{2,2}x_2+···+a_{n,2}x_n}{\partial x_1}&···&\frac{\partial a_{1,m}x_1+a_{2,m}x_2+···+a_{n,m}x_n}{\partial x_1}\\ \frac{\partial a_{1,1}x_1+a_{2,1}x_2+···+a_{n,1}x_n}{\partial x_2}& \frac{\partial a_{1,2}x_1+a_{2,2}x_2+···+a_{n,2}x_n}{\partial x_2}&···&\frac{\partial a_{1,m}x_1+a_{2,m}x_2+···+a_{n,m}x_n}{\partial x_2}\\ ···&···&···&···\\ \frac{\partial a_{1,1}x_1+a_{2,1}x_2+···+a_{n,1}x_n}{\partial x_n}& \frac{\partial a_{1,2}x_1+a_{2,2}x_2+···+a_{n,2}x_n}{\partial x_n}&···&\frac{\partial a_{1,m}x_1+a_{2,m}x_2+···+a_{n,m}x_n}{\partial x_n}\end{bmatrix}$
= $\begin{bmatrix} a_{1,1} & a_{1,2}&···&a_{1,m}\\ a_{2,1}&a_{2,2}&···&a_{2,m} \\ ···&···&···&···\\ a_{n,1}&a_{n,2}&···&a_{n,m} \end{bmatrix}$ = $A$

三、对于所有 $\in \mathbb{R^{n\times n}}$ ，都有 $\nabla_{\vec x} \vec x^\top A \vec x = (A+A^\top)\vec x$ ；

证明：设 $A_{(n,n)}$ = $\begin{bmatrix} a_{1,1}&a_{1,2}&···&a_{1,n} \\ a_{2,1}&a_{2,2}&···&a_{2,n} \\ ··· & ··· & ··· & ··· \\ a_{n,1} & a_{n,2} &···&a_{n,n} \end{bmatrix}$ ，
则 $\vec x^\top A$ = $\begin{bmatrix} a_{1,1}x_1+a_{2,1}x_2+···+a_{n,1}x_n & a_{1,2}x_1+a_{2,2}x_2+···+a_{n,2}x_n & ···&a_{1,n}x_1+a_{2,n}x_2+···+a_{n,n}x_n \end{bmatrix}$ ,
$\vec x^\top A \vec x$ = $\begin{bmatrix} \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} (a_{i,j}x_ix_j) \end{bmatrix}$ ,
$\nabla_{\vec x}\vec x^\top A \vec x$ = $\begin{bmatrix} \frac{\partial \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} (a_{i,j}x_ix_j)}{\partial x_1} \\ \frac{\partial \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} (a_{i,j}x_ix_j)}{\partial x_2} \\ ···\\ \frac{\partial \sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} (a_{i,j}x_ix_j)}{\partial x_n} \end{bmatrix}$ = $\begin{bmatrix} \sum\limits_{i=1}^{n}(a_{i,1}+a_{1,i})x_i \\ \sum\limits_{i=1}^{n}(a_{i,2}+a_{2,i})x_i \\ ···\\ \sum\limits_{i=1}^{n}(a_{i,n}+a_{n,i})x_i \\ \end{bmatrix}$
= $\begin{bmatrix} 2a_{1,1} & a_{1,2}+a_{2,1} & ···&a_{1,n}+a_{n,1} \\ a_{2,1}+a_{1,2} & 2a_{2,2} & ···&a_{2,n}+a_{n,2} \\ ···&···&···&···\\ a_{n,1}+a_{1,n} & a_{n,2}+a_{2,n} & ···&2a_{n,n} \\ \end{bmatrix} \begin{bmatrix} x_1\\ x_2\\ ···\\ x_n \end{bmatrix}$ = $(A+A^\top)\vec x$

四、 $\nabla_{\vec x} \Vert x \Vert ^2=\nabla_{\vec x}\vec x^\top\vec x = 2\vec x$ 。

证明： $\nabla_{\vec x}\Vert x \Vert ^2$ = $\nabla_{\vec x}\sqrt{x_1^2+x_2^2+···+x_n^n}^2$ = $\nabla_{\vec x}x_1^2+x_2^2+···+x_n^n$ = $\nabla_{\vec x}x^\top x$ ；
$\nabla_{\vec x}\Vert x \Vert ^2$ = $\nabla_{\vec x}\sqrt{x_1^2+x_2^2+···+x_n^n}^2$ = $\nabla_{\vec x}x_1^2+x_2^2+···+x_n^n$ = $\begin{bmatrix} 2x_1\\ 2x_2\\ ···\\ 2x_n \end{bmatrix}$ = $2 x$

同样，对于任何矩阵 $X$ ，都有 $\nabla_X \Vert X \Vert_F^2=2X$ 。正如我们之后将看到的，梯度对于设计深度学习中的优化算法有很大用处。

五、对于任何矩阵 $X$ ，都有 $\nabla_X \Vert X \Vert_F^2=2X$

证明：设 $X$ 为 $m\times n$ 的矩阵， $\begin{bmatrix} x_{1,1}& x_{1,2}&···&x_{1,n}\\ x_{2,1}& x_{2,2}&···&x_{2,n}\\ ···&···&···&···\\ x_{m,1}& x_{m,2}&···&x_{m,n}\\ \end{bmatrix}$ ，
则 $\Vert X \Vert_F^2$ = $\sqrt{\sum\limits_{i=1}^{m}\sum\limits_{j=1}^n x_{i,j}^2}^2$ = $\sum\limits_{i=1}^{m}\sum\limits_{j=1}^n x_{i,j}^2$ ，
$\nabla_X \Vert X \Vert_F^2$ = $\begin{bmatrix} 2x_{1,1}& 2x_{1,2}&···&2x_{1,n}\\ 2x_{2,1}& 2x_{2,2}&···&2x_{2,n}\\ ···&···&···&···\\ 2x_{m,1}& 2x_{m,2}&···&2x_{m,n}\\ \end{bmatrix}$ = $2 X$