矩阵求导

最新推荐文章于 2024-07-08 17:03:29 发布

杭州草民

最新推荐文章于 2024-07-08 17:03:29 发布

阅读量379

点赞数 2

分类专栏：机器学习文章标签：矩阵求导反向传播矩阵相乘反向传播

本文链接：https://blog.csdn.net/virtual_earth/article/details/90478925

版权

机器学习专栏收录该内容

2 篇文章 0 订阅

订阅专栏

矩阵在机器学习中用的比较多，比如反向传播的时候，本来矩阵相关运算的反向传播很简单，但是网上的资料都写的很复杂，我尽量以简单的形式来表述矩阵相关运算的反向传播过程。

矩阵相乘的反向传播
矩阵点乘的反向传播
矩阵乘向量的反向传播
向量乘矩阵的反向传播
矩阵乘标量的反向传播

本文用表示标量， $\textbf{x}$ 表示向量，表示矩阵， $\mathbf{x}$ 表示列向量， $\mathbf{x^{^{T}}}$ 表示行向量

矩阵相乘反向传播

设为3行2列的矩阵，为2行4列的矩阵，为的结果矩阵，为3行4列，将其分别表示出来，则：

$X=\begin{bmatrix} x_{11} &x_{12} \\ x_{21} &x_{22} \\ x_{31} &x_{32} \end{bmatrix}$

$Y=\begin{bmatrix} y_{11} &y_{12} &y_{13} &y_{14} \\ y_{21} &y_{22} &y_{23} &y_{24} \end{bmatrix}$

$Z=\begin{bmatrix} z_{11} &z_{12} &z_{13} &z_{14} \\ z_{21} &z_{22} &z_{23} &z_{24} \\ z_{31} &z_{32} &z_{33} &z_{34} \end{bmatrix}$

其中 $z_{ij}=\sum_{k=1}^{2} x_{ik}y_{kj},i=1,2,3; j=1,2,3,4$

则loss对X的偏导数为：

${\frac{\partial l}{\partial X}} =\begin{bmatrix} ^{\frac{\partial l}{\partial x_{11}}}&^{\frac{\partial l}{\partial x_{12}}} \\ ^{\frac{\partial l}{\partial x_{21}}}&^{\frac{\partial l}{\partial x_{22}}} \\ ^{\frac{\partial l}{\partial x_{31}}}&^{\frac{\partial l}{\partial x_{32}}} \end{bmatrix} = \begin{bmatrix} \sum_{i=1,j=1}^{3,4}{\frac{\partial l}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial x_{11}}} & \sum_{i=1,j=1}^{3,4}{\frac{\partial l}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial x_{12}}}\\ \sum_{i=1,j=1}^{3,4}{\frac{\partial l}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial x_{21}}}& \sum_{i=1,j=1}^{3,4}{\frac{\partial l}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial x_{22}}}\\ \sum_{i=1,j=1}^{3,4}{\frac{\partial l}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial x_{31}}}& \sum_{i=1,j=1}^{3,4}{\frac{\partial l}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial x_{32}}} \end{bmatrix}$ （1）

所以 ${\frac{\partial l}{\partial X}}={\frac{\partial l}{\partial Z}} {\frac{\partial Z}{\partial X}} ={\frac{\partial l}{\partial Z}} Y^{T}$

同理可证 ${\frac{\partial l}{\partial Y}}={\frac{\partial l}{\partial Z}} {\frac{\partial Z}{\partial Y}} =X^{T}{\frac{\partial l}{\partial Z}}$

矩阵点乘的反向传播

同理，Z=Xdot(Y),设

$X=\begin{bmatrix} x_{11} &x_{12} \\ x_{21} &x_{22} \\ x_{31} &x_{32} \end{bmatrix}$

$Y=\begin{bmatrix} y_{11} &y_{12} \\ y_{21} &y_{22} \\ y_{31} &y_{32} \end{bmatrix}$

$Z=\begin{bmatrix} z_{11} &z_{12} \\ z_{21} &z_{22} \\ z_{31} &z_{32} \end{bmatrix}$

其中 $z_{ij}=x_{ij}y_{ij},i=1,2,3; j=1,2;$

${\frac{\partial l}{\partial X}} =\begin{bmatrix} ^{\frac{\partial l}{\partial x_{11}}}&^{\frac{\partial l}{\partial x_{12}}} \\ ^{\frac{\partial l}{\partial x_{21}}}&^{\frac{\partial l}{\partial x_{22}}} \\ ^{\frac{\partial l}{\partial x_{31}}}&^{\frac{\partial l}{\partial x_{32}}} \end{bmatrix} = \begin{bmatrix} {\frac{\partial l}{\partial z_{11}}\frac{\partial z_{11}}{\partial x_{11}}} & {\frac{\partial l}{\partial z_{12}}\frac{\partial z_{12}}{\partial x_{12}}}\\ {\frac{\partial l}{\partial z_{21}}\frac{\partial z_{21}}{\partial x_{21}}}& {\frac{\partial l}{\partial z_{22}}\frac{\partial z_{22}}{\partial x_{22}}}\\ {\frac{\partial l}{\partial z_{31}}\frac{\partial z_{31}}{\partial x_{31}}}& {\frac{\partial l}{\partial z_{32}}\frac{\partial z_{32}}{\partial x_{32}}} \end{bmatrix}=\frac{\partial l}{\partial Z}dot(Y)$

同理

${\frac{\partial l}{\partial Y}}=\frac{\partial l}{\partial Z}dot(X)$

矩阵乘向量的反向传播

$\textbf{y}=W\textbf{x}$ ，设W为3行2列，x为2行1列，y3行1列：
$W=\begin{bmatrix} w_{11} &w_{12} \\ w_{21} &w_{22} \\w_{31} &w_{32} \end{bmatrix}$

$\textbf{x}=\begin{bmatrix} x11\\ x21 \end{bmatrix}$

$\textbf{y}=\begin{bmatrix} y11\\ y21 \\y31\end{bmatrix}$

${\frac{\partial l}{\partial \textbf{x}}} =\begin{bmatrix} ^{\frac{\partial l}{\partial x_{11}}}\\ ^{\frac{\partial l}{\partial x_{21}}} \end{bmatrix} = \begin{bmatrix} {\frac{\partial l}{\partial y_{11}}\frac{\partial y_{11}}{\partial x_{11}}}+{\frac{\partial l}{\partial y_{21}}\frac{\partial y_{21}}{\partial x_{11}}} +{\frac{\partial l}{\partial y_{31}}\frac{\partial y_{31}}{\partial x_{11}}} \\ {\frac{\partial l}{\partial y_{11}}\frac{\partial y_{11}}{\partial x_{21}}}+{\frac{\partial l}{\partial y_{21}}\frac{\partial y_{21}}{\partial x_{21}}}+{\frac{\partial l}{\partial y_{31}}\frac{\partial y_{31}}{\partial x_{21}}}\end{bmatrix}=\begin{bmatrix} {\frac{\partial l}{\partial y_{11}}}w_{11}+{\frac{\partial l}{\partial y_{21}}}w_{21}+{\frac{\partial l}{\partial y_{31}}}w_{31}\\ {\frac{\partial l}{\partial y_{11}}}w_{12}+{\frac{\partial l}{\partial y_{21}}}w_{22}+{\frac{\partial l}{\partial y_{31}}}w_{32} \end{bmatrix}=W^{T}{\frac{\partial l}{\partial \textbf{y}}}$

同理可证 ${\frac{\partial l}{\partial W}} ={\frac{\partial l}{\partial \textbf{y}}}\mathbf{x^{T}}$

向量乘矩阵的反向传播

$\textbf{y}=\textbf{x}W$ ，设W为3行2列，x为1行3列，y为1行2列：
$W=\begin{bmatrix} w_{11} &w_{12} \\ w_{21} &w_{22} \\w_{31} &w_{32} \end{bmatrix}$

$\textbf{x}=\begin{bmatrix} x11& x12 & x13\end{bmatrix}$

$\textbf{y}=\begin{bmatrix} y11& y12\end{bmatrix}$

${\frac{\partial l}{\partial \textbf{x}}} =\begin{bmatrix} \frac{\partial l}{\partial x_{11}}& \frac{\partial l}{\partial x_{12}} & \frac{\partial l}{\partial x_{13}}\end{bmatrix} =\begin{bmatrix} \frac{\partial l}{\partial y_{11}}\frac{\partial y_{11}}{\partial x_{11}}+ \frac{\partial l}{\partial y_{12}}\frac{\partial y_{12}}{\partial x_{11}}& \frac{\partial l}{\partial y_{11}}\frac{\partial y_{11}}{\partial x_{12}}+\frac{\partial l}{\partial y_{12}}\frac{\partial y_{12}}{\partial x_{12}} & \frac{\partial l}{\partial y_{11}}\frac{\partial y_{11}}{\partial x_{13}}+ \frac{\partial l}{\partial y_{12}}\frac{\partial y_{12}}{\partial x_{13}} \end{bmatrix}=\begin{bmatrix} \frac{\partial l}{\partial y_{11}}w_{11}+ \frac{\partial l}{\partial y_{12}}w_{12}& \frac{\partial l}{\partial y_{11}}w_{21}+ \frac{\partial l}{\partial y_{12}}w_{22} & \frac{\partial l}{\partial y_{11}}w_{31}+ \frac{\partial l}{\partial y_{12}}w_{32} \end{bmatrix}={\frac{\partial l}{\partial \textbf{y}}}W^{T}$

同理可证 ${\frac{\partial l}{\partial W}} =\mathbf{x^{T}}{\frac{\partial l}{\partial \textbf{y}}}$

矩阵乘标量的反向传播

Y=xW ，设W为3行2列，x为标量，y为3行2列：
$W=\begin{bmatrix} w_{11} &w_{12} \\ w_{21} &w_{22} \\w_{31} &w_{32} \end{bmatrix}$

x=x11

$Y=\begin{bmatrix} y_{11} &y_{12} \\ y_{21} &y_{22} \\y_{31} &y_{32} \end{bmatrix}$

${\frac{\partial l}{\partial x}} =\frac{\partial l}{\partial y_{11}}\frac{\partial y_{11}}{\partial x}+\frac{\partial l}{\partial y_{12}}\frac{\partial y_{12}}{\partial x}+\frac{\partial l}{\partial y_{21}}\frac{\partial y_{21}}{\partial x}+\frac{\partial l}{\partial y_{22}}\frac{\partial y_{22}}{\partial x}+\frac{\partial l}{\partial y_{31}}\frac{\partial y_{31}}{\partial x}+\frac{\partial l}{\partial y_{32}}\frac{\partial y_{32}}{\partial x}=\frac{\partial l}{\partial y_{11}}w_{11}+\frac{\partial l}{\partial y_{12}}w_{12}+\frac{\partial l}{\partial y_{21}}w_{21}+\frac{\partial l}{\partial y_{22}}w_{22}+\frac{\partial l}{\partial y_{31}}w_{31}+\frac{\partial l}{\partial y_{32}}w_{32}=\sum ({\frac{\partial l}{\partial y}}\cdot W)$

${\frac{\partial l}{\partial W}} =\begin{bmatrix} ^{\frac{\partial l}{\partial w_{11}}}&^{\frac{\partial l}{\partial w_{12}}} \\ ^{\frac{\partial l}{\partial w_{21}}}&^{\frac{\partial l}{\partial w_{22}}} \\ ^{\frac{\partial l}{\partial w_{31}}}&^{\frac{\partial l}{\partial w_{32}}} \end{bmatrix} =\begin{bmatrix} \frac{\partial l}{\partial y_{11}}\frac{\partial y_{11}}{\partial w_{11}}& \frac{\partial l}{\partial y_{12}}\frac{\partial y_{12}}{\partial w_{12}} \\ \frac{\partial l}{\partial y_{21}}\frac{\partial y_{21}}{\partial w_{21}} &\frac{\partial l}{\partial y_{22}}\frac{\partial y_{22}}{\partial w_{22}} \\ \frac{\partial l}{\partial y_{31}}\frac{\partial y_{31}}{\partial w_{31}}& \frac{\partial l}{\partial y_{32}}\frac{\partial y_{32}}{\partial w_{32}} \end{bmatrix}=x\cdot {\frac{\partial l}{\partial Y}}$