矩阵微分_矩阵的微分-CSDN博客

本文链接：https://blog.csdn.net/xfzero/article/details/80336043

矩阵微分

标签（空格分隔）：矩阵微分

矩阵微分
在学习机器学习算法时,发现对矩阵求导很不熟悉,去看了张贤达的矩阵分析,发现标量对矩阵求导的问题说的很清楚.关于如何求解 hessian矩阵,日后再来补上.
重要的事情说三遍
损失函数是标量函数!!!
损失函数是标量函数!!!
损失函数是标量函数!!!

主要介绍实值函数相对于实向量变量或者矩阵变量的偏导.这里首先对变元和函数符号做统一的规定以便后面介绍.
$\pmb x = [x_1, ..., x_m]^T \in R^m$ 为实向量变元
$\pmb X = [\pmb x_1, ..., \pmb x_m]^T \in R^{m \times n}$ 为矩阵变元
$f(\pmb x) \in R 为实值标量函数,其变元\pmb x \in R^m,记做f:R^{m} \to R$
$f(\pmb X) \in R 为实值标量函数,其变元\pmb X \in R^{m \times n},记做f:R^{m \times n} \to R$
$\pmb f(\pmb x) \in R^p 为p维实列向量函数,其变元\pmb x \in R^m,记做f:R^{m} \to R^p$
$\pmb f(\pmb X) \in R^p 为p维实列向量函数,其变元\pmb X \in R^{m \times n},记做f:R^{m \times n} \to R^p$
$\pmb F(\pmb x) \in R^{p \times q} 为p \times q 实矩阵函数,其变元\pmb x \in R^m,记做f:R^{m} \to R^{p \times q}$
$\pmb F(\pmb X) \in R^{p \times q} 为p \times q 实矩阵函数,其变元\pmb X \in R^{m \times n},记做f:R^{m \times n} \to R^{p \times q}$

Jacobian 矩阵

采用 $1 \times m$ 行向量作为偏导算子,记为

D x x = d e f [\partial \partial x 1, . . ., \partial \partial x m]

$D_{\pmb x} \overset{def}{=} [ \cfrac {\partial}{\partial x_1}, ..., \cfrac {\partial }{\partial x_m}]$

实值标量函数 $f(\pmb x) 在 \pmb x 的偏导向量为 1\times m 行向量,定义如下$

D x x f (x x) = \partial f ( x x ) \partial x x T = [\partial f ( x x ) \partial x 1, . . ., \partial f ( x x ) \partial x m]

$D_{\pmb x} f(\pmb x ) = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T} = [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ..., \cfrac {\partial f(\pmb x)}{\partial x_m}]$

当实值标量函数 $f(\pmb X)的变元是p \times q$ 维矩阵的时候,他有两种定义:Jacobian矩阵和行向量偏导.他的Jacobian矩阵定义为如下

D X X f (X X) = \partial f ( X X ) \partial X X T

$D_{\pmb X} f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial \pmb X^T}$ 而他的行向量偏导定义为

D v e c X X f (X X) = \partial f ( X X ) \partial v e c ( X X ) T = [\partial f ( x x ) \partial x 1, . . . \partial f ( x x ) \partial x m 1, . . ., \partial f ( x x ) \partial x 1 n, . . ., \partial f ( x x ) \partial x m n]

$D_{vec \pmb X } f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial vec(\pmb X)^T}= [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ...\cfrac {\partial f(\pmb x)}{\partial x_{m1}},..., \cfrac {\partial f(\pmb x)}{\partial x_{1n}},...,\cfrac {\partial f(\pmb x)}{\partial x_{mn}}]$ 这里需要注意的是,实值标量函数

f(XX) f ( X X ) $f(\pmb X)$ 的Jacobian矩阵的转置

DTXXf(XX) D X X T f ( X X ) $D_{\pmb X}^Tf(\pmb X)$ 的列向量化后即为他的行向量偏导

DvecXXf(XX) D v e c X X f ( X X ) $D_{vec \pmb X}f(\pmb X)$ ,这是后面介绍Jacobian矩阵相关知识的基础

当FF(XX)为p×q实矩阵函数时,定义他的Jacobian矩阵如下当 F F ( X X ) 为 p × q 实矩阵函数时 , 定义他的 J a c o b i a n 矩阵如下 $当\pmb F(\pmb X)为p \times q 实矩阵函数时,定义他的Jacobian矩阵如下$

D X X F F (X X) = d e f \partial v e c ( F F ( X X ) ) \partial ( v e c X X ) T

$D_{\pmb X} \pmb F(\pmb X)\overset{def}{=} \cfrac {\partial vec(\pmb F(\pmb X))}{\partial (vec \pmb X)^T}$

梯度矩阵

采用列向量形式的偏导算子称为列向量偏导算子,也称为梯度算子
采用 $m \times 1$ 向量作为偏导算子,记为

\nabla x x = d e f [\partial \partial x 1, . . ., \partial \partial x m] T

$\nabla _{\pmb x} \overset{def}{=} [ \cfrac {\partial}{\partial x_1}, ..., \cfrac {\partial }{\partial x_m}]^T$

实值标量函数 $f(\pmb x) 在 \pmb x 的梯度向量为 m\times 1 列向量,定义如下$

\nabla x x f (x x) = [\partial f ( x x ) \partial x 1, . . ., \partial f ( x x ) \partial x m] T

$\nabla _{\pmb x} f(\pmb x) = [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ..., \cfrac {\partial f(\pmb x)}{\partial x_m}]^T$

实值标量函数 $f(\pmb X) 的变元 \pmb X 列向量化后,可以定义其矩阵变元\pmb X 的梯度向量为$

\nabla v e c X X f (X X) = \partial f ( X X ) \partial v e c ( X X ) = [\partial f ( x x ) \partial x 1, . . . \partial f ( x x ) \partial x m 1, . . ., \partial f ( x x ) \partial x 1 n, . . ., \partial f ( x x ) \partial x m n] T

$\nabla _{vec \pmb X} f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial vec(\pmb X)}= [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ...\cfrac {\partial f(\pmb x)}{\partial x_{m1}},..., \cfrac {\partial f(\pmb x)}{\partial x_{1n}},...,\cfrac {\partial f(\pmb x)}{\partial x_{mn}}]^T$
定义

f(XX)的其关于矩阵变元XX的梯度矩阵为 f ( X X ) 的其关于矩阵变元 X X 的梯度矩阵为 $f(\pmb X) 的其关于矩阵变元\pmb X 的梯度矩阵为$

\nabla X X f (X X) = \partial f ( X X ) \partial X X

$\nabla _{\pmb X} f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial \pmb X}$ 比较

f(XX) f ( X X ) $f(\pmb X)$ 的梯度矩阵和Jacobian矩阵,可以发现梯度矩阵即为Jacobian矩阵的转置
当实值标量函数数

f(XX)的变元是p×q维矩阵的时候,他的梯度向量定义为如下 f ( X X ) 的变元是 p × q 维矩阵的时候 , 他的梯度向量定义为如下 $f(\pmb X)的变元是p \times q维矩阵的时候,他的梯度向量定义为如下$

\nabla X X F F (X X) = d e f \partial v e c ( F F ( X X ) ) \partial ( v e c X X ) T

$\nabla_{\pmb X} \pmb F(\pmb X)\overset{def}{=} \cfrac {\partial vec(\pmb F(\pmb X))}{\partial (vec \pmb X)^T}$

标量函数 $f(\pmb x)$ 与Jacobian矩阵

以向量为变元的标量函数 $f(\pmb x)$ 的全微分形式可以写为

d f (x x) = \partial f ( x x ) \partial x 1 d x 1 + . . . + \partial f ( x x ) \partial x m d x m = \partial f ( x x ) \partial x x T d x x

$df(\pmb x) = \cfrac {\partial f(\pmb x)}{\partial x_1} dx_1 + ... + \cfrac {\partial f(\pmb x)}{\partial x_m} dx_m = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T} d \pmb x$

记AA=∂f(xx)∂xxT 记 A A = ∂ f ( x x ) ∂ x x T $记\pmb A = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T}$ ,则有如下等价关系

d f (x x) = t r (A d x x) ⟺ D x x f (x x) = \partial f ( x x ) \partial x x T = A

$df(\pmb x) = tr(Ad \pmb x) \iff D_{\pmb x} f(\pmb x ) = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T} = A$
也就是标量函数

f(xx) f ( x x ) $f(\pmb x)$ 的Jacobian矩阵和微分矩阵存在着等价关系

标量函数 $f(\pmb X)$ 与Jacobian矩阵

标量函数 $f(\pmb X)$ 和上面类似,其全微分可以写成如下形式

d f (X X) = \partial f ( X X ) \partial x x T 1 d x x 1 + . . . + \partial f ( X X ) \partial x x T n d x x n = \partial f ( X X ) \partial v e c T ( ( X ) d (v e c X X) = D v e c X X f (X X) d (v e c X X)

$\begin{align*} df(\pmb X) & = \cfrac {\partial f(\pmb X)}{\partial \pmb x_1^T} d \pmb x_1 + ... + \cfrac {\partial f(\pmb X)}{\partial \pmb x_n^T} d \pmb x_n \\ &= \cfrac {\partial f(\pmb X)}{\partial vec^T \pmb (X)} d (vec \pmb X) \\ &= D_{vec \pmb X} f(\pmb X) d(vec \pmb X) \end{align*}$
这里再利用行向量偏导和Jacobian矩阵的关系

DvecXXf(XX)=(vec(DTXXf(XX)))T，并令A=DTXXf(XX) D v e c X X f ( X X ) = ( v e c ( D X X T f ( X X ) ) ) T ，并令 A = D X X T f ( X X ) $D_{vec \pmb X} f(\pmb X) = (vec (D_{\pmb X}^T f(\pmb X)))^T，并令A=D_{\pmb X}^T f(\pmb X)$ 可以得到

d f (X X) = (v e c (A T)) T d (v e c X X)

$\begin{align*} df(\pmb X) & = (vec(A^T))^T d(vec \pmb X) \end{align*}$
由向量化算子vec与迹函数的关系式

tr(BTC)=(vec(B))Tvec(C),令B=AT,C=dXX t r ( B T C ) = ( v e c ( B ) ) T v e c ( C ) , 令 B = A T , C = d X X $tr(B^TC) = (vec(B))^Tvec(C),令B = A^T, C = d \pmb X$ ,则上式可以重写为

d f (X X) = t r (A A d X X)

$df(\pmb X) = tr(\pmb A d \pmb X)$

综合以上,可以得到如下结论:
Jacobian矩阵可以通过以下式子等价确定

d f (x x) = t r (A d x x) ⟺ D x x f (x x) = A d f (X X) = t r (A d X X) ⟺ D X X f (X X) = A

$df(\pmb x) = tr(A d \pmb x) \iff D_{\pmb x} f(\pmb x) = A \\ df(\pmb X) = tr(A d \pmb X) \iff D_{\pmb X} f(\pmb X) = A$

矩阵微分 $df(\pmb X)$ 可以通过简单的变化转化为矩阵微分的标准形式 $df(\pmb X) = tr(A d \pmb X)$ .再由Jacobian和梯度矩阵的关系,进一步可以得到梯度矩阵.因而对求解梯度矩阵可以由矩阵微分的标准形式 $df(\pmb X) = tr(A d \pmb X)$ 得到.
如下:
对于 $tr(\pmb X^T \pmb X)$ 我们可以得到

d t r (X X T X X) = t r (d (X X T X X)) = t r (d (X X) T X X + X X T d X X) = t r (d (X X) T X X) + t r (X X T d X X) = t r (X X T d (X X)) + t r (X X T d X X) = t r (2 X X T d (X X))

$\begin{align*} d tr(\pmb X^T \pmb X) & = tr(d(\pmb X^T \pmb X)) \\ &= tr(d(\pmb X )^T \pmb X + \pmb X^T d \pmb X)\\ &= tr(d(\pmb X )^T \pmb X) +tr( \pmb X^T d \pmb X) \\ &= tr(\pmb X ^ T d(\pmb X )) +tr( \pmb X^T d \pmb X) \\ &= tr(2 \pmb X ^ T d(\pmb X )) \end{align*}$
由以上结论,可以得到