（二）实值函数相对于向量的梯度

最新推荐文章于 2023-07-13 15:15:26 发布

HawardScut

最新推荐文章于 2023-07-13 15:15:26 发布

阅读量7.1k

点赞数 3

分类专栏：机器学习中的数学

本文链接：https://blog.csdn.net/hao5335156/article/details/80322623

版权

机器学习中的数学专栏收录该内容

6 篇文章 2 订阅

订阅专栏

1、定义

以n维向量 $x$ 为变元的实标量函数 $f(x)$ 相对于 $x$ 的梯度结果为n*1列向量，定义为

▽_{x} f (x) = [\begin{matrix} \frac{\partial f (x)}{\partial x_{1}} \\ \frac{\partial f (x)}{\partial x_{2}} \\ ⋮ \\ \frac{\partial f (x)}{\partial x_{n}} \end{matrix}] = {[\begin{matrix} \frac{\partial f (x)}{\partial x_{1}}, & \frac{\partial f (x)}{\partial x_{2}}, & . . ., & \frac{\partial f (x)}{\partial x_{n}} \end{matrix}]}^{T} = \frac{\partial f (x)}{\partial x}

$\bigtriangledown_{x}f(x)=\begin{bmatrix}\frac{\partial f(x)}{\partial x_{1}} \\ \frac{\partial f(x)}{\partial x_{2}} \\ \vdots \\ \frac{\partial f(x)}{\partial x_{n}} \end{bmatrix}=\begin{bmatrix} \frac{\partial f(x)}{\partial x_{1}},&\frac{\partial f(x)}{\partial x_{2}},&...,&\frac{\partial f(x)}{\partial x_{n}} \end{bmatrix}^T=\frac{\partial f(x)}{\partial x}$

其中， $x$ 默认为列向量， $x^T$ 默认为行向量。

2、拓展定义

2.1 实标量函数 $f(x)$ 相对于1*n行向量 $x^T$ 的梯度结果为1*n行向量，定义为

\partial f ( x ) \partial x T = [\partial f ( x ) \partial x 1, \partial f ( x ) \partial x 2, . . ., \partial f ( x ) \partial x n] = ▽ x T f (x)

$\frac{\partial f(x)}{\partial x^T}=\begin{bmatrix} \frac{\partial f(x)}{\partial x_{1}},&\frac{\partial f(x)}{\partial x_{2}},&...,&\frac{\partial f(x)}{\partial x_{n}} \end{bmatrix}=\bigtriangledown_{x^T}f(x)$

2.2 m维行向量函数 $f(x)=\begin{bmatrix} f_{1}(x),&f_{2}(x),&…,&f_{m}(x) \end{bmatrix}$ 相对n维实向量 $x$ ( $x$ 默认是列向量)的梯度为一 $n*m$ 矩阵，定义为

\partial f ( x ) \partial x = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ \partial f 1 ( x ) \partial x 1 \partial f 1 ( x ) \partial x 2 ⋮ \partial f 1 ( x ) \partial x n \partial f 2 ( x ) \partial x 1 \partial f 2 ( x ) \partial x 2 ⋮ \partial f 2 ( x ) \partial x n . . . . . . . . . \partial f m ( x ) \partial x 1 \partial f m ( x ) \partial x 2 ⋮ \partial f m ( x ) \partial x n ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ = ▽ x f (x)

$\frac{\partial f(x)}{\partial x}=\begin{bmatrix} \frac{\partial f_{1}(x)}{\partial x_{1}}&\frac{\partial f_{2}(x)}{\partial x_{1}}&...&\frac{\partial f_{m}(x)}{\partial x_{1}}\\ \frac{\partial f_{1}(x)}{\partial x_{2}}&\frac{\partial f_{2}(x)}{\partial x_{2}}&...&\frac{\partial f_{m}(x)}{\partial x_{2}}\\ \vdots &\vdots& &\vdots \\ \frac{\partial f_{1}(x)}{\partial x_{n}}&\frac{\partial f_{2}(x)}{\partial x_{n}}&...&\frac{\partial f_{m}(x)}{\partial x_{n}}\\ \end{bmatrix} =\bigtriangledown_{x}f(x)$
两个特例：
（1）若m*1向量函数

f(x)=y=[y1,y2,…,ym]T f ( x ) = y = [ y 1 , y 2 , … , y m ] T $f(x)=y=\begin{bmatrix} y_{1},&y_{2},&…,&y_{m} \end{bmatrix}^T$ ，其中

y1,y2,…,ym y 1 , y 2 , … , y m $y_{1},y_{2},…,y_{m}$ 是向量的标量函数。则一阶梯度

\partial y \partial x T = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ \partial y 1 \partial x 1 \partial y 2 \partial x 1 ⋮ \partial y m \partial x 1 \partial y 1 \partial x 2 \partial y 2 \partial x 2 ⋮ \partial y m \partial x 2 . . . . . . . . . \partial y 1 \partial x n \partial y 2 \partial x n ⋮ \partial y m \partial x n ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥

$\frac{\partial y}{\partial x^T}=\begin{bmatrix} \frac{\partial y_{1}}{\partial x_{1}}&\frac{\partial y_{1}}{\partial x_{2}}&...&\frac{\partial y_{1}}{\partial x_{n}}\\ \frac{\partial y_{2}}{\partial x_{1}}&\frac{\partial y_{2}}{\partial x_{2}}&...&\frac{\partial y_{2}}{\partial x_{n}}\\ \vdots &\vdots& &\vdots \\ \frac{\partial y_{m}}{\partial x_{1}}&\frac{\partial y_{m}}{\partial x_{2}}&...&\frac{\partial y_{m}}{\partial x_{n}}\\ \end{bmatrix}$
是一个

m∗n m ∗ n $m*n$ 矩阵，称为 向量函数

y=[y1,y2,...,ym]T y = [ y 1 , y 2 , . . . , y m ] T $y=\begin{bmatrix}y_{1},&y_{2},&...,&y_{m}\end{bmatrix}^T$ 的 Jacobi矩阵。
（2）若

f(x)=[x1,x2,...,xn] f ( x ) = [ x 1 , x 2 , . . . , x n ] $f(x)=[x_{1},x_{2},...,x_{n}]$ ，则

\partial x T \partial x = I = \partial f ( x ) \partial x （ I 是 单 位 矩 阵 ）

$\frac{\partial x^T}{\partial x}=I=\frac{\partial f(x)}{\partial x}（I是单位矩阵）$
例如:

x=[x1,x2,x3]T x = [ x 1 , x 2 , x 3 ] T $x=[x_{1},x_{2},x_{3}]^T$ ，则

xT=[x1,x2,x3] x T = [ x 1 , x 2 , x 3 ] $x^T=[x_{1},x_{2},x_{3}]$ ，所以

\partial x T \partial x = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ \partial x 1 \partial x 1 \partial x 1 \partial x 2 \partial x 1 \partial x 3 \partial x 2 \partial x 1 \partial x 2 \partial x 2 \partial x 2 \partial x 3 \partial x 3 \partial x 1 \partial x 3 \partial x 2 \partial x 3 \partial x 3 ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ = ⎡ ⎣ ⎢ 100010001 ⎤ ⎦ ⎥ = I

$\frac{\partial x^T}{\partial x}=\begin{bmatrix} \frac{\partial x_{1}}{\partial x_{1}}&\frac{\partial x_{2}}{\partial x_{1}}&\frac{\partial x_{3}}{\partial x_{1}}\\ \frac{\partial x_{1}}{\partial x_{2}}&\frac{\partial x_{2}}{\partial x_{2}}&\frac{\partial x_{3}}{\partial x_{2}}\\ \frac{\partial x_{1}}{\partial x_{3}}&\frac{\partial x_{2}}{\partial x_{3}}&\frac{\partial x_{3}}{\partial x_{3}} \end{bmatrix}=\begin{bmatrix} 1& 0& 0\\ 0 & 1 & 0\\ 0& 0 & 1 \end{bmatrix}=I$
公式 $\frac{\partial x^T}{\partial x}=I（I是单位矩阵）$ 非常有用

3、导出的基本公式

$A$ 和 $y$ 均与向量 $x$ 无关，有

（1） $\frac{\partial x^TAy}{\partial x}=\frac{\partial x^T}{\partial x}Ay=Ay$

（2） $\frac{\partial y^TAx}{\partial x}=A^Ty$

（3） $\frac{\partial x^TAx}{\partial x}=Ax+A^Tx$

（4） $\frac{\partial x^TAx}{\partial x}=2Ax(A为对称矩阵,转置矩阵等于本身)$

注：矩阵的转置

$(A T) T = A$ $(A^T)^T=A$

$(A + B) T = A T + B T$ $(A+B)^T=A^T+B^T$

$(λ A) T = λ A T$ $(\lambda A)^T=\lambda A^T$

$(A B) T = B T A T$ $(AB)^T=B^TA^T$
参考
[1]《矩阵分析与运用》第5章
[2]《线性代数》第二版