数学基础——矩阵向量求导

最新推荐文章于 2022-09-25 15:07:01 发布

brycezou

最新推荐文章于 2022-09-25 15:07:01 发布

阅读量695

点赞数 2

分类专栏：机器学习文章标签：机器学习

本文链接：https://blog.csdn.net/gnehcuoz/article/details/53181880

版权

机器学习专栏收录该内容

2 篇文章 0 订阅

订阅专栏

数学基础—矩阵向量求导

brycezou@163.com

1、基本求导法则

用青色表示矩阵，用紫红色表示向量，用黑色表示标量

$y = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ y 1 y 2 ⋮ y m ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ x = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ x 1 x 2 ⋮ x n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟$ $\color{Fuchsia}{\mathbf{y}}=\left(\begin{matrix}y_1\\ y_2\\ \vdots\\y_m\end{matrix}\right)\ \ \ \ \color{Fuchsia}{\mathbf{x}}=\left(\begin{matrix}x_1\\ x_2\\ \vdots\\x_n\end{matrix}\right)$
1）向量 $\mathbf{y}$ 对标量 $x$ 求导

$\partial y \partial x = (\partial y 1 \partial x \partial y 2 \partial x \dots \partial y m \partial x)$ $\frac{\color{Fuchsia}{\partial \mathbf{y}}}{\partial x}= \left(\begin{matrix} \frac{\partial y_1}{\partial x} & \frac{\partial y_2}{\partial x} & \cdots & \frac{\partial y_m}{\partial x} \end{matrix} \right)$
2）标量 $y$ 对向量 $\mathbf{x}$ 求导

$\partial y \partial x = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ \partial y \partial x 1 \partial y \partial x 2 ⋮ \partial y \partial x n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟$ $\frac{\partial y}{\color{Fuchsia}{\partial \mathbf{x}}}= \left(\begin{matrix} \frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2}\\ \vdots\\ \frac{\partial y}{\partial x_n}\\ \ \end{matrix}\right)$
3）向量 $\mathbf{y}$ 对向量 $\mathbf{x}$ 求导

$\partial y \partial x = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ \partial y 1 \partial x 1 \partial y 1 \partial x 2 ⋮ \partial y 1 \partial x n \partial y 2 \partial x 1 \partial y 2 \partial x 2 ⋮ \partial y 2 \partial x n \dots \dots ⋱ \dots \partial y m \partial x 1 \partial y m \partial x 2 ⋮ \partial y m \partial x n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟$ $\frac{\color{Fuchsia}{\partial \mathbf{y}}}{\color{Fuchsia}{\partial \mathbf{x}}}= \left(\begin{matrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_m}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_2}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_1}{\partial x_n} & \frac{\partial y_2}{\partial x_n} & \cdots & \frac{\partial y_m}{\partial x_n}\\ \ & \ & \ & \ \end{matrix}\right)$
4）标量 $y$ 对矩阵 $\mathbf{X}$ 求导

$\partial y \partial X = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ \partial y \partial x 11 \partial y \partial x 21 ⋮ \partial y \partial x m 1 \partial y \partial x 12 \partial y \partial x 22 ⋮ \partial y \partial x m 2 \dots \dots ⋱ \dots \partial y \partial x 1 n \partial y \partial x 2 n ⋮ \partial y \partial x m n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟$ $\frac{\partial y}{\color{Cyan}{\partial \mathbf{X}}}= \left(\begin{matrix} \frac{\partial y}{\partial x_{11}} & \frac{\partial y}{\partial x_{12}} & \cdots & \frac{\partial y}{\partial x_{1n}}\\ \frac{\partial y}{\partial x_{21}} & \frac{\partial y}{\partial x_{22}} & \cdots & \frac{\partial y}{\partial x_{2n}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y}{\partial x_{m1}} & \frac{\partial y}{\partial x_{m2}} & \cdots & \frac{\partial y}{\partial x_{mn}}\\ \ & \ & \ & \ \end{matrix}\right)$
5）矩阵 $\mathbf{Y}$ 对标量 $x$ 求导

$\partial Y \partial x = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ \partial y 11 \partial x \partial y 12 \partial x ⋮ \partial y 1 n \partial x \partial y 21 \partial x \partial y 22 \partial x ⋮ \partial y 2 n \partial x \dots \dots ⋱ \dots \partial y m 1 \partial x \partial y m 2 \partial x ⋮ \partial y m n \partial x ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟$ $\frac{\color{Cyan}{\partial \mathbf{Y}}}{\partial x}= \left(\begin{matrix} \frac{\partial y_{11}}{\partial x} & \frac{\partial y_{21}}{\partial x} & \cdots & \frac{\partial y_{m1}}{\partial x}\\ \frac{\partial y_{12}}{\partial x} & \frac{\partial y_{22}}{\partial x} & \cdots & \frac{\partial y_{m2}}{\partial x}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_{1n}}{\partial x} & \frac{\partial y_{2n}}{\partial x} & \cdots & \frac{\partial y_{mn}}{\partial x} \end{matrix}\right)$
直观上看，对标量求导，结果的形式都要进行转置；而标量对向量和矩阵求导则位置保持不变。

2、定义求导法

令矩阵

$A = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ a 11 a 21 ⋮ a m 1 a 12 a 22 ⋮ a m 2 \dots \dots ⋱ \dots a 1 n a 2 n ⋮ a m n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ y = A x$ $\color{Cyan}{\mathbf{A}}=\left(\begin{matrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & \cdots & a_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{matrix}\right) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \color{Fuchsia}{\mathbf{y}}=\color{Cyan}{\mathbf{A}}\color{Fuchsia}{\mathbf{x}}$
因为

$y i = (A x) i = a i 1 x 1 + a i 2 x 2 + a i 3 x 3 + \dots + a i n x n$ $y_i=(\color{Cyan}{\mathbf{A}}\color{Fuchsia}{\mathbf{x}})_i=a_{i1}x_1+a_{i2}x_2+a_{i3}x_3+\cdots+a_{in}x_n$
所以

$\partial y \partial x = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ \partial y 1 \partial x 1 \partial y 1 \partial x 2 ⋮ \partial y 1 \partial x n \partial y 2 \partial x 1 \partial y 2 \partial x 2 ⋮ \partial y 2 \partial x n \dots \dots ⋱ \dots \partial y m \partial x 1 \partial y m \partial x 2 ⋮ \partial y m \partial x n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ \partial ( A x ) 1 \partial x 1 \partial ( A x ) 1 \partial x 2 ⋮ \partial ( A x ) 1 \partial x n \partial ( A x ) 2 \partial x 1 \partial ( A x ) 2 \partial x 2 ⋮ \partial ( A x ) 2 \partial x n \dots \dots ⋱ \dots \partial ( A x ) m \partial x 1 \partial ( A x ) m \partial x 2 ⋮ \partial ( A x ) m \partial x n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ a 11 a 12 ⋮ a 1 n a 21 a 22 ⋮ a 2 n \dots \dots ⋱ \dots a m 1 a m 2 ⋮ a m n ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ = A T$ $\frac{\color{Fuchsia}{\partial \mathbf{y}}}{\color{Fuchsia}{\partial \mathbf{x}}}= \left(\begin{matrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_m}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_2}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_1}{\partial x_n} & \frac{\partial y_2}{\partial x_n} & \cdots & \frac{\partial y_m}{\partial x_n}\\ \ & \ & \ & \ \end{matrix}\right)= \left(\begin{matrix} \frac{\partial (Ax)_1}{\partial x_1} & \frac{\partial (Ax)_2}{\partial x_1} & \cdots & \frac{\partial (Ax)_m}{\partial x_1}\\ \frac{\partial (Ax)_1}{\partial x_2} & \frac{\partial (Ax)_2}{\partial x_2} & \cdots & \frac{\partial (Ax)_m}{\partial x_2}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial (Ax)_1}{\partial x_n} & \frac{\partial (Ax)_2}{\partial x_n} & \cdots & \frac{\partial (Ax)_m}{\partial x_n}\\ \ & \ & \ & \ \end{matrix}\right)= \left(\begin{matrix} a_{11} & a_{21} & \cdots & a_{m1}\\ a_{12} & a_{22} & \cdots & a_{m2}\\ \vdots & \vdots & \ddots & \vdots\\ a_{1n} & a_{2n} & \cdots & a_{mn} \end{matrix}\right) =\color{Cyan}{\mathbf{A}}^T$

3、维度分析法

令 $\mathbf{A}\in\mathbb{R}^{m\times n},\ \mathbf{u}\in\mathbb{R}^{n\times 1}, \mathbf{x}\in\mathbb{R}^{p\times 1}$ ，且 $\mathbf{A}$ 与 $\mathbf{x}$ 无关，求 $\frac{\partial \mathbf{Au}}{\partial\mathbf{x}}$ 。
分析：因为 $\mathbf{Au}\in\mathbb{R}^{m\times 1}$ ，所以 $\frac{\partial \mathbf{Au}}{\partial\mathbf{x}}\in\mathbb{R}^{p\times m}$ 。又因为结果中必然包含 $\mathbf{A}\in\mathbb{R}^{m\times n}$ ，且与 $\frac{\partial \mathbf{u}}{\partial\mathbf{x}}\in\mathbb{R}^{p\times n}$ 有关，所以通过矩阵维度比较容易得出

\partial A u \partial x = \partial u \partial x A T

$\frac{\partial \mathbf{Au}}{\partial\mathbf{x}}=\frac{\partial\mathbf{u}}{\partial\mathbf{x}}\mathbf{A}^{\mathrm{T}}$

下面综合运用维度分析法练习求导

$\partial x T A y \partial x, A \in ℝ m \times n, x \in ℝ m \times 1, y \in ℝ n \times 1$ $\frac{\partial\mathbf{x^TAy}}{\partial \mathbf{x}},\mathbf{A}\in\mathbb{R}^{m\times n},\mathbf{x}\in\mathbb{R}^{m\times1},\mathbf{y}\in\mathbb{R}^{n\times1}$
由非精确的乘积法则可得

[\partial x T A y \partial x] m \times 1 = \partial ( x T A ) y \partial x = > [\partial x T A \partial x] m \times n [y] n \times 1 + [x T A] 1 \times n [\partial y \partial x] m \times n

$\left[\frac{\partial\mathbf{x^TAy}}{\partial \mathbf{x}}\right]_{m\times 1}=\frac{\partial\mathbf{(x^TA)y}}{\partial \mathbf{x}}=> \left[\frac{\partial\mathbf{x^TA}}{\partial \mathbf{x}}\right]_{m\times n}\left[\mathbf{y}\right]_{n\times 1}+ \left[\mathbf{x^TA}\right]_{1\times n}\left[\frac{\partial \mathbf{y}}{\partial \mathbf{x}}\right]_{m\times n}$

通过等式两边的维度比较可得

$[\partial x T A y \partial x] m \times 1 = [A] m \times n [y] n \times 1 + [\partial y \partial x] m \times n [A T x] n \times 1 = A y + \partial y \partial x A T x$ $\left[\frac{\partial\mathbf{x^TAy}}{\partial \mathbf{x}}\right]_{m\times 1}= \left[\mathbf{A}\right]_{m\times n}\left[\mathbf{y}\right]_{n\times 1}+ \left[\frac{\partial \mathbf{y}}{\partial \mathbf{x}}\right]_{m\times n} \left[\mathbf{A^Tx}\right]_{n\times 1}= \mathbf{Ay}+\frac{\partial \mathbf{y}}{\partial \mathbf{x}}\mathbf{A^Tx}$
令 $\mathbf{y}=\mathbf{x}$ 可得常用的公式

$\partial x T A x \partial x = (A + A T) x$ $\frac{\partial\mathbf{x^TAx}}{\partial \mathbf{x}}=(\mathbf{A+A^T})\ \mathbf{x}$

主要参考资料 http://xuehy.github.io/2014/04/18/2014-04-18-matrixcalc/

brycezou

关注

2
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
数学基础——矩阵向量求导

数学基础—矩阵向量求导 brycezou@163.com 1、基本求导法则用青色表示矩阵，用紫红色表示向量，用黑色表示标量 y=⎛⎝⎜⎜⎜⎜y1y2⋮ym⎞⎠⎟⎟⎟⎟ x=⎛⎝⎜⎜⎜⎜x1x2⋮xn⎞⎠⎟⎟⎟⎟\color{Fuchsia}{\mathbf{y}}=\left(\begin{matrix}y_1\\ y_2\\ \vdots\\y_m\end{matrix}
复制链接

扫一扫