Derivatives of matrix

最新推荐文章于 2020-03-10 11:09:56 发布

艳艳儿

最新推荐文章于 2020-03-10 11:09:56 发布

阅读量502

点赞数

分类专栏： linear algebra

本文链接：https://blog.csdn.net/COMEYAN/article/details/50514610

版权

linear algebra 专栏收录该内容

9 篇文章 1 订阅

订阅专栏

matrix V scalar
scalar V matrix
vector V vector
Examples
Reference

1. matrix V scalar

\partial A ( x ) \partial x = [\partial A ( x ) i j \partial x] i, j

$\begin{equation} \frac{\partial A(x)}{\partial x} = \left[\frac{\partial A(x)_{ij}}{\partial x}\right]_{i,j} \end{equation}$

2. scalar V matrix

f ( A ) \partial A = [\partial f ( x ) \partial A i j] i, j

$\begin{equation} \frac{f(A)}{\partial A} = \left[\frac{\partial f(x)}{\partial A_{ij}}\right]_{i,j} \end{equation}$

3. vector V vector

g \to ( V \to ) \partial V \to = ⎡ ⎣ ⎢ g \to ( V \to ) i \partial V \to j ⎤ ⎦ ⎥ i, j

$\begin{equation} \frac{\overrightarrow g( \overrightarrow V)}{\partial \overrightarrow V} = \left[\frac{\overrightarrow g( \overrightarrow V)_i}{\partial \overrightarrow V_j}\right]_{i,j} \end{equation}$

4. Examples

4.1 Derivative of inverse matrix

\Rightarrow \Rightarrow \Rightarrow A - 1 A = I \partial A - 1 A \partial x = 0 \partial A - 1 \partial x A + A - 1 \partial A \partial x = 0 \partial A - 1 \partial x = - A - 1 \partial A \partial x A - 1

$\begin{align} & A^{-1}A = I\\ \Rightarrow&\frac{\partial A^{-1}A }{\partial x}=0\\ \Rightarrow&\frac{\partial A^{-1}}{\partial x}A+A^{-1}\frac{\partial A}{\partial x}=0\\ \Rightarrow& \frac{\partial A^{-1}}{\partial x}=-A^{-1}\frac{\partial A}{\partial x}A^{-1} \end{align}$

For $x=A_{ij}$ with $A$ being nonsymmetric, we have

\partial A \partial A i j = I i j

$\frac{\partial A}{\partial A_{ij}}=I_{ij}$
and

\partial A - 1 \partial A i j = - A - 1 I i j A - 1

$\frac{\partial A^{-1}}{\partial A_{ij}} =-A^{-1}I_{ij}A^{-1}$
and
$\partial 2 A - 1 \partial A i j \partial A k r = \partial \partial A - 1 \partial A k r \partial A i j = - \partial A - 1 I k r A - 1 \partial A i j = - \partial A - 1 \partial A i j I k r A - 1 - A - 1 I k r \partial A - 1 I k r A - 1 \partial A i j = - (- A - 1 I i j A - 1) I k r A - 1 - A - 1 I k r (- A - 1 I i j A - 1) I k r = A - 1 I i j A - 1 I k r A - 1 + A - 1 I k r A - 1 I i j A - 1$ $\begin{align} \frac{\partial^2 A^{-1}}{\partial A_{ij}\partial A_{kr}} &=\frac{\partial \frac{\partial A^{-1}}{\partial A_{kr}}}{\partial A_{ij}} =\frac{-\partial A^{-1}I_{kr}A^{-1}}{\partial A_{ij}}\\ &=- \frac{\partial A^{-1}}{\partial A_{ij}}I_{kr}A^{-1} - A^{-1}I_{kr}\frac{\partial A^{-1}I_{kr}A^{-1}}{\partial A_{ij}}\\ &=-\left(-A^{-1}I_{ij}A^{-1}\right)I_{kr}A^{-1} - A^{-1}I_{kr}\left(-A^{-1}I_{ij}A^{-1}\right)I_{kr}\\ &=A^{-1}I_{ij}A^{-1}I_{kr}A^{-1} +A^{-1}I_{kr}A^{-1}I_{ij}A^{-1} \end{align}$

For $x=A_{ij}$ with $A$ being symmetric, we have

$\partial A \partial A i j = I i j + I j i - δ j i I j i = d I * i j$ $\frac{\partial A}{\partial A_{ij}}=I_{ij}+I_{ji} - \delta_{ji}I_{ji}\overset{d}{=}I_{ij}^*$

Then all can be replaced by $I_{ij}^*$

4.2 Derivative of trace

If A is not symmetric:

$\partial tr ( A ) \partial A = [\partial tr ( A ) \partial A i j] i j = [tr (\partial A \partial A i j)] i j = tr [I i j] i j = I n$ $\begin{align} \frac{\partial \text{tr}(A)}{\partial A} = \left[\frac{\partial \text{tr}(A)}{\partial A_{ij}}\right]_{ij} = \left[\text{tr}\left(\frac{\partial A}{\partial A_{ij}}\right)\right]_{ij} =\text{tr}[I_{ij}]_{ij} = I_n \end{align}$
If A is symmetric:
$\partial tr ( A ) \partial A = [\partial tr ( A ) \partial A i j] i j = [tr (\partial A \partial A i j)] i j = tr [I i j + I j i + δ j i I j i] i j = I n$ $\begin{align} \frac{\partial \text{tr}(A)}{\partial A} = \left[\frac{\partial \text{tr}(A)}{\partial A_{ij}}\right]_{ij} = \left[\text{tr}\left(\frac{\partial A}{\partial A_{ij}}\right)\right]_{ij} = \text{tr}\left[I_{ij} + I_{ji}+\delta_{ji}I_{ji}\right]_{ij} = I_{n} \end{align}$

If S is not symmetric:

$\Rightarrow \Rightarrow f = tr (S - 1 B) \partial f \partial S i j = \partial tr ( S - 1 B ) \partial S i j = tr (\partial S - 1 B S i j) = - tr (S - 1 I i j S - 1 B) = tr (I i j S - 1 B S - 1) =    Θ = S - 1 B S - 1 tr (I i j Θ) = Θ j i \partial tr ( S - 1 B ) \partial S = - (S - 1 B S - 1) T$ $\begin{align} &f = \text{tr}\left(S^{-1}B\right)\\ \Rightarrow &\frac{\partial f}{\partial S_{ij}} = \frac{\partial \text{tr}\left(S^{-1}B\right)}{\partial S_{ij}}=\text{tr}\left(\frac{\partial S^{-1}B}{S_{ij}}\right) = - \text{tr}\left(S^{-1}I_{ij}S^{-1}B\right) \\ &= \text{tr}\left(I_{ij}S^{-1}BS^{-1}\right) \underbrace{=}_{\Theta=S^{-1}BS^{-1}}\text{tr}\left(I_{ij}\Theta\right) = \Theta_{ji}\\ \Rightarrow &\frac{\partial \text{tr}\left(S^{-1}B\right)}{\partial S} = - \left(S^{-1}BS^{-1}\right)^T \end{align}$

If S is symmetric:

$\Rightarrow \Rightarrow f = tr (S - 1 B) \partial f \partial S i j = \partial tr ( S - 1 B ) \partial S i j = tr (\partial S - 1 B S i j) = - tr (I * i j Θ) = - [Θ j i + Θ i j - δ i j Θ i j] \partial tr ( S - 1 B ) \partial S = - [S - 1 (B + B T) S - 1 - diag (S - 1 B S - 1)]$ $\begin{align} &f = \text{tr}\left(S^{-1}B\right)\\ \Rightarrow &\frac{\partial f}{\partial S_{ij}} = \frac{\partial \text{tr}\left(S^{-1}B\right)}{\partial S_{ij}}=\text{tr}\left(\frac{\partial S^{-1}B}{S_{ij}}\right) = - \text{tr}\left(I_{ij}^*\Theta\right) = -[\Theta_{ji}+\Theta_{ij} - \delta_{ij}\Theta_{ij}]\\ \Rightarrow &\frac{\partial \text{tr}\left(S^{-1}B\right)}{\partial S} = - \left[S^{-1}(B+B^T)S^{-1}- \text{diag}\left(S^{-1}BS^{-1}\right)\right] \end{align}$

4.3 Applications

$\partial tr ( A B ) \partial A = B T$ $\frac{\partial \text{tr}(AB)}{\partial A} = B^T$
$\partial tr ( A T B ) \partial A = B$ $\frac{\partial \text{tr}(A^TB)}{\partial A} = B$

$\partial tr ( A T S A ) \partial A = [\partial tr ( A T S A ) \partial A i j] i j = [tr (I j i S A + A T S I i j)] i j = S A + S A T A = (S + S T) A$ $\begin{align} \frac{\partial \text{tr}(A^T S A)}{\partial A} =\left[\frac{\partial \text{tr}(A^T S A)}{\partial A_{ij}} \right]_{ij} =\left[\text{tr}\left(I_{ji}SA+A^T S I_{ij}\right)\right]_{ij} =SA+SA^TA = (S+S^T)A \end{align}$

S is symmetric

$\partial tr ( ( A T S A ) - 1 R ) \partial A = ⎡ ⎣ ⎢ ⎢ ⎢ \partial tr ( ( A T S A ) - 1 R ) \partial A i j ⎤ ⎦ ⎥ ⎥ ⎥ i j = ⎡ ⎣ ⎢ ⎢ tr ⎛ ⎝ ⎜ ⎜ \partial ( A T S A ) - 1 R \partial A i j ⎞ ⎠ ⎟ ⎟ ⎤ ⎦ ⎥ ⎥ i j = [tr (- (A T S A) - 1 \partial A T S A \partial A i j (A T S A) - 1 R)] i j = [tr (- (A T S A) - 1 (I j i S A + A T S I i j) (A T S A) - 1 R)] i j = [- tr (I j i S A (A T S A) - 1 R (A T S A) - 1)] i j + [- tr ((A T S A) - 1 R (A T S A) - 1 A T S I i j)] i j = - S A (A T S A) - 1 R (A T S A) - 1 - S A (A T S A) - 1 R T (A T S A) - 1 = - S A (A T S A) - 1 (R + R T) (A T S A) - 1$ $\begin{align} &\frac{\partial \text{tr}\left(\left(A^T S A\right)^{-1}R\right)}{\partial A} \\ &= \left[\frac{\partial \text{tr}\left(\left(A^T S A\right)^{-1}R\right)}{\partial A_{ij}} \right]_{ij}=\left[\text{tr}\left(\frac{\partial \left(A^T S A\right)^{-1} R }{\partial A_{ij}}\right)\right]_{ij} \\ &=\left[\text{tr}\left(-\left(A^T S A\right)^{-1}\frac{\partial A^T S A}{\partial A_{ij}}\left(A^T S A\right)^{-1} R\right)\right]_{ij}\\ &=\left[\text{tr}\left(-\left(A^T S A\right)^{-1}\left(I_{ji}SA +A^T S I_{ij}\right)\left(A^T S A\right)^{-1} R\right)\right]_{ij}\\ &=\left[- \text{tr}\left(I_{ji}SA\left(A^T S A\right)^{-1} R \left(A^T S A\right)^{-1}\right)\right]_{ij}\\ & +\left[- \text{tr}\left(\left(A^T S A\right)^{-1} R\left(A^T S A\right)^{-1}A^T S I_{ij}\right)\right]_{ij}\\ &=-SA\left(A^T S A\right)^{-1} R \left(A^T S A\right)^{-1} - SA\left(A^T S A\right)^{-1} R^T \left(A^T S A\right)^{-1}\\ &=-SA\left(A^T S A\right)^{-1} \left(R+R^T\right) \left(A^T S A\right)^{-1} \end{align}$

$\partial tr ( ( A T S 2 A ) - 1 ( A T S 1 A ) ) \partial A = \partial tr ( ( A T 1 S 2 A 1 ) - 1 ( A T S 1 A ) ) \partial A ∣ ∣ ∣ ∣ ∣ ∣ A 1 = A + \partial tr ( ( A T S 2 A ) - 1 ( A T 2 S 1 A 2 ) ) \partial A ∣ ∣ ∣ ∣ ∣ ∣ A 2 = A = - 2 S 2 A (A T S 2 A) - 1 (A T S 1 A) (A T S 2 A) - 1 + 2 S 1 A (A T S 2 A) - 1$ $\begin{align} &\frac{\partial \text{tr}\left( \left(A^TS_2 A\right)^{-1}\left(A^TS_1A\right)\right)}{\partial A}\\ &=\left.\frac{\partial \text{tr}\left( \left(A_1^TS_2 A_1\right)^{-1}\left(A^TS_1A\right)\right)}{\partial A}\right|_{A_1=A}\\ &+\left.\frac{\partial \text{tr}\left( \left(A^TS_2 A\right)^{-1}\left(A_2^TS_1A_2\right)\right)}{\partial A}\right|_{A_2=A}\\ &=-2S_2A \left(A^TS_2 A\right)^{-1}\left(A^TS_1A\right)\left(A^TS_2 A\right)^{-1}+2S_1A\left(A^TS_2 A\right)^{-1} \end{align}$

4.4 Derivative of determinant

$\partial | R | \partial R i j = C i j$ $\frac{\partial |R|}{\partial R_{ij}}=C_{ij}$
which is its cofactor(from the cofactor expansion). Then
$\partial | R | \partial R = C$ $\frac{\partial |R|}{\partial R}=C$
which is the adjoint matrix. Then from the fact that
$| R | I = R C T$ $|R|I=R C^T$
we have
$\partial | R | \partial R = | R | R - 1 T$ $\frac{\partial |R|}{\partial R}=|R|{R^{-1}}^T$

Furthermore

$\partial ln | R | \partial R = 1 | R | \partial | R | \partial R = R - 1 T$ $\frac{\partial \text{ln}|R|}{\partial R}=\frac{1}{|R|}\frac{\partial |R|}{\partial R}={R^{-1}}^T$

5. Reference

reference of book “Introduction to Statistical Pattern Recognition, second edition.”

艳艳儿

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Derivatives of matrix

matrix V scalarscalar V matrixvector V vectorexamples1 derivative of inverse matrix2 derivative of trace1. matrix V scalar∂A(x)∂x=[∂A(x)ij∂x]i,j\begin{equation}\frac{\partial A(x)}{\partial x} =
复制链接

扫一扫