Matrix Derivative

Matrix Derivative

1.1 Definition

For a function $f(X)$, where $X\in R^{m\times n}$ is a matrix and $f(X)\in R$ is scalar. We define the derivative of the function as $\frac{\partial f}{\partial X}=\left[\frac{\partial f}{\partial X_{ij}}\right]$, where $\frac{\partial f}{\partial X}$ is a matrix with the same shape as $X$. However, sometimes it’s not easy to calculate the result of it, since we have to calculate the derivative of the elements one by one. Next we will see how to use the trace trick to simplify the result.

1.2 Trace of Matrix

We define the trace of square matrix $A\in R^{n\times n}$, which has the same number of rows as columns, as $tr(A)={\sum }_{i=1}^n{A}_{ii}$, where $A_{ii},i=1,...,n$ is the diagonal elements of $A$. The trace of a square matrix is the sum of its diagonal elements. For a scalar $a$, we regard it as a matrix whose shape is $1\times 1$, and then $tr(a)=a$.
The trace is not so trivial so far. While the multiplication of two matrices
$$tr(A^{T}B)=\sum _{i=0}^n\sum _{j=0}^m{A}_{ij}{B}_{ij}\text{\hspace{1em}(1)}$$
where $A,B\in R^{m\times n}$. It is the sum of the multiplication of corresponding elements. In fact, it is similar with the inner product of two vectors.
To verify equation $(1)$ is not difficult, we can use $C=A^{T}B$, then $c_{ii}=a_i^T{b}_i$, where $c_{ii}$ is the $i$th rows and $i$th column of $C$, $a_i^T$ is the $i$th row of $A^T$, e.q. $i$th column of $A$, and $b_i$ is $i$th column of $B$. So $c_{ii}$ is the inner product of $i$th column of $A$ and $i$th column of $B$.
Then $tr(A^{T}B)={\sum }_{i=0}^n{c}_{ii}$ is the “inner product” of $A$ and $B$.

Here we list some trace tricks which are useful.

• $a=tr(a)$, where $a$ is a scalar.
• $tr(A^T)=tr(A)$.
• $tr(A\pm B)=tr(A)\pm tr(B)$.
• $tr(A^{T}B)=tr(B{A}^T)={\sum }_{i,j}{A}_{ij}{B}_{ij}$, where $A,B\in R^{n\times n}$.
• $tr(ABC)=tr(BCA)=tr(CAB)$, where $A\in R^{r\times m},B\in R^{m\times n},C\in R^{n\times r}$. Particularly, if $a\in R^m,c\in R^n$, $a$ and $c$ are both column vectors. Then $tr(a^{T}Bc)=tr(c{a}^{T}B)=tr(Bc{a}^T)$.
• $tr(A^T(B⨀C)=tr((A⨀B{)}^{T}C)$, where $A,B,C\in R^{m\times n}$, and $⨀$ means multiply of element-wise.

1.3 Trace and Derivative

You should mention that in this article the $f$ is function which map a scalar, vector or matrix to a scalar. Besides, the vector is column vector by default.
First we show the total differential of scalar function. If there is a function $f(x)$, then $df(x)=f^{\prime }(x)dx$, where $x\in R$.
Second, if $x\in R^n$ is scalar, then $df(x)=\nabla f^{T}dx$, where $\nabla f\in R^n$ is the gradient of $f(x)$, its $i$th component is $(\nabla f{)}_i=\frac{\partial f}{\partial x_i}$. $dx\in R^n$ is also a vector and $(dx{)}_i=d{x}_i$. We can also rewrite $df(x)={\sum }_i\frac{\partial f}{\partial x_i}d{x}_i$.
At last, similar to the vector condition, the total differential of $f(X)$ can be write as
$$df(X)=\sum _{i,j}\frac{\partial f}{\partial X_{ij}}d{X}_{ij}\text{\hspace{1em}(2)}$$
Where $X\in R^{m\times n}$ and $f(X)$ is a scalar. $\frac{\partial f}{\partial X}$ and $dX$ are both matrices with shape $m\times n$ and $(\frac{\partial f}{\partial X}{)}_{ij}=\frac{\partial f}{\partial X_{ij}}$ and $(dX{)}_{ij}=d{X}_{ij}$.

Compare equation $(1)$ and equation $(2)$, we can see that
$$df(X)=tr\left({\left(\frac{\partial f}{\partial X}\right)}^{T}dX\right)\text{\hspace{1em}(3)}$$
Equation $(3)$ inspires us that if we can calculate the total differential of $f$ with the format of $(3)$, then we can get the derivative of $f$, which is $\frac{\partial f}{\partial X}$. The trick is that using trace and let $dX$ at the end of the equation, just like eqution (3).

1.4 Total Differential

If $f=uv$, then $df=vdu+udv$, where $u,v$ are scalars.
If $u,v\in R^n$ and $f=u^{T}v={\sum }_i{u}_i{v}_i$, where $u_i$ and $v_i$ is the $i$th element of $u$ and $v$ respectively. Then
$$\begin{array}{rl}df& =d(\sum _i{u}_i{v}_i)\\ & =\sum _{i}d(u_i{v}_i)\\ & =\sum _i(u_{i}d{v}_i+v_{i}d{u}_i)\\ & =\sum _i{u}_{i}d{v}_i+\sum _i{v}_{i}d{u}_i\\ & =u^{T}dv+v^{T}du\end{array}$$
Similarly, if $U,V\in R^{m\times n}$, and $f=tr(U^{T}V)$, then
$$df=tr(Vd{U}^T)+tr(U^{T}dV)$$

1.5 Examples:

(1) Linear Combination

$f(X)=a^{T}Xb$, where $a\in R^m,b\in R^n,X\in R^{m\times n}$. We need to calculate $\frac{\partial f}{\partial X}$. Hint: $f$ is scalar and $df$ is a scalar as well, then $df=tr(df)$
Solution:
$$df=tr(df)=tr(d(a^{T}Xb))=tr(a^{T}dXb)=tr(b{a}^{T}dX)$$
Compare equation (3), we have $(\frac{\partial f}{\partial X}{)}^T=b{a}^T$, which means $\frac{\partial f}{\partial X}=a{b}^T$.
We can check it by the shape, since $x\in R^{m\times n}$, then $\frac{\partial f}{\partial X}\in R^{m\times n}$ as well. Besides, $a{b}^n\in R^{m\times n}$ can be verified easily. It show that the result is reasonable.

(2) Least Square

$l(w)=\Vert Xw-y{\Vert }^2$, find $\frac{\partial l}{\partial w}$. In the least square method, $X\in R^{m\times n}$ is a data matrix, $w\in R^n$ is weight vector, $y\in R^m$ is target label.
Solution:
$$l=\Vert Xw-y{\Vert }^2=(Xw-y{)}^T(Xw-y)$$
Notice that $Xw-y$ is a vector now.
$$\begin{array}{rl}tr(dl)& =dl\\ & =(Xdw{)}^T(Xw-y)+(Xw-y{)}^T(Xdw)\\ & =2(Xw-y{)}^{T}Xdw\end{array}$$
Then $\frac{\partial l}{\partial w}=2X^T(Xw-y)$.

(3) PCA

$l(w)=w^T\Sigma w+\lambda (1-w^{T}w)$, where $w\in R^n,\Sigma \in R^{n\times n}$, $\Sigma$ is symmetric, which means ${\Sigma }^T=\Sigma$, and $\lambda$ is a scalar. Find $\frac{\partial l}{\partial w}$.
Solution:
$$\begin{array}{rl}dl& =tr(dl)\\ & =tr(d{w}^T\Sigma w)+tr(w^T\Sigma dw)-\lambda (tr(d{w}^{T}w)+tr(w^{T}dw))\\ & =tr(w^T{\Sigma }^{T}dw)+tr(w^T\Sigma dw)-\lambda (tr(w^{T}dw)+tr(w^{T}dw))\\ & =tr(2(\Sigma w-\lambda w{)}^{T}dw)\end{array}$$
Then we have $\frac{\partial l}{\partial w}=2(\Sigma w-\lambda w)$.

(4) SVM

What th SVM need do is find the proper weight $w$, which
$$\begin{gathered} \underset{w}{\min }\frac{1}{2}\Vert w{\Vert }^2 \\ s.t.1-y_i(w^T{x}_i)\le 0,i=1,2,3,\cdots ,M \end{gathered}$$
Where w , x i ∈ R n , y i ∈ { − 1 , 1 } w,x_i\in R^n, y_i\in \{-1,1\} w,xi​∈Rn,yi​∈{−1,1}, $M$ is the number of data. The Lagrangian
$$L(w,\lambda )=\frac{1}{2}\Vert w{\Vert }^2+\sum _{i=1}^M{\lambda }_i(1-y_i(w^T{x}_i))$$
where $\lambda \in R^M$ and ${\lambda }_i$ is $i$th component of $\lambda$, here $\lambda \succcurlyeq 0$, which means ${\lambda }_i\ge 0$.
Find $\frac{\partial L}{\partial w}$.
Solution:
$$\begin{array}{r}\frac{\partial L}{\partial w}=w-\sum _{i=1}^M{\lambda }_i{y}_i{x}_i\end{array}$$
I find it is simple enough to use the trace tricks…