cs224笔记: Lecture 3a Matrix Calculus

Matrix Calculus

1 Jacobian Matrix

假设有函数$f:{\mathbb{R}}^n\to {\mathbb{R}}^m$，即将一个长度为$n$的向量映射成长度为$m$的向量，
$$\mathbf{f}(\mathbf{x})=[f_1(x_1,x_2,...,x_n),f_2(x_1,x_2,...,x_n),...,f_m(x_1,x_2,...,x_n)]$$
Jacobian矩阵是一个$m\times n$的矩阵，定义如下:
$$\frac{\partial \mathbf{f}}{\partial \mathbf{x}}=\left(\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& ...& \frac{\partial f_1}{\partial x_n}\\ ...& ...& ...\\ \frac{\partial f_m}{\partial x_1}& ...& \frac{\partial f_m}{\partial x_n}\end{array}\right)$$
即$(\frac{\partial f}{\partial x}{)}_{ij}=\frac{\partial f_i}{\partial x_j}$，Jacobian矩阵相乘就可以实现链式法则(chain rule)的运算。

2 有用的等式

(1) $\mathbf{z}=\mathbf{W}\mathbf{x}$，则$\frac{\partial \mathbf{z}}{\partial \mathbf{x}}=\mathbf{W}$

假设$\mathbf{W}\in {\mathbb{R}}^{n\times m}$，可以看作是将$m$维向量映射到$n$维向量，展开为标量形式为：
$$\left(\begin{array}{c}{z}_1\\ z_2\\ ...\\ z_n\end{array}\right)=\left(\begin{array}{cccc}{W}_{11}& W_{12}& ...& W_{1m}\\ ...& ...& ...& ...\\ W_{n1}& W_{n2}& ...& W_{nm}\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ ...\\ x_m\end{array}\right)$$

$$z_i=\sum _{k=1}^m{W}_{ik}{x}_k$$

所以Jacobian为$n\times m$:
$$(\frac{\partial \mathbf{z}}{\partial \mathbf{x}}{)}_{ij}=\frac{\partial z_i}{\partial x_j}=\frac{\partial }{\partial x_j}\sum _{k=1}^m{W}_{ik}{x}_k=\sum _{k=1}^m{W}_{ik}\frac{\partial }{\partial x_j}{x}_k=W_{ij}$$
因此有，$\frac{\partial \mathbf{z}}{\partial \mathbf{x}}=\mathbf{W}$

(2)行向量(row vector)，$\mathbf{x}$ ($1\times n$)和$\mathbf{z}$ ($1\times m$)，$\mathbf{z}=\mathbf{x}\mathbf{W}$，则$\frac{\partial \mathbf{z}}{\partial \mathbf{x}}={\mathbf{W}}^T$

同上Jacobian为:
$$(\frac{\partial \mathbf{z}}{\partial \mathbf{x}}{)}_{ij}=\frac{\partial z_i}{\partial x_j}=\frac{\partial }{\partial x_j}\sum _{k=1}^n{x}_k{W}_{ki}=\sum _{k=1}^n{W}_{ki}\frac{\partial }{\partial x_j}{x}_k=W_{ji}$$

因此，$\frac{\partial \mathbf{z}}{\partial \mathbf{x}}={\mathbf{W}}^T$

(3)$\mathbf{z}=\mathbf{x}$，$\frac{\partial \mathbf{z}}{\partial \mathbf{x}}=\mathbf{I}$

$$(\frac{\partial \mathbf{z}}{\partial \mathbf{x}}{)}_{ij}=\frac{\partial z_i}{\partial x_j}=\frac{\partial x_i}{\partial x_j}=\{\begin{array}{l}1,\text{if i=j}\\ 0,\text{otherwise}\end{array}$$

所以可以看出Jacobian矩阵对角线都是1，其他位置都为0，所以是一个单位矩阵(identity matrix)。

(4)$\mathbf{z}=f(\mathbf{x})$，$\frac{\partial \mathbf{z}}{\partial \mathbf{x}}=diag(f^{{}^{\prime }}(\mathbf{x}))$

有$z_i=f(x_i)$，所以：

$$(\frac{\partial \mathbf{z}}{\partial \mathbf{x}}{)}_{ij}=\frac{\partial z_i}{\partial x_j}=\frac{\partial }{\partial x_j}f(x_i)=\{\begin{array}{l}{f}^{{}^{\prime }}(x_i),\text{if i=j}\\ 0,\text{otherwise}\end{array}$$

Jacobian为对角线为$f^{{}^{\prime }}(x_i)$的对角矩阵(diagonal matrix)。

(5)Matrix times column vector with respect to the matrix

$\mathbf{z}=\mathbf{W}\mathbf{x}$，$\delta =\frac{\partial J}{\partial \mathbf{z}}$，$\frac{\partial J}{\partial \mathbf{W}}=\frac{\partial J}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{W}}={\delta }^T{x}^T$

假设我们有一个损失函数$J$(一个标量)，然后要计算它关于$\mathbf{W}\in {\mathbb{R}}^{n\times m}$的梯度，这样我们可以把$J$当作是一个关于$\mathbf{W}$的函数，有$nm$个输入($\mathbf{W}$的size)，和一个输出($J$)，所以$\frac{\partial J}{\partial \mathbf{W}}$的Jacobian是$1\times nm$的向量，但是在实践中，这个方式不是很有用，如果导数以下形式出现会更方便一些，因为它和$\mathbf{W}$的形式保持一致，直接与其相减就可以实现梯度下降，所以将它作为$\frac{\partial J}{\partial \mathbf{W}}$,
$$\frac{\partial J}{\partial \mathbf{W}}=\left(\begin{array}{cccc}\frac{\partial J}{\partial W_{11}}& \frac{\partial J}{\partial W_{12}}& ...& \frac{\partial J}{\partial W_{1m}}\\ ...& ...& ...& ...\\ \frac{\partial J}{\partial W_{n1}}& \frac{\partial J}{\partial W_{n2}}& ...& \frac{\partial J}{\partial W_{nm}}\end{array}\right)$$

$$\begin{gathered} z_k=\sum _{l=1}^m{W}_{kl}{x}_l \\ \frac{\partial z_k}{\partial W_{ij}}=\sum _{l=1}^m{x}_l\frac{\partial }{\partial W_{ij}}{W}_{kl} \end{gathered}$$

这里如果$i=k$且$j=l$，则 $\frac{\partial }{\partial w_{ij}}{w}_{kl}=1$，否则等于0。

$$\frac{\partial z}{\partial W_{ij}}=\left(\begin{array}{c}0\\ .\\ .\\ 0\\ x_j\\ 0\\ .\\ 0\end{array}\right)$$

$$\frac{\partial J}{\partial W_{ij}}=\frac{\partial J}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial W_{ij}}=\sum _{k=1}^n{\delta }_k\frac{\partial z_k}{\partial W_{ij}}={\delta }_i{x}_j$$

$x(m,1)$

$W(n,m)$

$z(n,1)$

$\delta =\frac{\partial J}{\partial z}(1,n)$

$\frac{\partial J}{\partial \mathbf{W}}={\delta }^T{x}^T(n,m)$

(6)Row vector time matrix with respect to the matrix
$\mathbf{z}=\mathbf{x}\mathbf{W}$，$\delta =\frac{\partial J}{\partial \mathbf{z}}$，$\frac{\partial J}{\partial \mathbf{W}}=\frac{\partial J}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{W}}=x^T\delta$

过程同(5)

(7)Cross-Entropy loss with respect to logits

$\hat{\mathbf{y}}=softmax(\theta )$, $J=CE(\mathbf{y},\hat{\mathbf{y}})$, $\frac{\partial J}{\partial \theta }=\hat{\mathbf{y}}-\mathbf{y}$

下面给出过程：
$$\hat{\mathbf{y}}=softmax(\theta )=\left(\begin{array}{c}\frac{exp({\theta }_1)}{{\sum }_{i=1}^n{\theta }_i}\\ .\\ .\\ .\\ \frac{exp({\theta }_n)}{{\sum }_{i=1}^n{\theta }_i}\end{array}\right)\in {\mathbb{R}}^n$$

$$J=CE(\mathbf{y},\hat{\mathbf{y}})=-{\mathbf{y}}^T\log (\hat{\mathbf{y}})$$

因为$y$代表正确标签，会使用one-hot编码，所以只有正确的位置为1，其他位置为0，如下：
$$\mathbf{y}=\left(\begin{array}{c}0\\ ...\\ 0\\ 1\\ 0\\ ...\\ 0\end{array}\right)$$
不失一般性，令$y_k=1$，即第k个为正确值，所以$J$如下：
$$J=-y_k\log ({\hat{y}}_k)=-\log ({\hat{y}}_k)=-\log (\frac{exp({\theta }_k)}{{\sum }_{i=1}^{n}exp({\theta }_i)})=-{\theta }_k+\log (\sum _{i=1}^{n}exp({\theta }_i))$$
则，
$$\begin{array}{rl}\frac{\partial J}{\partial {\theta }_i}& =\frac{\partial }{\partial {\theta }_i}-\log ({\hat{y}}_k)\\ & =\frac{\partial }{\partial {\theta }_i}(-{\theta }_k+\log (\sum _{x=1}^{n}exp({\theta }_x))\\ & =-\frac{\partial }{\partial {\theta }_i}{\theta }_k+\frac{\partial }{\partial {\theta }_i}\log (\sum _{x=1}^{n}exp({\theta }_x))\\ & =-\frac{\partial }{\partial {\theta }_i}{\theta }_k+\frac{1}{{\sum }_{x=1}^{n}exp({\theta }_x)}\frac{\partial }{\partial {\theta }_i}\sum _{x=1}^{n}exp({\theta }_x)\\ & =-\frac{\partial }{\partial {\theta }_i}{\theta }_k+\frac{1}{{\sum }_{x=1}^{n}exp({\theta }_x)}exp({\theta }_i)\\ & =-\frac{\partial }{\partial {\theta }_i}{\theta }_k+{\hat{y}}_i\end{array}$$
等式前半部分只有$i=k$时为1，其他情况为0，所以有：
$$\frac{\partial J}{\partial \theta }=-\mathbf{y}+\hat{\mathbf{y}}$$