Computing Nerual Network Gradients

1. Matrix times column vector with respect to the column vector $(z = Wx$ ,what is $\frac{d z}{d x})$
   where $W \in R^{n\times m},x \in R^m$
   

   $$\frac{d z}{d x} = W$$

2. Row vector times matrix with respect to the row vector ($z = xW$,what is $\frac{d z}{d x}$)
   where $W \in R^{n\times m},x\in R^n$
   

   $$\frac{d z}{d x} = W^T$$

3. A vector with itself ($z = x$，what is $\frac{d z}{d x}$?) this is just the identity matrix:
   

   $$\frac{d z}{d x} = I$$
   this term will disappear because a matrix multiplied by the identity matrix does not change

4. An elementwise function applied a vector($z = f(x)$，what is $\frac{d z}{d x}$?)
   

   $$(\frac{d z}{d x})_{ij} = \frac{d z}{d x} = \frac{d}{d x_j}f(x_i) = \begin{cases} f'(x_i), & \text{if $i=j$} \\ 0, & \text{if otherwise} \end{cases}$$
   we can write this as $\frac{d z}{d x} = diag(f'(x))$。since multiplication by a diagonal matrix is the same as doing elementwise multipication by the diagonal，we could also write $\circ f'(x)$ when applying the chain rule

5. Matrix times column vector with respect to the matrix($z = Wx,\delta=\frac{d J}{d z}$ what is $\frac{d J}{d W}=\frac{d J}{d z}\frac{dz}{d W}=\delta\frac{d z}{d W}$?) where $W \in R^{n*m},x \in R^m,z \in R^n$
   

   $$z_k=\sum_{l=1}^mW_{kl}x_l$$
   $$\frac{d z_k}{d W_{ij}}=\sum_{l=1}^mx_l\frac{d}{d W_{ij}}W_{kl}$$
   Note that $\frac{d}{d W_{ij}}W_{kl}=1$ if $i=k$ and $j=l$ and 0 if otherwise。so if $k \neq i$ everything in the sum is zero and the gradient is zero，Otherwise，the only nonzero element of the sum is when l=j . so
   $$\frac{d z_k}{d W_{ij}}=x_{j}$$
   Now let’s compute
   $$\frac{d J}{d W_{ij}}=\frac{d J}{d z}\frac{d z}{d w_{ij}}=\delta\frac{d z}{d W_{ij}}=\sum_{k=1}^m\delta_k\frac{d z_k}{d W_{ij}}=\delta_ix_j$$
   (the only nonzero term in the sum is $\delta_i\frac{d z_i}{d W_{ij}}$). To get $\frac{d J}{d W}$ we want a matrix where entry $(i,j)$ is $\delta_{i}x_j$. This matrix is equal to the outer product
   $$\frac{d J}{d W} = \delta x^T$$

6. Row vector time matrix with respect to the matrix($z=xW, \delta=\frac{d J}{d z}$ what is $\frac{d J}{d W}=\delta\frac{d z}{d W}$?) where $W \in R^{n*m},x \in R^{1*n},z \in R^{1*m}$
   A similar computation to (5）shows that
   

   $$\frac{d J}{d W}=x^T\delta$$

7. Cross-entropy loss with respect to logits($\hat{y}=softmax(\theta),J=CE(y.\hat{y}),$what is $\frac{d J}{d \theta}$?)
   $$\frac{d J}{d \theta}=\hat{y}-y$$