CMU 11-785 L13 Recurrent Networks

Modelling Series

• In many situations one must consider a series of inputs to produce an output

  • Outputs too may be a series

• Finite response model

  • Can use convolutional neural net applied to series data (slide)

    • Also called a Time-Delay neural network

  • Something that happens today only affects the output of the system for days into the future

    • $$Y_t=f\left(X_t,X_{t-1},\dots ,X_{t-N}\right)$$

• Infinite response systems

  • Systems often have long-term dependencies

  • What happens today can continue to affect the output forever

    • $$Y_t=f\left(X_t,X_{t-1},\dots ,X_{t-\infty }\right)$$

Infinite response systems

• A one-tap NARX network

  • 「nonlinear autoregressive network with exogenous inputs」
  • $$Y_t=f(X_t,Y_{t-1})$$
  • An input at t=0 affects outputs forever
    

• An explicit memory variable whose job it is to remember

  • $$\begin{array}{c}{m}_t=r\left(y_{t-1},h_{t-1}^{\prime },m_{t-1}\right)\\ \\ h_t=f\left(x_t,m_t\right)\\ \\ y_t=g\left(h_t\right)\end{array}$$

• Jordan Network

  • Memory unit simply retains a running average of past outputs
  • Memory has fixed structure; does not “learn” to remember

• Elman Networks

  • Separate memory state from output
  • Only the weight from the memory unit to the hidden unit is learned
    • But during training no gradient is backpropagated over the “1” link (Just cloned state)

• Problem

  • “Simple” (or partially recurrent) because during learning current error does not actually propagate to the past

State-space model

$$\begin{array}{c}{h}_t=f\left(x_t,h_{t-1}\right)\\ \\ y_t=g\left(h_t\right)\end{array}$$

• $h_t$ is the state of the network
  • Model directly embeds the memory in the state
  • State summarizes information about the entire past
• Recurrent neural network

Variants

• All columns are identical

• The simplest structures are most popular

Recurrent neural network

Forward pass

Backward pass

• BPTT
  • Back Propagation Through Time
  • Defining a divergence between the actual and desired output sequences
  • Backpropagating gradients over the entire chain of recursion
    • Backpropagation through time
  • Pooling gradients with respect to individual parameters over time

Notion

• The divergence computed is between the sequence of outputs by the network and the desired sequence of outputs
  • DIV is a scalar function of a …series… of vectors
  • This is not just the sum of the divergences at individual times
• $Y(t)$ is the output at time $t$
  • $Y_i(t)$ is the ith output
• $Z^{(2)}(t)$ is the pre-activation value of the neurons at the output layer at time $t$
• $h(t)$ is the output of the hidden layer at time $t$

BPTT

• $Y(t)$ is a column vector
• $DIV$ is a scalar
• $\frac{dDiv}{dY(t)}$ is a row vector

Derivative at time $T$

1. Compute $\frac{dDIV}{d{Y}_i(T)}$ for all $i$

   • In general we will be required to compute $\frac{dDIV}{d{Y}_i(t)}$ for all $i$ and $t$ as we will see

     • This can be a source of significant difficulty in many scenarios

   • Special case, when the overall divergence is a simple sum of local divergences at each time

     • $$\frac{dDIV}{d{Y}_i(t)}=\frac{dDiv(t)}{d{Y}_i(t)}$$

2. Compute ${\nabla }_{Z^{(2)}(T)}DIV$

   • $${\nabla }_{Z^{(2)}(T)}DIV={\nabla }_{Y(T)}DIV{\nabla }_{Z^{(2)}(T)}Y(T)$$

   • For scalar output activation

     • $$\frac{dDIV}{d{Z}_i^{(2)}(T)}=\frac{dDIV}{d{Y}_i(T)}\frac{d{Y}_i(T)}{d{Z}_i^{(2)}(T)}$$

   • For vector output activation

     • $$\frac{dDIV}{d{Z}_i^{(2)}(T)}=\sum _i\frac{dDIV}{d{Y}_j(T)}\frac{d{Y}_j(T)}{d{Z}_i^{(2)}(T)}$$

3. Compute ${\nabla }_{h_{(}T)}DIV$

   • $$W^{(2)}h(T)=Z^{(2)}(T)$$

   • $$\frac{dDIV}{d{h}_i(T)}=\sum _j\frac{dDIV}{d{Z}_j^{(2)}(T)}\frac{d{Z}_j^{(2)}(T)}{d{h}_i(T)}=\sum _j{w}_{ij}^{(2)}\frac{dDIV}{d{Z}_j^{(2)}(T)}$$

   • $${\nabla }_{h(T)}DIV={\nabla }_{Z^{(2)}(T)}DIV{W}^{(2)}$$

4. Compute ${\nabla }_{W^{(2)}}DIV$

   • $$\frac{dDIV}{d{w}_{ij}^{(2)}}=\frac{dDIV}{d{Z}_j^{(2)}(T)}{h}_i(T)$$

   • $${\nabla }_{W^{(2)}}DIV=h(T){\nabla }_{Z^{(2)}(T)}DIV$$

5. Compute ${\nabla }_{Z^{(1)}(T)}DIV$

   • $$\frac{dDIV}{d{Z}_i^{(1)}(T)}=\frac{dDIV}{d{h}_i(T)}\frac{d{h}_i(T)}{d{Z}_i^{(1)}(T)}$$

   • $${\nabla }_{Z^{(1)}(T)}DIV={\nabla }_{h(T)}DIV{\nabla }_{Z^{(1)}(T)}h(T)$$

6. Compute ${\nabla }_{W^{(1)}}DIV$

   • $$W^{(1)}X(T)+W^{(11)}h(T-1)=Z^{(1)}(T)$$

   • $$\frac{dDIV}{d{w}_{ij}^{(1)}}=\frac{dDIV}{d{Z}_j^{(1)}(T)}{X}_i(T)$$

   • $${\nabla }_{W^{(1)}}DIV=X(T){\nabla }_{Z^{(1)}(T)}DIV$$

7. Compute ${\nabla }_{W^{(11)}}DIV$

   • $$\frac{dDIV}{d{w}_{ii}^{(11)}}=\frac{dDIV}{d{Z}_i^{(1)}(T)}{h}_i(T-1)$$

   • $${\nabla }_W^{(11)}DIV=h(T-1){\nabla }_{Z^{(1)}(T)}DIV$$

Derivative at time $T-1$

1. Compute ${\nabla }_{Z^{(2)}(T-1)}DIV$

   • $${\nabla }_{Z^{(2)}(T-1)}DIV={\nabla }_{Y(T-1)}DIV{\nabla }_{Z^{(2)}(T-1)}Y(T-1)$$

   • For scalar output activation

     • $$\frac{dDIV}{d{Z}_i^{(2)}(T-1)}=\frac{dDIV}{d{Y}_i(T-1)}\frac{d{Y}_i(T-1)}{d{Z}_i^{(2)}(T-1)}$$

   • For vector output activation

     • $$\frac{dDIV}{d{Z}_i^{(2)}(T-1)}=\sum _j\frac{dDIV}{d{Y}_j(T-1)}\frac{d{Y}_j(T-1)}{d{Z}_i^{(2)}(T-1)}$$

2. Compute ${\nabla }_{h_{(}T-1)}DIV$

   • $$\frac{dDIV}{d{h}_i(T-1)}=\sum _j{w}_{ij}^{(2)}\frac{dDIV}{d{Z}_j^{(2)}(T-1)}+\sum _j{w}_{ij}^{(11)}\frac{dDIV}{d{Z}_j^{(1)}(T)}$$

   • $${\nabla }_{h(T-1)}DIV={\nabla }_{Z^{(2)}(T-1)}DIV{W}^{(2)}+{\nabla }_{Z^{(1)}(T)}DIV{W}^{(11)}$$

3. Compute ${\nabla }_{W^{(2)}}DIV$

   • $$\frac{dDIV}{d{w}_{ij}^{(2)}}+=\frac{dDIV}{d{Z}_j^{(2)}(T-1)}{h}_i(T-1)$$

   • $${\nabla }_{W^{(2)}}DIV+=h(T-1){\nabla }_{Z^{(2)}(T-1)}DIV$$

4. Compute ${\nabla }_{Z^{(1)}(T-1)}DIV$

   • $$\frac{dDIV}{d{Z}_i^{(1)}(T-1)}=\frac{dDIV}{d{h}_i(T-1)}\frac{d{h}_i(T-1)}{d{Z}_i^{(1)}(T-1)}$$

   • $${\nabla }_{Z^{(1)}(T-1)}DIV={\nabla }_{h(T-1)}DIV{\nabla }_{Z^{(1)}(T-1)}h(T-1)$$

5. Compute ${\nabla }_{W^{(1)}}DIV$

   • $$\frac{dDIV}{d{w}_{ij}^{(1)}}+=\frac{dDIV}{d{Z}_j^{(1)}(T-1)}{X}_i(T-1)$$

   • $${\nabla }_{W^{(1)}}DIV+=X(T-1){\nabla }_{Z^{(1)}(T-1)}DIV$$

6. Compute ${\nabla }_{W^{(11)}}DIV$

   • $$\frac{dDIV}{d{w}_{ij}^{(11)}}+=\frac{dDIV}{d{Z}_j^{(1)}(T-1)}{h}_i(T-2)$$

   • $${\nabla }_{W^{(11)}}DIV+=h(T-2){\nabla }_{Z^{(1)}(T-1)}DIV$$

Back Propagation Through Time

$$\frac{dDIV}{d{h}_i(-1)}=\sum _i{w}_{ij}^{(11)}\frac{dDIV}{d{Z}_j^{(1)}(0)}$$

$$\frac{dDIV}{d{h}_i^{(k)}(t)}=\sum _j{w}_{i,j}^{(k+1)}\frac{dDIV}{d{Z}_j^{(k+1)}(t)}+\sum _j{w}_{i,j}^{(k,k)}\frac{dDIV}{d{Z}_j^{(k)}(t+1)}$$

$$\frac{dDIV}{d{Z}_i^{(k)}(t)}=\frac{dDIV}{d{h}_i^{(k)}(t)}{f}_k^{\prime }\left(Z_i^{(k)}(t)\right)$$

$$\frac{dDIV}{d{w}_{ij}^{(1)}}=\sum _t\frac{dDIV}{d{Z}_j^{(1)}(t)}{X}_i(t)$$

$$\frac{dDIV}{d{w}_{ij}^{(11)}}=\sum _t\frac{dDIV}{d{Z}_j^{(1)}(t)}{h}_i(t-1)$$

Algorithm

Bidirectional RNN

• Two independent RNN
• Clearly, this is not an online process and requires the entire input data
• It is easy to learning two RNN independently
  • Forward pass: Compute both forward and backward networks and final output
  • Backpropagation
    • A basic backprop routine that we will call
    • Two calls to the routine within a higher-level wrapper