Modern Recurrent Neural Networks

Reference:

https://d2l.ai/chapter_recurrent-modern/index.html 9.1-9.4

Motivation

• Some tokens may not carry pertinent observation $\to$ Mechanism for skipping such tokens in the latent state representation
• There is a logical break between parts of a sequence $\to$ Means of resetting our internal state representation
• An early observation is highly significant for predicting all future observations $\to$ Mechanisms for storing vital early information in a memory cell

Gated Recurrent Units (GRU)

Cho, K., Van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014). On the properties of neural machine translation: encoder-decoder approaches. arXiv preprint arXiv:1409.1259.

GRU supports gating of the hidden state, which decides when a hidden state should be updated and also when it should be reset.

Reset Gate and Update Gate

• Reset gate:
  • controls how much of the previous state we might still want to remember
  • helps capture short-term dependencies in sequences
• Update gate:
  • controls how much of the new state is just a copy of the old state
  • helps capture long-term dependencies in sequences

Given the input of the current time step and the hidden state of the previous time step, the outputs of two gates are given by two fully-connected layers with a sigmoid activation function:

${\mathbf{X}}_t\in {\mathbb{R}}^{n\times d}$: a minibatch with batch size $n$ and $d$ inputs

${\mathbf{H}}_{t-1}\in {\mathbb{R}}^{n\times h}$: the hidden state of the previous time step ($h$=number of hidden units)

${\mathbf{R}}_t\in {\mathbb{R}}^{n\times h}$: the reset gate

${\mathbf{Z}}_t\in {\mathbb{R}}^{n\times h}$: the update gate

${\mathbf{W}}_{xr},{\mathbf{W}}_{xz}\in {\mathbb{R}}^{d\times h}$ and ${\mathbf{W}}_{hr},{\mathbf{W}}_{hz}\in {\mathbb{R}}^{h\times h}$: weight parameters

${\mathbf{b}}_r,{\mathbf{b}}_z\in {\mathbb{R}}^{1\times h}$: biases
$$\begin{gathered} {\mathbf{R}}_t=\sigma ({\mathbf{X}}_t{\mathbf{W}}_{xr}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{hr}+{\mathbf{b}}_r) \\ {\mathbf{Z}}_t=\sigma ({\mathbf{X}}_t{\mathbf{W}}_{xz}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{hz}+{\mathbf{b}}_z)\text{\hspace{1em}(GRU.1)} \end{gathered}$$

Hidden State

Next, let us integrate the reset gate ${\mathbf{R}}_t$ with the regular latent state updating mechanism. It leads to the following candidate hidden state ${\tilde{\mathbf{H}}}_t\in {\mathbb{R}}^{n\times h}$ at time step $t$:
$${\tilde{\mathbf{H}}}_t=\tanh ({\mathbf{X}}_t{\mathbf{W}}_{xh}+\left({\mathbf{R}}_t\odot {\mathbf{H}}_{t-1}\right){\mathbf{W}}_{hh}+{\mathbf{b}}_h)\text{\hspace{1em}(GRU.2)}$$
where ${\mathbf{W}}_{xh}\in {\mathbb{R}}^{d\times h}$ and ${\mathbf{W}}_{hh}\in {\mathbb{R}}^{h\times h}$ are weight parameters, ${\mathbf{b}}_h\in {\mathbb{R}}^{1\times h}$ is the bias, and the symbol $\odot$ is the Hadamard (elementwise) product operator.

Finally, we need to incorporate the effect of the update gate ${\mathbf{Z}}_t$. This determines the extent to which the new hidden state ${\mathbf{H}}_t\in {\mathbb{R}}^{n\times h}$ is just the old state ${\mathbf{H}}_{t-1}$ and by how much the new candidate state ${\tilde{\mathbf{H}}}_t$ is used:
$${\mathbf{H}}_t={\mathbf{Z}}_t\odot {\mathbf{H}}_{t-1}+(1-{\mathbf{Z}}_t)\odot {\tilde{\mathbf{H}}}_t\text{\hspace{1em}(GRU.3)}$$

Long Short-Term Memory (LSTM)

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–1780.

LSTM introduces a memory cell that has the same shape as the hidden state (some literatures consider the memory cell as a special type of the hidden state), engineered to record additional information.

Input Gate, Forget Gate, and Output Gate

To control the memory cell we need a number of gates.

• Input gate: to decide when to read data into the cell
• Output gate: to read out the entries from the cell
• Forget gate: to reset the content of the cell

The motivation for such a design is the same as that of GRUs, namely to be able to decide when to remember and when to ignore inputs in the hidden state via a dedicated mechanism.

Just like in GRUs, the data feeding into the LSTM gates are the input at the current time step and the hidden state of the previous time step. They are processed by three fully-connected layers with a sigmoid activation function to compute the values of the input, forget. and output gates. As a result, values of the three gates are in the range of $(0,1)$.

${\mathbf{I}}_t\in {\mathbb{R}}^{n\times h}$: the input gate

${\mathbf{O}}_t\in {\mathbb{R}}^{n\times h}$: the output gate

${\mathbf{F}}_t\in {\mathbb{R}}^{n\times h}$: the forget gate

${\mathbf{W}}_{xi},{\mathbf{W}}_{xo},{\mathbf{W}}_{xf}\in {\mathbb{R}}^{d\times h}$ and ${\mathbf{W}}_{hi},{\mathbf{W}}_{ho},{\mathbf{W}}_{hf}\in {\mathbb{R}}^{h\times h}$: weight parameters

${\mathbf{b}}_i,{\mathbf{b}}_o,{\mathbf{b}}_f\in {\mathbb{R}}^{1\times h}$: biases
$$\begin{array}{rl}{\mathbf{I}}_t& =\sigma ({\mathbf{X}}_t{\mathbf{W}}_{xi}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{hi}+{\mathbf{b}}_i)\\ {\mathbf{O}}_t& =\sigma ({\mathbf{X}}_t{\mathbf{W}}_{xo}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{ho}+{\mathbf{b}}_o)\\ {\mathbf{F}}_t& =\sigma ({\mathbf{X}}_t{\mathbf{W}}_{xf}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{hf}+{\mathbf{b}}_f)\end{array}\text{\hspace{1em}(LSTM.1)}$$

Memory Cell

Next we design the memory cell. Since we have not specified the action of the various gates yet, we first introduce the candidate memory cell ${\tilde{\mathbf{C}}}_t\in {\mathbb{R}}^{n\times h}$. Its computation is similar to that of the three gates described above, but using a $\tanh$ function with a value range for $(-1,1)$ as the activation function:
$${\tilde{\mathbf{C}}}_t=\tanh ({\mathbf{X}}_t{\mathbf{W}}_{xc}+{\mathbf{H}}_{t-1}{\mathbf{W}}_{hc}+{\mathbf{b}}_c)\text{\hspace{1em}(LSTM.2)}$$
where ${\mathbf{W}}_{xc}\in {\mathbb{R}}^{d\times h}$ and ${\mathbf{W}}_{hc}\in {\mathbb{R}}^{h\times h}$ are the weight parameters and ${\mathbf{b}}_c\in {\mathbb{R}}^{1\times h}$ is a bias parameter.

In GRUs, we have a mechanism to govern input and forgetting (or skipping). Similarly, in LSTMs we have two dedicated gates for such purposes: the input gate ${\mathbf{I}}_t$ governs how much we take new data into account via ${\tilde{\mathbf{C}}}_t$ and the forget gate ${\mathbf{F}}_t$ addresses how much of the old memory cell content ${\mathbf{C}}_{t-1}\in {\mathbb{R}}^{n\times h}$ we retain:
$${\mathbf{C}}_t={\mathbf{F}}_t\odot {\mathbf{C}}_{t-1}+{\mathbf{I}}_t\odot {\tilde{\mathbf{C}}}_t\text{\hspace{1em}(LSTM.3)}$$
This design is introduced to alleviate the vanishing gradient problem and to better capture long range dependencies within sequences.

Hidden State

Last, we need to define how to compute the hidden state ${\mathbf{H}}_t\in {\mathbb{R}}^{n\times h}$. This is where the output gate comes into play. In LSTM it is simply a gated version of the $\tanh$ of the memory cell. This ensures that the values of ${\mathbf{H}}_t$ are always in the interval $(-1,1)$:
$${\mathbf{H}}_t={\mathbf{O}}_t\odot \tanh ({\mathbf{C}}_t)\text{\hspace{1em}(LSTM.4)}$$
Whenever the output gate approximates 1 we effectively pass all memory information through to the predictor, whereas for the output gate close to 0 we retain all the information only within the memory cell and perform no further processing.

Deep Recurrent Neural Networks

We could stack multiple layers of RNNs on top of each other. Each hidden state is continuously passed to both the next time step of the current layer and the current time step of the next layer.

${\mathbf{X}}_t\in {\mathbb{R}}^{n\times d}$: a minibatch with batch size $n$ and $d$ inputs

${\mathbf{H}}_t^{(l)}\in {\mathbb{R}}^{n\times h}$: the hidden state of the $l^{th}$ hidden layer ($h$=number of hidden units), ${\mathbf{H}}_t^{(0)}={\mathbf{X}}_t$.
$${\mathbf{H}}_t^{(l)}={\varphi }_l({\mathbf{H}}_t^{(l-1)}{\mathbf{W}}_{xh}^{(l)}+{\mathbf{H}}_{t-1}^{(l)}{\mathbf{W}}_{hh}^{(l)}+{\mathbf{b}}_h^{(l)})\text{\hspace{1em}(DRNN.1)}$$
In the end, the calculation of the output layer is only based on the hidden state of the final $L^{th}$ hidden layer:
$${\mathbf{O}}_t={\mathbf{H}}_t^{(L)}{\mathbf{W}}_{hq}+{\mathbf{b}}_q\text{\hspace{1em}(DRNN.2)}$$

Bidirectional Recurrent Neural Networks

Schuster, M., & Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11), 2673–2681.

To motivate why one might pick this specific architecture, we can first take a detour to probabilistic models.

Dynamic Programming in Hidden Markov Models

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-0rEvAioR-1630835800664)(https://d2l.ai/_images/hmm.svg)]

For a sequence of $T$ observations we have the following joint probability distribution over the observed and hidden states:
$$P(x_1,\cdots ,x_T,h_1,\cdots ,h_T)=\prod _{t=1}^{T}P(h_t|h_{t-1})P(x_t|h_t)\text{\hspace{1em}(HMM.1)}$$
where $P(h_1|h_0)=P(h_1)$.

Now assume that we observe all $x_i$ with the exception of some $x_j$ and it is our goal to compute $P(x_j|x_{-j})$, where $x_{-j}=(x_1,\cdots ,x_{j-1},x_j,\cdots ,x_T)$.

Since there is no latent variable in $P(x_j|x_{-j})$, we consider summing over all the possible combinations of choices for $h_1,\cdots ,h_T$. In case any $h_i$ can take on $k$ distinct values, this means that we need to sum over $k^T$ terms, which is usually impossible! Fortunately there is an elegant solution for this: dynamic programming.

• Forward recursion
  $$\begin{array}{rl}P(x_1,\dots ,x_T)=& \sum _{h_1,\dots ,h_T}P(x_1,\dots ,x_T,h_1,\dots ,h_T)\\ =& \sum _{h_1,\dots ,h_T}\prod _{t=1}^{T}P(h_t\mid h_{t-1})P(x_t\mid h_t)\\ =& \sum _{h_2,\dots ,h_T}\underset{{\pi }_2(h_2)\stackrel{\mathrm{d}\mathrm{e}\mathrm{f}}{=}}{\underbrace{\left[\sum _{h_1}P(h_1)P(x_1\mid h_1)P(h_2\mid h_1)\right]}}P(x_2\mid h_2)\prod _{t=3}^{T}P(h_t\mid h_{t-1})P(x_t\mid h_t)\\ =& \sum _{h_3,\dots ,h_T}\underset{{\pi }_3(h_3)\stackrel{\mathrm{d}\mathrm{e}\mathrm{f}}{=}}{\underbrace{\left[\sum _{h_2}{\pi }_2(h_2)P(x_2\mid h_2)P(h_3\mid h_2)\right]}}P(x_3\mid h_3)\prod _{t=4}^{T}P(h_t\mid h_{t-1})P(x_t\mid h_t)\\ =& \dots \\ =& \sum _{h_T}{\pi }_T(h_T)P(x_T\mid h_T).\end{array}\text{\hspace{1em}(HMM.2)}$$
  In general we have the forward recursion as
  $${\pi }_{t+1}(h_{t+1})=\sum _{h_t}{\pi }_t(h_t)P(x_t\mid h_t)P(h_{t+1}\mid h_t)\text{\hspace{1em}(HMM.3)}$$
  The recursion is initialized as ${\pi }_1(h_1)=P(h_1)$. In abstract terms this can be written as ${\pi }_{t+1}=f({\pi }_t,x_t)$, where $f$ is some learnable function. This looks very much like the update equation in the latent variable models we discussed so far in the context of RNNs!

• Backward recursion
  $$\begin{array}{rl}P(x_1,\dots ,x_T)=& \sum _{h_1,\dots ,h_T}P(x_1,\dots ,x_T,h_1,\dots ,h_T)\\ =& \sum _{h_1,\dots ,h_T}\prod _{t=1}^{T-1}P(h_t\mid h_{t-1})P(x_t\mid h_t)\cdot P(h_T\mid h_{T-1})P(x_T\mid h_T)\\ =& \sum _{h_1,\dots ,h_{T-1}}\prod _{t=1}^{T-1}P(h_t\mid h_{t-1})P(x_t\mid h_t)\cdot \underset{{\rho }_{T-1}(h_{T-1})\stackrel{\mathrm{d}\mathrm{e}\mathrm{f}}{=}}{\underbrace{\left[\sum _{h_T}P(h_T\mid h_{T-1})P(x_T\mid h_T)\right]}}\\ =& \sum _{h_1,\dots ,h_{T-2}}\prod _{t=1}^{T-2}P(h_t\mid h_{t-1})P(x_t\mid h_t)\cdot \underset{{\rho }_{T-2}(h_{T-2})\stackrel{\mathrm{d}\mathrm{e}\mathrm{f}}{=}}{\underbrace{\left[\sum _{h_{T-1}}P(h_{T-1}\mid h_{T-2})P(x_{T-1}\mid h_{T-1}){\rho }_{T-1}(h_{T-1})\right]}}\\ =& \dots \\ =& \sum _{h_1}P(h_1)P(x_1\mid h_1){\rho }_1(h_1)\end{array}\text{\hspace{1em}(HMM.4)}$$
  We can thus write the backward recursion as
  $${\rho }_{t-1}(h_{t-1})=\sum _{h_t}P(h_t\mid h_{t-1})P(x_t\mid h_t){\rho }_t(h_t)\text{\hspace{1em}(HMM.5)}$$
  with initialization ${\rho }_T(h_T)=1$. Note that in abstract terms the backward recursion can be written as ${\rho }_{t-1}=g({\rho }_t,x_t)$, where $g$ is a learnable function. Again, this looks very much like an update equation, just running backwards unlike what we have seen so far in RNNs.

• Forward and backward recursion
  $$\begin{array}{rl}& P(x_1,\dots ,x_T)\\ =& \sum _{h_1,\dots ,h_T}\prod _{t=1}^{T}P(h_t\mid h_{t-1})P(x_t\mid h_t)\\ =& \sum _{h_j}\left[\left(\sum _{h_1,\dots ,h_{j-1}}\left[\prod _{t=1}^{T}P(h_t\mid h_{t-1})P(x_t\mid h_t)\right]P(h_j\mid h_{j-1})\right)P(x_j\mid h_j)\left(\sum _{h_{j+1},\dots ,h_T}\left[\prod _{t=1}^{T}P(h_t\mid h_{t-1})P(x_t\mid h_t)\right]\right)\right]\\ =& \sum _{h_j}{\pi }_j(h_j)P(x_j\mid h_j)\rho (h_j)\end{array}\text{\hspace{1em}(HMM.6)}$$

From the results above, we are able to compute
$$P(x_j|x_{-j})\propto \sum _{h_j}{\pi }_j(h_j)P(x_j\mid h_j)\rho (h_j)\text{\hspace{1em}(HMM.7)}$$
These recursions allow us to sum over $T$ latent variables in $\mathcal{O}(kT)$ (linear) time over all values of $(h_1,\cdots ,h_T)$ rather than in exponential time.

Bidirectional Model

If we want to have a mechanism in RNNs that offers comparable look-ahead ability as in hidden Markov models, we need to modify the RNN design that we have seen so far. Instead of running an RNN only in the forward mode starting from the first token, we start another one from the last token running from back to front. Bidirectional RNNs add a hidden layer that passes information in a backward direction to more flexibly process such information.

In fact, this is not too dissimilar to the forward and backward recursions in the dynamic programming of hidden Markov models.

${\mathbf{X}}_t\in {\mathbb{R}}^{n\times d}$: a minibatch with batch size $n$ and $d$ inputs

${\overrightarrow{\mathbf{H}}}_t\in {\mathbb{R}}^{n\times h},{\overleftarrow{\mathbf{H}}}_t\in {\mathbb{R}}^{n\times h}$: the forward and backward hidden states for this time step

${\mathbf{W}}_{xh}^{(f)}\in {\mathbb{R}}^{d\times h},{\mathbf{W}}_{hh}^{(f)}\in {\mathbb{R}}^{h\times h},{\mathbf{W}}_{xh}^{(b)}\in {\mathbb{R}}^{d\times h},{\mathbf{W}}_{hh}^{(b)}\in {\mathbb{R}}^{h\times h}$: weights

${\mathbf{b}}_h^{(f)}\in {\mathbb{R}}^{1\times h},{\mathbf{b}}_h^{(b)}\in {\mathbb{R}}^{1\times h}$: biases
$$\begin{array}{rl}{\overrightarrow{\mathbf{H}}}_t& =\varphi ({\mathbf{X}}_t{\mathbf{W}}_{xh}^{(f)}+{\overrightarrow{\mathbf{H}}}_{t-1}{\mathbf{W}}_{hh}^{(f)}+{\mathbf{b}}_h^{(f)}),\\ {\overleftarrow{\mathbf{H}}}_t& =\varphi ({\mathbf{X}}_t{\mathbf{W}}_{xh}^{(b)}+{\overleftarrow{\mathbf{H}}}_{t+1}{\mathbf{W}}_{hh}^{(b)}+{\mathbf{b}}_h^{(b)}),\end{array}\text{\hspace{1em}(BM.1)}$$
Last, the output layer computes the output ${\mathbf{O}}_t\in {\mathbb{R}}^{n\times q}$ (number of outputs: $q$)
$${\mathbf{O}}_t={\mathbf{H}}_t{\mathbf{W}}_{hq}+{\mathbf{b}}_q\text{\hspace{1em}(BM.2)}$$
Here, the weight matrix ${\mathbf{W}}_{hq}\in {\mathbb{R}}^{2h\times q}$ and the bias ${\mathbf{b}}_q\in {\mathbb{R}}^{1\times q}$ are the model parameters of the output layer. In fact, the two directions can have different numbers of hidden units.