RNN, LSTM 图文详解

往期文章链接目录

Sequence Data

There are many sequence data in applications. Here are some examples

• Machine translation

  • from text sequence to text sequence.

• Text Summarization

  • from text sequence to text sequence.

• Sentiment classification

  • from text sequence to categories.

• Music Generation

  • from nothing or some simple stuff (character, integer, etc) to wave sequence.

• Name entity recognition (NER)

  • From text sequence to label sequence.

Why not use a standard neural network for sequence tasks

• Inputs, outputs can be different lengths in different examples. This can be solved by standard neural network by paddings with the maximum lengths but it’s not a good solution since there would be too many parameters.

• Doesn’t share features learned across different positions of text/sequence. Note that Convolutional Neural Network (CNN) is a good example of parameter sharing, so we should have a similar model for sequence data.

RNN

A Recurrent Neural Network (RNN) can be thought of as multiple copies of the same network, each passing a message to a successor. This chain-like nature reveals that recurrent neural networks are intimately related to sequences. Therefore, they’re the natural architecture of neural network to use for sequence data. Note that it also allows previous outputs to be used as inputs.

For each time step $t$, the activation $a^{<t>}$ and the output $y^{<t>}$ are expressed as follows:

• $a^{<t>}=g_1\left(W_{aa}{a}^{<t-1>}+W_{ax}{x}^{<t>}+b_a\right)$

• $y^{<t>}=g_2\left(W_{ya}{a}^{<t>}+b_y\right)$

These calculation could be visualized in the following figure

Note:

• dimension of $W_{aa}$: (number of hidden neurons, number of hidden neurons)

• dimension of $W_{ax}$: (number of hidden neurons, length of $x$)

• dimension of $W_{ya}$: (length of $y$, number of hidden neurons)

• The weight matrix $W_{aa}$ is the memory the RNN is trying to maintain from the previous layers.

• dimension of $b_a$: (number of hidden neurons, 1)

• dimension of $b_y$: (length of y