Recurrent Neural Networks

Examples of Sequence Data

• Speech Recognition
• Music Generation
• Sentiment Classification
• DNA Sequence Analysis
• Machine Translation
• Video Activity Recognition
• Name Entity Recognition

Notation

Symbol	Meaning
$X ^{(i) <t>}$	The $t$ th element in the input sequence for training example $i$
$Y ^{(i) <t>}$	The $t$ th element in the output sequence for training example $i$
$T ^{(i)} _{X}$	Input sequence length for training example $i$
$T ^{(i)} _{y}$	Output sequence length for training example $i$

Recurrent Neural Network Model

Why not standard network?

1. Inputs, outputs can be different lengths in different examples.
2. Doesn’t share features across different features of text.

RNN Unit

$a ^{<t>} = g \left ( W _{a a} a ^{<t - 1>} + W _{a x} x ^{<t>} + b _{a} \right )$
$\hat y ^{<t>} = g \left (W _{y a} a ^{<t >} + b _{y} \right )$
Let $W _{a} = \begin{pmatrix} W _{a a} W _{a x} \end{pmatrix}, \begin{bmatrix} a ^{<t - 1>}, x ^{<t>} \end{bmatrix} = \begin{pmatrix} a ^{<t - 1>} \\ x ^{<t>} \end{pmatrix} , W _{y} = W _{y a},$ then
$a ^{<t>} = g \left ( W _{a} \begin{bmatrix} a ^{<t - 1>}, x ^{<t>} \end{bmatrix} + b _{a} \right )$
$\hat y ^{<t>} = g \left (W _{y} a ^{<t >} + b _{y} \right )$

Forward Propagation

Different Types of RNNs

Type	Example
Many-to-many, $T_{x} = T_{y}$	Name entity recognition
Many-to-one	Sentiment classification
One-to-one
One-to-many	Music generation
Many-to-many, $T_{x} \neq T_{y}$	Machine translation

1. Many-to-many, $T_{x} = T_{y}$

2. Many-to-one

3. One-to-one

4. One-to-many

5. Many-to-many, $T_{x} \neq T_{y}$

Gated Recurrent Unit (GRU)

$\tilde c ^{<t>} = \tanh \left ( W _{c} \left [ \Gamma _{r} * c ^{<t - 1>}, x ^{<t>} \right ] + b _{c} \right )$
Update Gate: $\Gamma _{u} = \sigma \left ( W _{u} \left [ c ^{<t - 1>}, x ^{<t>} \right ] + b _{u} \right )$
Relevant Gate: $\Gamma _{r} = \sigma \left ( W _{r} \left [ c ^{<t - 1>}, x ^{<t>} \right ] + b _{r} \right )$
Memory cell value: $c ^{<t>} = \Gamma _{u} * \tilde c ^{<t>} + \left ( 1 - \Gamma _{u} \right ) * c ^{<t - 1>}$
$a ^{<t>} = c ^{<t>}$

Long Short Term Memory (LSTM)

$\tilde c ^{<t>} = \tanh \left ( W _{c} \left [ a ^{<t - 1>}, x ^{<t>} \right ] + b _{c} \right )$
Update Gate: $\Gamma _{u} = \sigma \left ( W _{u} \left [ a ^{<t - 1>}, x ^{<t>} \right ] + b _{u} \right )$
Forget Gate: $\Gamma _{f} = \sigma \left ( W _{f} \left [ a ^{<t - 1>}, x ^{<t>} \right ] + b _{f} \right )$
Output Gate: $\Gamma _{o} = \sigma \left ( W _{o} \left [ a ^{<t - 1>}, x ^{<t>} \right ] + b _{o} \right )$
Memory Cell: $c ^{<t>} = \Gamma _{u} * \tilde c ^{<t>} + \Gamma _{f} * c ^{<t - 1>}$
$a ^{<t>} = \Gamma _{o} * \tanh c ^{<t>}$