Introduction to Recurrent Neural Networks

What is RNN

The networks are recurrent because they performance same computations for all the elements of a sequence of input, and the output of each element dependents, in addition to current input, from all the previous commutations.

Why RNN

• Sequential type information of the inputs
  Video Analysis
  Speech Recognition
  Machine Translation
• RNN have proved to have excellent performance in such problems

RNN Procedure

Sigmoid Gradient

The Vanish Gradient Problem

Consider the recurrent networks:

$$h_t = \sigma(Ux_t+Vh_{t-1})$$
then,
$$h_3 = \sigma(Ux_3+V(\sigma(Ux_2+V(\sigma(Ux_1)))))$$
$$\frac{\partial E_3}{\partial U}=\frac{\partial E_3}{\partial out_3}\frac{\partial out_3}{\partial h_3}\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial h_1}\frac{\partial h_1}{\partial U}$$

LSTM Cell

• Input Gate
  

  $$g = tanh(b^g+x_tU^g+h_{t-1}V^g)$$
  $$i=\sigma(b^i+x_tU^i+h_{t-1}V^i)$$
  $$out_i=g\circ i$$

• forget gate
  

  $$f = \sigma(b^f+x_tU^f+h_{t-1}V^f)$$
  $$s_t = s_{t-1}\circ f + g\circ i$$

• output gate
  

  $$o = \sigma(b^o+x_tU^o+h_{t-1}V^o)$$
  $$h_t = tanh(s_t)\circ o$$

Reducing The Problem

$$\frac{\partial s_t}{\partial s_{t-1}} = f$$

Reference

• http://adventuresinmachinelearning.com/recurrent-neural-networks-lstm-tutorial-tensorflow/
• Deep Learning with Tensorflow