Recurrent Neural Networks

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/phoenix198425/article/details/79955663

Examples of Sequence Data

  • Speech Recognition
  • Music Generation
  • Sentiment Classification
  • DNA Sequence Analysis
  • Machine Translation
  • Video Activity Recognition
  • Name Entity Recognition

Notation

Symbol Meaning
X(i)<t> The t th element in the input sequence for training example i
Y(i)<t> The t th element in the output sequence for training example i
TX(i) Input sequence length for training example i
Ty(i) Output sequence length for training example i

Recurrent Neural Network Model

Why not standard network?

  1. Inputs, outputs can be different lengths in different examples.
  2. Doesn’t share features across different features of text.

RNN Unit

a<t>=g(Waaa<t1>+Waxx<t>+ba)
y^<t>=g(Wyaa<t>+by)
Let Wa=(WaaWax),[a<t1>,x<t>]=(a<t1>x<t>),Wy=Wya, then
a<t>=g(Wa[a<t1>,x<t>]+ba)
y^<t>=g(Wya<t>+by)

Forward Propagation

Different Types of RNNs

Type Example
Many-to-many, Tx=Ty Name entity recognition
Many-to-one Sentiment classification
One-to-one
One-to-many Music generation
Many-to-many, TxTy Machine translation

1. Many-to-many, Tx=Ty

2. Many-to-one

3. One-to-one

4. One-to-many

5. Many-to-many, TxTy

Gated Recurrent Unit (GRU)


c~<t>=tanh(Wc[Γrc<t1>,x<t>]+bc)
Update Gate: Γu=σ(Wu[c<t1>,x<t>]+bu)
Relevant Gate: Γr=σ(Wr[c<t1>,x<t>]+br)
Memory cell value: c<t>=Γuc~<t>+(1Γu)c<t1>
a<t>=c<t>

Long Short Term Memory (LSTM)


c~<t>=tanh(Wc[a<t1>,x<t>]+bc)
Update Gate: Γu=σ(Wu[a<t1>,x<t>]+bu)
Forget Gate: Γf=σ(Wf[a<t1>,x<t>]+bf)
Output Gate: Γo=σ(Wo[a<t1>,x<t>]+bo)
Memory Cell: c<t>=Γuc~<t>+Γfc<t1>
a<t>=Γotanhc<t>

阅读更多

没有更多推荐了,返回首页