Recurrent Neural Networks

原创 2018年04月16日 06:28:37

Examples of Sequence Data

  • Speech Recognition
  • Music Generation
  • Sentiment Classification
  • DNA Sequence Analysis
  • Machine Translation
  • Video Activity Recognition
  • Name Entity Recognition

Notation

Symbol Meaning
X(i)<t> The t th element in the input sequence for training example i
Y(i)<t> The t th element in the output sequence for training example i
TX(i) Input sequence length for training example i
Ty(i) Output sequence length for training example i

Recurrent Neural Network Model

Why not standard network?

  1. Inputs, outputs can be different lengths in different examples.
  2. Doesn’t share features across different features of text.

RNN Unit

a<t>=g(Waaa<t1>+Waxx<t>+ba)
y^<t>=g(Wyaa<t>+by)
Let Wa=(WaaWax),[a<t1>,x<t>]=(a<t1>x<t>),Wy=Wya, then
a<t>=g(Wa[a<t1>,x<t>]+ba)
y^<t>=g(Wya<t>+by)

Forward Propagation

Different Types of RNNs

Type Example
Many-to-many, Tx=Ty Name entity recognition
Many-to-one Sentiment classification
One-to-one
One-to-many Music generation
Many-to-many, TxTy Machine translation

1. Many-to-many, Tx=Ty

2. Many-to-one

3. One-to-one

4. One-to-many

5. Many-to-many, TxTy

Gated Recurrent Unit (GRU)


c~<t>=tanh(Wc[Γrc<t1>,x<t>]+bc)
Update Gate: Γu=σ(Wu[c<t1>,x<t>]+bu)
Relevant Gate: Γr=σ(Wr[c<t1>,x<t>]+br)
Memory cell value: c<t>=Γuc~<t>+(1Γu)c<t1>
a<t>=c<t>

Long Short Term Memory (LSTM)


c~<t>=tanh(Wc[a<t1>,x<t>]+bc)
Update Gate: Γu=σ(Wu[a<t1>,x<t>]+bu)
Forget Gate: Γf=σ(Wf[a<t1>,x<t>]+bf)
Output Gate: Γo=σ(Wo[a<t1>,x<t>]+bo)
Memory Cell: c<t>=Γuc~<t>+Γfc<t1>
a<t>=Γotanhc<t>

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/phoenix198425/article/details/79955663

ACL 2016 Accepted Papers 会议论文

ACL 2016 Accepted Papers Long papers Short papers System demonstrations   Long Papers...
  • GarfieldEr007
  • GarfieldEr007
  • 2016-06-15 19:42:22
  • 7088

Recurrent Neural Network 学习之路

Recurrent Neural Network 学习之路 1. Read paper: "A critical review of Recurrent Neural Networks for Seq...
  • yangyangyang20092010
  • yangyangyang20092010
  • 2015-12-21 20:33:28
  • 3316

论文阅读:Pixel Recurrent Neural Networks

Introduction   PixelRNN和PixelCNN都没有做独立性假设。   通过一层256维的softmax层来将像素值当做离散分布来建模。 Math Model gener...
  • qq184861643
  • qq184861643
  • 2018-01-10 11:16:26
  • 84

【每周一文】Supervised Sequence Labelling with Recurrent Neural Networks

概述 序列化标注是NLP领域非常常见的问题,很多问题都可以归结为序列化标注问题,例如分类可以看做多对一的标注;词性标注属于一对一的标注;机器翻译属于多对多的标注。 深度学习在NLP上取得...
  • fangqingan_java
  • fangqingan_java
  • 2016-02-16 23:47:55
  • 2462

Recurrent Neural Networks for Prediction(pdf)

  • 2008年10月15日 14:16
  • 5.61MB
  • 下载

《session-based recommendations with recurrent neural networks》ICLR 2016 阅读笔记

文章简介: 该文提出了利用RNN建模一个session间用户的点击序列。该方法完全利用用户在当前session里的反馈去做推荐,相比原依赖用户历史记录的推荐能在解决冷启动问题上更为简洁有效。 模型 文...
  • studyless
  • studyless
  • 2017-03-03 22:42:59
  • 899

论文笔记:session-based recommendations with recurrent neural networks

占坑中。。。。。。。。。。。。。。作者尝试了 常见的 point-wise rank loss,即认为负样本为 0, 正样本为 1 的loss function,发现训练出来的模型并不稳定,原因可能因...
  • chris_xy
  • chris_xy
  • 2017-09-24 21:41:34
  • 399

【Learning Notes】Quasi-recurrent Neural Networks

Quasi-recurrent Neural Networks (QRNN)
  • JackyTintin
  • JackyTintin
  • 2017-09-12 20:02:14
  • 1262

论文《Recurrent Convolutional Neural Networks for Text Classification》总结

《Recurrent Convolutional Neural Networks for Text Classification》论文来源:Lai, S., Xu, L., Liu, K., & Zh...
  • rxt2012kc
  • rxt2012kc
  • 2017-06-26 22:05:13
  • 2412

Conditional Random Fields as Recurrent Neural Networks

牛津,斯坦福,IDL的一篇论文介绍:1.传统的用于Object Recognition的CNN很难转换为用于segmentation的 pixel-wise predicting:1)感受野过大以及p...
  • meanme
  • meanme
  • 2016-03-09 21:42:46
  • 2413
收藏助手
不良信息举报
您举报文章:Recurrent Neural Networks
举报原因:
原因补充:

(最多只允许输入30个字)