keras attention code

原创 2018年04月17日 09:33:09
#lstm+attention
inputs = Input(shape=(TIME_STEPS, INPUT_DIM,))
lstm_units = 32
lstm_out = LSTM(lstm_units, return_sequences=True)(inputs)

a = Permute((2, 1))(lstm_out)
a = Reshape((input_dim, TIME_STEPS))(a) # this line is not useful. It's just to know which dimension is what.
a = Dense(TIME_STEPS, activation='softmax')(a)
if SINGLE_ATTENTION_VECTOR:
    a = Lambda(lambda x: K.mean(x, axis=1), name='dim_reduction')(a)
    a = RepeatVector(input_dim)(a)
a_probs = Permute((2, 1), name='attention_vec')(a)
output_attention_mul = merge([inputs, a_probs], name='attention_mul', mode='mul')

attention_mul = Flatten()(output_attention_mul)
output = Dense(1, activation='sigmoid')(attention_mul)
model = Model(input=[inputs], output=output)
#######################################

#attention+lstm
inputs = Input(shape=(TIME_STEPS, INPUT_DIM,))

a = Permute((2, 1))(inputs)
a = Reshape((input_dim, TIME_STEPS))(a) # this line is not useful. It's just to know which dimension is what.
a = Dense(TIME_STEPS, activation='softmax')(a)
if SINGLE_ATTENTION_VECTOR:
    a = Lambda(lambda x: K.mean(x, axis=1), name='dim_reduction')(a)
    a = RepeatVector(input_dim)(a)
a_probs = Permute((2, 1), name='attention_vec')(a)
output_attention_mul = merge([inputs, a_probs], name='attention_mul', mode='mul')

attention_mul = LSTM(lstm_units, return_sequences=False)(output_attention_mul)
output = Dense(1, activation='sigmoid')(attention_mul)
model = Model(input=[inputs], output=output)
#########################################

keras系列︱seq2seq系列相关实现与案例(feedback、peek、attention类型)

之前在看《Semi-supervised Sequence Learning》这篇文章的时候对seq2seq半监督的方式做文本分类的方式产生了一定兴趣,于是开始简单研究了seq2seq。先来简单说一下...
  • sinat_26917383
  • sinat_26917383
  • 2017-07-13 19:36:26
  • 6620

seq2seq 的 keras 实现

上一篇 seq2seq 入门 提到了 cho 和 Sutskever 的两篇论文,今天来看一下如何用 keras 建立 seq2seq。 第一个 LSTM 为 Encoder,只在序列结束时输出一个...
  • aliceyangxi1987
  • aliceyangxi1987
  • 2017-06-18 12:13:40
  • 4175

记忆网络之Hierarchical Memory Networks(架构分层)

记忆网络之Hierarchical Memory Networks(架构分层)这是中科院去年发表的一篇论文“Hierarchical Memory Networks for Answer Select...
  • liuchonge
  • liuchonge
  • 2017-10-18 20:03:29
  • 888

知乎--LSTM(挺全的)

首页发现话题 登录 加入知乎 有哪些LSTM(Long Short Term Memory)和RNN(Recu...
  • vbskj
  • vbskj
  • 2017-05-12 11:16:47
  • 7052

keras for attention

keras还没有官方实现attention机制,有些attention的个人实现,在mnist数据集上做了下实验。模型是双向lstm+attention+dropout,话说双向lstm本身就很强大了...
  • u010041824
  • u010041824
  • 2017-12-20 17:44:42
  • 859

attention 机制入门

在下面这两篇文章中都有提到 attention 机制: 使聊天机器人的对话更有营养 如何自动生成文章摘要今天来看看 attention 是什么。下面这篇论文算是在NLP中第一个使用attenti...
  • aliceyangxi1987
  • aliceyangxi1987
  • 2017-07-28 22:53:31
  • 3049

Bi-Directional Attention Flow For Machine Comprehension

关键词bi-directional attention来源arXiv 2016.11.05问题利用 multi-stage 信息对文章进行编码,同时尝试两个方向上的 attention 来提高 RC ...
  • u014300008
  • u014300008
  • 2016-11-15 08:48:04
  • 1715

CVPR 2015 papers

From:   http://www.pamitc.org/cvpr15/program.php?from=timeline&isappinstalled=0 Monday, Ju...
  • xuyuhua1985
  • xuyuhua1985
  • 2015-06-27 16:30:51
  • 24139

深度学习笔记——Attention Model(注意力模型)学习总结

Attention Model(注意力模型)学习总结,包括soft Attention Model,Global Attention Model和Local Attention Model,静态AM,...
  • mpk_no1
  • mpk_no1
  • 2017-08-06 21:49:46
  • 20890

tensorflow 学习笔记: recurrent models of visual attention

tf.truncated_normal: tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None,...
  • alwaystry
  • alwaystry
  • 2017-03-17 15:54:18
  • 873
收藏助手
不良信息举报
您举报文章:keras attention code
举报原因:
原因补充:

(最多只允许输入30个字)