tf.nn.bidirectional_dynamic_rnn
tf.nn.bidirectional_dynamic_rnn(
cell_fw,
cell_bw,
inputs,
sequence_length=None,
initial_state_fw=None,
initial_state_bw=None,
dtype=None,
parallel_iterations=None,
swap_memory=False,
time_major=False,
scope=None
)
输入
- cell_fw:一个RNNCell实例,用于前向。
- cell_bw: 一个RNNCell实例,用于后向。
- Inputs:RNN的输入。一般的shape为[batch_size,seq_len,embedding_size]
- sequence_length:可选参数。一个大小为[batch_size]的int32/int64类型向量。表示每个输入样本长度,如时间步长,dynamic可以处理不定长的,但我做的实验意义不大。
- Initial_state:_fw:可选参数。一个针对前向RNN的初始状态。[batch_size, cell_fw.state_size]
- Initial_state_bw: 可选参数。一个针对后向RNN的初始状态。[batch_size, cell_fw.state_size]
输出
一个(outputs,output_states)元组,其中:
- outputs 是一个一个(output_fw,output_bw)元组,里面包括前向和后向的结果,每一个结果都包含所有的时刻的输出结果 shape为[batch_size,num_step,num_hidden]
- output_states 一个(output_state_fw,output_state_bw)的元组,每一个结果里面的都包含两个状态,只包含最后一个时刻(前后、后项的最后一个)的两个输出,所以没有n_step 包含两个结果 ,一个是 C(最后一个时刻后 LSTM里面最上面的通道,通常用作下一个时刻的输入),H(最后一个时刻的输出,可以从下面的代码验证中得到验证)
加Attention机制
思想: 利用最后一层的输出(因为最后一层融合了所有的信息) 跟每一层进行相乘得出attention权重,然后利用权重把每一步的结果都加起来得到最终的一个结果
代码验证
import tensorflow as tf
import numpy as np
tf.reset_default_graph()
# Bi-LSTM(Attention) Parameters
embedding_dim = 2
n_hidden = 5 # number of hidden units in one cell
# 3 words sentences (=sequence_length is 3)
sentences = ["i love you", "he loves me", "she likes baseball", "i hate you", "sorry for that", "this is awful"]
word_list = " ".join(sentences).split()
word_list = list(set(word_list))
word_dict = {w: i for i, w in enumerate(word_list)}
vocab_size = len(word_dict)
input_batch = []
for sen in sentences:
input_batch.append(np.asarray([word_dict[n] for n in sen.split()]))
target_batch = []
for out in labels:
target_batch.append(np.eye(n_class)[out]) # ONE-HOT : To using Tensor Softmax Loss function
# LSTM Model
X = tf.placeholder(tf.int32, [None, n_step])
Y = tf.placeholder(tf.int32, [None, n_class])
out = tf.Variable(tf.random_normal([n_hidden * 2, n_class]))
embedding = tf.Variable(tf.random_uniform([vocab_size, embedding_dim]))
input = tf.nn.embedding_lookup(embedding, X) # [batch_size, len_seq3, embedding_dim2]
lstm_fw_cell = tf.nn.rnn_cell.LSTMCell(n_hidden) # 5
lstm_bw_cell = tf.nn.rnn_cell.LSTMCell(n_hidden)
# output : [batch_size, len_seq, n_hidden], states : [batch_size, n_hidden]
output, final_state = tf.nn.bidirectional_dynamic_rnn(lstm_fw_cell,lstm_bw_cell, input, dtype=tf.float32)
# Attention
# Training
with tf.Session() as sess:
init = tf.global_variables_initializer()
sess.run(init)
#print(input_batch.shape)
output, final_state = sess.run([output, final_state], feed_dict={X: input_batch, Y: target_batch})
#print(output.shape())
print("output_fw-----------------------")
print(output[0])
print("output_bw-----------------------")
print(output[1])
print("=-----------------*****************************************------------------=")
print("output_fw_C-----------------------")
print(final_state[0][0])
print("output_fw_H-----------------------")
print(final_state[0][1])
print("output_bw_C-----------------------")
print(final_state[1][0])
print("output_bw_H-----------------------")
print(final_state[1][1])
#结果
output_fw-----------------------
[[[ 0.00394321 -0.02351782 -0.03405523 0.00305696 0.00405202]
[ 0.06117373 -0.08493181 -0.10548183 0.0173593 0.01642234]
[ 0.09302159 -0.14045352 -0.18056425 0.02292664 0.02151979]]
[[ 0.00897689 -0.0114818 -0.01073236 0.0026193 0.00251714]
[-0.01570433 -0.01093796 -0.02816775 -0.00127712 0.00047153]
[ 0.03564881 -0.08152001 -0.12945394 0.01309824 0.01412579]]
[[ 0.0022017 -0.0556691 -0.09676469 0.00533017 0.00813764]
[-0.02163241 -0.03916193 -0.09379555 0.00117761 0.0033054 ]
[-0.02534943 -0.03482693 -0.09665176 0.00129206 0.00241509]]
[[ 0.00394321 -0.02351782 -0.03405523 0.00305696 0.00405202]
[ 0.0251001 -0.04374754 -0.05323521 0.00777274 0.00770803]
[ 0.06088063 -0.10670137 -0.14363492 0.01596653 0.01687609]]
[[-0.02074501 -0.03782491 -0.08010133 0.00106625 0.0042809 ]
[-0.02030762 -0.02580994 -0.06796815 0.00064477 0.00200086]
[ 0.01159763 -0.09142994 -0.1786155 0.01095188 0.01238727]]
[[ 0.03423732 -0.04540066 -0.0511797 0.0099178 0.00961432]
[ 0.03746802 -0.09071141 -0.12944394 0.01133813 0.0131082 ]
[ 0.04090166 -0.07015887 -0.0894173 0.00954981 0.00913159]]]
output_bw-----------------------
[[[ 0.02337044 0.12136219 0.05673089 -0.01224817 0.0593852 ]
[ 0.01874987 0.09703043 0.0713386 -0.05112801 0.09921131]
[ 0.03311865 0.06603951 0.02200164 -0.0013612 0.04199017]]
[[ 0.04622315 0.08167503 -0.02331962 0.02814881 -0.01469414]
[ 0.06475978 0.1021239 -0.02712514 0.05594093 -0.01951348]
[ 0.02701424 0.06634663 0.03368881 -0.01682826 0.05579737]]
[[ 0.17932206 0.11006899 -0.17011283 0.17438392 -0.14428209]
[ 0.13817243 0.08259486 -0.13932668 0.15030426 -0.13008778]
[ 0.06183491 0.04734965 -0.05742453 0.07186026 -0.05271984]]
[[ 0.03479569 0.09810638 0.01812333 0.01501928 0.02510414]
[ 0.02493659 0.08395203 0.03107387 -0.00838013 0.04572592]
[ 0.03311865 0.06603951 0.02200164 -0.0013612 0.04199017]]
[[ 0.16409379 0.1282753 -0.13091424 0.17192362 -0.10831039]
[ 0.09441089 0.10732475 -0.05974932 0.08499315 -0.04574324]
[ 0.06810828 0.07835089 -0.01157835 0.05307872 0.00895631]]
[[ 0.0615179 0.09249893 -0.00614712 0.01541615 0.02442793]
[ 0.07529346 0.07565043 -0.03864583 0.07073426 -0.02094099]
[ 0.01069495 0.01040417 -0.01082753 0.01262293 -0.00998173]]]
=-----------------*****************************************------------------=
output_fw_C-----------------------
[[ 0.18077272 -0.25065544 -0.2970943 0.05401518 0.04332165]
[ 0.0666052 -0.14184129 -0.21055089 0.03050351 0.02707803]
[-0.04982244 -0.06662414 -0.17368984 0.00284229 0.00523079]
[ 0.11556551 -0.18880183 -0.23572075 0.03740995 0.03326564]
[ 0.0214364 -0.15807846 -0.28311607 0.0270236 0.02515054]
[ 0.08471472 -0.13929133 -0.17238948 0.01954564 0.01927523]]
output_fw_H-----------------------
[[ 0.09302159 -0.14045352 -0.18056425 0.02292664 0.02151979]
[ 0.03564881 -0.08152001 -0.12945394 0.01309824 0.01412579]
[-0.02534943 -0.03482693 -0.09665176 0.00129206 0.00241509]
[ 0.06088063 -0.10670137 -0.14363492 0.01596653 0.01687609]
[ 0.01159763 -0.09142994 -0.1786155 0.01095188 0.01238727]
[ 0.04090166 -0.07015887 -0.0894173 0.00954981 0.00913159]]
output_bw_C-----------------------
[[ 0.04923753 0.29065222 0.10612877 -0.02204721 0.12080507]
[ 0.09414361 0.17565773 -0.04700623 0.05542824 -0.02873464]
[ 0.3784477 0.3159748 -0.2940823 0.28821698 -0.27351445]
[ 0.07218795 0.23126517 0.03378158 0.02722929 0.05014303]
[ 0.32902873 0.36530167 -0.22143884 0.27972797 -0.21702379]
[ 0.13681822 0.22507632 -0.01161141 0.02861859 0.04536273]]
output_bw_H-----------------------
[[ 0.02337044 0.12136219 0.05673089 -0.01224817 0.0593852 ]
[ 0.04622315 0.08167503 -0.02331962 0.02814881 -0.01469414]
[ 0.17932206 0.11006899 -0.17011283 0.17438392 -0.14428209]
[ 0.03479569 0.09810638 0.01812333 0.01501928 0.02510414]
[ 0.16409379 0.1282753 -0.13091424 0.17192362 -0.10831039]
[ 0.0615179 0.09249893 -0.00614712 0.01541615 0.02442793]]