转载自吴恩达老师深度学习课程作业notebook
Neural Machine Translation
Welcome to your first programming assignment for this week!
You will build a Neural Machine Translation (NMT) model to translate human readable dates (“25th of June, 2009”) into machine readable dates (“2009-06-25”). You will do this using an attention model, one of the most sophisticated sequence to sequence models.
This notebook was produced together with NVIDIA’s Deep Learning Institute.
Let’s load all the packages you will need for this assignment.
from keras.layers import Bidirectional, Concatenate, Permute, Dot, Input, LSTM, Multiply
from keras.layers import RepeatVector, Dense, Activation, Lambda
from keras.optimizers import Adam
from keras.utils import to_categorical
from keras.models import load_model, Model
import keras.backend as K
import numpy as np
from faker import Faker
import random
from tqdm import tqdm
from babel.dates import format_date
from nmt_utils import *
import matplotlib.pyplot as plt
%matplotlib inline
Using TensorFlow backend.
1 - Translating human readable dates into machine readable dates
The model you will build here could be used to translate from one language to another, such as translating from English to Hindi. However, language translation requires massive datasets and usually takes days of training on GPUs. To give you a place to experiment with these models even without using massive datasets, we will instead use a simpler “date translation” task.
The network will input a date written in a variety of possible formats (e.g. “the 29th of August 1958”, “03/30/1968”, “24 JUNE 1987”) and translate them into standardized, machine readable dates (e.g. “1958-08-29”, “1968-03-30”, “1987-06-24”). We will have the network learn to output dates in the common machine-readable format YYYY-MM-DD.
1.1 - Dataset
We will train the model on a dataset of 10000 human readable dates and their equivalent, standardized, machine readable dates. Let’s run the following cells to load the dataset and print some examples.
m = 10000
dataset, human_vocab, machine_vocab, inv_machine_vocab = load_dataset(m)
100%|█████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 10457.93it/s]
print (type(dataset),dataset[:5])
print (human_vocab,len(human_vocab))
print (machine_vocab,len(machine_vocab))
print (inv_machine_vocab,len(inv_machine_vocab))
<class 'list'> [('9 may 1998', '1998-05-09'), ('10.09.70', '1970-09-10'), ('4/28/90', '1990-04-28'), ('thursday january 26 1995', '1995-01-26'), ('monday march 7 1983', '1983-03-07')]
{' ': 0, '.': 1, '/': 2, '0': 3, '1': 4, '2': 5, '3': 6, '4': 7, '5': 8, '6': 9, '7': 10, '8': 11, '9': 12, 'a': 13, 'b': 14, 'c': 15, 'd': 16, 'e': 17, 'f': 18, 'g': 19, 'h': 20, 'i': 21, 'j': 22, 'l': 23, 'm': 24, 'n': 25, 'o': 26, 'p': 27, 'r': 28, 's': 29, 't': 30, 'u': 31, 'v': 32, 'w': 33, 'y': 34, '<unk>': 35, '<pad>': 36} 37
{'-': 0, '0': 1, '1': 2, '2': 3, '3': 4, '4': 5, '5': 6, '6': 7, '7': 8, '8': 9, '9': 10} 11
{0: '-', 1: '0', 2: '1', 3: '2', 4: '3', 5: '4', 6: '5', 7: '6', 8: '7', 9: '8', 10: '9'} 11
You’ve loaded:
dataset: a list of tuples of (human readable date, machine readable date)human_vocab: a python dictionary mapping all characters used in the human readable dates to an integer-valued indexmachine_vocab: a python dictionary mapping all characters used in machine readable dates to an integer-valued index. These indices are not necessarily consistent withhuman_vocab.inv_machine_vocab: the inverse dictionary ofmachine_vocab, mapping from indices back to characters.
Let’s preprocess the data and map the raw text data into the index values. We will also use Tx=30 (which we assume is the maximum length of the human readable date; if we get a longer input, we would have to truncate it) and Ty=10 (since “YYYY-MM-DD” is 10 characters long).
'''
zip() 函数用于将可迭代的对象作为参数，将对象中对应的元素打包成一个个元组，然后返回由这些元组组成的列表
如果各个迭代器的元素个数不一致，则返回列表长度与最短的对象相同，利用 * 号操作符，可以将元组解压为列表
'''
X_test, Y_test = zip(*dataset)
print (X_test[:5],'\n',Y_test[:5])
dataset_test = zip(X_test[:5],Y_test[:5])
print (dataset_test,type(dataset_test))
for i,j in dataset_test:
print (i,j)
('9 may 1998', '10.09.70', '4/28/90', 'thursday january 26 1995', 'monday march 7 1983')
('1998-05-09', '1970-09-10', '1990-04-28', '1995-01-26', '1983-03-07')
<zip object at 0x0000029F439FB6C8> <class 'zip'>
9 may 1998 1998-05-09
10.09.70 1970-09-10
4/28/90 1990-04-28
thursday january 26 1995 1995-01-26
monday march 7 1983 1983-03-07
'''
将string按vocab方式重新编码，长度统一为length，空缺补全为pad
'''
def string_to_int(string, length, vocab):
"""
Converts all strings in the vocabulary into a list of integers representing the positions of the
input string's characters in the "vocab"
Arguments:
string -- input string, e.g. 'Wed 10 Jul 2007'
length -- the number of time steps you'd like, determines if the output will be padded or cut
vocab -- vocabulary, dictionary used to index every character of your "string"
Returns:
rep -- list of integers (or '<unk>') (size = length) representing the position of the string's character in the vocabulary
"""
#make lower to standardize
string = string.lower()
string = string.replace(',','')
if len(string) > length:
string = string[:length]
rep = list(map(lambda x: vocab.get(x, '<unk>'), string))
if len(string) < length:
rep += [vocab['<pad>']] * (length - len(string))
#print (rep)
return rep
X_test1 = np.array([string_to_int(i, Tx, human_vocab) for i in X_test])
print (X_test[:5])
print (human_vocab)
print (X_test1[:5])
('9 may 1998', '10.09.70', '4/28/90', 'thursday january 26 1995', 'monday march 7 1983')
{' ': 0, '.': 1, '/': 2, '0': 3, '1': 4, '2': 5, '3': 6, '4': 7, '5': 8, '6': 9, '7': 10, '8': 11, '9': 12, 'a': 13, 'b': 14, 'c': 15, 'd': 16, 'e': 17, 'f': 18, 'g': 19, 'h': 20, 'i': 21, 'j': 22, 'l': 23, 'm': 24, 'n': 25, 'o': 26, 'p': 27, 'r': 28, 's': 29, 't': 30, 'u': 31, 'v': 32, 'w': 33, 'y': 34, '<unk>': 35, '<pad>': 36}
[[12 0 24 13 34 0 4 12 12 11 36 36 36 36 36 36 36 36 36 36 36 36 36 36
36 36 36 36 36 36]
[ 4 3 1 3 12 1 10 3 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36
36 36 36 36 36 36]
[ 7 2 5 11 2 12 3 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36
36 36 36 36 36 36]
[30 20 31 28 29 16 13 34 0 22 13 25 31 13 28 34 0 5 9 0 4 12 12 8
36 36 36 36 36 36]
[24 26 25 16 13 34 0 24 13 28 15 20 0 10 0 4 12 11 6 36 36 36 36 36
36 36 36 36 36 36]]
'''keras中to_categorical,将列表转化为独热编码矩阵'''
b = [0,1,2,3,4]
#调用to_categorical将b按照9个类别来进行转换
c = to_categorical(b, 9)
print (c)
d = to_categorical(b)
print (d)
[[1. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 1. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 1. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 1. 0. 0. 0. 0.]]
[[1. 0. 0. 0. 0.]
[0. 1. 0. 0. 0.]
[0. 0. 1. 0. 0.]
[0. 0. 0. 1. 0.]
[0. 0. 0. 0. 1.]]
'''
python 的map函数
它接收一个函数 f 和一个 list，并通过把函数 f 依次作用在 list 的每个元素上，得到一个新的 list 并返回
map()函数不改变原有的 list，而是返回一个新的 list
将英文名字到第一个字母大写，后面的字母都小写
'''
def format_name(s):
s1=s[0:1].upper()+s[1:].lower();
return s1;
name_list = ['adam', 'LISA', 'barT']
map_test = map(format_name,name_list )
print (name_list)
print (type(map_test),'\n',list(map_test))
['adam', 'LISA', 'barT']
<class 'map'>
['Adam', 'Lisa', 'Bart']
def preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty):
X, Y = zip(*dataset) #此时的X为人类输入的各种时间形式，Y为对应的固定的机器时间形式
X = np.array([string_to_int(i, Tx, human_vocab) for i in X])
Y = [string_to_int(t, Ty, machine_vocab) for t in Y]
Xoh = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), X)))
Yoh = np.array(list(map(lambda x: to_categorical(x, num_classes=len(machine_vocab)), Y)))
return X, np.array(Y), Xoh, Yoh
Tx = 30
Ty = 10
X, Y, Xoh, Yoh = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty)
print("X.shape:", X[:5],X.shape)
print("Y.shape:", Y[:5],Y.shape)
print("Xoh.shape:", Xoh.shape)
print("Yoh.shape:", Yoh.shape)
X.shape: [[12 0 24 13 34 0 4 12 12 11 36 36 36 36 36 36 36 36 36 36 36 36 36 36
36 36 36 36 36 36]
[ 4 3 1 3 12 1 10 3 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36
36 36 36 36 36 36]
[ 7 2 5 11 2 12 3 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36
36 36 36 36 36 36]
[30 20 31 28 29 16 13 34 0 22 13 25 31 13 28 34 0 5 9 0 4 12 12 8
36 36 36 36 36 36]
[24 26 25 16 13 34 0 24 13 28 15 20 0 10 0 4 12 11 6 36 36 36 36 36
36 36 36 36 36 36]] (10000, 30)
Y.shape: [[ 2 10 10 9 0 1 6 0 1 10]
[ 2 10 8 1 0 1 10 0 2 1]
[ 2 10 10 1 0 1 5 0 3 9]
[ 2 10 10 6 0 1 2 0 3 7]
[ 2 10 9 4 0 1 4 0 1 8]] (10000, 10)
Xoh.shape: (10000, 30, 37)
Yoh.shape: (10000, 10, 11)
You now have:
X: a processed version of the human readable dates in the training set, where each character is replaced by an index mapped to the character viahuman_vocab. Each date is further padded to T x T_x Tx values with a special character (< pad >).X.shape = (m, Tx)Y: a processed version of the machine readable dates in the training set, where each character is replaced by the index it is mapped to inmachine_vocab. You should haveY.shape = (m, Ty).Xoh: one-hot version ofX, the “1” entry’s index is mapped to the character thanks tohuman_vocab.Xoh.shape = (m, Tx, len(human_vocab))Yoh: one-hot version ofY, the “1” entry’s index is mapped to the character thanks tomachine_vocab.Yoh.shape = (m, Tx, len(machine_vocab)). Here,len(machine_vocab) = 11since there are 11 characters (’-’ as well as 0-9).
Lets also look at some examples of preprocessed training examples. Feel free to play with index in the cell below to navigate the dataset and see how source/target dates are preprocessed.
index = 0
print("Source date:", dataset[index][0])
print("Target date:", dataset[index][1])
print()
print("Source after preprocessing (indices):", X[index])
print("Target after preprocessing (indices):", Y[index])
print()
print("Source after preprocessing (one-hot):", Xoh[index])
print("Target after preprocessing (one-hot):", Yoh[index])
Source date: 9 may 1998
Target date: 1998-05-09
Source after preprocessing (indices): [12 0 24 13 34 0 4 12 12 11 36 36 36 36 36 36 36 36 36 36 36 36 36 36
36 36 36 36 36 36]
Target after preprocessing (indices): [ 2 10 10 9 0 1 6 0 1 10]
Source after preprocessing (one-hot): [[0. 0. 0. ... 0. 0. 0.]
[1. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 1.]
[0. 0. 0. ... 0. 0. 1.]
[0. 0. 0. ... 0. 0. 1.]]
Target after preprocessing (one-hot): [[0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]
[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]
2 - Neural machine translation with attention
If you had to translate a book’s paragraph from French to English, you would not read the whole paragraph, then close the book and translate. Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down.
The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step.
2.1 - Attention mechanism
In this part, you will implement the attention mechanism presented in the lecture videos. Here is a figure to remind you how the model works. The diagram on the left shows the attention model. The diagram on the right shows what one “Attention” step does to calculate the attention variables α ⟨ t , t ′ ⟩ \alpha^{\langle t, t' \rangle} α⟨t,t′⟩, which are used to compute the context variable c o n t e x t ⟨ t ⟩ context^{\langle t \rangle} context⟨t⟩ for each timestep in the output ( t = 1 , … , T y t=1, \ldots, T_y t=1,…,Ty).
Here are some properties of the model that you may notice:
-
There are two separate LSTMs in this model (see diagram on the left). Because the one at the bottom of the picture is a Bi-directional LSTM and comes before the attention mechanism, we will call it pre-attention Bi-LSTM. The LSTM at the top of the diagram comes after the attention mechanism, so we will call it the post-attention LSTM. The pre-attention Bi-LSTM goes through T x T_x Tx time steps; the post-attention LSTM goes through T y T_y Ty time steps.
-
The post-attention LSTM passes s ⟨ t ⟩ , c ⟨ t ⟩ s^{\langle t \rangle}, c^{\langle t \rangle} s⟨t⟩,c⟨t⟩ from one time step to the next. In the lecture videos, we were using only a basic RNN for the post-activation sequence model, so the state captured by the RNN output activations s ⟨ t ⟩ s^{\langle t\rangle} s⟨t⟩. But since we are using an LSTM here, the LSTM has both the output activation s ⟨ t ⟩ s^{\langle t\rangle} s⟨t⟩ and the hidden cell state c ⟨ t ⟩ c^{\langle t\rangle} c⟨t⟩. However, unlike previous text generation examples (such as Dinosaurus in week 1), in this model the post-activation LSTM at time t t t does will not take the specific generated y ⟨ t − 1 ⟩ y^{\langle t-1 \rangle} y⟨t−1⟩ as input; it only takes s ⟨ t ⟩ s^{\langle t\rangle} s⟨t⟩ and c ⟨ t ⟩ c^{\langle t\rangle} c⟨t⟩ as input. We have designed the model this way, because (unlike language generation where adjacent characters are highly correlated) there isn’t as strong a dependency between the previous character and the next character in a YYYY-MM-DD date.
-
We use a ⟨ t ⟩ = [ a → ⟨ t ⟩ ; a ← ⟨ t ⟩ ] a^{\langle t \rangle} = [\overrightarrow{a}^{\langle t \rangle}; \overleftarrow{a}^{\langle t \rangle}] a⟨t⟩=[a⟨t⟩;a⟨t⟩] to represent the concatenation of the activations of both the forward-direction and backward-directions of the pre-attention Bi-LSTM.
-
The diagram on the right uses a
RepeatVectornode to copy s ⟨ t − 1 ⟩ s^{\langle t-1 \rangle} s⟨t−1⟩'s value T x T_x Tx times, and thenConcatenationto concatenate s ⟨ t − 1 ⟩ s^{\langle t-1 \rangle} s⟨t−1⟩ and a ⟨ t ⟩ a^{\langle t \rangle} a⟨t⟩ to compute e ⟨ t , t ′ e^{\langle t, t'} e⟨t,t′, which is then passed through a softmax to compute α ⟨ t , t ′ ⟩ \alpha^{\langle t, t' \rangle} α⟨t,t′⟩. We’ll explain how to useRepeatVectorandConcatenationin Keras below.
Lets implement this model. You will start by implementing two functions: one_step_attention() and model().
1) one_step_attention(): At step
t
t
t, given all the hidden states of the Bi-LSTM (
[
a
<
1
>
,
a
<
2
>
,
.
.
.
,
a
<
T
x
>
]
[a^{<1>},a^{<2>}, ..., a^{<T_x>}]
[a<1>,a<2>,...,a<Tx>]) and the previous hidden state of the second LSTM (
s
<
t
−
1
>
s^{<t-1>}
s<t−1>), one_step_attention() will compute the attention weights (
[
α
<
t
,
1
>
,
α
<
t
,
2
>
,
.
.
.
,
α
<
t
,
T
x
>
]
[\alpha^{<t,1>},\alpha^{<t,2>}, ..., \alpha^{<t,T_x>}]
[α<t,1>,α<t,2>,...,α<t,Tx>]) and output the context vector (see Figure 1 (right) for details):
(1)
c
o
n
t
e
x
t
<
t
>
=
∑
t
′
=
0
T
x
α
<
t
,
t
′
>
a
<
t
′
>
context^{<t>} = \sum_{t' = 0}^{T_x} \alpha^{<t,t'>}a^{<t'>}\tag{1}
context<t>=t′=0∑Txα<t,t′>a<t′>(1)
Note that we are denoting the attention in this notebook c o n t e x t ⟨ t ⟩ context^{\langle t \rangle} context⟨t⟩. In the lecture videos, the context was denoted c ⟨ t ⟩ c^{\langle t \rangle} c⟨t⟩, but here we are calling it c o n t e x t ⟨ t ⟩ context^{\langle t \rangle} context⟨t⟩ to avoid confusion with the (post-attention) LSTM’s internal memory cell variable, which is sometimes also denoted c ⟨ t ⟩ c^{\langle t \rangle} c⟨t⟩.
2) model(): Implements the entire model. It first runs the input through a Bi-LSTM to get back
[
a
<
1
>
,
a
<
2
>
,
.
.
.
,
a
<
T
x
>
]
[a^{<1>},a^{<2>}, ..., a^{<T_x>}]
[a<1>,a<2>,...,a<Tx>]. Then, it calls one_step_attention()
T
y
T_y
Ty times (for loop). At each iteration of this loop, it gives the computed context vector
c
<
t
>
c^{<t>}
c<t> to the second LSTM, and runs the output of the LSTM through a dense layer with softmax activation to generate a prediction
y
^
<
t
>
\hat{y}^{<t>}
y^<t>.
Exercise: Implement one_step_attention(). The function model() will call the layers in one_step_attention()
T
y
T_y
Ty using a for-loop, and it is important that all
T
y
T_y
Ty copies have the same weights. I.e., it should not re-initiaiize the weights every time. In other words, all
T
y
T_y
Ty steps should have shared weights. Here’s how you can implement layers with shareable weights in Keras:
- Define the layer objects (as global variables for examples).
- Call these objects when propagating the input.
We have defined the layers you need as global variables. Please run the following cells to create them. Please check the Keras documentation to make sure you understand what these layers are: RepeatVector(), Concatenate(), Dense(), Activation(), Dot().
a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6], [7, 8]])
c = np.concatenate((a, b), axis = 0)
print (a,'\n',b)
print ('* '*5)
print (c)
d = np.concatenate((a, b), axis = -1)
print ('* '*5)
print (d)
[[1 2]
[3 4]]
[[5 6]
[7 8]]
* * * * *
[[1 2]
[3 4]
[5 6]
[7 8]]
* * * * *
[[1 2 5 6]
[3 4 7 8]]
a = np.array([2,3])
b = np.array([4,5])
# c = np.dot(a,b)
c = np.multiply(a,b)
print (c)
d = np.sum(c)
print (d)
[ 8 15]
23
# Defined shared layers as global variables
repeator = RepeatVector(Tx)
'''RepeatVector层将输入重复n次
输入shape：形如（nb_samples, features）的2D张量
输出shape：形如（nb_samples, n, features）的3D张量
'''
concatenator = Concatenate(axis=-1)
densor = Dense(1, activation = "relu")
activator = Activation(softmax, name='attention_weights') # We are using a custom softmax(axis = 1) loaded in this notebook
dotor = Dot(axes = 1)
Now you can use these layers to implement one_step_attention(). In order to propagate a Keras tensor object X through one of these layers, use layer(X) (or layer([X,Y]) if it requires multiple inputs.), e.g. densor(X) will propagate X through the Dense(1) layer defined above.
# GRADED FUNCTION: one_step_attention
def one_step_attention(a, s_prev):
"""
Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights
"alphas" and the hidden states "a" of the Bi-LSTM.
Arguments:
a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a)
s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s)
Returns:
context -- context vector, input of the next (post-attetion) LSTM cell
"""
### START CODE HERE ###
# Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states "a" (≈ 1 line)
s_prev = repeator(s_prev)
# Use concatenator to concatenate a and s_prev on the last axis (≈ 1 line)
concat = concatenator([a, s_prev])
# Use densor to propagate concat through a small fully-connected neural network to compute the "energies" variable e. (≈1 lines)
e = densor(concat)
# Use activator and e to compute the attention weights "alphas" (≈ 1 line)
alphas = activator(e)
# Use dotor together with "alphas" and "a" to compute the context vector to be given to the next (post-attention) LSTM-cell (≈ 1 line)
context = dotor([alphas, a])
### END CODE HERE ###
return context
You will be able to check the expected output of one_step_attention() after you’ve coded the model() function.
Exercise: Implement model() as explained in figure 2 and the text above. Again, we have defined global layers that will share weights to be used in model().
n_a = 64
n_s = 128
post_activation_LSTM_cell = LSTM(n_s, return_state = True)
output_layer = Dense(len(machine_vocab), activation=softmax)
Now you can use these layers
T
y
T_y
Ty times in a for loop to generate the outputs, and their parameters will not be reinitialized. You will have to carry out the following steps:
-
Propagate the input into a Bidirectional LSTM
-
Iterate for t = 0 , … , T y − 1 t = 0, \dots, T_y-1 t=0,…,Ty−1:
- Call
one_step_attention()on [ α < t , 1 > , α < t , 2 > , . . . , α < t , T x > ] [\alpha^{<t,1>},\alpha^{<t,2>}, ..., \alpha^{<t,T_x>}] [α<t,1>,α<t,2>,...,α<t,Tx>] and s < t − 1 > s^{<t-1>} s<t−1> to get the context vector c o n t e x t < t > context^{<t>} context<t>. - Give
c
o
n
t
e
x
t
<
t
>
context^{<t>}
context<t> to the post-attention LSTM cell. Remember pass in the previous hidden-state
s
⟨
t
−
1
⟩
s^{\langle t-1\rangle}
s⟨t−1⟩ and cell-states
c
⟨
t
−
1
⟩
c^{\langle t-1\rangle}
c⟨t−1⟩ of this LSTM using
initial_state= [previous hidden state, previous cell state]. Get back the new hidden state s < t > s^{<t>} s<t> and the new cell state c < t > c^{<t>} c<t>. - Apply a softmax layer to s < t > s^{<t>} s<t>, get the output.
- Save the output by adding it to the list of outputs.
- Call
-
Create your Keras model instance, it should have three inputs (“inputs”, s < 0 > s^{<0>} s<0> and c < 0 > c^{<0>} c<0>) and output the list of “outputs”.
# GRADED FUNCTION: model
def model(Tx, Ty, n_a, n_s, human_vocab_size, machine_vocab_size):
"""
Arguments:
Tx -- length of the input sequence
Ty -- length of the output sequence
n_a -- hidden state size of the Bi-LSTM
n_s -- hidden state size of the post-attention LSTM
human_vocab_size -- size of the python dictionary "human_vocab"
machine_vocab_size -- size of the python dictionary "machine_vocab"
Returns:
model -- Keras model instance
"""
# Define the inputs of your model with a shape (Tx,)
# Define s0 and c0, initial hidden state for the decoder LSTM of shape (n_s,)
X = Input(shape=(Tx, human_vocab_size))
s0 = Input(shape=(n_s,), name='s0')
c0 = Input(shape=(n_s,), name='c0')
s = s0
c = c0
# Initialize empty list of outputs
outputs = []
### START CODE HERE ###
# Step 1: Define your pre-attention Bi-LSTM. Remember to use return_sequences=True. (≈ 1 line)
a = Bidirectional(LSTM(n_a, return_sequences=True))(X)
# Step 2: Iterate for Ty steps
for t in range(Ty):
# Step 2.A: Perform one step of the attention mechanism to get back the context vector at step t (≈ 1 line)
context = one_step_attention(a, s)
# Step 2.B: Apply the post-attention LSTM cell to the "context" vector.
# Don't forget to pass: initial_state = [hidden state, cell state] (≈ 1 line)
s, _, c = post_activation_LSTM_cell(context, initial_state= [s, c])
# Step 2.C: Apply Dense layer to the hidden state output of the post-attention LSTM (≈ 1 line)
out = output_layer(s)
# Step 2.D: Append "out" to the "outputs" list (≈ 1 line)
outputs.append(out)
# Step 3: Create model instance taking three inputs and returning the list of outputs. (≈ 1 line)
model = Model(inputs = [X, s0,c0], outputs = outputs )
### END CODE HERE ###
return model
Run the following cell to create your model.
model = model(Tx, Ty, n_a, n_s, len(human_vocab), len(machine_vocab))
Let’s get a summary of the model to check if it matches the expected output.
model.summary()
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_3 (InputLayer) (None, 30, 37) 0
__________________________________________________________________________________________________
s0 (InputLayer) (None, 128) 0
__________________________________________________________________________________________________
bidirectional_3 (Bidirectional) (None, 30, 128) 52224 input_3[0][0]
__________________________________________________________________________________________________
repeat_vector_1 (RepeatVector) (None, 30, 128) 0 s0[0][0]
lstm_1[0][0]
lstm_1[1][0]
lstm_1[2][0]
lstm_1[3][0]
lstm_1[4][0]
lstm_1[5][0]
lstm_1[6][0]
lstm_1[7][0]
lstm_1[8][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 30, 256) 0 bidirectional_3[0][0]
repeat_vector_1[2][0]
bidirectional_3[0][0]
repeat_vector_1[3][0]
bidirectional_3[0][0]
repeat_vector_1[4][0]
bidirectional_3[0][0]
repeat_vector_1[5][0]
bidirectional_3[0][0]
repeat_vector_1[6][0]
bidirectional_3[0][0]
repeat_vector_1[7][0]
bidirectional_3[0][0]
repeat_vector_1[8][0]
bidirectional_3[0][0]
repeat_vector_1[9][0]
bidirectional_3[0][0]
repeat_vector_1[10][0]
bidirectional_3[0][0]
repeat_vector_1[11][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 30, 1) 257 concatenate_1[2][0]
concatenate_1[3][0]
concatenate_1[4][0]
concatenate_1[5][0]
concatenate_1[6][0]
concatenate_1[7][0]
concatenate_1[8][0]
concatenate_1[9][0]
concatenate_1[10][0]
concatenate_1[11][0]
__________________________________________________________________________________________________
attention_weights (Activation) (None, 30, 1) 0 dense_1[1][0]
dense_1[2][0]
dense_1[3][0]
dense_1[4][0]
dense_1[5][0]
dense_1[6][0]
dense_1[7][0]
dense_1[8][0]
dense_1[9][0]
dense_1[10][0]
__________________________________________________________________________________________________
dot_1 (Dot) (None, 1, 128) 0 attention_weights[0][0]
bidirectional_3[0][0]
attention_weights[1][0]
bidirectional_3[0][0]
attention_weights[2][0]
bidirectional_3[0][0]
attention_weights[3][0]
bidirectional_3[0][0]
attention_weights[4][0]
bidirectional_3[0][0]
attention_weights[5][0]
bidirectional_3[0][0]
attention_weights[6][0]
bidirectional_3[0][0]
attention_weights[7][0]
bidirectional_3[0][0]
attention_weights[8][0]
bidirectional_3[0][0]
attention_weights[9][0]
bidirectional_3[0][0]
__________________________________________________________________________________________________
c0 (InputLayer) (None, 128) 0
__________________________________________________________________________________________________
lstm_1 (LSTM) [(None, 128), (None, 131584 dot_1[0][0]
s0[0][0]
c0[0][0]
dot_1[1][0]
lstm_1[0][0]
lstm_1[0][2]
dot_1[2][0]
lstm_1[1][0]
lstm_1[1][2]
dot_1[3][0]
lstm_1[2][0]
lstm_1[2][2]
dot_1[4][0]
lstm_1[3][0]
lstm_1[3][2]
dot_1[5][0]
lstm_1[4][0]
lstm_1[4][2]
dot_1[6][0]
lstm_1[5][0]
lstm_1[5][2]
dot_1[7][0]
lstm_1[6][0]
lstm_1[6][2]
dot_1[8][0]
lstm_1[7][0]
lstm_1[7][2]
dot_1[9][0]
lstm_1[8][0]
lstm_1[8][2]
__________________________________________________________________________________________________
dense_2 (Dense) (None, 11) 1419 lstm_1[0][0]
lstm_1[1][0]
lstm_1[2][0]
lstm_1[3][0]
lstm_1[4][0]
lstm_1[5][0]
lstm_1[6][0]
lstm_1[7][0]
lstm_1[8][0]
lstm_1[9][0]
==================================================================================================
Total params: 185,484
Trainable params: 185,484
Non-trainable params: 0
__________________________________________________________________________________________________
Expected Output:
Here is the summary you should see
Total params:
185,484Trainable params:
185,484Non-trainable params:
0bidirectional_1’s output shape
(None, 30, 128)repeat_vector_1’s output shape
(None, 30, 128)concatenate_1’s output shape
(None, 30, 256)attention_weights’s output shape
(None, 30, 1)dot_1’s output shape
(None, 1, 128)dense_2’s output shape
(None, 11)As usual, after creating your model in Keras, you need to compile it and define what loss, optimizer and metrics your are want to use. Compile your model using categorical_crossentropy loss, a custom Adam optimizer (learning rate = 0.005,
β
1
=
0.9
\beta_1 = 0.9
β1=0.9,
β
2
=
0.999
\beta_2 = 0.999
β2=0.999, decay = 0.01) and ['accuracy'] metrics:
### START CODE HERE ### (≈2 lines)
opt = model.compile(optimizer=Adam(lr=0.005, beta_1=0.9, beta_2=0.999, decay=0.01),
metrics=['accuracy'],
loss='categorical_crossentropy')
opt
### END CODE HERE ###
The last step is to define all your inputs and outputs to fit the model:
- You already have X of shape ( m = 10000 , T x = 30 ) (m = 10000, T_x = 30) (m=10000,Tx=30) containing the training examples.
- You need to create
s0andc0to initialize yourpost_activation_LSTM_cellwith 0s. - Given the
model()you coded, you need the “outputs” to be a list of 11 elements of shape (m, T_y). So that:outputs[i][0], ..., outputs[i][Ty]represent the true labels (characters) corresponding to the i t h i^{th} ith training example (X[i]). More generally,outputs[i][j]is the true label of the j t h j^{th} jth character in the i t h i^{th} ith training example.
s0 = np.zeros((m, n_s))
c0 = np.zeros((m, n_s))
outputs = list(Yoh.swapaxes(0,1))
Let’s now fit the model and run it for one epoch.
model.fit([Xoh, s0, c0], outputs, epochs=1, batch_size=100)
Epoch 1/1
2800/10000 [=======>......................] - ETA: 42:40 - loss: 24.1277 - dense_2_loss: 2.3966 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.0300 - dense_2_acc_2: 0.0500 - dense_2_acc_3: 0.1000 - dense_2_acc_4: 0.0000e+00 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.1000 - dense_2_acc_7: 0.0000e+00 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.07 - ETA: 21:24 - loss: 23.8081 - dense_2_loss: 2.4068 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.0200 - dense_2_acc_2: 0.0250 - dense_2_acc_3: 0.0500 - dense_2_acc_4: 0.4950 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0500 - dense_2_acc_7: 0.5000 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.0350 - ETA: 14:17 - loss: 23.4265 - dense_2_loss: 2.4244 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.0167 - dense_2_acc_2: 0.0167 - dense_2_acc_3: 0.0333 - dense_2_acc_4: 0.6633 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0333 - dense_2_acc_7: 0.6667 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.02 - ETA: 10:44 - loss: 23.0623 - dense_2_loss: 2.5977 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.0125 - dense_2_acc_2: 0.0125 - dense_2_acc_3: 0.0250 - dense_2_acc_4: 0.7475 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0250 - dense_2_acc_7: 0.7500 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.01 - ETA: 8:35 - loss: 22.8817 - dense_2_loss: 2.7836 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.0100 - dense_2_acc_2: 0.0100 - dense_2_acc_3: 0.0200 - dense_2_acc_4: 0.7980 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0200 - dense_2_acc_7: 0.8000 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.0140 - ETA: 7:10 - loss: 22.7298 - dense_2_loss: 2.8403 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.0617 - dense_2_acc_2: 0.0083 - dense_2_acc_3: 0.0167 - dense_2_acc_4: 0.8317 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0167 - dense_2_acc_7: 0.8333 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.011 - ETA: 6:09 - loss: 22.5800 - dense_2_loss: 2.8415 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.1057 - dense_2_acc_2: 0.0286 - dense_2_acc_3: 0.0143 - dense_2_acc_4: 0.8557 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0143 - dense_2_acc_7: 0.8571 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.010 - ETA: 5:22 - loss: 22.4674 - dense_2_loss: 2.8144 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.1500 - dense_2_acc_2: 0.0550 - dense_2_acc_3: 0.0125 - dense_2_acc_4: 0.8738 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0125 - dense_2_acc_7: 0.8750 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.008 - ETA: 4:47 - loss: 22.3656 - dense_2_loss: 2.8019 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.1767 - dense_2_acc_2: 0.0722 - dense_2_acc_3: 0.0111 - dense_2_acc_4: 0.8878 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0111 - dense_2_acc_7: 0.8889 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.007 - ETA: 4:18 - loss: 22.2621 - dense_2_loss: 2.8046 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2010 - dense_2_acc_2: 0.0890 - dense_2_acc_3: 0.0100 - dense_2_acc_4: 0.8990 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0100 - dense_2_acc_7: 0.9000 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.007 - ETA: 3:54 - loss: 22.1678 - dense_2_loss: 2.8016 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2173 - dense_2_acc_2: 0.0945 - dense_2_acc_3: 0.0182 - dense_2_acc_4: 0.9082 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0091 - dense_2_acc_7: 0.9091 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.006 - ETA: 3:35 - loss: 22.0908 - dense_2_loss: 2.8025 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2292 - dense_2_acc_2: 0.1042 - dense_2_acc_3: 0.0233 - dense_2_acc_4: 0.9158 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0083 - dense_2_acc_7: 0.9167 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.005 - ETA: 3:18 - loss: 21.9966 - dense_2_loss: 2.7944 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2454 - dense_2_acc_2: 0.1115 - dense_2_acc_3: 0.0338 - dense_2_acc_4: 0.9223 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0077 - dense_2_acc_7: 0.9231 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.005 - ETA: 3:04 - loss: 21.8974 - dense_2_loss: 2.8001 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2564 - dense_2_acc_2: 0.1200 - dense_2_acc_3: 0.0436 - dense_2_acc_4: 0.9279 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0071 - dense_2_acc_7: 0.9286 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.005 - ETA: 2:51 - loss: 21.8088 - dense_2_loss: 2.7991 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2620 - dense_2_acc_2: 0.1247 - dense_2_acc_3: 0.0513 - dense_2_acc_4: 0.9327 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0067 - dense_2_acc_7: 0.9333 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.004 - ETA: 2:40 - loss: 21.7262 - dense_2_loss: 2.8001 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2637 - dense_2_acc_2: 0.1256 - dense_2_acc_3: 0.0556 - dense_2_acc_4: 0.9369 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0063 - dense_2_acc_7: 0.9375 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.004 - ETA: 2:31 - loss: 21.6509 - dense_2_loss: 2.7937 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.2835 - dense_2_acc_2: 0.1306 - dense_2_acc_3: 0.0588 - dense_2_acc_4: 0.9406 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0059 - dense_2_acc_7: 0.9412 - dense_2_acc_8: 0.0000e+00 - dense_2_acc_9: 0.010 - ETA: 2:22 - loss: 21.5728 - dense_2_loss: 2.7850 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.3044 - dense_2_acc_2: 0.1372 - dense_2_acc_3: 0.0633 - dense_2_acc_4: 0.9439 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0056 - dense_2_acc_7: 0.9444 - dense_2_acc_8: 0.0194 - dense_2_acc_9: 0.0172 - ETA: 2:14 - loss: 21.4888 - dense_2_loss: 2.7692 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.3184 - dense_2_acc_2: 0.1479 - dense_2_acc_3: 0.0668 - dense_2_acc_4: 0.9468 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0053 - dense_2_acc_7: 0.9474 - dense_2_acc_8: 0.0358 - dense_2_acc_9: 0.022 - ETA: 2:07 - loss: 21.4178 - dense_2_loss: 2.7619 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.3345 - dense_2_acc_2: 0.1520 - dense_2_acc_3: 0.0685 - dense_2_acc_4: 0.9495 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0050 - dense_2_acc_7: 0.9500 - dense_2_acc_8: 0.0340 - dense_2_acc_9: 0.029 - ETA: 2:01 - loss: 21.3456 - dense_2_loss: 2.7532 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.3452 - dense_2_acc_2: 0.1600 - dense_2_acc_3: 0.0652 - dense_2_acc_4: 0.9519 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0048 - dense_2_acc_7: 0.9524 - dense_2_acc_8: 0.0348 - dense_2_acc_9: 0.036 - ETA: 1:55 - loss: 21.2648 - dense_2_loss: 2.7484 - dense_2_acc: 0.0000e+00 - dense_2_acc_1: 0.3536 - dense_2_acc_2: 0.1645 - dense_2_acc_3: 0.0623 - dense_2_acc_4: 0.9541 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0045 - dense_2_acc_7: 0.9541 - dense_2_acc_8: 0.0482 - dense_2_acc_9: 0.040 - ETA: 1:50 - loss: 21.1789 - dense_2_loss: 2.7442 - dense_2_acc: 0.0130 - dense_2_acc_1: 0.3670 - dense_2_acc_2: 0.1683 - dense_2_acc_3: 0.0596 - dense_2_acc_4: 0.9561 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0043 - dense_2_acc_7: 0.9561 - dense_2_acc_8: 0.0596 - dense_2_acc_9: 0.0443 - ETA: 1:45 - loss: 21.1287 - dense_2_loss: 2.7555 - dense_2_acc: 0.0354 - dense_2_acc_1: 0.3796 - dense_2_acc_2: 0.1692 - dense_2_acc_3: 0.0604 - dense_2_acc_4: 0.9579 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0042 - dense_2_acc_7: 0.9579 - dense_2_acc_8: 0.0571 - dense_2_acc_9: 0.045 - ETA: 1:40 - loss: 21.0564 - dense_2_loss: 2.7616 - dense_2_acc: 0.0584 - dense_2_acc_1: 0.3888 - dense_2_acc_2: 0.1708 - dense_2_acc_3: 0.0616 - dense_2_acc_4: 0.9596 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0040 - dense_2_acc_7: 0.9596 - dense_2_acc_8: 0.0740 - dense_2_acc_9: 0.048 - ETA: 1:36 - loss: 21.0089 - dense_2_loss: 2.7677 - dense_2_acc: 0.0769 - dense_2_acc_1: 0.3946 - dense_2_acc_2: 0.1758 - dense_2_acc_3: 0.0592 - dense_2_acc_4: 0.9612 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0038 - dense_2_acc_7: 0.9227 - dense_2_acc_8: 0.0865 - dense_2_acc_9: 0.052 - ETA: 1:32 - loss: 20.9454 - dense_2_loss: 2.7648 - dense_2_acc: 0.0963 - dense_2_acc_1: 0.4022 - dense_2_acc_2: 0.1789 - dense_2_acc_3: 0.0570 - dense_2_acc_4: 0.9626 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0037 - dense_2_acc_7: 0.9256 - dense_2_acc_8: 0.0948 - dense_2_acc_9: 0.053 - ETA: 1:28 - loss: 20.8728 - dense_2_loss: 2.7637 - dense_2_acc: 0.1171 - dense_2_acc_1: 0.4121 - dense_2_acc_2: 0.1832 - dense_2_acc_3: 0.0550 - dense_2_acc_4: 0.9639 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0036 - dense_2_acc_7: 0.9282 - dense_2_acc_8: 0.0914 - dense_2_acc_9: 0.055 5700/10000 [================>.............] - ETA: 1:25 - loss: 20.8057 - dense_2_loss: 2.7623 - dense_2_acc: 0.1338 - dense_2_acc_1: 0.4186 - dense_2_acc_2: 0.1848 - dense_2_acc_3: 0.0531 - dense_2_acc_4: 0.9652 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0034 - dense_2_acc_7: 0.9307 - dense_2_acc_8: 0.0883 - dense_2_acc_9: 0.056 - ETA: 1:21 - loss: 20.7516 - dense_2_loss: 2.7639 - dense_2_acc: 0.1507 - dense_2_acc_1: 0.4260 - dense_2_acc_2: 0.1850 - dense_2_acc_3: 0.0563 - dense_2_acc_4: 0.9663 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0043 - dense_2_acc_7: 0.9013 - dense_2_acc_8: 0.0943 - dense_2_acc_9: 0.058 - ETA: 1:18 - loss: 20.6786 - dense_2_loss: 2.7605 - dense_2_acc: 0.1674 - dense_2_acc_1: 0.4339 - dense_2_acc_2: 0.1890 - dense_2_acc_3: 0.0574 - dense_2_acc_4: 0.9674 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0048 - dense_2_acc_7: 0.8739 - dense_2_acc_8: 0.1000 - dense_2_acc_9: 0.061 - ETA: 1:16 - loss: 20.6031 - dense_2_loss: 2.7545 - dense_2_acc: 0.1844 - dense_2_acc_1: 0.4425 - dense_2_acc_2: 0.1897 - dense_2_acc_3: 0.0556 - dense_2_acc_4: 0.9684 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0047 - dense_2_acc_7: 0.8778 - dense_2_acc_8: 0.1025 - dense_2_acc_9: 0.065 - ETA: 1:13 - loss: 20.5385 - dense_2_loss: 2.7521 - dense_2_acc: 0.1961 - dense_2_acc_1: 0.4464 - dense_2_acc_2: 0.1888 - dense_2_acc_3: 0.0539 - dense_2_acc_4: 0.9694 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0045 - dense_2_acc_7: 0.8815 - dense_2_acc_8: 0.1036 - dense_2_acc_9: 0.066 - ETA: 1:10 - loss: 20.4770 - dense_2_loss: 2.7514 - dense_2_acc: 0.2088 - dense_2_acc_1: 0.4518 - dense_2_acc_2: 0.1888 - dense_2_acc_3: 0.0524 - dense_2_acc_4: 0.9703 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0044 - dense_2_acc_7: 0.8821 - dense_2_acc_8: 0.1085 - dense_2_acc_9: 0.067 - ETA: 1:08 - loss: 20.4158 - dense_2_loss: 2.7548 - dense_2_acc: 0.2197 - dense_2_acc_1: 0.4557 - dense_2_acc_2: 0.1897 - dense_2_acc_3: 0.0509 - dense_2_acc_4: 0.9711 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0043 - dense_2_acc_7: 0.8774 - dense_2_acc_8: 0.1166 - dense_2_acc_9: 0.068 - ETA: 1:06 - loss: 20.3419 - dense_2_loss: 2.7505 - dense_2_acc: 0.2317 - dense_2_acc_1: 0.4611 - dense_2_acc_2: 0.1911 - dense_2_acc_3: 0.0497 - dense_2_acc_4: 0.9719 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0042 - dense_2_acc_7: 0.8803 - dense_2_acc_8: 0.1219 - dense_2_acc_9: 0.072 - ETA: 1:03 - loss: 20.2682 - dense_2_loss: 2.7477 - dense_2_acc: 0.2419 - dense_2_acc_1: 0.4651 - dense_2_acc_2: 0.1978 - dense_2_acc_3: 0.0503 - dense_2_acc_4: 0.9727 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0041 - dense_2_acc_7: 0.8835 - dense_2_acc_8: 0.1192 - dense_2_acc_9: 0.076 - ETA: 1:01 - loss: 20.2016 - dense_2_loss: 2.7495 - dense_2_acc: 0.2524 - dense_2_acc_1: 0.4697 - dense_2_acc_2: 0.2018 - dense_2_acc_3: 0.0521 - dense_2_acc_4: 0.9734 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0039 - dense_2_acc_7: 0.8861 - dense_2_acc_8: 0.1232 - dense_2_acc_9: 0.078 - ETA: 59s - loss: 20.1238 - dense_2_loss: 2.7484 - dense_2_acc: 0.2636 - dense_2_acc_1: 0.4754 - dense_2_acc_2: 0.2046 - dense_2_acc_3: 0.0518 - dense_2_acc_4: 0.9741 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0038 - dense_2_acc_7: 0.8872 - dense_2_acc_8: 0.1295 - dense_2_acc_9: 0.080 - ETA: 57s - loss: 20.0421 - dense_2_loss: 2.7483 - dense_2_acc: 0.2740 - dense_2_acc_1: 0.4805 - dense_2_acc_2: 0.2067 - dense_2_acc_3: 0.0517 - dense_2_acc_4: 0.9748 - dense_2_acc_5: 0.0000e+00 - dense_2_acc_6: 0.0038 - dense_2_acc_7: 0.8898 - dense_2_acc_8: 0.1325 - dense_2_acc_9: 0.08 - ETA: 55s - loss: 19.9692 - dense_2_loss: 2.7465 - dense_2_acc: 0.2812 - dense_2_acc_1: 0.4827 - dense_2_acc_2: 0.2080 - dense_2_acc_3: 0.0510 - dense_2_acc_4: 0.9754 - dense_2_acc_5: 2.4390e-04 - dense_2_acc_6: 0.0037 - dense_2_acc_7: 0.8924 - dense_2_acc_8: 0.1305 - dense_2_acc_9: 0.08 - ETA: 53s - loss: 19.8887 - dense_2_loss: 2.7419 - dense_2_acc: 0.2888 - dense_2_acc_1: 0.4879 - dense_2_acc_2: 0.2114 - dense_2_acc_3: 0.0502 - dense_2_acc_4: 0.9736 - dense_2_acc_5: 0.0112 - dense_2_acc_6: 0.0038 - dense_2_acc_7: 0.8943 - dense_2_acc_8: 0.1379 - dense_2_acc_9: 0.0843 - ETA: 52s - loss: 19.8212 - dense_2_loss: 2.7372 - dense_2_acc: 0.2958 - dense_2_acc_1: 0.4909 - dense_2_acc_2: 0.2137 - dense_2_acc_3: 0.0509 - dense_2_acc_4: 0.9742 - dense_2_acc_5: 0.0207 - dense_2_acc_6: 0.0056 - dense_2_acc_7: 0.8851 - dense_2_acc_8: 0.1440 - dense_2_acc_9: 0.08 - ETA: 50s - loss: 19.7435 - dense_2_loss: 2.7330 - dense_2_acc: 0.3023 - dense_2_acc_1: 0.4995 - dense_2_acc_2: 0.2159 - dense_2_acc_3: 0.0509 - dense_2_acc_4: 0.9720 - dense_2_acc_5: 0.0366 - dense_2_acc_6: 0.0055 - dense_2_acc_7: 0.8877 - dense_2_acc_8: 0.1423 - dense_2_acc_9: 0.08 - ETA: 48s - loss: 19.6641 - dense_2_loss: 2.7296 - dense_2_acc: 0.3084 - dense_2_acc_1: 0.5076 - dense_2_acc_2: 0.2187 - dense_2_acc_3: 0.0500 - dense_2_acc_4: 0.9727 - dense_2_acc_5: 0.0460 - dense_2_acc_6: 0.0053 - dense_2_acc_7: 0.8902 - dense_2_acc_8: 0.1396 - dense_2_acc_9: 0.08 - ETA: 47s - loss: 19.5714 - dense_2_loss: 2.7247 - dense_2_acc: 0.3167 - dense_2_acc_1: 0.5117 - dense_2_acc_2: 0.2211 - dense_2_acc_3: 0.0498 - dense_2_acc_4: 0.9733 - dense_2_acc_5: 0.0498 - dense_2_acc_6: 0.0061 - dense_2_acc_7: 0.8926 - dense_2_acc_8: 0.1400 - dense_2_acc_9: 0.09 - ETA: 45s - loss: 19.4835 - dense_2_loss: 2.7211 - dense_2_acc: 0.3236 - dense_2_acc_1: 0.5162 - dense_2_acc_2: 0.2236 - dense_2_acc_3: 0.0491 - dense_2_acc_4: 0.9738 - dense_2_acc_5: 0.0632 - dense_2_acc_6: 0.0066 - dense_2_acc_7: 0.8947 - dense_2_acc_8: 0.1455 - dense_2_acc_9: 0.09 - ETA: 44s - loss: 19.3891 - dense_2_loss: 2.7151 - dense_2_acc: 0.3283 - dense_2_acc_1: 0.5215 - dense_2_acc_2: 0.2254 - dense_2_acc_3: 0.0487 - dense_2_acc_4: 0.9744 - dense_2_acc_5: 0.0788 - dense_2_acc_6: 0.0090 - dense_2_acc_7: 0.8969 - dense_2_acc_8: 0.1506 - dense_2_acc_9: 0.08 - ETA: 42s - loss: 19.2951 - dense_2_loss: 2.7118 - dense_2_acc: 0.3347 - dense_2_acc_1: 0.5290 - dense_2_acc_2: 0.2286 - dense_2_acc_3: 0.0500 - dense_2_acc_4: 0.9749 - dense_2_acc_5: 0.0937 - dense_2_acc_6: 0.0104 - dense_2_acc_7: 0.8990 - dense_2_acc_8: 0.1522 - dense_2_acc_9: 0.09 - ETA: 41s - loss: 19.1908 - dense_2_loss: 2.7057 - dense_2_acc: 0.3400 - dense_2_acc_1: 0.5364 - dense_2_acc_2: 0.2324 - dense_2_acc_3: 0.0502 - dense_2_acc_4: 0.9754 - dense_2_acc_5: 0.1076 - dense_2_acc_6: 0.0120 - dense_2_acc_7: 0.9010 - dense_2_acc_8: 0.1578 - dense_2_acc_9: 0.09 - ETA: 40s - loss: 19.0835 - dense_2_loss: 2.6996 - dense_2_acc: 0.3447 - dense_2_acc_1: 0.5398 - dense_2_acc_2: 0.2331 - dense_2_acc_3: 0.0510 - dense_2_acc_4: 0.9759 - dense_2_acc_5: 0.1224 - dense_2_acc_6: 0.0127 - dense_2_acc_7: 0.9029 - dense_2_acc_8: 0.1647 - dense_2_acc_9: 0.09 - ETA: 38s - loss: 18.9628 - dense_2_loss: 2.6919 - dense_2_acc: 0.3496 - dense_2_acc_1: 0.5442 - dense_2_acc_2: 0.2367 - dense_2_acc_3: 0.0527 - dense_2_acc_4: 0.9763 - dense_2_acc_5: 0.1315 - dense_2_acc_6: 0.0140 - dense_2_acc_7: 0.9048 - dense_2_acc_8: 0.1710 - dense_2_acc_9: 0.09 - ETA: 37s - loss: 18.8454 - dense_2_loss: 2.6848 - dense_2_acc: 0.3545 - dense_2_acc_1: 0.5517 - dense_2_acc_2: 0.2400 - dense_2_acc_3: 0.0542 - dense_2_acc_4: 0.9766 - dense_2_acc_5: 0.1396 - dense_2_acc_6: 0.0155 - dense_2_acc_7: 0.9066 - dense_2_acc_8: 0.1777 - dense_2_acc_9: 0.09 - ETA: 36s - loss: 18.7326 - dense_2_loss: 2.6778 - dense_2_acc: 0.3594 - dense_2_acc_1: 0.5583 - dense_2_acc_2: 0.2443 - dense_2_acc_3: 0.0543 - dense_2_acc_4: 0.9770 - dense_2_acc_5: 0.1530 - dense_2_acc_6: 0.0169 - dense_2_acc_7: 0.9083 - dense_2_acc_8: 0.1817 - dense_2_acc_9: 0.09 - ETA: 35s - loss: 18.6201 - dense_2_loss: 2.6725 - dense_2_acc: 0.3676 - dense_2_acc_1: 0.5636 - dense_2_acc_2: 0.2462 - dense_2_acc_3: 0.0545 - dense_2_acc_4: 0.9775 - dense_2_acc_5: 0.1644 - dense_2_acc_6: 0.0180 - dense_2_acc_7: 0.9100 - dense_2_acc_8: 0.1847 - dense_2_acc_9: 0.09 - ETA: 34s - loss: 18.4984 - dense_2_loss: 2.6661 - dense_2_acc: 0.3755 - dense_2_acc_1: 0.5695 - dense_2_acc_2: 0.2487 - dense_2_acc_3: 0.0543 - dense_2_acc_4: 0.9779 - dense_2_acc_5: 0.1752 - dense_2_acc_6: 0.0195 - dense_2_acc_7: 0.9116 - dense_2_acc_8: 0.1886 - dense_2_acc_9: 0.10 - ETA: 33s - loss: 18.3760 - dense_2_loss: 2.6586 - dense_2_acc: 0.3818 - dense_2_acc_1: 0.5751 - dense_2_acc_2: 0.2533 - dense_2_acc_3: 0.0551 - dense_2_acc_4: 0.9782 - dense_2_acc_5: 0.1861 - dense_2_acc_6: 0.0202 - dense_2_acc_7: 0.9132 - dense_2_acc_8: 0.1933 - dense_2_acc_9: 0.1018 8600/10000 [========================>.....] - ETA: 31s - loss: 18.2533 - dense_2_loss: 2.6513 - dense_2_acc: 0.3881 - dense_2_acc_1: 0.5819 - dense_2_acc_2: 0.2564 - dense_2_acc_3: 0.0571 - dense_2_acc_4: 0.9786 - dense_2_acc_5: 0.1959 - dense_2_acc_6: 0.0222 - dense_2_acc_7: 0.9147 - dense_2_acc_8: 0.1972 - dense_2_acc_9: 0.10 - ETA: 30s - loss: 18.1366 - dense_2_loss: 2.6453 - dense_2_acc: 0.3919 - dense_2_acc_1: 0.5878 - dense_2_acc_2: 0.2605 - dense_2_acc_3: 0.0576 - dense_2_acc_4: 0.9790 - dense_2_acc_5: 0.2053 - dense_2_acc_6: 0.0234 - dense_2_acc_7: 0.9161 - dense_2_acc_8: 0.2010 - dense_2_acc_9: 0.10 - ETA: 29s - loss: 18.0232 - dense_2_loss: 2.6397 - dense_2_acc: 0.3962 - dense_2_acc_1: 0.5935 - dense_2_acc_2: 0.2643 - dense_2_acc_3: 0.0582 - dense_2_acc_4: 0.9793 - dense_2_acc_5: 0.2150 - dense_2_acc_6: 0.0260 - dense_2_acc_7: 0.9175 - dense_2_acc_8: 0.2055 - dense_2_acc_9: 0.10 - ETA: 28s - loss: 17.9099 - dense_2_loss: 2.6332 - dense_2_acc: 0.4013 - dense_2_acc_1: 0.5990 - dense_2_acc_2: 0.2659 - dense_2_acc_3: 0.0593 - dense_2_acc_4: 0.9797 - dense_2_acc_5: 0.2239 - dense_2_acc_6: 0.0287 - dense_2_acc_7: 0.9189 - dense_2_acc_8: 0.2100 - dense_2_acc_9: 0.10 - ETA: 27s - loss: 17.7965 - dense_2_loss: 2.6278 - dense_2_acc: 0.4087 - dense_2_acc_1: 0.6052 - dense_2_acc_2: 0.2687 - dense_2_acc_3: 0.0603 - dense_2_acc_4: 0.9800 - dense_2_acc_5: 0.2335 - dense_2_acc_6: 0.0318 - dense_2_acc_7: 0.9202 - dense_2_acc_8: 0.2147 - dense_2_acc_9: 0.10 - ETA: 26s - loss: 17.6890 - dense_2_loss: 2.6234 - dense_2_acc: 0.4149 - dense_2_acc_1: 0.6110 - dense_2_acc_2: 0.2708 - dense_2_acc_3: 0.0608 - dense_2_acc_4: 0.9803 - dense_2_acc_5: 0.2422 - dense_2_acc_6: 0.0343 - dense_2_acc_7: 0.9214 - dense_2_acc_8: 0.2184 - dense_2_acc_9: 0.10 - ETA: 25s - loss: 17.5786 - dense_2_loss: 2.6180 - dense_2_acc: 0.4202 - dense_2_acc_1: 0.6169 - dense_2_acc_2: 0.2730 - dense_2_acc_3: 0.0612 - dense_2_acc_4: 0.9806 - dense_2_acc_5: 0.2522 - dense_2_acc_6: 0.0372 - dense_2_acc_7: 0.9227 - dense_2_acc_8: 0.2231 - dense_2_acc_9: 0.11 - ETA: 24s - loss: 17.4770 - dense_2_loss: 2.6140 - dense_2_acc: 0.4252 - dense_2_acc_1: 0.6215 - dense_2_acc_2: 0.2755 - dense_2_acc_3: 0.0620 - dense_2_acc_4: 0.9809 - dense_2_acc_5: 0.2614 - dense_2_acc_6: 0.0402 - dense_2_acc_7: 0.9238 - dense_2_acc_8: 0.2260 - dense_2_acc_9: 0.11 - ETA: 24s - loss: 17.3709 - dense_2_loss: 2.6080 - dense_2_acc: 0.4309 - dense_2_acc_1: 0.6262 - dense_2_acc_2: 0.2779 - dense_2_acc_3: 0.0641 - dense_2_acc_4: 0.9812 - dense_2_acc_5: 0.2698 - dense_2_acc_6: 0.0438 - dense_2_acc_7: 0.9250 - dense_2_acc_8: 0.2294 - dense_2_acc_9: 0.11 - ETA: 23s - loss: 17.2727 - dense_2_loss: 2.6021 - dense_2_acc: 0.4385 - dense_2_acc_1: 0.6304 - dense_2_acc_2: 0.2804 - dense_2_acc_3: 0.0646 - dense_2_acc_4: 0.9815 - dense_2_acc_5: 0.2773 - dense_2_acc_6: 0.0464 - dense_2_acc_7: 0.9261 - dense_2_acc_8: 0.2315 - dense_2_acc_9: 0.11 - ETA: 22s - loss: 17.1750 - dense_2_loss: 2.5982 - dense_2_acc: 0.4443 - dense_2_acc_1: 0.6347 - dense_2_acc_2: 0.2832 - dense_2_acc_3: 0.0657 - dense_2_acc_4: 0.9818 - dense_2_acc_5: 0.2840 - dense_2_acc_6: 0.0491 - dense_2_acc_7: 0.9272 - dense_2_acc_8: 0.2326 - dense_2_acc_9: 0.11 - ETA: 21s - loss: 17.0829 - dense_2_loss: 2.5942 - dense_2_acc: 0.4486 - dense_2_acc_1: 0.6383 - dense_2_acc_2: 0.2857 - dense_2_acc_3: 0.0662 - dense_2_acc_4: 0.9820 - dense_2_acc_5: 0.2913 - dense_2_acc_6: 0.0504 - dense_2_acc_7: 0.9283 - dense_2_acc_8: 0.2354 - dense_2_acc_9: 0.11 - ETA: 20s - loss: 16.9904 - dense_2_loss: 2.5892 - dense_2_acc: 0.4544 - dense_2_acc_1: 0.6426 - dense_2_acc_2: 0.2889 - dense_2_acc_3: 0.0670 - dense_2_acc_4: 0.9823 - dense_2_acc_5: 0.2979 - dense_2_acc_6: 0.0511 - dense_2_acc_7: 0.9293 - dense_2_acc_8: 0.2381 - dense_2_acc_9: 0.11 - ETA: 19s - loss: 16.8972 - dense_2_loss: 2.5839 - dense_2_acc: 0.4603 - dense_2_acc_1: 0.6466 - dense_2_acc_2: 0.2901 - dense_2_acc_3: 0.0676 - dense_2_acc_4: 0.9825 - dense_2_acc_5: 0.3059 - dense_2_acc_6: 0.0541 - dense_2_acc_7: 0.9303 - dense_2_acc_8: 0.2431 - dense_2_acc_9: 0.11 - ETA: 18s - loss: 16.8050 - dense_2_loss: 2.5801 - dense_2_acc: 0.4664 - dense_2_acc_1: 0.6513 - dense_2_acc_2: 0.2925 - dense_2_acc_3: 0.0690 - dense_2_acc_4: 0.9828 - dense_2_acc_5: 0.3131 - dense_2_acc_6: 0.0578 - dense_2_acc_7: 0.9313 - dense_2_acc_8: 0.2457 - dense_2_acc_9: 0.11 - ETA: 18s - loss: 16.7156 - dense_2_loss: 2.5765 - dense_2_acc: 0.4719 - dense_2_acc_1: 0.6551 - dense_2_acc_2: 0.2942 - dense_2_acc_3: 0.0697 - dense_2_acc_4: 0.9830 - dense_2_acc_5: 0.3203 - dense_2_acc_6: 0.0603 - dense_2_acc_7: 0.9322 - dense_2_acc_8: 0.2479 - dense_2_acc_9: 0.11 - ETA: 17s - loss: 16.6263 - dense_2_loss: 2.5717 - dense_2_acc: 0.4773 - dense_2_acc_1: 0.6585 - dense_2_acc_2: 0.2947 - dense_2_acc_3: 0.0709 - dense_2_acc_4: 0.9832 - dense_2_acc_5: 0.3282 - dense_2_acc_6: 0.0623 - dense_2_acc_7: 0.9331 - dense_2_acc_8: 0.2508 - dense_2_acc_9: 0.11 - ETA: 16s - loss: 16.5363 - dense_2_loss: 2.5667 - dense_2_acc: 0.4831 - dense_2_acc_1: 0.6629 - dense_2_acc_2: 0.2981 - dense_2_acc_3: 0.0724 - dense_2_acc_4: 0.9835 - dense_2_acc_5: 0.3352 - dense_2_acc_6: 0.0644 - dense_2_acc_7: 0.9340 - dense_2_acc_8: 0.2532 - dense_2_acc_9: 0.11 - ETA: 15s - loss: 16.4502 - dense_2_loss: 2.5629 - dense_2_acc: 0.4891 - dense_2_acc_1: 0.6668 - dense_2_acc_2: 0.3000 - dense_2_acc_3: 0.0728 - dense_2_acc_4: 0.9837 - dense_2_acc_5: 0.3421 - dense_2_acc_6: 0.0668 - dense_2_acc_7: 0.9349 - dense_2_acc_8: 0.2571 - dense_2_acc_9: 0.11 - ETA: 14s - loss: 16.3647 - dense_2_loss: 2.5588 - dense_2_acc: 0.4948 - dense_2_acc_1: 0.6706 - dense_2_acc_2: 0.3016 - dense_2_acc_3: 0.0732 - dense_2_acc_4: 0.9839 - dense_2_acc_5: 0.3483 - dense_2_acc_6: 0.0700 - dense_2_acc_7: 0.9357 - dense_2_acc_8: 0.2590 - dense_2_acc_9: 0.11 - ETA: 14s - loss: 16.2836 - dense_2_loss: 2.5547 - dense_2_acc: 0.5004 - dense_2_acc_1: 0.6742 - dense_2_acc_2: 0.3029 - dense_2_acc_3: 0.0738 - dense_2_acc_4: 0.9841 - dense_2_acc_5: 0.3545 - dense_2_acc_6: 0.0723 - dense_2_acc_7: 0.9365 - dense_2_acc_8: 0.2603 - dense_2_acc_9: 0.11 - ETA: 13s - loss: 16.2021 - dense_2_loss: 2.5504 - dense_2_acc: 0.5058 - dense_2_acc_1: 0.6777 - dense_2_acc_2: 0.3047 - dense_2_acc_3: 0.0748 - dense_2_acc_4: 0.9843 - dense_2_acc_5: 0.3610 - dense_2_acc_6: 0.0743 - dense_2_acc_7: 0.9373 - dense_2_acc_8: 0.2633 - dense_2_acc_9: 0.11 - ETA: 12s - loss: 16.1254 - dense_2_loss: 2.5468 - dense_2_acc: 0.5113 - dense_2_acc_1: 0.6811 - dense_2_acc_2: 0.3076 - dense_2_acc_3: 0.0763 - dense_2_acc_4: 0.9845 - dense_2_acc_5: 0.3654 - dense_2_acc_6: 0.0757 - dense_2_acc_7: 0.9381 - dense_2_acc_8: 0.2664 - dense_2_acc_9: 0.12 - ETA: 12s - loss: 16.0473 - dense_2_loss: 2.5433 - dense_2_acc: 0.5172 - dense_2_acc_1: 0.6848 - dense_2_acc_2: 0.3101 - dense_2_acc_3: 0.0779 - dense_2_acc_4: 0.9847 - dense_2_acc_5: 0.3712 - dense_2_acc_6: 0.0769 - dense_2_acc_7: 0.9389 - dense_2_acc_8: 0.2691 - dense_2_acc_9: 0.12 - ETA: 11s - loss: 15.9667 - dense_2_loss: 2.5385 - dense_2_acc: 0.5228 - dense_2_acc_1: 0.6883 - dense_2_acc_2: 0.3124 - dense_2_acc_3: 0.0790 - dense_2_acc_4: 0.9849 - dense_2_acc_5: 0.3773 - dense_2_acc_6: 0.0796 - dense_2_acc_7: 0.9396 - dense_2_acc_8: 0.2724 - dense_2_acc_9: 0.12 - ETA: 10s - loss: 15.8941 - dense_2_loss: 2.5346 - dense_2_acc: 0.5277 - dense_2_acc_1: 0.6913 - dense_2_acc_2: 0.3140 - dense_2_acc_3: 0.0800 - dense_2_acc_4: 0.9851 - dense_2_acc_5: 0.3822 - dense_2_acc_6: 0.0817 - dense_2_acc_7: 0.9404 - dense_2_acc_8: 0.2727 - dense_2_acc_9: 0.12 - ETA: 9s - loss: 15.8199 - dense_2_loss: 2.5315 - dense_2_acc: 0.5329 - dense_2_acc_1: 0.6948 - dense_2_acc_2: 0.3154 - dense_2_acc_3: 0.0807 - dense_2_acc_4: 0.9852 - dense_2_acc_5: 0.3863 - dense_2_acc_6: 0.0840 - dense_2_acc_7: 0.9411 - dense_2_acc_8: 0.2746 - dense_2_acc_9: 0.1227 - ETA: 9s - loss: 15.7468 - dense_2_loss: 2.5278 - dense_2_acc: 0.5374 - dense_2_acc_1: 0.6978 - dense_2_acc_2: 0.3175 - dense_2_acc_3: 0.0818 - dense_2_acc_4: 0.9854 - dense_2_acc_5: 0.3925 - dense_2_acc_6: 0.0856 - dense_2_acc_7: 0.9418 - dense_2_acc_8: 0.2772 - dense_2_acc_9: 0.123 - ETA: 8s - loss: 15.6769 - dense_2_loss: 2.5247 - dense_2_acc: 0.5424 - dense_2_acc_1: 0.7006 - dense_2_acc_2: 0.3203 - dense_2_acc_3: 0.0826 - dense_2_acc_4: 0.9856 - dense_2_acc_5: 0.3983 - dense_2_acc_6: 0.0863 - dense_2_acc_7: 0.9424 - dense_2_acc_8: 0.2791 - dense_2_acc_9: 0.124110000/10000 [==============================] - ETA: 7s - loss: 15.6056 - dense_2_loss: 2.5214 - dense_2_acc: 0.5474 - dense_2_acc_1: 0.7037 - dense_2_acc_2: 0.3225 - dense_2_acc_3: 0.0845 - dense_2_acc_4: 0.9857 - dense_2_acc_5: 0.4041 - dense_2_acc_6: 0.0878 - dense_2_acc_7: 0.9431 - dense_2_acc_8: 0.2801 - dense_2_acc_9: 0.124 - ETA: 7s - loss: 15.5356 - dense_2_loss: 2.5176 - dense_2_acc: 0.5520 - dense_2_acc_1: 0.7066 - dense_2_acc_2: 0.3250 - dense_2_acc_3: 0.0855 - dense_2_acc_4: 0.9859 - dense_2_acc_5: 0.4085 - dense_2_acc_6: 0.0908 - dense_2_acc_7: 0.9438 - dense_2_acc_8: 0.2811 - dense_2_acc_9: 0.125 - ETA: 6s - loss: 15.4686 - dense_2_loss: 2.5153 - dense_2_acc: 0.5566 - dense_2_acc_1: 0.7094 - dense_2_acc_2: 0.3280 - dense_2_acc_3: 0.0862 - dense_2_acc_4: 0.9861 - dense_2_acc_5: 0.4135 - dense_2_acc_6: 0.0926 - dense_2_acc_7: 0.9444 - dense_2_acc_8: 0.2827 - dense_2_acc_9: 0.125 - ETA: 6s - loss: 15.4006 - dense_2_loss: 2.5121 - dense_2_acc: 0.5613 - dense_2_acc_1: 0.7122 - dense_2_acc_2: 0.3301 - dense_2_acc_3: 0.0868 - dense_2_acc_4: 0.9862 - dense_2_acc_5: 0.4178 - dense_2_acc_6: 0.0944 - dense_2_acc_7: 0.9450 - dense_2_acc_8: 0.2842 - dense_2_acc_9: 0.126 - ETA: 5s - loss: 15.3357 - dense_2_loss: 2.5098 - dense_2_acc: 0.5657 - dense_2_acc_1: 0.7152 - dense_2_acc_2: 0.3318 - dense_2_acc_3: 0.0869 - dense_2_acc_4: 0.9864 - dense_2_acc_5: 0.4227 - dense_2_acc_6: 0.0963 - dense_2_acc_7: 0.9456 - dense_2_acc_8: 0.2855 - dense_2_acc_9: 0.126 - ETA: 4s - loss: 15.2700 - dense_2_loss: 2.5066 - dense_2_acc: 0.5699 - dense_2_acc_1: 0.7177 - dense_2_acc_2: 0.3342 - dense_2_acc_3: 0.0875 - dense_2_acc_4: 0.9865 - dense_2_acc_5: 0.4282 - dense_2_acc_6: 0.0985 - dense_2_acc_7: 0.9462 - dense_2_acc_8: 0.2877 - dense_2_acc_9: 0.127 - ETA: 4s - loss: 15.2047 - dense_2_loss: 2.5040 - dense_2_acc: 0.5744 - dense_2_acc_1: 0.7206 - dense_2_acc_2: 0.3362 - dense_2_acc_3: 0.0887 - dense_2_acc_4: 0.9867 - dense_2_acc_5: 0.4325 - dense_2_acc_6: 0.1005 - dense_2_acc_7: 0.9468 - dense_2_acc_8: 0.2894 - dense_2_acc_9: 0.127 - ETA: 3s - loss: 15.1409 - dense_2_loss: 2.5010 - dense_2_acc: 0.5785 - dense_2_acc_1: 0.7233 - dense_2_acc_2: 0.3386 - dense_2_acc_3: 0.0905 - dense_2_acc_4: 0.9868 - dense_2_acc_5: 0.4376 - dense_2_acc_6: 0.1016 - dense_2_acc_7: 0.9473 - dense_2_acc_8: 0.2907 - dense_2_acc_9: 0.127 - ETA: 2s - loss: 15.0788 - dense_2_loss: 2.4982 - dense_2_acc: 0.5825 - dense_2_acc_1: 0.7258 - dense_2_acc_2: 0.3396 - dense_2_acc_3: 0.0924 - dense_2_acc_4: 0.9869 - dense_2_acc_5: 0.4428 - dense_2_acc_6: 0.1034 - dense_2_acc_7: 0.9479 - dense_2_acc_8: 0.2913 - dense_2_acc_9: 0.128 - ETA: 2s - loss: 15.0172 - dense_2_loss: 2.4953 - dense_2_acc: 0.5868 - dense_2_acc_1: 0.7284 - dense_2_acc_2: 0.3416 - dense_2_acc_3: 0.0931 - dense_2_acc_4: 0.9871 - dense_2_acc_5: 0.4477 - dense_2_acc_6: 0.1049 - dense_2_acc_7: 0.9484 - dense_2_acc_8: 0.2929 - dense_2_acc_9: 0.128 - ETA: 1s - loss: 14.9589 - dense_2_loss: 2.4922 - dense_2_acc: 0.5902 - dense_2_acc_1: 0.7304 - dense_2_acc_2: 0.3425 - dense_2_acc_3: 0.0939 - dense_2_acc_4: 0.9872 - dense_2_acc_5: 0.4527 - dense_2_acc_6: 0.1068 - dense_2_acc_7: 0.9490 - dense_2_acc_8: 0.2956 - dense_2_acc_9: 0.129 - ETA: 1s - loss: 14.9002 - dense_2_loss: 2.4889 - dense_2_acc: 0.5941 - dense_2_acc_1: 0.7331 - dense_2_acc_2: 0.3437 - dense_2_acc_3: 0.0951 - dense_2_acc_4: 0.9873 - dense_2_acc_5: 0.4574 - dense_2_acc_6: 0.1077 - dense_2_acc_7: 0.9495 - dense_2_acc_8: 0.2986 - dense_2_acc_9: 0.130 - ETA: 0s - loss: 14.8397 - dense_2_loss: 2.4853 - dense_2_acc: 0.5982 - dense_2_acc_1: 0.7357 - dense_2_acc_2: 0.3449 - dense_2_acc_3: 0.0959 - dense_2_acc_4: 0.9875 - dense_2_acc_5: 0.4622 - dense_2_acc_6: 0.1087 - dense_2_acc_7: 0.9500 - dense_2_acc_8: 0.3017 - dense_2_acc_9: 0.130 - 57s 6ms/step - loss: 14.7845 - dense_2_loss: 2.4831 - dense_2_acc: 0.6015 - dense_2_acc_1: 0.7375 - dense_2_acc_2: 0.3465 - dense_2_acc_3: 0.0964 - dense_2_acc_4: 0.9876 - dense_2_acc_5: 0.4672 - dense_2_acc_6: 0.1109 - dense_2_acc_7: 0.9505 - dense_2_acc_8: 0.3037 - dense_2_acc_9: 0.1310
<keras.callbacks.History at 0x22258007d30>
While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples:
We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.)
model.load_weights('models/model.h5')
You can now see the results on new examples.
EXAMPLES = ['3 May 1979', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001']
for example in EXAMPLES:
source = string_to_int(example, Tx, human_vocab)
source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), source))).swapaxes(0,1)
prediction = model.predict([source, s0, c0])
prediction = np.argmax(prediction, axis = -1)
output = [inv_machine_vocab[int(i)] for i in prediction]
print("source:", example)
print("output:", ''.join(output))
You can also change these examples to test with your own examples. The next part will give you a better sense on what the attention mechanism is doing–i.e., what part of the input the network is paying attention to when generating a particular output character.
3 - Visualizing Attention (Optional / Ungraded)
Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (say the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can visualize what part of the output is looking at what part of the input.
Consider the task of translating “Saturday 9 May 2018” to “2018-05-09”. If we visualize the computed α ⟨ t , t ′ ⟩ \alpha^{\langle t, t' \rangle} α⟨t,t′⟩ we get this:
Notice how the output ignores the “Saturday” portion of the input. None of the output timesteps are paying much attention to that portion of the input. We see also that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input’s “18” in order to generate “2018.”
3.1 - Getting the activations from the network
Lets now visualize the attention values in your network. We’ll propagate an example through the network, then visualize the values of α ⟨ t , t ′ ⟩ \alpha^{\langle t, t' \rangle} α⟨t,t′⟩.
To figure out where the attention values are located, let’s start by printing a summary of the model .
model.summary()
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_3 (InputLayer) (None, 30, 37) 0
__________________________________________________________________________________________________
s0 (InputLayer) (None, 128) 0
__________________________________________________________________________________________________
bidirectional_3 (Bidirectional) (None, 30, 128) 52224 input_3[0][0]
__________________________________________________________________________________________________
repeat_vector_1 (RepeatVector) (None, 30, 128) 0 s0[0][0]
lstm_1[0][0]
lstm_1[1][0]
lstm_1[2][0]
lstm_1[3][0]
lstm_1[4][0]
lstm_1[5][0]
lstm_1[6][0]
lstm_1[7][0]
lstm_1[8][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 30, 256) 0 bidirectional_3[0][0]
repeat_vector_1[2][0]
bidirectional_3[0][0]
repeat_vector_1[3][0]
bidirectional_3[0][0]
repeat_vector_1[4][0]
bidirectional_3[0][0]
repeat_vector_1[5][0]
bidirectional_3[0][0]
repeat_vector_1[6][0]
bidirectional_3[0][0]
repeat_vector_1[7][0]
bidirectional_3[0][0]
repeat_vector_1[8][0]
bidirectional_3[0][0]
repeat_vector_1[9][0]
bidirectional_3[0][0]
repeat_vector_1[10][0]
bidirectional_3[0][0]
repeat_vector_1[11][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 30, 1) 257 concatenate_1[2][0]
concatenate_1[3][0]
concatenate_1[4][0]
concatenate_1[5][0]
concatenate_1[6][0]
concatenate_1[7][0]
concatenate_1[8][0]
concatenate_1[9][0]
concatenate_1[10][0]
concatenate_1[11][0]
__________________________________________________________________________________________________
attention_weights (Activation) (None, 30, 1) 0 dense_1[1][0]
dense_1[2][0]
dense_1[3][0]
dense_1[4][0]
dense_1[5][0]
dense_1[6][0]
dense_1[7][0]
dense_1[8][0]
dense_1[9][0]
dense_1[10][0]
__________________________________________________________________________________________________
dot_1 (Dot) (None, 1, 128) 0 attention_weights[0][0]
bidirectional_3[0][0]
attention_weights[1][0]
bidirectional_3[0][0]
attention_weights[2][0]
bidirectional_3[0][0]
attention_weights[3][0]
bidirectional_3[0][0]
attention_weights[4][0]
bidirectional_3[0][0]
attention_weights[5][0]
bidirectional_3[0][0]
attention_weights[6][0]
bidirectional_3[0][0]
attention_weights[7][0]
bidirectional_3[0][0]
attention_weights[8][0]
bidirectional_3[0][0]
attention_weights[9][0]
bidirectional_3[0][0]
__________________________________________________________________________________________________
c0 (InputLayer) (None, 128) 0
__________________________________________________________________________________________________
lstm_1 (LSTM) [(None, 128), (None, 131584 dot_1[0][0]
s0[0][0]
c0[0][0]
dot_1[1][0]
lstm_1[0][0]
lstm_1[0][2]
dot_1[2][0]
lstm_1[1][0]
lstm_1[1][2]
dot_1[3][0]
lstm_1[2][0]
lstm_1[2][2]
dot_1[4][0]
lstm_1[3][0]
lstm_1[3][2]
dot_1[5][0]
lstm_1[4][0]
lstm_1[4][2]
dot_1[6][0]
lstm_1[5][0]
lstm_1[5][2]
dot_1[7][0]
lstm_1[6][0]
lstm_1[6][2]
dot_1[8][0]
lstm_1[7][0]
lstm_1[7][2]
dot_1[9][0]
lstm_1[8][0]
lstm_1[8][2]
__________________________________________________________________________________________________
dense_2 (Dense) (None, 11) 1419 lstm_1[0][0]
lstm_1[1][0]
lstm_1[2][0]
lstm_1[3][0]
lstm_1[4][0]
lstm_1[5][0]
lstm_1[6][0]
lstm_1[7][0]
lstm_1[8][0]
lstm_1[9][0]
==================================================================================================
Total params: 185,484
Trainable params: 185,484
Non-trainable params: 0
__________________________________________________________________________________________________
Navigate through the output of model.summary() above. You can see that the layer named attention_weights outputs the alphas of shape (m, 30, 1) before dot_2 computes the context vector for every time step
t
=
0
,
…
,
T
y
−
1
t = 0, \ldots, T_y-1
t=0,…,Ty−1. Lets get the activations from this layer.
The function attention_map() pulls out the attention values from your model and plots them.
attention_map = plot_attention_map(model, human_vocab, inv_machine_vocab, "Tuesday April 08 1993", num = 6, n_s = 128)
<Figure size 432x288 with 0 Axes>
On the generated plot you can observe the values of the attention weights for each character of the predicted output. Examine this plot and check that where the network is paying attention makes sense to you.
In the date translation application, you will observe that most of the time attention helps predict the year, and hasn’t much impact on predicting the day/month.
Congratulations!
You have come to the end of this assignment
Here’s what you should remember from this notebook:
- Machine translation models can be used to map from one sequence to another. They are useful not just for translating human languages (like French->English) but also for tasks like date format translation.
- An attention mechanism allows a network to focus on the most relevant parts of the input when producing a specific part of the output.
- A network using an attention mechanism can translate from inputs of length T x T_x Tx to outputs of length T y T_y Ty, where T x T_x Tx and T y T_y Ty can be different.
- You can visualize attention weights α ⟨ t , t ′ ⟩ \alpha^{\langle t,t' \rangle} α⟨t,t′⟩ to see what the network is paying attention to while generating each output.
Congratulations on finishing this assignment! You are now able to implement an attention model and use it to learn complex mappings from one sequence to another.
扫一扫
56
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
