今天将的还是cs224d 的problem set2 的第三部分习题，
原来国外大学的系统难度真的如此之大，相比之下还是默默地再天朝继续搬砖吧
下面讲述一下RNN语言建模的数学公式:
给出一串连续的词x1,x2...xt关于预测其后面紧跟的词xt+1的建模方式是:
vj是词库中的某个词。实现一个循环神经网络，此网络利用隐层中的反馈信息对"历史记录"x1,x2...xt进行建模:
$h^{(0)}=h_{0}\epsilon R^{D_{h}}$是隐藏层的初始化向量
$x^{(t)}L$是以$x^{(t)}$one-hot行向量与嵌入矩阵L的乘积
这个one-hot行向量就是当前处理词汇的索引
是词嵌入矩阵,
$L$是词嵌入矩阵
$I$是输入词表征矩阵
$H$是隐藏转换矩阵
$U$是输出词表征矩阵
$b_{1}$ $b_{2}$是偏置值
$d$是词嵌入的维数
|V|代表词库的规模
$D_{h}$是隐层的维数
输出向量
是面向整个词库的概率分布，我们需要最优化交叉熵（非正则化的）的损失率：
使用困惑度来评估语言模型的性能，其定义形式如下：
梯度:
该模型中各个变量进行最优化迭代的时候的梯度如下所示:
初始化所有的上面这些需要训练的参数的值
然后通过对每一个词进行训练，安装上述公司求出每个参数的导数值
然后使用梯度下降方法对其进行更新
将新得到的参数代入到模型中，如果损失的值小于初始设定的值则停止迭代，否则继续进行迭代
下面是一张RNNLM的结构图
上面这张是第二层RNN节点的结构图
上面这张是在RNN的变量上面应用Dropout的结构，降低模型过拟合的误差，第一层RNN的dropout结构
上面这张是第一层RNN的结构图
（注意前方高能，一大批天书即将来袭）
''' Created on 2017年9月26日 @author: weizhen ''' import getpass import sys import time import numpy as np from copy import deepcopy from utils import calculate_perplexity, get_ptb_dataset, Vocab from utils import ptb_iterator, sample import tensorflow as tf from model import LanguageModel from tensorflow.contrib.legacy_seq2seq.python.ops.seq2seq import sequence_loss class Config(object): """储存超参数和数据信息""" batch_size = 64 embed_size = 50 hidden_size = 100 num_steps = 10 max_epochs = 16 early_stopping = 2 dropout = 0.9 lr = 0.001 class RNNLM_Model(LanguageModel): def load_data(self, debug=False): """加载词向量并且训练 train/dev/test 数据""" self.vocab = Vocab() self.vocab.construct(get_ptb_dataset('train')) self.encoded_train = np.array([self.vocab.encode(word) for word in get_ptb_dataset('train')], dtype=np.int32) self.encoded_valid = np.array([self.vocab.encode(word) for word in get_ptb_dataset('valid')], dtype=np.int32) self.encoded_test = np.array([self.vocab.encode(word) for word in get_ptb_dataset('test')]) if debug: num_debug = 1024 self.encoded_train = self.encoded_train[:num_debug] self.encoded_valid = self.encoded_valid[:num_debug] self.encoded_test = self.encoded_test[:num_debug] def add_placeholders(self): """生成placeholder 变量来表示输入的 tensors 这些placeholder 被用来在模型的其他地方被填充 并且在训练的过程中会被填充 input_placeholder:Input placeholder shape (None,num_steps),type tf.int32 labels_placeholder:label placeholder shape (None,num_steps) type tf.float32 dropout_placeholder:dropput value placeholder (scalar), type tf.float32 """ self.input_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Input') self.labels_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Target') self.dropout_placeholder = tf.placeholder(tf.float32, name='Dropout') def add_embedding(self): """添加词嵌入层 Hint : 这一层应该用input_placeholder 来索引词嵌入 Hint : 你或许能发现tf.nn.embedding_lookup 是有用的 Hint : 你或许能发现tf.split , tf.squeeze 是有用的在构造tensor 的输入的时候 Hint : 下面是你需要创建的变量的维度 L:(len(self.vocab),embed_size) Returns: inputs:一个训练次数的列表,每一个元素应该是 一个张量 大小是 (batch_size,embed_size) tf.split(dimension,num_split,input) dimension表示输入张量的哪一个维度, 如果是0就表示对第0维度进行切割, num_split就是切割的数量, 如果是2就表示输入张量被切成2份, 每一份是一个列表 tf.squeeze(input,squeeze_dims=None,name=None) 从tensor中删除所有大小是1的维度 example: t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t))==>[2,3] t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t,[2,4]))==>[1,2,3,1] tf.nn.embedding_lookup 将词的索引映射到词的向量 """ with tf.device('/cpu:0'): embedding = tf.get_variable('Embedding', [len(self.vocab), self.config.embed_size], trainable=True) inputs = tf.nn.embedding_lookup(embedding, self.input_placeholder) inputs = [tf.squeeze(x, [1]) for x in tf.split(inputs, self.config.num_steps, 1)] return inputs def add_projection(self, rnn_outputs): """添加一个投影层 投影层将隐藏层的表示变换到整个词向量上的分布式表示 Hint:下面是你需要去创建的维度 U(hidden_size,len(vocab)) b_2:(len(vocab),) 参数: rnn_outputs:一个训练次数的列表，每一个元素应该是一个张量 大小是(batch_size,embed_size) Returns: outputs:一个长度的列表，每一个元素是一个张量(batch_size,len(vocab)) """ with tf.variable_scope('Projection'): U = tf.get_variable('Matrix', [self.config.hidden_size, len(self.vocab)]) proj_b = tf.get_variable('Bias', [len(self.vocab)]) outputs = [tf.matmul(o, U) + proj_b for o in rnn_outputs] return outputs def add_loss_op(self, output): """将损失添加到目标函数上面 Hint:使用tensorflow.python.ops.seq2seq.sequence_loss 来实现序列损失 参数: 输出:一个张量 大小是 (None,self.vocab) 返回: 损失：一个0-d大小的张量 """ all_ones = [tf.ones([self.config.batch_size * self.config.num_steps])] cross_entropy = sequence_loss([output], [tf.reshape(self.labels_placeholder, [-1])], all_ones, len(self.vocab)) tf.add_to_collection('total_loss', cross_entropy) loss = tf.add_n(tf.get_collection('total_loss')) return loss def add_training_op(self, loss): """将目标损失添加到计算图上 创建一个优化器并且应用梯度下降到所有的训练变量上面 Hint:使用tf.train.AdamOptimizer 对于这个模型 使用optimizer.minimize() 会返回一个train_op的对象 参数: loss: 损失张量，来自于cross_entropy_loss 交叉熵损失 返回: train_op:训练的目标 """ with tf.variable_scope("Optimizer") as scope: train_op = tf.train.AdamOptimizer(self.config.lr).minimize(loss) return train_op def __init__(self, config): self.config = config self.load_data(debug=False) self.add_placeholders() self.inputs = self.add_embedding() self.rnn_outputs = self.add_model(self.inputs) self.outputs = self.add_projection(self.rnn_outputs) # 我们想去检验下一个词预测得多好 # 我们把o转变成float64 位 因为如果不这样就会有数值问题 # sum(output of softmax) = 1.00000298179 并且不是 1 self.predictions = [tf.nn.softmax(tf.cast(o, 'float64')) for o in self.outputs] # 将输出值转变成 len(vocab) 的大小 output = tf.reshape(tf.concat(self.outputs, 1), [-1, len(self.vocab)]) self.calculate_loss = self.add_loss_op(output) self.train_step = self.add_training_op(self.calculate_loss) def add_model(self, inputs): """创建RNN LM 模型 在下面的实现里面你需要去实现RNN LM 模型的等式 Hint: 使用一个零向量 大小是 (batch_size,hidden_size) 作为初始的RNN的状态 Hint: 将最后RNN输出 作为实例变量 self.final_state Hint : 确保将dropout应用到 输入和输出的 变量上面 Hint : 使用变量域 RNN 来定义 RNN变量 Hint : 表现一个明显的 for-loop 在输入上面 你可以使用scope.reuse_variable() 来确定权重 在每一次迭代都是相同的 确保不会在第一次循环的时候调用这个，因为没有变量会被初始化 Hint : 下面变量的不同的维度 ， 你需要去创建的 H: (hidden_size,hidden_size) I: (embed_size,hidden_size) b_1:(hidden_size,) Args: inputs:一个记录num_steps的列表，里边的每一个元素应该是一个张量 大小是(batch_size,embed_size)的大小 Returns:返回 outputs:一个记录num_steps的列表，里面每一个元素应该是一个张量 大小是(batch_size,hidden_size) """ with tf.variable_scope('InputDropout'): inputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in inputs] with tf.variable_scope('RNN') as scope: self.initial_state = tf.zeros([self.config.batch_size, self.config.hidden_size]) state = self.initial_state rnn_outputs = [] for tstep, current_input in enumerate(inputs): if tstep > 0: scope.reuse_variables() RNN_H = tf.get_variable('HMatrix', [self.config.hidden_size, self.config.hidden_size]) RNN_I = tf.get_variable('IMatrix', [self.config.embed_size, self.config.hidden_size]) RNN_b = tf.get_variable('B', [self.config.hidden_size]) state = tf.nn.sigmoid(tf.matmul(state, RNN_H) + tf.matmul(current_input, RNN_I) + RNN_b) rnn_outputs.append(state) self.final_state = rnn_outputs[-1] with tf.variable_scope('RNNDropout'): rnn_outputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in rnn_outputs] return rnn_outputs def run_epoch(self, session, data, train_op=None, verbose=10): config = self.config dp = config.dropout if not train_op: train_op = tf.no_op() dp = 1 total_steps = sum(1 for x in ptb_iterator(data, config.batch_size, config.num_steps)) total_loss = [] state = self.initial_state.eval() for step, (x, y) in enumerate(ptb_iterator(data, config.batch_size, config.num_steps)): # 我们需要通过初始状态，并且从最终状态中抽取数据来进行填充 # RNN 合适的 历史 feed = {self.input_placeholder: x, self.labels_placeholder: y, self.initial_state: state, self.dropout_placeholder: dp } loss, state, _ = session.run([self.calculate_loss, self.final_state, train_op], feed_dict=feed) total_loss.append(loss) if verbose and step % verbose == 0: sys.stdout.write('\r{} / {} : pp = {} '.format(step, total_steps, np.exp(np.mean(total_loss)))) sys.stdout.flush() if verbose: sys.stdout.write('\r') return np.exp(np.mean(total_loss)) def generate_text(session, model, config, starting_text='<eos>', stop_length=100, stop_tokens=None, temp=1.0): """从模型自动生成文字 Hint:创建一个feed-dictionary 并且使用sess.run()方法去执行这个模型 你会需要使用model.initial_state 作为一个键传递给feed_dict Hint:得到model.final_state 和 model.predictions[-1]. 在add_model()方法中设置model.final_state 。 model.predictions 是在 __init__方法中设置的 Hint:在模型的训练中存储输出的参数值，和预测的y_pred的值 参数: Args: session : tf.Session() object model : Object of type RNNLM Model config : A Config() object starting_text:Initial text passed to model Returns: output : List of word idxs """ state = model.initial_state.eval() # Imagine tokens as a batch size of one, length of len(tokens[0]) tokens = [model.vocab.encode(word) for word in starting_text.split()] for i in range(stop_length): feed = {model.input_placeholder: [tokens[-1:]], model.initial_state: state, model.dropout_placeholder: 1} state, y_pred = session.run([model.final_state, model.predictions[-1]], feed_dict=feed) next_word_idx = sample(y_pred[0], temperature=temp) tokens.append(next_word_idx) if stop_tokens and model.vocab.decode(tokens[-1]) in stop_tokens: break output = [model.vocab.decode(word_idx) for word_idx in tokens] return output def generate_sentence(session, model, config, *args, **kwargs): """方便从模型来生成句子""" return generate_text(session, model, config, *args, stop_tokens=['<eos>'], **kwargs) def test_RNNLM(): config = Config() gen_config = deepcopy(config) gen_config.batch_size = gen_config.num_steps = 1 # 创建训练模型，并且生成模型 with tf.variable_scope('RNNLM',reuse=None) as scope: model = RNNLM_Model(config) # 这个指示gen_model来重新使用相同的变量作为以上的模型 scope.reuse_variables() gen_model = RNNLM_Model(gen_config) init = tf.global_variables_initializer() saver = tf.train.Saver() with tf.Session() as session: best_val_pp = float('inf') best_val_epoch = 0 session.run(init) for epoch in range(config.max_epochs): print('Epoch {0}'.format(epoch)) start = time.time() train_pp = model.run_epoch(session, model.encoded_train, train_op=model.train_step) valid_pp = model.run_epoch(session, model.encoded_valid) print('Training perplexity: {0}'.format(train_pp)) print('Validation perplexity:{0}'.format(valid_pp)) if valid_pp < best_val_pp: best_val_pp = valid_pp best_val_epoch = epoch saver.save(session, './ptb_rnnlm.weights') if epoch - best_val_epoch > config.early_stopping: break print('Total time : {0}'.format(time.time() - start)) saver.restore(session, 'ptb_rnnlm.weights') test_pp = model.run_epoch(session, model.encoded_test) print('=-=' * 5) print('Test perplexity: {0} '.format(test_pp)) print('=-=' * 5) starting_text = 'in palo alto' while starting_text: print(' '.join(generate_sentence(session, gen_model, gen_config, starting_text=starting_text, temp=1.0))) #starting_text = raw_input('>') if __name__ == "__main__": test_RNNLM()
（其实也不算是天书啦，比高数简单多啦，比数学分析那是简单了好几十万倍了呀）
下面是训练的Log
1380 / 1452 : pp = 266.20892333984375 1390 / 1452 : pp = 265.94439697265625 1400 / 1452 : pp = 265.66845703125 1410 / 1452 : pp = 265.5393981933594 1420 / 1452 : pp = 265.32489013671875 1430 / 1452 : pp = 265.2019348144531 1440 / 1452 : pp = 265.13720703125 1450 / 1452 : pp = 264.954833984375 0 / 115 : pp = 296.9217224121094 10 / 115 : pp = 282.02130126953125 20 / 115 : pp = 279.76800537109375 30 / 115 : pp = 276.4101257324219 40 / 115 : pp = 276.2939147949219 50 / 115 : pp = 270.73565673828125 60 / 115 : pp = 269.88134765625 70 / 115 : pp = 266.8675231933594 80 / 115 : pp = 263.6731872558594 90 / 115 : pp = 260.8569030761719 100 / 115 : pp = 256.3356628417969 110 / 115 : pp = 255.1026611328125 Training perplexity: 264.9092102050781 Validation perplexity:254.84902954101562 Total time : 41.65332388877869 Epoch 3 0 / 1452 : pp = 327.0847473144531 10 / 1452 : pp = 273.9620056152344 20 / 1452 : pp = 270.22943115234375 30 / 1452 : pp = 263.5213317871094 40 / 1452 : pp = 264.0644836425781 50 / 1452 : pp = 258.6029968261719 60 / 1452 : pp = 257.04290771484375 70 / 1452 : pp = 257.59161376953125 80 / 1452 : pp = 256.7600402832031 90 / 1452 : pp = 254.5120391845703 100 / 1452 : pp = 252.44725036621094 110 / 1452 : pp = 250.13954162597656 120 / 1452 : pp = 249.91647338867188 130 / 1452 : pp = 249.50460815429688 140 / 1452 : pp = 247.67440795898438 150 / 1452 : pp = 247.19090270996094 160 / 1452 : pp = 247.8919219970703 170 / 1452 : pp = 247.54322814941406 180 / 1452 : pp = 246.17623901367188 190 / 1452 : pp = 245.78330993652344 200 / 1452 : pp = 246.80552673339844 210 / 1452 : pp = 246.3059844970703 220 / 1452 : pp = 246.19021606445312 230 / 1452 : pp = 246.70140075683594 240 / 1452 : pp = 246.3099822998047 250 / 1452 : pp = 245.1745147705078 260 / 1452 : pp = 244.17384338378906 270 / 1452 : pp = 242.57363891601562 280 / 1452 : pp = 242.8500213623047 290 / 1452 : pp = 243.0492706298828 300 / 1452 : pp = 243.1466522216797 310 / 1452 : pp = 242.89044189453125 320 / 1452 : pp = 243.08045959472656 330 / 1452 : pp = 243.32235717773438 340 / 1452 : pp = 242.34715270996094 350 / 1452 : pp = 242.80972290039062 360 / 1452 : pp = 242.5345458984375 370 / 1452 : pp = 242.0083465576172 380 / 1452 : pp = 241.22708129882812 390 / 1452 : pp = 241.24398803710938 400 / 1452 : pp = 240.63473510742188 410 / 1452 : pp = 240.94094848632812 420 / 1452 : pp = 241.19717407226562 430 / 1452 : pp = 240.8896026611328 440 / 1452 : pp = 240.7772979736328 450 / 1452 : pp = 240.45913696289062 460 / 1452 : pp = 240.06674194335938 470 / 1452 : pp = 239.42198181152344 480 / 1452 : pp = 238.39271545410156 490 / 1452 : pp = 238.0517120361328 500 / 1452 : pp = 237.31752014160156 510 / 1452 : pp = 237.1197967529297 520 / 1452 : pp = 236.64865112304688 530 / 1452 : pp = 236.004638671875 540 / 1452 : pp = 235.192626953125 550 / 1452 : pp = 234.6700439453125 560 / 1452 : pp = 234.1914825439453 570 / 1452 : pp = 233.80899047851562 580 / 1452 : pp = 233.3753662109375 590 / 1452 : pp = 232.8699188232422 600 / 1452 : pp = 232.2629852294922 610 / 1452 : pp = 231.8668212890625 620 / 1452 : pp = 231.478515625 630 / 1452 : pp = 231.0444793701172 640 / 1452 : pp = 231.2737579345703 650 / 1452 : pp = 231.28114318847656 660 / 1452 : pp = 231.4324951171875 670 / 1452 : pp = 231.48513793945312 680 / 1452 : pp = 231.45932006835938 690 / 1452 : pp = 231.17738342285156 700 / 1452 : pp = 231.00570678710938 710 / 1452 : pp = 231.03810119628906 720 / 1452 : pp = 230.96131896972656 730 / 1452 : pp = 230.91110229492188 740 / 1452 : pp = 231.13539123535156 750 / 1452 : pp = 231.04393005371094 760 / 1452 : pp = 231.03489685058594 770 / 1452 : pp = 231.19744873046875 780 / 1452 : pp = 231.26625061035156 790 / 1452 : pp = 231.38714599609375 800 / 1452 : pp = 231.24441528320312 810 / 1452 : pp = 231.16824340820312 820 / 1452 : pp = 231.11831665039062 830 / 1452 : pp = 231.34886169433594 840 / 1452 : pp = 231.221923828125 850 / 1452 : pp = 231.2562255859375 860 / 1452 : pp = 231.26492309570312 870 / 1452 : pp = 231.1961212158203 880 / 1452 : pp = 231.30506896972656 890 / 1452 : pp = 231.24728393554688 900 / 1452 : pp = 231.15744018554688 910 / 1452 : pp = 231.20175170898438 920 / 1452 : pp = 231.25534057617188 930 / 1452 : pp = 231.09461975097656 940 / 1452 : pp = 231.12612915039062 950 / 1452 : pp = 231.0475616455078 960 / 1452 : pp = 230.86056518554688 970 / 1452 : pp = 230.80377197265625 980 / 1452 : pp = 230.4598846435547 990 / 1452 : pp = 230.24559020996094 1000 / 1452 : pp = 229.91030883789062 1010 / 1452 : pp = 229.9349822998047 1020 / 1452 : pp = 230.01470947265625 1030 / 1452 : pp = 229.8909149169922 1040 / 1452 : pp = 229.9403533935547 1050 / 1452 : pp = 229.84815979003906 1060 / 1452 : pp = 229.60377502441406 1070 / 1452 : pp = 229.74647521972656 1080 / 1452 : pp = 229.80410766601562 1090 / 1452 : pp = 229.78733825683594 1100 / 1452 : pp = 229.64549255371094 1110 / 1452 : pp = 229.26255798339844 1120 / 1452 : pp = 229.00262451171875 1130 / 1452 : pp = 228.6716766357422 1140 / 1452 : pp = 228.55067443847656 1150 / 1452 : pp = 228.61563110351562 1160 / 1452 : pp = 228.50958251953125 1170 / 1452 : pp = 228.3498992919922 1180 / 1452 : pp = 228.29786682128906 1190 / 1452 : pp = 228.33204650878906 1200 / 1452 : pp = 228.27369689941406 1210 / 1452 : pp = 228.11831665039062 1220 / 1452 : pp = 228.21775817871094 1230 / 1452 : pp = 228.3170166015625 1240 / 1452 : pp = 228.22134399414062 1250 / 1452 : pp = 228.3769073486328 1260 / 1452 : pp = 228.37527465820312 1270 / 1452 : pp = 228.33694458007812 1280 / 1452 : pp = 228.27108764648438 1290 / 1452 : pp = 228.1731414794922 1300 / 1452 : pp = 228.12200927734375 1310 / 1452 : pp = 228.10275268554688 1320 / 1452 : pp = 227.9289093017578 1330 / 1452 : pp = 227.77723693847656 1340 / 1452 : pp = 227.79623413085938 1350 / 1452 : pp = 227.7408447265625 1360 / 1452 : pp = 227.72586059570312 1370 / 1452 : pp = 227.49728393554688 1380 / 1452 : pp = 227.37940979003906 1390 / 1452 : pp = 227.20166015625 1400 / 1452 : pp = 227.018310546875 1410 / 1452 : pp = 226.95651245117188 1420 / 1452 : pp = 226.8065643310547 1430 / 1452 : pp = 226.7261199951172 1440 / 1452 : pp = 226.7193145751953 1450 / 1452 : pp = 226.61068725585938 0 / 115 : pp = 269.342041015625 10 / 115 : pp = 255.03016662597656 20 / 115 : pp = 253.8992919921875 30 / 115 : pp = 251.04025268554688 40 / 115 : pp = 250.51756286621094 50 / 115 : pp = 245.3595428466797 60 / 115 : pp = 244.4713897705078 70 / 115 : pp = 241.2674560546875 80 / 115 : pp = 238.3473663330078 90 / 115 : pp = 235.56423950195312 100 / 115 : pp = 231.2281036376953 110 / 115 : pp = 229.8423614501953 Training perplexity: 226.5760040283203 Validation perplexity:229.59939575195312 Total time : 42.202677726745605 Epoch 4 0 / 1452 : pp = 282.2423095703125 10 / 1452 : pp = 240.16258239746094 20 / 1452 : pp = 236.12203979492188 30 / 1452 : pp = 230.3953857421875 40 / 1452 : pp = 231.8789825439453 50 / 1452 : pp = 227.26612854003906 60 / 1452 : pp = 226.22061157226562 70 / 1452 : pp = 227.01885986328125 80 / 1452 : pp = 226.2459716796875 90 / 1452 : pp = 224.3211669921875 100 / 1452 : pp = 222.65615844726562 110 / 1452 : pp = 220.70326232910156 120 / 1452 : pp = 220.42288208007812 130 / 1452 : pp = 219.8100128173828 140 / 1452 : pp = 218.04432678222656 150 / 1452 : pp = 217.31639099121094 160 / 1452 : pp = 217.86349487304688 170 / 1452 : pp = 217.46597290039062 180 / 1452 : pp = 216.3349151611328 190 / 1452 : pp = 216.12240600585938 200 / 1452 : pp = 216.97842407226562 210 / 1452 : pp = 216.51014709472656 220 / 1452 : pp = 216.46751403808594 230 / 1452 : pp = 216.80126953125 240 / 1452 : pp = 216.45965576171875 250 / 1452 : pp = 215.5008544921875 260 / 1452 : pp = 214.62210083007812 270 / 1452 : pp = 213.29183959960938 280 / 1452 : pp = 213.5621337890625 290 / 1452 : pp = 213.80657958984375 300 / 1452 : pp = 213.8963165283203 310 / 1452 : pp = 213.60653686523438 320 / 1452 : pp = 213.85877990722656 330 / 1452 : pp = 214.07345581054688 340 / 1452 : pp = 213.25421142578125 350 / 1452 : pp = 213.68019104003906 360 / 1452 : pp = 213.41717529296875 370 / 1452 : pp = 213.04920959472656 380 / 1452 : pp = 212.39019775390625 390 / 1452 : pp = 212.4908905029297 400 / 1452 : pp = 212.01914978027344 410 / 1452 : pp = 212.36903381347656 420 / 1452 : pp = 212.6802520751953 430 / 1452 : pp = 212.42697143554688 440 / 1452 : pp = 212.42990112304688 450 / 1452 : pp = 212.14524841308594 460 / 1452 : pp = 211.7836151123047 470 / 1452 : pp = 211.17282104492188 480 / 1452 : pp = 210.27903747558594 490 / 1452 : pp = 209.95211791992188 500 / 1452 : pp = 209.28302001953125 510 / 1452 : pp = 209.1029815673828 520 / 1452 : pp = 208.73855590820312 530 / 1452 : pp = 208.19700622558594 540 / 1452 : pp = 207.4554443359375 550 / 1452 : pp = 207.0062255859375 560 / 1452 : pp = 206.59739685058594 570 / 1452 : pp = 206.27874755859375 580 / 1452 : pp = 205.87144470214844 590 / 1452 : pp = 205.43545532226562 600 / 1452 : pp = 204.90940856933594 610 / 1452 : pp = 204.5686798095703 620 / 1452 : pp = 204.22862243652344 630 / 1452 : pp = 203.8448028564453 640 / 1452 : pp = 204.06576538085938 650 / 1452 : pp = 204.0941925048828 660 / 1452 : pp = 204.22103881835938 670 / 1452 : pp = 204.289794921875 680 / 1452 : pp = 204.3115234375 690 / 1452 : pp = 204.10284423828125 700 / 1452 : pp = 203.99757385253906 710 / 1452 : pp = 204.04971313476562 720 / 1452 : pp = 204.03152465820312 730 / 1452 : pp = 203.99046325683594 740 / 1452 : pp = 204.19786071777344 750 / 1452 : pp = 204.1642608642578 760 / 1452 : pp = 204.19435119628906 770 / 1452 : pp = 204.37786865234375 780 / 1452 : pp = 204.4965057373047 790 / 1452 : pp = 204.6479034423828 800 / 1452 : pp = 204.56117248535156 810 / 1452 : pp = 204.52284240722656 820 / 1452 : pp = 204.50978088378906 830 / 1452 : pp = 204.7531280517578 840 / 1452 : pp = 204.64468383789062 850 / 1452 : pp = 204.71348571777344 860 / 1452 : pp = 204.7399444580078 870 / 1452 : pp = 204.69406127929688 880 / 1452 : pp = 204.7965850830078 890 / 1452 : pp = 204.7594757080078 900 / 1452 : pp = 204.71446228027344 910 / 1452 : pp = 204.7590789794922 920 / 1452 : pp = 204.85772705078125 930 / 1452 : pp = 204.7428741455078 940 / 1452 : pp = 204.8068389892578 950 / 1452 : pp = 204.75791931152344 960 / 1452 : pp = 204.63815307617188 970 / 1452 : pp = 204.60760498046875 980 / 1452 : pp = 204.34347534179688 990 / 1452 : pp = 204.151611328125 1000 / 1452 : pp = 203.8665771484375 1010 / 1452 : pp = 203.9164581298828 1020 / 1452 : pp = 204.0184783935547 1030 / 1452 : pp = 203.95166015625 1040 / 1452 : pp = 204.03045654296875 1050 / 1452 : pp = 203.95846557617188 1060 / 1452 : pp = 203.77114868164062 1070 / 1452 : pp = 203.93260192871094 1080 / 1452 : pp = 204.00048828125 1090 / 1452 : pp = 204.00233459472656 1100 / 1452 : pp = 203.8960418701172 1110 / 1452 : pp = 203.5987548828125 1120 / 1452 : pp = 203.38392639160156 1130 / 1452 : pp = 203.08872985839844 1140 / 1452 : pp = 203.01272583007812 1150 / 1452 : pp = 203.0865936279297 1160 / 1452 : pp = 203.02308654785156 1170 / 1452 : pp = 202.9125518798828 1180 / 1452 : pp = 202.9097442626953 1190 / 1452 : pp = 202.98252868652344 1200 / 1452 : pp = 202.95387268066406 1210 / 1452 : pp = 202.851318359375 1220 / 1452 : pp = 202.97671508789062 1230 / 1452 : pp = 203.1051025390625 1240 / 1452 : pp = 203.0526123046875 1250 / 1452 : pp = 203.21417236328125 1260 / 1452 : pp = 203.23617553710938 1270 / 1452 : pp = 203.22802734375 1280 / 1452 : pp = 203.20846557617188 1290 / 1452 : pp = 203.15362548828125 1300 / 1452 : pp = 203.14315795898438 1310 / 1452 : pp = 203.15264892578125 1320 / 1452 : pp = 203.02801513671875 1330 / 1452 : pp = 202.92977905273438 1340 / 1452 : pp = 202.95484924316406 1350 / 1452 : pp = 202.9335479736328 1360 / 1452 : pp = 202.955322265625 1370 / 1452 : pp = 202.7740478515625 1380 / 1452 : pp = 202.68569946289062 1390 / 1452 : pp = 202.55816650390625 1400 / 1452 : pp = 202.41651916503906 1410 / 1452 : pp = 202.38494873046875 1420 / 1452 : pp = 202.27593994140625 1430 / 1452 : pp = 202.21826171875 1440 / 1452 : pp = 202.23272705078125 1450 / 1452 : pp = 202.16099548339844 0 / 115 : pp = 253.23211669921875 10 / 115 : pp = 237.62506103515625 20 / 115 : pp = 237.60557556152344 30 / 115 : pp = 234.9273223876953 40 / 115 : pp = 234.30519104003906 50 / 115 : pp = 229.43960571289062 60 / 115 : pp = 228.6050567626953 70 / 115 : pp = 225.2646484375 80 / 115 : pp = 222.55935668945312 90 / 115 : pp = 219.83255004882812 100 / 115 : pp = 215.5491485595703 110 / 115 : pp = 214.07937622070312 Training perplexity: 202.1349639892578 Validation perplexity:213.85256958007812 Total time : 42.10724234580994 Epoch 5 0 / 1452 : pp = 255.92384338378906 10 / 1452 : pp = 219.5322265625 20 / 1452 : pp = 214.36212158203125 30 / 1452 : pp = 209.12620544433594 40 / 1452 : pp = 210.04193115234375 50 / 1452 : pp = 205.77398681640625 60 / 1452 : pp = 204.8201141357422 70 / 1452 : pp = 205.3955841064453 80 / 1452 : pp = 204.8386688232422 90 / 1452 : pp = 203.21194458007812 100 / 1452 : pp = 201.87643432617188 110 / 1452 : pp = 200.10122680664062 120 / 1452 : pp = 199.82012939453125 130 / 1452 : pp = 199.11192321777344 140 / 1452 : pp = 197.51919555664062 150 / 1452 : pp = 197.03567504882812 160 / 1452 : pp = 197.4231414794922 170 / 1452 : pp = 197.09571838378906 180 / 1452 : pp = 196.17665100097656 190 / 1452 : pp = 196.0064697265625 200 / 1452 : pp = 196.7347869873047 210 / 1452 : pp = 196.3063507080078 220 / 1452 : pp = 196.21388244628906 230 / 1452 : pp = 196.5252227783203 240 / 1452 : pp = 196.203125 250 / 1452 : pp = 195.3251953125 260 / 1452 : pp = 194.53335571289062 270 / 1452 : pp = 193.3546142578125 280 / 1452 : pp = 193.59420776367188 290 / 1452 : pp = 193.83297729492188 300 / 1452 : pp = 193.98489379882812 310 / 1452 : pp = 193.68414306640625 320 / 1452 : pp = 193.89065551757812 330 / 1452 : pp = 194.0518798828125 340 / 1452 : pp = 193.32888793945312 350 / 1452 : pp = 193.76219177246094 360 / 1452 : pp = 193.56106567382812 370 / 1452 : pp = 193.28179931640625 380 / 1452 : pp = 192.7037811279297 390 / 1452 : pp = 192.8145294189453 400 / 1452 : pp = 192.43325805664062 410 / 1452 : pp = 192.81527709960938 420 / 1452 : pp = 193.13760375976562 430 / 1452 : pp = 192.9148712158203 440 / 1452 : pp = 192.92526245117188 450 / 1452 : pp = 192.70083618164062 460 / 1452 : pp = 192.36647033691406 470 / 1452 : pp = 191.85394287109375 480 / 1452 : pp = 191.07244873046875 490 / 1452 : pp = 190.75401306152344 500 / 1452 : pp = 190.1843719482422 510 / 1452 : pp = 190.03334045410156 520 / 1452 : pp = 189.72938537597656 530 / 1452 : pp = 189.25889587402344 540 / 1452 : pp = 188.59315490722656 550 / 1452 : pp = 188.19313049316406 560 / 1452 : pp = 187.80621337890625 570 / 1452 : pp = 187.5229034423828 580 / 1452 : pp = 187.1091766357422 590 / 1452 : pp = 186.72592163085938 600 / 1452 : pp = 186.2238006591797 610 / 1452 : pp = 185.89695739746094 620 / 1452 : pp = 185.60989379882812 630 / 1452 : pp = 185.2689208984375 640 / 1452 : pp = 185.47567749023438 650 / 1452 : pp = 185.5127410888672 660 / 1452 : pp = 185.64627075195312 670 / 1452 : pp = 185.71311950683594 680 / 1452 : pp = 185.72569274902344 690 / 1452 : pp = 185.56459045410156 700 / 1452 : pp = 185.48681640625 710 / 1452 : pp = 185.5458221435547 720 / 1452 : pp = 185.5598907470703 730 / 1452 : pp = 185.5335235595703 740 / 1452 : pp = 185.73995971679688 750 / 1452 : pp = 185.744384765625 760 / 1452 : pp = 185.81268310546875 770 / 1452 : pp = 186.00088500976562 780 / 1452 : pp = 186.14443969726562 790 / 1452 : pp = 186.30764770507812 800 / 1452 : pp = 186.2595977783203 810 / 1452 : pp = 186.23028564453125 820 / 1452 : pp = 186.23997497558594 830 / 1452 : pp = 186.49057006835938 840 / 1452 : pp = 186.43331909179688 850 / 1452 : pp = 186.48887634277344 860 / 1452 : pp = 186.51502990722656 870 / 1452 : pp = 186.5167999267578 880 / 1452 : pp = 186.62400817871094 890 / 1452 : pp = 186.6103973388672 900 / 1452 : pp = 186.58111572265625 910 / 1452 : pp = 186.64126586914062 920 / 1452 : pp = 186.7366180419922 930 / 1452 : pp = 186.65719604492188 940 / 1452 : pp = 186.71755981445312 950 / 1452 : pp = 186.6977996826172 960 / 1452 : pp = 186.62774658203125 970 / 1452 : pp = 186.62115478515625 980 / 1452 : pp = 186.3773193359375 990 / 1452 : pp = 186.23109436035156 1000 / 1452 : pp = 185.99227905273438 1010 / 1452 : pp = 186.0488739013672 1020 / 1452 : pp = 186.1744384765625 1030 / 1452 : pp = 186.1162109375 1040 / 1452 : pp = 186.18899536132812 1050 / 1452 : pp = 186.1549072265625 1060 / 1452 : pp = 186.01419067382812 1070 / 1452 : pp = 186.17364501953125 1080 / 1452 : pp = 186.27061462402344 1090 / 1452 : pp = 186.28428649902344 1100 / 1452 : pp = 186.2150115966797 1110 / 1452 : pp = 185.95103454589844 1120 / 1452 : pp = 185.77423095703125 1130 / 1452 : pp = 185.5232696533203 1140 / 1452 : pp = 185.4607391357422 1150 / 1452 : pp = 185.56077575683594 1160 / 1452 : pp = 185.53343200683594 1170 / 1452 : pp = 185.46453857421875 1180 / 1452 : pp = 185.4741668701172 1190 / 1452 : pp = 185.5594482421875 1200 / 1452 : pp = 185.53785705566406 1210 / 1452 : pp = 185.4576416015625 1220 / 1452 : pp = 185.5943145751953 1230 / 1452 : pp = 185.7483673095703 1240 / 1452 : pp = 185.70762634277344 1250 / 1452 : pp = 185.8568115234375 1260 / 1452 : pp = 185.90635681152344 1270 / 1452 : pp = 185.8961639404297 1280 / 1452 : pp = 185.89199829101562 1290 / 1452 : pp = 185.85911560058594 1300 / 1452 : pp = 185.86097717285156 1310 / 1452 : pp = 185.88739013671875 1320 / 1452 : pp = 185.79248046875 1330 / 1452 : pp = 185.69700622558594 1340 / 1452 : pp = 185.7310028076172 1350 / 1452 : pp = 185.72613525390625 1360 / 1452 : pp = 185.76829528808594 1370 / 1452 : pp = 185.6322021484375 1380 / 1452 : pp = 185.56378173828125 1390 / 1452 : pp = 185.4654998779297 1400 / 1452 : pp = 185.35110473632812 1410 / 1452 : pp = 185.33917236328125 1420 / 1452 : pp = 185.2509002685547 1430 / 1452 : pp = 185.20436096191406 1440 / 1452 : pp = 185.2254638671875 1450 / 1452 : pp = 185.16542053222656 0 / 115 : pp = 242.26800537109375 10 / 115 : pp = 226.12258911132812 20 / 115 : pp = 226.4702606201172 30 / 115 : pp = 223.982666015625 40 / 115 : pp = 223.376953125 50 / 115 : pp = 218.65716552734375 60 / 115 : pp = 217.95306396484375 70 / 115 : pp = 214.5392303466797 80 / 115 : pp = 212.07525634765625 90 / 115 : pp = 209.40631103515625 100 / 115 : pp = 205.1455078125 110 / 115 : pp = 203.6289520263672 Training perplexity: 185.14476013183594 Validation perplexity:203.3822784423828 Total time : 42.47052240371704 Epoch 6 0 / 1452 : pp = 233.56707763671875 10 / 1452 : pp = 202.6468505859375 20 / 1452 : pp = 198.2734375 30 / 1452 : pp = 193.47442626953125 40 / 1452 : pp = 195.17147827148438 50 / 1452 : pp = 191.5596923828125 60 / 1452 : pp = 190.4825897216797 70 / 1452 : pp = 191.07681274414062 80 / 1452 : pp = 190.339599609375 90 / 1452 : pp = 188.98277282714844 100 / 1452 : pp = 187.74757385253906 110 / 1452 : pp = 186.10104370117188 120 / 1452 : pp = 185.7500457763672 130 / 1452 : pp = 184.90707397460938 140 / 1452 : pp = 183.340087890625 150 / 1452 : pp = 182.70840454101562 160 / 1452 : pp = 183.1043701171875 170 / 1452 : pp = 182.69776916503906 180 / 1452 : pp = 181.88400268554688 190 / 1452 : pp = 181.8062286376953 200 / 1452 : pp = 182.4969940185547 210 / 1452 : pp = 182.10572814941406 220 / 1452 : pp = 181.9981689453125 230 / 1452 : pp = 182.3802490234375 240 / 1452 : pp = 182.03636169433594 250 / 1452 : pp = 181.23712158203125 260 / 1452 : pp = 180.53726196289062 270 / 1452 : pp = 179.53567504882812 280 / 1452 : pp = 179.70208740234375 290 / 1452 : pp = 179.977783203125 300 / 1452 : pp = 180.16600036621094 310 / 1452 : pp = 179.87294006347656 320 / 1452 : pp = 180.11849975585938 330 / 1452 : pp = 180.31838989257812 340 / 1452 : pp = 179.56759643554688 350 / 1452 : pp = 179.97134399414062 360 / 1452 : pp = 179.80030822753906 370 / 1452 : pp = 179.52085876464844 380 / 1452 : pp = 178.98228454589844 390 / 1452 : pp = 179.0868682861328 400 / 1452 : pp = 178.74569702148438 410 / 1452 : pp = 179.1776580810547 420 / 1452 : pp = 179.5055389404297 430 / 1452 : pp = 179.3883056640625 440 / 1452 : pp = 179.42279052734375 450 / 1452 : pp = 179.2106475830078 460 / 1452 : pp = 178.85311889648438 470 / 1452 : pp = 178.33840942382812 480 / 1452 : pp = 177.60350036621094 490 / 1452 : pp = 177.30335998535156 500 / 1452 : pp = 176.72222900390625 510 / 1452 : pp = 176.6067352294922 520 / 1452 : pp = 176.33998107910156 530 / 1452 : pp = 175.93162536621094 540 / 1452 : pp = 175.30657958984375 550 / 1452 : pp = 174.9462432861328 560 / 1452 : pp = 174.5836639404297 570 / 1452 : pp = 174.31431579589844 580 / 1452 : pp = 173.92300415039062 590 / 1452 : pp = 173.55856323242188 600 / 1452 : pp = 173.08277893066406 610 / 1452 : pp = 172.75930786132812 620 / 1452 : pp = 172.53192138671875 630 / 1452 : pp = 172.20652770996094 640 / 1452 : pp = 172.37454223632812 650 / 1452 : pp = 172.39845275878906 660 / 1452 : pp = 172.52255249023438 670 / 1452 : pp = 172.60935974121094 680 / 1452 : pp = 172.6611328125 690 / 1452 : pp = 172.53118896484375 700 / 1452 : pp = 172.4709014892578 710 / 1452 : pp = 172.5406494140625 720 / 1452 : pp = 172.55447387695312 730 / 1452 : pp = 172.5330047607422 740 / 1452 : pp = 172.7061767578125 750 / 1452 : pp = 172.71054077148438 760 / 1452 : pp = 172.77743530273438 770 / 1452 : pp = 172.95481872558594 780 / 1452 : pp = 173.11265563964844 790 / 1452 : pp = 173.2832794189453 800 / 1452 : pp = 173.2537841796875 810 / 1452 : pp = 173.22164916992188 820 / 1452 : pp = 173.24148559570312 830 / 1452 : pp = 173.48228454589844 840 / 1452 : pp = 173.43753051757812 850 / 1452 : pp = 173.505615234375 860 / 1452 : pp = 173.5214080810547 870 / 1452 : pp = 173.5009002685547 880 / 1452 : pp = 173.6202392578125 890 / 1452 : pp = 173.622802734375 900 / 1452 : pp = 173.5987091064453 910 / 1452 : pp = 173.68316650390625 920 / 1452 : pp = 173.77330017089844 930 / 1452 : pp = 173.72018432617188 940 / 1452 : pp = 173.79351806640625 950 / 1452 : pp = 173.7653350830078 960 / 1452 : pp = 173.7102508544922 970 / 1452 : pp = 173.69766235351562 980 / 1452 : pp = 173.4836883544922 990 / 1452 : pp = 173.3550262451172 1000 / 1452 : pp = 173.14816284179688 1010 / 1452 : pp = 173.20777893066406 1020 / 1452 : pp = 173.3390655517578 1030 / 1452 : pp = 173.2884063720703 1040 / 1452 : pp = 173.38015747070312 1050 / 1452 : pp = 173.35592651367188 1060 / 1452 : pp = 173.2260284423828 1070 / 1452 : pp = 173.39321899414062 1080 / 1452 : pp = 173.4879913330078 1090 / 1452 : pp = 173.5231475830078 1100 / 1452 : pp = 173.47177124023438 1110 / 1452 : pp = 173.24453735351562 1120 / 1452 : pp = 173.09408569335938 1130 / 1452 : pp = 172.86627197265625 1140 / 1452 : pp = 172.8234100341797 1150 / 1452 : pp = 172.92843627929688 1160 / 1452 : pp = 172.90065002441406 1170 / 1452 : pp = 172.8550567626953 1180 / 1452 : pp = 172.8810272216797 1190 / 1452 : pp = 172.97312927246094 1200 / 1452 : pp = 172.9776611328125 1210 / 1452 : pp = 172.89413452148438 1220 / 1452 : pp = 173.0257568359375 1230 / 1452 : pp = 173.1847381591797 1240 / 1452 : pp = 173.1756591796875 1250 / 1452 : pp = 173.32138061523438 1260 / 1452 : pp = 173.37229919433594 1270 / 1452 : pp = 173.36891174316406 1280 / 1452 : pp = 173.36337280273438 1290 / 1452 : pp = 173.3444366455078 1300 / 1452 : pp = 173.36138916015625 1310 / 1452 : pp = 173.4015655517578 1320 / 1452 : pp = 173.31790161132812 1330 / 1452 : pp = 173.24710083007812 1340 / 1452 : pp = 173.27212524414062 1350 / 1452 : pp = 173.27674865722656 1360 / 1452 : pp = 173.32749938964844 1370 / 1452 : pp = 173.20472717285156 1380 / 1452 : pp = 173.14889526367188 1390 / 1452 : pp = 173.0755157470703 1400 / 1452 : pp = 172.9678497314453 1410 / 1452 : pp = 172.9612579345703 1420 / 1452 : pp = 172.8872833251953 1430 / 1452 : pp = 172.84805297851562 1440 / 1452 : pp = 172.87252807617188 1450 / 1452 : pp = 172.82505798339844 0 / 115 : pp = 236.35635375976562 10 / 115 : pp = 219.06166076660156 20 / 115 : pp = 219.7670440673828 30 / 115 : pp = 217.33587646484375 40 / 115 : pp = 216.6626739501953 50 / 115 : pp = 212.04734802246094 60 / 115 : pp = 211.42068481445312 70 / 115 : pp = 207.9592742919922 80 / 115 : pp = 205.6216583251953 90 / 115 : pp = 202.93597412109375 100 / 115 : pp = 198.62583923339844 110 / 115 : pp = 196.97216796875 Training perplexity: 172.80404663085938 Validation perplexity:196.6871337890625 Total time : 41.52522921562195 Epoch 7 0 / 1452 : pp = 219.23231506347656 10 / 1452 : pp = 192.07225036621094 20 / 1452 : pp = 187.48464965820312 30 / 1452 : pp = 182.9149932861328 40 / 1452 : pp = 184.2945098876953 50 / 1452 : pp = 180.78492736816406 60 / 1452 : pp = 179.377197265625 70 / 1452 : pp = 180.0273895263672 80 / 1452 : pp = 179.2517547607422 90 / 1452 : pp = 177.77540588378906 100 / 1452 : pp = 176.6474151611328 110 / 1452 : pp = 174.84066772460938 120 / 1452 : pp = 174.46890258789062 130 / 1452 : pp = 173.64573669433594 140 / 1452 : pp = 172.17483520507812 150 / 1452 : pp = 171.57041931152344 160 / 1452 : pp = 171.92059326171875 170 / 1452 : pp = 171.5497283935547 180 / 1452 : pp = 170.77249145507812 190 / 1452 : pp = 170.72103881835938 200 / 1452 : pp = 171.336181640625 210 / 1452 : pp = 170.98524475097656 220 / 1452 : pp = 170.99771118164062 230 / 1452 : pp = 171.39918518066406 240 / 1452 : pp = 171.09925842285156 250 / 1452 : pp = 170.39962768554688 260 / 1452 : pp = 169.7328643798828 270 / 1452 : pp = 168.72225952148438 280 / 1452 : pp = 168.92552185058594 290 / 1452 : pp = 169.20147705078125 300 / 1452 : pp = 169.40338134765625 310 / 1452 : pp = 169.12057495117188 320 / 1452 : pp = 169.31236267089844 330 / 1452 : pp = 169.49945068359375 340 / 1452 : pp = 168.8396759033203 350 / 1452 : pp = 169.25917053222656 360 / 1452 : pp = 169.09388732910156 370 / 1452 : pp = 168.84323120117188 380 / 1452 : pp = 168.3832550048828 390 / 1452 : pp = 168.48275756835938 400 / 1452 : pp = 168.19972229003906 410 / 1452 : pp = 168.5838623046875 420 / 1452 : pp = 168.91119384765625 430 / 1452 : pp = 168.80836486816406 440 / 1452 : pp = 168.90264892578125 450 / 1452 : pp = 168.68589782714844 460 / 1452 : pp = 168.3704071044922 470 / 1452 : pp = 167.90394592285156 480 / 1452 : pp = 167.23373413085938 490 / 1452 : pp = 166.9560546875 500 / 1452 : pp = 166.43161010742188 510 / 1452 : pp = 166.320068359375 520 / 1452 : pp = 166.05902099609375 530 / 1452 : pp = 165.71714782714844 540 / 1452 : pp = 165.10398864746094 550 / 1452 : pp = 164.80430603027344 560 / 1452 : pp = 164.4687042236328 570 / 1452 : pp = 164.2272491455078 580 / 1452 : pp = 163.84312438964844 590 / 1452 : pp = 163.46035766601562 600 / 1452 : pp = 163.01559448242188 610 / 1452 : pp = 162.74134826660156 620 / 1452 : pp = 162.50267028808594 630 / 1452 : pp = 162.2018280029297 640 / 1452 : pp = 162.37130737304688 650 / 1452 : pp = 162.3895721435547 660 / 1452 : pp = 162.51351928710938 670 / 1452 : pp = 162.57684326171875 680 / 1452 : pp = 162.6346893310547 690 / 1452 : pp = 162.5135955810547 700 / 1452 : pp = 162.47052001953125 710 / 1452 : pp = 162.539794921875 720 / 1452 : pp = 162.55381774902344 730 / 1452 : pp = 162.5297088623047 740 / 1452 : pp = 162.71652221679688 750 / 1452 : pp = 162.740966796875 760 / 1452 : pp = 162.79754638671875 770 / 1452 : pp = 162.9949951171875 780 / 1452 : pp = 163.17868041992188 790 / 1452 : pp = 163.33055114746094 800 / 1452 : pp = 163.31591796875 810 / 1452 : pp = 163.2859344482422 820 / 1452 : pp = 163.2958984375 830 / 1452 : pp = 163.528564453125 840 / 1452 : pp = 163.47610473632812 850 / 1452 : pp = 163.5260772705078 860 / 1452 : pp = 163.55352783203125 870 / 1452 : pp = 163.55718994140625 880 / 1452 : pp = 163.67523193359375 890 / 1452 : pp = 163.6920166015625 900 / 1452 : pp = 163.67710876464844 910 / 1452 : pp = 163.7476806640625 920 / 1452 : pp = 163.84803771972656 930 / 1452 : pp = 163.8114013671875 940 / 1452 : pp = 163.86663818359375 950 / 1452 : pp = 163.83531188964844 960 / 1452 : pp = 163.79945373535156 970 / 1452 : pp = 163.80320739746094 980 / 1452 : pp = 163.5953369140625 990 / 1452 : pp = 163.48382568359375 1000 / 1452 : pp = 163.2642822265625 1010 / 1452 : pp = 163.32113647460938 1020 / 1452 : pp = 163.44204711914062 1030 / 1452 : pp = 163.40206909179688 1040 / 1452 : pp = 163.4915313720703 1050 / 1452 : pp = 163.47096252441406 1060 / 1452 : pp = 163.3601531982422 1070 / 1452 : pp = 163.5138397216797 1080 / 1452 : pp = 163.6189727783203 1090 / 1452 : pp = 163.6471405029297 1100 / 1452 : pp = 163.60406494140625 1110 / 1452 : pp = 163.40736389160156 1120 / 1452 : pp = 163.26841735839844 1130 / 1452 : pp = 163.0680694580078 1140 / 1452 : pp = 163.04591369628906 1150 / 1452 : pp = 163.15478515625 1160 / 1452 : pp = 163.1380615234375 1170 / 1452 : pp = 163.09303283691406 1180 / 1452 : pp = 163.14149475097656 1190 / 1452 : pp = 163.2374267578125 1200 / 1452 : pp = 163.2394561767578 1210 / 1452 : pp = 163.17835998535156 1220 / 1452 : pp = 163.32347106933594 1230 / 1452 : pp = 163.4639434814453 1240 / 1452 : pp = 163.4611358642578 1250 / 1452 : pp = 163.60687255859375 1260 / 1452 : pp = 163.67227172851562 1270 / 1452 : pp = 163.67515563964844 1280 / 1452 : pp = 163.6881103515625 1290 / 1452 : pp = 163.66648864746094 1300 / 1452 : pp = 163.69287109375 1310 / 1452 : pp = 163.7276153564453 1320 / 1452 : pp = 163.6551055908203 1330 / 1452 : pp = 163.58901977539062 1340 / 1452 : pp = 163.6205291748047 1350 / 1452 : pp = 163.63824462890625 1360 / 1452 : pp = 163.69334411621094 1370 / 1452 : pp = 163.5885467529297 1380 / 1452 : pp = 163.54049682617188 1390 / 1452 : pp = 163.4760284423828 1400 / 1452 : pp = 163.38897705078125 1410 / 1452 : pp = 163.3974609375 1420 / 1452 : pp = 163.35009765625 1430 / 1452 : pp = 163.32191467285156 1440 / 1452 : pp = 163.35220336914062 1450 / 1452 : pp = 163.3201904296875 0 / 115 : pp = 232.2108154296875 10 / 115 : pp = 214.35496520996094 20 / 115 : pp = 215.20510864257812 30 / 115 : pp = 212.82754516601562 40 / 115 : pp = 212.0598907470703 50 / 115 : pp = 207.5095672607422 60 / 115 : pp = 206.86976623535156 70 / 115 : pp = 203.36016845703125 80 / 115 : pp = 201.11538696289062 90 / 115 : pp = 198.52120971679688 100 / 115 : pp = 194.1772003173828 110 / 115 : pp = 192.41224670410156 Training perplexity: 163.29916381835938 Validation perplexity:192.09552001953125 Total time : 41.78096055984497 Epoch 8 0 / 1452 : pp = 201.77548217773438 10 / 1452 : pp = 180.4141082763672 20 / 1452 : pp = 176.41432189941406 30 / 1452 : pp = 172.7764434814453 40 / 1452 : pp = 174.69166564941406 50 / 1452 : pp = 171.2933807373047 60 / 1452 : pp = 170.08010864257812 70 / 1452 : pp = 170.6719512939453 80 / 1452 : pp = 170.07589721679688 90 / 1452 : pp = 168.7478485107422 100 / 1452 : pp = 167.57081604003906 110 / 1452 : pp = 166.06971740722656 120 / 1452 : pp = 165.73374938964844 130 / 1452 : pp = 164.80674743652344 140 / 1452 : pp = 163.32821655273438 150 / 1452 : pp = 162.6752471923828 160 / 1452 : pp = 163.02049255371094 170 / 1452 : pp = 162.64120483398438 180 / 1452 : pp = 161.95529174804688 190 / 1452 : pp = 161.91954040527344 200 / 1452 : pp = 162.5446014404297 210 / 1452 : pp = 162.2645721435547 220 / 1452 : pp = 162.3128662109375 230 / 1452 : pp = 162.65872192382812 240 / 1452 : pp = 162.40948486328125 250 / 1452 : pp = 161.75787353515625 260 / 1452 : pp = 161.15213012695312 270 / 1452 : pp = 160.22256469726562 280 / 1452 : pp = 160.3651123046875 290 / 1452 : pp = 160.63780212402344 300 / 1452 : pp = 160.80026245117188 310 / 1452 : pp = 160.54383850097656 320 / 1452 : pp = 160.7539520263672 330 / 1452 : pp = 160.94317626953125 340 / 1452 : pp = 160.3373565673828 350 / 1452 : pp = 160.71763610839844 360 / 1452 : pp = 160.60960388183594 370 / 1452 : pp = 160.37527465820312 380 / 1452 : pp = 159.92990112304688 390 / 1452 : pp = 160.0165557861328 400 / 1452 : pp = 159.75697326660156 410 / 1452 : pp = 160.15274047851562 420 / 1452 : pp = 160.48390197753906 430 / 1452 : pp = 160.4031982421875 440 / 1452 : pp = 160.4693603515625 450 / 1452 : pp = 160.28016662597656 460 / 1452 : pp = 159.94004821777344 470 / 1452 : pp = 159.48257446289062 480 / 1452 : pp = 158.87998962402344 490 / 1452 : pp = 158.59765625 500 / 1452 : pp = 158.10865783691406 510 / 1452 : pp = 157.96795654296875 520 / 1452 : pp = 157.7591552734375 530 / 1452 : pp = 157.42648315429688 540 / 1452 : pp = 156.85348510742188 550 / 1452 : pp = 156.5618438720703 560 / 1452 : pp = 156.24905395507812 570 / 1452 : pp = 155.9994354248047 580 / 1452 : pp = 155.612060546875 590 / 1452 : pp = 155.25830078125 600 / 1452 : pp = 154.8464813232422 610 / 1452 : pp = 154.5833282470703 620 / 1452 : pp = 154.38040161132812 630 / 1452 : pp = 154.0767364501953 640 / 1452 : pp = 154.2534637451172 650 / 1452 : pp = 154.25875854492188 660 / 1452 : pp = 154.35874938964844 670 / 1452 : pp = 154.4289093017578 680 / 1452 : pp = 154.51412963867188 690 / 1452 : pp = 154.41676330566406 700 / 1452 : pp = 154.37892150878906 710 / 1452 : pp = 154.4234619140625 720 / 1452 : pp = 154.4586639404297 730 / 1452 : pp = 154.4351806640625 740 / 1452 : pp = 154.6002197265625 750 / 1452 : pp = 154.65684509277344 760 / 1452 : pp = 154.73318481445312 770 / 1452 : pp = 154.92935180664062 780 / 1452 : pp = 155.1021728515625 790 / 1452 : pp = 155.24757385253906 800 / 1452 : pp = 155.223876953125 810 / 1452 : pp = 155.2095184326172 820 / 1452 : pp = 155.24009704589844 830 / 1452 : pp = 155.4519500732422 840 / 1452 : pp = 155.3947296142578 850 / 1452 : pp = 155.45306396484375 860 / 1452 : pp = 155.4661102294922 870 / 1452 : pp = 155.45765686035156 880 / 1452 : pp = 155.58758544921875 890 / 1452 : pp = 155.59373474121094 900 / 1452 : pp = 155.59254455566406 910 / 1452 : pp = 155.66854858398438 920 / 1452 : pp = 155.75942993164062 930 / 1452 : pp = 155.73350524902344 940 / 1452 : pp = 155.80740356445312 950 / 1452 : pp = 155.7733917236328 960 / 1452 : pp = 155.73565673828125 970 / 1452 : pp = 155.74404907226562 980 / 1452 : pp = 155.55902099609375 990 / 1452 : pp = 155.45675659179688 1000 / 1452 : pp = 155.2649688720703 1010 / 1452 : pp = 155.31332397460938 1020 / 1452 : pp = 155.44979858398438 1030 / 1452 : pp = 155.4137725830078 1040 / 1452 : pp = 155.49012756347656 1050 / 1452 : pp = 155.46054077148438 1060 / 1452 : pp = 155.3616943359375 1070 / 1452 : pp = 155.5286865234375 1080 / 1452 : pp = 155.63743591308594 1090 / 1452 : pp = 155.6842803955078 1100 / 1452 : pp = 155.65599060058594 1110 / 1452 : pp = 155.4827880859375 1120 / 1452 : pp = 155.35450744628906 1130 / 1452 : pp = 155.1777801513672 1140 / 1452 : pp = 155.15994262695312 1150 / 1452 : pp = 155.26193237304688 1160 / 1452 : pp = 155.26214599609375 1170 / 1452 : pp = 155.23231506347656 1180 / 1452 : pp = 155.29266357421875 1190 / 1452 : pp = 155.37680053710938 1200 / 1452 : pp = 155.3736114501953 1210 / 1452 : pp = 155.3380584716797 1220 / 1452 : pp = 155.474853515625 1230 / 1452 : pp = 155.62986755371094 1240 / 1452 : pp = 155.62831115722656 1250 / 1452 : pp = 155.77101135253906 1260 / 1452 : pp = 155.83445739746094 1270 / 1452 : pp = 155.845458984375 1280 / 1452 : pp = 155.8556365966797 1290 / 1452 : pp = 155.8556365966797 1300 / 1452 : pp = 155.8843994140625 1310 / 1452 : pp = 155.92417907714844 1320 / 1452 : pp = 155.8560791015625 1330 / 1452 : pp = 155.80636596679688 1340 / 1452 : pp = 155.84344482421875 1350 / 1452 : pp = 155.8706512451172 1360 / 1452 : pp = 155.9273681640625 1370 / 1452 : pp = 155.83140563964844 1380 / 1452 : pp = 155.7911376953125 1390 / 1452 : pp = 155.7401885986328 1400 / 1452 : pp = 155.6622314453125 1410 / 1452 : pp = 155.68531799316406 1420 / 1452 : pp = 155.64041137695312 1430 / 1452 : pp = 155.62216186523438 1440 / 1452 : pp = 155.6437530517578 1450 / 1452 : pp = 155.62757873535156 0 / 115 : pp = 228.70111083984375 10 / 115 : pp = 211.03330993652344 20 / 115 : pp = 212.24957275390625 30 / 115 : pp = 209.8839569091797 40 / 115 : pp = 209.11045837402344 50 / 115 : pp = 204.66351318359375 60 / 115 : pp = 204.03366088867188 70 / 115 : pp = 200.46681213378906 80 / 115 : pp = 198.24404907226562 90 / 115 : pp = 195.63223266601562 100 / 115 : pp = 191.18345642089844 110 / 115 : pp = 189.31134033203125 Training perplexity: 155.61154174804688 Validation perplexity:188.94537353515625 Total time : 42.13483738899231 Epoch 9 0 / 1452 : pp = 197.80628967285156 10 / 1452 : pp = 172.6316680908203 20 / 1452 : pp = 168.6739959716797 30 / 1452 : pp = 164.4781036376953 40 / 1452 : pp = 166.1627960205078 50 / 1452 : pp = 163.05197143554688 60 / 1452 : pp = 161.87924194335938 70 / 1452 : pp = 162.5297088623047 80 / 1452 : pp = 161.7450714111328 90 / 1452 : pp = 160.6148223876953 100 / 1452 : pp = 159.73289489746094 110 / 1452 : pp = 158.4092254638672 120 / 1452 : pp = 158.04653930664062 130 / 1452 : pp = 157.13563537597656 140 / 1452 : pp = 155.71798706054688 150 / 1452 : pp = 155.19161987304688 160 / 1452 : pp = 155.42718505859375 170 / 1452 : pp = 155.0531463623047 180 / 1452 : pp = 154.46897888183594 190 / 1452 : pp = 154.4127197265625 200 / 1452 : pp = 154.97154235839844 210 / 1452 : pp = 154.70169067382812 220 / 1452 : pp = 154.72816467285156 230 / 1452 : pp = 155.03799438476562 240 / 1452 : pp = 154.85601806640625 250 / 1452 : pp = 154.28016662597656 260 / 1452 : pp = 153.7699432373047 270 / 1452 : pp = 152.90948486328125 280 / 1452 : pp = 153.0459747314453 290 / 1452 : pp = 153.298095703125 300 / 1452 : pp = 153.45716857910156 310 / 1452 : pp = 153.22195434570312 320 / 1452 : pp = 153.41664123535156 330 / 1452 : pp = 153.66542053222656 340 / 1452 : pp = 153.06378173828125 350 / 1452 : pp = 153.43923950195312 360 / 1452 : pp = 153.31381225585938 370 / 1452 : pp = 153.13473510742188 380 / 1452 : pp = 152.75267028808594 390 / 1452 : pp = 152.85504150390625 400 / 1452 : pp = 152.62342834472656 410 / 1452 : pp = 153.03152465820312 420 / 1452 : pp = 153.39161682128906 430 / 1452 : pp = 153.30364990234375 440 / 1452 : pp = 153.37896728515625 450 / 1452 : pp = 153.18988037109375 460 / 1452 : pp = 152.88478088378906 470 / 1452 : pp = 152.4380340576172 480 / 1452 : pp = 151.86618041992188 490 / 1452 : pp = 151.5962371826172 500 / 1452 : pp = 151.11614990234375 510 / 1452 : pp = 150.99830627441406 520 / 1452 : pp = 150.8135986328125 530 / 1452 : pp = 150.500732421875 540 / 1452 : pp = 149.9623260498047 550 / 1452 : pp = 149.68028259277344 560 / 1452 : pp = 149.3885040283203 570 / 1452 : pp = 149.140380859375 580 / 1452 : pp = 148.76876831054688 590 / 1452 : pp = 148.43368530273438 600 / 1452 : pp = 148.02598571777344 610 / 1452 : pp = 147.7869110107422 620 / 1452 : pp = 147.59796142578125 630 / 1452 : pp = 147.30068969726562 640 / 1452 : pp = 147.45240783691406 650 / 1452 : pp = 147.4651336669922 660 / 1452 : pp = 147.5808563232422 670 / 1452 : pp = 147.65582275390625 680 / 1452 : pp = 147.7360382080078 690 / 1452 : pp = 147.63075256347656 700 / 1452 : pp = 147.6066131591797 710 / 1452 : pp = 147.7024383544922 720 / 1452 : pp = 147.7445526123047 730 / 1452 : pp = 147.72279357910156 740 / 1452 : pp = 147.87107849121094 750 / 1452 : pp = 147.91436767578125 760 / 1452 : pp = 147.9857635498047 770 / 1452 : pp = 148.18206787109375 780 / 1452 : pp = 148.3845672607422 790 / 1452 : pp = 148.5517120361328 800 / 1452 : pp = 148.54002380371094 810 / 1452 : pp = 148.51119995117188 820 / 1452 : pp = 148.5664520263672 830 / 1452 : pp = 148.7821044921875 840 / 1452 : pp = 148.72486877441406 850 / 1452 : pp = 148.77452087402344 860 / 1452 : pp = 148.80076599121094 870 / 1452 : pp = 148.79701232910156 880 / 1452 : pp = 148.9181671142578 890 / 1452 : pp = 148.94537353515625 900 / 1452 : pp = 148.9435272216797 910 / 1452 : pp = 149.02102661132812 920 / 1452 : pp = 149.1085968017578 930 / 1452 : pp = 149.06893920898438 940 / 1452 : pp = 149.1317138671875 950 / 1452 : pp = 149.1232452392578 960 / 1452 : pp = 149.10354614257812 970 / 1452 : pp = 149.11656188964844 980 / 1452 : pp = 148.94259643554688 990 / 1452 : pp = 148.8236846923828 1000 / 1452 : pp = 148.633056640625 1010 / 1452 : pp = 148.6830291748047 1020 / 1452 : pp = 148.8126220703125 1030 / 1452 : pp = 148.78089904785156 1040 / 1452 : pp = 148.8600311279297 1050 / 1452 : pp = 148.8486785888672 1060 / 1452 : pp = 148.7664337158203 1070 / 1452 : pp = 148.9337921142578 1080 / 1452 : pp = 149.04441833496094 1090 / 1452 : pp = 149.07284545898438 1100 / 1452 : pp = 149.03318786621094 1110 / 1452 : pp = 148.86428833007812 1120 / 1452 : pp = 148.7332305908203 1130 / 1452 : pp = 148.5670166015625 1140 / 1452 : pp = 148.54661560058594 1150 / 1452 : pp = 148.64219665527344 1160 / 1452 : pp = 148.6490020751953 1170 / 1452 : pp = 148.62420654296875 1180 / 1452 : pp = 148.67665100097656 1190 / 1452 : pp = 148.7633056640625 1200 / 1452 : pp = 148.7782745361328 1210 / 1452 : pp = 148.72500610351562 1220 / 1452 : pp = 148.87493896484375 1230 / 1452 : pp = 149.039794921875 1240 / 1452 : pp = 149.04000854492188 1250 / 1452 : pp = 149.17054748535156 1260 / 1452 : pp = 149.23863220214844 1270 / 1452 : pp = 149.2436065673828 1280 / 1452 : pp = 149.25086975097656 1290 / 1452 : pp = 149.24147033691406 1300 / 1452 : pp = 149.27413940429688 1310 / 1452 : pp = 149.32077026367188 1320 / 1452 : pp = 149.27301025390625 1330 / 1452 : pp = 149.23080444335938 1340 / 1452 : pp = 149.25791931152344 1350 / 1452 : pp = 149.2841033935547 1360 / 1452 : pp = 149.337158203125 1370 / 1452 : pp = 149.2467498779297 1380 / 1452 : pp = 149.21351623535156 1390 / 1452 : pp = 149.15403747558594 1400 / 1452 : pp = 149.0877685546875 1410 / 1452 : pp = 149.110595703125 1420 / 1452 : pp = 149.07241821289062 1430 / 1452 : pp = 149.05166625976562 1440 / 1452 : pp = 149.0776824951172 1450 / 1452 : pp = 149.06771850585938 0 / 115 : pp = 227.0559844970703 10 / 115 : pp = 208.7002410888672 20 / 115 : pp = 210.38775634765625 30 / 115 : pp = 207.9513397216797 40 / 115 : pp = 207.12994384765625 50 / 115 : pp = 202.70811462402344 60 / 115 : pp = 202.05787658691406 70 / 115 : pp = 198.3761444091797 80 / 115 : pp = 196.17637634277344 90 / 115 : pp = 193.5880126953125 100 / 115 : pp = 189.0758819580078 110 / 115 : pp = 187.07528686523438 Training perplexity: 149.0502471923828 Validation perplexity:186.6911163330078 Total time : 47.274805545806885 Epoch 10 0 / 1452 : pp = 181.8408203125 10 / 1452 : pp = 164.99664306640625 20 / 1452 : pp = 161.8847198486328 30 / 1452 : pp = 158.30064392089844 40 / 1452 : pp = 160.13914489746094 50 / 1452 : pp = 157.58743286132812 60 / 1452 : pp = 156.11871337890625 70 / 1452 : pp = 156.82948303222656 80 / 1452 : pp = 156.2889862060547 90 / 1452 : pp = 155.04833984375 100 / 1452 : pp = 154.09327697753906 110 / 1452 : pp = 152.5070343017578 120 / 1452 : pp = 152.20750427246094 130 / 1452 : pp = 151.3399200439453 140 / 1452 : pp = 149.90740966796875 150 / 1452 : pp = 149.345703125 160 / 1452 : pp = 149.59814453125 170 / 1452 : pp = 149.26539611816406 180 / 1452 : pp = 148.624267578125 190 / 1452 : pp = 148.58819580078125 200 / 1452 : pp = 149.09552001953125 210 / 1452 : pp = 148.8439178466797 220 / 1452 : pp = 148.86605834960938 230 / 1452 : pp = 149.1971435546875 240 / 1452 : pp = 148.96533203125 250 / 1452 : pp = 148.4253387451172 260 / 1452 : pp = 147.9200897216797 270 / 1452 : pp = 147.08816528320312 280 / 1452 : pp = 147.24366760253906 290 / 1452 : pp = 147.52182006835938 300 / 1452 : pp = 147.72222900390625 310 / 1452 : pp = 147.50486755371094 320 / 1452 : pp = 147.73892211914062 330 / 1452 : pp = 147.9404754638672 340 / 1452 : pp = 147.37803649902344 350 / 1452 : pp = 147.6969451904297 360 / 1452 : pp = 147.5704345703125 370 / 1452 : pp = 147.38674926757812 380 / 1452 : pp = 147.03970336914062 390 / 1452 : pp = 147.14231872558594 400 / 1452 : pp = 146.91656494140625 410 / 1452 : pp = 147.34059143066406 420 / 1452 : pp = 147.68496704101562 430 / 1452 : pp = 147.61195373535156 440 / 1452 : pp = 147.68405151367188 450 / 1452 : pp = 147.4711151123047 460 / 1452 : pp = 147.1927032470703 470 / 1452 : pp = 146.72970581054688 480 / 1452 : pp = 146.17173767089844 490 / 1452 : pp = 145.9028778076172 500 / 1452 : pp = 145.42721557617188 510 / 1452 : pp = 145.3111114501953 520 / 1452 : pp = 145.11460876464844 530 / 1452 : pp = 144.81488037109375 540 / 1452 : pp = 144.263916015625 550 / 1452 : pp = 143.997802734375 560 / 1452 : pp = 143.71766662597656 570 / 1452 : pp = 143.47451782226562 580 / 1452 : pp = 143.08474731445312 590 / 1452 : pp = 142.77920532226562 600 / 1452 : pp = 142.39573669433594 610 / 1452 : pp = 142.14906311035156 620 / 1452 : pp = 141.9574432373047 630 / 1452 : pp = 141.67369079589844 640 / 1452 : pp = 141.81556701660156 650 / 1452 : pp = 141.81759643554688 660 / 1452 : pp = 141.9339599609375 670 / 1452 : pp = 142.01248168945312 680 / 1452 : pp = 142.08773803710938 690 / 1452 : pp = 142.00328063964844 700 / 1452 : pp = 141.98086547851562 710 / 1452 : pp = 142.0632781982422 720 / 1452 : pp = 142.10372924804688 730 / 1452 : pp = 142.08055114746094 740 / 1452 : pp = 142.23619079589844 750 / 1452 : pp = 142.2660369873047 760 / 1452 : pp = 142.34678649902344 770 / 1452 : pp = 142.5257568359375 780 / 1452 : pp = 142.70025634765625 790 / 1452 : pp = 142.8614044189453 800 / 1452 : pp = 142.84573364257812 810 / 1452 : pp = 142.8250274658203 820 / 1452 : pp = 142.8540496826172 830 / 1452 : pp = 143.06053161621094 840 / 1452 : pp = 143.0423126220703 850 / 1452 : pp = 143.09634399414062 860 / 1452 : pp = 143.10487365722656 870 / 1452 : pp = 143.0884246826172 880 / 1452 : pp = 143.19387817382812 890 / 1452 : pp = 143.236083984375 900 / 1452 : pp = 143.23390197753906 910 / 1452 : pp = 143.29537963867188 920 / 1452 : pp = 143.3722686767578 930 / 1452 : pp = 143.33795166015625 940 / 1452 : pp = 143.40618896484375 950 / 1452 : pp = 143.3929901123047 960 / 1452 : pp = 143.3693389892578 970 / 1452 : pp = 143.39736938476562 980 / 1452 : pp = 143.2371063232422 990 / 1452 : pp = 143.13893127441406 1000 / 1452 : pp = 142.9658660888672 1010 / 1452 : pp = 143.01544189453125 1020 / 1452 : pp = 143.152587890625 1030 / 1452 : pp = 143.11334228515625 1040 / 1452 : pp = 143.19020080566406 1050 / 1452 : pp = 143.18234252929688 1060 / 1452 : pp = 143.092041015625 1070 / 1452 : pp = 143.24449157714844 1080 / 1452 : pp = 143.34828186035156 1090 / 1452 : pp = 143.38739013671875 1100 / 1452 : pp = 143.37432861328125 1110 / 1452 : pp = 143.20596313476562 1120 / 1452 : pp = 143.07969665527344 1130 / 1452 : pp = 142.92041015625 1140 / 1452 : pp = 142.90902709960938 1150 / 1452 : pp = 143.00732421875 1160 / 1452 : pp = 143.01182556152344 1170 / 1452 : pp = 142.9925994873047 1180 / 1452 : pp = 143.06080627441406 1190 / 1452 : pp = 143.14337158203125 1200 / 1452 : pp = 143.16644287109375 1210 / 1452 : pp = 143.1259002685547 1220 / 1452 : pp = 143.2671661376953 1230 / 1452 : pp = 143.4210968017578 1240 / 1452 : pp = 143.4327850341797 1250 / 1452 : pp = 143.5699920654297 1260 / 1452 : pp = 143.63771057128906 1270 / 1452 : pp = 143.65798950195312 1280 / 1452 : pp = 143.68251037597656 1290 / 1452 : pp = 143.68045043945312 1300 / 1452 : pp = 143.72293090820312 1310 / 1452 : pp = 143.77015686035156 1320 / 1452 : pp = 143.71910095214844 1330 / 1452 : pp = 143.68792724609375 1340 / 1452 : pp = 143.7241668701172 1350 / 1452 : pp = 143.7570037841797 1360 / 1452 : pp = 143.81829833984375 1370 / 1452 : pp = 143.7487030029297 1380 / 1452 : pp = 143.7196502685547 1390 / 1452 : pp = 143.67359924316406 1400 / 1452 : pp = 143.60592651367188 1410 / 1452 : pp = 143.62620544433594 1420 / 1452 : pp = 143.5905303955078 1430 / 1452 : pp = 143.55799865722656 1440 / 1452 : pp = 143.5891571044922 1450 / 1452 : pp = 143.5869598388672 0 / 115 : pp = 226.9864959716797 10 / 115 : pp = 207.8067169189453 20 / 115 : pp = 209.68667602539062 30 / 115 : pp = 207.1610565185547 40 / 115 : pp = 206.3247833251953 50 / 115 : pp = 201.77403259277344 60 / 115 : pp = 201.07098388671875 70 / 115 : pp = 197.33335876464844 80 / 115 : pp = 195.12513732910156 90 / 115 : pp = 192.5349578857422 100 / 115 : pp = 187.90072631835938 110 / 115 : pp = 185.81240844726562 Training perplexity: 143.57354736328125 Validation perplexity:185.40573120117188 Total time : 46.14846849441528 Epoch 11 0 / 1452 : pp = 181.93162536621094 10 / 1452 : pp = 159.94607543945312 20 / 1452 : pp = 156.83673095703125 30 / 1452 : pp = 153.75843811035156 40 / 1452 : pp = 155.18362426757812 50 / 1452 : pp = 152.39529418945312 60 / 1452 : pp = 151.18772888183594 70 / 1452 : pp = 151.9004364013672 80 / 1452 : pp = 151.30239868164062 90 / 1452 : pp = 150.1591033935547 100 / 1452 : pp = 149.18618774414062 110 / 1452 : pp = 147.72653198242188 120 / 1452 : pp = 147.4357452392578 130 / 1452 : pp = 146.41372680664062 140 / 1452 : pp = 145.0057373046875 150 / 1452 : pp = 144.39447021484375 160 / 1452 : pp = 144.5330047607422 170 / 1452 : pp = 144.23593139648438 180 / 1452 : pp = 143.63990783691406 190 / 1452 : pp = 143.63812255859375 200 / 1452 : pp = 144.1143798828125 210 / 1452 : pp = 143.88278198242188 220 / 1452 : pp = 143.92518615722656 230 / 1452 : pp = 144.24032592773438 240 / 1452 : pp = 143.94110107421875 250 / 1452 : pp = 143.3688507080078 260 / 1452 : pp = 142.8829345703125 270 / 1452 : pp = 142.11952209472656 280 / 1452 : pp = 142.19415283203125 290 / 1452 : pp = 142.51889038085938 300 / 1452 : pp = 142.70494079589844 310 / 1452 : pp = 142.51426696777344 320 / 1452 : pp = 142.70106506347656 330 / 1452 : pp = 142.88014221191406 340 / 1452 : pp = 142.3287353515625 350 / 1452 : pp = 142.6169891357422 360 / 1452 : pp = 142.51971435546875 370 / 1452 : pp = 142.33566284179688 380 / 1452 : pp = 142.04161071777344 390 / 1452 : pp = 142.13551330566406 400 / 1452 : pp = 141.9499969482422 410 / 1452 : pp = 142.3361358642578 420 / 1452 : pp = 142.64065551757812 430 / 1452 : pp = 142.5511016845703 440 / 1452 : pp = 142.6728973388672 450 / 1452 : pp = 142.47030639648438 460 / 1452 : pp = 142.1704864501953 470 / 1452 : pp = 141.73390197753906 480 / 1452 : pp = 141.23020935058594 490 / 1452 : pp = 140.9759521484375 500 / 1452 : pp = 140.51609802246094 510 / 1452 : pp = 140.40545654296875 520 / 1452 : pp = 140.1936492919922 530 / 1452 : pp = 139.8929443359375 540 / 1452 : pp = 139.3696746826172 550 / 1452 : pp = 139.13217163085938 560 / 1452 : pp = 138.85247802734375 570 / 1452 : pp = 138.6092987060547 580 / 1452 : pp = 138.2471160888672 590 / 1452 : pp = 137.9485626220703 600 / 1452 : pp = 137.57379150390625 610 / 1452 : pp = 137.31576538085938 620 / 1452 : pp = 137.14230346679688 630 / 1452 : pp = 136.87405395507812 640 / 1452 : pp = 137.02928161621094 650 / 1452 : pp = 137.0481719970703 660 / 1452 : pp = 137.1595001220703 670 / 1452 : pp = 137.21124267578125 680 / 1452 : pp = 137.2671356201172 690 / 1452 : pp = 137.19410705566406 700 / 1452 : pp = 137.1850128173828 710 / 1452 : pp = 137.26058959960938 720 / 1452 : pp = 137.30726623535156 730 / 1452 : pp = 137.28048706054688 740 / 1452 : pp = 137.4352569580078 750 / 1452 : pp = 137.4680938720703 760 / 1452 : pp = 137.5524139404297 770 / 1452 : pp = 137.73829650878906 780 / 1452 : pp = 137.90882873535156 790 / 1452 : pp = 138.05865478515625 800 / 1452 : pp = 138.0673370361328 810 / 1452 : pp = 138.03909301757812 820 / 1452 : pp = 138.084716796875 830 / 1452 : pp = 138.27989196777344 840 / 1452 : pp = 138.23545837402344 850 / 1452 : pp = 138.30343627929688 860 / 1452 : pp = 138.3339080810547 870 / 1452 : pp = 138.32835388183594 880 / 1452 : pp = 138.4450225830078 890 / 1452 : pp = 138.47157287597656 900 / 1452 : pp = 138.46304321289062 910 / 1452 : pp = 138.55618286132812 920 / 1452 : pp = 138.64512634277344 930 / 1452 : pp = 138.6160430908203 940 / 1452 : pp = 138.66932678222656 950 / 1452 : pp = 138.6573028564453 960 / 1452 : pp = 138.6463165283203 970 / 1452 : pp = 138.67059326171875 980 / 1452 : pp = 138.50999450683594 990 / 1452 : pp = 138.42430114746094 1000 / 1452 : pp = 138.25344848632812 1010 / 1452 : pp = 138.3004608154297 1020 / 1452 : pp = 138.4243621826172 1030 / 1452 : pp = 138.40713500976562 1040 / 1452 : pp = 138.47129821777344 1050 / 1452 : pp = 138.45928955078125 1060 / 1452 : pp = 138.3919677734375 1070 / 1452 : pp = 138.5287628173828 1080 / 1452 : pp = 138.62298583984375 1090 / 1452 : pp = 138.6699981689453 1100 / 1452 : pp = 138.64849853515625 1110 / 1452 : pp = 138.49191284179688 1120 / 1452 : pp = 138.37355041503906 1130 / 1452 : pp = 138.2216796875 1140 / 1452 : pp = 138.21534729003906 1150 / 1452 : pp = 138.30963134765625 1160 / 1452 : pp = 138.316162109375 1170 / 1452 : pp = 138.3023681640625 1180 / 1452 : pp = 138.36932373046875 1190 / 1452 : pp = 138.45960998535156 1200 / 1452 : pp = 138.4866180419922 1210 / 1452 : pp = 138.45730590820312 1220 / 1452 : pp = 138.60031127929688 1230 / 1452 : pp = 138.75485229492188 1240 / 1452 : pp = 138.7751007080078 1250 / 1452 : pp = 138.91221618652344 1260 / 1452 : pp = 138.9815216064453 1270 / 1452 : pp = 138.9919891357422 1280 / 1452 : pp = 139.0243377685547 1290 / 1452 : pp = 139.02725219726562 1300 / 1452 : pp = 139.0701446533203 1310 / 1452 : pp = 139.1090850830078 1320 / 1452 : pp = 139.06027221679688 1330 / 1452 : pp = 139.0338134765625 1340 / 1452 : pp = 139.06385803222656 1350 / 1452 : pp = 139.09608459472656 1360 / 1452 : pp = 139.1609649658203 1370 / 1452 : pp = 139.0869903564453 1380 / 1452 : pp = 139.0604705810547 1390 / 1452 : pp = 139.01670837402344 1400 / 1452 : pp = 138.94393920898438 1410 / 1452 : pp = 138.97323608398438 1420 / 1452 : pp = 138.9404296875 1430 / 1452 : pp = 138.90943908691406 1440 / 1452 : pp = 138.94268798828125 1450 / 1452 : pp = 138.93991088867188 0 / 115 : pp = 225.55990600585938 10 / 115 : pp = 207.0504608154297 20 / 115 : pp = 208.98306274414062 30 / 115 : pp = 206.28396606445312 40 / 115 : pp = 205.35386657714844 50 / 115 : pp = 200.7255401611328 60 / 115 : pp = 200.0526580810547 70 / 115 : pp = 196.33087158203125 80 / 115 : pp = 194.12110900878906 90 / 115 : pp = 191.52816772460938 100 / 115 : pp = 186.7974395751953 110 / 115 : pp = 184.59829711914062 Training perplexity: 138.9222869873047 Validation perplexity:184.18101501464844 Total time : 43.92928600311279 Epoch 12 0 / 1452 : pp = 173.0251007080078 10 / 1452 : pp = 152.98446655273438 20 / 1452 : pp = 150.43128967285156 30 / 1452 : pp = 147.5819854736328 40 / 1452 : pp = 149.4164276123047 50 / 1452 : pp = 146.70816040039062 60 / 1452 : pp = 145.557861328125 70 / 1452 : pp = 146.50473022460938 80 / 1452 : pp = 145.83200073242188 90 / 1452 : pp = 144.84402465820312 100 / 1452 : pp = 144.0390167236328 110 / 1452 : pp = 142.66514587402344 120 / 1452 : pp = 142.3549346923828 130 / 1452 : pp = 141.4630126953125 140 / 1452 : pp = 140.2266082763672 150 / 1452 : pp = 139.67518615722656 160 / 1452 : pp = 139.90414428710938 170 / 1452 : pp = 139.5490264892578 180 / 1452 : pp = 138.91969299316406 190 / 1452 : pp = 138.89234924316406 200 / 1452 : pp = 139.40908813476562 210 / 1452 : pp = 139.19068908691406 220 / 1452 : pp = 139.35513305664062 230 / 1452 : pp = 139.5464324951172 240 / 1452 : pp = 139.3047637939453 250 / 1452 : pp = 138.7708740234375 260 / 1452 : pp = 138.29188537597656 270 / 1452 : pp = 137.4787139892578 280 / 1452 : pp = 137.6367950439453 290 / 1452 : pp = 137.98513793945312 300 / 1452 : pp = 138.17819213867188 310 / 1452 : pp = 137.943359375 320 / 1452 : pp = 138.12060546875 330 / 1452 : pp = 138.29037475585938 340 / 1452 : pp = 137.77606201171875 350 / 1452 : pp = 138.06378173828125 360 / 1452 : pp = 137.99000549316406 370 / 1452 : pp = 137.81922912597656 380 / 1452 : pp = 137.52159118652344 390 / 1452 : pp = 137.61782836914062 400 / 1452 : pp = 137.4178924560547 410 / 1452 : pp = 137.82632446289062 420 / 1452 : pp = 138.17567443847656 430 / 1452 : pp = 138.11863708496094 440 / 1452 : pp = 138.215087890625 450 / 1452 : pp = 137.9976348876953 460 / 1452 : pp = 137.6929168701172 470 / 1452 : pp = 137.25416564941406 480 / 1452 : pp = 136.75140380859375 490 / 1452 : pp = 136.51712036132812 500 / 1452 : pp = 136.0896453857422 510 / 1452 : pp = 135.97048950195312 520 / 1452 : pp = 135.7760009765625 530 / 1452 : pp = 135.50389099121094 540 / 1452 : pp = 135.01437377929688 550 / 1452 : pp = 134.7666015625 560 / 1452 : pp = 134.48973083496094 570 / 1452 : pp = 134.22853088378906 580 / 1452 : pp = 133.88455200195312 590 / 1452 : pp = 133.5808868408203 600 / 1452 : pp = 133.22975158691406 610 / 1452 : pp = 132.99591064453125 620 / 1452 : pp = 132.79502868652344 630 / 1452 : pp = 132.5094451904297 640 / 1452 : pp = 132.62892150878906 650 / 1452 : pp = 132.63499450683594 660 / 1452 : pp = 132.7379913330078 670 / 1452 : pp = 132.79046630859375 680 / 1452 : pp = 132.85842895507812 690 / 1452 : pp = 132.80364990234375 700 / 1452 : pp = 132.80477905273438 710 / 1452 : pp = 132.90170288085938 720 / 1452 : pp = 132.92971801757812 730 / 1452 : pp = 132.9019012451172 740 / 1452 : pp = 133.04811096191406 750 / 1452 : pp = 133.10877990722656 760 / 1452 : pp = 133.19189453125 770 / 1452 : pp = 133.3564910888672 780 / 1452 : pp = 133.54000854492188 790 / 1452 : pp = 133.69239807128906 800 / 1452 : pp = 133.68495178222656 810 / 1452 : pp = 133.67971801757812 820 / 1452 : pp = 133.7035675048828 830 / 1452 : pp = 133.89329528808594 840 / 1452 : pp = 133.850341796875 850 / 1452 : pp = 133.90390014648438 860 / 1452 : pp = 133.9090118408203 870 / 1452 : pp = 133.89974975585938 880 / 1452 : pp = 134.0077667236328 890 / 1452 : pp = 134.03485107421875 900 / 1452 : pp = 134.0261688232422 910 / 1452 : pp = 134.10255432128906 920 / 1452 : pp = 134.17291259765625 930 / 1452 : pp = 134.14796447753906 940 / 1452 : pp = 134.20925903320312 950 / 1452 : pp = 134.19281005859375 960 / 1452 : pp = 134.17745971679688 970 / 1452 : pp = 134.18653869628906 980 / 1452 : pp = 134.03192138671875 990 / 1452 : pp = 133.94349670410156 1000 / 1452 : pp = 133.79685974121094 1010 / 1452 : pp = 133.8438262939453 1020 / 1452 : pp = 133.9608612060547 1030 / 1452 : pp = 133.93934631347656 1040 / 1452 : pp = 134.02833557128906 1050 / 1452 : pp = 134.01734924316406 1060 / 1452 : pp = 133.95346069335938 1070 / 1452 : pp = 134.10205078125 1080 / 1452 : pp = 134.2030487060547 1090 / 1452 : pp = 134.23696899414062 1100 / 1452 : pp = 134.2230224609375 1110 / 1452 : pp = 134.0829315185547 1120 / 1452 : pp = 133.980224609375 1130 / 1452 : pp = 133.83815002441406 1140 / 1452 : pp = 133.8366241455078 1150 / 1452 : pp = 133.92108154296875 1160 / 1452 : pp = 133.94375610351562 1170 / 1452 : pp = 133.9360809326172 1180 / 1452 : pp = 133.99684143066406 1190 / 1452 : pp = 134.0944366455078 1200 / 1452 : pp = 134.11676025390625 1210 / 1452 : pp = 134.0911102294922 1220 / 1452 : pp = 134.22763061523438 1230 / 1452 : pp = 134.38043212890625 1240 / 1452 : pp = 134.39817810058594 1250 / 1452 : pp = 134.5367431640625 1260 / 1452 : pp = 134.593017578125 1270 / 1452 : pp = 134.61497497558594 1280 / 1452 : pp = 134.6423797607422 1290 / 1452 : pp = 134.64340209960938 1300 / 1452 : pp = 134.68026733398438 1310 / 1452 : pp = 134.73556518554688 1320 / 1452 : pp = 134.69021606445312 1330 / 1452 : pp = 134.66131591796875 1340 / 1452 : pp = 134.69393920898438 1350 / 1452 : pp = 134.7328643798828 1360 / 1452 : pp = 134.79405212402344 1370 / 1452 : pp = 134.71237182617188 1380 / 1452 : pp = 134.6885528564453 1390 / 1452 : pp = 134.65110778808594 1400 / 1452 : pp = 134.59584045410156 1410 / 1452 : pp = 134.6193389892578 1420 / 1452 : pp = 134.58338928222656 1430 / 1452 : pp = 134.559326171875 1440 / 1452 : pp = 134.59507751464844 1450 / 1452 : pp = 134.59365844726562 0 / 115 : pp = 226.0741729736328 10 / 115 : pp = 207.00494384765625 20 / 115 : pp = 209.26976013183594 30 / 115 : pp = 206.44662475585938 40 / 115 : pp = 205.47268676757812 50 / 115 : pp = 200.7876739501953 60 / 115 : pp = 200.13414001464844 70 / 115 : pp = 196.35549926757812 80 / 115 : pp = 194.10777282714844 90 / 115 : pp = 191.47467041015625 100 / 115 : pp = 186.61351013183594 110 / 115 : pp = 184.30374145507812 Training perplexity: 134.57826232910156 Validation perplexity:183.8900146484375 Total time : 45.410256147384644 Epoch 13 0 / 1452 : pp = 169.39393615722656 10 / 1452 : pp = 150.13232421875 20 / 1452 : pp = 147.60450744628906 30 / 1452 : pp = 144.64317321777344 40 / 1452 : pp = 146.47427368164062 50 / 1452 : pp = 143.929443359375 60 / 1452 : pp = 142.8344268798828 70 / 1452 : pp = 143.45248413085938 80 / 1452 : pp = 142.5418701171875 90 / 1452 : pp = 141.6178436279297 100 / 1452 : pp = 140.70127868652344 110 / 1452 : pp = 139.2852325439453 120 / 1452 : pp = 138.8017120361328 130 / 1452 : pp = 137.85629272460938 140 / 1452 : pp = 136.51718139648438 150 / 1452 : pp = 136.03619384765625 160 / 1452 : pp = 136.154296875 170 / 1452 : pp = 135.67037963867188 180 / 1452 : pp = 135.0376739501953 190 / 1452 : pp = 134.9230499267578 200 / 1452 : pp = 135.4241180419922 210 / 1452 : pp = 135.24581909179688 220 / 1452 : pp = 135.37957763671875 230 / 1452 : pp = 135.67652893066406 240 / 1452 : pp = 135.4161834716797 250 / 1452 : pp = 134.90895080566406 260 / 1452 : pp = 134.46754455566406 270 / 1452 : pp = 133.68577575683594 280 / 1452 : pp = 133.86770629882812 290 / 1452 : pp = 134.18475341796875 300 / 1452 : pp = 134.39132690429688 310 / 1452 : pp = 134.19985961914062 320 / 1452 : pp = 134.37998962402344 330 / 1452 : pp = 134.5557403564453 340 / 1452 : pp = 134.00686645507812 350 / 1452 : pp = 134.27749633789062 360 / 1452 : pp = 134.20286560058594 370 / 1452 : pp = 134.042724609375 380 / 1452 : pp = 133.74398803710938 390 / 1452 : pp = 133.83584594726562 400 / 1452 : pp = 133.64382934570312 410 / 1452 : pp = 134.02366638183594 420 / 1452 : pp = 134.35415649414062 430 / 1452 : pp = 134.310546875 440 / 1452 : pp = 134.3634490966797 450 / 1452 : pp = 134.15602111816406 460 / 1452 : pp = 133.86578369140625 470 / 1452 : pp = 133.43414306640625 480 / 1452 : pp = 132.90310668945312 490 / 1452 : pp = 132.646240234375 500 / 1452 : pp = 132.1982421875 510 / 1452 : pp = 132.04200744628906 520 / 1452 : pp = 131.86940002441406 530 / 1452 : pp = 131.59841918945312 540 / 1452 : pp = 131.12356567382812 550 / 1452 : pp = 130.887939453125 560 / 1452 : pp = 130.6210174560547 570 / 1452 : pp = 130.37826538085938 580 / 1452 : pp = 130.0374755859375 590 / 1452 : pp = 129.75979614257812 600 / 1452 : pp = 129.38308715820312 610 / 1452 : pp = 129.16685485839844 620 / 1452 : pp = 129.0115509033203 630 / 1452 : pp = 128.75152587890625 640 / 1452 : pp = 128.87295532226562 650 / 1452 : pp = 128.88734436035156 660 / 1452 : pp = 128.98275756835938 670 / 1452 : pp = 129.0487060546875 680 / 1452 : pp = 129.11013793945312 690 / 1452 : pp = 129.0646514892578 700 / 1452 : pp = 129.06280517578125 710 / 1452 : pp = 129.1343994140625 720 / 1452 : pp = 129.18582153320312 730 / 1452 : pp = 129.15138244628906 740 / 1452 : pp = 129.29811096191406 750 / 1452 : pp = 129.339599609375 760 / 1452 : pp = 129.4257354736328 770 / 1452 : pp = 129.61631774902344 780 / 1452 : pp = 129.802734375 790 / 1452 : pp = 129.96804809570312 800 / 1452 : pp = 129.95187377929688 810 / 1452 : pp = 129.92417907714844 820 / 1452 : pp = 129.9774627685547 830 / 1452 : pp = 130.1638946533203 840 / 1452 : pp = 130.13095092773438 850 / 1452 : pp = 130.16595458984375 860 / 1452 : pp = 130.173828125 870 / 1452 : pp = 130.170166015625 880 / 1452 : pp = 130.27032470703125 890 / 1452 : pp = 130.3022003173828 900 / 1452 : pp = 130.3071746826172 910 / 1452 : pp = 130.37939453125 920 / 1452 : pp = 130.46229553222656 930 / 1452 : pp = 130.43846130371094 940 / 1452 : pp = 130.50889587402344 950 / 1452 : pp = 130.50086975097656 960 / 1452 : pp = 130.4833221435547 970 / 1452 : pp = 130.50814819335938 980 / 1452 : pp = 130.35577392578125 990 / 1452 : pp = 130.26759338378906 1000 / 1452 : pp = 130.1064453125 1010 / 1452 : pp = 130.1472625732422 1020 / 1452 : pp = 130.27169799804688 1030 / 1452 : pp = 130.25100708007812 1040 / 1452 : pp = 130.30816650390625 1050 / 1452 : pp = 130.29803466796875 1060 / 1452 : pp = 130.2242431640625 1070 / 1452 : pp = 130.35906982421875 1080 / 1452 : pp = 130.45103454589844 1090 / 1452 : pp = 130.49838256835938 1100 / 1452 : pp = 130.484130859375 1110 / 1452 : pp = 130.35316467285156 1120 / 1452 : pp = 130.24697875976562 1130 / 1452 : pp = 130.10804748535156 1140 / 1452 : pp = 130.1076202392578 1150 / 1452 : pp = 130.195068359375 1160 / 1452 : pp = 130.19674682617188 1170 / 1452 : pp = 130.18321228027344 1180 / 1452 : pp = 130.24623107910156 1190 / 1452 : pp = 130.33905029296875 1200 / 1452 : pp = 130.3650360107422 1210 / 1452 : pp = 130.34588623046875 1220 / 1452 : pp = 130.4850616455078 1230 / 1452 : pp = 130.63160705566406 1240 / 1452 : pp = 130.64674377441406 1250 / 1452 : pp = 130.77078247070312 1260 / 1452 : pp = 130.8397674560547 1270 / 1452 : pp = 130.8511199951172 1280 / 1452 : pp = 130.88967895507812 1290 / 1452 : pp = 130.9040985107422 1300 / 1452 : pp = 130.93511962890625 1310 / 1452 : pp = 130.9759063720703 1320 / 1452 : pp = 130.92800903320312 1330 / 1452 : pp = 130.9105224609375 1340 / 1452 : pp = 130.929443359375 1350 / 1452 : pp = 130.96153259277344 1360 / 1452 : pp = 131.02381896972656 1370 / 1452 : pp = 130.9545440673828 1380 / 1452 : pp = 130.9344940185547 1390 / 1452 : pp = 130.9055938720703 1400 / 1452 : pp = 130.85386657714844 1410 / 1452 : pp = 130.8874969482422 1420 / 1452 : pp = 130.85928344726562 1430 / 1452 : pp = 130.83995056152344 1440 / 1452 : pp = 130.86659240722656 1450 / 1452 : pp = 130.86839294433594 0 / 115 : pp = 227.78428649902344 10 / 115 : pp = 207.609619140625 20 / 115 : pp = 209.92459106445312 30 / 115 : pp = 206.96240234375 40 / 115 : pp = 205.9295654296875 50 / 115 : pp = 201.0296630859375 60 / 115 : pp = 200.38059997558594 70 / 115 : pp = 196.55764770507812 80 / 115 : pp = 194.31735229492188 90 / 115 : pp = 191.66146850585938 100 / 115 : pp = 186.70437622070312 110 / 115 : pp = 184.3171844482422 Training perplexity: 130.85043334960938 Validation perplexity:183.88186645507812 Total time : 45.345656394958496 Epoch 14 0 / 1452 : pp = 164.82191467285156 10 / 1452 : pp = 146.39089965820312 20 / 1452 : pp = 142.93240356445312 30 / 1452 : pp = 140.3113555908203 40 / 1452 : pp = 142.39939880371094 50 / 1452 : pp = 139.70162963867188 60 / 1452 : pp = 138.73023986816406 70 / 1452 : pp = 139.2675018310547 80 / 1452 : pp = 138.47824096679688 90 / 1452 : pp = 137.40432739257812 100 / 1452 : pp = 136.47793579101562 110 / 1452 : pp = 135.2294464111328 120 / 1452 : pp = 134.80728149414062 130 / 1452 : pp = 133.89822387695312 140 / 1452 : pp = 132.54141235351562 150 / 1452 : pp = 132.10025024414062 160 / 1452 : pp = 132.21829223632812 170 / 1452 : pp = 131.8765106201172 180 / 1452 : pp = 131.37515258789062 190 / 1452 : pp = 131.31622314453125 200 / 1452 : pp = 131.78297424316406 210 / 1452 : pp = 131.5507354736328 220 / 1452 : pp = 131.7002410888672 230 / 1452 : pp = 131.9277801513672 240 / 1452 : pp = 131.72166442871094 250 / 1452 : pp = 131.225830078125 260 / 1452 : pp = 130.7496337890625 270 / 1452 : pp = 129.9896697998047 280 / 1452 : pp = 130.10594177246094 290 / 1452 : pp = 130.41644287109375 300 / 1452 : pp = 130.5982208251953 310 / 1452 : pp = 130.36329650878906 320 / 1452 : pp = 130.5633544921875 330 / 1452 : pp = 130.77252197265625 340 / 1452 : pp = 130.273193359375 350 / 1452 : pp = 130.47889709472656 360 / 1452 : pp = 130.4348602294922 370 / 1452 : pp = 130.28126525878906 380 / 1452 : pp = 130.02786254882812 390 / 1452 : pp = 130.1564483642578 400 / 1452 : pp = 129.98440551757812 410 / 1452 : pp = 130.37721252441406 420 / 1452 : pp = 130.71859741210938 430 / 1452 : pp = 130.65939331054688 440 / 1452 : pp = 130.72987365722656 450 / 1452 : pp = 130.56272888183594 460 / 1452 : pp = 130.28195190429688 470 / 1452 : pp = 129.90936279296875 480 / 1452 : pp = 129.42857360839844 490 / 1452 : pp = 129.18077087402344 500 / 1452 : pp = 128.7588348388672 510 / 1452 : pp = 128.6303253173828 520 / 1452 : pp = 128.47616577148438 530 / 1452 : pp = 128.21148681640625 540 / 1452 : pp = 127.7218017578125 550 / 1452 : pp = 127.50067138671875 560 / 1452 : pp = 127.27574157714844 570 / 1452 : pp = 127.05399322509766 580 / 1452 : pp = 126.73983001708984 590 / 1452 : pp = 126.43692779541016 600 / 1452 : pp = 126.06050109863281 610 / 1452 : pp = 125.82952880859375 620 / 1452 : pp = 125.66295623779297 630 / 1452 : pp = 125.39354705810547 640 / 1452 : pp = 125.49463653564453 650 / 1452 : pp = 125.48816680908203 660 / 1452 : pp = 125.58712005615234 670 / 1452 : pp = 125.65978240966797 680 / 1452 : pp = 125.71456146240234 690 / 1452 : pp = 125.66937255859375 700 / 1452 : pp = 125.65900421142578 710 / 1452 : pp = 125.7271499633789 720 / 1452 : pp = 125.77758026123047 730 / 1452 : pp = 125.74129486083984 740 / 1452 : pp = 125.8759765625 750 / 1452 : pp = 125.91793823242188 760 / 1452 : pp = 125.99595642089844 770 / 1452 : pp = 126.18113708496094 780 / 1452 : pp = 126.35147094726562 790 / 1452 : pp = 126.50797271728516 800 / 1452 : pp = 126.49759674072266 810 / 1452 : pp = 126.48113250732422 820 / 1452 : pp = 126.52528381347656 830 / 1452 : pp = 126.705810546875 840 / 1452 : pp = 126.67517852783203 850 / 1452 : pp = 126.74176025390625 860 / 1452 : pp = 126.74151611328125 870 / 1452 : pp = 126.73414611816406 880 / 1452 : pp = 126.83026885986328 890 / 1452 : pp = 126.88519287109375 900 / 1452 : pp = 126.88053894042969 910 / 1452 : pp = 126.97138214111328 920 / 1452 : pp = 127.04660034179688 930 / 1452 : pp = 127.03763580322266 940 / 1452 : pp = 127.1126480102539 950 / 1452 : pp = 127.09610748291016 960 / 1452 : pp = 127.0873794555664 970 / 1452 : pp = 127.10343933105469 980 / 1452 : pp = 126.96441650390625 990 / 1452 : pp = 126.88519287109375 1000 / 1452 : pp = 126.7336654663086 1010 / 1452 : pp = 126.77796936035156 1020 / 1452 : pp = 126.89826202392578 1030 / 1452 : pp = 126.88761138916016 1040 / 1452 : pp = 126.95309448242188 1050 / 1452 : pp = 126.96478271484375 1060 / 1452 : pp = 126.89324188232422 1070 / 1452 : pp = 127.03242492675781 1080 / 1452 : pp = 127.13228607177734 1090 / 1452 : pp = 127.173095703125 1100 / 1452 : pp = 127.15975189208984 1110 / 1452 : pp = 127.0392074584961 1120 / 1452 : pp = 126.94032287597656 1130 / 1452 : pp = 126.80693054199219 1140 / 1452 : pp = 126.81315612792969 1150 / 1452 : pp = 126.90467834472656 1160 / 1452 : pp = 126.91236114501953 1170 / 1452 : pp = 126.90897369384766 1180 / 1452 : pp = 126.98052215576172 1190 / 1452 : pp = 127.07483673095703 1200 / 1452 : pp = 127.10216522216797 1210 / 1452 : pp = 127.08258819580078 1220 / 1452 : pp = 127.22943878173828 1230 / 1452 : pp = 127.38563537597656 1240 / 1452 : pp = 127.40538024902344 1250 / 1452 : pp = 127.53369140625 1260 / 1452 : pp = 127.59293365478516 1270 / 1452 : pp = 127.61489868164062 1280 / 1452 : pp = 127.6484375 1290 / 1452 : pp = 127.65257263183594 1300 / 1452 : pp = 127.69329833984375 1310 / 1452 : pp = 127.74549102783203 1320 / 1452 : pp = 127.7043228149414 1330 / 1452 : pp = 127.6866683959961 1340 / 1452 : pp = 127.70913696289062 1350 / 1452 : pp = 127.73233795166016 1360 / 1452 : pp = 127.7855224609375 1370 / 1452 : pp = 127.71918487548828 1380 / 1452 : pp = 127.69987487792969 1390 / 1452 : pp = 127.6697998046875 1400 / 1452 : pp = 127.61137390136719 1410 / 1452 : pp = 127.6404037475586 1420 / 1452 : pp = 127.61094665527344 1430 / 1452 : pp = 127.58216857910156 1440 / 1452 : pp = 127.61477661132812 1450 / 1452 : pp = 127.61964416503906 0 / 115 : pp = 228.21578979492188 10 / 115 : pp = 208.11244201660156 20 / 115 : pp = 210.688232421875 30 / 115 : pp = 207.62408447265625 40 / 115 : pp = 206.45184326171875 50 / 115 : pp = 201.52760314941406 60 / 115 : pp = 200.7784881591797 70 / 115 : pp = 196.83067321777344 80 / 115 : pp = 194.6357879638672 90 / 115 : pp = 191.9783935546875 100 / 115 : pp = 186.8787841796875 110 / 115 : pp = 184.35252380371094 Training perplexity: 127.60413360595703 Validation perplexity:183.8877410888672 Total time : 41.6636528968811 Epoch 15 0 / 1452 : pp = 156.81654357910156 10 / 1452 : pp = 142.1070556640625 20 / 1452 : pp = 139.55076599121094 30 / 1452 : pp = 136.63551330566406 40 / 1452 : pp = 138.5840606689453 50 / 1452 : pp = 136.052734375 60 / 1452 : pp = 134.93019104003906 70 / 1452 : pp = 135.65206909179688 80 / 1452 : pp = 135.2620086669922 90 / 1452 : pp = 134.314697265625 100 / 1452 : pp = 133.4916229248047 110 / 1452 : pp = 132.26052856445312 120 / 1452 : pp = 131.7714080810547 130 / 1452 : pp = 130.77365112304688 140 / 1452 : pp = 129.5411834716797 150 / 1452 : pp = 129.0791778564453 160 / 1452 : pp = 129.21920776367188 170 / 1452 : pp = 128.7528839111328 180 / 1452 : pp = 128.22279357910156 190 / 1452 : pp = 128.18177795410156 200 / 1452 : pp = 128.58758544921875 210 / 1452 : pp = 128.3906707763672 220 / 1452 : pp = 128.5266571044922 230 / 1452 : pp = 128.80563354492188 240 / 1452 : pp = 128.61886596679688 250 / 1452 : pp = 128.13172912597656 260 / 1452 : pp = 127.69220733642578 270 / 1452 : pp = 126.96150970458984 280 / 1452 : pp = 127.04702758789062 290 / 1452 : pp = 127.33565521240234 300 / 1452 : pp = 127.55929565429688 310 / 1452 : pp = 127.38514709472656 320 / 1452 : pp = 127.52171325683594 330 / 1452 : pp = 127.68690490722656 340 / 1452 : pp = 127.18340301513672 350 / 1452 : pp = 127.4073257446289 360 / 1452 : pp = 127.30432891845703 370 / 1452 : pp = 127.17618560791016 380 / 1452 : pp = 126.92579650878906 390 / 1452 : pp = 127.02473449707031 400 / 1452 : pp = 126.8515625 410 / 1452 : pp = 127.211669921875 420 / 1452 : pp = 127.51788330078125 430 / 1452 : pp = 127.47386169433594 440 / 1452 : pp = 127.57164001464844 450 / 1452 : pp = 127.3601303100586 460 / 1452 : pp = 127.09434509277344 470 / 1452 : pp = 126.71922302246094 480 / 1452 : pp = 126.24349212646484 490 / 1452 : pp = 125.98778533935547 500 / 1452 : pp = 125.59526824951172 510 / 1452 : pp = 125.4450912475586 520 / 1452 : pp = 125.29247283935547 530 / 1452 : pp = 125.03536224365234 540 / 1452 : pp = 124.5813980102539 550 / 1452 : pp = 124.33724212646484 560 / 1452 : pp = 124.08995819091797 570 / 1452 : pp = 123.86637878417969 580 / 1452 : pp = 123.53152465820312 590 / 1452 : pp = 123.20321655273438 600 / 1452 : pp = 122.85673522949219 610 / 1452 : pp = 122.64250946044922 620 / 1452 : pp = 122.4958724975586 630 / 1452 : pp = 122.22386169433594 640 / 1452 : pp = 122.31143188476562 650 / 1452 : pp = 122.30093383789062 660 / 1452 : pp = 122.39427947998047 670 / 1452 : pp = 122.45440673828125 680 / 1452 : pp = 122.51146697998047 690 / 1452 : pp = 122.4854736328125 700 / 1452 : pp = 122.48600006103516 710 / 1452 : pp = 122.56084442138672 720 / 1452 : pp = 122.59059143066406 730 / 1452 : pp = 122.55529022216797 740 / 1452 : pp = 122.69409942626953 750 / 1452 : pp = 122.76456451416016 760 / 1452 : pp = 122.84437561035156 770 / 1452 : pp = 123.02527618408203 780 / 1452 : pp = 123.20509338378906 790 / 1452 : pp = 123.36305236816406 800 / 1452 : pp = 123.36852264404297 810 / 1452 : pp = 123.36799621582031 820 / 1452 : pp = 123.39976501464844 830 / 1452 : pp = 123.59362030029297 840 / 1452 : pp = 123.56946563720703 850 / 1452 : pp = 123.63800811767578 860 / 1452 : pp = 123.63983917236328 870 / 1452 : pp = 123.64148712158203 880 / 1452 : pp = 123.7568588256836 890 / 1452 : pp = 123.7885513305664 900 / 1452 : pp = 123.79640197753906 910 / 1452 : pp = 123.86153411865234 920 / 1452 : pp = 123.92941284179688 930 / 1452 : pp = 123.9125747680664 940 / 1452 : pp = 123.95559692382812 950 / 1452 : pp = 123.93928527832031 960 / 1452 : pp = 123.94294738769531 970 / 1452 : pp = 123.95547485351562 980 / 1452 : pp = 123.8229751586914 990 / 1452 : pp = 123.73727416992188 1000 / 1452 : pp = 123.59091186523438 1010 / 1452 : pp = 123.634765625 1020 / 1452 : pp = 123.76506042480469 1030 / 1452 : pp = 123.75485229492188 1040 / 1452 : pp = 123.807861328125 1050 / 1452 : pp = 123.79156494140625 1060 / 1452 : pp = 123.73054504394531 1070 / 1452 : pp = 123.8615951538086 1080 / 1452 : pp = 123.96564483642578 1090 / 1452 : pp = 124.02104187011719 1100 / 1452 : pp = 124.012939453125 1110 / 1452 : pp = 123.87582397460938 1120 / 1452 : pp = 123.775390625 1130 / 1452 : pp = 123.63182067871094 1140 / 1452 : pp = 123.62391662597656 1150 / 1452 : pp = 123.71013641357422 1160 / 1452 : pp = 123.72423553466797 1170 / 1452 : pp = 123.71726989746094 1180 / 1452 : pp = 123.79032897949219 1190 / 1452 : pp = 123.87883758544922 1200 / 1452 : pp = 123.9125747680664 1210 / 1452 : pp = 123.90140533447266 1220 / 1452 : pp = 124.03245544433594 1230 / 1452 : pp = 124.19799041748047 1240 / 1452 : pp = 124.21469116210938 1250 / 1452 : pp = 124.34103393554688 1260 / 1452 : pp = 124.4041976928711 1270 / 1452 : pp = 124.42852020263672 1280 / 1452 : pp = 124.46656036376953 1290 / 1452 : pp = 124.4811019897461 1300 / 1452 : pp = 124.52384185791016 1310 / 1452 : pp = 124.57533264160156 1320 / 1452 : pp = 124.5398178100586 1330 / 1452 : pp = 124.52598571777344 1340 / 1452 : pp = 124.53311157226562 1350 / 1452 : pp = 124.57759094238281 1360 / 1452 : pp = 124.63385772705078 1370 / 1452 : pp = 124.58133697509766 1380 / 1452 : pp = 124.55769348144531 1390 / 1452 : pp = 124.54011535644531 1400 / 1452 : pp = 124.4884033203125 1410 / 1452 : pp = 124.51226806640625 1420 / 1452 : pp = 124.49683380126953 1430 / 1452 : pp = 124.4754638671875 1440 / 1452 : pp = 124.50164031982422 1450 / 1452 : pp = 124.50894165039062 0 / 115 : pp = 230.8488006591797 10 / 115 : pp = 209.2509002685547 20 / 115 : pp = 211.68577575683594 30 / 115 : pp = 208.44056701660156 40 / 115 : pp = 207.2039337158203 50 / 115 : pp = 202.1859588623047 60 / 115 : pp = 201.34739685058594 70 / 115 : pp = 197.4251251220703 80 / 115 : pp = 195.2623291015625 90 / 115 : pp = 192.592529296875 100 / 115 : pp = 187.39553833007812 110 / 115 : pp = 184.791259765625 Training perplexity: 124.4933853149414 Validation perplexity:184.32510375976562 Total time : 40.856229066848755 0 / 128 : pp = 184.6475067138672 10 / 128 : pp = 176.8856964111328 20 / 128 : pp = 164.3444366455078 30 / 128 : pp = 167.85472106933594 40 / 128 : pp = 169.25367736816406 50 / 128 : pp = 168.86561584472656 60 / 128 : pp = 168.11801147460938 70 / 128 : pp = 165.4105224609375 80 / 128 : pp = 162.91146850585938 90 / 128 : pp = 161.29742431640625 100 / 128 : pp = 162.45989990234375 110 / 128 : pp = 162.6834716796875 120 / 128 : pp = 164.3359832763672 =-==-==-==-==-= Test perplexity: 164.0149383544922 =-==-==-==-==-=
更详细的内容请参考下面链接
https://github.com/weizhenzhao/cs224d_nlp_problem_set2
今天将的还是cs224d 的problem set2 的第三部分习题，
原来国外大学的系统难度真的如此之大，相比之下还是默默地再天朝继续搬砖吧
下面讲述一下RNN语言建模的数学公式:
给出一串连续的词x1,x2...xt关于预测其后面紧跟的词xt+1的建模方式是:
vj是词库中的某个词。实现一个循环神经网络，此网络利用隐层中的反馈信息对"历史记录"x1,x2...xt进行建模:
$h^{(0)}=h_{0}\epsilon R^{D_{h}}$是隐藏层的初始化向量
$x^{(t)}L$是以$x^{(t)}$one-hot行向量与嵌入矩阵L的乘积
这个one-hot行向量就是当前处理词汇的索引
是词嵌入矩阵,
$L$是词嵌入矩阵
$I$是输入词表征矩阵
$H$是隐藏转换矩阵
$U$是输出词表征矩阵
$b_{1}$ $b_{2}$是偏置值
$d$是词嵌入的维数
|V|代表词库的规模
$D_{h}$是隐层的维数
输出向量
是面向整个词库的概率分布，我们需要最优化交叉熵（非正则化的）的损失率：
使用困惑度来评估语言模型的性能，其定义形式如下：
梯度:
该模型中各个变量进行最优化迭代的时候的梯度如下所示:
初始化所有的上面这些需要训练的参数的值
然后通过对每一个词进行训练，安装上述公司求出每个参数的导数值
然后使用梯度下降方法对其进行更新
将新得到的参数代入到模型中，如果损失的值小于初始设定的值则停止迭代，否则继续进行迭代
下面是一张RNNLM的结构图
上面这张是第二层RNN节点的结构图
上面这张是在RNN的变量上面应用Dropout的结构，降低模型过拟合的误差，第一层RNN的dropout结构
上面这张是第一层RNN的结构图
（注意前方高能，一大批天书即将来袭）
''' Created on 2017年9月26日 @author: weizhen ''' import getpass import sys import time import numpy as np from copy import deepcopy from utils import calculate_perplexity, get_ptb_dataset, Vocab from utils import ptb_iterator, sample import tensorflow as tf from model import LanguageModel from tensorflow.contrib.legacy_seq2seq.python.ops.seq2seq import sequence_loss class Config(object): """储存超参数和数据信息""" batch_size = 64 embed_size = 50 hidden_size = 100 num_steps = 10 max_epochs = 16 early_stopping = 2 dropout = 0.9 lr = 0.001 class RNNLM_Model(LanguageModel): def load_data(self, debug=False): """加载词向量并且训练 train/dev/test 数据""" self.vocab = Vocab() self.vocab.construct(get_ptb_dataset('train')) self.encoded_train = np.array([self.vocab.encode(word) for word in get_ptb_dataset('train')], dtype=np.int32) self.encoded_valid = np.array([self.vocab.encode(word) for word in get_ptb_dataset('valid')], dtype=np.int32) self.encoded_test = np.array([self.vocab.encode(word) for word in get_ptb_dataset('test')]) if debug: num_debug = 1024 self.encoded_train = self.encoded_train[:num_debug] self.encoded_valid = self.encoded_valid[:num_debug] self.encoded_test = self.encoded_test[:num_debug] def add_placeholders(self): """生成placeholder 变量来表示输入的 tensors 这些placeholder 被用来在模型的其他地方被填充 并且在训练的过程中会被填充 input_placeholder:Input placeholder shape (None,num_steps),type tf.int32 labels_placeholder:label placeholder shape (None,num_steps) type tf.float32 dropout_placeholder:dropput value placeholder (scalar), type tf.float32 """ self.input_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Input') self.labels_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Target') self.dropout_placeholder = tf.placeholder(tf.float32, name='Dropout') def add_embedding(self): """添加词嵌入层 Hint : 这一层应该用input_placeholder 来索引词嵌入 Hint : 你或许能发现tf.nn.embedding_lookup 是有用的 Hint : 你或许能发现tf.split , tf.squeeze 是有用的在构造tensor 的输入的时候 Hint : 下面是你需要创建的变量的维度 L:(len(self.vocab),embed_size) Returns: inputs:一个训练次数的列表,每一个元素应该是 一个张量 大小是 (batch_size,embed_size) tf.split(dimension,num_split,input) dimension表示输入张量的哪一个维度, 如果是0就表示对第0维度进行切割, num_split就是切割的数量, 如果是2就表示输入张量被切成2份, 每一份是一个列表 tf.squeeze(input,squeeze_dims=None,name=None) 从tensor中删除所有大小是1的维度 example: t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t))==>[2,3] t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t,[2,4]))==>[1,2,3,1] tf.nn.embedding_lookup 将词的索引映射到词的向量 """ with tf.device('/cpu:0'): embedding = tf.get_variable('Embedding', [len(self.vocab), self.config.embed_size], trainable=True) inputs = tf.nn.embedding_lookup(embedding, self.input_placeholder) inputs = [tf.squeeze(x, [1]) for x in tf.split(inputs, self.config.num_steps, 1)] return inputs def add_projection(self, rnn_outputs): """添加一个投影层 投影层将隐藏层的表示变换到整个词向量上的分布式表示 Hint:下面是你需要去创建的维度 U(hidden_size,len(vocab)) b_2:(len(vocab),) 参数: rnn_outputs:一个训练次数的列表，每一个元素应该是一个张量 大小是(batch_size,embed_size) Returns: outputs:一个长度的列表，每一个元素是一个张量(batch_size,len(vocab)) """ with tf.variable_scope('Projection'): U = tf.get_variable('Matrix', [self.config.hidden_size, len(self.vocab)]) proj_b = tf.get_variable('Bias', [len(self.vocab)]) outputs = [tf.matmul(o, U) + proj_b for o in rnn_outputs] return outputs def add_loss_op(self, output): """将损失添加到目标函数上面 Hint:使用tensorflow.python.ops.seq2seq.sequence_loss 来实现序列损失 参数: 输出:一个张量 大小是 (None,self.vocab) 返回: 损失：一个0-d大小的张量 """ all_ones = [tf.ones([self.config.batch_size * self.config.num_steps])] cross_entropy = sequence_loss([output], [tf.reshape(self.labels_placeholder, [-1])], all_ones, len(self.vocab)) tf.add_to_collection('total_loss', cross_entropy) loss = tf.add_n(tf.get_collection('total_loss')) return loss def add_training_op(self, loss): """将目标损失添加到计算图上 创建一个优化器并且应用梯度下降到所有的训练变量上面 Hint:使用tf.train.AdamOptimizer 对于这个模型 使用optimizer.minimize() 会返回一个train_op的对象 参数: loss: 损失张量，来自于cross_entropy_loss 交叉熵损失 返回: train_op:训练的目标 """ with tf.variable_scope("Optimizer") as scope: train_op = tf.train.AdamOptimizer(self.config.lr).minimize(loss) return train_op def __init__(self, config): self.config = config self.load_data(debug=False) self.add_placeholders() self.inputs = self.add_embedding() self.rnn_outputs = self.add_model(self.inputs) self.outputs = self.add_projection(self.rnn_outputs) # 我们想去检验下一个词预测得多好 # 我们把o转变成float64 位 因为如果不这样就会有数值问题 # sum(output of softmax) = 1.00000298179 并且不是 1 self.predictions = [tf.nn.softmax(tf.cast(o, 'float64')) for o in self.outputs] # 将输出值转变成 len(vocab) 的大小 output = tf.reshape(tf.concat(self.outputs, 1), [-1, len(self.vocab)]) self.calculate_loss = self.add_loss_op(output) self.train_step = self.add_training_op(self.calculate_loss) def add_model(self, inputs): """创建RNN LM 模型 在下面的实现里面你需要去实现RNN LM 模型的等式 Hint: 使用一个零向量 大小是 (batch_size,hidden_size) 作为初始的RNN的状态 Hint: 将最后RNN输出 作为实例变量 self.final_state Hint : 确保将dropout应用到 输入和输出的 变量上面 Hint : 使用变量域 RNN 来定义 RNN变量 Hint : 表现一个明显的 for-loop 在输入上面 你可以使用scope.reuse_variable() 来确定权重 在每一次迭代都是相同的 确保不会在第一次循环的时候调用这个，因为没有变量会被初始化 Hint : 下面变量的不同的维度 ， 你需要去创建的 H: (hidden_size,hidden_size) I: (embed_size,hidden_size) b_1:(hidden_size,) Args: inputs:一个记录num_steps的列表，里边的每一个元素应该是一个张量 大小是(batch_size,embed_size)的大小 Returns:返回 outputs:一个记录num_steps的列表，里面每一个元素应该是一个张量 大小是(batch_size,hidden_size) """ with tf.variable_scope('InputDropout'): inputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in inputs] with tf.variable_scope('RNN') as scope: self.initial_state = tf.zeros([self.config.batch_size, self.config.hidden_size]) state = self.initial_state rnn_outputs = [] for tstep, current_input in enumerate(inputs): if tstep > 0: scope.reuse_variables() RNN_H = tf.get_variable('HMatrix', [self.config.hidden_size, self.config.hidden_size]) RNN_I = tf.get_variable('IMatrix', [self.config.embed_size, self.config.hidden_size]) RNN_b = tf.get_variable('B', [self.config.hidden_size]) state = tf.nn.sigmoid(tf.matmul(state, RNN_H) + tf.matmul(current_input, RNN_I) + RNN_b) rnn_outputs.append(state) self.final_state = rnn_outputs[-1] with tf.variable_scope('RNNDropout'): rnn_outputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in rnn_outputs] return rnn_outputs def run_epoch(self, session, data, train_op=None, verbose=10): config = self.config dp = config.dropout if not train_op: train_op = tf.no_op() dp = 1 total_steps = sum(1 for x in ptb_iterator(data, config.batch_size, config.num_steps)) total_loss = [] state = self.initial_state.eval() for step, (x, y) in enumerate(ptb_iterator(data, config.batch_size, config.num_steps)): # 我们需要通过初始状态，并且从最终状态中抽取数据来进行填充 # RNN 合适的 历史 feed = {self.input_placeholder: x, self.labels_placeholder: y, self.initial_state: state, self.dropout_placeholder: dp } loss, state, _ = session.run([self.calculate_loss, self.final_state, train_op], feed_dict=feed) total_loss.append(loss) if verbose and step % verbose == 0: sys.stdout.write('\r{} / {} : pp = {} '.format(step, total_steps, np.exp(np.mean(total_loss)))) sys.stdout.flush() if verbose: sys.stdout.write('\r') return np.exp(np.mean(total_loss)) def generate_text(session, model, config, starting_text='<eos>', stop_length=100, stop_tokens=None, temp=1.0): """从模型自动生成文字 Hint:创建一个feed-dictionary 并且使用sess.run()方法去执行这个模型 你会需要使用model.initial_state 作为一个键传递给feed_dict Hint:得到model.final_state 和 model.predictions[-1]. 在add_model()方法中设置model.final_state 。 model.predictions 是在 __init__方法中设置的 Hint:在模型的训练中存储输出的参数值，和预测的y_pred的值 参数: Args: session : tf.Session() object model : Object of type RNNLM Model config : A Config() object starting_text:Initial text passed to model Returns: output : List of word idxs """ state = model.initial_state.eval() # Imagine tokens as a batch size of one, length of len(tokens[0]) tokens = [model.vocab.encode(word) for word in starting_text.split()] for i in range(stop_length): feed = {model.input_placeholder: [tokens[-1:]], model.initial_state: state, model.dropout_placeholder: 1} state, y_pred = session.run([model.final_state, model.predictions[-1]], feed_dict=feed) next_word_idx = sample(y_pred[0], temperature=temp) tokens.append(next_word_idx) if stop_tokens and model.vocab.decode(tokens[-1]) in stop_tokens: break output = [model.vocab.decode(word_idx) for word_idx in tokens] return output def generate_sentence(session, model, config, *args, **kwargs): """方便从模型来生成句子""" return generate_text(session, model, config, *args, stop_tokens=['<eos>'], **kwargs) def test_RNNLM(): config = Config() gen_config = deepcopy(config) gen_config.batch_size = gen_config.num_steps = 1 # 创建训练模型，并且生成模型 with tf.variable_scope('RNNLM',reuse=None) as scope: model = RNNLM_Model(config) # 这个指示gen_model来重新使用相同的变量作为以上的模型 scope.reuse_variables() gen_model = RNNLM_Model(gen_config) init = tf.global_variables_initializer() saver = tf.train.Saver() with tf.Session() as session: best_val_pp = float('inf') best_val_epoch = 0 session.run(init) for epoch in range(config.max_epochs): print('Epoch {0}'.format(epoch)) start = time.time() train_pp = model.run_epoch(session, model.encoded_train, train_op=model.train_step) valid_pp = model.run_epoch(session, model.encoded_valid) print('Training perplexity: {0}'.format(train_pp)) print('Validation perplexity:{0}'.format(valid_pp)) if valid_pp < best_val_pp: best_val_pp = valid_pp best_val_epoch = epoch saver.save(session, './ptb_rnnlm.weights') if epoch - best_val_epoch > config.early_stopping: break print('Total time : {0}'.format(time.time() - start)) saver.restore(session, 'ptb_rnnlm.weights') test_pp = model.run_epoch(session, model.encoded_test) print('=-=' * 5) print('Test perplexity: {0} '.format(test_pp)) print('=-=' * 5) starting_text = 'in palo alto' while starting_text: print(' '.join(generate_sentence(session, gen_model, gen_config, starting_text=starting_text, temp=1.0))) #starting_text = raw_input('>') if __name__ == "__main__": test_RNNLM()
（其实也不算是天书啦，比高数简单多啦，比数学分析那是简单了好几十万倍了呀）
下面是训练的Log
1380 / 1452 : pp = 266.20892333984375 1390 / 1452 : pp = 265.94439697265625 1400 / 1452 : pp = 265.66845703125 1410 / 1452 : pp = 265.5393981933594 1420 / 1452 : pp = 265.32489013671875 1430 / 1452 : pp = 265.2019348144531 1440 / 1452 : pp = 265.13720703125 1450 / 1452 : pp = 264.954833984375 0 / 115 : pp = 296.9217224121094 10 / 115 : pp = 282.02130126953125 20 / 115 : pp = 279.76800537109375 30 / 115 : pp = 276.4101257324219 40 / 115 : pp = 276.2939147949219 50 / 115 : pp = 270.73565673828125 60 / 115 : pp = 269.88134765625 70 / 115 : pp = 266.8675231933594 80 / 115 : pp = 263.6731872558594 90 / 115 : pp = 260.8569030761719 100 / 115 : pp = 256.3356628417969 110 / 115 : pp = 255.1026611328125 Training perplexity: 264.9092102050781 Validation perplexity:254.84902954101562 Total time : 41.65332388877869 Epoch 3 0 / 1452 : pp = 327.0847473144531 10 / 1452 : pp = 273.9620056152344 20 / 1452 : pp = 270.22943115234375 30 / 1452 : pp = 263.5213317871094 40 / 1452 : pp = 264.0644836425781 50 / 1452 : pp = 258.6029968261719 60 / 1452 : pp = 257.04290771484375 70 / 1452 : pp = 257.59161376953125 80 / 1452 : pp = 256.7600402832031 90 / 1452 : pp = 254.5120391845703 100 / 1452 : pp = 252.44725036621094 110 / 1452 : pp = 250.13954162597656 120 / 1452 : pp = 249.91647338867188 130 / 1452 : pp = 249.50460815429688 140 / 1452 : pp = 247.67440795898438 150 / 1452 : pp = 247.19090270996094 160 / 1452 : pp = 247.8919219970703 170 / 1452 : pp = 247.54322814941406 180 / 1452 : pp = 246.17623901367188 190 / 1452 : pp = 245.78330993652344 200 / 1452 : pp = 246.80552673339844 210 / 1452 : pp = 246.3059844970703 220 / 1452 : pp = 246.19021606445312 230 / 1452 : pp = 246.70140075683594 240 / 1452 : pp = 246.3099822998047 250 / 1452 : pp = 245.1745147705078 260 / 1452 : pp = 244.17384338378906 270 / 1452 : pp = 242.57363891601562 280 / 1452 : pp = 242.8500213623047 290 / 1452 : pp = 243.0492706298828 300 / 1452 : pp = 243.1466522216797 310 / 1452 : pp = 242.89044189453125 320 / 1452 : pp = 243.08045959472656 330 / 1452 : pp = 243.32235717773438 340 / 1452 : pp = 242.34715270996094 350 / 1452 : pp = 242.80972290039062 360 / 1452 : pp = 242.5345458984375 370 / 1452 : pp = 242.0083465576172 380 / 1452 : pp = 241.22708129882812 390 / 1452 : pp = 241.24398803710938 400 / 1452 : pp = 240.63473510742188 410 / 1452 : pp = 240.94094848632812 420 / 1452 : pp = 241.19717407226562 430 / 1452 : pp = 240.8896026611328 440 / 1452 : pp = 240.7772979736328 450 / 1452 : pp = 240.45913696289062 460 / 1452 : pp = 240.06674194335938 470 / 1452 : pp = 239.42198181152344 480 / 1452 : pp = 238.39271545410156 490 / 1452 : pp = 238.0517120361328 500 / 1452 : pp = 237.31752014160156 510 / 1452 : pp = 237.1197967529297 520 / 1452 : pp = 236.64865112304688 530 / 1452 : pp = 236.004638671875 540 / 1452 : pp = 235.192626953125 550 / 1452 : pp = 234.6700439453125 560 / 1452 : pp = 234.1914825439453 570 / 1452 : pp = 233.80899047851562 580 / 1452 : pp = 233.3753662109375 590 / 1452 : pp = 232.8699188232422 600 / 1452 : pp = 232.2629852294922 610 / 1452 : pp = 231.8668212890625 620 / 1452 : pp = 231.478515625 630 / 1452 : pp = 231.0444793701172 640 / 1452 : pp = 231.2737579345703 650 / 1452 : pp = 231.28114318847656 660 / 1452 : pp = 231.4324951171875 670 / 1452 : pp = 231.48513793945312 680 / 1452 : pp = 231.45932006835938 690 / 1452 : pp = 231.17738342285156 700 / 1452 : pp = 231.00570678710938 710 / 1452 : pp = 231.03810119628906 720 / 1452 : pp = 230.96131896972656 730 / 1452 : pp = 230.91110229492188 740 / 1452 : pp = 231.13539123535156 750 / 1452 : pp = 231.04393005371094 760 / 1452 : pp = 231.03489685058594 770 / 1452 : pp = 231.19744873046875 780 / 1452 : pp = 231.26625061035156 790 / 1452 : pp = 231.38714599609375 800 / 1452 : pp = 231.24441528320312 810 / 1452 : pp = 231.16824340820312 820 / 1452 : pp = 231.11831665039062 830 / 1452 : pp = 231.34886169433594 840 / 1452 : pp = 231.221923828125 850 / 1452 : pp = 231.2562255859375 860 / 1452 : pp = 231.26492309570312 870 / 1452 : pp = 231.1961212158203 880 / 1452 : pp = 231.30506896972656 890 / 1452 : pp = 231.24728393554688 900 / 1452 : pp = 231.15744018554688 910 / 1452 : pp = 231.20175170898438 920 / 1452 : pp = 231.25534057617188 930 / 1452 : pp = 231.09461975097656 940 / 1452 : pp = 231.12612915039062 950 / 1452 : pp = 231.0475616455078 960 / 1452 : pp = 230.86056518554688 970 / 1452 : pp = 230.80377197265625 980 / 1452 : pp = 230.4598846435547 990 / 1452 : pp = 230.24559020996094 1000 / 1452 : pp = 229.91030883789062 1010 / 1452 : pp = 229.9349822998047 1020 / 1452 : pp = 230.01470947265625 1030 / 1452 : pp = 229.8909149169922 1040 / 1452 : pp = 229.9403533935547 1050 / 1452 : pp = 229.84815979003906 1060 / 1452 : pp = 229.60377502441406 1070 / 1452 : pp = 229.74647521972656 1080 / 1452 : pp = 229.80410766601562 1090 / 1452 : pp = 229.78733825683594 1100 / 1452 : pp = 229.64549255371094 1110 / 1452 : pp = 229.26255798339844 1120 / 1452 : pp = 229.00262451171875 1130 / 1452 : pp = 228.6716766357422 1140 / 1452 : pp = 228.55067443847656 1150 / 1452 : pp = 228.61563110351562 1160 / 1452 : pp = 228.50958251953125 1170 / 1452 : pp = 228.3498992919922 1180 / 1452 : pp = 228.29786682128906 1190 / 1452 : pp = 228.33204650878906 1200 / 1452 : pp = 228.27369689941406 1210 / 1452 : pp = 228.11831665039062 1220 / 1452 : pp = 228.21775817871094 1230 / 1452 : pp = 228.3170166015625 1240 / 1452 : pp = 228.22134399414062 1250 / 1452 : pp = 228.3769073486328 1260 / 1452 : pp = 228.37527465820312 1270 / 1452 : pp = 228.33694458007812 1280 / 1452 : pp = 228.27108764648438 1290 / 1452 : pp = 228.1731414794922 1300 / 1452 : pp = 228.12200927734375 1310 / 1452 : pp = 228.10275268554688 1320 / 1452 : pp = 227.9289093017578 1330 / 1452 : pp = 227.77723693847656 1340 / 1452 : pp = 227.79623413085938 1350 / 1452 : pp = 227.7408447265625 1360 / 1452 : pp = 227.72586059570312 1370 / 1452 : pp = 227.49728393554688 1380 / 1452 : pp = 227.37940979003906 1390 / 1452 : pp = 227.20166015625 1400 / 1452 : pp = 227.018310546875 1410 / 1452 : pp = 226.95651245117188 1420 / 1452 : pp = 226.8065643310547 1430 / 1452 : pp = 226.7261199951172 1440 / 1452 : pp = 226.7193145751953 1450 / 1452 : pp = 226.61068725585938 0 / 115 : pp = 269.342041015625 10 / 115 : pp = 255.03016662597656 20 / 115 : pp = 253.8992919921875 30 / 115 : pp = 251.04025268554688 40 / 115 : pp = 250.51756286621094 50 / 115 : pp = 245.3595428466797 60 / 115 : pp = 244.4713897705078 70 / 115 : pp = 241.2674560546875 80 / 115 : pp = 238.3473663330078 90 / 115 : pp = 235.56423950195312 100 / 115 : pp = 231.2281036376953 110 / 115 : pp = 229.8423614501953 Training perplexity: 226.5760040283203 Validation perplexity:229.59939575195312 Total time : 42.202677726745605 Epoch 4 0 / 1452 : pp = 282.2423095703125 10 / 1452 : pp = 240.16258239746094 20 / 1452 : pp = 236.12203979492188 30 / 1452 : pp = 230.3953857421875 40 / 1452 : pp = 231.8789825439453 50 / 1452 : pp = 227.26612854003906 60 / 1452 : pp = 226.22061157226562 70 / 1452 : pp = 227.01885986328125 80 / 1452 : pp = 226.2459716796875 90 / 1452 : pp = 224.3211669921875 100 / 1452 : pp = 222.65615844726562 110 / 1452 : pp = 220.70326232910156 120 / 1452 : pp = 220.42288208007812 130 / 1452 : pp = 219.8100128173828 140 / 1452 : pp = 218.04432678222656 150 / 1452 : pp = 217.31639099121094 160 / 1452 : pp = 217.86349487304688 170 / 1452 : pp = 217.46597290039062 180 / 1452 : pp = 216.3349151611328 190 / 1452 : pp = 216.12240600585938 200 / 1452 : pp = 216.97842407226562 210 / 1452 : pp = 216.51014709472656 220 / 1452 : pp = 216.46751403808594 230 / 1452 : pp = 216.80126953125 240 / 1452 : pp = 216.45965576171875 250 / 1452 : pp = 215.5008544921875 260 / 1452 : pp = 214.62210083007812 270 / 1452 : pp = 213.29183959960938 280 / 1452 : pp = 213.5621337890625 290 / 1452 : pp = 213.80657958984375 300 / 1452 : pp = 213.8963165283203 310 / 1452 : pp = 213.60653686523438 320 / 1452 : pp = 213.85877990722656 330 / 1452 : pp = 214.07345581054688 340 / 1452 : pp = 213.25421142578125 350 / 1452 : pp = 213.68019104003906 360 / 1452 : pp = 213.41717529296875 370 / 1452 : pp = 213.04920959472656 380 / 1452 : pp = 212.39019775390625 390 / 1452 : pp = 212.4908905029297 400 / 1452 : pp = 212.01914978027344 410 / 1452 : pp = 212.36903381347656 420 / 1452 : pp = 212.6802520751953 430 / 1452 : pp = 212.42697143554688 440 / 1452 : pp = 212.42990112304688 450 / 1452 : pp = 212.14524841308594 460 / 1452 : pp = 211.7836151123047 470 / 1452 : pp = 211.17282104492188 480 / 1452 : pp = 210.27903747558594 490 / 1452 : pp = 209.95211791992188 500 / 1452 : pp = 209.28302001953125 510 / 1452 : pp = 209.1029815673828 520 / 1452 : pp = 208.73855590820312 530 / 1452 : pp = 208.19700622558594 540 / 1452 : pp = 207.4554443359375 550 / 1452 : pp = 207.0062255859375 560 / 1452 : pp = 206.59739685058594 570 / 1452 : pp = 206.27874755859375 580 / 1452 : pp = 205.87144470214844 590 / 1452 : pp = 205.43545532226562 600 / 1452 : pp = 204.90940856933594 610 / 1452 : pp = 204.5686798095703 620 / 1452 : pp = 204.22862243652344 630 / 1452 : pp = 203.8448028564453 640 / 1452 : pp = 204.06576538085938 650 / 1452 : pp = 204.0941925048828 660 / 1452 : pp = 204.22103881835938 670 / 1452 : pp = 204.289794921875 680 / 1452 : pp = 204.3115234375 690 / 1452 : pp = 204.10284423828125 700 / 1452 : pp = 203.99757385253906 710 / 1452 : pp = 204.04971313476562 720 / 1452 : pp = 204.03152465820312 730 / 1452 : pp = 203.99046325683594 740 / 1452 : pp = 204.19786071777344 750 / 1452 : pp = 204.1642608642578 760 / 1452 : pp = 204.19435119628906 770 / 1452 : pp = 204.37786865234375 780 / 1452 : pp = 204.4965057373047 790 / 1452 : pp = 204.6479034423828 800 / 1452 : pp = 204.56117248535156 810 / 1452 : pp = 204.52284240722656 820 / 1452 : pp = 204.50978088378906 830 / 1452 : pp = 204.7531280517578 840 / 1452 : pp = 204.64468383789062 850 / 1452 : pp = 204.71348571777344 860 / 1452 : pp = 204.7399444580078 870 / 1452 : pp = 204.69406127929688 880 / 1452 : pp = 204.7965850830078 890 / 1452 : pp = 204.7594757080078 900 / 1452 : pp = 204.71446228027344 910 / 1452 : pp = 204.7590789794922 920 / 1452 : pp = 204.85772705078125 930 / 1452 : pp = 204.7428741455078 940 / 1452 : pp = 204.8068389892578 950 / 1452 : pp = 204.75791931152344 960 / 1452 : pp = 204.63815307617188 970 / 1452 : pp = 204.60760498046875 980 / 1452 : pp = 204.34347534179688 990 / 1452 : pp = 204.151611328125 1000 / 1452 : pp = 203.8665771484375 1010 / 1452 : pp = 203.9164581298828 1020 / 1452 : pp = 204.0184783935547 1030 / 1452 : pp = 203.95166015625 1040 / 1452 : pp = 204.03045654296875 1050 / 1452 : pp = 203.95846557617188 1060 / 1452 : pp = 203.77114868164062 1070 / 1452 : pp = 203.93260192871094 1080 / 1452 : pp = 204.00048828125 1090 / 1452 : pp = 204.00233459472656 1100 / 1452 : pp = 203.8960418701172 1110 / 1452 : pp = 203.5987548828125 1120 / 1452 : pp = 203.38392639160156 1130 / 1452 : pp = 203.08872985839844 1140 / 1452 : pp = 203.01272583007812 1150 / 1452 : pp = 203.0865936279297 1160 / 1452 : pp = 203.02308654785156 1170 / 1452 : pp = 202.9125518798828 1180 / 1452 : pp = 202.9097442626953 1190 / 1452 : pp = 202.98252868652344 1200 / 1452 : pp = 202.95387268066406 1210 / 1452 : pp = 202.851318359375 1220 / 1452 : pp = 202.97671508789062 1230 / 1452 : pp = 203.1051025390625 1240 / 1452 : pp = 203.0526123046875 1250 / 1452 : pp = 203.21417236328125 1260 / 1452 : pp = 203.23617553710938 1270 / 1452 : pp = 203.22802734375 1280 / 1452 : pp = 203.20846557617188 1290 / 1452 : pp = 203.15362548828125 1300 / 1452 : pp = 203.14315795898438 1310 / 1452 : pp = 203.15264892578125 1320 / 1452 : pp = 203.02801513671875 1330 / 1452 : pp = 202.92977905273438 1340 / 1452 : pp = 202.95484924316406 1350 / 1452 : pp = 202.9335479736328 1360 / 1452 : pp = 202.955322265625 1370 / 1452 : pp = 202.7740478515625 1380 / 1452 : pp = 202.68569946289062 1390 / 1452 : pp = 202.55816650390625 1400 / 1452 : pp = 202.41651916503906 1410 / 1452 : pp = 202.38494873046875 1420 / 1452 : pp = 202.27593994140625 1430 / 1452 : pp = 202.21826171875 1440 / 1452 : pp = 202.23272705078125 1450 / 1452 : pp = 202.16099548339844 0 / 115 : pp = 253.23211669921875 10 / 115 : pp = 237.62506103515625 20 / 115 : pp = 237.60557556152344 30 / 115 : pp = 234.9273223876953 40 / 115 : pp = 234.30519104003906 50 / 115 : pp = 229.43960571289062 60 / 115 : pp = 228.6050567626953 70 / 115 : pp = 225.2646484375 80 / 115 : pp = 222.55935668945312 90 / 115 : pp = 219.83255004882812 100 / 115 : pp = 215.5491485595703 110 / 115 : pp = 214.07937622070312 Training perplexity: 202.1349639892578 Validation perplexity:213.85256958007812 Total time : 42.10724234580994 Epoch 5 0 / 1452 : pp = 255.92384338378906 10 / 1452 : pp = 219.5322265625 20 / 1452 : pp = 214.36212158203125 30 / 1452 : pp = 209.12620544433594 40 / 1452 : pp = 210.04193115234375 50 / 1452 : pp = 205.77398681640625 60 / 1452 : pp = 204.8201141357422 70 / 1452 : pp = 205.3955841064453 80 / 1452 : pp = 204.8386688232422 90 / 1452 : pp = 203.21194458007812 100 / 1452 : pp = 201.87643432617188 110 / 1452 : pp = 200.10122680664062 120 / 1452 : pp = 199.82012939453125 130 / 1452 : pp = 199.11192321777344 140 / 1452 : pp = 197.51919555664062 150 / 1452 : pp = 197.03567504882812 160 / 1452 : pp = 197.4231414794922 170 / 1452 : pp = 197.09571838378906 180 / 1452 : pp = 196.17665100097656 190 / 1452 : pp = 196.0064697265625 200 / 1452 : pp = 196.7347869873047 210 / 1452 : pp = 196.3063507080078 220 / 1452 : pp = 196.21388244628906 230 / 1452 : pp = 196.5252227783203 240 / 1452 : pp = 196.203125 250 / 1452 : pp = 195.3251953125 260 / 1452 : pp = 194.53335571289062 270 / 1452 : pp = 193.3546142578125 280 / 1452 : pp = 193.59420776367188 290 / 1452 : pp = 193.83297729492188 300 / 1452 : pp = 193.98489379882812 310 / 1452 : pp = 193.68414306640625 320 / 1452 : pp = 193.89065551757812 330 / 1452 : pp = 194.0518798828125 340 / 1452 : pp = 193.32888793945312 350 / 1452 : pp = 193.76219177246094 360 / 1452 : pp = 193.56106567382812 370 / 1452 : pp = 193.28179931640625 380 / 1452 : pp = 192.7037811279297 390 / 1452 : pp = 192.8145294189453 400 / 1452 : pp = 192.43325805664062 410 / 1452 : pp = 192.81527709960938 420 / 1452 : pp = 193.13760375976562 430 / 1452 : pp = 192.9148712158203 440 / 1452 : pp = 192.92526245117188 450 / 1452 : pp = 192.70083618164062 460 / 1452 : pp = 192.36647033691406 470 / 1452 : pp = 191.85394287109375 480 / 1452 : pp = 191.07244873046875 490 / 1452 : pp = 190.75401306152344 500 / 1452 : pp = 190.1843719482422 510 / 1452 : pp = 190.03334045410156 520 / 1452 : pp = 189.72938537597656 530 / 1452 : pp = 189.25889587402344 540 / 1452 : pp = 188.59315490722656 550 / 1452 : pp = 188.19313049316406 560 / 1452 : pp = 187.80621337890625 570 / 1452 : pp = 187.5229034423828 580 / 1452 : pp = 187.1091766357422 590 / 1452 : pp = 186.72592163085938 600 / 1452 : pp = 186.2238006591797 610 / 1452 : pp = 185.89695739746094 620 / 1452 : pp = 185.60989379882812 630 / 1452 : pp = 185.2689208984375 640 / 1452 : pp = 185.47567749023438 650 / 1452 : pp = 185.5127410888672 660 / 1452 : pp = 185.64627075195312 670 / 1452 : pp = 185.71311950683594 680 / 1452 : pp = 185.72569274902344 690 / 1452 : pp = 185.56459045410156 700 / 1452 : pp = 185.48681640625 710 / 1452 : pp = 185.5458221435547 720 / 1452 : pp = 185.5598907470703 730 / 1452 : pp = 185.5335235595703 740 / 1452 : pp = 185.73995971679688 750 / 1452 : pp = 185.744384765625 760 / 1452 : pp = 185.81268310546875 770 / 1452 : pp = 186.00088500976562 780 / 1452 : pp = 186.14443969726562 790 / 1452 : pp = 186.30764770507812 800 / 1452 : pp = 186.2595977783203 810 / 1452 : pp = 186.23028564453125 820 / 1452 : pp = 186.23997497558594 830 / 1452 : pp = 186.49057006835938 840 / 1452 : pp = 186.43331909179688 850 / 1452 : pp = 186.48887634277344 860 / 1452 : pp = 186.51502990722656 870 / 1452 : pp = 186.5167999267578 880 / 1452 : pp = 186.62400817871094 890 / 1452 : pp = 186.6103973388672 900 / 1452 : pp = 186.58111572265625 910 / 1452 : pp = 186.64126586914062 920 / 1452 : pp = 186.7366180419922 930 / 1452 : pp = 186.65719604492188 940 / 1452 : pp = 186.71755981445312 950 / 1452 : pp = 186.6977996826172 960 / 1452 : pp = 186.62774658203125 970 / 1452 : pp = 186.62115478515625 980 / 1452 : pp = 186.3773193359375 990 / 1452 : pp = 186.23109436035156 1000 / 1452 : pp = 185.99227905273438 1010 / 1452 : pp = 186.0488739013672 1020 / 1452 : pp = 186.1744384765625 1030 / 1452 : pp = 186.1162109375 1040 / 1452 : pp = 186.18899536132812 1050 / 1452 : pp = 186.1549072265625 1060 / 1452 : pp = 186.01419067382812 1070 / 1452 : pp = 186.17364501953125 1080 / 1452 : pp = 186.27061462402344 1090 / 1452 : pp = 186.28428649902344 1100 / 1452 : pp = 186.2150115966797 1110 / 1452 : pp = 185.95103454589844 1120 / 1452 : pp = 185.77423095703125 1130 / 1452 : pp = 185.5232696533203 1140 / 1452 : pp = 185.4607391357422 1150 / 1452 : pp = 185.56077575683594 1160 / 1452 : pp = 185.53343200683594 1170 / 1452 : pp = 185.46453857421875 1180 / 1452 : pp = 185.4741668701172 1190 / 1452 : pp = 185.5594482421875 1200 / 1452 : pp = 185.53785705566406 1210 / 1452 : pp = 185.4576416015625 1220 / 1452 : pp = 185.5943145751953 1230 / 1452 : pp = 185.7483673095703 1240 / 1452 : pp = 185.70762634277344 1250 / 1452 : pp = 185.8568115234375 1260 / 1452 : pp = 185.90635681152344 1270 / 1452 : pp = 185.8961639404297 1280 / 1452 : pp = 185.89199829101562 1290 / 1452 : pp = 185.85911560058594 1300 / 1452 : pp = 185.86097717285156 1310 / 1452 : pp = 185.88739013671875 1320 / 1452 : pp = 185.79248046875 1330 / 1452 : pp = 185.69700622558594 1340 / 1452 : pp = 185.7310028076172 1350 / 1452 : pp = 185.72613525390625 1360 / 1452 : pp = 185.76829528808594 1370 / 1452 : pp = 185.6322021484375 1380 / 1452 : pp = 185.56378173828125 1390 / 1452 : pp = 185.4654998779297 1400 / 1452 : pp = 185.35110473632812 1410 / 1452 : pp = 185.33917236328125 1420 / 1452 : pp = 185.2509002685547 1430 / 1452 : pp = 185.20436096191406 1440 / 1452 : pp = 185.2254638671875 1450 / 1452 : pp = 185.16542053222656 0 / 115 : pp = 242.26800537109375 10 / 115 : pp = 226.12258911132812 20 / 115 : pp = 226.4702606201172 30 / 115 : pp = 223.982666015625 40 / 115 : pp = 223.376953125 50 / 115 : pp = 218.65716552734375 60 / 115 : pp = 217.95306396484375 70 / 115 : pp = 214.5392303466797 80 / 115 : pp = 212.07525634765625 90 / 115 : pp = 209.40631103515625 100 / 115 : pp = 205.1455078125 110 / 115 : pp = 203.6289520263672 Training perplexity: 185.14476013183594 Validation perplexity:203.3822784423828 Total time : 42.47052240371704 Epoch 6 0 / 1452 : pp = 233.56707763671875 10 / 1452 : pp = 202.6468505859375 20 / 1452 : pp = 198.2734375 30 / 1452 : pp = 193.47442626953125 40 / 1452 : pp = 195.17147827148438 50 / 1452 : pp = 191.5596923828125 60 / 1452 : pp = 190.4825897216797 70 / 1452 : pp = 191.07681274414062 80 / 1452 : pp = 190.339599609375 90 / 1452 : pp = 188.98277282714844 100 / 1452 : pp = 187.74757385253906 110 / 1452 : pp = 186.10104370117188 120 / 1452 : pp = 185.7500457763672 130 / 1452 : pp = 184.90707397460938 140 / 1452 : pp = 183.340087890625 150 / 1452 : pp = 182.70840454101562 160 / 1452 : pp = 183.1043701171875 170 / 1452 : pp = 182.69776916503906 180 / 1452 : pp = 181.88400268554688 190 / 1452 : pp = 181.8062286376953 200 / 1452 : pp = 182.4969940185547 210 / 1452 : pp = 182.10572814941406 220 / 1452 : pp = 181.9981689453125 230 / 1452 : pp = 182.3802490234375 240 / 1452 : pp = 182.03636169433594 250 / 1452 : pp = 181.23712158203125 260 / 1452 : pp = 180.53726196289062 270 / 1452 : pp = 179.53567504882812 280 / 1452 : pp = 179.70208740234375 290 / 1452 : pp = 179.977783203125 300 / 1452 : pp = 180.16600036621094 310 / 1452 : pp = 179.87294006347656 320 / 1452 : pp = 180.11849975585938 330 / 1452 : pp = 180.31838989257812 340 / 1452 : pp = 179.56759643554688 350 / 1452 : pp = 179.97134399414062 360 / 1452 : pp = 179.80030822753906 370 / 1452 : pp = 179.52085876464844 380 / 1452 : pp = 178.98228454589844 390 / 1452 : pp = 179.0868682861328 400 / 1452 : pp = 178.74569702148438 410 / 1452 : pp = 179.1776580810547 420 / 1452 : pp = 179.5055389404297 430 / 1452 : pp = 179.3883056640625 440 / 1452 : pp = 179.42279052734375 450 / 1452 : pp = 179.2106475830078 460 / 1452 : pp = 178.85311889648438 470 / 1452 : pp = 178.33840942382812 480 / 1452 : pp = 177.60350036621094 490 / 1452 : pp = 177.30335998535156 500 / 1452 : pp = 176.72222900390625 510 / 1452 : pp = 176.6067352294922 520 / 1452 : pp = 176.33998107910156 530 / 1452 : pp = 175.93162536621094 540 / 1452 : pp = 175.30657958984375 550 / 1452 : pp = 174.9462432861328 560 / 1452 : pp = 174.5836639404297 570 / 1452 : pp = 174.31431579589844 580 / 1452 : pp = 173.92300415039062 590 / 1452 : pp = 173.55856323242188 600 / 1452 : pp = 173.08277893066406 610 / 1452 : pp = 172.75930786132812 620 / 1452 : pp = 172.53192138671875 630 / 1452 : pp = 172.20652770996094 640 / 1452 : pp = 172.37454223632812 650 / 1452 : pp = 172.39845275878906 660 / 1452 : pp = 172.52255249023438 670 / 1452 : pp = 172.60935974121094 680 / 1452 : pp = 172.6611328125 690 / 1452 : pp = 172.53118896484375 700 / 1452 : pp = 172.4709014892578 710 / 1452 : pp = 172.5406494140625 720 / 1452 : pp = 172.55447387695312 730 / 1452 : pp = 172.5330047607422 740 / 1452 : pp = 172.7061767578125 750 / 1452 : pp = 172.71054077148438 760 / 1452 : pp = 172.77743530273438 770 / 1452 : pp = 172.95481872558594 780 / 1452 : pp = 173.11265563964844 790 / 1452 : pp = 173.2832794189453 800 / 1452 : pp = 173.2537841796875 810 / 1452 : pp = 173.22164916992188 820 / 1452 : pp = 173.24148559570312 830 / 1452 : pp = 173.48228454589844 840 / 1452 : pp = 173.43753051757812 850 / 1452 : pp = 173.505615234375 860 / 1452 : pp = 173.5214080810547 870 / 1452 : pp = 173.5009002685547 880 / 1452 : pp = 173.6202392578125 890 / 1452 : pp = 173.622802734375 900 / 1452 : pp = 173.5987091064453 910 / 1452 : pp = 173.68316650390625 920 / 1452 : pp = 173.77330017089844 930 / 1452 : pp = 173.72018432617188 940 / 1452 : pp = 173.79351806640625 950 / 1452 : pp = 173.7653350830078 960 / 1452 : pp = 173.7102508544922 970 / 1452 : pp = 173.69766235351562 980 / 1452 : pp = 173.4836883544922 990 / 1452 : pp = 173.3550262451172 1000 / 1452 : pp = 173.14816284179688 1010 / 1452 : pp = 173.20777893066406 1020 / 1452 : pp = 173.3390655517578 1030 / 1452 : pp = 173.2884063720703 1040 / 1452 : pp = 173.38015747070312 1050 / 1452 : pp = 173.35592651367188 1060 / 1452 : pp = 173.2260284423828 1070 / 1452 : pp = 173.39321899414062 1080 / 1452 : pp = 173.4879913330078 1090 / 1452 : pp = 173.5231475830078 1100 / 1452 : pp = 173.47177124023438 1110 / 1452 : pp = 173.24453735351562 1120 / 1452 : pp = 173.09408569335938 1130 / 1452 : pp = 172.86627197265625 1140 / 1452 : pp = 172.8234100341797 1150 / 1452 : pp = 172.92843627929688 1160 / 1452 : pp = 172.90065002441406 1170 / 1452 : pp = 172.8550567626953 1180 / 1452 : pp = 172.8810272216797 1190 / 1452 : pp = 172.97312927246094 1200 / 1452 : pp = 172.9776611328125 1210 / 1452 : pp = 172.89413452148438 1220 / 1452 : pp = 173.0257568359375 1230 / 1452 : pp = 173.1847381591797 1240 / 1452 : pp = 173.1756591796875 1250 / 1452 : pp = 173.32138061523438 1260 / 1452 : pp = 173.37229919433594 1270 / 1452 : pp = 173.36891174316406 1280 / 1452 : pp = 173.36337280273438 1290 / 1452 : pp = 173.3444366455078 1300 / 1452 : pp = 173.36138916015625 1310 / 1452 : pp = 173.4015655517578 1320 / 1452 : pp = 173.31790161132812 1330 / 1452 : pp = 173.24710083007812 1340 / 1452 : pp = 173.27212524414062 1350 / 1452 : pp = 173.27674865722656 1360 / 1452 : pp = 173.32749938964844 1370 / 1452 : pp = 173.20472717285156 1380 / 1452 : pp = 173.14889526367188 1390 / 1452 : pp = 173.0755157470703 1400 / 1452 : pp = 172.9678497314453 1410 / 1452 : pp = 172.9612579345703 1420 / 1452 : pp = 172.8872833251953 1430 / 1452 : pp = 172.84805297851562 1440 / 1452 : pp = 172.87252807617188 1450 / 1452 : pp = 172.82505798339844 0 / 115 : pp = 236.35635375976562 10 / 115 : pp = 219.06166076660156 20 / 115 : pp = 219.7670440673828 30 / 115 : pp = 217.33587646484375 40 / 115 : pp = 216.6626739501953 50 / 115 : pp = 212.04734802246094 60 / 115 : pp = 211.42068481445312 70 / 115 : pp = 207.9592742919922 80 / 115 : pp = 205.6216583251953 90 / 115 : pp = 202.93597412109375 100 / 115 : pp = 198.62583923339844 110 / 115 : pp = 196.97216796875 Training perplexity: 172.80404663085938 Validation perplexity:196.6871337890625 Total time : 41.52522921562195 Epoch 7 0 / 1452 : pp = 219.23231506347656 10 / 1452 : pp = 192.07225036621094 20 / 1452 : pp = 187.48464965820312 30 / 1452 : pp = 182.9149932861328 40 / 1452 : pp = 184.2945098876953 50 / 1452 : pp = 180.78492736816406 60 / 1452 : pp = 179.377197265625 70 / 1452 : pp = 180.0273895263672 80 / 1452 : pp = 179.2517547607422 90 / 1452 : pp = 177.77540588378906 100 / 1452 : pp = 176.6474151611328 110 / 1452 : pp = 174.84066772460938 120 / 1452 : pp = 174.46890258789062 130 / 1452 : pp = 173.64573669433594 140 / 1452 : pp = 172.17483520507812 150 / 1452 : pp = 171.57041931152344 160 / 1452 : pp = 171.92059326171875 170 / 1452 : pp = 171.5497283935547 180 / 1452 : pp = 170.77249145507812 190 / 1452 : pp = 170.72103881835938 200 / 1452 : pp = 171.336181640625 210 / 1452 : pp = 170.98524475097656 220 / 1452 : pp = 170.99771118164062 230 / 1452 : pp = 171.39918518066406 240 / 1452 : pp = 171.09925842285156 250 / 1452 : pp = 170.39962768554688 260 / 1452 : pp = 169.7328643798828 270 / 1452 : pp = 168.72225952148438 280 / 1452 : pp = 168.92552185058594 290 / 1452 : pp = 169.20147705078125 300 / 1452 : pp = 169.40338134765625 310 / 1452 : pp = 169.12057495117188 320 / 1452 : pp = 169.31236267089844 330 / 1452 : pp = 169.49945068359375 340 / 1452 : pp = 168.8396759033203 350 / 1452 : pp = 169.25917053222656 360 / 1452 : pp = 169.09388732910156 370 / 1452 : pp = 168.84323120117188 380 / 1452 : pp = 168.3832550048828 390 / 1452 : pp = 168.48275756835938 400 / 1452 : pp = 168.19972229003906 410 / 1452 : pp = 168.5838623046875 420 / 1452 : pp = 168.91119384765625 430 / 1452 : pp = 168.80836486816406 440 / 1452 : pp = 168.90264892578125 450 / 1452 : pp = 168.68589782714844 460 / 1452 : pp = 168.3704071044922 470 / 1452 : pp = 167.90394592285156 480 / 1452 : pp = 167.23373413085938 490 / 1452 : pp = 166.9560546875 500 / 1452 : pp = 166.43161010742188 510 / 1452 : pp = 166.320068359375 520 / 1452 : pp = 166.05902099609375 530 / 1452 : pp = 165.71714782714844 540 / 1452 : pp = 165.10398864746094 550 / 1452 : pp = 164.80430603027344 560 / 1452 : pp = 164.4687042236328 570 / 1452 : pp = 164.2272491455078 580 / 1452 : pp = 163.84312438964844 590 / 1452 : pp = 163.46035766601562 600 / 1452 : pp = 163.01559448242188 610 / 1452 : pp = 162.74134826660156 620 / 1452 : pp = 162.50267028808594 630 / 1452 : pp = 162.2018280029297 640 / 1452 : pp = 162.37130737304688 650 / 1452 : pp = 162.3895721435547 660 / 1452 : pp = 162.51351928710938 670 / 1452 : pp = 162.57684326171875 680 / 1452 : pp = 162.6346893310547 690 / 1452 : pp = 162.5135955810547 700 / 1452 : pp = 162.47052001953125 710 / 1452 : pp = 162.539794921875 720 / 1452 : pp = 162.55381774902344 730 / 1452 : pp = 162.5297088623047 740 / 1452 : pp = 162.71652221679688 750 / 1452 : pp = 162.740966796875 760 / 1452 : pp = 162.79754638671875 770 / 1452 : pp = 162.9949951171875 780 / 1452 : pp = 163.17868041992188 790 / 1452 : pp = 163.33055114746094 800 / 1452 : pp = 163.31591796875 810 / 1452 : pp = 163.2859344482422 820 / 1452 : pp = 163.2958984375 830 / 1452 : pp = 163.528564453125 840 / 1452 : pp = 163.47610473632812 850 / 1452 : pp = 163.5260772705078 860 / 1452 : pp = 163.55352783203125 870 / 1452 : pp = 163.55718994140625 880 / 1452 : pp = 163.67523193359375 890 / 1452 : pp = 163.6920166015625 900 / 1452 : pp = 163.67710876464844 910 / 1452 : pp = 163.7476806640625 920 / 1452 : pp = 163.84803771972656 930 / 1452 : pp = 163.8114013671875 940 / 1452 : pp = 163.86663818359375 950 / 1452 : pp = 163.83531188964844 960 / 1452 : pp = 163.79945373535156 970 / 1452 : pp = 163.80320739746094 980 / 1452 : pp = 163.5953369140625 990 / 1452 : pp = 163.48382568359375 1000 / 1452 : pp = 163.2642822265625 1010 / 1452 : pp = 163.32113647460938 1020 / 1452 : pp = 163.44204711914062 1030 / 1452 : pp = 163.40206909179688 1040 / 1452 : pp = 163.4915313720703 1050 / 1452 : pp = 163.47096252441406 1060 / 1452 : pp = 163.3601531982422 1070 / 1452 : pp = 163.5138397216797 1080 / 1452 : pp = 163.6189727783203 1090 / 1452 : pp = 163.6471405029297 1100 / 1452 : pp = 163.60406494140625 1110 / 1452 : pp = 163.40736389160156 1120 / 1452 : pp = 163.26841735839844 1130 / 1452 : pp = 163.0680694580078 1140 / 1452 : pp = 163.04591369628906 1150 / 1452 : pp = 163.15478515625 1160 / 1452 : pp = 163.1380615234375 1170 / 1452 : pp = 163.09303283691406 1180 / 1452 : pp = 163.14149475097656 1190 / 1452 : pp = 163.2374267578125 1200 / 1452 : pp = 163.2394561767578 1210 / 1452 : pp = 163.17835998535156 1220 / 1452 : pp = 163.32347106933594 1230 / 1452 : pp = 163.4639434814453 1240 / 1452 : pp = 163.4611358642578 1250 / 1452 : pp = 163.60687255859375 1260 / 1452 : pp = 163.67227172851562 1270 / 1452 : pp = 163.67515563964844 1280 / 1452 : pp = 163.6881103515625 1290 / 1452 : pp = 163.66648864746094 1300 / 1452 : pp = 163.69287109375 1310 / 1452 : pp = 163.7276153564453 1320 / 1452 : pp = 163.6551055908203 1330 / 1452 : pp = 163.58901977539062 1340 / 1452 : pp = 163.6205291748047 1350 / 1452 : pp = 163.63824462890625 1360 / 1452 : pp = 163.69334411621094 1370 / 1452 : pp = 163.5885467529297 1380 / 1452 : pp = 163.54049682617188 1390 / 1452 : pp = 163.4760284423828 1400 / 1452 : pp = 163.38897705078125 1410 / 1452 : pp = 163.3974609375 1420 / 1452 : pp = 163.35009765625 1430 / 1452 : pp = 163.32191467285156 1440 / 1452 : pp = 163.35220336914062 1450 / 1452 : pp = 163.3201904296875 0 / 115 : pp = 232.2108154296875 10 / 115 : pp = 214.35496520996094 20 / 115 : pp = 215.20510864257812 30 / 115 : pp = 212.82754516601562 40 / 115 : pp = 212.0598907470703 50 / 115 : pp = 207.5095672607422 60 / 115 : pp = 206.86976623535156 70 / 115 : pp = 203.36016845703125 80 / 115 : pp = 201.11538696289062 90 / 115 : pp = 198.52120971679688 100 / 115 : pp = 194.1772003173828 110 / 115 : pp = 192.41224670410156 Training perplexity: 163.29916381835938 Validation perplexity:192.09552001953125 Total time : 41.78096055984497 Epoch 8 0 / 1452 : pp = 201.77548217773438 10 / 1452 : pp = 180.4141082763672 20 / 1452 : pp = 176.41432189941406 30 / 1452 : pp = 172.7764434814453 40 / 1452 : pp = 174.69166564941406 50 / 1452 : pp = 171.2933807373047 60 / 1452 : pp = 170.08010864257812 70 / 1452 : pp = 170.6719512939453 80 / 1452 : pp = 170.07589721679688 90 / 1452 : pp = 168.7478485107422 100 / 1452 : pp = 167.57081604003906 110 / 1452 : pp = 166.06971740722656 120 / 1452 : pp = 165.73374938964844 130 / 1452 : pp = 164.80674743652344 140 / 1452 : pp = 163.32821655273438 150 / 1452 : pp = 162.6752471923828 160 / 1452 : pp = 163.02049255371094 170 / 1452 : pp = 162.64120483398438 180 / 1452 : pp = 161.95529174804688 190 / 1452 : pp = 161.91954040527344 200 / 1452 : pp = 162.5446014404297 210 / 1452 : pp = 162.2645721435547 220 / 1452 : pp = 162.3128662109375 230 / 1452 : pp = 162.65872192382812 240 / 1452 : pp = 162.40948486328125 250 / 1452 : pp = 161.75787353515625 260 / 1452 : pp = 161.15213012695312 270 / 1452 : pp = 160.22256469726562 280 / 1452 : pp = 160.3651123046875 290 / 1452 : pp = 160.63780212402344 300 / 1452 : pp = 160.80026245117188 310 / 1452 : pp = 160.54383850097656 320 / 1452 : pp = 160.7539520263672 330 / 1452 : pp = 160.94317626953125 340 / 1452 : pp = 160.3373565673828 350 / 1452 : pp = 160.71763610839844 360 / 1452 : pp = 160.60960388183594 370 / 1452 : pp = 160.37527465820312 380 / 1452 : pp = 159.92990112304688 390 / 1452 : pp = 160.0165557861328 400 / 1452 : pp = 159.75697326660156 410 / 1452 : pp = 160.15274047851562 420 / 1452 : pp = 160.48390197753906 430 / 1452 : pp = 160.4031982421875 440 / 1452 : pp = 160.4693603515625 450 / 1452 : pp = 160.28016662597656 460 / 1452 : pp = 159.94004821777344 470 / 1452 : pp = 159.48257446289062 480 / 1452 : pp = 158.87998962402344 490 / 1452 : pp = 158.59765625 500 / 1452 : pp = 158.10865783691406 510 / 1452 : pp = 157.96795654296875 520 / 1452 : pp = 157.7591552734375 530 / 1452 : pp = 157.42648315429688 540 / 1452 : pp = 156.85348510742188 550 / 1452 : pp = 156.5618438720703 560 / 1452 : pp = 156.24905395507812 570 / 1452 : pp = 155.9994354248047 580 / 1452 : pp = 155.612060546875 590 / 1452 : pp = 155.25830078125 600 / 1452 : pp = 154.8464813232422 610 / 1452 : pp = 154.5833282470703 620 / 1452 : pp = 154.38040161132812 630 / 1452 : pp = 154.0767364501953 640 / 1452 : pp = 154.2534637451172 650 / 1452 : pp = 154.25875854492188 660 / 1452 : pp = 154.35874938964844 670 / 1452 : pp = 154.4289093017578 680 / 1452 : pp = 154.51412963867188 690 / 1452 : pp = 154.41676330566406 700 / 1452 : pp = 154.37892150878906 710 / 1452 : pp = 154.4234619140625 720 / 1452 : pp = 154.4586639404297 730 / 1452 : pp = 154.4351806640625 740 / 1452 : pp = 154.6002197265625 750 / 1452 : pp = 154.65684509277344 760 / 1452 : pp = 154.73318481445312 770 / 1452 : pp = 154.92935180664062 780 / 1452 : pp = 155.1021728515625 790 / 1452 : pp = 155.24757385253906 800 / 1452 : pp = 155.223876953125 810 / 1452 : pp = 155.2095184326172 820 / 1452 : pp = 155.24009704589844 830 / 1452 : pp = 155.4519500732422 840 / 1452 : pp = 155.3947296142578 850 / 1452 : pp = 155.45306396484375 860 / 1452 : pp = 155.4661102294922 870 / 1452 : pp = 155.45765686035156 880 / 1452 : pp = 155.58758544921875 890 / 1452 : pp = 155.59373474121094 900 / 1452 : pp = 155.59254455566406 910 / 1452 : pp = 155.66854858398438 920 / 1452 : pp = 155.75942993164062 930 / 1452 : pp = 155.73350524902344 940 / 1452 : pp = 155.80740356445312 950 / 1452 : pp = 155.7733917236328 960 / 1452 : pp = 155.73565673828125 970 / 1452 : pp = 155.74404907226562 980 / 1452 : pp = 155.55902099609375 990 / 1452 : pp = 155.45675659179688 1000 / 1452 : pp = 155.2649688720703 1010 / 1452 : pp = 155.31332397460938 1020 / 1452 : pp = 155.44979858398438 1030 / 1452 : pp = 155.4137725830078 1040 / 1452 : pp = 155.49012756347656 1050 / 1452 : pp = 155.46054077148438 1060 / 1452 : pp = 155.3616943359375 1070 / 1452 : pp = 155.5286865234375 1080 / 1452 : pp = 155.63743591308594 1090 / 1452 : pp = 155.6842803955078 1100 / 1452 : pp = 155.65599060058594 1110 / 1452 : pp = 155.4827880859375 1120 / 1452 : pp = 155.35450744628906 1130 / 1452 : pp = 155.1777801513672 1140 / 1452 : pp = 155.15994262695312 1150 / 1452 : pp = 155.26193237304688 1160 / 1452 : pp = 155.26214599609375 1170 / 1452 : pp = 155.23231506347656 1180 / 1452 : pp = 155.29266357421875 1190 / 1452 : pp = 155.37680053710938 1200 / 1452 : pp = 155.3736114501953 1210 / 1452 : pp = 155.3380584716797 1220 / 1452 : pp = 155.474853515625 1230 / 1452 : pp = 155.62986755371094 1240 / 1452 : pp = 155.62831115722656 1250 / 1452 : pp = 155.77101135253906 1260 / 1452 : pp = 155.83445739746094 1270 / 1452 : pp = 155.845458984375 1280 / 1452 : pp = 155.8556365966797 1290 / 1452 : pp = 155.8556365966797 1300 / 1452 : pp = 155.8843994140625 1310 / 1452 : pp = 155.92417907714844 1320 / 1452 : pp = 155.8560791015625 1330 / 1452 : pp = 155.80636596679688 1340 / 1452 : pp = 155.84344482421875 1350 / 1452 : pp = 155.8706512451172 1360 / 1452 : pp = 155.9273681640625 1370 / 1452 : pp = 155.83140563964844 1380 / 1452 : pp = 155.7911376953125 1390 / 1452 : pp = 155.7401885986328 1400 / 1452 : pp = 155.6622314453125 1410 / 1452 : pp = 155.68531799316406 1420 / 1452 : pp = 155.64041137695312 1430 / 1452 : pp = 155.62216186523438 1440 / 1452 : pp = 155.6437530517578 1450 / 1452 : pp = 155.62757873535156 0 / 115 : pp = 228.70111083984375 10 / 115 : pp = 211.03330993652344 20 / 115 : pp = 212.24957275390625 30 / 115 : pp = 209.8839569091797 40 / 115 : pp = 209.11045837402344 50 / 115 : pp = 204.66351318359375 60 / 115 : pp = 204.03366088867188 70 / 115 : pp = 200.46681213378906 80 / 115 : pp = 198.24404907226562 90 / 115 : pp = 195.63223266601562 100 / 115 : pp = 191.18345642089844 110 / 115 : pp = 189.31134033203125 Training perplexity: 155.61154174804688 Validation perplexity:188.94537353515625 Total time : 42.13483738899231 Epoch 9 0 / 1452 : pp = 197.80628967285156 10 / 1452 : pp = 172.6316680908203 20 / 1452 : pp = 168.6739959716797 30 / 1452 : pp = 164.4781036376953 40 / 1452 : pp = 166.1627960205078 50 / 1452 : pp = 163.05197143554688 60 / 1452 : pp = 161.87924194335938 70 / 1452 : pp = 162.5297088623047 80 / 1452 : pp = 161.7450714111328 90 / 1452 : pp = 160.6148223876953 100 / 1452 : pp = 159.73289489746094 110 / 1452 : pp = 158.4092254638672 120 / 1452 : pp = 158.04653930664062 130 / 1452 : pp = 157.13563537597656 140 / 1452 : pp = 155.71798706054688 150 / 1452 : pp = 155.19161987304688 160 / 1452 : pp = 155.42718505859375 170 / 1452 : pp = 155.0531463623047 180 / 1452 : pp = 154.46897888183594 190 / 1452 : pp = 154.4127197265625 200 / 1452 : pp = 154.97154235839844 210 / 1452 : pp = 154.70169067382812 220 / 1452 : pp = 154.72816467285156 230 / 1452 : pp = 155.03799438476562 240 / 1452 : pp = 154.85601806640625 250 / 1452 : pp = 154.28016662597656 260 / 1452 : pp = 153.7699432373047 270 / 1452 : pp = 152.90948486328125 280 / 1452 : pp = 153.0459747314453 290 / 1452 : pp = 153.298095703125 300 / 1452 : pp = 153.45716857910156 310 / 1452 : pp = 153.22195434570312 320 / 1452 : pp = 153.41664123535156 330 / 1452 : pp = 153.66542053222656 340 / 1452 : pp = 153.06378173828125 350 / 1452 : pp = 153.43923950195312 360 / 1452 : pp = 153.31381225585938 370 / 1452 : pp = 153.13473510742188 380 / 1452 : pp = 152.75267028808594 390 / 1452 : pp = 152.85504150390625 400 / 1452 : pp = 152.62342834472656 410 / 1452 : pp = 153.03152465820312 420 / 1452 : pp = 153.39161682128906 430 / 1452 : pp = 153.30364990234375 440 / 1452 : pp = 153.37896728515625 450 / 1452 : pp = 153.18988037109375 460 / 1452 : pp = 152.88478088378906 470 / 1452 : pp = 152.4380340576172 480 / 1452 : pp = 151.86618041992188 490 / 1452 : pp = 151.5962371826172 500 / 1452 : pp = 151.11614990234375 510 / 1452 : pp = 150.99830627441406 520 / 1452 : pp = 150.8135986328125 530 / 1452 : pp = 150.500732421875 540 / 1452 : pp = 149.9623260498047 550 / 1452 : pp = 149.68028259277344 560 / 1452 : pp = 149.3885040283203 570 / 1452 : pp = 149.140380859375 580 / 1452 : pp = 148.76876831054688 590 / 1452 : pp = 148.43368530273438 600 / 1452 : pp = 148.02598571777344 610 / 1452 : pp = 147.7869110107422 620 / 1452 : pp = 147.59796142578125 630 / 1452 : pp = 147.30068969726562 640 / 1452 : pp = 147.45240783691406 650 / 1452 : pp = 147.4651336669922 660 / 1452 : pp = 147.5808563232422 670 / 1452 : pp = 147.65582275390625 680 / 1452 : pp = 147.7360382080078 690 / 1452 : pp = 147.63075256347656 700 / 1452 : pp = 147.6066131591797 710 / 1452 : pp = 147.7024383544922 720 / 1452 : pp = 147.7445526123047 730 / 1452 : pp = 147.72279357910156 740 / 1452 : pp = 147.87107849121094 750 / 1452 : pp = 147.91436767578125 760 / 1452 : pp = 147.9857635498047 770 / 1452 : pp = 148.18206787109375 780 / 1452 : pp = 148.3845672607422 790 / 1452 : pp = 148.5517120361328 800 / 1452 : pp = 148.54002380371094 810 / 1452 : pp = 148.51119995117188 820 / 1452 : pp = 148.5664520263672 830 / 1452 : pp = 148.7821044921875 840 / 1452 : pp = 148.72486877441406 850 / 1452 : pp = 148.77452087402344 860 / 1452 : pp = 148.80076599121094 870 / 1452 : pp = 148.79701232910156 880 / 1452 : pp = 148.9181671142578 890 / 1452 : pp = 148.94537353515625 900 / 1452 : pp = 148.9435272216797 910 / 1452 : pp = 149.02102661132812 920 / 1452 : pp = 149.1085968017578 930 / 1452 : pp = 149.06893920898438 940 / 1452 : pp = 149.1317138671875 950 / 1452 : pp = 149.1232452392578 960 / 1452 : pp = 149.10354614257812 970 / 1452 : pp = 149.11656188964844 980 / 1452 : pp = 148.94259643554688 990 / 1452 : pp = 148.8236846923828 1000 / 1452 : pp = 148.633056640625 1010 / 1452 : pp = 148.6830291748047 1020 / 1452 : pp = 148.8126220703125 1030 / 1452 : pp = 148.78089904785156 1040 / 1452 : pp = 148.8600311279297 1050 / 1452 : pp = 148.8486785888672 1060 / 1452 : pp = 148.7664337158203 1070 / 1452 : pp = 148.9337921142578 1080 / 1452 : pp = 149.04441833496094 1090 / 1452 : pp = 149.07284545898438 1100 / 1452 : pp = 149.03318786621094 1110 / 1452 : pp = 148.86428833007812 1120 / 1452 : pp = 148.7332305908203 1130 / 1452 : pp = 148.5670166015625 1140 / 1452 : pp = 148.54661560058594 1150 / 1452 : pp = 148.64219665527344 1160 / 1452 : pp = 148.6490020751953 1170 / 1452 : pp = 148.62420654296875 1180 / 1452 : pp = 148.67665100097656 1190 / 1452 : pp = 148.7633056640625 1200 / 1452 : pp = 148.7782745361328 1210 / 1452 : pp = 148.72500610351562 1220 / 1452 : pp = 148.87493896484375 1230 / 1452 : pp = 149.039794921875 1240 / 1452 : pp = 149.04000854492188 1250 / 1452 : pp = 149.17054748535156 1260 / 1452 : pp = 149.23863220214844 1270 / 1452 : pp = 149.2436065673828 1280 / 1452 : pp = 149.25086975097656 1290 / 1452 : pp = 149.24147033691406 1300 / 1452 : pp = 149.27413940429688 1310 / 1452 : pp = 149.32077026367188 1320 / 1452 : pp = 149.27301025390625 1330 / 1452 : pp = 149.23080444335938 1340 / 1452 : pp = 149.25791931152344 1350 / 1452 : pp = 149.2841033935547 1360 / 1452 : pp = 149.337158203125 1370 / 1452 : pp = 149.2467498779297 1380 / 1452 : pp = 149.21351623535156 1390 / 1452 : pp = 149.15403747558594 1400 / 1452 : pp = 149.0877685546875 1410 / 1452 : pp = 149.110595703125 1420 / 1452 : pp = 149.07241821289062 1430 / 1452 : pp = 149.05166625976562 1440 / 1452 : pp = 149.0776824951172 1450 / 1452 : pp = 149.06771850585938 0 / 115 : pp = 227.0559844970703 10 / 115 : pp = 208.7002410888672 20 / 115 : pp = 210.38775634765625 30 / 115 : pp = 207.9513397216797 40 / 115 : pp = 207.12994384765625 50 / 115 : pp = 202.70811462402344 60 / 115 : pp = 202.05787658691406 70 / 115 : pp = 198.3761444091797 80 / 115 : pp = 196.17637634277344 90 / 115 : pp = 193.5880126953125 100 / 115 : pp = 189.0758819580078 110 / 115 : pp = 187.07528686523438 Training perplexity: 149.0502471923828 Validation perplexity:186.6911163330078 Total time : 47.274805545806885 Epoch 10 0 / 1452 : pp = 181.8408203125 10 / 1452 : pp = 164.99664306640625 20 / 1452 : pp = 161.8847198486328 30 / 1452 : pp = 158.30064392089844 40 / 1452 : pp = 160.13914489746094 50 / 1452 : pp = 157.58743286132812 60 / 1452 : pp = 156.11871337890625 70 / 1452 : pp = 156.82948303222656 80 / 1452 : pp = 156.2889862060547 90 / 1452 : pp = 155.04833984375 100 / 1452 : pp = 154.09327697753906 110 / 1452 : pp = 152.5070343017578 120 / 1452 : pp = 152.20750427246094 130 / 1452 : pp = 151.3399200439453 140 / 1452 : pp = 149.90740966796875 150 / 1452 : pp = 149.345703125 160 / 1452 : pp = 149.59814453125 170 / 1452 : pp = 149.26539611816406 180 / 1452 : pp = 148.624267578125 190 / 1452 : pp = 148.58819580078125 200 / 1452 : pp = 149.09552001953125 210 / 1452 : pp = 148.8439178466797 220 / 1452 : pp = 148.86605834960938 230 / 1452 : pp = 149.1971435546875 240 / 1452 : pp = 148.96533203125 250 / 1452 : pp = 148.4253387451172 260 / 1452 : pp = 147.9200897216797 270 / 1452 : pp = 147.08816528320312 280 / 1452 : pp = 147.24366760253906 290 / 1452 : pp = 147.52182006835938 300 / 1452 : pp = 147.72222900390625 310 / 1452 : pp = 147.50486755371094 320 / 1452 : pp = 147.73892211914062 330 / 1452 : pp = 147.9404754638672 340 / 1452 : pp = 147.37803649902344 350 / 1452 : pp = 147.6969451904297 360 / 1452 : pp = 147.5704345703125 370 / 1452 : pp = 147.38674926757812 380 / 1452 : pp = 147.03970336914062 390 / 1452 : pp = 147.14231872558594 400 / 1452 : pp = 146.91656494140625 410 / 1452 : pp = 147.34059143066406 420 / 1452 : pp = 147.68496704101562 430 / 1452 : pp = 147.61195373535156 440 / 1452 : pp = 147.68405151367188 450 / 1452 : pp = 147.4711151123047 460 / 1452 : pp = 147.1927032470703 470 / 1452 : pp = 146.72970581054688 480 / 1452 : pp = 146.17173767089844 490 / 1452 : pp = 145.9028778076172 500 / 1452 : pp = 145.42721557617188 510 / 1452 : pp = 145.3111114501953 520 / 1452 : pp = 145.11460876464844 530 / 1452 : pp = 144.81488037109375 540 / 1452 : pp = 144.263916015625 550 / 1452 : pp = 143.997802734375 560 / 1452 : pp = 143.71766662597656 570 / 1452 : pp = 143.47451782226562 580 / 1452 : pp = 143.08474731445312 590 / 1452 : pp = 142.77920532226562 600 / 1452 : pp = 142.39573669433594 610 / 1452 : pp = 142.14906311035156 620 / 1452 : pp = 141.9574432373047 630 / 1452 : pp = 141.67369079589844 640 / 1452 : pp = 141.81556701660156 650 / 1452 : pp = 141.81759643554688 660 / 1452 : pp = 141.9339599609375 670 / 1452 : pp = 142.01248168945312 680 / 1452 : pp = 142.08773803710938 690 / 1452 : pp = 142.00328063964844 700 / 1452 : pp = 141.98086547851562 710 / 1452 : pp = 142.0632781982422 720 / 1452 : pp = 142.10372924804688 730 / 1452 : pp = 142.08055114746094 740 / 1452 : pp = 142.23619079589844 750 / 1452 : pp = 142.2660369873047 760 / 1452 : pp = 142.34678649902344 770 / 1452 : pp = 142.5257568359375 780 / 1452 : pp = 142.70025634765625 790 / 1452 : pp = 142.8614044189453 800 / 1452 : pp = 142.84573364257812 810 / 1452 : pp = 142.8250274658203 820 / 1452 : pp = 142.8540496826172 830 / 1452 : pp = 143.06053161621094 840 / 1452 : pp = 143.0423126220703 850 / 1452 : pp = 143.09634399414062 860 / 1452 : pp = 143.10487365722656 870 / 1452 : pp = 143.0884246826172 880 / 1452 : pp = 143.19387817382812 890 / 1452 : pp = 143.236083984375 900 / 1452 : pp = 143.23390197753906 910 / 1452 : pp = 143.29537963867188 920 / 1452 : pp = 143.3722686767578 930 / 1452 : pp = 143.33795166015625 940 / 1452 : pp = 143.40618896484375 950 / 1452 : pp = 143.3929901123047 960 / 1452 : pp = 143.3693389892578 970 / 1452 : pp = 143.39736938476562 980 / 1452 : pp = 143.2371063232422 990 / 1452 : pp = 143.13893127441406 1000 / 1452 : pp = 142.9658660888672 1010 / 1452 : pp = 143.01544189453125 1020 / 1452 : pp = 143.152587890625 1030 / 1452 : pp = 143.11334228515625 1040 / 1452 : pp = 143.19020080566406 1050 / 1452 : pp = 143.18234252929688 1060 / 1452 : pp = 143.092041015625 1070 / 1452 : pp = 143.24449157714844 1080 / 1452 : pp = 143.34828186035156 1090 / 1452 : pp = 143.38739013671875 1100 / 1452 : pp = 143.37432861328125 1110 / 1452 : pp = 143.20596313476562 1120 / 1452 : pp = 143.07969665527344 1130 / 1452 : pp = 142.92041015625 1140 / 1452 : pp = 142.90902709960938 1150 / 1452 : pp = 143.00732421875 1160 / 1452 : pp = 143.01182556152344 1170 / 1452 : pp = 142.9925994873047 1180 / 1452 : pp = 143.06080627441406 1190 / 1452 : pp = 143.14337158203125 1200 / 1452 : pp = 143.16644287109375 1210 / 1452 : pp = 143.1259002685547 1220 / 1452 : pp = 143.2671661376953 1230 / 1452 : pp = 143.4210968017578 1240 / 1452 : pp = 143.4327850341797 1250 / 1452 : pp = 143.5699920654297 1260 / 1452 : pp = 143.63771057128906 1270 / 1452 : pp = 143.65798950195312 1280 / 1452 : pp = 143.68251037597656 1290 / 1452 : pp = 143.68045043945312 1300 / 1452 : pp = 143.72293090820312 1310 / 1452 : pp = 143.77015686035156 1320 / 1452 : pp = 143.71910095214844 1330 / 1452 : pp = 143.68792724609375 1340 / 1452 : pp = 143.7241668701172 1350 / 1452 : pp = 143.7570037841797 1360 / 1452 : pp = 143.81829833984375 1370 / 1452 : pp = 143.7487030029297 1380 / 1452 : pp = 143.7196502685547 1390 / 1452 : pp = 143.67359924316406 1400 / 1452 : pp = 143.60592651367188 1410 / 1452 : pp = 143.62620544433594 1420 / 1452 : pp = 143.5905303955078 1430 / 1452 : pp = 143.55799865722656 1440 / 1452 : pp = 143.5891571044922 1450 / 1452 : pp = 143.5869598388672 0 / 115 : pp = 226.9864959716797 10 / 115 : pp = 207.8067169189453 20 / 115 : pp = 209.68667602539062 30 / 115 : pp = 207.1610565185547 40 / 115 : pp = 206.3247833251953 50 / 115 : pp = 201.77403259277344 60 / 115 : pp = 201.07098388671875 70 / 115 : pp = 197.33335876464844 80 / 115 : pp = 195.12513732910156 90 / 115 : pp = 192.5349578857422 100 / 115 : pp = 187.90072631835938 110 / 115 : pp = 185.81240844726562 Training perplexity: 143.57354736328125 Validation perplexity:185.40573120117188 Total time : 46.14846849441528 Epoch 11 0 / 1452 : pp = 181.93162536621094 10 / 1452 : pp = 159.94607543945312 20 / 1452 : pp = 156.83673095703125 30 / 1452 : pp = 153.75843811035156 40 / 1452 : pp = 155.18362426757812 50 / 1452 : pp = 152.39529418945312 60 / 1452 : pp = 151.18772888183594 70 / 1452 : pp = 151.9004364013672 80 / 1452 : pp = 151.30239868164062 90 / 1452 : pp = 150.1591033935547 100 / 1452 : pp = 149.18618774414062 110 / 1452 : pp = 147.72653198242188 120 / 1452 : pp = 147.4357452392578 130 / 1452 : pp = 146.41372680664062 140 / 1452 : pp = 145.0057373046875 150 / 1452 : pp = 144.39447021484375 160 / 1452 : pp = 144.5330047607422 170 / 1452 : pp = 144.23593139648438 180 / 1452 : pp = 143.63990783691406 190 / 1452 : pp = 143.63812255859375 200 / 1452 : pp = 144.1143798828125 210 / 1452 : pp = 143.88278198242188 220 / 1452 : pp = 143.92518615722656 230 / 1452 : pp = 144.24032592773438 240 / 1452 : pp = 143.94110107421875 250 / 1452 : pp = 143.3688507080078 260 / 1452 : pp = 142.8829345703125 270 / 1452 : pp = 142.11952209472656 280 / 1452 : pp = 142.19415283203125 290 / 1452 : pp = 142.51889038085938 300 / 1452 : pp = 142.70494079589844 310 / 1452 : pp = 142.51426696777344 320 / 1452 : pp = 142.70106506347656 330 / 1452 : pp = 142.88014221191406 340 / 1452 : pp = 142.3287353515625 350 / 1452 : pp = 142.6169891357422 360 / 1452 : pp = 142.51971435546875 370 / 1452 : pp = 142.33566284179688 380 / 1452 : pp = 142.04161071777344 390 / 1452 : pp = 142.13551330566406 400 / 1452 : pp = 141.9499969482422 410 / 1452 : pp = 142.3361358642578 420 / 1452 : pp = 142.64065551757812 430 / 1452 : pp = 142.5511016845703 440 / 1452 : pp = 142.6728973388672 450 / 1452 : pp = 142.47030639648438 460 / 1452 : pp = 142.1704864501953 470 / 1452 : pp = 141.73390197753906 480 / 1452 : pp = 141.23020935058594 490 / 1452 : pp = 140.9759521484375 500 / 1452 : pp = 140.51609802246094 510 / 1452 : pp = 140.40545654296875 520 / 1452 : pp = 140.1936492919922 530 / 1452 : pp = 139.8929443359375 540 / 1452 : pp = 139.3696746826172 550 / 1452 : pp = 139.13217163085938 560 / 1452 : pp = 138.85247802734375 570 / 1452 : pp = 138.6092987060547 580 / 1452 : pp = 138.2471160888672 590 / 1452 : pp = 137.9485626220703 600 / 1452 : pp = 137.57379150390625 610 / 1452 : pp = 137.31576538085938 620 / 1452 : pp = 137.14230346679688 630 / 1452 : pp = 136.87405395507812 640 / 1452 : pp = 137.02928161621094 650 / 1452 : pp = 137.0481719970703 660 / 1452 : pp = 137.1595001220703 670 / 1452 : pp = 137.21124267578125 680 / 1452 : pp = 137.2671356201172 690 / 1452 : pp = 137.19410705566406 700 / 1452 : pp = 137.1850128173828 710 / 1452 : pp = 137.26058959960938 720 / 1452 : pp = 137.30726623535156 730 / 1452 : pp = 137.28048706054688 740 / 1452 : pp = 137.4352569580078 750 / 1452 : pp = 137.4680938720703 760 / 1452 : pp = 137.5524139404297 770 / 1452 : pp = 137.73829650878906 780 / 1452 : pp = 137.90882873535156 790 / 1452 : pp = 138.05865478515625 800 / 1452 : pp = 138.0673370361328 810 / 1452 : pp = 138.03909301757812 820 / 1452 : pp = 138.084716796875 830 / 1452 : pp = 138.27989196777344 840 / 1452 : pp = 138.23545837402344 850 / 1452 : pp = 138.30343627929688 860 / 1452 : pp = 138.3339080810547 870 / 1452 : pp = 138.32835388183594 880 / 1452 : pp = 138.4450225830078 890 / 1452 : pp = 138.47157287597656 900 / 1452 : pp = 138.46304321289062 910 / 1452 : pp = 138.55618286132812 920 / 1452 : pp = 138.64512634277344 930 / 1452 : pp = 138.6160430908203 940 / 1452 : pp = 138.66932678222656 950 / 1452 : pp = 138.6573028564453 960 / 1452 : pp = 138.6463165283203 970 / 1452 : pp = 138.67059326171875 980 / 1452 : pp = 138.50999450683594 990 / 1452 : pp = 138.42430114746094 1000 / 1452 : pp = 138.25344848632812 1010 / 1452 : pp = 138.3004608154297 1020 / 1452 : pp = 138.4243621826172 1030 / 1452 : pp = 138.40713500976562 1040 / 1452 : pp = 138.47129821777344 1050 / 1452 : pp = 138.45928955078125 1060 / 1452 : pp = 138.3919677734375 1070 / 1452 : pp = 138.5287628173828 1080 / 1452 : pp = 138.62298583984375 1090 / 1452 : pp = 138.6699981689453 1100 / 1452 : pp = 138.64849853515625 1110 / 1452 : pp = 138.49191284179688 1120 / 1452 : pp = 138.37355041503906 1130 / 1452 : pp = 138.2216796875 1140 / 1452 : pp = 138.21534729003906 1150 / 1452 : pp = 138.30963134765625 1160 / 1452 : pp = 138.316162109375 1170 / 1452 : pp = 138.3023681640625 1180 / 1452 : pp = 138.36932373046875 1190 / 1452 : pp = 138.45960998535156 1200 / 1452 : pp = 138.4866180419922 1210 / 1452 : pp = 138.45730590820312 1220 / 1452 : pp = 138.60031127929688 1230 / 1452 : pp = 138.75485229492188 1240 / 1452 : pp = 138.7751007080078 1250 / 1452 : pp = 138.91221618652344 1260 / 1452 : pp = 138.9815216064453 1270 / 1452 : pp = 138.9919891357422 1280 / 1452 : pp = 139.0243377685547 1290 / 1452 : pp = 139.02725219726562 1300 / 1452 : pp = 139.0701446533203 1310 / 1452 : pp = 139.1090850830078 1320 / 1452 : pp = 139.06027221679688 1330 / 1452 : pp = 139.0338134765625 1340 / 1452 : pp = 139.06385803222656 1350 / 1452 : pp = 139.09608459472656 1360 / 1452 : pp = 139.1609649658203 1370 / 1452 : pp = 139.0869903564453 1380 / 1452 : pp = 139.0604705810547 1390 / 1452 : pp = 139.01670837402344 1400 / 1452 : pp = 138.94393920898438 1410 / 1452 : pp = 138.97323608398438 1420 / 1452 : pp = 138.9404296875 1430 / 1452 : pp = 138.90943908691406 1440 / 1452 : pp = 138.94268798828125 1450 / 1452 : pp = 138.93991088867188 0 / 115 : pp = 225.55990600585938 10 / 115 : pp = 207.0504608154297 20 / 115 : pp = 208.98306274414062 30 / 115 : pp = 206.28396606445312 40 / 115 : pp = 205.35386657714844 50 / 115 : pp = 200.7255401611328 60 / 115 : pp = 200.0526580810547 70 / 115 : pp = 196.33087158203125 80 / 115 : pp = 194.12110900878906 90 / 115 : pp = 191.52816772460938 100 / 115 : pp = 186.7974395751953 110 / 115 : pp = 184.59829711914062 Training perplexity: 138.9222869873047 Validation perplexity:184.18101501464844 Total time : 43.92928600311279 Epoch 12 0 / 1452 : pp = 173.0251007080078 10 / 1452 : pp = 152.98446655273438 20 / 1452 : pp = 150.43128967285156 30 / 1452 : pp = 147.5819854736328 40 / 1452 : pp = 149.4164276123047 50 / 1452 : pp = 146.70816040039062 60 / 1452 : pp = 145.557861328125 70 / 1452 : pp = 146.50473022460938 80 / 1452 : pp = 145.83200073242188 90 / 1452 : pp = 144.84402465820312 100 / 1452 : pp = 144.0390167236328 110 / 1452 : pp = 142.66514587402344 120 / 1452 : pp = 142.3549346923828 130 / 1452 : pp = 141.4630126953125 140 / 1452 : pp = 140.2266082763672 150 / 1452 : pp = 139.67518615722656 160 / 1452 : pp = 139.90414428710938 170 / 1452 : pp = 139.5490264892578 180 / 1452 : pp = 138.91969299316406 190 / 1452 : pp = 138.89234924316406 200 / 1452 : pp = 139.40908813476562 210 / 1452 : pp = 139.19068908691406 220 / 1452 : pp = 139.35513305664062 230 / 1452 : pp = 139.5464324951172 240 / 1452 : pp = 139.3047637939453 250 / 1452 : pp = 138.7708740234375 260 / 1452 : pp = 138.29188537597656 270 / 1452 : pp = 137.4787139892578 280 / 1452 : pp = 137.6367950439453 290 / 1452 : pp = 137.98513793945312 300 / 1452 : pp = 138.17819213867188 310 / 1452 : pp = 137.943359375 320 / 1452 : pp = 138.12060546875 330 / 1452 : pp = 138.29037475585938 340 / 1452 : pp = 137.77606201171875 350 / 1452 : pp = 138.06378173828125 360 / 1452 : pp = 137.99000549316406 370 / 1452 : pp = 137.81922912597656 380 / 1452 : pp = 137.52159118652344 390 / 1452 : pp = 137.61782836914062 400 / 1452 : pp = 137.4178924560547 410 / 1452 : pp = 137.82632446289062 420 / 1452 : pp = 138.17567443847656 430 / 1452 : pp = 138.11863708496094 440 / 1452 : pp = 138.215087890625 450 / 1452 : pp = 137.9976348876953 460 / 1452 : pp = 137.6929168701172 470 / 1452 : pp = 137.25416564941406 480 / 1452 : pp = 136.75140380859375 490 / 1452 : pp = 136.51712036132812 500 / 1452 : pp = 136.0896453857422 510 / 1452 : pp = 135.97048950195312 520 / 1452 : pp = 135.7760009765625 530 / 1452 : pp = 135.50389099121094 540 / 1452 : pp = 135.01437377929688 550 / 1452 : pp = 134.7666015625 560 / 1452 : pp = 134.48973083496094 570 / 1452 : pp = 134.22853088378906 580 / 1452 : pp = 133.88455200195312 590 / 1452 : pp = 133.5808868408203 600 / 1452 : pp = 133.22975158691406 610 / 1452 : pp = 132.99591064453125 620 / 1452 : pp = 132.79502868652344 630 / 1452 : pp = 132.5094451904297 640 / 1452 : pp = 132.62892150878906 650 / 1452 : pp = 132.63499450683594 660 / 1452 : pp = 132.7379913330078 670 / 1452 : pp = 132.79046630859375 680 / 1452 : pp = 132.85842895507812 690 / 1452 : pp = 132.80364990234375 700 / 1452 : pp = 132.80477905273438 710 / 1452 : pp = 132.90170288085938 720 / 1452 : pp = 132.92971801757812 730 / 1452 : pp = 132.9019012451172 740 / 1452 : pp = 133.04811096191406 750 / 1452 : pp = 133.10877990722656 760 / 1452 : pp = 133.19189453125 770 / 1452 : pp = 133.3564910888672 780 / 1452 : pp = 133.54000854492188 790 / 1452 : pp = 133.69239807128906 800 / 1452 : pp = 133.68495178222656 810 / 1452 : pp = 133.67971801757812 820 / 1452 : pp = 133.7035675048828 830 / 1452 : pp = 133.89329528808594 840 / 1452 : pp = 133.850341796875 850 / 1452 : pp = 133.90390014648438 860 / 1452 : pp = 133.9090118408203 870 / 1452 : pp = 133.89974975585938 880 / 1452 : pp = 134.0077667236328 890 / 1452 : pp = 134.03485107421875 900 / 1452 : pp = 134.0261688232422 910 / 1452 : pp = 134.10255432128906 920 / 1452 : pp = 134.17291259765625 930 / 1452 : pp = 134.14796447753906 940 / 1452 : pp = 134.20925903320312 950 / 1452 : pp = 134.19281005859375 960 / 1452 : pp = 134.17745971679688 970 / 1452 : pp = 134.18653869628906 980 / 1452 : pp = 134.03192138671875 990 / 1452 : pp = 133.94349670410156 1000 / 1452 : pp = 133.79685974121094 1010 / 1452 : pp = 133.8438262939453 1020 / 1452 : pp = 133.9608612060547 1030 / 1452 : pp = 133.93934631347656 1040 / 1452 : pp = 134.02833557128906 1050 / 1452 : pp = 134.01734924316406 1060 / 1452 : pp = 133.95346069335938 1070 / 1452 : pp = 134.10205078125 1080 / 1452 : pp = 134.2030487060547 1090 / 1452 : pp = 134.23696899414062 1100 / 1452 : pp = 134.2230224609375 1110 / 1452 : pp = 134.0829315185547 1120 / 1452 : pp = 133.980224609375 1130 / 1452 : pp = 133.83815002441406 1140 / 1452 : pp = 133.8366241455078 1150 / 1452 : pp = 133.92108154296875 1160 / 1452 : pp = 133.94375610351562 1170 / 1452 : pp = 133.9360809326172 1180 / 1452 : pp = 133.99684143066406 1190 / 1452 : pp = 134.0944366455078 1200 / 1452 : pp = 134.11676025390625 1210 / 1452 : pp = 134.0911102294922 1220 / 1452 : pp = 134.22763061523438 1230 / 1452 : pp = 134.38043212890625 1240 / 1452 : pp = 134.39817810058594 1250 / 1452 : pp = 134.5367431640625 1260 / 1452 : pp = 134.593017578125 1270 / 1452 : pp = 134.61497497558594 1280 / 1452 : pp = 134.6423797607422 1290 / 1452 : pp = 134.64340209960938 1300 / 1452 : pp = 134.68026733398438 1310 / 1452 : pp = 134.73556518554688 1320 / 1452 : pp = 134.69021606445312 1330 / 1452 : pp = 134.66131591796875 1340 / 1452 : pp = 134.69393920898438 1350 / 1452 : pp = 134.7328643798828 1360 / 1452 : pp = 134.79405212402344 1370 / 1452 : pp = 134.71237182617188 1380 / 1452 : pp = 134.6885528564453 1390 / 1452 : pp = 134.65110778808594 1400 / 1452 : pp = 134.59584045410156 1410 / 1452 : pp = 134.6193389892578 1420 / 1452 : pp = 134.58338928222656 1430 / 1452 : pp = 134.559326171875 1440 / 1452 : pp = 134.59507751464844 1450 / 1452 : pp = 134.59365844726562 0 / 115 : pp = 226.0741729736328 10 / 115 : pp = 207.00494384765625 20 / 115 : pp = 209.26976013183594 30 / 115 : pp = 206.44662475585938 40 / 115 : pp = 205.47268676757812 50 / 115 : pp = 200.7876739501953 60 / 115 : pp = 200.13414001464844 70 / 115 : pp = 196.35549926757812 80 / 115 : pp = 194.10777282714844 90 / 115 : pp = 191.47467041015625 100 / 115 : pp = 186.61351013183594 110 / 115 : pp = 184.30374145507812 Training perplexity: 134.57826232910156 Validation perplexity:183.8900146484375 Total time : 45.410256147384644 Epoch 13 0 / 1452 : pp = 169.39393615722656 10 / 1452 : pp = 150.13232421875 20 / 1452 : pp = 147.60450744628906 30 / 1452 : pp = 144.64317321777344 40 / 1452 : pp = 146.47427368164062 50 / 1452 : pp = 143.929443359375 60 / 1452 : pp = 142.8344268798828 70 / 1452 : pp = 143.45248413085938 80 / 1452 : pp = 142.5418701171875 90 / 1452 : pp = 141.6178436279297 100 / 1452 : pp = 140.70127868652344 110 / 1452 : pp = 139.2852325439453 120 / 1452 : pp = 138.8017120361328 130 / 1452 : pp = 137.85629272460938 140 / 1452 : pp = 136.51718139648438 150 / 1452 : pp = 136.03619384765625 160 / 1452 : pp = 136.154296875 170 / 1452 : pp = 135.67037963867188 180 / 1452 : pp = 135.0376739501953 190 / 1452 : pp = 134.9230499267578 200 / 1452 : pp = 135.4241180419922 210 / 1452 : pp = 135.24581909179688 220 / 1452 : pp = 135.37957763671875 230 / 1452 : pp = 135.67652893066406 240 / 1452 : pp = 135.4161834716797 250 / 1452 : pp = 134.90895080566406 260 / 1452 : pp = 134.46754455566406 270 / 1452 : pp = 133.68577575683594 280 / 1452 : pp = 133.86770629882812 290 / 1452 : pp = 134.18475341796875 300 / 1452 : pp = 134.39132690429688 310 / 1452 : pp = 134.19985961914062 320 / 1452 : pp = 134.37998962402344 330 / 1452 : pp = 134.5557403564453 340 / 1452 : pp = 134.00686645507812 350 / 1452 : pp = 134.27749633789062 360 / 1452 : pp = 134.20286560058594 370 / 1452 : pp = 134.042724609375 380 / 1452 : pp = 133.74398803710938 390 / 1452 : pp = 133.83584594726562 400 / 1452 : pp = 133.64382934570312 410 / 1452 : pp = 134.02366638183594 420 / 1452 : pp = 134.35415649414062 430 / 1452 : pp = 134.310546875 440 / 1452 : pp = 134.3634490966797 450 / 1452 : pp = 134.15602111816406 460 / 1452 : pp = 133.86578369140625 470 / 1452 : pp = 133.43414306640625 480 / 1452 : pp = 132.90310668945312 490 / 1452 : pp = 132.646240234375 500 / 1452 : pp = 132.1982421875 510 / 1452 : pp = 132.04200744628906 520 / 1452 : pp = 131.86940002441406 530 / 1452 : pp = 131.59841918945312 540 / 1452 : pp = 131.12356567382812 550 / 1452 : pp = 130.887939453125 560 / 1452 : pp = 130.6210174560547 570 / 1452 : pp = 130.37826538085938 580 / 1452 : pp = 130.0374755859375 590 / 1452 : pp = 129.75979614257812 600 / 1452 : pp = 129.38308715820312 610 / 1452 : pp = 129.16685485839844 620 / 1452 : pp = 129.0115509033203 630 / 1452 : pp = 128.75152587890625 640 / 1452 : pp = 128.87295532226562 650 / 1452 : pp = 128.88734436035156 660 / 1452 : pp = 128.98275756835938 670 / 1452 : pp = 129.0487060546875 680 / 1452 : pp = 129.11013793945312 690 / 1452 : pp = 129.0646514892578 700 / 1452 : pp = 129.06280517578125 710 / 1452 : pp = 129.1343994140625 720 / 1452 : pp = 129.18582153320312 730 / 1452 : pp = 129.15138244628906 740 / 1452 : pp = 129.29811096191406 750 / 1452 : pp = 129.339599609375 760 / 1452 : pp = 129.4257354736328 770 / 1452 : pp = 129.61631774902344 780 / 1452 : pp = 129.802734375 790 / 1452 : pp = 129.96804809570312 800 / 1452 : pp = 129.95187377929688 810 / 1452 : pp = 129.92417907714844 820 / 1452 : pp = 129.9774627685547 830 / 1452 : pp = 130.1638946533203 840 / 1452 : pp = 130.13095092773438 850 / 1452 : pp = 130.16595458984375 860 / 1452 : pp = 130.173828125 870 / 1452 : pp = 130.170166015625 880 / 1452 : pp = 130.27032470703125 890 / 1452 : pp = 130.3022003173828 900 / 1452 : pp = 130.3071746826172 910 / 1452 : pp = 130.37939453125 920 / 1452 : pp = 130.46229553222656 930 / 1452 : pp = 130.43846130371094 940 / 1452 : pp = 130.50889587402344 950 / 1452 : pp = 130.50086975097656 960 / 1452 : pp = 130.4833221435547 970 / 1452 : pp = 130.50814819335938 980 / 1452 : pp = 130.35577392578125 990 / 1452 : pp = 130.26759338378906 1000 / 1452 : pp = 130.1064453125 1010 / 1452 : pp = 130.1472625732422 1020 / 1452 : pp = 130.27169799804688 1030 / 1452 : pp = 130.25100708007812 1040 / 1452 : pp = 130.30816650390625 1050 / 1452 : pp = 130.29803466796875 1060 / 1452 : pp = 130.2242431640625 1070 / 1452 : pp = 130.35906982421875 1080 / 1452 : pp = 130.45103454589844 1090 / 1452 : pp = 130.49838256835938 1100 / 1452 : pp = 130.484130859375 1110 / 1452 : pp = 130.35316467285156 1120 / 1452 : pp = 130.24697875976562 1130 / 1452 : pp = 130.10804748535156 1140 / 1452 : pp = 130.1076202392578 1150 / 1452 : pp = 130.195068359375 1160 / 1452 : pp = 130.19674682617188 1170 / 1452 : pp = 130.18321228027344 1180 / 1452 : pp = 130.24623107910156 1190 / 1452 : pp = 130.33905029296875 1200 / 1452 : pp = 130.3650360107422 1210 / 1452 : pp = 130.34588623046875 1220 / 1452 : pp = 130.4850616455078 1230 / 1452 : pp = 130.63160705566406 1240 / 1452 : pp = 130.64674377441406 1250 / 1452 : pp = 130.77078247070312 1260 / 1452 : pp = 130.8397674560547 1270 / 1452 : pp = 130.8511199951172 1280 / 1452 : pp = 130.88967895507812 1290 / 1452 : pp = 130.9040985107422 1300 / 1452 : pp = 130.93511962890625 1310 / 1452 : pp = 130.9759063720703 1320 / 1452 : pp = 130.92800903320312 1330 / 1452 : pp = 130.9105224609375 1340 / 1452 : pp = 130.929443359375 1350 / 1452 : pp = 130.96153259277344 1360 / 1452 : pp = 131.02381896972656 1370 / 1452 : pp = 130.9545440673828 1380 / 1452 : pp = 130.9344940185547 1390 / 1452 : pp = 130.9055938720703 1400 / 1452 : pp = 130.85386657714844 1410 / 1452 : pp = 130.8874969482422 1420 / 1452 : pp = 130.85928344726562 1430 / 1452 : pp = 130.83995056152344 1440 / 1452 : pp = 130.86659240722656 1450 / 1452 : pp = 130.86839294433594 0 / 115 : pp = 227.78428649902344 10 / 115 : pp = 207.609619140625 20 / 115 : pp = 209.92459106445312 30 / 115 : pp = 206.96240234375 40 / 115 : pp = 205.9295654296875 50 / 115 : pp = 201.0296630859375 60 / 115 : pp = 200.38059997558594 70 / 115 : pp = 196.55764770507812 80 / 115 : pp = 194.31735229492188 90 / 115 : pp = 191.66146850585938 100 / 115 : pp = 186.70437622070312 110 / 115 : pp = 184.3171844482422 Training perplexity: 130.85043334960938 Validation perplexity:183.88186645507812 Total time : 45.345656394958496 Epoch 14 0 / 1452 : pp = 164.82191467285156 10 / 1452 : pp = 146.39089965820312 20 / 1452 : pp = 142.93240356445312 30 / 1452 : pp = 140.3113555908203 40 / 1452 : pp = 142.39939880371094 50 / 1452 : pp = 139.70162963867188 60 / 1452 : pp = 138.73023986816406 70 / 1452 : pp = 139.2675018310547 80 / 1452 : pp = 138.47824096679688 90 / 1452 : pp = 137.40432739257812 100 / 1452 : pp = 136.47793579101562 110 / 1452 : pp = 135.2294464111328 120 / 1452 : pp = 134.80728149414062 130 / 1452 : pp = 133.89822387695312 140 / 1452 : pp = 132.54141235351562 150 / 1452 : pp = 132.10025024414062 160 / 1452 : pp = 132.21829223632812 170 / 1452 : pp = 131.8765106201172 180 / 1452 : pp = 131.37515258789062 190 / 1452 : pp = 131.31622314453125 200 / 1452 : pp = 131.78297424316406 210 / 1452 : pp = 131.5507354736328 220 / 1452 : pp = 131.7002410888672 230 / 1452 : pp = 131.9277801513672 240 / 1452 : pp = 131.72166442871094 250 / 1452 : pp = 131.225830078125 260 / 1452 : pp = 130.7496337890625 270 / 1452 : pp = 129.9896697998047 280 / 1452 : pp = 130.10594177246094 290 / 1452 : pp = 130.41644287109375 300 / 1452 : pp = 130.5982208251953 310 / 1452 : pp = 130.36329650878906 320 / 1452 : pp = 130.5633544921875 330 / 1452 : pp = 130.77252197265625 340 / 1452 : pp = 130.273193359375 350 / 1452 : pp = 130.47889709472656 360 / 1452 : pp = 130.4348602294922 370 / 1452 : pp = 130.28126525878906 380 / 1452 : pp = 130.02786254882812 390 / 1452 : pp = 130.1564483642578 400 / 1452 : pp = 129.98440551757812 410 / 1452 : pp = 130.37721252441406 420 / 1452 : pp = 130.71859741210938 430 / 1452 : pp = 130.65939331054688 440 / 1452 : pp = 130.72987365722656 450 / 1452 : pp = 130.56272888183594 460 / 1452 : pp = 130.28195190429688 470 / 1452 : pp = 129.90936279296875 480 / 1452 : pp = 129.42857360839844 490 / 1452 : pp = 129.18077087402344 500 / 1452 : pp = 128.7588348388672 510 / 1452 : pp = 128.6303253173828 520 / 1452 : pp = 128.47616577148438 530 / 1452 : pp = 128.21148681640625 540 / 1452 : pp = 127.7218017578125 550 / 1452 : pp = 127.50067138671875 560 / 1452 : pp = 127.27574157714844 570 / 1452 : pp = 127.05399322509766 580 / 1452 : pp = 126.73983001708984 590 / 1452 : pp = 126.43692779541016 600 / 1452 : pp = 126.06050109863281 610 / 1452 : pp = 125.82952880859375 620 / 1452 : pp = 125.66295623779297 630 / 1452 : pp = 125.39354705810547 640 / 1452 : pp = 125.49463653564453 650 / 1452 : pp = 125.48816680908203 660 / 1452 : pp = 125.58712005615234 670 / 1452 : pp = 125.65978240966797 680 / 1452 : pp = 125.71456146240234 690 / 1452 : pp = 125.66937255859375 700 / 1452 : pp = 125.65900421142578 710 / 1452 : pp = 125.7271499633789 720 / 1452 : pp = 125.77758026123047 730 / 1452 : pp = 125.74129486083984 740 / 1452 : pp = 125.8759765625 750 / 1452 : pp = 125.91793823242188 760 / 1452 : pp = 125.99595642089844 770 / 1452 : pp = 126.18113708496094 780 / 1452 : pp = 126.35147094726562 790 / 1452 : pp = 126.50797271728516 800 / 1452 : pp = 126.49759674072266 810 / 1452 : pp = 126.48113250732422 820 / 1452 : pp = 126.52528381347656 830 / 1452 : pp = 126.705810546875 840 / 1452 : pp = 126.67517852783203 850 / 1452 : pp = 126.74176025390625 860 / 1452 : pp = 126.74151611328125 870 / 1452 : pp = 126.73414611816406 880 / 1452 : pp = 126.83026885986328 890 / 1452 : pp = 126.88519287109375 900 / 1452 : pp = 126.88053894042969 910 / 1452 : pp = 126.97138214111328 920 / 1452 : pp = 127.04660034179688 930 / 1452 : pp = 127.03763580322266 940 / 1452 : pp = 127.1126480102539 950 / 1452 : pp = 127.09610748291016 960 / 1452 : pp = 127.0873794555664 970 / 1452 : pp = 127.10343933105469 980 / 1452 : pp = 126.96441650390625 990 / 1452 : pp = 126.88519287109375 1000 / 1452 : pp = 126.7336654663086 1010 / 1452 : pp = 126.77796936035156 1020 / 1452 : pp = 126.89826202392578 1030 / 1452 : pp = 126.88761138916016 1040 / 1452 : pp = 126.95309448242188 1050 / 1452 : pp = 126.96478271484375 1060 / 1452 : pp = 126.89324188232422 1070 / 1452 : pp = 127.03242492675781 1080 / 1452 : pp = 127.13228607177734 1090 / 1452 : pp = 127.173095703125 1100 / 1452 : pp = 127.15975189208984 1110 / 1452 : pp = 127.0392074584961 1120 / 1452 : pp = 126.94032287597656 1130 / 1452 : pp = 126.80693054199219 1140 / 1452 : pp = 126.81315612792969 1150 / 1452 : pp = 126.90467834472656 1160 / 1452 : pp = 126.91236114501953 1170 / 1452 : pp = 126.90897369384766 1180 / 1452 : pp = 126.98052215576172 1190 / 1452 : pp = 127.07483673095703 1200 / 1452 : pp = 127.10216522216797 1210 / 1452 : pp = 127.08258819580078 1220 / 1452 : pp = 127.22943878173828 1230 / 1452 : pp = 127.38563537597656 1240 / 1452 : pp = 127.40538024902344 1250 / 1452 : pp = 127.53369140625 1260 / 1452 : pp = 127.59293365478516 1270 / 1452 : pp = 127.61489868164062 1280 / 1452 : pp = 127.6484375 1290 / 1452 : pp = 127.65257263183594 1300 / 1452 : pp = 127.69329833984375 1310 / 1452 : pp = 127.74549102783203 1320 / 1452 : pp = 127.7043228149414 1330 / 1452 : pp = 127.6866683959961 1340 / 1452 : pp = 127.70913696289062 1350 / 1452 : pp = 127.73233795166016 1360 / 1452 : pp = 127.7855224609375 1370 / 1452 : pp = 127.71918487548828 1380 / 1452 : pp = 127.69987487792969 1390 / 1452 : pp = 127.6697998046875 1400 / 1452 : pp = 127.61137390136719 1410 / 1452 : pp = 127.6404037475586 1420 / 1452 : pp = 127.61094665527344 1430 / 1452 : pp = 127.58216857910156 1440 / 1452 : pp = 127.61477661132812 1450 / 1452 : pp = 127.61964416503906 0 / 115 : pp = 228.21578979492188 10 / 115 : pp = 208.11244201660156 20 / 115 : pp = 210.688232421875 30 / 115 : pp = 207.62408447265625 40 / 115 : pp = 206.45184326171875 50 / 115 : pp = 201.52760314941406 60 / 115 : pp = 200.7784881591797 70 / 115 : pp = 196.83067321777344 80 / 115 : pp = 194.6357879638672 90 / 115 : pp = 191.9783935546875 100 / 115 : pp = 186.8787841796875 110 / 115 : pp = 184.35252380371094 Training perplexity: 127.60413360595703 Validation perplexity:183.8877410888672 Total time : 41.6636528968811 Epoch 15 0 / 1452 : pp = 156.81654357910156 10 / 1452 : pp = 142.1070556640625 20 / 1452 : pp = 139.55076599121094 30 / 1452 : pp = 136.63551330566406 40 / 1452 : pp = 138.5840606689453 50 / 1452 : pp = 136.052734375 60 / 1452 : pp = 134.93019104003906 70 / 1452 : pp = 135.65206909179688 80 / 1452 : pp = 135.2620086669922 90 / 1452 : pp = 134.314697265625 100 / 1452 : pp = 133.4916229248047 110 / 1452 : pp = 132.26052856445312 120 / 1452 : pp = 131.7714080810547 130 / 1452 : pp = 130.77365112304688 140 / 1452 : pp = 129.5411834716797 150 / 1452 : pp = 129.0791778564453 160 / 1452 : pp = 129.21920776367188 170 / 1452 : pp = 128.7528839111328 180 / 1452 : pp = 128.22279357910156 190 / 1452 : pp = 128.18177795410156 200 / 1452 : pp = 128.58758544921875 210 / 1452 : pp = 128.3906707763672 220 / 1452 : pp = 128.5266571044922 230 / 1452 : pp = 128.80563354492188 240 / 1452 : pp = 128.61886596679688 250 / 1452 : pp = 128.13172912597656 260 / 1452 : pp = 127.69220733642578 270 / 1452 : pp = 126.96150970458984 280 / 1452 : pp = 127.04702758789062 290 / 1452 : pp = 127.33565521240234 300 / 1452 : pp = 127.55929565429688 310 / 1452 : pp = 127.38514709472656 320 / 1452 : pp = 127.52171325683594 330 / 1452 : pp = 127.68690490722656 340 / 1452 : pp = 127.18340301513672 350 / 1452 : pp = 127.4073257446289 360 / 1452 : pp = 127.30432891845703 370 / 1452 : pp = 127.17618560791016 380 / 1452 : pp = 126.92579650878906 390 / 1452 : pp = 127.02473449707031 400 / 1452 : pp = 126.8515625 410 / 1452 : pp = 127.211669921875 420 / 1452 : pp = 127.51788330078125 430 / 1452 : pp = 127.47386169433594 440 / 1452 : pp = 127.57164001464844 450 / 1452 : pp = 127.3601303100586 460 / 1452 : pp = 127.09434509277344 470 / 1452 : pp = 126.71922302246094 480 / 1452 : pp = 126.24349212646484 490 / 1452 : pp = 125.98778533935547 500 / 1452 : pp = 125.59526824951172 510 / 1452 : pp = 125.4450912475586 520 / 1452 : pp = 125.29247283935547 530 / 1452 : pp = 125.03536224365234 540 / 1452 : pp = 124.5813980102539 550 / 1452 : pp = 124.33724212646484 560 / 1452 : pp = 124.08995819091797 570 / 1452 : pp = 123.86637878417969 580 / 1452 : pp = 123.53152465820312 590 / 1452 : pp = 123.20321655273438 600 / 1452 : pp = 122.85673522949219 610 / 1452 : pp = 122.64250946044922 620 / 1452 : pp = 122.4958724975586 630 / 1452 : pp = 122.22386169433594 640 / 1452 : pp = 122.31143188476562 650 / 1452 : pp = 122.30093383789062 660 / 1452 : pp = 122.39427947998047 670 / 1452 : pp = 122.45440673828125 680 / 1452 : pp = 122.51146697998047 690 / 1452 : pp = 122.4854736328125 700 / 1452 : pp = 122.48600006103516 710 / 1452 : pp = 122.56084442138672 720 / 1452 : pp = 122.59059143066406 730 / 1452 : pp = 122.55529022216797 740 / 1452 : pp = 122.69409942626953 750 / 1452 : pp = 122.76456451416016 760 / 1452 : pp = 122.84437561035156 770 / 1452 : pp = 123.02527618408203 780 / 1452 : pp = 123.20509338378906 790 / 1452 : pp = 123.36305236816406 800 / 1452 : pp = 123.36852264404297 810 / 1452 : pp = 123.36799621582031 820 / 1452 : pp = 123.39976501464844 830 / 1452 : pp = 123.59362030029297 840 / 1452 : pp = 123.56946563720703 850 / 1452 : pp = 123.63800811767578 860 / 1452 : pp = 123.63983917236328 870 / 1452 : pp = 123.64148712158203 880 / 1452 : pp = 123.7568588256836 890 / 1452 : pp = 123.7885513305664 900 / 1452 : pp = 123.79640197753906 910 / 1452 : pp = 123.86153411865234 920 / 1452 : pp = 123.92941284179688 930 / 1452 : pp = 123.9125747680664 940 / 1452 : pp = 123.95559692382812 950 / 1452 : pp = 123.93928527832031 960 / 1452 : pp = 123.94294738769531 970 / 1452 : pp = 123.95547485351562 980 / 1452 : pp = 123.8229751586914 990 / 1452 : pp = 123.73727416992188 1000 / 1452 : pp = 123.59091186523438 1010 / 1452 : pp = 123.634765625 1020 / 1452 : pp = 123.76506042480469 1030 / 1452 : pp = 123.75485229492188 1040 / 1452 : pp = 123.807861328125 1050 / 1452 : pp = 123.79156494140625 1060 / 1452 : pp = 123.73054504394531 1070 / 1452 : pp = 123.8615951538086 1080 / 1452 : pp = 123.96564483642578 1090 / 1452 : pp = 124.02104187011719 1100 / 1452 : pp = 124.012939453125 1110 / 1452 : pp = 123.87582397460938 1120 / 1452 : pp = 123.775390625 1130 / 1452 : pp = 123.63182067871094 1140 / 1452 : pp = 123.62391662597656 1150 / 1452 : pp = 123.71013641357422 1160 / 1452 : pp = 123.72423553466797 1170 / 1452 : pp = 123.71726989746094 1180 / 1452 : pp = 123.79032897949219 1190 / 1452 : pp = 123.87883758544922 1200 / 1452 : pp = 123.9125747680664 1210 / 1452 : pp = 123.90140533447266 1220 / 1452 : pp = 124.03245544433594 1230 / 1452 : pp = 124.19799041748047 1240 / 1452 : pp = 124.21469116210938 1250 / 1452 : pp = 124.34103393554688 1260 / 1452 : pp = 124.4041976928711 1270 / 1452 : pp = 124.42852020263672 1280 / 1452 : pp = 124.46656036376953 1290 / 1452 : pp = 124.4811019897461 1300 / 1452 : pp = 124.52384185791016 1310 / 1452 : pp = 124.57533264160156 1320 / 1452 : pp = 124.5398178100586 1330 / 1452 : pp = 124.52598571777344 1340 / 1452 : pp = 124.53311157226562 1350 / 1452 : pp = 124.57759094238281 1360 / 1452 : pp = 124.63385772705078 1370 / 1452 : pp = 124.58133697509766 1380 / 1452 : pp = 124.55769348144531 1390 / 1452 : pp = 124.54011535644531 1400 / 1452 : pp = 124.4884033203125 1410 / 1452 : pp = 124.51226806640625 1420 / 1452 : pp = 124.49683380126953 1430 / 1452 : pp = 124.4754638671875 1440 / 1452 : pp = 124.50164031982422 1450 / 1452 : pp = 124.50894165039062 0 / 115 : pp = 230.8488006591797 10 / 115 : pp = 209.2509002685547 20 / 115 : pp = 211.68577575683594 30 / 115 : pp = 208.44056701660156 40 / 115 : pp = 207.2039337158203 50 / 115 : pp = 202.1859588623047 60 / 115 : pp = 201.34739685058594 70 / 115 : pp = 197.4251251220703 80 / 115 : pp = 195.2623291015625 90 / 115 : pp = 192.592529296875 100 / 115 : pp = 187.39553833007812 110 / 115 : pp = 184.791259765625 Training perplexity: 124.4933853149414 Validation perplexity:184.32510375976562 Total time : 40.856229066848755 0 / 128 : pp = 184.6475067138672 10 / 128 : pp = 176.8856964111328 20 / 128 : pp = 164.3444366455078 30 / 128 : pp = 167.85472106933594 40 / 128 : pp = 169.25367736816406 50 / 128 : pp = 168.86561584472656 60 / 128 : pp = 168.11801147460938 70 / 128 : pp = 165.4105224609375 80 / 128 : pp = 162.91146850585938 90 / 128 : pp = 161.29742431640625 100 / 128 : pp = 162.45989990234375 110 / 128 : pp = 162.6834716796875 120 / 128 : pp = 164.3359832763672 =-==-==-==-==-= Test perplexity: 164.0149383544922 =-==-==-==-==-=
更详细的内容请参考下面链接
https://github.com/weizhenzhao/cs224d_nlp_problem_set2
今天将的还是cs224d 的problem set2 的第三部分习题，
原来国外大学的系统难度真的如此之大，相比之下还是默默地再天朝继续搬砖吧
下面讲述一下RNN语言建模的数学公式:
给出一串连续的词x1,x2...xt关于预测其后面紧跟的词xt+1的建模方式是:
vj是词库中的某个词。实现一个循环神经网络，此网络利用隐层中的反馈信息对"历史记录"x1,x2...xt进行建模:
$h^{(0)}=h_{0}\epsilon R^{D_{h}}$是隐藏层的初始化向量
$x^{(t)}L$是以$x^{(t)}$one-hot行向量与嵌入矩阵L的乘积
这个one-hot行向量就是当前处理词汇的索引
是词嵌入矩阵,
$L$是词嵌入矩阵
$I$是输入词表征矩阵
$H$是隐藏转换矩阵
$U$是输出词表征矩阵
$b_{1}$ $b_{2}$是偏置值
$d$是词嵌入的维数
|V|代表词库的规模
$D_{h}$是隐层的维数
输出向量
是面向整个词库的概率分布，我们需要最优化交叉熵（非正则化的）的损失率：
使用困惑度来评估语言模型的性能，其定义形式如下：
梯度:
该模型中各个变量进行最优化迭代的时候的梯度如下所示:
初始化所有的上面这些需要训练的参数的值
然后通过对每一个词进行训练，安装上述公司求出每个参数的导数值
然后使用梯度下降方法对其进行更新
将新得到的参数代入到模型中，如果损失的值小于初始设定的值则停止迭代，否则继续进行迭代
下面是一张RNNLM的结构图
上面这张是第二层RNN节点的结构图
上面这张是在RNN的变量上面应用Dropout的结构，降低模型过拟合的误差，第一层RNN的dropout结构
上面这张是第一层RNN的结构图
（注意前方高能，一大批天书即将来袭）
''' Created on 2017年9月26日 @author: weizhen ''' import getpass import sys import time import numpy as np from copy import deepcopy from utils import calculate_perplexity, get_ptb_dataset, Vocab from utils import ptb_iterator, sample import tensorflow as tf from model import LanguageModel from tensorflow.contrib.legacy_seq2seq.python.ops.seq2seq import sequence_loss class Config(object): """储存超参数和数据信息""" batch_size = 64 embed_size = 50 hidden_size = 100 num_steps = 10 max_epochs = 16 early_stopping = 2 dropout = 0.9 lr = 0.001 class RNNLM_Model(LanguageModel): def load_data(self, debug=False): """加载词向量并且训练 train/dev/test 数据""" self.vocab = Vocab() self.vocab.construct(get_ptb_dataset('train')) self.encoded_train = np.array([self.vocab.encode(word) for word in get_ptb_dataset('train')], dtype=np.int32) self.encoded_valid = np.array([self.vocab.encode(word) for word in get_ptb_dataset('valid')], dtype=np.int32) self.encoded_test = np.array([self.vocab.encode(word) for word in get_ptb_dataset('test')]) if debug: num_debug = 1024 self.encoded_train = self.encoded_train[:num_debug] self.encoded_valid = self.encoded_valid[:num_debug] self.encoded_test = self.encoded_test[:num_debug] def add_placeholders(self): """生成placeholder 变量来表示输入的 tensors 这些placeholder 被用来在模型的其他地方被填充 并且在训练的过程中会被填充 input_placeholder:Input placeholder shape (None,num_steps),type tf.int32 labels_placeholder:label placeholder shape (None,num_steps) type tf.float32 dropout_placeholder:dropput value placeholder (scalar), type tf.float32 """ self.input_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Input') self.labels_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Target') self.dropout_placeholder = tf.placeholder(tf.float32, name='Dropout') def add_embedding(self): """添加词嵌入层 Hint : 这一层应该用input_placeholder 来索引词嵌入 Hint : 你或许能发现tf.nn.embedding_lookup 是有用的 Hint : 你或许能发现tf.split , tf.squeeze 是有用的在构造tensor 的输入的时候 Hint : 下面是你需要创建的变量的维度 L:(len(self.vocab),embed_size) Returns: inputs:一个训练次数的列表,每一个元素应该是 一个张量 大小是 (batch_size,embed_size) tf.split(dimension,num_split,input) dimension表示输入张量的哪一个维度, 如果是0就表示对第0维度进行切割, num_split就是切割的数量, 如果是2就表示输入张量被切成2份, 每一份是一个列表 tf.squeeze(input,squeeze_dims=None,name=None) 从tensor中删除所有大小是1的维度 example: t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t))==>[2,3] t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t,[2,4]))==>[1,2,3,1] tf.nn.embedding_lookup 将词的索引映射到词的向量 """ with tf.device('/cpu:0'): embedding = tf.get_variable('Embedding', [len(self.vocab), self.config.embed_size], trainable=True) inputs = tf.nn.embedding_lookup(embedding, self.input_placeholder) inputs = [tf.squeeze(x, [1]) for x in tf.split(inputs, self.config.num_steps, 1)] return inputs def add_projection(self, rnn_outputs): """添加一个投影层 投影层将隐藏层的表示变换到整个词向量上的分布式表示 Hint:下面是你需要去创建的维度 U(hidden_size,len(vocab)) b_2:(len(vocab),) 参数: rnn_outputs:一个训练次数的列表，每一个元素应该是一个张量 大小是(batch_size,embed_size) Returns: outputs:一个长度的列表，每一个元素是一个张量(batch_size,len(vocab)) """ with tf.variable_scope('Projection'): U = tf.get_variable('Matrix', [self.config.hidden_size, len(self.vocab)]) proj_b = tf.get_variable('Bias', [len(self.vocab)]) outputs = [tf.matmul(o, U) + proj_b for o in rnn_outputs] return outputs def add_loss_op(self, output): """将损失添加到目标函数上面 Hint:使用tensorflow.python.ops.seq2seq.sequence_loss 来实现序列损失 参数: 输出:一个张量 大小是 (None,self.vocab) 返回: 损失：一个0-d大小的张量 """ all_ones = [tf.ones([self.config.batch_size * self.config.num_steps])] cross_entropy = sequence_loss([output], [tf.reshape(self.labels_placeholder, [-1])], all_ones, len(self.vocab)) tf.add_to_collection('total_loss', cross_entropy) loss = tf.add_n(tf.get_collection('total_loss')) return loss def add_training_op(self, loss): """将目标损失添加到计算图上 创建一个优化器并且应用梯度下降到所有的训练变量上面 Hint:使用tf.train.AdamOptimizer 对于这个模型 使用optimizer.minimize() 会返回一个train_op的对象 参数: loss: 损失张量，来自于cross_entropy_loss 交叉熵损失 返回: train_op:训练的目标 """ with tf.variable_scope("Optimizer") as scope: train_op = tf.train.AdamOptimizer(self.config.lr).minimize(loss) return train_op def __init__(self, config): self.config = config self.load_data(debug=False) self.add_placeholders() self.inputs = self.add_embedding() self.rnn_outputs = self.add_model(self.inputs) self.outputs = self.add_projection(self.rnn_outputs) # 我们想去检验下一个词预测得多好 # 我们把o转变成float64 位 因为如果不这样就会有数值问题 # sum(output of softmax) = 1.00000298179 并且不是 1 self.predictions = [tf.nn.softmax(tf.cast(o, 'float64')) for o in self.outputs] # 将输出值转变成 len(vocab) 的大小 output = tf.reshape(tf.concat(self.outputs, 1), [-1, len(self.vocab)]) self.calculate_loss = self.add_loss_op(output) self.train_step = self.add_training_op(self.calculate_loss) def add_model(self, inputs): """创建RNN LM 模型 在下面的实现里面你需要去实现RNN LM 模型的等式 Hint: 使用一个零向量 大小是 (batch_size,hidden_size) 作为初始的RNN的状态 Hint: 将最后RNN输出 作为实例变量 self.final_state Hint : 确保将dropout应用到 输入和输出的 变量上面 Hint : 使用变量域 RNN 来定义 RNN变量 Hint : 表现一个明显的 for-loop 在输入上面 你可以使用scope.reuse_variable() 来确定权重 在每一次迭代都是相同的 确保不会在第一次循环的时候调用这个，因为没有变量会被初始化 Hint : 下面变量的不同的维度 ， 你需要去创建的 H: (hidden_size,hidden_size) I: (embed_size,hidden_size) b_1:(hidden_size,) Args: inputs:一个记录num_steps的列表，里边的每一个元素应该是一个张量 大小是(batch_size,embed_size)的大小 Returns:返回 outputs:一个记录num_steps的列表，里面每一个元素应该是一个张量 大小是(batch_size,hidden_size) """ with tf.variable_scope('InputDropout'): inputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in inputs] with tf.variable_scope('RNN') as scope: self.initial_state = tf.zeros([self.config.batch_size, self.config.hidden_size]) state = self.initial_state rnn_outputs = [] for tstep, current_input in enumerate(inputs): if tstep > 0: scope.reuse_variables() RNN_H = tf.get_variable('HMatrix', [self.config.hidden_size, self.config.hidden_size]) RNN_I = tf.get_variable('IMatrix', [self.config.embed_size, self.config.hidden_size]) RNN_b = tf.get_variable('B', [self.config.hidden_size]) state = tf.nn.sigmoid(tf.matmul(state, RNN_H) + tf.matmul(current_input, RNN_I) + RNN_b) rnn_outputs.append(state) self.final_state = rnn_outputs[-1] with tf.variable_scope('RNNDropout'): rnn_outputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in rnn_outputs] return rnn_outputs def run_epoch(self, session, data, train_op=None, verbose=10): config = self.config dp = config.dropout if not train_op: train_op = tf.no_op() dp = 1 total_steps = sum(1 for x in ptb_iterator(data, config.batch_size, config.num_steps)) total_loss = [] state = self.initial_state.eval() for step, (x, y) in enumerate(ptb_iterator(data, config.batch_size, config.num_steps)): # 我们需要通过初始状态，并且从最终状态中抽取数据来进行填充 # RNN 合适的 历史 feed = {self.input_placeholder: x, self.labels_placeholder: y, self.initial_state: state, self.dropout_placeholder: dp } loss, state, _ = session.run([self.calculate_loss, self.final_state, train_op], feed_dict=feed) total_loss.append(loss) if verbose and step % verbose == 0: sys.stdout.write('\r{} / {} : pp = {} '.format(step, total_steps, np.exp(np.mean(total_loss)))) sys.stdout.flush() if verbose: sys.stdout.write('\r') return np.exp(np.mean(total_loss)) def generate_text(session, model, config, starting_text='<eos>', stop_length=100, stop_tokens=None, temp=1.0): """从模型自动生成文字 Hint:创建一个feed-dictionary 并且使用sess.run()方法去执行这个模型 你会需要使用model.initial_state 作为一个键传递给feed_dict Hint:得到model.final_state 和 model.predictions[-1]. 在add_model()方法中设置model.final_state 。 model.predictions 是在 __init__方法中设置的 Hint:在模型的训练中存储输出的参数值，和预测的y_pred的值 参数: Args: session : tf.Session() object model : Object of type RNNLM Model config : A Config() object starting_text:Initial text passed to model Returns: output : List of word idxs """ state = model.initial_state.eval() # Imagine tokens as a batch size of one, length of len(tokens[0]) tokens = [model.vocab.encode(word) for word in starting_text.split()] for i in range(stop_length): feed = {model.input_placeholder: [tokens[-1:]], model.initial_state: state, model.dropout_placeholder: 1} state, y_pred = session.run([model.final_state, model.predictions[-1]], feed_dict=feed) next_word_idx = sample(y_pred[0], temperature=temp) tokens.append(next_word_idx) if stop_tokens and model.vocab.decode(tokens[-1]) in stop_tokens: break output = [model.vocab.decode(word_idx) for word_idx in tokens] return output def generate_sentence(session, model, config, *args, **kwargs): """方便从模型来生成句子""" return generate_text(session, model, config, *args, stop_tokens=['<eos>'], **kwargs) def test_RNNLM(): config = Config() gen_config = deepcopy(config) gen_config.batch_size = gen_config.num_steps = 1 # 创建训练模型，并且生成模型 with tf.variable_scope('RNNLM',reuse=None) as scope: model = RNNLM_Model(config) # 这个指示gen_model来重新使用相同的变量作为以上的模型 scope.reuse_variables() gen_model = RNNLM_Model(gen_config) init = tf.global_variables_initializer() saver = tf.train.Saver() with tf.Session() as session: best_val_pp = float('inf') best_val_epoch = 0 session.run(init) for epoch in range(config.max_epochs): print('Epoch {0}'.format(epoch)) start = time.time() train_pp = model.run_epoch(session, model.encoded_train, train_op=model.train_step) valid_pp = model.run_epoch(session, model.encoded_valid) print('Training perplexity: {0}'.format(train_pp)) print('Validation perplexity:{0}'.format(valid_pp)) if valid_pp < best_val_pp: best_val_pp = valid_pp best_val_epoch = epoch saver.save(session, './ptb_rnnlm.weights') if epoch - best_val_epoch > config.early_stopping: break print('Total time : {0}'.format(time.time() - start)) saver.restore(session, 'ptb_rnnlm.weights') test_pp = model.run_epoch(session, model.encoded_test) print('=-=' * 5) print('Test perplexity: {0} '.format(test_pp)) print('=-=' * 5) starting_text = 'in palo alto' while starting_text: print(' '.join(generate_sentence(session, gen_model, gen_config, starting_text=starting_text, temp=1.0))) #starting_text = raw_input('>') if __name__ == "__main__": test_RNNLM()
（其实也不算是天书啦，比高数简单多啦，比数学分析那是简单了好几十万倍了呀）
下面是训练的Log
1380 / 1452 : pp = 266.20892333984375 1390 / 1452 : pp = 265.94439697265625 1400 / 1452 : pp = 265.66845703125 1410 / 1452 : pp = 265.5393981933594 1420 / 1452 : pp = 265.32489013671875 1430 / 1452 : pp = 265.2019348144531 1440 / 1452 : pp = 265.13720703125 1450 / 1452 : pp = 264.954833984375 0 / 115 : pp = 296.9217224121094 10 / 115 : pp = 282.02130126953125 20 / 115 : pp = 279.76800537109375 30 / 115 : pp = 276.4101257324219 40 / 115 : pp = 276.2939147949219 50 / 115 : pp = 270.73565673828125 60 / 115 : pp = 269.88134765625 70 / 115 : pp = 266.8675231933594 80 / 115 : pp = 263.6731872558594 90 / 115 : pp = 260.8569030761719 100 / 115 : pp = 256.3356628417969 110 / 115 : pp = 255.1026611328125 Training perplexity: 264.9092102050781 Validation perplexity:254.84902954101562 Total time : 41.65332388877869 Epoch 3 0 / 1452 : pp = 327.0847473144531 10 / 1452 : pp = 273.9620056152344 20 / 1452 : pp = 270.22943115234375 30 / 1452 : pp = 263.5213317871094 40 / 1452 : pp = 264.0644836425781 50 / 1452 : pp = 258.6029968261719 60 / 1452 : pp = 257.04290771484375 70 / 1452 : pp = 257.59161376953125 80 / 1452 : pp = 256.7600402832031 90 / 1452 : pp = 254.5120391845703 100 / 1452 : pp = 252.44725036621094 110 / 1452 : pp = 250.13954162597656 120 / 1452 : pp = 249.91647338867188 130 / 1452 : pp = 249.50460815429688 140 / 1452 : pp = 247.67440795898438 150 / 1452 : pp = 247.19090270996094 160 / 1452 : pp = 247.8919219970703 170 / 1452 : pp = 247.54322814941406 180 / 1452 : pp = 246.17623901367188 190 / 1452 : pp = 245.78330993652344 200 / 1452 : pp = 246.80552673339844 210 / 1452 : pp = 246.3059844970703 220 / 1452 : pp = 246.19021606445312 230 / 1452 : pp = 246.70140075683594 240 / 1452 : pp = 246.3099822998047 250 / 1452 : pp = 245.1745147705078 260 / 1452 : pp = 244.17384338378906 270 / 1452 : pp = 242.57363891601562 280 / 1452 : pp = 242.8500213623047 290 / 1452 : pp = 243.0492706298828 300 / 1452 : pp = 243.1466522216797 310 / 1452 : pp = 242.89044189453125 320 / 1452 : pp = 243.08045959472656 330 / 1452 : pp = 243.32235717773438 340 / 1452 : pp = 242.34715270996094 350 / 1452 : pp = 242.80972290039062 360 / 1452 : pp = 242.5345458984375 370 / 1452 : pp = 242.0083465576172 380 / 1452 : pp = 241.22708129882812 390 / 1452 : pp = 241.24398803710938 400 / 1452 : pp = 240.63473510742188 410 / 1452 : pp = 240.94094848632812 420 / 1452 : pp = 241.19717407226562 430 / 1452 : pp = 240.8896026611328 440 / 1452 : pp = 240.7772979736328 450 / 1452 : pp = 240.45913696289062 460 / 1452 : pp = 240.06674194335938 470 / 1452 : pp = 239.42198181152344 480 / 1452 : pp = 238.39271545410156 490 / 1452 : pp = 238.0517120361328 500 / 1452 : pp = 237.31752014160156 510 / 1452 : pp = 237.1197967529297 520 / 1452 : pp = 236.64865112304688 530 / 1452 : pp = 236.004638671875 540 / 1452 : pp = 235.192626953125 550 / 1452 : pp = 234.6700439453125 560 / 1452 : pp = 234.1914825439453 570 / 1452 : pp = 233.80899047851562 580 / 1452 : pp = 233.3753662109375 590 / 1452 : pp = 232.8699188232422 600 / 1452 : pp = 232.2629852294922 610 / 1452 : pp = 231.8668212890625 620 / 1452 : pp = 231.478515625 630 / 1452 : pp = 231.0444793701172 640 / 1452 : pp = 231.2737579345703 650 / 1452 : pp = 231.28114318847656 660 / 1452 : pp = 231.4324951171875 670 / 1452 : pp = 231.48513793945312 680 / 1452 : pp = 231.45932006835938 690 / 1452 : pp = 231.17738342285156 700 / 1452 : pp = 231.00570678710938 710 / 1452 : pp = 231.03810119628906 720 / 1452 : pp = 230.96131896972656 730 / 1452 : pp = 230.91110229492188 740 / 1452 : pp = 231.13539123535156 750 / 1452 : pp = 231.04393005371094 760 / 1452 : pp = 231.03489685058594 770 / 1452 : pp = 231.19744873046875 780 / 1452 : pp = 231.26625061035156 790 / 1452 : pp = 231.38714599609375 800 / 1452 : pp = 231.24441528320312 810 / 1452 : pp = 231.16824340820312 820 / 1452 : pp = 231.11831665039062 830 / 1452 : pp = 231.34886169433594 840 / 1452 : pp = 231.221923828125 850 / 1452 : pp = 231.2562255859375 860 / 1452 : pp = 231.26492309570312 870 / 1452 : pp = 231.1961212158203 880 / 1452 : pp = 231.30506896972656 890 / 1452 : pp = 231.24728393554688 900 / 1452 : pp = 231.15744018554688 910 / 1452 : pp = 231.20175170898438 920 / 1452 : pp = 231.25534057617188 930 / 1452 : pp = 231.09461975097656 940 / 1452 : pp = 231.12612915039062 950 / 1452 : pp = 231.0475616455078 960 / 1452 : pp = 230.86056518554688 970 / 1452 : pp = 230.80377197265625 980 / 1452 : pp = 230.4598846435547 990 / 1452 : pp = 230.24559020996094 1000 / 1452 : pp = 229.91030883789062 1010 / 1452 : pp = 229.9349822998047 1020 / 1452 : pp = 230.01470947265625 1030 / 1452 : pp = 229.8909149169922 1040 / 1452 : pp = 229.9403533935547 1050 / 1452 : pp = 229.84815979003906 1060 / 1452 : pp = 229.60377502441406 1070 / 1452 : pp = 229.74647521972656 1080 / 1452 : pp = 229.80410766601562 1090 / 1452 : pp = 229.78733825683594 1100 / 1452 : pp = 229.64549255371094 1110 / 1452 : pp = 229.26255798339844 1120 / 1452 : pp = 229.00262451171875 1130 / 1452 : pp = 228.6716766357422 1140 / 1452 : pp = 228.55067443847656 1150 / 1452 : pp = 228.61563110351562 1160 / 1452 : pp = 228.50958251953125 1170 / 1452 : pp = 228.3498992919922 1180 / 1452 : pp = 228.29786682128906 1190 / 1452 : pp = 228.33204650878906 1200 / 1452 : pp = 228.27369689941406 1210 / 1452 : pp = 228.11831665039062 1220 / 1452 : pp = 228.21775817871094 1230 / 1452 : pp = 228.3170166015625 1240 / 1452 : pp = 228.22134399414062 1250 / 1452 : pp = 228.3769073486328 1260 / 1452 : pp = 228.37527465820312 1270 / 1452 : pp = 228.33694458007812 1280 / 1452 : pp = 228.27108764648438 1290 / 1452 : pp = 228.1731414794922 1300 / 1452 : pp = 228.12200927734375 1310 / 1452 : pp = 228.10275268554688 1320 / 1452 : pp = 227.9289093017578 1330 / 1452 : pp = 227.77723693847656 1340 / 1452 : pp = 227.79623413085938 1350 / 1452 : pp = 227.7408447265625 1360 / 1452 : pp = 227.72586059570312 1370 / 1452 : pp = 227.49728393554688 1380 / 1452 : pp = 227.37940979003906 1390 / 1452 : pp = 227.20166015625 1400 / 1452 : pp = 227.018310546875 1410 / 1452 : pp = 226.95651245117188 1420 / 1452 : pp = 226.8065643310547 1430 / 1452 : pp = 226.7261199951172 1440 / 1452 : pp = 226.7193145751953 1450 / 1452 : pp = 226.61068725585938 0 / 115 : pp = 269.342041015625 10 / 115 : pp = 255.03016662597656 20 / 115 : pp = 253.8992919921875 30 / 115 : pp = 251.04025268554688 40 / 115 : pp = 250.51756286621094 50 / 115 : pp = 245.3595428466797 60 / 115 : pp = 244.4713897705078 70 / 115 : pp = 241.2674560546875 80 / 115 : pp = 238.3473663330078 90 / 115 : pp = 235.56423950195312 100 / 115 : pp = 231.2281036376953 110 / 115 : pp = 229.8423614501953 Training perplexity: 226.5760040283203 Validation perplexity:229.59939575195312 Total time : 42.202677726745605 Epoch 4 0 / 1452 : pp = 282.2423095703125 10 / 1452 : pp = 240.16258239746094 20 / 1452 : pp = 236.12203979492188 30 / 1452 : pp = 230.3953857421875 40 / 1452 : pp = 231.8789825439453 50 / 1452 : pp = 227.26612854003906 60 / 1452 : pp = 226.22061157226562 70 / 1452 : pp = 227.01885986328125 80 / 1452 : pp = 226.2459716796875 90 / 1452 : pp = 224.3211669921875 100 / 1452 : pp = 222.65615844726562 110 / 1452 : pp = 220.70326232910156 120 / 1452 : pp = 220.42288208007812 130 / 1452 : pp = 219.8100128173828 140 / 1452 : pp = 218.04432678222656 150 / 1452 : pp = 217.31639099121094 160 / 1452 : pp = 217.86349487304688 170 / 1452 : pp = 217.46597290039062 180 / 1452 : pp = 216.3349151611328 190 / 1452 : pp = 216.12240600585938 200 / 1452 : pp = 216.97842407226562 210 / 1452 : pp = 216.51014709472656 220 / 1452 : pp = 216.46751403808594 230 / 1452 : pp = 216.80126953125 240 / 1452 : pp = 216.45965576171875 250 / 1452 : pp = 215.5008544921875 260 / 1452 : pp = 214.62210083007812 270 / 1452 : pp = 213.29183959960938 280 / 1452 : pp = 213.5621337890625 290 / 1452 : pp = 213.80657958984375 300 / 1452 : pp = 213.8963165283203 310 / 1452 : pp = 213.60653686523438 320 / 1452 : pp = 213.85877990722656 330 / 1452 : pp = 214.07345581054688 340 / 1452 : pp = 213.25421142578125 350 / 1452 : pp = 213.68019104003906 360 / 1452 : pp = 213.41717529296875 370 / 1452 : pp = 213.04920959472656 380 / 1452 : pp = 212.39019775390625 390 / 1452 : pp = 212.4908905029297 400 / 1452 : pp = 212.01914978027344 410 / 1452 : pp = 212.36903381347656 420 / 1452 : pp = 212.6802520751953 430 / 1452 : pp = 212.42697143554688 440 / 1452 : pp = 212.42990112304688 450 / 1452 : pp = 212.14524841308594 460 / 1452 : pp = 211.7836151123047 470 / 1452 : pp = 211.17282104492188 480 / 1452 : pp = 210.27903747558594 490 / 1452 : pp = 209.95211791992188 500 / 1452 : pp = 209.28302001953125 510 / 1452 : pp = 209.1029815673828 520 / 1452 : pp = 208.73855590820312 530 / 1452 : pp = 208.19700622558594 540 / 1452 : pp = 207.4554443359375 550 / 1452 : pp = 207.0062255859375 560 / 1452 : pp = 206.59739685058594 570 / 1452 : pp = 206.27874755859375 580 / 1452 : pp = 205.87144470214844 590 / 1452 : pp = 205.43545532226562 600 / 1452 : pp = 204.90940856933594 610 / 1452 : pp = 204.5686798095703 620 / 1452 : pp = 204.22862243652344 630 / 1452 : pp = 203.8448028564453 640 / 1452 : pp = 204.06576538085938 650 / 1452 : pp = 204.0941925048828 660 / 1452 : pp = 204.22103881835938 670 / 1452 : pp = 204.289794921875 680 / 1452 : pp = 204.3115234375 690 / 1452 : pp = 204.10284423828125 700 / 1452 : pp = 203.99757385253906 710 / 1452 : pp = 204.04971313476562 720 / 1452 : pp = 204.03152465820312 730 / 1452 : pp = 203.99046325683594 740 / 1452 : pp = 204.19786071777344 750 / 1452 : pp = 204.1642608642578 760 / 1452 : pp = 204.19435119628906 770 / 1452 : pp = 204.37786865234375 780 / 1452 : pp = 204.4965057373047 790 / 1452 : pp = 204.6479034423828 800 / 1452 : pp = 204.56117248535156 810 / 1452 : pp = 204.52284240722656 820 / 1452 : pp = 204.50978088378906 830 / 1452 : pp = 204.7531280517578 840 / 1452 : pp = 204.64468383789062 850 / 1452 : pp = 204.71348571777344 860 / 1452 : pp = 204.7399444580078 870 / 1452 : pp = 204.69406127929688 880 / 1452 : pp = 204.7965850830078 890 / 1452 : pp = 204.7594757080078 900 / 1452 : pp = 204.71446228027344 910 / 1452 : pp = 204.7590789794922 920 / 1452 : pp = 204.85772705078125 930 / 1452 : pp = 204.7428741455078 940 / 1452 : pp = 204.8068389892578 950 / 1452 : pp = 204.75791931152344 960 / 1452 : pp = 204.63815307617188 970 / 1452 : pp = 204.60760498046875 980 / 1452 : pp = 204.34347534179688 990 / 1452 : pp = 204.151611328125 1000 / 1452 : pp = 203.8665771484375 1010 / 1452 : pp = 203.9164581298828 1020 / 1452 : pp = 204.0184783935547 1030 / 1452 : pp = 203.95166015625 1040 / 1452 : pp = 204.03045654296875 1050 / 1452 : pp = 203.95846557617188 1060 / 1452 : pp = 203.77114868164062 1070 / 1452 : pp = 203.93260192871094 1080 / 1452 : pp = 204.00048828125 1090 / 1452 : pp = 204.00233459472656 1100 / 1452 : pp = 203.8960418701172 1110 / 1452 : pp = 203.5987548828125 1120 / 1452 : pp = 203.38392639160156 1130 / 1452 : pp = 203.08872985839844 1140 / 1452 : pp = 203.01272583007812 1150 / 1452 : pp = 203.0865936279297 1160 / 1452 : pp = 203.02308654785156 1170 / 1452 : pp = 202.9125518798828 1180 / 1452 : pp = 202.9097442626953 1190 / 1452 : pp = 202.98252868652344 1200 / 1452 : pp = 202.95387268066406 1210 / 1452 : pp = 202.851318359375 1220 / 1452 : pp = 202.97671508789062 1230 / 1452 : pp = 203.1051025390625 1240 / 1452 : pp = 203.0526123046875 1250 / 1452 : pp = 203.21417236328125 1260 / 1452 : pp = 203.23617553710938 1270 / 1452 : pp = 203.22802734375 1280 / 1452 : pp = 203.20846557617188 1290 / 1452 : pp = 203.15362548828125 1300 / 1452 : pp = 203.14315795898438 1310 / 1452 : pp = 203.15264892578125 1320 / 1452 : pp = 203.02801513671875 1330 / 1452 : pp = 202.92977905273438 1340 / 1452 : pp = 202.95484924316406 1350 / 1452 : pp = 202.9335479736328 1360 / 1452 : pp = 202.955322265625 1370 / 1452 : pp = 202.7740478515625 1380 / 1452 : pp = 202.68569946289062 1390 / 1452 : pp = 202.55816650390625 1400 / 1452 : pp = 202.41651916503906 1410 / 1452 : pp = 202.38494873046875 1420 / 1452 : pp = 202.27593994140625 1430 / 1452 : pp = 202.21826171875 1440 / 1452 : pp = 202.23272705078125 1450 / 1452 : pp = 202.16099548339844 0 / 115 : pp = 253.23211669921875 10 / 115 : pp = 237.62506103515625 20 / 115 : pp = 237.60557556152344 30 / 115 : pp = 234.9273223876953 40 / 115 : pp = 234.30519104003906 50 / 115 : pp = 229.43960571289062 60 / 115 : pp = 228.6050567626953 70 / 115 : pp = 225.2646484375 80 / 115 : pp = 222.55935668945312 90 / 115 : pp = 219.83255004882812 100 / 115 : pp = 215.5491485595703 110 / 115 : pp = 214.07937622070312 Training perplexity: 202.1349639892578 Validation perplexity:213.85256958007812 Total time : 42.10724234580994 Epoch 5 0 / 1452 : pp = 255.92384338378906 10 / 1452 : pp = 219.5322265625 20 / 1452 : pp = 214.36212158203125 30 / 1452 : pp = 209.12620544433594 40 / 1452 : pp = 210.04193115234375 50 / 1452 : pp = 205.77398681640625 60 / 1452 : pp = 204.8201141357422 70 / 1452 : pp = 205.3955841064453 80 / 1452 : pp = 204.8386688232422 90 / 1452 : pp = 203.21194458007812 100 / 1452 : pp = 201.87643432617188 110 / 1452 : pp = 200.10122680664062 120 / 1452 : pp = 199.82012939453125 130 / 1452 : pp = 199.11192321777344 140 / 1452 : pp = 197.51919555664062 150 / 1452 : pp = 197.03567504882812 160 / 1452 : pp = 197.4231414794922 170 / 1452 : pp = 197.09571838378906 180 / 1452 : pp = 196.17665100097656 190 / 1452 : pp = 196.0064697265625 200 / 1452 : pp = 196.7347869873047 210 / 1452 : pp = 196.3063507080078 220 / 1452 : pp = 196.21388244628906 230 / 1452 : pp = 196.5252227783203 240 / 1452 : pp = 196.203125 250 / 1452 : pp = 195.3251953125 260 / 1452 : pp = 194.53335571289062 270 / 1452 : pp = 193.3546142578125 280 / 1452 : pp = 193.59420776367188 290 / 1452 : pp = 193.83297729492188 300 / 1452 : pp = 193.98489379882812 310 / 1452 : pp = 193.68414306640625 320 / 1452 : pp = 193.89065551757812 330 / 1452 : pp = 194.0518798828125 340 / 1452 : pp = 193.32888793945312 350 / 1452 : pp = 193.76219177246094 360 / 1452 : pp = 193.56106567382812 370 / 1452 : pp = 193.28179931640625 380 / 1452 : pp = 192.7037811279297 390 / 1452 : pp = 192.8145294189453 400 / 1452 : pp = 192.43325805664062 410 / 1452 : pp = 192.81527709960938 420 / 1452 : pp = 193.13760375976562 430 / 1452 : pp = 192.9148712158203 440 / 1452 : pp = 192.92526245117188 450 / 1452 : pp = 192.70083618164062 460 / 1452 : pp = 192.36647033691406 470 / 1452 : pp = 191.85394287109375 480 / 1452 : pp = 191.07244873046875 490 / 1452 : pp = 190.75401306152344 500 / 1452 : pp = 190.1843719482422 510 / 1452 : pp = 190.03334045410156 520 / 1452 : pp = 189.72938537597656 530 / 1452 : pp = 189.25889587402344 540 / 1452 : pp = 188.59315490722656 550 / 1452 : pp = 188.19313049316406 560 / 1452 : pp = 187.80621337890625 570 / 1452 : pp = 187.5229034423828 580 / 1452 : pp = 187.1091766357422 590 / 1452 : pp = 186.72592163085938 600 / 1452 : pp = 186.2238006591797 610 / 1452 : pp = 185.89695739746094 620 / 1452 : pp = 185.60989379882812 630 / 1452 : pp = 185.2689208984375 640 / 1452 : pp = 185.47567749023438 650 / 1452 : pp = 185.5127410888672 660 / 1452 : pp = 185.64627075195312 670 / 1452 : pp = 185.71311950683594 680 / 1452 : pp = 185.72569274902344 690 / 1452 : pp = 185.56459045410156 700 / 1452 : pp = 185.48681640625 710 / 1452 : pp = 185.5458221435547 720 / 1452 : pp = 185.5598907470703 730 / 1452 : pp = 185.5335235595703 740 / 1452 : pp = 185.73995971679688 750 / 1452 : pp = 185.744384765625 760 / 1452 : pp = 185.81268310546875 770 / 1452 : pp = 186.00088500976562 780 / 1452 : pp = 186.14443969726562 790 / 1452 : pp = 186.30764770507812 800 / 1452 : pp = 186.2595977783203 810 / 1452 : pp = 186.23028564453125 820 / 1452 : pp = 186.23997497558594 830 / 1452 : pp = 186.49057006835938 840 / 1452 : pp = 186.43331909179688 850 / 1452 : pp = 186.48887634277344 860 / 1452 : pp = 186.51502990722656 870 / 1452 : pp = 186.5167999267578 880 / 1452 : pp = 186.62400817871094 890 / 1452 : pp = 186.6103973388672 900 / 1452 : pp = 186.58111572265625 910 / 1452 : pp = 186.64126586914062 920 / 1452 : pp = 186.7366180419922 930 / 1452 : pp = 186.65719604492188 940 / 1452 : pp = 186.71755981445312 950 / 1452 : pp = 186.6977996826172 960 / 1452 : pp = 186.62774658203125 970 / 1452 : pp = 186.62115478515625 980 / 1452 : pp = 186.3773193359375 990 / 1452 : pp = 186.23109436035156 1000 / 1452 : pp = 185.99227905273438 1010 / 1452 : pp = 186.0488739013672 1020 / 1452 : pp = 186.1744384765625 1030 / 1452 : pp = 186.1162109375 1040 / 1452 : pp = 186.18899536132812 1050 / 1452 : pp = 186.1549072265625 1060 / 1452 : pp = 186.01419067382812 1070 / 1452 : pp = 186.17364501953125 1080 / 1452 : pp = 186.27061462402344 1090 / 1452 : pp = 186.28428649902344 1100 / 1452 : pp = 186.2150115966797 1110 / 1452 : pp = 185.95103454589844 1120 / 1452 : pp = 185.77423095703125 1130 / 1452 : pp = 185.5232696533203 1140 / 1452 : pp = 185.4607391357422 1150 / 1452 : pp = 185.56077575683594 1160 / 1452 : pp = 185.53343200683594 1170 / 1452 : pp = 185.46453857421875 1180 / 1452 : pp = 185.4741668701172 1190 / 1452 : pp = 185.5594482421875 1200 / 1452 : pp = 185.53785705566406 1210 / 1452 : pp = 185.4576416015625 1220 / 1452 : pp = 185.5943145751953 1230 / 1452 : pp = 185.7483673095703 1240 / 1452 : pp = 185.70762634277344 1250 / 1452 : pp = 185.8568115234375 1260 / 1452 : pp = 185.90635681152344 1270 / 1452 : pp = 185.8961639404297 1280 / 1452 : pp = 185.89199829101562 1290 / 1452 : pp = 185.85911560058594 1300 / 1452 : pp = 185.86097717285156 1310 / 1452 : pp = 185.88739013671875 1320 / 1452 : pp = 185.79248046875 1330 / 1452 : pp = 185.69700622558594 1340 / 1452 : pp = 185.7310028076172 1350 / 1452 : pp = 185.72613525390625 1360 / 1452 : pp = 185.76829528808594 1370 / 1452 : pp = 185.6322021484375 1380 / 1452 : pp = 185.56378173828125 1390 / 1452 : pp = 185.4654998779297 1400 / 1452 : pp = 185.35110473632812 1410 / 1452 : pp = 185.33917236328125 1420 / 1452 : pp = 185.2509002685547 1430 / 1452 : pp = 185.20436096191406 1440 / 1452 : pp = 185.2254638671875 1450 / 1452 : pp = 185.16542053222656 0 / 115 : pp = 242.26800537109375 10 / 115 : pp = 226.12258911132812 20 / 115 : pp = 226.4702606201172 30 / 115 : pp = 223.982666015625 40 / 115 : pp = 223.376953125 50 / 115 : pp = 218.65716552734375 60 / 115 : pp = 217.95306396484375 70 / 115 : pp = 214.5392303466797 80 / 115 : pp = 212.07525634765625 90 / 115 : pp = 209.40631103515625 100 / 115 : pp = 205.1455078125 110 / 115 : pp = 203.6289520263672 Training perplexity: 185.14476013183594 Validation perplexity:203.3822784423828 Total time : 42.47052240371704 Epoch 6 0 / 1452 : pp = 233.56707763671875 10 / 1452 : pp = 202.6468505859375 20 / 1452 : pp = 198.2734375 30 / 1452 : pp = 193.47442626953125 40 / 1452 : pp = 195.17147827148438 50 / 1452 : pp = 191.5596923828125 60 / 1452 : pp = 190.4825897216797 70 / 1452 : pp = 191.07681274414062 80 / 1452 : pp = 190.339599609375 90 / 1452 : pp = 188.98277282714844 100 / 1452 : pp = 187.74757385253906 110 / 1452 : pp = 186.10104370117188 120 / 1452 : pp = 185.7500457763672 130 / 1452 : pp = 184.90707397460938 140 / 1452 : pp = 183.340087890625 150 / 1452 : pp = 182.70840454101562 160 / 1452 : pp = 183.1043701171875 170 / 1452 : pp = 182.69776916503906 180 / 1452 : pp = 181.88400268554688 190 / 1452 : pp = 181.8062286376953 200 / 1452 : pp = 182.4969940185547 210 / 1452 : pp = 182.10572814941406 220 / 1452 : pp = 181.9981689453125 230 / 1452 : pp = 182.3802490234375 240 / 1452 : pp = 182.03636169433594 250 / 1452 : pp = 181.23712158203125 260 / 1452 : pp = 180.53726196289062 270 / 1452 : pp = 179.53567504882812 280 / 1452 : pp = 179.70208740234375 290 / 1452 : pp = 179.977783203125 300 / 1452 : pp = 180.16600036621094 310 / 1452 : pp = 179.87294006347656 320 / 1452 : pp = 180.11849975585938 330 / 1452 : pp = 180.31838989257812 340 / 1452 : pp = 179.56759643554688 350 / 1452 : pp = 179.97134399414062 360 / 1452 : pp = 179.80030822753906 370 / 1452 : pp = 179.52085876464844 380 / 1452 : pp = 178.98228454589844 390 / 1452 : pp = 179.0868682861328 400 / 1452 : pp = 178.74569702148438 410 / 1452 : pp = 179.1776580810547 420 / 1452 : pp = 179.5055389404297 430 / 1452 : pp = 179.3883056640625 440 / 1452 : pp = 179.42279052734375 450 / 1452 : pp = 179.2106475830078 460 / 1452 : pp = 178.85311889648438 470 / 1452 : pp = 178.33840942382812 480 / 1452 : pp = 177.60350036621094 490 / 1452 : pp = 177.30335998535156 500 / 1452 : pp = 176.72222900390625 510 / 1452 : pp = 176.6067352294922 520 / 1452 : pp = 176.33998107910156 530 / 1452 : pp = 175.93162536621094 540 / 1452 : pp = 175.30657958984375 550 / 1452 : pp = 174.9462432861328 560 / 1452 : pp = 174.5836639404297 570 / 1452 : pp = 174.31431579589844 580 / 1452 : pp = 173.92300415039062 590 / 1452 : pp = 173.55856323242188 600 / 1452 : pp = 173.08277893066406 610 / 1452 : pp = 172.75930786132812 620 / 1452 : pp = 172.53192138671875 630 / 1452 : pp = 172.20652770996094 640 / 1452 : pp = 172.37454223632812 650 / 1452 : pp = 172.39845275878906 660 / 1452 : pp = 172.52255249023438 670 / 1452 : pp = 172.60935974121094 680 / 1452 : pp = 172.6611328125 690 / 1452 : pp = 172.53118896484375 700 / 1452 : pp = 172.4709014892578 710 / 1452 : pp = 172.5406494140625 720 / 1452 : pp = 172.55447387695312 730 / 1452 : pp = 172.5330047607422 740 / 1452 : pp = 172.7061767578125 750 / 1452 : pp = 172.71054077148438 760 / 1452 : pp = 172.77743530273438 770 / 1452 : pp = 172.95481872558594 780 / 1452 : pp = 173.11265563964844 790 / 1452 : pp = 173.2832794189453 800 / 1452 : pp = 173.2537841796875 810 / 1452 : pp = 173.22164916992188 820 / 1452 : pp = 173.24148559570312 830 / 1452 : pp = 173.48228454589844 840 / 1452 : pp = 173.43753051757812 850 / 1452 : pp = 173.505615234375 860 / 1452 : pp = 173.5214080810547 870 / 1452 : pp = 173.5009002685547 880 / 1452 : pp = 173.6202392578125 890 / 1452 : pp = 173.622802734375 900 / 1452 : pp = 173.5987091064453 910 / 1452 : pp = 173.68316650390625 920 / 1452 : pp = 173.77330017089844 930 / 1452 : pp = 173.72018432617188 940 / 1452 : pp = 173.79351806640625 950 / 1452 : pp = 173.7653350830078 960 / 1452 : pp = 173.7102508544922 970 / 1452 : pp = 173.69766235351562 980 / 1452 : pp = 173.4836883544922 990 / 1452 : pp = 173.3550262451172 1000 / 1452 : pp = 173.14816284179688 1010 / 1452 : pp = 173.20777893066406 1020 / 1452 : pp = 173.3390655517578 1030 / 1452 : pp = 173.2884063720703 1040 / 1452 : pp = 173.38015747070312 1050 / 1452 : pp = 173.35592651367188 1060 / 1452 : pp = 173.2260284423828 1070 / 1452 : pp = 173.39321899414062 1080 / 1452 : pp = 173.4879913330078 1090 / 1452 : pp = 173.5231475830078 1100 / 1452 : pp = 173.47177124023438 1110 / 1452 : pp = 173.24453735351562 1120 / 1452 : pp = 173.09408569335938 1130 / 1452 : pp = 172.86627197265625 1140 / 1452 : pp = 172.8234100341797 1150 / 1452 : pp = 172.92843627929688 1160 / 1452 : pp = 172.90065002441406 1170 / 1452 : pp = 172.8550567626953 1180 / 1452 : pp = 172.8810272216797 1190 / 1452 : pp = 172.97312927246094 1200 / 1452 : pp = 172.9776611328125 1210 / 1452 : pp = 172.89413452148438 1220 / 1452 : pp = 173.0257568359375 1230 / 1452 : pp = 173.1847381591797 1240 / 1452 : pp = 173.1756591796875 1250 / 1452 : pp = 173.32138061523438 1260 / 1452 : pp = 173.37229919433594 1270 / 1452 : pp = 173.36891174316406 1280 / 1452 : pp = 173.36337280273438 1290 / 1452 : pp = 173.3444366455078 1300 / 1452 : pp = 173.36138916015625 1310 / 1452 : pp = 173.4015655517578 1320 / 1452 : pp = 173.31790161132812 1330 / 1452 : pp = 173.24710083007812 1340 / 1452 : pp = 173.27212524414062 1350 / 1452 : pp = 173.27674865722656 1360 / 1452 : pp = 173.32749938964844 1370 / 1452 : pp = 173.20472717285156 1380 / 1452 : pp = 173.14889526367188 1390 / 1452 : pp = 173.0755157470703 1400 / 1452 : pp = 172.9678497314453 1410 / 1452 : pp = 172.9612579345703 1420 / 1452 : pp = 172.8872833251953 1430 / 1452 : pp = 172.84805297851562 1440 / 1452 : pp = 172.87252807617188 1450 / 1452 : pp = 172.82505798339844 0 / 115 : pp = 236.35635375976562 10 / 115 : pp = 219.06166076660156 20 / 115 : pp = 219.7670440673828 30 / 115 : pp = 217.33587646484375 40 / 115 : pp = 216.6626739501953 50 / 115 : pp = 212.04734802246094 60 / 115 : pp = 211.42068481445312 70 / 115 : pp = 207.9592742919922 80 / 115 : pp = 205.6216583251953 90 / 115 : pp = 202.93597412109375 100 / 115 : pp = 198.62583923339844 110 / 115 : pp = 196.97216796875 Training perplexity: 172.80404663085938 Validation perplexity:196.6871337890625 Total time : 41.52522921562195 Epoch 7 0 / 1452 : pp = 219.23231506347656 10 / 1452 : pp = 192.07225036621094 20 / 1452 : pp = 187.48464965820312 30 / 1452 : pp = 182.9149932861328 40 / 1452 : pp = 184.2945098876953 50 / 1452 : pp = 180.78492736816406 60 / 1452 : pp = 179.377197265625 70 / 1452 : pp = 180.0273895263672 80 / 1452 : pp = 179.2517547607422 90 / 1452 : pp = 177.77540588378906 100 / 1452 : pp = 176.6474151611328 110 / 1452 : pp = 174.84066772460938 120 / 1452 : pp = 174.46890258789062 130 / 1452 : pp = 173.64573669433594 140 / 1452 : pp = 172.17483520507812 150 / 1452 : pp = 171.57041931152344 160 / 1452 : pp = 171.92059326171875 170 / 1452 : pp = 171.5497283935547 180 / 1452 : pp = 170.77249145507812 190 / 1452 : pp = 170.72103881835938 200 / 1452 : pp = 171.336181640625 210 / 1452 : pp = 170.98524475097656 220 / 1452 : pp = 170.99771118164062 230 / 1452 : pp = 171.39918518066406 240 / 1452 : pp = 171.09925842285156 250 / 1452 : pp = 170.39962768554688 260 / 1452 : pp = 169.7328643798828 270 / 1452 : pp = 168.72225952148438 280 / 1452 : pp = 168.92552185058594 290 / 1452 : pp = 169.20147705078125 300 / 1452 : pp = 169.40338134765625 310 / 1452 : pp = 169.12057495117188 320 / 1452 : pp = 169.31236267089844 330 / 1452 : pp = 169.49945068359375 340 / 1452 : pp = 168.8396759033203 350 / 1452 : pp = 169.25917053222656 360 / 1452 : pp = 169.09388732910156 370 / 1452 : pp = 168.84323120117188 380 / 1452 : pp = 168.3832550048828 390 / 1452 : pp = 168.48275756835938 400 / 1452 : pp = 168.19972229003906 410 / 1452 : pp = 168.5838623046875 420 / 1452 : pp = 168.91119384765625 430 / 1452 : pp = 168.80836486816406 440 / 1452 : pp = 168.90264892578125 450 / 1452 : pp = 168.68589782714844 460 / 1452 : pp = 168.3704071044922 470 / 1452 : pp = 167.90394592285156 480 / 1452 : pp = 167.23373413085938 490 / 1452 : pp = 166.9560546875 500 / 1452 : pp = 166.43161010742188 510 / 1452 : pp = 166.320068359375 520 / 1452 : pp = 166.05902099609375 530 / 1452 : pp = 165.71714782714844 540 / 1452 : pp = 165.10398864746094 550 / 1452 : pp = 164.80430603027344 560 / 1452 : pp = 164.4687042236328 570 / 1452 : pp = 164.2272491455078 580 / 1452 : pp = 163.84312438964844 590 / 1452 : pp = 163.46035766601562 600 / 1452 : pp = 163.01559448242188 610 / 1452 : pp = 162.74134826660156 620 / 1452 : pp = 162.50267028808594 630 / 1452 : pp = 162.2018280029297 640 / 1452 : pp = 162.37130737304688 650 / 1452 : pp = 162.3895721435547 660 / 1452 : pp = 162.51351928710938 670 / 1452 : pp = 162.57684326171875 680 / 1452 : pp = 162.6346893310547 690 / 1452 : pp = 162.5135955810547 700 / 1452 : pp = 162.47052001953125 710 / 1452 : pp = 162.539794921875 720 / 1452 : pp = 162.55381774902344 730 / 1452 : pp = 162.5297088623047 740 / 1452 : pp = 162.71652221679688 750 / 1452 : pp = 162.740966796875 760 / 1452 : pp = 162.79754638671875 770 / 1452 : pp = 162.9949951171875 780 / 1452 : pp = 163.17868041992188 790 / 1452 : pp = 163.33055114746094 800 / 1452 : pp = 163.31591796875 810 / 1452 : pp = 163.2859344482422 820 / 1452 : pp = 163.2958984375 830 / 1452 : pp = 163.528564453125 840 / 1452 : pp = 163.47610473632812 850 / 1452 : pp = 163.5260772705078 860 / 1452 : pp = 163.55352783203125 870 / 1452 : pp = 163.55718994140625 880 / 1452 : pp = 163.67523193359375 890 / 1452 : pp = 163.6920166015625 900 / 1452 : pp = 163.67710876464844 910 / 1452 : pp = 163.7476806640625 920 / 1452 : pp = 163.84803771972656 930 / 1452 : pp = 163.8114013671875 940 / 1452 : pp = 163.86663818359375 950 / 1452 : pp = 163.83531188964844 960 / 1452 : pp = 163.79945373535156 970 / 1452 : pp = 163.80320739746094 980 / 1452 : pp = 163.5953369140625 990 / 1452 : pp = 163.48382568359375 1000 / 1452 : pp = 163.2642822265625 1010 / 1452 : pp = 163.32113647460938 1020 / 1452 : pp = 163.44204711914062 1030 / 1452 : pp = 163.40206909179688 1040 / 1452 : pp = 163.4915313720703 1050 / 1452 : pp = 163.47096252441406 1060 / 1452 : pp = 163.3601531982422 1070 / 1452 : pp = 163.5138397216797 1080 / 1452 : pp = 163.6189727783203 1090 / 1452 : pp = 163.6471405029297 1100 / 1452 : pp = 163.60406494140625 1110 / 1452 : pp = 163.40736389160156 1120 / 1452 : pp = 163.26841735839844 1130 / 1452 : pp = 163.0680694580078 1140 / 1452 : pp = 163.04591369628906 1150 / 1452 : pp = 163.15478515625 1160 / 1452 : pp = 163.1380615234375 1170 / 1452 : pp = 163.09303283691406 1180 / 1452 : pp = 163.14149475097656 1190 / 1452 : pp = 163.2374267578125 1200 / 1452 : pp = 163.2394561767578 1210 / 1452 : pp = 163.17835998535156 1220 / 1452 : pp = 163.32347106933594 1230 / 1452 : pp = 163.4639434814453 1240 / 1452 : pp = 163.4611358642578 1250 / 1452 : pp = 163.60687255859375 1260 / 1452 : pp = 163.67227172851562 1270 / 1452 : pp = 163.67515563964844 1280 / 1452 : pp = 163.6881103515625 1290 / 1452 : pp = 163.66648864746094 1300 / 1452 : pp = 163.69287109375 1310 / 1452 : pp = 163.7276153564453 1320 / 1452 : pp = 163.6551055908203 1330 / 1452 : pp = 163.58901977539062 1340 / 1452 : pp = 163.6205291748047 1350 / 1452 : pp = 163.63824462890625 1360 / 1452 : pp = 163.69334411621094 1370 / 1452 : pp = 163.5885467529297 1380 / 1452 : pp = 163.54049682617188 1390 / 1452 : pp = 163.4760284423828 1400 / 1452 : pp = 163.38897705078125 1410 / 1452 : pp = 163.3974609375 1420 / 1452 : pp = 163.35009765625 1430 / 1452 : pp = 163.32191467285156 1440 / 1452 : pp = 163.35220336914062 1450 / 1452 : pp = 163.3201904296875 0 / 115 : pp = 232.2108154296875 10 / 115 : pp = 214.35496520996094 20 / 115 : pp = 215.20510864257812 30 / 115 : pp = 212.82754516601562 40 / 115 : pp = 212.0598907470703 50 / 115 : pp = 207.5095672607422 60 / 115 : pp = 206.86976623535156 70 / 115 : pp = 203.36016845703125 80 / 115 : pp = 201.11538696289062 90 / 115 : pp = 198.52120971679688 100 / 115 : pp = 194.1772003173828 110 / 115 : pp = 192.41224670410156 Training perplexity: 163.29916381835938 Validation perplexity:192.09552001953125 Total time : 41.78096055984497 Epoch 8 0 / 1452 : pp = 201.77548217773438 10 / 1452 : pp = 180.4141082763672 20 / 1452 : pp = 176.41432189941406 30 / 1452 : pp = 172.7764434814453 40 / 1452 : pp = 174.69166564941406 50 / 1452 : pp = 171.2933807373047 60 / 1452 : pp = 170.08010864257812 70 / 1452 : pp = 170.6719512939453 80 / 1452 : pp = 170.07589721679688 90 / 1452 : pp = 168.7478485107422 100 / 1452 : pp = 167.57081604003906 110 / 1452 : pp = 166.06971740722656 120 / 1452 : pp = 165.73374938964844 130 / 1452 : pp = 164.80674743652344 140 / 1452 : pp = 163.32821655273438 150 / 1452 : pp = 162.6752471923828 160 / 1452 : pp = 163.02049255371094 170 / 1452 : pp = 162.64120483398438 180 / 1452 : pp = 161.95529174804688 190 / 1452 : pp = 161.91954040527344 200 / 1452 : pp = 162.5446014404297 210 / 1452 : pp = 162.2645721435547 220 / 1452 : pp = 162.3128662109375 230 / 1452 : pp = 162.65872192382812 240 / 1452 : pp = 162.40948486328125 250 / 1452 : pp = 161.75787353515625 260 / 1452 : pp = 161.15213012695312 270 / 1452 : pp = 160.22256469726562 280 / 1452 : pp = 160.3651123046875 290 / 1452 : pp = 160.63780212402344 300 / 1452 : pp = 160.80026245117188 310 / 1452 : pp = 160.54383850097656 320 / 1452 : pp = 160.7539520263672 330 / 1452 : pp = 160.94317626953125 340 / 1452 : pp = 160.3373565673828 350 / 1452 : pp = 160.71763610839844 360 / 1452 : pp = 160.60960388183594 370 / 1452 : pp = 160.37527465820312 380 / 1452 : pp = 159.92990112304688 390 / 1452 : pp = 160.0165557861328 400 / 1452 : pp = 159.75697326660156 410 / 1452 : pp = 160.15274047851562 420 / 1452 : pp = 160.48390197753906 430 / 1452 : pp = 160.4031982421875 440 / 1452 : pp = 160.4693603515625 450 / 1452 : pp = 160.28016662597656 460 / 1452 : pp = 159.94004821777344 470 / 1452 : pp = 159.48257446289062 480 / 1452 : pp = 158.87998962402344 490 / 1452 : pp = 158.59765625 500 / 1452 : pp = 158.10865783691406 510 / 1452 : pp = 157.96795654296875 520 / 1452 : pp = 157.7591552734375 530 / 1452 : pp = 157.42648315429688 540 / 1452 : pp = 156.85348510742188 550 / 1452 : pp = 156.5618438720703 560 / 1452 : pp = 156.24905395507812 570 / 1452 : pp = 155.9994354248047 580 / 1452 : pp = 155.612060546875 590 / 1452 : pp = 155.25830078125 600 / 1452 : pp = 154.8464813232422 610 / 1452 : pp = 154.5833282470703 620 / 1452 : pp = 154.38040161132812 630 / 1452 : pp = 154.0767364501953 640 / 1452 : pp = 154.2534637451172 650 / 1452 : pp = 154.25875854492188 660 / 1452 : pp = 154.35874938964844 670 / 1452 : pp = 154.4289093017578 680 / 1452 : pp = 154.51412963867188 690 / 1452 : pp = 154.41676330566406 700 / 1452 : pp = 154.37892150878906 710 / 1452 : pp = 154.4234619140625 720 / 1452 : pp = 154.4586639404297 730 / 1452 : pp = 154.4351806640625 740 / 1452 : pp = 154.6002197265625 750 / 1452 : pp = 154.65684509277344 760 / 1452 : pp = 154.73318481445312 770 / 1452 : pp = 154.92935180664062 780 / 1452 : pp = 155.1021728515625 790 / 1452 : pp = 155.24757385253906 800 / 1452 : pp = 155.223876953125 810 / 1452 : pp = 155.2095184326172 820 / 1452 : pp = 155.24009704589844 830 / 1452 : pp = 155.4519500732422 840 / 1452 : pp = 155.3947296142578 850 / 1452 : pp = 155.45306396484375 860 / 1452 : pp = 155.4661102294922 870 / 1452 : pp = 155.45765686035156 880 / 1452 : pp = 155.58758544921875 890 / 1452 : pp = 155.59373474121094 900 / 1452 : pp = 155.59254455566406 910 / 1452 : pp = 155.66854858398438 920 / 1452 : pp = 155.75942993164062 930 / 1452 : pp = 155.73350524902344 940 / 1452 : pp = 155.80740356445312 950 / 1452 : pp = 155.7733917236328 960 / 1452 : pp = 155.73565673828125 970 / 1452 : pp = 155.74404907226562 980 / 1452 : pp = 155.55902099609375 990 / 1452 : pp = 155.45675659179688 1000 / 1452 : pp = 155.2649688720703 1010 / 1452 : pp = 155.31332397460938 1020 / 1452 : pp = 155.44979858398438 1030 / 1452 : pp = 155.4137725830078 1040 / 1452 : pp = 155.49012756347656 1050 / 1452 : pp = 155.46054077148438 1060 / 1452 : pp = 155.3616943359375 1070 / 1452 : pp = 155.5286865234375 1080 / 1452 : pp = 155.63743591308594 1090 / 1452 : pp = 155.6842803955078 1100 / 1452 : pp = 155.65599060058594 1110 / 1452 : pp = 155.4827880859375 1120 / 1452 : pp = 155.35450744628906 1130 / 1452 : pp = 155.1777801513672 1140 / 1452 : pp = 155.15994262695312 1150 / 1452 : pp = 155.26193237304688 1160 / 1452 : pp = 155.26214599609375 1170 / 1452 : pp = 155.23231506347656 1180 / 1452 : pp = 155.29266357421875 1190 / 1452 : pp = 155.37680053710938 1200 / 1452 : pp = 155.3736114501953 1210 / 1452 : pp = 155.3380584716797 1220 / 1452 : pp = 155.474853515625 1230 / 1452 : pp = 155.62986755371094 1240 / 1452 : pp = 155.62831115722656 1250 / 1452 : pp = 155.77101135253906 1260 / 1452 : pp = 155.83445739746094 1270 / 1452 : pp = 155.845458984375 1280 / 1452 : pp = 155.8556365966797 1290 / 1452 : pp = 155.8556365966797 1300 / 1452 : pp = 155.8843994140625 1310 / 1452 : pp = 155.92417907714844 1320 / 1452 : pp = 155.8560791015625 1330 / 1452 : pp = 155.80636596679688 1340 / 1452 : pp = 155.84344482421875 1350 / 1452 : pp = 155.8706512451172 1360 / 1452 : pp = 155.9273681640625 1370 / 1452 : pp = 155.83140563964844 1380 / 1452 : pp = 155.7911376953125 1390 / 1452 : pp = 155.7401885986328 1400 / 1452 : pp = 155.6622314453125 1410 / 1452 : pp = 155.68531799316406 1420 / 1452 : pp = 155.64041137695312 1430 / 1452 : pp = 155.62216186523438 1440 / 1452 : pp = 155.6437530517578 1450 / 1452 : pp = 155.62757873535156 0 / 115 : pp = 228.70111083984375 10 / 115 : pp = 211.03330993652344 20 / 115 : pp = 212.24957275390625 30 / 115 : pp = 209.8839569091797 40 / 115 : pp = 209.11045837402344 50 / 115 : pp = 204.66351318359375 60 / 115 : pp = 204.03366088867188 70 / 115 : pp = 200.46681213378906 80 / 115 : pp = 198.24404907226562 90 / 115 : pp = 195.63223266601562 100 / 115 : pp = 191.18345642089844 110 / 115 : pp = 189.31134033203125 Training perplexity: 155.61154174804688 Validation perplexity:188.94537353515625 Total time : 42.13483738899231 Epoch 9 0 / 1452 : pp = 197.80628967285156 10 / 1452 : pp = 172.6316680908203 20 / 1452 : pp = 168.6739959716797 30 / 1452 : pp = 164.4781036376953 40 / 1452 : pp = 166.1627960205078 50 / 1452 : pp = 163.05197143554688 60 / 1452 : pp = 161.87924194335938 70 / 1452 : pp = 162.5297088623047 80 / 1452 : pp = 161.7450714111328 90 / 1452 : pp = 160.6148223876953 100 / 1452 : pp = 159.73289489746094 110 / 1452 : pp = 158.4092254638672 120 / 1452 : pp = 158.04653930664062 130 / 1452 : pp = 157.13563537597656 140 / 1452 : pp = 155.71798706054688 150 / 1452 : pp = 155.19161987304688 160 / 1452 : pp = 155.42718505859375 170 / 1452 : pp = 155.0531463623047 180 / 1452 : pp = 154.46897888183594 190 / 1452 : pp = 154.4127197265625 200 / 1452 : pp = 154.97154235839844 210 / 1452 : pp = 154.70169067382812 220 / 1452 : pp = 154.72816467285156 230 / 1452 : pp = 155.03799438476562 240 / 1452 : pp = 154.85601806640625 250 / 1452 : pp = 154.28016662597656 260 / 1452 : pp = 153.7699432373047 270 / 1452 : pp = 152.90948486328125 280 / 1452 : pp = 153.0459747314453 290 / 1452 : pp = 153.298095703125 300 / 1452 : pp = 153.45716857910156 310 / 1452 : pp = 153.22195434570312 320 / 1452 : pp = 153.41664123535156 330 / 1452 : pp = 153.66542053222656 340 / 1452 : pp = 153.06378173828125 350 / 1452 : pp = 153.43923950195312 360 / 1452 : pp = 153.31381225585938 370 / 1452 : pp = 153.13473510742188 380 / 1452 : pp = 152.75267028808594 390 / 1452 : pp = 152.85504150390625 400 / 1452 : pp = 152.62342834472656 410 / 1452 : pp = 153.03152465820312 420 / 1452 : pp = 153.39161682128906 430 / 1452 : pp = 153.30364990234375 440 / 1452 : pp = 153.37896728515625 450 / 1452 : pp = 153.18988037109375 460 / 1452 : pp = 152.88478088378906 470 / 1452 : pp = 152.4380340576172 480 / 1452 : pp = 151.86618041992188 490 / 1452 : pp = 151.5962371826172 500 / 1452 : pp = 151.11614990234375 510 / 1452 : pp = 150.99830627441406 520 / 1452 : pp = 150.8135986328125 530 / 1452 : pp = 150.500732421875 540 / 1452 : pp = 149.9623260498047 550 / 1452 : pp = 149.68028259277344 560 / 1452 : pp = 149.3885040283203 570 / 1452 : pp = 149.140380859375 580 / 1452 : pp = 148.76876831054688 590 / 1452 : pp = 148.43368530273438 600 / 1452 : pp = 148.02598571777344 610 / 1452 : pp = 147.7869110107422 620 / 1452 : pp = 147.59796142578125 630 / 1452 : pp = 147.30068969726562 640 / 1452 : pp = 147.45240783691406 650 / 1452 : pp = 147.4651336669922 660 / 1452 : pp = 147.5808563232422 670 / 1452 : pp = 147.65582275390625 680 / 1452 : pp = 147.7360382080078 690 / 1452 : pp = 147.63075256347656 700 / 1452 : pp = 147.6066131591797 710 / 1452 : pp = 147.7024383544922 720 / 1452 : pp = 147.7445526123047 730 / 1452 : pp = 147.72279357910156 740 / 1452 : pp = 147.87107849121094 750 / 1452 : pp = 147.91436767578125 760 / 1452 : pp = 147.9857635498047 770 / 1452 : pp = 148.18206787109375 780 / 1452 : pp = 148.3845672607422 790 / 1452 : pp = 148.5517120361328 800 / 1452 : pp = 148.54002380371094 810 / 1452 : pp = 148.51119995117188 820 / 1452 : pp = 148.5664520263672 830 / 1452 : pp = 148.7821044921875 840 / 1452 : pp = 148.72486877441406 850 / 1452 : pp = 148.77452087402344 860 / 1452 : pp = 148.80076599121094 870 / 1452 : pp = 148.79701232910156 880 / 1452 : pp = 148.9181671142578 890 / 1452 : pp = 148.94537353515625 900 / 1452 : pp = 148.9435272216797 910 / 1452 : pp = 149.02102661132812 920 / 1452 : pp = 149.1085968017578 930 / 1452 : pp = 149.06893920898438 940 / 1452 : pp = 149.1317138671875 950 / 1452 : pp = 149.1232452392578 960 / 1452 : pp = 149.10354614257812 970 / 1452 : pp = 149.11656188964844 980 / 1452 : pp = 148.94259643554688 990 / 1452 : pp = 148.8236846923828 1000 / 1452 : pp = 148.633056640625 1010 / 1452 : pp = 148.6830291748047 1020 / 1452 : pp = 148.8126220703125 1030 / 1452 : pp = 148.78089904785156 1040 / 1452 : pp = 148.8600311279297 1050 / 1452 : pp = 148.8486785888672 1060 / 1452 : pp = 148.7664337158203 1070 / 1452 : pp = 148.9337921142578 1080 / 1452 : pp = 149.04441833496094 1090 / 1452 : pp = 149.07284545898438 1100 / 1452 : pp = 149.03318786621094 1110 / 1452 : pp = 148.86428833007812 1120 / 1452 : pp = 148.7332305908203 1130 / 1452 : pp = 148.5670166015625 1140 / 1452 : pp = 148.54661560058594 1150 / 1452 : pp = 148.64219665527344 1160 / 1452 : pp = 148.6490020751953 1170 / 1452 : pp = 148.62420654296875 1180 / 1452 : pp = 148.67665100097656 1190 / 1452 : pp = 148.7633056640625 1200 / 1452 : pp = 148.7782745361328 1210 / 1452 : pp = 148.72500610351562 1220 / 1452 : pp = 148.87493896484375 1230 / 1452 : pp = 149.039794921875 1240 / 1452 : pp = 149.04000854492188 1250 / 1452 : pp = 149.17054748535156 1260 / 1452 : pp = 149.23863220214844 1270 / 1452 : pp = 149.2436065673828 1280 / 1452 : pp = 149.25086975097656 1290 / 1452 : pp = 149.24147033691406 1300 / 1452 : pp = 149.27413940429688 1310 / 1452 : pp = 149.32077026367188 1320 / 1452 : pp = 149.27301025390625 1330 / 1452 : pp = 149.23080444335938 1340 / 1452 : pp = 149.25791931152344 1350 / 1452 : pp = 149.2841033935547 1360 / 1452 : pp = 149.337158203125 1370 / 1452 : pp = 149.2467498779297 1380 / 1452 : pp = 149.21351623535156 1390 / 1452 : pp = 149.15403747558594 1400 / 1452 : pp = 149.0877685546875 1410 / 1452 : pp = 149.110595703125 1420 / 1452 : pp = 149.07241821289062 1430 / 1452 : pp = 149.05166625976562 1440 / 1452 : pp = 149.0776824951172 1450 / 1452 : pp = 149.06771850585938 0 / 115 : pp = 227.0559844970703 10 / 115 : pp = 208.7002410888672 20 / 115 : pp = 210.38775634765625 30 / 115 : pp = 207.9513397216797 40 / 115 : pp = 207.12994384765625 50 / 115 : pp = 202.70811462402344 60 / 115 : pp = 202.05787658691406 70 / 115 : pp = 198.3761444091797 80 / 115 : pp = 196.17637634277344 90 / 115 : pp = 193.5880126953125 100 / 115 : pp = 189.0758819580078 110 / 115 : pp = 187.07528686523438 Training perplexity: 149.0502471923828 Validation perplexity:186.6911163330078 Total time : 47.274805545806885 Epoch 10 0 / 1452 : pp = 181.8408203125 10 / 1452 : pp = 164.99664306640625 20 / 1452 : pp = 161.8847198486328 30 / 1452 : pp = 158.30064392089844 40 / 1452 : pp = 160.13914489746094 50 / 1452 : pp = 157.58743286132812 60 / 1452 : pp = 156.11871337890625 70 / 1452 : pp = 156.82948303222656 80 / 1452 : pp = 156.2889862060547 90 / 1452 : pp = 155.04833984375 100 / 1452 : pp = 154.09327697753906 110 / 1452 : pp = 152.5070343017578 120 / 1452 : pp = 152.20750427246094 130 / 1452 : pp = 151.3399200439453 140 / 1452 : pp = 149.90740966796875 150 / 1452 : pp = 149.345703125 160 / 1452 : pp = 149.59814453125 170 / 1452 : pp = 149.26539611816406 180 / 1452 : pp = 148.624267578125 190 / 1452 : pp = 148.58819580078125 200 / 1452 : pp = 149.09552001953125 210 / 1452 : pp = 148.8439178466797 220 / 1452 : pp = 148.86605834960938 230 / 1452 : pp = 149.1971435546875 240 / 1452 : pp = 148.96533203125 250 / 1452 : pp = 148.4253387451172 260 / 1452 : pp = 147.9200897216797 270 / 1452 : pp = 147.08816528320312 280 / 1452 : pp = 147.24366760253906 290 / 1452 : pp = 147.52182006835938 300 / 1452 : pp = 147.72222900390625 310 / 1452 : pp = 147.50486755371094 320 / 1452 : pp = 147.73892211914062 330 / 1452 : pp = 147.9404754638672 340 / 1452 : pp = 147.37803649902344 350 / 1452 : pp = 147.6969451904297 360 / 1452 : pp = 147.5704345703125 370 / 1452 : pp = 147.38674926757812 380 / 1452 : pp = 147.03970336914062 390 / 1452 : pp = 147.14231872558594 400 / 1452 : pp = 146.91656494140625 410 / 1452 : pp = 147.34059143066406 420 / 1452 : pp = 147.68496704101562 430 / 1452 : pp = 147.61195373535156 440 / 1452 : pp = 147.68405151367188 450 / 1452 : pp = 147.4711151123047 460 / 1452 : pp = 147.1927032470703 470 / 1452 : pp = 146.72970581054688 480 / 1452 : pp = 146.17173767089844 490 / 1452 : pp = 145.9028778076172 500 / 1452 : pp = 145.42721557617188 510 / 1452 : pp = 145.3111114501953 520 / 1452 : pp = 145.11460876464844 530 / 1452 : pp = 144.81488037109375 540 / 1452 : pp = 144.263916015625 550 / 1452 : pp = 143.997802734375 560 / 1452 : pp = 143.71766662597656 570 / 1452 : pp = 143.47451782226562 580 / 1452 : pp = 143.08474731445312 590 / 1452 : pp = 142.77920532226562 600 / 1452 : pp = 142.39573669433594 610 / 1452 : pp = 142.14906311035156 620 / 1452 : pp = 141.9574432373047 630 / 1452 : pp = 141.67369079589844 640 / 1452 : pp = 141.81556701660156 650 / 1452 : pp = 141.81759643554688 660 / 1452 : pp = 141.9339599609375 670 / 1452 : pp = 142.01248168945312 680 / 1452 : pp = 142.08773803710938 690 / 1452 : pp = 142.00328063964844 700 / 1452 : pp = 141.98086547851562 710 / 1452 : pp = 142.0632781982422 720 / 1452 : pp = 142.10372924804688 730 / 1452 : pp = 142.08055114746094 740 / 1452 : pp = 142.23619079589844 750 / 1452 : pp = 142.2660369873047 760 / 1452 : pp = 142.34678649902344 770 / 1452 : pp = 142.5257568359375 780 / 1452 : pp = 142.70025634765625 790 / 1452 : pp = 142.8614044189453 800 / 1452 : pp = 142.84573364257812 810 / 1452 : pp = 142.8250274658203 820 / 1452 : pp = 142.8540496826172 830 / 1452 : pp = 143.06053161621094 840 / 1452 : pp = 143.0423126220703 850 / 1452 : pp = 143.09634399414062 860 / 1452 : pp = 143.10487365722656 870 / 1452 : pp = 143.0884246826172 880 / 1452 : pp = 143.19387817382812 890 / 1452 : pp = 143.236083984375 900 / 1452 : pp = 143.23390197753906 910 / 1452 : pp = 143.29537963867188 920 / 1452 : pp = 143.3722686767578 930 / 1452 : pp = 143.33795166015625 940 / 1452 : pp = 143.40618896484375 950 / 1452 : pp = 143.3929901123047 960 / 1452 : pp = 143.3693389892578 970 / 1452 : pp = 143.39736938476562 980 / 1452 : pp = 143.2371063232422 990 / 1452 : pp = 143.13893127441406 1000 / 1452 : pp = 142.9658660888672 1010 / 1452 : pp = 143.01544189453125 1020 / 1452 : pp = 143.152587890625 1030 / 1452 : pp = 143.11334228515625 1040 / 1452 : pp = 143.19020080566406 1050 / 1452 : pp = 143.18234252929688 1060 / 1452 : pp = 143.092041015625 1070 / 1452 : pp = 143.24449157714844 1080 / 1452 : pp = 143.34828186035156 1090 / 1452 : pp = 143.38739013671875 1100 / 1452 : pp = 143.37432861328125 1110 / 1452 : pp = 143.20596313476562 1120 / 1452 : pp = 143.07969665527344 1130 / 1452 : pp = 142.92041015625 1140 / 1452 : pp = 142.90902709960938 1150 / 1452 : pp = 143.00732421875 1160 / 1452 : pp = 143.01182556152344 1170 / 1452 : pp = 142.9925994873047 1180 / 1452 : pp = 143.06080627441406 1190 / 1452 : pp = 143.14337158203125 1200 / 1452 : pp = 143.16644287109375 1210 / 1452 : pp = 143.1259002685547 1220 / 1452 : pp = 143.2671661376953 1230 / 1452 : pp = 143.4210968017578 1240 / 1452 : pp = 143.4327850341797 1250 / 1452 : pp = 143.5699920654297 1260 / 1452 : pp = 143.63771057128906 1270 / 1452 : pp = 143.65798950195312 1280 / 1452 : pp = 143.68251037597656 1290 / 1452 : pp = 143.68045043945312 1300 / 1452 : pp = 143.72293090820312 1310 / 1452 : pp = 143.77015686035156 1320 / 1452 : pp = 143.71910095214844 1330 / 1452 : pp = 143.68792724609375 1340 / 1452 : pp = 143.7241668701172 1350 / 1452 : pp = 143.7570037841797 1360 / 1452 : pp = 143.81829833984375 1370 / 1452 : pp = 143.7487030029297 1380 / 1452 : pp = 143.7196502685547 1390 / 1452 : pp = 143.67359924316406 1400 / 1452 : pp = 143.60592651367188 1410 / 1452 : pp = 143.62620544433594 1420 / 1452 : pp = 143.5905303955078 1430 / 1452 : pp = 143.55799865722656 1440 / 1452 : pp = 143.5891571044922 1450 / 1452 : pp = 143.5869598388672 0 / 115 : pp = 226.9864959716797 10 / 115 : pp = 207.8067169189453 20 / 115 : pp = 209.68667602539062 30 / 115 : pp = 207.1610565185547 40 / 115 : pp = 206.3247833251953 50 / 115 : pp = 201.77403259277344 60 / 115 : pp = 201.07098388671875 70 / 115 : pp = 197.33335876464844 80 / 115 : pp = 195.12513732910156 90 / 115 : pp = 192.5349578857422 100 / 115 : pp = 187.90072631835938 110 / 115 : pp = 185.81240844726562 Training perplexity: 143.57354736328125 Validation perplexity:185.40573120117188 Total time : 46.14846849441528 Epoch 11 0 / 1452 : pp = 181.93162536621094 10 / 1452 : pp = 159.94607543945312 20 / 1452 : pp = 156.83673095703125 30 / 1452 : pp = 153.75843811035156 40 / 1452 : pp = 155.18362426757812 50 / 1452 : pp = 152.39529418945312 60 / 1452 : pp = 151.18772888183594 70 / 1452 : pp = 151.9004364013672 80 / 1452 : pp = 151.30239868164062 90 / 1452 : pp = 150.1591033935547 100 / 1452 : pp = 149.18618774414062 110 / 1452 : pp = 147.72653198242188 120 / 1452 : pp = 147.4357452392578 130 / 1452 : pp = 146.41372680664062 140 / 1452 : pp = 145.0057373046875 150 / 1452 : pp = 144.39447021484375 160 / 1452 : pp = 144.5330047607422 170 / 1452 : pp = 144.23593139648438 180 / 1452 : pp = 143.63990783691406 190 / 1452 : pp = 143.63812255859375 200 / 1452 : pp = 144.1143798828125 210 / 1452 : pp = 143.88278198242188 220 / 1452 : pp = 143.92518615722656 230 / 1452 : pp = 144.24032592773438 240 / 1452 : pp = 143.94110107421875 250 / 1452 : pp = 143.3688507080078 260 / 1452 : pp = 142.8829345703125 270 / 1452 : pp = 142.11952209472656 280 / 1452 : pp = 142.19415283203125 290 / 1452 : pp = 142.51889038085938 300 / 1452 : pp = 142.70494079589844 310 / 1452 : pp = 142.51426696777344 320 / 1452 : pp = 142.70106506347656 330 / 1452 : pp = 142.88014221191406 340 / 1452 : pp = 142.3287353515625 350 / 1452 : pp = 142.6169891357422 360 / 1452 : pp = 142.51971435546875 370 / 1452 : pp = 142.33566284179688 380 / 1452 : pp = 142.04161071777344 390 / 1452 : pp = 142.13551330566406 400 / 1452 : pp = 141.9499969482422 410 / 1452 : pp = 142.3361358642578 420 / 1452 : pp = 142.64065551757812 430 / 1452 : pp = 142.5511016845703 440 / 1452 : pp = 142.6728973388672 450 / 1452 : pp = 142.47030639648438 460 / 1452 : pp = 142.1704864501953 470 / 1452 : pp = 141.73390197753906 480 / 1452 : pp = 141.23020935058594 490 / 1452 : pp = 140.9759521484375 500 / 1452 : pp = 140.51609802246094 510 / 1452 : pp = 140.40545654296875 520 / 1452 : pp = 140.1936492919922 530 / 1452 : pp = 139.8929443359375 540 / 1452 : pp = 139.3696746826172 550 / 1452 : pp = 139.13217163085938 560 / 1452 : pp = 138.85247802734375 570 / 1452 : pp = 138.6092987060547 580 / 1452 : pp = 138.2471160888672 590 / 1452 : pp = 137.9485626220703 600 / 1452 : pp = 137.57379150390625 610 / 1452 : pp = 137.31576538085938 620 / 1452 : pp = 137.14230346679688 630 / 1452 : pp = 136.87405395507812 640 / 1452 : pp = 137.02928161621094 650 / 1452 : pp = 137.0481719970703 660 / 1452 : pp = 137.1595001220703 670 / 1452 : pp = 137.21124267578125 680 / 1452 : pp = 137.2671356201172 690 / 1452 : pp = 137.19410705566406 700 / 1452 : pp = 137.1850128173828 710 / 1452 : pp = 137.26058959960938 720 / 1452 : pp = 137.30726623535156 730 / 1452 : pp = 137.28048706054688 740 / 1452 : pp = 137.4352569580078 750 / 1452 : pp = 137.4680938720703 760 / 1452 : pp = 137.5524139404297 770 / 1452 : pp = 137.73829650878906 780 / 1452 : pp = 137.90882873535156 790 / 1452 : pp = 138.05865478515625 800 / 1452 : pp = 138.0673370361328 810 / 1452 : pp = 138.03909301757812 820 / 1452 : pp = 138.084716796875 830 / 1452 : pp = 138.27989196777344 840 / 1452 : pp = 138.23545837402344 850 / 1452 : pp = 138.30343627929688 860 / 1452 : pp = 138.3339080810547 870 / 1452 : pp = 138.32835388183594 880 / 1452 : pp = 138.4450225830078 890 / 1452 : pp = 138.47157287597656 900 / 1452 : pp = 138.46304321289062 910 / 1452 : pp = 138.55618286132812 920 / 1452 : pp = 138.64512634277344 930 / 1452 : pp = 138.6160430908203 940 / 1452 : pp = 138.66932678222656 950 / 1452 : pp = 138.6573028564453 960 / 1452 : pp = 138.6463165283203 970 / 1452 : pp = 138.67059326171875 980 / 1452 : pp = 138.50999450683594 990 / 1452 : pp = 138.42430114746094 1000 / 1452 : pp = 138.25344848632812 1010 / 1452 : pp = 138.3004608154297 1020 / 1452 : pp = 138.4243621826172 1030 / 1452 : pp = 138.40713500976562 1040 / 1452 : pp = 138.47129821777344 1050 / 1452 : pp = 138.45928955078125 1060 / 1452 : pp = 138.3919677734375 1070 / 1452 : pp = 138.5287628173828 1080 / 1452 : pp = 138.62298583984375 1090 / 1452 : pp = 138.6699981689453 1100 / 1452 : pp = 138.64849853515625 1110 / 1452 : pp = 138.49191284179688 1120 / 1452 : pp = 138.37355041503906 1130 / 1452 : pp = 138.2216796875 1140 / 1452 : pp = 138.21534729003906 1150 / 1452 : pp = 138.30963134765625 1160 / 1452 : pp = 138.316162109375 1170 / 1452 : pp = 138.3023681640625 1180 / 1452 : pp = 138.36932373046875 1190 / 1452 : pp = 138.45960998535156 1200 / 1452 : pp = 138.4866180419922 1210 / 1452 : pp = 138.45730590820312 1220 / 1452 : pp = 138.60031127929688 1230 / 1452 : pp = 138.75485229492188 1240 / 1452 : pp = 138.7751007080078 1250 / 1452 : pp = 138.91221618652344 1260 / 1452 : pp = 138.9815216064453 1270 / 1452 : pp = 138.9919891357422 1280 / 1452 : pp = 139.0243377685547 1290 / 1452 : pp = 139.02725219726562 1300 / 1452 : pp = 139.0701446533203 1310 / 1452 : pp = 139.1090850830078 1320 / 1452 : pp = 139.06027221679688 1330 / 1452 : pp = 139.0338134765625 1340 / 1452 : pp = 139.06385803222656 1350 / 1452 : pp = 139.09608459472656 1360 / 1452 : pp = 139.1609649658203 1370 / 1452 : pp = 139.0869903564453 1380 / 1452 : pp = 139.0604705810547 1390 / 1452 : pp = 139.01670837402344 1400 / 1452 : pp = 138.94393920898438 1410 / 1452 : pp = 138.97323608398438 1420 / 1452 : pp = 138.9404296875 1430 / 1452 : pp = 138.90943908691406 1440 / 1452 : pp = 138.94268798828125 1450 / 1452 : pp = 138.93991088867188 0 / 115 : pp = 225.55990600585938 10 / 115 : pp = 207.0504608154297 20 / 115 : pp = 208.98306274414062 30 / 115 : pp = 206.28396606445312 40 / 115 : pp = 205.35386657714844 50 / 115 : pp = 200.7255401611328 60 / 115 : pp = 200.0526580810547 70 / 115 : pp = 196.33087158203125 80 / 115 : pp = 194.12110900878906 90 / 115 : pp = 191.52816772460938 100 / 115 : pp = 186.7974395751953 110 / 115 : pp = 184.59829711914062 Training perplexity: 138.9222869873047 Validation perplexity:184.18101501464844 Total time : 43.92928600311279 Epoch 12 0 / 1452 : pp = 173.0251007080078 10 / 1452 : pp = 152.98446655273438 20 / 1452 : pp = 150.43128967285156 30 / 1452 : pp = 147.5819854736328 40 / 1452 : pp = 149.4164276123047 50 / 1452 : pp = 146.70816040039062 60 / 1452 : pp = 145.557861328125 70 / 1452 : pp = 146.50473022460938 80 / 1452 : pp = 145.83200073242188 90 / 1452 : pp = 144.84402465820312 100 / 1452 : pp = 144.0390167236328 110 / 1452 : pp = 142.66514587402344 120 / 1452 : pp = 142.3549346923828 130 / 1452 : pp = 141.4630126953125 140 / 1452 : pp = 140.2266082763672 150 / 1452 : pp = 139.67518615722656 160 / 1452 : pp = 139.90414428710938 170 / 1452 : pp = 139.5490264892578 180 / 1452 : pp = 138.91969299316406 190 / 1452 : pp = 138.89234924316406 200 / 1452 : pp = 139.40908813476562 210 / 1452 : pp = 139.19068908691406 220 / 1452 : pp = 139.35513305664062 230 / 1452 : pp = 139.5464324951172 240 / 1452 : pp = 139.3047637939453 250 / 1452 : pp = 138.7708740234375 260 / 1452 : pp = 138.29188537597656 270 / 1452 : pp = 137.4787139892578 280 / 1452 : pp = 137.6367950439453 290 / 1452 : pp = 137.98513793945312 300 / 1452 : pp = 138.17819213867188 310 / 1452 : pp = 137.943359375 320 / 1452 : pp = 138.12060546875 330 / 1452 : pp = 138.29037475585938 340 / 1452 : pp = 137.77606201171875 350 / 1452 : pp = 138.06378173828125 360 / 1452 : pp = 137.99000549316406 370 / 1452 : pp = 137.81922912597656 380 / 1452 : pp = 137.52159118652344 390 / 1452 : pp = 137.61782836914062 400 / 1452 : pp = 137.4178924560547 410 / 1452 : pp = 137.82632446289062 420 / 1452 : pp = 138.17567443847656 430 / 1452 : pp = 138.11863708496094 440 / 1452 : pp = 138.215087890625 450 / 1452 : pp = 137.9976348876953 460 / 1452 : pp = 137.6929168701172 470 / 1452 : pp = 137.25416564941406 480 / 1452 : pp = 136.75140380859375 490 / 1452 : pp = 136.51712036132812 500 / 1452 : pp = 136.0896453857422 510 / 1452 : pp = 135.97048950195312 520 / 1452 : pp = 135.7760009765625 530 / 1452 : pp = 135.50389099121094 540 / 1452 : pp = 135.01437377929688 550 / 1452 : pp = 134.7666015625 560 / 1452 : pp = 134.48973083496094 570 / 1452 : pp = 134.22853088378906 580 / 1452 : pp = 133.88455200195312 590 / 1452 : pp = 133.5808868408203 600 / 1452 : pp = 133.22975158691406 610 / 1452 : pp = 132.99591064453125 620 / 1452 : pp = 132.79502868652344 630 / 1452 : pp = 132.5094451904297 640 / 1452 : pp = 132.62892150878906 650 / 1452 : pp = 132.63499450683594 660 / 1452 : pp = 132.7379913330078 670 / 1452 : pp = 132.79046630859375 680 / 1452 : pp = 132.85842895507812 690 / 1452 : pp = 132.80364990234375 700 / 1452 : pp = 132.80477905273438 710 / 1452 : pp = 132.90170288085938 720 / 1452 : pp = 132.92971801757812 730 / 1452 : pp = 132.9019012451172 740 / 1452 : pp = 133.04811096191406 750 / 1452 : pp = 133.10877990722656 760 / 1452 : pp = 133.19189453125 770 / 1452 : pp = 133.3564910888672 780 / 1452 : pp = 133.54000854492188 790 / 1452 : pp = 133.69239807128906 800 / 1452 : pp = 133.68495178222656 810 / 1452 : pp = 133.67971801757812 820 / 1452 : pp = 133.7035675048828 830 / 1452 : pp = 133.89329528808594 840 / 1452 : pp = 133.850341796875 850 / 1452 : pp = 133.90390014648438 860 / 1452 : pp = 133.9090118408203 870 / 1452 : pp = 133.89974975585938 880 / 1452 : pp = 134.0077667236328 890 / 1452 : pp = 134.03485107421875 900 / 1452 : pp = 134.0261688232422 910 / 1452 : pp = 134.10255432128906 920 / 1452 : pp = 134.17291259765625 930 / 1452 : pp = 134.14796447753906 940 / 1452 : pp = 134.20925903320312 950 / 1452 : pp = 134.19281005859375 960 / 1452 : pp = 134.17745971679688 970 / 1452 : pp = 134.18653869628906 980 / 1452 : pp = 134.03192138671875 990 / 1452 : pp = 133.94349670410156 1000 / 1452 : pp = 133.79685974121094 1010 / 1452 : pp = 133.8438262939453 1020 / 1452 : pp = 133.9608612060547 1030 / 1452 : pp = 133.93934631347656 1040 / 1452 : pp = 134.02833557128906 1050 / 1452 : pp = 134.01734924316406 1060 / 1452 : pp = 133.95346069335938 1070 / 1452 : pp = 134.10205078125 1080 / 1452 : pp = 134.2030487060547 1090 / 1452 : pp = 134.23696899414062 1100 / 1452 : pp = 134.2230224609375 1110 / 1452 : pp = 134.0829315185547 1120 / 1452 : pp = 133.980224609375 1130 / 1452 : pp = 133.83815002441406 1140 / 1452 : pp = 133.8366241455078 1150 / 1452 : pp = 133.92108154296875 1160 / 1452 : pp = 133.94375610351562 1170 / 1452 : pp = 133.9360809326172 1180 / 1452 : pp = 133.99684143066406 1190 / 1452 : pp = 134.0944366455078 1200 / 1452 : pp = 134.11676025390625 1210 / 1452 : pp = 134.0911102294922 1220 / 1452 : pp = 134.22763061523438 1230 / 1452 : pp = 134.38043212890625 1240 / 1452 : pp = 134.39817810058594 1250 / 1452 : pp = 134.5367431640625 1260 / 1452 : pp = 134.593017578125 1270 / 1452 : pp = 134.61497497558594 1280 / 1452 : pp = 134.6423797607422 1290 / 1452 : pp = 134.64340209960938 1300 / 1452 : pp = 134.68026733398438 1310 / 1452 : pp = 134.73556518554688 1320 / 1452 : pp = 134.69021606445312 1330 / 1452 : pp = 134.66131591796875 1340 / 1452 : pp = 134.69393920898438 1350 / 1452 : pp = 134.7328643798828 1360 / 1452 : pp = 134.79405212402344 1370 / 1452 : pp = 134.71237182617188 1380 / 1452 : pp = 134.6885528564453 1390 / 1452 : pp = 134.65110778808594 1400 / 1452 : pp = 134.59584045410156 1410 / 1452 : pp = 134.6193389892578 1420 / 1452 : pp = 134.58338928222656 1430 / 1452 : pp = 134.559326171875 1440 / 1452 : pp = 134.59507751464844 1450 / 1452 : pp = 134.59365844726562 0 / 115 : pp = 226.0741729736328 10 / 115 : pp = 207.00494384765625 20 / 115 : pp = 209.26976013183594 30 / 115 : pp = 206.44662475585938 40 / 115 : pp = 205.47268676757812 50 / 115 : pp = 200.7876739501953 60 / 115 : pp = 200.13414001464844 70 / 115 : pp = 196.35549926757812 80 / 115 : pp = 194.10777282714844 90 / 115 : pp = 191.47467041015625 100 / 115 : pp = 186.61351013183594 110 / 115 : pp = 184.30374145507812 Training perplexity: 134.57826232910156 Validation perplexity:183.8900146484375 Total time : 45.410256147384644 Epoch 13 0 / 1452 : pp = 169.39393615722656 10 / 1452 : pp = 150.13232421875 20 / 1452 : pp = 147.60450744628906 30 / 1452 : pp = 144.64317321777344 40 / 1452 : pp = 146.47427368164062 50 / 1452 : pp = 143.929443359375 60 / 1452 : pp = 142.8344268798828 70 / 1452 : pp = 143.45248413085938 80 / 1452 : pp = 142.5418701171875 90 / 1452 : pp = 141.6178436279297 100 / 1452 : pp = 140.70127868652344 110 / 1452 : pp = 139.2852325439453 120 / 1452 : pp = 138.8017120361328 130 / 1452 : pp = 137.85629272460938 140 / 1452 : pp = 136.51718139648438 150 / 1452 : pp = 136.03619384765625 160 / 1452 : pp = 136.154296875 170 / 1452 : pp = 135.67037963867188 180 / 1452 : pp = 135.0376739501953 190 / 1452 : pp = 134.9230499267578 200 / 1452 : pp = 135.4241180419922 210 / 1452 : pp = 135.24581909179688 220 / 1452 : pp = 135.37957763671875 230 / 1452 : pp = 135.67652893066406 240 / 1452 : pp = 135.4161834716797 250 / 1452 : pp = 134.90895080566406 260 / 1452 : pp = 134.46754455566406 270 / 1452 : pp = 133.68577575683594 280 / 1452 : pp = 133.86770629882812 290 / 1452 : pp = 134.18475341796875 300 / 1452 : pp = 134.39132690429688 310 / 1452 : pp = 134.19985961914062 320 / 1452 : pp = 134.37998962402344 330 / 1452 : pp = 134.5557403564453 340 / 1452 : pp = 134.00686645507812 350 / 1452 : pp = 134.27749633789062 360 / 1452 : pp = 134.20286560058594 370 / 1452 : pp = 134.042724609375 380 / 1452 : pp = 133.74398803710938 390 / 1452 : pp = 133.83584594726562 400 / 1452 : pp = 133.64382934570312 410 / 1452 : pp = 134.02366638183594 420 / 1452 : pp = 134.35415649414062 430 / 1452 : pp = 134.310546875 440 / 1452 : pp = 134.3634490966797 450 / 1452 : pp = 134.15602111816406 460 / 1452 : pp = 133.86578369140625 470 / 1452 : pp = 133.43414306640625 480 / 1452 : pp = 132.90310668945312 490 / 1452 : pp = 132.646240234375 500 / 1452 : pp = 132.1982421875 510 / 1452 : pp = 132.04200744628906 520 / 1452 : pp = 131.86940002441406 530 / 1452 : pp = 131.59841918945312 540 / 1452 : pp = 131.12356567382812 550 / 1452 : pp = 130.887939453125 560 / 1452 : pp = 130.6210174560547 570 / 1452 : pp = 130.37826538085938 580 / 1452 : pp = 130.0374755859375 590 / 1452 : pp = 129.75979614257812 600 / 1452 : pp = 129.38308715820312 610 / 1452 : pp = 129.16685485839844 620 / 1452 : pp = 129.0115509033203 630 / 1452 : pp = 128.75152587890625 640 / 1452 : pp = 128.87295532226562 650 / 1452 : pp = 128.88734436035156 660 / 1452 : pp = 128.98275756835938 670 / 1452 : pp = 129.0487060546875 680 / 1452 : pp = 129.11013793945312 690 / 1452 : pp = 129.0646514892578 700 / 1452 : pp = 129.06280517578125 710 / 1452 : pp = 129.1343994140625 720 / 1452 : pp = 129.18582153320312 730 / 1452 : pp = 129.15138244628906 740 / 1452 : pp = 129.29811096191406 750 / 1452 : pp = 129.339599609375 760 / 1452 : pp = 129.4257354736328 770 / 1452 : pp = 129.61631774902344 780 / 1452 : pp = 129.802734375 790 / 1452 : pp = 129.96804809570312 800 / 1452 : pp = 129.95187377929688 810 / 1452 : pp = 129.92417907714844 820 / 1452 : pp = 129.9774627685547 830 / 1452 : pp = 130.1638946533203 840 / 1452 : pp = 130.13095092773438 850 / 1452 : pp = 130.16595458984375 860 / 1452 : pp = 130.173828125 870 / 1452 : pp = 130.170166015625 880 / 1452 : pp = 130.27032470703125 890 / 1452 : pp = 130.3022003173828 900 / 1452 : pp = 130.3071746826172 910 / 1452 : pp = 130.37939453125 920 / 1452 : pp = 130.46229553222656 930 / 1452 : pp = 130.43846130371094 940 / 1452 : pp = 130.50889587402344 950 / 1452 : pp = 130.50086975097656 960 / 1452 : pp = 130.4833221435547 970 / 1452 : pp = 130.50814819335938 980 / 1452 : pp = 130.35577392578125 990 / 1452 : pp = 130.26759338378906 1000 / 1452 : pp = 130.1064453125 1010 / 1452 : pp = 130.1472625732422 1020 / 1452 : pp = 130.27169799804688 1030 / 1452 : pp = 130.25100708007812 1040 / 1452 : pp = 130.30816650390625 1050 / 1452 : pp = 130.29803466796875 1060 / 1452 : pp = 130.2242431640625 1070 / 1452 : pp = 130.35906982421875 1080 / 1452 : pp = 130.45103454589844 1090 / 1452 : pp = 130.49838256835938 1100 / 1452 : pp = 130.484130859375 1110 / 1452 : pp = 130.35316467285156 1120 / 1452 : pp = 130.24697875976562 1130 / 1452 : pp = 130.10804748535156 1140 / 1452 : pp = 130.1076202392578 1150 / 1452 : pp = 130.195068359375 1160 / 1452 : pp = 130.19674682617188 1170 / 1452 : pp = 130.18321228027344 1180 / 1452 : pp = 130.24623107910156 1190 / 1452 : pp = 130.33905029296875 1200 / 1452 : pp = 130.3650360107422 1210 / 1452 : pp = 130.34588623046875 1220 / 1452 : pp = 130.4850616455078 1230 / 1452 : pp = 130.63160705566406 1240 / 1452 : pp = 130.64674377441406 1250 / 1452 : pp = 130.77078247070312 1260 / 1452 : pp = 130.8397674560547 1270 / 1452 : pp = 130.8511199951172 1280 / 1452 : pp = 130.88967895507812 1290 / 1452 : pp = 130.9040985107422 1300 / 1452 : pp = 130.93511962890625 1310 / 1452 : pp = 130.9759063720703 1320 / 1452 : pp = 130.92800903320312 1330 / 1452 : pp = 130.9105224609375 1340 / 1452 : pp = 130.929443359375 1350 / 1452 : pp = 130.96153259277344 1360 / 1452 : pp = 131.02381896972656 1370 / 1452 : pp = 130.9545440673828 1380 / 1452 : pp = 130.9344940185547 1390 / 1452 : pp = 130.9055938720703 1400 / 1452 : pp = 130.85386657714844 1410 / 1452 : pp = 130.8874969482422 1420 / 1452 : pp = 130.85928344726562 1430 / 1452 : pp = 130.83995056152344 1440 / 1452 : pp = 130.86659240722656 1450 / 1452 : pp = 130.86839294433594 0 / 115 : pp = 227.78428649902344 10 / 115 : pp = 207.609619140625 20 / 115 : pp = 209.92459106445312 30 / 115 : pp = 206.96240234375 40 / 115 : pp = 205.9295654296875 50 / 115 : pp = 201.0296630859375 60 / 115 : pp = 200.38059997558594 70 / 115 : pp = 196.55764770507812 80 / 115 : pp = 194.31735229492188 90 / 115 : pp = 191.66146850585938 100 / 115 : pp = 186.70437622070312 110 / 115 : pp = 184.3171844482422 Training perplexity: 130.85043334960938 Validation perplexity:183.88186645507812 Total time : 45.345656394958496 Epoch 14 0 / 1452 : pp = 164.82191467285156 10 / 1452 : pp = 146.39089965820312 20 / 1452 : pp = 142.93240356445312 30 / 1452 : pp = 140.3113555908203 40 / 1452 : pp = 142.39939880371094 50 / 1452 : pp = 139.70162963867188 60 / 1452 : pp = 138.73023986816406 70 / 1452 : pp = 139.2675018310547 80 / 1452 : pp = 138.47824096679688 90 / 1452 : pp = 137.40432739257812 100 / 1452 : pp = 136.47793579101562 110 / 1452 : pp = 135.2294464111328 120 / 1452 : pp = 134.80728149414062 130 / 1452 : pp = 133.89822387695312 140 / 1452 : pp = 132.54141235351562 150 / 1452 : pp = 132.10025024414062 160 / 1452 : pp = 132.21829223632812 170 / 1452 : pp = 131.8765106201172 180 / 1452 : pp = 131.37515258789062 190 / 1452 : pp = 131.31622314453125 200 / 1452 : pp = 131.78297424316406 210 / 1452 : pp = 131.5507354736328 220 / 1452 : pp = 131.7002410888672 230 / 1452 : pp = 131.9277801513672 240 / 1452 : pp = 131.72166442871094 250 / 1452 : pp = 131.225830078125 260 / 1452 : pp = 130.7496337890625 270 / 1452 : pp = 129.9896697998047 280 / 1452 : pp = 130.10594177246094 290 / 1452 : pp = 130.41644287109375 300 / 1452 : pp = 130.5982208251953 310 / 1452 : pp = 130.36329650878906 320 / 1452 : pp = 130.5633544921875 330 / 1452 : pp = 130.77252197265625 340 / 1452 : pp = 130.273193359375 350 / 1452 : pp = 130.47889709472656 360 / 1452 : pp = 130.4348602294922 370 / 1452 : pp = 130.28126525878906 380 / 1452 : pp = 130.02786254882812 390 / 1452 : pp = 130.1564483642578 400 / 1452 : pp = 129.98440551757812 410 / 1452 : pp = 130.37721252441406 420 / 1452 : pp = 130.71859741210938 430 / 1452 : pp = 130.65939331054688 440 / 1452 : pp = 130.72987365722656 450 / 1452 : pp = 130.56272888183594 460 / 1452 : pp = 130.28195190429688 470 / 1452 : pp = 129.90936279296875 480 / 1452 : pp = 129.42857360839844 490 / 1452 : pp = 129.18077087402344 500 / 1452 : pp = 128.7588348388672 510 / 1452 : pp = 128.6303253173828 520 / 1452 : pp = 128.47616577148438 530 / 1452 : pp = 128.21148681640625 540 / 1452 : pp = 127.7218017578125 550 / 1452 : pp = 127.50067138671875 560 / 1452 : pp = 127.27574157714844 570 / 1452 : pp = 127.05399322509766 580 / 1452 : pp = 126.73983001708984 590 / 1452 : pp = 126.43692779541016 600 / 1452 : pp = 126.06050109863281 610 / 1452 : pp = 125.82952880859375 620 / 1452 : pp = 125.66295623779297 630 / 1452 : pp = 125.39354705810547 640 / 1452 : pp = 125.49463653564453 650 / 1452 : pp = 125.48816680908203 660 / 1452 : pp = 125.58712005615234 670 / 1452 : pp = 125.65978240966797 680 / 1452 : pp = 125.71456146240234 690 / 1452 : pp = 125.66937255859375 700 / 1452 : pp = 125.65900421142578 710 / 1452 : pp = 125.7271499633789 720 / 1452 : pp = 125.77758026123047 730 / 1452 : pp = 125.74129486083984 740 / 1452 : pp = 125.8759765625 750 / 1452 : pp = 125.91793823242188 760 / 1452 : pp = 125.99595642089844 770 / 1452 : pp = 126.18113708496094 780 / 1452 : pp = 126.35147094726562 790 / 1452 : pp = 126.50797271728516 800 / 1452 : pp = 126.49759674072266 810 / 1452 : pp = 126.48113250732422 820 / 1452 : pp = 126.52528381347656 830 / 1452 : pp = 126.705810546875 840 / 1452 : pp = 126.67517852783203 850 / 1452 : pp = 126.74176025390625 860 / 1452 : pp = 126.74151611328125 870 / 1452 : pp = 126.73414611816406 880 / 1452 : pp = 126.83026885986328 890 / 1452 : pp = 126.88519287109375 900 / 1452 : pp = 126.88053894042969 910 / 1452 : pp = 126.97138214111328 920 / 1452 : pp = 127.04660034179688 930 / 1452 : pp = 127.03763580322266 940 / 1452 : pp = 127.1126480102539 950 / 1452 : pp = 127.09610748291016 960 / 1452 : pp = 127.0873794555664 970 / 1452 : pp = 127.10343933105469 980 / 1452 : pp = 126.96441650390625 990 / 1452 : pp = 126.88519287109375 1000 / 1452 : pp = 126.7336654663086 1010 / 1452 : pp = 126.77796936035156 1020 / 1452 : pp = 126.89826202392578 1030 / 1452 : pp = 126.88761138916016 1040 / 1452 : pp = 126.95309448242188 1050 / 1452 : pp = 126.96478271484375 1060 / 1452 : pp = 126.89324188232422 1070 / 1452 : pp = 127.03242492675781 1080 / 1452 : pp = 127.13228607177734 1090 / 1452 : pp = 127.173095703125 1100 / 1452 : pp = 127.15975189208984 1110 / 1452 : pp = 127.0392074584961 1120 / 1452 : pp = 126.94032287597656 1130 / 1452 : pp = 126.80693054199219 1140 / 1452 : pp = 126.81315612792969 1150 / 1452 : pp = 126.90467834472656 1160 / 1452 : pp = 126.91236114501953 1170 / 1452 : pp = 126.90897369384766 1180 / 1452 : pp = 126.98052215576172 1190 / 1452 : pp = 127.07483673095703 1200 / 1452 : pp = 127.10216522216797 1210 / 1452 : pp = 127.08258819580078 1220 / 1452 : pp = 127.22943878173828 1230 / 1452 : pp = 127.38563537597656 1240 / 1452 : pp = 127.40538024902344 1250 / 1452 : pp = 127.53369140625 1260 / 1452 : pp = 127.59293365478516 1270 / 1452 : pp = 127.61489868164062 1280 / 1452 : pp = 127.6484375 1290 / 1452 : pp = 127.65257263183594 1300 / 1452 : pp = 127.69329833984375 1310 / 1452 : pp = 127.74549102783203 1320 / 1452 : pp = 127.7043228149414 1330 / 1452 : pp = 127.6866683959961 1340 / 1452 : pp = 127.70913696289062 1350 / 1452 : pp = 127.73233795166016 1360 / 1452 : pp = 127.7855224609375 1370 / 1452 : pp = 127.71918487548828 1380 / 1452 : pp = 127.69987487792969 1390 / 1452 : pp = 127.6697998046875 1400 / 1452 : pp = 127.61137390136719 1410 / 1452 : pp = 127.6404037475586 1420 / 1452 : pp = 127.61094665527344 1430 / 1452 : pp = 127.58216857910156 1440 / 1452 : pp = 127.61477661132812 1450 / 1452 : pp = 127.61964416503906 0 / 115 : pp = 228.21578979492188 10 / 115 : pp = 208.11244201660156 20 / 115 : pp = 210.688232421875 30 / 115 : pp = 207.62408447265625 40 / 115 : pp = 206.45184326171875 50 / 115 : pp = 201.52760314941406 60 / 115 : pp = 200.7784881591797 70 / 115 : pp = 196.83067321777344 80 / 115 : pp = 194.6357879638672 90 / 115 : pp = 191.9783935546875 100 / 115 : pp = 186.8787841796875 110 / 115 : pp = 184.35252380371094 Training perplexity: 127.60413360595703 Validation perplexity:183.8877410888672 Total time : 41.6636528968811 Epoch 15 0 / 1452 : pp = 156.81654357910156 10 / 1452 : pp = 142.1070556640625 20 / 1452 : pp = 139.55076599121094 30 / 1452 : pp = 136.63551330566406 40 / 1452 : pp = 138.5840606689453 50 / 1452 : pp = 136.052734375 60 / 1452 : pp = 134.93019104003906 70 / 1452 : pp = 135.65206909179688 80 / 1452 : pp = 135.2620086669922 90 / 1452 : pp = 134.314697265625 100 / 1452 : pp = 133.4916229248047 110 / 1452 : pp = 132.26052856445312 120 / 1452 : pp = 131.7714080810547 130 / 1452 : pp = 130.77365112304688 140 / 1452 : pp = 129.5411834716797 150 / 1452 : pp = 129.0791778564453 160 / 1452 : pp = 129.21920776367188 170 / 1452 : pp = 128.7528839111328 180 / 1452 : pp = 128.22279357910156 190 / 1452 : pp = 128.18177795410156 200 / 1452 : pp = 128.58758544921875 210 / 1452 : pp = 128.3906707763672 220 / 1452 : pp = 128.5266571044922 230 / 1452 : pp = 128.80563354492188 240 / 1452 : pp = 128.61886596679688 250 / 1452 : pp = 128.13172912597656 260 / 1452 : pp = 127.69220733642578 270 / 1452 : pp = 126.96150970458984 280 / 1452 : pp = 127.04702758789062 290 / 1452 : pp = 127.33565521240234 300 / 1452 : pp = 127.55929565429688 310 / 1452 : pp = 127.38514709472656 320 / 1452 : pp = 127.52171325683594 330 / 1452 : pp = 127.68690490722656 340 / 1452 : pp = 127.18340301513672 350 / 1452 : pp = 127.4073257446289 360 / 1452 : pp = 127.30432891845703 370 / 1452 : pp = 127.17618560791016 380 / 1452 : pp = 126.92579650878906 390 / 1452 : pp = 127.02473449707031 400 / 1452 : pp = 126.8515625 410 / 1452 : pp = 127.211669921875 420 / 1452 : pp = 127.51788330078125 430 / 1452 : pp = 127.47386169433594 440 / 1452 : pp = 127.57164001464844 450 / 1452 : pp = 127.3601303100586 460 / 1452 : pp = 127.09434509277344 470 / 1452 : pp = 126.71922302246094 480 / 1452 : pp = 126.24349212646484 490 / 1452 : pp = 125.98778533935547 500 / 1452 : pp = 125.59526824951172 510 / 1452 : pp = 125.4450912475586 520 / 1452 : pp = 125.29247283935547 530 / 1452 : pp = 125.03536224365234 540 / 1452 : pp = 124.5813980102539 550 / 1452 : pp = 124.33724212646484 560 / 1452 : pp = 124.08995819091797 570 / 1452 : pp = 123.86637878417969 580 / 1452 : pp = 123.53152465820312 590 / 1452 : pp = 123.20321655273438 600 / 1452 : pp = 122.85673522949219 610 / 1452 : pp = 122.64250946044922 620 / 1452 : pp = 122.4958724975586 630 / 1452 : pp = 122.22386169433594 640 / 1452 : pp = 122.31143188476562 650 / 1452 : pp = 122.30093383789062 660 / 1452 : pp = 122.39427947998047 670 / 1452 : pp = 122.45440673828125 680 / 1452 : pp = 122.51146697998047 690 / 1452 : pp = 122.4854736328125 700 / 1452 : pp = 122.48600006103516 710 / 1452 : pp = 122.56084442138672 720 / 1452 : pp = 122.59059143066406 730 / 1452 : pp = 122.55529022216797 740 / 1452 : pp = 122.69409942626953 750 / 1452 : pp = 122.76456451416016 760 / 1452 : pp = 122.84437561035156 770 / 1452 : pp = 123.02527618408203 780 / 1452 : pp = 123.20509338378906 790 / 1452 : pp = 123.36305236816406 800 / 1452 : pp = 123.36852264404297 810 / 1452 : pp = 123.36799621582031 820 / 1452 : pp = 123.39976501464844 830 / 1452 : pp = 123.59362030029297 840 / 1452 : pp = 123.56946563720703 850 / 1452 : pp = 123.63800811767578 860 / 1452 : pp = 123.63983917236328 870 / 1452 : pp = 123.64148712158203 880 / 1452 : pp = 123.7568588256836 890 / 1452 : pp = 123.7885513305664 900 / 1452 : pp = 123.79640197753906 910 / 1452 : pp = 123.86153411865234 920 / 1452 : pp = 123.92941284179688 930 / 1452 : pp = 123.9125747680664 940 / 1452 : pp = 123.95559692382812 950 / 1452 : pp = 123.93928527832031 960 / 1452 : pp = 123.94294738769531 970 / 1452 : pp = 123.95547485351562 980 / 1452 : pp = 123.8229751586914 990 / 1452 : pp = 123.73727416992188 1000 / 1452 : pp = 123.59091186523438 1010 / 1452 : pp = 123.634765625 1020 / 1452 : pp = 123.76506042480469 1030 / 1452 : pp = 123.75485229492188 1040 / 1452 : pp = 123.807861328125 1050 / 1452 : pp = 123.79156494140625 1060 / 1452 : pp = 123.73054504394531 1070 / 1452 : pp = 123.8615951538086 1080 / 1452 : pp = 123.96564483642578 1090 / 1452 : pp = 124.02104187011719 1100 / 1452 : pp = 124.012939453125 1110 / 1452 : pp = 123.87582397460938 1120 / 1452 : pp = 123.775390625 1130 / 1452 : pp = 123.63182067871094 1140 / 1452 : pp = 123.62391662597656 1150 / 1452 : pp = 123.71013641357422 1160 / 1452 : pp = 123.72423553466797 1170 / 1452 : pp = 123.71726989746094 1180 / 1452 : pp = 123.79032897949219 1190 / 1452 : pp = 123.87883758544922 1200 / 1452 : pp = 123.9125747680664 1210 / 1452 : pp = 123.90140533447266 1220 / 1452 : pp = 124.03245544433594 1230 / 1452 : pp = 124.19799041748047 1240 / 1452 : pp = 124.21469116210938 1250 / 1452 : pp = 124.34103393554688 1260 / 1452 : pp = 124.4041976928711 1270 / 1452 : pp = 124.42852020263672 1280 / 1452 : pp = 124.46656036376953 1290 / 1452 : pp = 124.4811019897461 1300 / 1452 : pp = 124.52384185791016 1310 / 1452 : pp = 124.57533264160156 1320 / 1452 : pp = 124.5398178100586 1330 / 1452 : pp = 124.52598571777344 1340 / 1452 : pp = 124.53311157226562 1350 / 1452 : pp = 124.57759094238281 1360 / 1452 : pp = 124.63385772705078 1370 / 1452 : pp = 124.58133697509766 1380 / 1452 : pp = 124.55769348144531 1390 / 1452 : pp = 124.54011535644531 1400 / 1452 : pp = 124.4884033203125 1410 / 1452 : pp = 124.51226806640625 1420 / 1452 : pp = 124.49683380126953 1430 / 1452 : pp = 124.4754638671875 1440 / 1452 : pp = 124.50164031982422 1450 / 1452 : pp = 124.50894165039062 0 / 115 : pp = 230.8488006591797 10 / 115 : pp = 209.2509002685547 20 / 115 : pp = 211.68577575683594 30 / 115 : pp = 208.44056701660156 40 / 115 : pp = 207.2039337158203 50 / 115 : pp = 202.1859588623047 60 / 115 : pp = 201.34739685058594 70 / 115 : pp = 197.4251251220703 80 / 115 : pp = 195.2623291015625 90 / 115 : pp = 192.592529296875 100 / 115 : pp = 187.39553833007812 110 / 115 : pp = 184.791259765625 Training perplexity: 124.4933853149414 Validation perplexity:184.32510375976562 Total time : 40.856229066848755 0 / 128 : pp = 184.6475067138672 10 / 128 : pp = 176.8856964111328 20 / 128 : pp = 164.3444366455078 30 / 128 : pp = 167.85472106933594 40 / 128 : pp = 169.25367736816406 50 / 128 : pp = 168.86561584472656 60 / 128 : pp = 168.11801147460938 70 / 128 : pp = 165.4105224609375 80 / 128 : pp = 162.91146850585938 90 / 128 : pp = 161.29742431640625 100 / 128 : pp = 162.45989990234375 110 / 128 : pp = 162.6834716796875 120 / 128 : pp = 164.3359832763672 =-==-==-==-==-= Test perplexity: 164.0149383544922 =-==-==-==-==-=
更详细的内容请参考下面链接
https://github.com/weizhenzhao/cs224d_nlp_problem_set2
今天将的还是cs224d 的problem set2 的第三部分习题，
原来国外大学的系统难度真的如此之大，相比之下还是默默地再天朝继续搬砖吧
下面讲述一下RNN语言建模的数学公式:
给出一串连续的词x1,x2...xt关于预测其后面紧跟的词xt+1的建模方式是:
vj是词库中的某个词。实现一个循环神经网络，此网络利用隐层中的反馈信息对"历史记录"x1,x2...xt进行建模:
$h^{(0)}=h_{0}\epsilon R^{D_{h}}$是隐藏层的初始化向量
$x^{(t)}L$是以$x^{(t)}$one-hot行向量与嵌入矩阵L的乘积
这个one-hot行向量就是当前处理词汇的索引
是词嵌入矩阵,
$L$是词嵌入矩阵
$I$是输入词表征矩阵
$H$是隐藏转换矩阵
$U$是输出词表征矩阵
$b_{1}$ $b_{2}$是偏置值
$d$是词嵌入的维数
|V|代表词库的规模
$D_{h}$是隐层的维数
输出向量
是面向整个词库的概率分布，我们需要最优化交叉熵（非正则化的）的损失率：
使用困惑度来评估语言模型的性能，其定义形式如下：
梯度:
该模型中各个变量进行最优化迭代的时候的梯度如下所示:
初始化所有的上面这些需要训练的参数的值
然后通过对每一个词进行训练，安装上述公司求出每个参数的导数值
然后使用梯度下降方法对其进行更新
将新得到的参数代入到模型中，如果损失的值小于初始设定的值则停止迭代，否则继续进行迭代
下面是一张RNNLM的结构图
上面这张是第二层RNN节点的结构图
上面这张是在RNN的变量上面应用Dropout的结构，降低模型过拟合的误差，第一层RNN的dropout结构
上面这张是第一层RNN的结构图
（注意前方高能，一大批天书即将来袭）
''' Created on 2017年9月26日 @author: weizhen ''' import getpass import sys import time import numpy as np from copy import deepcopy from utils import calculate_perplexity, get_ptb_dataset, Vocab from utils import ptb_iterator, sample import tensorflow as tf from model import LanguageModel from tensorflow.contrib.legacy_seq2seq.python.ops.seq2seq import sequence_loss class Config(object): """储存超参数和数据信息""" batch_size = 64 embed_size = 50 hidden_size = 100 num_steps = 10 max_epochs = 16 early_stopping = 2 dropout = 0.9 lr = 0.001 class RNNLM_Model(LanguageModel): def load_data(self, debug=False): """加载词向量并且训练 train/dev/test 数据""" self.vocab = Vocab() self.vocab.construct(get_ptb_dataset('train')) self.encoded_train = np.array([self.vocab.encode(word) for word in get_ptb_dataset('train')], dtype=np.int32) self.encoded_valid = np.array([self.vocab.encode(word) for word in get_ptb_dataset('valid')], dtype=np.int32) self.encoded_test = np.array([self.vocab.encode(word) for word in get_ptb_dataset('test')]) if debug: num_debug = 1024 self.encoded_train = self.encoded_train[:num_debug] self.encoded_valid = self.encoded_valid[:num_debug] self.encoded_test = self.encoded_test[:num_debug] def add_placeholders(self): """生成placeholder 变量来表示输入的 tensors 这些placeholder 被用来在模型的其他地方被填充 并且在训练的过程中会被填充 input_placeholder:Input placeholder shape (None,num_steps),type tf.int32 labels_placeholder:label placeholder shape (None,num_steps) type tf.float32 dropout_placeholder:dropput value placeholder (scalar), type tf.float32 """ self.input_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Input') self.labels_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Target') self.dropout_placeholder = tf.placeholder(tf.float32, name='Dropout') def add_embedding(self): """添加词嵌入层 Hint : 这一层应该用input_placeholder 来索引词嵌入 Hint : 你或许能发现tf.nn.embedding_lookup 是有用的 Hint : 你或许能发现tf.split , tf.squeeze 是有用的在构造tensor 的输入的时候 Hint : 下面是你需要创建的变量的维度 L:(len(self.vocab),embed_size) Returns: inputs:一个训练次数的列表,每一个元素应该是 一个张量 大小是 (batch_size,embed_size) tf.split(dimension,num_split,input) dimension表示输入张量的哪一个维度, 如果是0就表示对第0维度进行切割, num_split就是切割的数量, 如果是2就表示输入张量被切成2份, 每一份是一个列表 tf.squeeze(input,squeeze_dims=None,name=None) 从tensor中删除所有大小是1的维度 example: t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t))==>[2,3] t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t,[2,4]))==>[1,2,3,1] tf.nn.embedding_lookup 将词的索引映射到词的向量 """ with tf.device('/cpu:0'): embedding = tf.get_variable('Embedding', [len(self.vocab), self.config.embed_size], trainable=True) inputs = tf.nn.embedding_lookup(embedding, self.input_placeholder) inputs = [tf.squeeze(x, [1]) for x in tf.split(inputs, self.config.num_steps, 1)] return inputs def add_projection(self, rnn_outputs): """添加一个投影层 投影层将隐藏层的表示变换到整个词向量上的分布式表示 Hint:下面是你需要去创建的维度 U(hidden_size,len(vocab)) b_2:(len(vocab),) 参数: rnn_outputs:一个训练次数的列表，每一个元素应该是一个张量 大小是(batch_size,embed_size) Returns: outputs:一个长度的列表，每一个元素是一个张量(batch_size,len(vocab)) """ with tf.variable_scope('Projection'): U = tf.get_variable('Matrix', [self.config.hidden_size, len(self.vocab)]) proj_b = tf.get_variable('Bias', [len(self.vocab)]) outputs = [tf.matmul(o, U) + proj_b for o in rnn_outputs] return outputs def add_loss_op(self, output): """将损失添加到目标函数上面 Hint:使用tensorflow.python.ops.seq2seq.sequence_loss 来实现序列损失 参数: 输出:一个张量 大小是 (None,self.vocab) 返回: 损失：一个0-d大小的张量 """ all_ones = [tf.ones([self.config.batch_size * self.config.num_steps])] cross_entropy = sequence_loss([output], [tf.reshape(self.labels_placeholder, [-1])], all_ones, len(self.vocab)) tf.add_to_collection('total_loss', cross_entropy) loss = tf.add_n(tf.get_collection('total_loss')) return loss def add_training_op(self, loss): """将目标损失添加到计算图上 创建一个优化器并且应用梯度下降到所有的训练变量上面 Hint:使用tf.train.AdamOptimizer 对于这个模型 使用optimizer.minimize() 会返回一个train_op的对象 参数: loss: 损失张量，来自于cross_entropy_loss 交叉熵损失 返回: train_op:训练的目标 """ with tf.variable_scope("Optimizer") as scope: train_op = tf.train.AdamOptimizer(self.config.lr).minimize(loss) return train_op def __init__(self, config): self.config = config self.load_data(debug=False) self.add_placeholders() self.inputs = self.add_embedding() self.rnn_outputs = self.add_model(self.inputs) self.outputs = self.add_projection(self.rnn_outputs) # 我们想去检验下一个词预测得多好 # 我们把o转变成float64 位 因为如果不这样就会有数值问题 # sum(output of softmax) = 1.00000298179 并且不是 1 self.predictions = [tf.nn.softmax(tf.cast(o, 'float64')) for o in self.outputs] # 将输出值转变成 len(vocab) 的大小 output = tf.reshape(tf.concat(self.outputs, 1), [-1, len(self.vocab)]) self.calculate_loss = self.add_loss_op(output) self.train_step = self.add_training_op(self.calculate_loss) def add_model(self, inputs): """创建RNN LM 模型 在下面的实现里面你需要去实现RNN LM 模型的等式 Hint: 使用一个零向量 大小是 (batch_size,hidden_size) 作为初始的RNN的状态 Hint: 将最后RNN输出 作为实例变量 self.final_state Hint : 确保将dropout应用到 输入和输出的 变量上面 Hint : 使用变量域 RNN 来定义 RNN变量 Hint : 表现一个明显的 for-loop 在输入上面 你可以使用scope.reuse_variable() 来确定权重 在每一次迭代都是相同的 确保不会在第一次循环的时候调用这个，因为没有变量会被初始化 Hint : 下面变量的不同的维度 ， 你需要去创建的 H: (hidden_size,hidden_size) I: (embed_size,hidden_size) b_1:(hidden_size,) Args: inputs:一个记录num_steps的列表，里边的每一个元素应该是一个张量 大小是(batch_size,embed_size)的大小 Returns:返回 outputs:一个记录num_steps的列表，里面每一个元素应该是一个张量 大小是(batch_size,hidden_size) """ with tf.variable_scope('InputDropout'): inputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in inputs] with tf.variable_scope('RNN') as scope: self.initial_state = tf.zeros([self.config.batch_size, self.config.hidden_size]) state = self.initial_state rnn_outputs = [] for tstep, current_input in enumerate(inputs): if tstep > 0: scope.reuse_variables() RNN_H = tf.get_variable('HMatrix', [self.config.hidden_size, self.config.hidden_size]) RNN_I = tf.get_variable('IMatrix', [self.config.embed_size, self.config.hidden_size]) RNN_b = tf.get_variable('B', [self.config.hidden_size]) state = tf.nn.sigmoid(tf.matmul(state, RNN_H) + tf.matmul(current_input, RNN_I) + RNN_b) rnn_outputs.append(state) self.final_state = rnn_outputs[-1] with tf.variable_scope('RNNDropout'): rnn_outputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in rnn_outputs] return rnn_outputs def run_epoch(self, session, data, train_op=None, verbose=10): config = self.config dp = config.dropout if not train_op: train_op = tf.no_op() dp = 1 total_steps = sum(1 for x in ptb_iterator(data, config.batch_size, config.num_steps)) total_loss = [] state = self.initial_state.eval() for step, (x, y) in enumerate(ptb_iterator(data, config.batch_size, config.num_steps)): # 我们需要通过初始状态，并且从最终状态中抽取数据来进行填充 # RNN 合适的 历史 feed = {self.input_placeholder: x, self.labels_placeholder: y, self.initial_state: state, self.dropout_placeholder: dp } loss, state, _ = session.run([self.calculate_loss, self.final_state, train_op], feed_dict=feed) total_loss.append(loss) if verbose and step % verbose == 0: sys.stdout.write('\r{} / {} : pp = {} '.format(step, total_steps, np.exp(np.mean(total_loss)))) sys.stdout.flush() if verbose: sys.stdout.write('\r') return np.exp(np.mean(total_loss)) def generate_text(session, model, config, starting_text='<eos>', stop_length=100, stop_tokens=None, temp=1.0): """从模型自动生成文字 Hint:创建一个feed-dictionary 并且使用sess.run()方法去执行这个模型 你会需要使用model.initial_state 作为一个键传递给feed_dict Hint:得到model.final_state 和 model.predictions[-1]. 在add_model()方法中设置model.final_state 。 model.predictions 是在 __init__方法中设置的 Hint:在模型的训练中存储输出的参数值，和预测的y_pred的值 参数: Args: session : tf.Session() object model : Object of type RNNLM Model config : A Config() object starting_text:Initial text passed to model Returns: output : List of word idxs """ state = model.initial_state.eval() # Imagine tokens as a batch size of one, length of len(tokens[0]) tokens = [model.vocab.encode(word) for word in starting_text.split()] for i in range(stop_length): feed = {model.input_placeholder: [tokens[-1:]], model.initial_state: state, model.dropout_placeholder: 1} state, y_pred = session.run([model.final_state, model.predictions[-1]], feed_dict=feed) next_word_idx = sample(y_pred[0], temperature=temp) tokens.append(next_word_idx) if stop_tokens and model.vocab.decode(tokens[-1]) in stop_tokens: break output = [model.vocab.decode(word_idx) for word_idx in tokens] return output def generate_sentence(session, model, config, *args, **kwargs): """方便从模型来生成句子""" return generate_text(session, model, config, *args, stop_tokens=['<eos>'], **kwargs) def test_RNNLM(): config = Config() gen_config = deepcopy(config) gen_config.batch_size = gen_config.num_steps = 1 # 创建训练模型，并且生成模型 with tf.variable_scope('RNNLM',reuse=None) as scope: model = RNNLM_Model(config) # 这个指示gen_model来重新使用相同的变量作为以上的模型 scope.reuse_variables() gen_model = RNNLM_Model(gen_config) init = tf.global_variables_initializer() saver = tf.train.Saver() with tf.Session() as session: best_val_pp = float('inf') best_val_epoch = 0 session.run(init) for epoch in range(config.max_epochs): print('Epoch {0}'.format(epoch)) start = time.time() train_pp = model.run_epoch(session, model.encoded_train, train_op=model.train_step) valid_pp = model.run_epoch(session, model.encoded_valid) print('Training perplexity: {0}'.format(train_pp)) print('Validation perplexity:{0}'.format(valid_pp)) if valid_pp < best_val_pp: best_val_pp = valid_pp best_val_epoch = epoch saver.save(session, './ptb_rnnlm.weights') if epoch - best_val_epoch > config.early_stopping: break print('Total time : {0}'.format(time.time() - start)) saver.restore(session, 'ptb_rnnlm.weights') test_pp = model.run_epoch(session, model.encoded_test) print('=-=' * 5) print('Test perplexity: {0} '.format(test_pp)) print('=-=' * 5) starting_text = 'in palo alto' while starting_text: print(' '.join(generate_sentence(session, gen_model, gen_config, starting_text=starting_text, temp=1.0))) #starting_text = raw_input('>') if __name__ == "__main__": test_RNNLM()
（其实也不算是天书啦，比高数简单多啦，比数学分析那是简单了好几十万倍了呀）
下面是训练的Log
1380 / 1452 : pp = 266.20892333984375 1390 / 1452 : pp = 265.94439697265625 1400 / 1452 : pp = 265.66845703125 1410 / 1452 : pp = 265.5393981933594 1420 / 1452 : pp = 265.32489013671875 1430 / 1452 : pp = 265.2019348144531 1440 / 1452 : pp = 265.13720703125 1450 / 1452 : pp = 264.954833984375 0 / 115 : pp = 296.9217224121094 10 / 115 : pp = 282.02130126953125 20 / 115 : pp = 279.76800537109375 30 / 115 : pp = 276.4101257324219 40 / 115 : pp = 276.2939147949219 50 / 115 : pp = 270.73565673828125 60 / 115 : pp = 269.88134765625 70 / 115 : pp = 266.8675231933594 80 / 115 : pp = 263.6731872558594 90 / 115 : pp = 260.8569030761719 100 / 115 : pp = 256.3356628417969 110 / 115 : pp = 255.1026611328125 Training perplexity: 264.9092102050781 Validation perplexity:254.84902954101562 Total time : 41.65332388877869 Epoch 3 0 / 1452 : pp = 327.0847473144531 10 / 1452 : pp = 273.9620056152344 20 / 1452 : pp = 270.22943115234375 30 / 1452 : pp = 263.5213317871094 40 / 1452 : pp = 264.0644836425781 50 / 1452 : pp = 258.6029968261719 60 / 1452 : pp = 257.04290771484375 70 / 1452 : pp = 257.59161376953125 80 / 1452 : pp = 256.7600402832031 90 / 1452 : pp = 254.5120391845703 100 / 1452 : pp = 252.44725036621094 110 / 1452 : pp = 250.13954162597656 120 / 1452 : pp = 249.91647338867188 130 / 1452 : pp = 249.50460815429688 140 / 1452 : pp = 247.67440795898438 150 / 1452 : pp = 247.19090270996094 160 / 1452 : pp = 247.8919219970703 170 / 1452 : pp = 247.54322814941406 180 / 1452 : pp = 246.17623901367188 190 / 1452 : pp = 245.78330993652344 200 / 1452 : pp = 246.80552673339844 210 / 1452 : pp = 246.3059844970703 220 / 1452 : pp = 246.19021606445312 230 / 1452 : pp = 246.70140075683594 240 / 1452 : pp = 246.3099822998047 250 / 1452 : pp = 245.1745147705078 260 / 1452 : pp = 244.17384338378906 270 / 1452 : pp = 242.57363891601562 280 / 1452 : pp = 242.8500213623047 290 / 1452 : pp = 243.0492706298828 300 / 1452 : pp = 243.1466522216797 310 / 1452 : pp = 242.89044189453125 320 / 1452 : pp = 243.08045959472656 330 / 1452 : pp = 243.32235717773438 340 / 1452 : pp = 242.34715270996094 350 / 1452 : pp = 242.80972290039062 360 / 1452 : pp = 242.5345458984375 370 / 1452 : pp = 242.0083465576172 380 / 1452 : pp = 241.22708129882812 390 / 1452 : pp = 241.24398803710938 400 / 1452 : pp = 240.63473510742188 410 / 1452 : pp = 240.94094848632812 420 / 1452 : pp = 241.19717407226562 430 / 1452 : pp = 240.8896026611328 440 / 1452 : pp = 240.7772979736328 450 / 1452 : pp = 240.45913696289062 460 / 1452 : pp = 240.06674194335938 470 / 1452 : pp = 239.42198181152344 480 / 1452 : pp = 238.39271545410156 490 / 1452 : pp = 238.0517120361328 500 / 1452 : pp = 237.31752014160156 510 / 1452 : pp = 237.1197967529297 520 / 1452 : pp = 236.64865112304688 530 / 1452 : pp = 236.004638671875 540 / 1452 : pp = 235.192626953125 550 / 1452 : pp = 234.6700439453125 560 / 1452 : pp = 234.1914825439453 570 / 1452 : pp = 233.80899047851562 580 / 1452 : pp = 233.3753662109375 590 / 1452 : pp = 232.8699188232422 600 / 1452 : pp = 232.2629852294922 610 / 1452 : pp = 231.8668212890625 620 / 1452 : pp = 231.478515625 630 / 1452 : pp = 231.0444793701172 640 / 1452 : pp = 231.2737579345703 650 / 1452 : pp = 231.28114318847656 660 / 1452 : pp = 231.4324951171875 670 / 1452 : pp = 231.48513793945312 680 / 1452 : pp = 231.45932006835938 690 / 1452 : pp = 231.17738342285156 700 / 1452 : pp = 231.00570678710938 710 / 1452 : pp = 231.03810119628906 720 / 1452 : pp = 230.96131896972656 730 / 1452 : pp = 230.91110229492188 740 / 1452 : pp = 231.13539123535156 750 / 1452 : pp = 231.04393005371094 760 / 1452 : pp = 231.03489685058594 770 / 1452 : pp = 231.19744873046875 780 / 1452 : pp = 231.26625061035156 790 / 1452 : pp = 231.38714599609375 800 / 1452 : pp = 231.24441528320312 810 / 1452 : pp = 231.16824340820312 820 / 1452 : pp = 231.11831665039062 830 / 1452 : pp = 231.34886169433594 840 / 1452 : pp = 231.221923828125 850 / 1452 : pp = 231.2562255859375 860 / 1452 : pp = 231.26492309570312 870 / 1452 : pp = 231.1961212158203 880 / 1452 : pp = 231.30506896972656 890 / 1452 : pp = 231.24728393554688 900 / 1452 : pp = 231.15744018554688 910 / 1452 : pp = 231.20175170898438 920 / 1452 : pp = 231.25534057617188 930 / 1452 : pp = 231.09461975097656 940 / 1452 : pp = 231.12612915039062 950 / 1452 : pp = 231.0475616455078 960 / 1452 : pp = 230.86056518554688 970 / 1452 : pp = 230.80377197265625 980 / 1452 : pp = 230.4598846435547 990 / 1452 : pp = 230.24559020996094 1000 / 1452 : pp = 229.91030883789062 1010 / 1452 : pp = 229.9349822998047 1020 / 1452 : pp = 230.01470947265625 1030 / 1452 : pp = 229.8909149169922 1040 / 1452 : pp = 229.9403533935547 1050 / 1452 : pp = 229.84815979003906 1060 / 1452 : pp = 229.60377502441406 1070 / 1452 : pp = 229.74647521972656 1080 / 1452 : pp = 229.80410766601562 1090 / 1452 : pp = 229.78733825683594 1100 / 1452 : pp = 229.64549255371094 1110 / 1452 : pp = 229.26255798339844 1120 / 1452 : pp = 229.00262451171875 1130 / 1452 : pp = 228.6716766357422 1140 / 1452 : pp = 228.55067443847656 1150 / 1452 : pp = 228.61563110351562 1160 / 1452 : pp = 228.50958251953125 1170 / 1452 : pp = 228.3498992919922 1180 / 1452 : pp = 228.29786682128906 1190 / 1452 : pp = 228.33204650878906 1200 / 1452 : pp = 228.27369689941406 1210 / 1452 : pp = 228.11831665039062 1220 / 1452 : pp = 228.21775817871094 1230 / 1452 : pp = 228.3170166015625 1240 / 1452 : pp = 228.22134399414062 1250 / 1452 : pp = 228.3769073486328 1260 / 1452 : pp = 228.37527465820312 1270 / 1452 : pp = 228.33694458007812 1280 / 1452 : pp = 228.27108764648438 1290 / 1452 : pp = 228.1731414794922 1300 / 1452 : pp = 228.12200927734375 1310 / 1452 : pp = 228.10275268554688 1320 / 1452 : pp = 227.9289093017578 1330 / 1452 : pp = 227.77723693847656 1340 / 1452 : pp = 227.79623413085938 1350 / 1452 : pp = 227.7408447265625 1360 / 1452 : pp = 227.72586059570312 1370 / 1452 : pp = 227.49728393554688 1380 / 1452 : pp = 227.37940979003906 1390 / 1452 : pp = 227.20166015625 1400 / 1452 : pp = 227.018310546875 1410 / 1452 : pp = 226.95651245117188 1420 / 1452 : pp = 226.8065643310547 1430 / 1452 : pp = 226.7261199951172 1440 / 1452 : pp = 226.7193145751953 1450 / 1452 : pp = 226.61068725585938 0 / 115 : pp = 269.342041015625 10 / 115 : pp = 255.03016662597656 20 / 115 : pp = 253.8992919921875 30 / 115 : pp = 251.04025268554688 40 / 115 : pp = 250.51756286621094 50 / 115 : pp = 245.3595428466797 60 / 115 : pp = 244.4713897705078 70 / 115 : pp = 241.2674560546875 80 / 115 : pp = 238.3473663330078 90 / 115 : pp = 235.56423950195312 100 / 115 : pp = 231.2281036376953 110 / 115 : pp = 229.8423614501953 Training perplexity: 226.5760040283203 Validation perplexity:229.59939575195312 Total time : 42.202677726745605 Epoch 4 0 / 1452 : pp = 282.2423095703125 10 / 1452 : pp = 240.16258239746094 20 / 1452 : pp = 236.12203979492188 30 / 1452 : pp = 230.3953857421875 40 / 1452 : pp = 231.8789825439453 50 / 1452 : pp = 227.26612854003906 60 / 1452 : pp = 226.22061157226562 70 / 1452 : pp = 227.01885986328125 80 / 1452 : pp = 226.2459716796875 90 / 1452 : pp = 224.3211669921875 100 / 1452 : pp = 222.65615844726562 110 / 1452 : pp = 220.70326232910156 120 / 1452 : pp = 220.42288208007812 130 / 1452 : pp = 219.8100128173828 140 / 1452 : pp = 218.04432678222656 150 / 1452 : pp = 217.31639099121094 160 / 1452 : pp = 217.86349487304688 170 / 1452 : pp = 217.46597290039062 180 / 1452 : pp = 216.3349151611328 190 / 1452 : pp = 216.12240600585938 200 / 1452 : pp = 216.97842407226562 210 / 1452 : pp = 216.51014709472656 220 / 1452 : pp = 216.46751403808594 230 / 1452 : pp = 216.80126953125 240 / 1452 : pp = 216.45965576171875 250 / 1452 : pp = 215.5008544921875 260 / 1452 : pp = 214.62210083007812 270 / 1452 : pp = 213.29183959960938 280 / 1452 : pp = 213.5621337890625 290 / 1452 : pp = 213.80657958984375 300 / 1452 : pp = 213.8963165283203 310 / 1452 : pp = 213.60653686523438 320 / 1452 : pp = 213.85877990722656 330 / 1452 : pp = 214.07345581054688 340 / 1452 : pp = 213.25421142578125 350 / 1452 : pp = 213.68019104003906 360 / 1452 : pp = 213.41717529296875 370 / 1452 : pp = 213.04920959472656 380 / 1452 : pp = 212.39019775390625 390 / 1452 : pp = 212.4908905029297 400 / 1452 : pp = 212.01914978027344 410 / 1452 : pp = 212.36903381347656 420 / 1452 : pp = 212.6802520751953 430 / 1452 : pp = 212.42697143554688 440 / 1452 : pp = 212.42990112304688 450 / 1452 : pp = 212.14524841308594 460 / 1452 : pp = 211.7836151123047 470 / 1452 : pp = 211.17282104492188 480 / 1452 : pp = 210.27903747558594 490 / 1452 : pp = 209.95211791992188 500 / 1452 : pp = 209.28302001953125 510 / 1452 : pp = 209.1029815673828 520 / 1452 : pp = 208.73855590820312 530 / 1452 : pp = 208.19700622558594 540 / 1452 : pp = 207.4554443359375 550 / 1452 : pp = 207.0062255859375 560 / 1452 : pp = 206.59739685058594 570 / 1452 : pp = 206.27874755859375 580 / 1452 : pp = 205.87144470214844 590 / 1452 : pp = 205.43545532226562 600 / 1452 : pp = 204.90940856933594 610 / 1452 : pp = 204.5686798095703 620 / 1452 : pp = 204.22862243652344 630 / 1452 : pp = 203.8448028564453 640 / 1452 : pp = 204.06576538085938 650 / 1452 : pp = 204.0941925048828 660 / 1452 : pp = 204.22103881835938 670 / 1452 : pp = 204.289794921875 680 / 1452 : pp = 204.3115234375 690 / 1452 : pp = 204.10284423828125 700 / 1452 : pp = 203.99757385253906 710 / 1452 : pp = 204.04971313476562 720 / 1452 : pp = 204.03152465820312 730 / 1452 : pp = 203.99046325683594 740 / 1452 : pp = 204.19786071777344 750 / 1452 : pp = 204.1642608642578 760 / 1452 : pp = 204.19435119628906 770 / 1452 : pp = 204.37786865234375 780 / 1452 : pp = 204.4965057373047 790 / 1452 : pp = 204.6479034423828 800 / 1452 : pp = 204.56117248535156 810 / 1452 : pp = 204.52284240722656 820 / 1452 : pp = 204.50978088378906 830 / 1452 : pp = 204.7531280517578 840 / 1452 : pp = 204.64468383789062 850 / 1452 : pp = 204.71348571777344 860 / 1452 : pp = 204.7399444580078 870 / 1452 : pp = 204.69406127929688 880 / 1452 : pp = 204.7965850830078 890 / 1452 : pp = 204.7594757080078 900 / 1452 : pp = 204.71446228027344 910 / 1452 : pp = 204.7590789794922 920 / 1452 : pp = 204.85772705078125 930 / 1452 : pp = 204.7428741455078 940 / 1452 : pp = 204.8068389892578 950 / 1452 : pp = 204.75791931152344 960 / 1452 : pp = 204.63815307617188 970 / 1452 : pp = 204.60760498046875 980 / 1452 : pp = 204.34347534179688 990 / 1452 : pp = 204.151611328125 1000 / 1452 : pp = 203.8665771484375 1010 / 1452 : pp = 203.9164581298828 1020 / 1452 : pp = 204.0184783935547 1030 / 1452 : pp = 203.95166015625 1040 / 1452 : pp = 204.03045654296875 1050 / 1452 : pp = 203.95846557617188 1060 / 1452 : pp = 203.77114868164062 1070 / 1452 : pp = 203.93260192871094 1080 / 1452 : pp = 204.00048828125 1090 / 1452 : pp = 204.00233459472656 1100 / 1452 : pp = 203.8960418701172 1110 / 1452 : pp = 203.5987548828125 1120 / 1452 : pp = 203.38392639160156 1130 / 1452 : pp = 203.08872985839844 1140 / 1452 : pp = 203.01272583007812 1150 / 1452 : pp = 203.0865936279297 1160 / 1452 : pp = 203.02308654785156 1170 / 1452 : pp = 202.9125518798828 1180 / 1452 : pp = 202.9097442626953 1190 / 1452 : pp = 202.98252868652344 1200 / 1452 : pp = 202.95387268066406 1210 / 1452 : pp = 202.851318359375 1220 / 1452 : pp = 202.97671508789062 1230 / 1452 : pp = 203.1051025390625 1240 / 1452 : pp = 203.0526123046875 1250 / 1452 : pp = 203.21417236328125 1260 / 1452 : pp = 203.23617553710938 1270 / 1452 : pp = 203.22802734375 1280 / 1452 : pp = 203.20846557617188 1290 / 1452 : pp = 203.15362548828125 1300 / 1452 : pp = 203.14315795898438 1310 / 1452 : pp = 203.15264892578125 1320 / 1452 : pp = 203.02801513671875 1330 / 1452 : pp = 202.92977905273438 1340 / 1452 : pp = 202.95484924316406 1350 / 1452 : pp = 202.9335479736328 1360 / 1452 : pp = 202.955322265625 1370 / 1452 : pp = 202.7740478515625 1380 / 1452 : pp = 202.68569946289062 1390 / 1452 : pp = 202.55816650390625 1400 / 1452 : pp = 202.41651916503906 1410 / 1452 : pp = 202.38494873046875 1420 / 1452 : pp = 202.27593994140625 1430 / 1452 : pp = 202.21826171875 1440 / 1452 : pp = 202.23272705078125 1450 / 1452 : pp = 202.16099548339844 0 / 115 : pp = 253.23211669921875 10 / 115 : pp = 237.62506103515625 20 / 115 : pp = 237.60557556152344 30 / 115 : pp = 234.9273223876953 40 / 115 : pp = 234.30519104003906 50 / 115 : pp = 229.43960571289062 60 / 115 : pp = 228.6050567626953 70 / 115 : pp = 225.2646484375 80 / 115 : pp = 222.55935668945312 90 / 115 : pp = 219.83255004882812 100 / 115 : pp = 215.5491485595703 110 / 115 : pp = 214.07937622070312 Training perplexity: 202.1349639892578 Validation perplexity:213.85256958007812 Total time : 42.10724234580994 Epoch 5 0 / 1452 : pp = 255.92384338378906 10 / 1452 : pp = 219.5322265625 20 / 1452 : pp = 214.36212158203125 30 / 1452 : pp = 209.12620544433594 40 / 1452 : pp = 210.04193115234375 50 / 1452 : pp = 205.77398681640625 60 / 1452 : pp = 204.8201141357422 70 / 1452 : pp = 205.3955841064453 80 / 1452 : pp = 204.8386688232422 90 / 1452 : pp = 203.21194458007812 100 / 1452 : pp = 201.87643432617188 110 / 1452 : pp = 200.10122680664062 120 / 1452 : pp = 199.82012939453125 130 / 1452 : pp = 199.11192321777344 140 / 1452 : pp = 197.51919555664062 150 / 1452 : pp = 197.03567504882812 160 / 1452 : pp = 197.4231414794922 170 / 1452 : pp = 197.09571838378906 180 / 1452 : pp = 196.17665100097656 190 / 1452 : pp = 196.0064697265625 200 / 1452 : pp = 196.7347869873047 210 / 1452 : pp = 196.3063507080078 220 / 1452 : pp = 196.21388244628906 230 / 1452 : pp = 196.5252227783203 240 / 1452 : pp = 196.203125 250 / 1452 : pp = 195.3251953125 260 / 1452 : pp = 194.53335571289062 270 / 1452 : pp = 193.3546142578125 280 / 1452 : pp = 193.59420776367188 290 / 1452 : pp = 193.83297729492188 300 / 1452 : pp = 193.98489379882812 310 / 1452 : pp = 193.68414306640625 320 / 1452 : pp = 193.89065551757812 330 / 1452 : pp = 194.0518798828125 340 / 1452 : pp = 193.32888793945312 350 / 1452 : pp = 193.76219177246094 360 / 1452 : pp = 193.56106567382812 370 / 1452 : pp = 193.28179931640625 380 / 1452 : pp = 192.7037811279297 390 / 1452 : pp = 192.8145294189453 400 / 1452 : pp = 192.43325805664062 410 / 1452 : pp = 192.81527709960938 420 / 1452 : pp = 193.13760375976562 430 / 1452 : pp = 192.9148712158203 440 / 1452 : pp = 192.92526245117188 450 / 1452 : pp = 192.70083618164062 460 / 1452 : pp = 192.36647033691406 470 / 1452 : pp = 191.85394287109375 480 / 1452 : pp = 191.07244873046875 490 / 1452 : pp = 190.75401306152344 500 / 1452 : pp = 190.1843719482422 510 / 1452 : pp = 190.03334045410156 520 / 1452 : pp = 189.72938537597656 530 / 1452 : pp = 189.25889587402344 540 / 1452 : pp = 188.59315490722656 550 / 1452 : pp = 188.19313049316406 560 / 1452 : pp = 187.80621337890625 570 / 1452 : pp = 187.5229034423828 580 / 1452 : pp = 187.1091766357422 590 / 1452 : pp = 186.72592163085938 600 / 1452 : pp = 186.2238006591797 610 / 1452 : pp = 185.89695739746094 620 / 1452 : pp = 185.60989379882812 630 / 1452 : pp = 185.2689208984375 640 / 1452 : pp = 185.47567749023438 650 / 1452 : pp = 185.5127410888672 660 / 1452 : pp = 185.64627075195312 670 / 1452 : pp = 185.71311950683594 680 / 1452 : pp = 185.72569274902344 690 / 1452 : pp = 185.56459045410156 700 / 1452 : pp = 185.48681640625 710 / 1452 : pp = 185.5458221435547 720 / 1452 : pp = 185.5598907470703 730 / 1452 : pp = 185.5335235595703 740 / 1452 : pp = 185.73995971679688 750 / 1452 : pp = 185.744384765625 760 / 1452 : pp = 185.81268310546875 770 / 1452 : pp = 186.00088500976562 780 / 1452 : pp = 186.14443969726562 790 / 1452 : pp = 186.30764770507812 800 / 1452 : pp = 186.2595977783203 810 / 1452 : pp = 186.23028564453125 820 / 1452 : pp = 186.23997497558594 830 / 1452 : pp = 186.49057006835938 840 / 1452 : pp = 186.43331909179688 850 / 1452 : pp = 186.48887634277344 860 / 1452 : pp = 186.51502990722656 870 / 1452 : pp = 186.5167999267578 880 / 1452 : pp = 186.62400817871094 890 / 1452 : pp = 186.6103973388672 900 / 1452 : pp = 186.58111572265625 910 / 1452 : pp = 186.64126586914062 920 / 1452 : pp = 186.7366180419922 930 / 1452 : pp = 186.65719604492188 940 / 1452 : pp = 186.71755981445312 950 / 1452 : pp = 186.6977996826172 960 / 1452 : pp = 186.62774658203125 970 / 1452 : pp = 186.62115478515625 980 / 1452 : pp = 186.3773193359375 990 / 1452 : pp = 186.23109436035156 1000 / 1452 : pp = 185.99227905273438 1010 / 1452 : pp = 186.0488739013672 1020 / 1452 : pp = 186.1744384765625 1030 / 1452 : pp = 186.1162109375 1040 / 1452 : pp = 186.18899536132812 1050 / 1452 : pp = 186.1549072265625 1060 / 1452 : pp = 186.01419067382812 1070 / 1452 : pp = 186.17364501953125 1080 / 1452 : pp = 186.27061462402344 1090 / 1452 : pp = 186.28428649902344 1100 / 1452 : pp = 186.2150115966797 1110 / 1452 : pp = 185.95103454589844 1120 / 1452 : pp = 185.77423095703125 1130 / 1452 : pp = 185.5232696533203 1140 / 1452 : pp = 185.4607391357422 1150 / 1452 : pp = 185.56077575683594 1160 / 1452 : pp = 185.53343200683594 1170 / 1452 : pp = 185.46453857421875 1180 / 1452 : pp = 185.4741668701172 1190 / 1452 : pp = 185.5594482421875 1200 / 1452 : pp = 185.53785705566406 1210 / 1452 : pp = 185.4576416015625 1220 / 1452 : pp = 185.5943145751953 1230 / 1452 : pp = 185.7483673095703 1240 / 1452 : pp = 185.70762634277344 1250 / 1452 : pp = 185.8568115234375 1260 / 1452 : pp = 185.90635681152344 1270 / 1452 : pp = 185.8961639404297 1280 / 1452 : pp = 185.89199829101562 1290 / 1452 : pp = 185.85911560058594 1300 / 1452 : pp = 185.86097717285156 1310 / 1452 : pp = 185.88739013671875 1320 / 1452 : pp = 185.79248046875 1330 / 1452 : pp = 185.69700622558594 1340 / 1452 : pp = 185.7310028076172 1350 / 1452 : pp = 185.72613525390625 1360 / 1452 : pp = 185.76829528808594 1370 / 1452 : pp = 185.6322021484375 1380 / 1452 : pp = 185.56378173828125 1390 / 1452 : pp = 185.4654998779297 1400 / 1452 : pp = 185.35110473632812 1410 / 1452 : pp = 185.33917236328125 1420 / 1452 : pp = 185.2509002685547 1430 / 1452 : pp = 185.20436096191406 1440 / 1452 : pp = 185.2254638671875 1450 / 1452 : pp = 185.16542053222656 0 / 115 : pp = 242.26800537109375 10 / 115 : pp = 226.12258911132812 20 / 115 : pp = 226.4702606201172 30 / 115 : pp = 223.982666015625 40 / 115 : pp = 223.376953125 50 / 115 : pp = 218.65716552734375 60 / 115 : pp = 217.95306396484375 70 / 115 : pp = 214.5392303466797 80 / 115 : pp = 212.07525634765625 90 / 115 : pp = 209.40631103515625 100 / 115 : pp = 205.1455078125 110 / 115 : pp = 203.6289520263672 Training perplexity: 185.14476013183594 Validation perplexity:203.3822784423828 Total time : 42.47052240371704 Epoch 6 0 / 1452 : pp = 233.56707763671875 10 / 1452 : pp = 202.6468505859375 20 / 1452 : pp = 198.2734375 30 / 1452 : pp = 193.47442626953125 40 / 1452 : pp = 195.17147827148438 50 / 1452 : pp = 191.5596923828125 60 / 1452 : pp = 190.4825897216797 70 / 1452 : pp = 191.07681274414062 80 / 1452 : pp = 190.339599609375 90 / 1452 : pp = 188.98277282714844 100 / 1452 : pp = 187.74757385253906 110 / 1452 : pp = 186.10104370117188 120 / 1452 : pp = 185.7500457763672 130 / 1452 : pp = 184.90707397460938 140 / 1452 : pp = 183.340087890625 150 / 1452 : pp = 182.70840454101562 160 / 1452 : pp = 183.1043701171875 170 / 1452 : pp = 182.69776916503906 180 / 1452 : pp = 181.88400268554688 190 / 1452 : pp = 181.8062286376953 200 / 1452 : pp = 182.4969940185547 210 / 1452 : pp = 182.10572814941406 220 / 1452 : pp = 181.9981689453125 230 / 1452 : pp = 182.3802490234375 240 / 1452 : pp = 182.03636169433594 250 / 1452 : pp = 181.23712158203125 260 / 1452 : pp = 180.53726196289062 270 / 1452 : pp = 179.53567504882812 280 / 1452 : pp = 179.70208740234375 290 / 1452 : pp = 179.977783203125 300 / 1452 : pp = 180.16600036621094 310 / 1452 : pp = 179.87294006347656 320 / 1452 : pp = 180.11849975585938 330 / 1452 : pp = 180.31838989257812 340 / 1452 : pp = 179.56759643554688 350 / 1452 : pp = 179.97134399414062 360 / 1452 : pp = 179.80030822753906 370 / 1452 : pp = 179.52085876464844 380 / 1452 : pp = 178.98228454589844 390 / 1452 : pp = 179.0868682861328 400 / 1452 : pp = 178.74569702148438 410 / 1452 : pp = 179.1776580810547 420 / 1452 : pp = 179.5055389404297 430 / 1452 : pp = 179.3883056640625 440 / 1452 : pp = 179.42279052734375 450 / 1452 : pp = 179.2106475830078 460 / 1452 : pp = 178.85311889648438 470 / 1452 : pp = 178.33840942382812 480 / 1452 : pp = 177.60350036621094 490 / 1452 : pp = 177.30335998535156 500 / 1452 : pp = 176.72222900390625 510 / 1452 : pp = 176.6067352294922 520 / 1452 : pp = 176.33998107910156 530 / 1452 : pp = 175.93162536621094 540 / 1452 : pp = 175.30657958984375 550 / 1452 : pp = 174.9462432861328 560 / 1452 : pp = 174.5836639404297 570 / 1452 : pp = 174.31431579589844 580 / 1452 : pp = 173.92300415039062 590 / 1452 : pp = 173.55856323242188 600 / 1452 : pp = 173.08277893066406 610 / 1452 : pp = 172.75930786132812 620 / 1452 : pp = 172.53192138671875 630 / 1452 : pp = 172.20652770996094 640 / 1452 : pp = 172.37454223632812 650 / 1452 : pp = 172.39845275878906 660 / 1452 : pp = 172.52255249023438 670 / 1452 : pp = 172.60935974121094 680 / 1452 : pp = 172.6611328125 690 / 1452 : pp = 172.53118896484375 700 / 1452 : pp = 172.4709014892578 710 / 1452 : pp = 172.5406494140625 720 / 1452 : pp = 172.55447387695312 730 / 1452 : pp = 172.5330047607422 740 / 1452 : pp = 172.7061767578125 750 / 1452 : pp = 172.71054077148438 760 / 1452 : pp = 172.77743530273438 770 / 1452 : pp = 172.95481872558594 780 / 1452 : pp = 173.11265563964844 790 / 1452 : pp = 173.2832794189453 800 / 1452 : pp = 173.2537841796875 810 / 1452 : pp = 173.22164916992188 820 / 1452 : pp = 173.24148559570312 830 / 1452 : pp = 173.48228454589844 840 / 1452 : pp = 173.43753051757812 850 / 1452 : pp = 173.505615234375 860 / 1452 : pp = 173.5214080810547 870 / 1452 : pp = 173.5009002685547 880 / 1452 : pp = 173.6202392578125 890 / 1452 : pp = 173.622802734375 900 / 1452 : pp = 173.5987091064453 910 / 1452 : pp = 173.68316650390625 920 / 1452 : pp = 173.77330017089844 930 / 1452 : pp = 173.72018432617188 940 / 1452 : pp = 173.79351806640625 950 / 1452 : pp = 173.7653350830078 960 / 1452 : pp = 173.7102508544922 970 / 1452 : pp = 173.69766235351562 980 / 1452 : pp = 173.4836883544922 990 / 1452 : pp = 173.3550262451172 1000 / 1452 : pp = 173.14816284179688 1010 / 1452 : pp = 173.20777893066406 1020 / 1452 : pp = 173.3390655517578 1030 / 1452 : pp = 173.2884063720703 1040 / 1452 : pp = 173.38015747070312 1050 / 1452 : pp = 173.35592651367188 1060 / 1452 : pp = 173.2260284423828 1070 / 1452 : pp = 173.39321899414062 1080 / 1452 : pp = 173.4879913330078 1090 / 1452 : pp = 173.5231475830078 1100 / 1452 : pp = 173.47177124023438 1110 / 1452 : pp = 173.24453735351562 1120 / 1452 : pp = 173.09408569335938 1130 / 1452 : pp = 172.86627197265625 1140 / 1452 : pp = 172.8234100341797 1150 / 1452 : pp = 172.92843627929688 1160 / 1452 : pp = 172.90065002441406 1170 / 1452 : pp = 172.8550567626953 1180 / 1452 : pp = 172.8810272216797 1190 / 1452 : pp = 172.97312927246094 1200 / 1452 : pp = 172.9776611328125 1210 / 1452 : pp = 172.89413452148438 1220 / 1452 : pp = 173.0257568359375 1230 / 1452 : pp = 173.1847381591797 1240 / 1452 : pp = 173.1756591796875 1250 / 1452 : pp = 173.32138061523438 1260 / 1452 : pp = 173.37229919433594 1270 / 1452 : pp = 173.36891174316406 1280 / 1452 : pp = 173.36337280273438 1290 / 1452 : pp = 173.3444366455078 1300 / 1452 : pp = 173.36138916015625 1310 / 1452 : pp = 173.4015655517578 1320 / 1452 : pp = 173.31790161132812 1330 / 1452 : pp = 173.24710083007812 1340 / 1452 : pp = 173.27212524414062 1350 / 1452 : pp = 173.27674865722656 1360 / 1452 : pp = 173.32749938964844 1370 / 1452 : pp = 173.20472717285156 1380 / 1452 : pp = 173.14889526367188 1390 / 1452 : pp = 173.0755157470703 1400 / 1452 : pp = 172.9678497314453 1410 / 1452 : pp = 172.9612579345703 1420 / 1452 : pp = 172.8872833251953 1430 / 1452 : pp = 172.84805297851562 1440 / 1452 : pp = 172.87252807617188 1450 / 1452 : pp = 172.82505798339844 0 / 115 : pp = 236.35635375976562 10 / 115 : pp = 219.06166076660156 20 / 115 : pp = 219.7670440673828 30 / 115 : pp = 217.33587646484375 40 / 115 : pp = 216.6626739501953 50 / 115 : pp = 212.04734802246094 60 / 115 : pp = 211.42068481445312 70 / 115 : pp = 207.9592742919922 80 / 115 : pp = 205.6216583251953 90 / 115 : pp = 202.93597412109375 100 / 115 : pp = 198.62583923339844 110 / 115 : pp = 196.97216796875 Training perplexity: 172.80404663085938 Validation perplexity:196.6871337890625 Total time : 41.52522921562195 Epoch 7 0 / 1452 : pp = 219.23231506347656 10 / 1452 : pp = 192.07225036621094 20 / 1452 : pp = 187.48464965820312 30 / 1452 : pp = 182.9149932861328 40 / 1452 : pp = 184.2945098876953 50 / 1452 : pp = 180.78492736816406 60 / 1452 : pp = 179.377197265625 70 / 1452 : pp = 180.0273895263672 80 / 1452 : pp = 179.2517547607422 90 / 1452 : pp = 177.77540588378906 100 / 1452 : pp = 176.6474151611328 110 / 1452 : pp = 174.84066772460938 120 / 1452 : pp = 174.46890258789062 130 / 1452 : pp = 173.64573669433594 140 / 1452 : pp = 172.17483520507812 150 / 1452 : pp = 171.57041931152344 160 / 1452 : pp = 171.92059326171875 170 / 1452 : pp = 171.5497283935547 180 / 1452 : pp = 170.77249145507812 190 / 1452 : pp = 170.72103881835938 200 / 1452 : pp = 171.336181640625 210 / 1452 : pp = 170.98524475097656 220 / 1452 : pp = 170.99771118164062 230 / 1452 : pp = 171.39918518066406 240 / 1452 : pp = 171.09925842285156 250 / 1452 : pp = 170.39962768554688 260 / 1452 : pp = 169.7328643798828 270 / 1452 : pp = 168.72225952148438 280 / 1452 : pp = 168.92552185058594 290 / 1452 : pp = 169.20147705078125 300 / 1452 : pp = 169.40338134765625 310 / 1452 : pp = 169.12057495117188 320 / 1452 : pp = 169.31236267089844 330 / 1452 : pp = 169.49945068359375 340 / 1452 : pp = 168.8396759033203 350 / 1452 : pp = 169.25917053222656 360 / 1452 : pp = 169.09388732910156 370 / 1452 : pp = 168.84323120117188 380 / 1452 : pp = 168.3832550048828 390 / 1452 : pp = 168.48275756835938 400 / 1452 : pp = 168.19972229003906 410 / 1452 : pp = 168.5838623046875 420 / 1452 : pp = 168.91119384765625 430 / 1452 : pp = 168.80836486816406 440 / 1452 : pp = 168.90264892578125 450 / 1452 : pp = 168.68589782714844 460 / 1452 : pp = 168.3704071044922 470 / 1452 : pp = 167.90394592285156 480 / 1452 : pp = 167.23373413085938 490 / 1452 : pp = 166.9560546875 500 / 1452 : pp = 166.43161010742188 510 / 1452 : pp = 166.320068359375 520 / 1452 : pp = 166.05902099609375 530 / 1452 : pp = 165.71714782714844 540 / 1452 : pp = 165.10398864746094 550 / 1452 : pp = 164.80430603027344 560 / 1452 : pp = 164.4687042236328 570 / 1452 : pp = 164.2272491455078 580 / 1452 : pp = 163.84312438964844 590 / 1452 : pp = 163.46035766601562 600 / 1452 : pp = 163.01559448242188 610 / 1452 : pp = 162.74134826660156 620 / 1452 : pp = 162.50267028808594 630 / 1452 : pp = 162.2018280029297 640 / 1452 : pp = 162.37130737304688 650 / 1452 : pp = 162.3895721435547 660 / 1452 : pp = 162.51351928710938 670 / 1452 : pp = 162.57684326171875 680 / 1452 : pp = 162.6346893310547 690 / 1452 : pp = 162.5135955810547 700 / 1452 : pp = 162.47052001953125 710 / 1452 : pp = 162.539794921875 720 / 1452 : pp = 162.55381774902344 730 / 1452 : pp = 162.5297088623047 740 / 1452 : pp = 162.71652221679688 750 / 1452 : pp = 162.740966796875 760 / 1452 : pp = 162.79754638671875 770 / 1452 : pp = 162.9949951171875 780 / 1452 : pp = 163.17868041992188 790 / 1452 : pp = 163.33055114746094 800 / 1452 : pp = 163.31591796875 810 / 1452 : pp = 163.2859344482422 820 / 1452 : pp = 163.2958984375 830 / 1452 : pp = 163.528564453125 840 / 1452 : pp = 163.47610473632812 850 / 1452 : pp = 163.5260772705078 860 / 1452 : pp = 163.55352783203125 870 / 1452 : pp = 163.55718994140625 880 / 1452 : pp = 163.67523193359375 890 / 1452 : pp = 163.6920166015625 900 / 1452 : pp = 163.67710876464844 910 / 1452 : pp = 163.7476806640625 920 / 1452 : pp = 163.84803771972656 930 / 1452 : pp = 163.8114013671875 940 / 1452 : pp = 163.86663818359375 950 / 1452 : pp = 163.83531188964844 960 / 1452 : pp = 163.79945373535156 970 / 1452 : pp = 163.80320739746094 980 / 1452 : pp = 163.5953369140625 990 / 1452 : pp = 163.48382568359375 1000 / 1452 : pp = 163.2642822265625 1010 / 1452 : pp = 163.32113647460938 1020 / 1452 : pp = 163.44204711914062 1030 / 1452 : pp = 163.40206909179688 1040 / 1452 : pp = 163.4915313720703 1050 / 1452 : pp = 163.47096252441406 1060 / 1452 : pp = 163.3601531982422 1070 / 1452 : pp = 163.5138397216797 1080 / 1452 : pp = 163.6189727783203 1090 / 1452 : pp = 163.6471405029297 1100 / 1452 : pp = 163.60406494140625 1110 / 1452 : pp = 163.40736389160156 1120 / 1452 : pp = 163.26841735839844 1130 / 1452 : pp = 163.0680694580078 1140 / 1452 : pp = 163.04591369628906 1150 / 1452 : pp = 163.15478515625 1160 / 1452 : pp = 163.1380615234375 1170 / 1452 : pp = 163.09303283691406 1180 / 1452 : pp = 163.14149475097656 1190 / 1452 : pp = 163.2374267578125 1200 / 1452 : pp = 163.2394561767578 1210 / 1452 : pp = 163.17835998535156 1220 / 1452 : pp = 163.32347106933594 1230 / 1452 : pp = 163.4639434814453 1240 / 1452 : pp = 163.4611358642578 1250 / 1452 : pp = 163.60687255859375 1260 / 1452 : pp = 163.67227172851562 1270 / 1452 : pp = 163.67515563964844 1280 / 1452 : pp = 163.6881103515625 1290 / 1452 : pp = 163.66648864746094 1300 / 1452 : pp = 163.69287109375 1310 / 1452 : pp = 163.7276153564453 1320 / 1452 : pp = 163.6551055908203 1330 / 1452 : pp = 163.58901977539062 1340 / 1452 : pp = 163.6205291748047 1350 / 1452 : pp = 163.63824462890625 1360 / 1452 : pp = 163.69334411621094 1370 / 1452 : pp = 163.5885467529297 1380 / 1452 : pp = 163.54049682617188 1390 / 1452 : pp = 163.4760284423828 1400 / 1452 : pp = 163.38897705078125 1410 / 1452 : pp = 163.3974609375 1420 / 1452 : pp = 163.35009765625 1430 / 1452 : pp = 163.32191467285156 1440 / 1452 : pp = 163.35220336914062 1450 / 1452 : pp = 163.3201904296875 0 / 115 : pp = 232.2108154296875 10 / 115 : pp = 214.35496520996094 20 / 115 : pp = 215.20510864257812 30 / 115 : pp = 212.82754516601562 40 / 115 : pp = 212.0598907470703 50 / 115 : pp = 207.5095672607422 60 / 115 : pp = 206.86976623535156 70 / 115 : pp = 203.36016845703125 80 / 115 : pp = 201.11538696289062 90 / 115 : pp = 198.52120971679688 100 / 115 : pp = 194.1772003173828 110 / 115 : pp = 192.41224670410156 Training perplexity: 163.29916381835938 Validation perplexity:192.09552001953125 Total time : 41.78096055984497 Epoch 8 0 / 1452 : pp = 201.77548217773438 10 / 1452 : pp = 180.4141082763672 20 / 1452 : pp = 176.41432189941406 30 / 1452 : pp = 172.7764434814453 40 / 1452 : pp = 174.69166564941406 50 / 1452 : pp = 171.2933807373047 60 / 1452 : pp = 170.08010864257812 70 / 1452 : pp = 170.6719512939453 80 / 1452 : pp = 170.07589721679688 90 / 1452 : pp = 168.7478485107422 100 / 1452 : pp = 167.57081604003906 110 / 1452 : pp = 166.06971740722656 120 / 1452 : pp = 165.73374938964844 130 / 1452 : pp = 164.80674743652344 140 / 1452 : pp = 163.32821655273438 150 / 1452 : pp = 162.6752471923828 160 / 1452 : pp = 163.02049255371094 170 / 1452 : pp = 162.64120483398438 180 / 1452 : pp = 161.95529174804688 190 / 1452 : pp = 161.91954040527344 200 / 1452 : pp = 162.5446014404297 210 / 1452 : pp = 162.2645721435547 220 / 1452 : pp = 162.3128662109375 230 / 1452 : pp = 162.65872192382812 240 / 1452 : pp = 162.40948486328125 250 / 1452 : pp = 161.75787353515625 260 / 1452 : pp = 161.15213012695312 270 / 1452 : pp = 160.22256469726562 280 / 1452 : pp = 160.3651123046875 290 / 1452 : pp = 160.63780212402344 300 / 1452 : pp = 160.80026245117188 310 / 1452 : pp = 160.54383850097656 320 / 1452 : pp = 160.7539520263672 330 / 1452 : pp = 160.94317626953125 340 / 1452 : pp = 160.3373565673828 350 / 1452 : pp = 160.71763610839844 360 / 1452 : pp = 160.60960388183594 370 / 1452 : pp = 160.37527465820312 380 / 1452 : pp = 159.92990112304688 390 / 1452 : pp = 160.0165557861328 400 / 1452 : pp = 159.75697326660156 410 / 1452 : pp = 160.15274047851562 420 / 1452 : pp = 160.48390197753906 430 / 1452 : pp = 160.4031982421875 440 / 1452 : pp = 160.4693603515625 450 / 1452 : pp = 160.28016662597656 460 / 1452 : pp = 159.94004821777344 470 / 1452 : pp = 159.48257446289062 480 / 1452 : pp = 158.87998962402344 490 / 1452 : pp = 158.59765625 500 / 1452 : pp = 158.10865783691406 510 / 1452 : pp = 157.96795654296875 520 / 1452 : pp = 157.7591552734375 530 / 1452 : pp = 157.42648315429688 540 / 1452 : pp = 156.85348510742188 550 / 1452 : pp = 156.5618438720703 560 / 1452 : pp = 156.24905395507812 570 / 1452 : pp = 155.9994354248047 580 / 1452 : pp = 155.612060546875 590 / 1452 : pp = 155.25830078125 600 / 1452 : pp = 154.8464813232422 610 / 1452 : pp = 154.5833282470703 620 / 1452 : pp = 154.38040161132812 630 / 1452 : pp = 154.0767364501953 640 / 1452 : pp = 154.2534637451172 650 / 1452 : pp = 154.25875854492188 660 / 1452 : pp = 154.35874938964844 670 / 1452 : pp = 154.4289093017578 680 / 1452 : pp = 154.51412963867188 690 / 1452 : pp = 154.41676330566406 700 / 1452 : pp = 154.37892150878906 710 / 1452 : pp = 154.4234619140625 720 / 1452 : pp = 154.4586639404297 730 / 1452 : pp = 154.4351806640625 740 / 1452 : pp = 154.6002197265625 750 / 1452 : pp = 154.65684509277344 760 / 1452 : pp = 154.73318481445312 770 / 1452 : pp = 154.92935180664062 780 / 1452 : pp = 155.1021728515625 790 / 1452 : pp = 155.24757385253906 800 / 1452 : pp = 155.223876953125 810 / 1452 : pp = 155.2095184326172 820 / 1452 : pp = 155.24009704589844 830 / 1452 : pp = 155.4519500732422 840 / 1452 : pp = 155.3947296142578 850 / 1452 : pp = 155.45306396484375 860 / 1452 : pp = 155.4661102294922 870 / 1452 : pp = 155.45765686035156 880 / 1452 : pp = 155.58758544921875 890 / 1452 : pp = 155.59373474121094 900 / 1452 : pp = 155.59254455566406 910 / 1452 : pp = 155.66854858398438 920 / 1452 : pp = 155.75942993164062 930 / 1452 : pp = 155.73350524902344 940 / 1452 : pp = 155.80740356445312 950 / 1452 : pp = 155.7733917236328 960 / 1452 : pp = 155.73565673828125 970 / 1452 : pp = 155.74404907226562 980 / 1452 : pp = 155.55902099609375 990 / 1452 : pp = 155.45675659179688 1000 / 1452 : pp = 155.2649688720703 1010 / 1452 : pp = 155.31332397460938 1020 / 1452 : pp = 155.44979858398438 1030 / 1452 : pp = 155.4137725830078 1040 / 1452 : pp = 155.49012756347656 1050 / 1452 : pp = 155.46054077148438 1060 / 1452 : pp = 155.3616943359375 1070 / 1452 : pp = 155.5286865234375 1080 / 1452 : pp = 155.63743591308594 1090 / 1452 : pp = 155.6842803955078 1100 / 1452 : pp = 155.65599060058594 1110 / 1452 : pp = 155.4827880859375 1120 / 1452 : pp = 155.35450744628906 1130 / 1452 : pp = 155.1777801513672 1140 / 1452 : pp = 155.15994262695312 1150 / 1452 : pp = 155.26193237304688 1160 / 1452 : pp = 155.26214599609375 1170 / 1452 : pp = 155.23231506347656 1180 / 1452 : pp = 155.29266357421875 1190 / 1452 : pp = 155.37680053710938 1200 / 1452 : pp = 155.3736114501953 1210 / 1452 : pp = 155.3380584716797 1220 / 1452 : pp = 155.474853515625 1230 / 1452 : pp = 155.62986755371094 1240 / 1452 : pp = 155.62831115722656 1250 / 1452 : pp = 155.77101135253906 1260 / 1452 : pp = 155.83445739746094 1270 / 1452 : pp = 155.845458984375 1280 / 1452 : pp = 155.8556365966797 1290 / 1452 : pp = 155.8556365966797 1300 / 1452 : pp = 155.8843994140625 1310 / 1452 : pp = 155.92417907714844 1320 / 1452 : pp = 155.8560791015625 1330 / 1452 : pp = 155.80636596679688 1340 / 1452 : pp = 155.84344482421875 1350 / 1452 : pp = 155.8706512451172 1360 / 1452 : pp = 155.9273681640625 1370 / 1452 : pp = 155.83140563964844 1380 / 1452 : pp = 155.7911376953125 1390 / 1452 : pp = 155.7401885986328 1400 / 1452 : pp = 155.6622314453125 1410 / 1452 : pp = 155.68531799316406 1420 / 1452 : pp = 155.64041137695312 1430 / 1452 : pp = 155.62216186523438 1440 / 1452 : pp = 155.6437530517578 1450 / 1452 : pp = 155.62757873535156 0 / 115 : pp = 228.70111083984375 10 / 115 : pp = 211.03330993652344 20 / 115 : pp = 212.24957275390625 30 / 115 : pp = 209.8839569091797 40 / 115 : pp = 209.11045837402344 50 / 115 : pp = 204.66351318359375 60 / 115 : pp = 204.03366088867188 70 / 115 : pp = 200.46681213378906 80 / 115 : pp = 198.24404907226562 90 / 115 : pp = 195.63223266601562 100 / 115 : pp = 191.18345642089844 110 / 115 : pp = 189.31134033203125 Training perplexity: 155.61154174804688 Validation perplexity:188.94537353515625 Total time : 42.13483738899231 Epoch 9 0 / 1452 : pp = 197.80628967285156 10 / 1452 : pp = 172.6316680908203 20 / 1452 : pp = 168.6739959716797 30 / 1452 : pp = 164.4781036376953 40 / 1452 : pp = 166.1627960205078 50 / 1452 : pp = 163.05197143554688 60 / 1452 : pp = 161.87924194335938 70 / 1452 : pp = 162.5297088623047 80 / 1452 : pp = 161.7450714111328 90 / 1452 : pp = 160.6148223876953 100 / 1452 : pp = 159.73289489746094 110 / 1452 : pp = 158.4092254638672 120 / 1452 : pp = 158.04653930664062 130 / 1452 : pp = 157.13563537597656 140 / 1452 : pp = 155.71798706054688 150 / 1452 : pp = 155.19161987304688 160 / 1452 : pp = 155.42718505859375 170 / 1452 : pp = 155.0531463623047 180 / 1452 : pp = 154.46897888183594 190 / 1452 : pp = 154.4127197265625 200 / 1452 : pp = 154.97154235839844 210 / 1452 : pp = 154.70169067382812 220 / 1452 : pp = 154.72816467285156 230 / 1452 : pp = 155.03799438476562 240 / 1452 : pp = 154.85601806640625 250 / 1452 : pp = 154.28016662597656 260 / 1452 : pp = 153.7699432373047 270 / 1452 : pp = 152.90948486328125 280 / 1452 : pp = 153.0459747314453 290 / 1452 : pp = 153.298095703125 300 / 1452 : pp = 153.45716857910156 310 / 1452 : pp = 153.22195434570312 320 / 1452 : pp = 153.41664123535156 330 / 1452 : pp = 153.66542053222656 340 / 1452 : pp = 153.06378173828125 350 / 1452 : pp = 153.43923950195312 360 / 1452 : pp = 153.31381225585938 370 / 1452 : pp = 153.13473510742188 380 / 1452 : pp = 152.75267028808594 390 / 1452 : pp = 152.85504150390625 400 / 1452 : pp = 152.62342834472656 410 / 1452 : pp = 153.03152465820312 420 / 1452 : pp = 153.39161682128906 430 / 1452 : pp = 153.30364990234375 440 / 1452 : pp = 153.37896728515625 450 / 1452 : pp = 153.18988037109375 460 / 1452 : pp = 152.88478088378906 470 / 1452 : pp = 152.4380340576172 480 / 1452 : pp = 151.86618041992188 490 / 1452 : pp = 151.5962371826172 500 / 1452 : pp = 151.11614990234375 510 / 1452 : pp = 150.99830627441406 520 / 1452 : pp = 150.8135986328125 530 / 1452 : pp = 150.500732421875 540 / 1452 : pp = 149.9623260498047 550 / 1452 : pp = 149.68028259277344 560 / 1452 : pp = 149.3885040283203 570 / 1452 : pp = 149.140380859375 580 / 1452 : pp = 148.76876831054688 590 / 1452 : pp = 148.43368530273438 600 / 1452 : pp = 148.02598571777344 610 / 1452 : pp = 147.7869110107422 620 / 1452 : pp = 147.59796142578125 630 / 1452 : pp = 147.30068969726562 640 / 1452 : pp = 147.45240783691406 650 / 1452 : pp = 147.4651336669922 660 / 1452 : pp = 147.5808563232422 670 / 1452 : pp = 147.65582275390625 680 / 1452 : pp = 147.7360382080078 690 / 1452 : pp = 147.63075256347656 700 / 1452 : pp = 147.6066131591797 710 / 1452 : pp = 147.7024383544922 720 / 1452 : pp = 147.7445526123047 730 / 1452 : pp = 147.72279357910156 740 / 1452 : pp = 147.87107849121094 750 / 1452 : pp = 147.91436767578125 760 / 1452 : pp = 147.9857635498047 770 / 1452 : pp = 148.18206787109375 780 / 1452 : pp = 148.3845672607422 790 / 1452 : pp = 148.5517120361328 800 / 1452 : pp = 148.54002380371094 810 / 1452 : pp = 148.51119995117188 820 / 1452 : pp = 148.5664520263672 830 / 1452 : pp = 148.7821044921875 840 / 1452 : pp = 148.72486877441406 850 / 1452 : pp = 148.77452087402344 860 / 1452 : pp = 148.80076599121094 870 / 1452 : pp = 148.79701232910156 880 / 1452 : pp = 148.9181671142578 890 / 1452 : pp = 148.94537353515625 900 / 1452 : pp = 148.9435272216797 910 / 1452 : pp = 149.02102661132812 920 / 1452 : pp = 149.1085968017578 930 / 1452 : pp = 149.06893920898438 940 / 1452 : pp = 149.1317138671875 950 / 1452 : pp = 149.1232452392578 960 / 1452 : pp = 149.10354614257812 970 / 1452 : pp = 149.11656188964844 980 / 1452 : pp = 148.94259643554688 990 / 1452 : pp = 148.8236846923828 1000 / 1452 : pp = 148.633056640625 1010 / 1452 : pp = 148.6830291748047 1020 / 1452 : pp = 148.8126220703125 1030 / 1452 : pp = 148.78089904785156 1040 / 1452 : pp = 148.8600311279297 1050 / 1452 : pp = 148.8486785888672 1060 / 1452 : pp = 148.7664337158203 1070 / 1452 : pp = 148.9337921142578 1080 / 1452 : pp = 149.04441833496094 1090 / 1452 : pp = 149.07284545898438 1100 / 1452 : pp = 149.03318786621094 1110 / 1452 : pp = 148.86428833007812 1120 / 1452 : pp = 148.7332305908203 1130 / 1452 : pp = 148.5670166015625 1140 / 1452 : pp = 148.54661560058594 1150 / 1452 : pp = 148.64219665527344 1160 / 1452 : pp = 148.6490020751953 1170 / 1452 : pp = 148.62420654296875 1180 / 1452 : pp = 148.67665100097656 1190 / 1452 : pp = 148.7633056640625 1200 / 1452 : pp = 148.7782745361328 1210 / 1452 : pp = 148.72500610351562 1220 / 1452 : pp = 148.87493896484375 1230 / 1452 : pp = 149.039794921875 1240 / 1452 : pp = 149.04000854492188 1250 / 1452 : pp = 149.17054748535156 1260 / 1452 : pp = 149.23863220214844 1270 / 1452 : pp = 149.2436065673828 1280 / 1452 : pp = 149.25086975097656 1290 / 1452 : pp = 149.24147033691406 1300 / 1452 : pp = 149.27413940429688 1310 / 1452 : pp = 149.32077026367188 1320 / 1452 : pp = 149.27301025390625 1330 / 1452 : pp = 149.23080444335938 1340 / 1452 : pp = 149.25791931152344 1350 / 1452 : pp = 149.2841033935547 1360 / 1452 : pp = 149.337158203125 1370 / 1452 : pp = 149.2467498779297 1380 / 1452 : pp = 149.21351623535156 1390 / 1452 : pp = 149.15403747558594 1400 / 1452 : pp = 149.0877685546875 1410 / 1452 : pp = 149.110595703125 1420 / 1452 : pp = 149.07241821289062 1430 / 1452 : pp = 149.05166625976562 1440 / 1452 : pp = 149.0776824951172 1450 / 1452 : pp = 149.06771850585938 0 / 115 : pp = 227.0559844970703 10 / 115 : pp = 208.7002410888672 20 / 115 : pp = 210.38775634765625 30 / 115 : pp = 207.9513397216797 40 / 115 : pp = 207.12994384765625 50 / 115 : pp = 202.70811462402344 60 / 115 : pp = 202.05787658691406 70 / 115 : pp = 198.3761444091797 80 / 115 : pp = 196.17637634277344 90 / 115 : pp = 193.5880126953125 100 / 115 : pp = 189.0758819580078 110 / 115 : pp = 187.07528686523438 Training perplexity: 149.0502471923828 Validation perplexity:186.6911163330078 Total time : 47.274805545806885 Epoch 10 0 / 1452 : pp = 181.8408203125 10 / 1452 : pp = 164.99664306640625 20 / 1452 : pp = 161.8847198486328 30 / 1452 : pp = 158.30064392089844 40 / 1452 : pp = 160.13914489746094 50 / 1452 : pp = 157.58743286132812 60 / 1452 : pp = 156.11871337890625 70 / 1452 : pp = 156.82948303222656 80 / 1452 : pp = 156.2889862060547 90 / 1452 : pp = 155.04833984375 100 / 1452 : pp = 154.09327697753906 110 / 1452 : pp = 152.5070343017578 120 / 1452 : pp = 152.20750427246094 130 / 1452 : pp = 151.3399200439453 140 / 1452 : pp = 149.90740966796875 150 / 1452 : pp = 149.345703125 160 / 1452 : pp = 149.59814453125 170 / 1452 : pp = 149.26539611816406 180 / 1452 : pp = 148.624267578125 190 / 1452 : pp = 148.58819580078125 200 / 1452 : pp = 149.09552001953125 210 / 1452 : pp = 148.8439178466797 220 / 1452 : pp = 148.86605834960938 230 / 1452 : pp = 149.1971435546875 240 / 1452 : pp = 148.96533203125 250 / 1452 : pp = 148.4253387451172 260 / 1452 : pp = 147.9200897216797 270 / 1452 : pp = 147.08816528320312 280 / 1452 : pp = 147.24366760253906 290 / 1452 : pp = 147.52182006835938 300 / 1452 : pp = 147.72222900390625 310 / 1452 : pp = 147.50486755371094 320 / 1452 : pp = 147.73892211914062 330 / 1452 : pp = 147.9404754638672 340 / 1452 : pp = 147.37803649902344 350 / 1452 : pp = 147.6969451904297 360 / 1452 : pp = 147.5704345703125 370 / 1452 : pp = 147.38674926757812 380 / 1452 : pp = 147.03970336914062 390 / 1452 : pp = 147.14231872558594 400 / 1452 : pp = 146.91656494140625 410 / 1452 : pp = 147.34059143066406 420 / 1452 : pp = 147.68496704101562 430 / 1452 : pp = 147.61195373535156 440 / 1452 : pp = 147.68405151367188 450 / 1452 : pp = 147.4711151123047 460 / 1452 : pp = 147.1927032470703 470 / 1452 : pp = 146.72970581054688 480 / 1452 : pp = 146.17173767089844 490 / 1452 : pp = 145.9028778076172 500 / 1452 : pp = 145.42721557617188 510 / 1452 : pp = 145.3111114501953 520 / 1452 : pp = 145.11460876464844 530 / 1452 : pp = 144.81488037109375 540 / 1452 : pp = 144.263916015625 550 / 1452 : pp = 143.997802734375 560 / 1452 : pp = 143.71766662597656 570 / 1452 : pp = 143.47451782226562 580 / 1452 : pp = 143.08474731445312 590 / 1452 : pp = 142.77920532226562 600 / 1452 : pp = 142.39573669433594 610 / 1452 : pp = 142.14906311035156 620 / 1452 : pp = 141.9574432373047 630 / 1452 : pp = 141.67369079589844 640 / 1452 : pp = 141.81556701660156 650 / 1452 : pp = 141.81759643554688 660 / 1452 : pp = 141.9339599609375 670 / 1452 : pp = 142.01248168945312 680 / 1452 : pp = 142.08773803710938 690 / 1452 : pp = 142.00328063964844 700 / 1452 : pp = 141.98086547851562 710 / 1452 : pp = 142.0632781982422 720 / 1452 : pp = 142.10372924804688 730 / 1452 : pp = 142.08055114746094 740 / 1452 : pp = 142.23619079589844 750 / 1452 : pp = 142.2660369873047 760 / 1452 : pp = 142.34678649902344 770 / 1452 : pp = 142.5257568359375 780 / 1452 : pp = 142.70025634765625 790 / 1452 : pp = 142.8614044189453 800 / 1452 : pp = 142.84573364257812 810 / 1452 : pp = 142.8250274658203 820 / 1452 : pp = 142.8540496826172 830 / 1452 : pp = 143.06053161621094 840 / 1452 : pp = 143.0423126220703 850 / 1452 : pp = 143.09634399414062 860 / 1452 : pp = 143.10487365722656 870 / 1452 : pp = 143.0884246826172 880 / 1452 : pp = 143.19387817382812 890 / 1452 : pp = 143.236083984375 900 / 1452 : pp = 143.23390197753906 910 / 1452 : pp = 143.29537963867188 920 / 1452 : pp = 143.3722686767578 930 / 1452 : pp = 143.33795166015625 940 / 1452 : pp = 143.40618896484375 950 / 1452 : pp = 143.3929901123047 960 / 1452 : pp = 143.3693389892578 970 / 1452 : pp = 143.39736938476562 980 / 1452 : pp = 143.2371063232422 990 / 1452 : pp = 143.13893127441406 1000 / 1452 : pp = 142.9658660888672 1010 / 1452 : pp = 143.01544189453125 1020 / 1452 : pp = 143.152587890625 1030 / 1452 : pp = 143.11334228515625 1040 / 1452 : pp = 143.19020080566406 1050 / 1452 : pp = 143.18234252929688 1060 / 1452 : pp = 143.092041015625 1070 / 1452 : pp = 143.24449157714844 1080 / 1452 : pp = 143.34828186035156 1090 / 1452 : pp = 143.38739013671875 1100 / 1452 : pp = 143.37432861328125 1110 / 1452 : pp = 143.20596313476562 1120 / 1452 : pp = 143.07969665527344 1130 / 1452 : pp = 142.92041015625 1140 / 1452 : pp = 142.90902709960938 1150 / 1452 : pp = 143.00732421875 1160 / 1452 : pp = 143.01182556152344 1170 / 1452 : pp = 142.9925994873047 1180 / 1452 : pp = 143.06080627441406 1190 / 1452 : pp = 143.14337158203125 1200 / 1452 : pp = 143.16644287109375 1210 / 1452 : pp = 143.1259002685547 1220 / 1452 : pp = 143.2671661376953 1230 / 1452 : pp = 143.4210968017578 1240 / 1452 : pp = 143.4327850341797 1250 / 1452 : pp = 143.5699920654297 1260 / 1452 : pp = 143.63771057128906 1270 / 1452 : pp = 143.65798950195312 1280 / 1452 : pp = 143.68251037597656 1290 / 1452 : pp = 143.68045043945312 1300 / 1452 : pp = 143.72293090820312 1310 / 1452 : pp = 143.77015686035156 1320 / 1452 : pp = 143.71910095214844 1330 / 1452 : pp = 143.68792724609375 1340 / 1452 : pp = 143.7241668701172 1350 / 1452 : pp = 143.7570037841797 1360 / 1452 : pp = 143.81829833984375 1370 / 1452 : pp = 143.7487030029297 1380 / 1452 : pp = 143.7196502685547 1390 / 1452 : pp = 143.67359924316406 1400 / 1452 : pp = 143.60592651367188 1410 / 1452 : pp = 143.62620544433594 1420 / 1452 : pp = 143.5905303955078 1430 / 1452 : pp = 143.55799865722656 1440 / 1452 : pp = 143.5891571044922 1450 / 1452 : pp = 143.5869598388672 0 / 115 : pp = 226.9864959716797 10 / 115 : pp = 207.8067169189453 20 / 115 : pp = 209.68667602539062 30 / 115 : pp = 207.1610565185547 40 / 115 : pp = 206.3247833251953 50 / 115 : pp = 201.77403259277344 60 / 115 : pp = 201.07098388671875 70 / 115 : pp = 197.33335876464844 80 / 115 : pp = 195.12513732910156 90 / 115 : pp = 192.5349578857422 100 / 115 : pp = 187.90072631835938 110 / 115 : pp = 185.81240844726562 Training perplexity: 143.57354736328125 Validation perplexity:185.40573120117188 Total time : 46.14846849441528 Epoch 11 0 / 1452 : pp = 181.93162536621094 10 / 1452 : pp = 159.94607543945312 20 / 1452 : pp = 156.83673095703125 30 / 1452 : pp = 153.75843811035156 40 / 1452 : pp = 155.18362426757812 50 / 1452 : pp = 152.39529418945312 60 / 1452 : pp = 151.18772888183594 70 / 1452 : pp = 151.9004364013672 80 / 1452 : pp = 151.30239868164062 90 / 1452 : pp = 150.1591033935547 100 / 1452 : pp = 149.18618774414062 110 / 1452 : pp = 147.72653198242188 120 / 1452 : pp = 147.4357452392578 130 / 1452 : pp = 146.41372680664062 140 / 1452 : pp = 145.0057373046875 150 / 1452 : pp = 144.39447021484375 160 / 1452 : pp = 144.5330047607422 170 / 1452 : pp = 144.23593139648438 180 / 1452 : pp = 143.63990783691406 190 / 1452 : pp = 143.63812255859375 200 / 1452 : pp = 144.1143798828125 210 / 1452 : pp = 143.88278198242188 220 / 1452 : pp = 143.92518615722656 230 / 1452 : pp = 144.24032592773438 240 / 1452 : pp = 143.94110107421875 250 / 1452 : pp = 143.3688507080078 260 / 1452 : pp = 142.8829345703125 270 / 1452 : pp = 142.11952209472656 280 / 1452 : pp = 142.19415283203125 290 / 1452 : pp = 142.51889038085938 300 / 1452 : pp = 142.70494079589844 310 / 1452 : pp = 142.51426696777344 320 / 1452 : pp = 142.70106506347656 330 / 1452 : pp = 142.88014221191406 340 / 1452 : pp = 142.3287353515625 350 / 1452 : pp = 142.6169891357422 360 / 1452 : pp = 142.51971435546875 370 / 1452 : pp = 142.33566284179688 380 / 1452 : pp = 142.04161071777344 390 / 1452 : pp = 142.13551330566406 400 / 1452 : pp = 141.9499969482422 410 / 1452 : pp = 142.3361358642578 420 / 1452 : pp = 142.64065551757812 430 / 1452 : pp = 142.5511016845703 440 / 1452 : pp = 142.6728973388672 450 / 1452 : pp = 142.47030639648438 460 / 1452 : pp = 142.1704864501953 470 / 1452 : pp = 141.73390197753906 480 / 1452 : pp = 141.23020935058594 490 / 1452 : pp = 140.9759521484375 500 / 1452 : pp = 140.51609802246094 510 / 1452 : pp = 140.40545654296875 520 / 1452 : pp = 140.1936492919922 530 / 1452 : pp = 139.8929443359375 540 / 1452 : pp = 139.3696746826172 550 / 1452 : pp = 139.13217163085938 560 / 1452 : pp = 138.85247802734375 570 / 1452 : pp = 138.6092987060547 580 / 1452 : pp = 138.2471160888672 590 / 1452 : pp = 137.9485626220703 600 / 1452 : pp = 137.57379150390625 610 / 1452 : pp = 137.31576538085938 620 / 1452 : pp = 137.14230346679688 630 / 1452 : pp = 136.87405395507812 640 / 1452 : pp = 137.02928161621094 650 / 1452 : pp = 137.0481719970703 660 / 1452 : pp = 137.1595001220703 670 / 1452 : pp = 137.21124267578125 680 / 1452 : pp = 137.2671356201172 690 / 1452 : pp = 137.19410705566406 700 / 1452 : pp = 137.1850128173828 710 / 1452 : pp = 137.26058959960938 720 / 1452 : pp = 137.30726623535156 730 / 1452 : pp = 137.28048706054688 740 / 1452 : pp = 137.4352569580078 750 / 1452 : pp = 137.4680938720703 760 / 1452 : pp = 137.5524139404297 770 / 1452 : pp = 137.73829650878906 780 / 1452 : pp = 137.90882873535156 790 / 1452 : pp = 138.05865478515625 800 / 1452 : pp = 138.0673370361328 810 / 1452 : pp = 138.03909301757812 820 / 1452 : pp = 138.084716796875 830 / 1452 : pp = 138.27989196777344 840 / 1452 : pp = 138.23545837402344 850 / 1452 : pp = 138.30343627929688 860 / 1452 : pp = 138.3339080810547 870 / 1452 : pp = 138.32835388183594 880 / 1452 : pp = 138.4450225830078 890 / 1452 : pp = 138.47157287597656 900 / 1452 : pp = 138.46304321289062 910 / 1452 : pp = 138.55618286132812 920 / 1452 : pp = 138.64512634277344 930 / 1452 : pp = 138.6160430908203 940 / 1452 : pp = 138.66932678222656 950 / 1452 : pp = 138.6573028564453 960 / 1452 : pp = 138.6463165283203 970 / 1452 : pp = 138.67059326171875 980 / 1452 : pp = 138.50999450683594 990 / 1452 : pp = 138.42430114746094 1000 / 1452 : pp = 138.25344848632812 1010 / 1452 : pp = 138.3004608154297 1020 / 1452 : pp = 138.4243621826172 1030 / 1452 : pp = 138.40713500976562 1040 / 1452 : pp = 138.47129821777344 1050 / 1452 : pp = 138.45928955078125 1060 / 1452 : pp = 138.3919677734375 1070 / 1452 : pp = 138.5287628173828 1080 / 1452 : pp = 138.62298583984375 1090 / 1452 : pp = 138.6699981689453 1100 / 1452 : pp = 138.64849853515625 1110 / 1452 : pp = 138.49191284179688 1120 / 1452 : pp = 138.37355041503906 1130 / 1452 : pp = 138.2216796875 1140 / 1452 : pp = 138.21534729003906 1150 / 1452 : pp = 138.30963134765625 1160 / 1452 : pp = 138.316162109375 1170 / 1452 : pp = 138.3023681640625 1180 / 1452 : pp = 138.36932373046875 1190 / 1452 : pp = 138.45960998535156 1200 / 1452 : pp = 138.4866180419922 1210 / 1452 : pp = 138.45730590820312 1220 / 1452 : pp = 138.60031127929688 1230 / 1452 : pp = 138.75485229492188 1240 / 1452 : pp = 138.7751007080078 1250 / 1452 : pp = 138.91221618652344 1260 / 1452 : pp = 138.9815216064453 1270 / 1452 : pp = 138.9919891357422 1280 / 1452 : pp = 139.0243377685547 1290 / 1452 : pp = 139.02725219726562 1300 / 1452 : pp = 139.0701446533203 1310 / 1452 : pp = 139.1090850830078 1320 / 1452 : pp = 139.06027221679688 1330 / 1452 : pp = 139.0338134765625 1340 / 1452 : pp = 139.06385803222656 1350 / 1452 : pp = 139.09608459472656 1360 / 1452 : pp = 139.1609649658203 1370 / 1452 : pp = 139.0869903564453 1380 / 1452 : pp = 139.0604705810547 1390 / 1452 : pp = 139.01670837402344 1400 / 1452 : pp = 138.94393920898438 1410 / 1452 : pp = 138.97323608398438 1420 / 1452 : pp = 138.9404296875 1430 / 1452 : pp = 138.90943908691406 1440 / 1452 : pp = 138.94268798828125 1450 / 1452 : pp = 138.93991088867188 0 / 115 : pp = 225.55990600585938 10 / 115 : pp = 207.0504608154297 20 / 115 : pp = 208.98306274414062 30 / 115 : pp = 206.28396606445312 40 / 115 : pp = 205.35386657714844 50 / 115 : pp = 200.7255401611328 60 / 115 : pp = 200.0526580810547 70 / 115 : pp = 196.33087158203125 80 / 115 : pp = 194.12110900878906 90 / 115 : pp = 191.52816772460938 100 / 115 : pp = 186.7974395751953 110 / 115 : pp = 184.59829711914062 Training perplexity: 138.9222869873047 Validation perplexity:184.18101501464844 Total time : 43.92928600311279 Epoch 12 0 / 1452 : pp = 173.0251007080078 10 / 1452 : pp = 152.98446655273438 20 / 1452 : pp = 150.43128967285156 30 / 1452 : pp = 147.5819854736328 40 / 1452 : pp = 149.4164276123047 50 / 1452 : pp = 146.70816040039062 60 / 1452 : pp = 145.557861328125 70 / 1452 : pp = 146.50473022460938 80 / 1452 : pp = 145.83200073242188 90 / 1452 : pp = 144.84402465820312 100 / 1452 : pp = 144.0390167236328 110 / 1452 : pp = 142.66514587402344 120 / 1452 : pp = 142.3549346923828 130 / 1452 : pp = 141.4630126953125 140 / 1452 : pp = 140.2266082763672 150 / 1452 : pp = 139.67518615722656 160 / 1452 : pp = 139.90414428710938 170 / 1452 : pp = 139.5490264892578 180 / 1452 : pp = 138.91969299316406 190 / 1452 : pp = 138.89234924316406 200 / 1452 : pp = 139.40908813476562 210 / 1452 : pp = 139.19068908691406 220 / 1452 : pp = 139.35513305664062 230 / 1452 : pp = 139.5464324951172 240 / 1452 : pp = 139.3047637939453 250 / 1452 : pp = 138.7708740234375 260 / 1452 : pp = 138.29188537597656 270 / 1452 : pp = 137.4787139892578 280 / 1452 : pp = 137.6367950439453 290 / 1452 : pp = 137.98513793945312 300 / 1452 : pp = 138.17819213867188 310 / 1452 : pp = 137.943359375 320 / 1452 : pp = 138.12060546875 330 / 1452 : pp = 138.29037475585938 340 / 1452 : pp = 137.77606201171875 350 / 1452 : pp = 138.06378173828125 360 / 1452 : pp = 137.99000549316406 370 / 1452 : pp = 137.81922912597656 380 / 1452 : pp = 137.52159118652344 390 / 1452 : pp = 137.61782836914062 400 / 1452 : pp = 137.4178924560547 410 / 1452 : pp = 137.82632446289062 420 / 1452 : pp = 138.17567443847656 430 / 1452 : pp = 138.11863708496094 440 / 1452 : pp = 138.215087890625 450 / 1452 : pp = 137.9976348876953 460 / 1452 : pp = 137.6929168701172 470 / 1452 : pp = 137.25416564941406 480 / 1452 : pp = 136.75140380859375 490 / 1452 : pp = 136.51712036132812 500 / 1452 : pp = 136.0896453857422 510 / 1452 : pp = 135.97048950195312 520 / 1452 : pp = 135.7760009765625 530 / 1452 : pp = 135.50389099121094 540 / 1452 : pp = 135.01437377929688 550 / 1452 : pp = 134.7666015625 560 / 1452 : pp = 134.48973083496094 570 / 1452 : pp = 134.22853088378906 580 / 1452 : pp = 133.88455200195312 590 / 1452 : pp = 133.5808868408203 600 / 1452 : pp = 133.22975158691406 610 / 1452 : pp = 132.99591064453125 620 / 1452 : pp = 132.79502868652344 630 / 1452 : pp = 132.5094451904297 640 / 1452 : pp = 132.62892150878906 650 / 1452 : pp = 132.63499450683594 660 / 1452 : pp = 132.7379913330078 670 / 1452 : pp = 132.79046630859375 680 / 1452 : pp = 132.85842895507812 690 / 1452 : pp = 132.80364990234375 700 / 1452 : pp = 132.80477905273438 710 / 1452 : pp = 132.90170288085938 720 / 1452 : pp = 132.92971801757812 730 / 1452 : pp = 132.9019012451172 740 / 1452 : pp = 133.04811096191406 750 / 1452 : pp = 133.10877990722656 760 / 1452 : pp = 133.19189453125 770 / 1452 : pp = 133.3564910888672 780 / 1452 : pp = 133.54000854492188 790 / 1452 : pp = 133.69239807128906 800 / 1452 : pp = 133.68495178222656 810 / 1452 : pp = 133.67971801757812 820 / 1452 : pp = 133.7035675048828 830 / 1452 : pp = 133.89329528808594 840 / 1452 : pp = 133.850341796875 850 / 1452 : pp = 133.90390014648438 860 / 1452 : pp = 133.9090118408203 870 / 1452 : pp = 133.89974975585938 880 / 1452 : pp = 134.0077667236328 890 / 1452 : pp = 134.03485107421875 900 / 1452 : pp = 134.0261688232422 910 / 1452 : pp = 134.10255432128906 920 / 1452 : pp = 134.17291259765625 930 / 1452 : pp = 134.14796447753906 940 / 1452 : pp = 134.20925903320312 950 / 1452 : pp = 134.19281005859375 960 / 1452 : pp = 134.17745971679688 970 / 1452 : pp = 134.18653869628906 980 / 1452 : pp = 134.03192138671875 990 / 1452 : pp = 133.94349670410156 1000 / 1452 : pp = 133.79685974121094 1010 / 1452 : pp = 133.8438262939453 1020 / 1452 : pp = 133.9608612060547 1030 / 1452 : pp = 133.93934631347656 1040 / 1452 : pp = 134.02833557128906 1050 / 1452 : pp = 134.01734924316406 1060 / 1452 : pp = 133.95346069335938 1070 / 1452 : pp = 134.10205078125 1080 / 1452 : pp = 134.2030487060547 1090 / 1452 : pp = 134.23696899414062 1100 / 1452 : pp = 134.2230224609375 1110 / 1452 : pp = 134.0829315185547 1120 / 1452 : pp = 133.980224609375 1130 / 1452 : pp = 133.83815002441406 1140 / 1452 : pp = 133.8366241455078 1150 / 1452 : pp = 133.92108154296875 1160 / 1452 : pp = 133.94375610351562 1170 / 1452 : pp = 133.9360809326172 1180 / 1452 : pp = 133.99684143066406 1190 / 1452 : pp = 134.0944366455078 1200 / 1452 : pp = 134.11676025390625 1210 / 1452 : pp = 134.0911102294922 1220 / 1452 : pp = 134.22763061523438 1230 / 1452 : pp = 134.38043212890625 1240 / 1452 : pp = 134.39817810058594 1250 / 1452 : pp = 134.5367431640625 1260 / 1452 : pp = 134.593017578125 1270 / 1452 : pp = 134.61497497558594 1280 / 1452 : pp = 134.6423797607422 1290 / 1452 : pp = 134.64340209960938 1300 / 1452 : pp = 134.68026733398438 1310 / 1452 : pp = 134.73556518554688 1320 / 1452 : pp = 134.69021606445312 1330 / 1452 : pp = 134.66131591796875 1340 / 1452 : pp = 134.69393920898438 1350 / 1452 : pp = 134.7328643798828 1360 / 1452 : pp = 134.79405212402344 1370 / 1452 : pp = 134.71237182617188 1380 / 1452 : pp = 134.6885528564453 1390 / 1452 : pp = 134.65110778808594 1400 / 1452 : pp = 134.59584045410156 1410 / 1452 : pp = 134.6193389892578 1420 / 1452 : pp = 134.58338928222656 1430 / 1452 : pp = 134.559326171875 1440 / 1452 : pp = 134.59507751464844 1450 / 1452 : pp = 134.59365844726562 0 / 115 : pp = 226.0741729736328 10 / 115 : pp = 207.00494384765625 20 / 115 : pp = 209.26976013183594 30 / 115 : pp = 206.44662475585938 40 / 115 : pp = 205.47268676757812 50 / 115 : pp = 200.7876739501953 60 / 115 : pp = 200.13414001464844 70 / 115 : pp = 196.35549926757812 80 / 115 : pp = 194.10777282714844 90 / 115 : pp = 191.47467041015625 100 / 115 : pp = 186.61351013183594 110 / 115 : pp = 184.30374145507812 Training perplexity: 134.57826232910156 Validation perplexity:183.8900146484375 Total time : 45.410256147384644 Epoch 13 0 / 1452 : pp = 169.39393615722656 10 / 1452 : pp = 150.13232421875 20 / 1452 : pp = 147.60450744628906 30 / 1452 : pp = 144.64317321777344 40 / 1452 : pp = 146.47427368164062 50 / 1452 : pp = 143.929443359375 60 / 1452 : pp = 142.8344268798828 70 / 1452 : pp = 143.45248413085938 80 / 1452 : pp = 142.5418701171875 90 / 1452 : pp = 141.6178436279297 100 / 1452 : pp = 140.70127868652344 110 / 1452 : pp = 139.2852325439453 120 / 1452 : pp = 138.8017120361328 130 / 1452 : pp = 137.85629272460938 140 / 1452 : pp = 136.51718139648438 150 / 1452 : pp = 136.03619384765625 160 / 1452 : pp = 136.154296875 170 / 1452 : pp = 135.67037963867188 180 / 1452 : pp = 135.0376739501953 190 / 1452 : pp = 134.9230499267578 200 / 1452 : pp = 135.4241180419922 210 / 1452 : pp = 135.24581909179688 220 / 1452 : pp = 135.37957763671875 230 / 1452 : pp = 135.67652893066406 240 / 1452 : pp = 135.4161834716797 250 / 1452 : pp = 134.90895080566406 260 / 1452 : pp = 134.46754455566406 270 / 1452 : pp = 133.68577575683594 280 / 1452 : pp = 133.86770629882812 290 / 1452 : pp = 134.18475341796875 300 / 1452 : pp = 134.39132690429688 310 / 1452 : pp = 134.19985961914062 320 / 1452 : pp = 134.37998962402344 330 / 1452 : pp = 134.5557403564453 340 / 1452 : pp = 134.00686645507812 350 / 1452 : pp = 134.27749633789062 360 / 1452 : pp = 134.20286560058594 370 / 1452 : pp = 134.042724609375 380 / 1452 : pp = 133.74398803710938 390 / 1452 : pp = 133.83584594726562 400 / 1452 : pp = 133.64382934570312 410 / 1452 : pp = 134.02366638183594 420 / 1452 : pp = 134.35415649414062 430 / 1452 : pp = 134.310546875 440 / 1452 : pp = 134.3634490966797 450 / 1452 : pp = 134.15602111816406 460 / 1452 : pp = 133.86578369140625 470 / 1452 : pp = 133.43414306640625 480 / 1452 : pp = 132.90310668945312 490 / 1452 : pp = 132.646240234375 500 / 1452 : pp = 132.1982421875 510 / 1452 : pp = 132.04200744628906 520 / 1452 : pp = 131.86940002441406 530 / 1452 : pp = 131.59841918945312 540 / 1452 : pp = 131.12356567382812 550 / 1452 : pp = 130.887939453125 560 / 1452 : pp = 130.6210174560547 570 / 1452 : pp = 130.37826538085938 580 / 1452 : pp = 130.0374755859375 590 / 1452 : pp = 129.75979614257812 600 / 1452 : pp = 129.38308715820312 610 / 1452 : pp = 129.16685485839844 620 / 1452 : pp = 129.0115509033203 630 / 1452 : pp = 128.75152587890625 640 / 1452 : pp = 128.87295532226562 650 / 1452 : pp = 128.88734436035156 660 / 1452 : pp = 128.98275756835938 670 / 1452 : pp = 129.0487060546875 680 / 1452 : pp = 129.11013793945312 690 / 1452 : pp = 129.0646514892578 700 / 1452 : pp = 129.06280517578125 710 / 1452 : pp = 129.1343994140625 720 / 1452 : pp = 129.18582153320312 730 / 1452 : pp = 129.15138244628906 740 / 1452 : pp = 129.29811096191406 750 / 1452 : pp = 129.339599609375 760 / 1452 : pp = 129.4257354736328 770 / 1452 : pp = 129.61631774902344 780 / 1452 : pp = 129.802734375 790 / 1452 : pp = 129.96804809570312 800 / 1452 : pp = 129.95187377929688 810 / 1452 : pp = 129.92417907714844 820 / 1452 : pp = 129.9774627685547 830 / 1452 : pp = 130.1638946533203 840 / 1452 : pp = 130.13095092773438 850 / 1452 : pp = 130.16595458984375 860 / 1452 : pp = 130.173828125 870 / 1452 : pp = 130.170166015625 880 / 1452 : pp = 130.27032470703125 890 / 1452 : pp = 130.3022003173828 900 / 1452 : pp = 130.3071746826172 910 / 1452 : pp = 130.37939453125 920 / 1452 : pp = 130.46229553222656 930 / 1452 : pp = 130.43846130371094 940 / 1452 : pp = 130.50889587402344 950 / 1452 : pp = 130.50086975097656 960 / 1452 : pp = 130.4833221435547 970 / 1452 : pp = 130.50814819335938 980 / 1452 : pp = 130.35577392578125 990 / 1452 : pp = 130.26759338378906 1000 / 1452 : pp = 130.1064453125 1010 / 1452 : pp = 130.1472625732422 1020 / 1452 : pp = 130.27169799804688 1030 / 1452 : pp = 130.25100708007812 1040 / 1452 : pp = 130.30816650390625 1050 / 1452 : pp = 130.29803466796875 1060 / 1452 : pp = 130.2242431640625 1070 / 1452 : pp = 130.35906982421875 1080 / 1452 : pp = 130.45103454589844 1090 / 1452 : pp = 130.49838256835938 1100 / 1452 : pp = 130.484130859375 1110 / 1452 : pp = 130.35316467285156 1120 / 1452 : pp = 130.24697875976562 1130 / 1452 : pp = 130.10804748535156 1140 / 1452 : pp = 130.1076202392578 1150 / 1452 : pp = 130.195068359375 1160 / 1452 : pp = 130.19674682617188 1170 / 1452 : pp = 130.18321228027344 1180 / 1452 : pp = 130.24623107910156 1190 / 1452 : pp = 130.33905029296875 1200 / 1452 : pp = 130.3650360107422 1210 / 1452 : pp = 130.34588623046875 1220 / 1452 : pp = 130.4850616455078 1230 / 1452 : pp = 130.63160705566406 1240 / 1452 : pp = 130.64674377441406 1250 / 1452 : pp = 130.77078247070312 1260 / 1452 : pp = 130.8397674560547 1270 / 1452 : pp = 130.8511199951172 1280 / 1452 : pp = 130.88967895507812 1290 / 1452 : pp = 130.9040985107422 1300 / 1452 : pp = 130.93511962890625 1310 / 1452 : pp = 130.9759063720703 1320 / 1452 : pp = 130.92800903320312 1330 / 1452 : pp = 130.9105224609375 1340 / 1452 : pp = 130.929443359375 1350 / 1452 : pp = 130.96153259277344 1360 / 1452 : pp = 131.02381896972656 1370 / 1452 : pp = 130.9545440673828 1380 / 1452 : pp = 130.9344940185547 1390 / 1452 : pp = 130.9055938720703 1400 / 1452 : pp = 130.85386657714844 1410 / 1452 : pp = 130.8874969482422 1420 / 1452 : pp = 130.85928344726562 1430 / 1452 : pp = 130.83995056152344 1440 / 1452 : pp = 130.86659240722656 1450 / 1452 : pp = 130.86839294433594 0 / 115 : pp = 227.78428649902344 10 / 115 : pp = 207.609619140625 20 / 115 : pp = 209.92459106445312 30 / 115 : pp = 206.96240234375 40 / 115 : pp = 205.9295654296875 50 / 115 : pp = 201.0296630859375 60 / 115 : pp = 200.38059997558594 70 / 115 : pp = 196.55764770507812 80 / 115 : pp = 194.31735229492188 90 / 115 : pp = 191.66146850585938 100 / 115 : pp = 186.70437622070312 110 / 115 : pp = 184.3171844482422 Training perplexity: 130.85043334960938 Validation perplexity:183.88186645507812 Total time : 45.345656394958496 Epoch 14 0 / 1452 : pp = 164.82191467285156 10 / 1452 : pp = 146.39089965820312 20 / 1452 : pp = 142.93240356445312 30 / 1452 : pp = 140.3113555908203 40 / 1452 : pp = 142.39939880371094 50 / 1452 : pp = 139.70162963867188 60 / 1452 : pp = 138.73023986816406 70 / 1452 : pp = 139.2675018310547 80 / 1452 : pp = 138.47824096679688 90 / 1452 : pp = 137.40432739257812 100 / 1452 : pp = 136.47793579101562 110 / 1452 : pp = 135.2294464111328 120 / 1452 : pp = 134.80728149414062 130 / 1452 : pp = 133.89822387695312 140 / 1452 : pp = 132.54141235351562 150 / 1452 : pp = 132.10025024414062 160 / 1452 : pp = 132.21829223632812 170 / 1452 : pp = 131.8765106201172 180 / 1452 : pp = 131.37515258789062 190 / 1452 : pp = 131.31622314453125 200 / 1452 : pp = 131.78297424316406 210 / 1452 : pp = 131.5507354736328 220 / 1452 : pp = 131.7002410888672 230 / 1452 : pp = 131.9277801513672 240 / 1452 : pp = 131.72166442871094 250 / 1452 : pp = 131.225830078125 260 / 1452 : pp = 130.7496337890625 270 / 1452 : pp = 129.9896697998047 280 / 1452 : pp = 130.10594177246094 290 / 1452 : pp = 130.41644287109375 300 / 1452 : pp = 130.5982208251953 310 / 1452 : pp = 130.36329650878906 320 / 1452 : pp = 130.5633544921875 330 / 1452 : pp = 130.77252197265625 340 / 1452 : pp = 130.273193359375 350 / 1452 : pp = 130.47889709472656 360 / 1452 : pp = 130.4348602294922 370 / 1452 : pp = 130.28126525878906 380 / 1452 : pp = 130.02786254882812 390 / 1452 : pp = 130.1564483642578 400 / 1452 : pp = 129.98440551757812 410 / 1452 : pp = 130.37721252441406 420 / 1452 : pp = 130.71859741210938 430 / 1452 : pp = 130.65939331054688 440 / 1452 : pp = 130.72987365722656 450 / 1452 : pp = 130.56272888183594 460 / 1452 : pp = 130.28195190429688 470 / 1452 : pp = 129.90936279296875 480 / 1452 : pp = 129.42857360839844 490 / 1452 : pp = 129.18077087402344 500 / 1452 : pp = 128.7588348388672 510 / 1452 : pp = 128.6303253173828 520 / 1452 : pp = 128.47616577148438 530 / 1452 : pp = 128.21148681640625 540 / 1452 : pp = 127.7218017578125 550 / 1452 : pp = 127.50067138671875 560 / 1452 : pp = 127.27574157714844 570 / 1452 : pp = 127.05399322509766 580 / 1452 : pp = 126.73983001708984 590 / 1452 : pp = 126.43692779541016 600 / 1452 : pp = 126.06050109863281 610 / 1452 : pp = 125.82952880859375 620 / 1452 : pp = 125.66295623779297 630 / 1452 : pp = 125.39354705810547 640 / 1452 : pp = 125.49463653564453 650 / 1452 : pp = 125.48816680908203 660 / 1452 : pp = 125.58712005615234 670 / 1452 : pp = 125.65978240966797 680 / 1452 : pp = 125.71456146240234 690 / 1452 : pp = 125.66937255859375 700 / 1452 : pp = 125.65900421142578 710 / 1452 : pp = 125.7271499633789 720 / 1452 : pp = 125.77758026123047 730 / 1452 : pp = 125.74129486083984 740 / 1452 : pp = 125.8759765625 750 / 1452 : pp = 125.91793823242188 760 / 1452 : pp = 125.99595642089844 770 / 1452 : pp = 126.18113708496094 780 / 1452 : pp = 126.35147094726562 790 / 1452 : pp = 126.50797271728516 800 / 1452 : pp = 126.49759674072266 810 / 1452 : pp = 126.48113250732422 820 / 1452 : pp = 126.52528381347656 830 / 1452 : pp = 126.705810546875 840 / 1452 : pp = 126.67517852783203 850 / 1452 : pp = 126.74176025390625 860 / 1452 : pp = 126.74151611328125 870 / 1452 : pp = 126.73414611816406 880 / 1452 : pp = 126.83026885986328 890 / 1452 : pp = 126.88519287109375 900 / 1452 : pp = 126.88053894042969 910 / 1452 : pp = 126.97138214111328 920 / 1452 : pp = 127.04660034179688 930 / 1452 : pp = 127.03763580322266 940 / 1452 : pp = 127.1126480102539 950 / 1452 : pp = 127.09610748291016 960 / 1452 : pp = 127.0873794555664 970 / 1452 : pp = 127.10343933105469 980 / 1452 : pp = 126.96441650390625 990 / 1452 : pp = 126.88519287109375 1000 / 1452 : pp = 126.7336654663086 1010 / 1452 : pp = 126.77796936035156 1020 / 1452 : pp = 126.89826202392578 1030 / 1452 : pp = 126.88761138916016 1040 / 1452 : pp = 126.95309448242188 1050 / 1452 : pp = 126.96478271484375 1060 / 1452 : pp = 126.89324188232422 1070 / 1452 : pp = 127.03242492675781 1080 / 1452 : pp = 127.13228607177734 1090 / 1452 : pp = 127.173095703125 1100 / 1452 : pp = 127.15975189208984 1110 / 1452 : pp = 127.0392074584961 1120 / 1452 : pp = 126.94032287597656 1130 / 1452 : pp = 126.80693054199219 1140 / 1452 : pp = 126.81315612792969 1150 / 1452 : pp = 126.90467834472656 1160 / 1452 : pp = 126.91236114501953 1170 / 1452 : pp = 126.90897369384766 1180 / 1452 : pp = 126.98052215576172 1190 / 1452 : pp = 127.07483673095703 1200 / 1452 : pp = 127.10216522216797 1210 / 1452 : pp = 127.08258819580078 1220 / 1452 : pp = 127.22943878173828 1230 / 1452 : pp = 127.38563537597656 1240 / 1452 : pp = 127.40538024902344 1250 / 1452 : pp = 127.53369140625 1260 / 1452 : pp = 127.59293365478516 1270 / 1452 : pp = 127.61489868164062 1280 / 1452 : pp = 127.6484375 1290 / 1452 : pp = 127.65257263183594 1300 / 1452 : pp = 127.69329833984375 1310 / 1452 : pp = 127.74549102783203 1320 / 1452 : pp = 127.7043228149414 1330 / 1452 : pp = 127.6866683959961 1340 / 1452 : pp = 127.70913696289062 1350 / 1452 : pp = 127.73233795166016 1360 / 1452 : pp = 127.7855224609375 1370 / 1452 : pp = 127.71918487548828 1380 / 1452 : pp = 127.69987487792969 1390 / 1452 : pp = 127.6697998046875 1400 / 1452 : pp = 127.61137390136719 1410 / 1452 : pp = 127.6404037475586 1420 / 1452 : pp = 127.61094665527344 1430 / 1452 : pp = 127.58216857910156 1440 / 1452 : pp = 127.61477661132812 1450 / 1452 : pp = 127.61964416503906 0 / 115 : pp = 228.21578979492188 10 / 115 : pp = 208.11244201660156 20 / 115 : pp = 210.688232421875 30 / 115 : pp = 207.62408447265625 40 / 115 : pp = 206.45184326171875 50 / 115 : pp = 201.52760314941406 60 / 115 : pp = 200.7784881591797 70 / 115 : pp = 196.83067321777344 80 / 115 : pp = 194.6357879638672 90 / 115 : pp = 191.9783935546875 100 / 115 : pp = 186.8787841796875 110 / 115 : pp = 184.35252380371094 Training perplexity: 127.60413360595703 Validation perplexity:183.8877410888672 Total time : 41.6636528968811 Epoch 15 0 / 1452 : pp = 156.81654357910156 10 / 1452 : pp = 142.1070556640625 20 / 1452 : pp = 139.55076599121094 30 / 1452 : pp = 136.63551330566406 40 / 1452 : pp = 138.5840606689453 50 / 1452 : pp = 136.052734375 60 / 1452 : pp = 134.93019104003906 70 / 1452 : pp = 135.65206909179688 80 / 1452 : pp = 135.2620086669922 90 / 1452 : pp = 134.314697265625 100 / 1452 : pp = 133.4916229248047 110 / 1452 : pp = 132.26052856445312 120 / 1452 : pp = 131.7714080810547 130 / 1452 : pp = 130.77365112304688 140 / 1452 : pp = 129.5411834716797 150 / 1452 : pp = 129.0791778564453 160 / 1452 : pp = 129.21920776367188 170 / 1452 : pp = 128.7528839111328 180 / 1452 : pp = 128.22279357910156 190 / 1452 : pp = 128.18177795410156 200 / 1452 : pp = 128.58758544921875 210 / 1452 : pp = 128.3906707763672 220 / 1452 : pp = 128.5266571044922 230 / 1452 : pp = 128.80563354492188 240 / 1452 : pp = 128.61886596679688 250 / 1452 : pp = 128.13172912597656 260 / 1452 : pp = 127.69220733642578 270 / 1452 : pp = 126.96150970458984 280 / 1452 : pp = 127.04702758789062 290 / 1452 : pp = 127.33565521240234 300 / 1452 : pp = 127.55929565429688 310 / 1452 : pp = 127.38514709472656 320 / 1452 : pp = 127.52171325683594 330 / 1452 : pp = 127.68690490722656 340 / 1452 : pp = 127.18340301513672 350 / 1452 : pp = 127.4073257446289 360 / 1452 : pp = 127.30432891845703 370 / 1452 : pp = 127.17618560791016 380 / 1452 : pp = 126.92579650878906 390 / 1452 : pp = 127.02473449707031 400 / 1452 : pp = 126.8515625 410 / 1452 : pp = 127.211669921875 420 / 1452 : pp = 127.51788330078125 430 / 1452 : pp = 127.47386169433594 440 / 1452 : pp = 127.57164001464844 450 / 1452 : pp = 127.3601303100586 460 / 1452 : pp = 127.09434509277344 470 / 1452 : pp = 126.71922302246094 480 / 1452 : pp = 126.24349212646484 490 / 1452 : pp = 125.98778533935547 500 / 1452 : pp = 125.59526824951172 510 / 1452 : pp = 125.4450912475586 520 / 1452 : pp = 125.29247283935547 530 / 1452 : pp = 125.03536224365234 540 / 1452 : pp = 124.5813980102539 550 / 1452 : pp = 124.33724212646484 560 / 1452 : pp = 124.08995819091797 570 / 1452 : pp = 123.86637878417969 580 / 1452 : pp = 123.53152465820312 590 / 1452 : pp = 123.20321655273438 600 / 1452 : pp = 122.85673522949219 610 / 1452 : pp = 122.64250946044922 620 / 1452 : pp = 122.4958724975586 630 / 1452 : pp = 122.22386169433594 640 / 1452 : pp = 122.31143188476562 650 / 1452 : pp = 122.30093383789062 660 / 1452 : pp = 122.39427947998047 670 / 1452 : pp = 122.45440673828125 680 / 1452 : pp = 122.51146697998047 690 / 1452 : pp = 122.4854736328125 700 / 1452 : pp = 122.48600006103516 710 / 1452 : pp = 122.56084442138672 720 / 1452 : pp = 122.59059143066406 730 / 1452 : pp = 122.55529022216797 740 / 1452 : pp = 122.69409942626953 750 / 1452 : pp = 122.76456451416016 760 / 1452 : pp = 122.84437561035156 770 / 1452 : pp = 123.02527618408203 780 / 1452 : pp = 123.20509338378906 790 / 1452 : pp = 123.36305236816406 800 / 1452 : pp = 123.36852264404297 810 / 1452 : pp = 123.36799621582031 820 / 1452 : pp = 123.39976501464844 830 / 1452 : pp = 123.59362030029297 840 / 1452 : pp = 123.56946563720703 850 / 1452 : pp = 123.63800811767578 860 / 1452 : pp = 123.63983917236328 870 / 1452 : pp = 123.64148712158203 880 / 1452 : pp = 123.7568588256836 890 / 1452 : pp = 123.7885513305664 900 / 1452 : pp = 123.79640197753906 910 / 1452 : pp = 123.86153411865234 920 / 1452 : pp = 123.92941284179688 930 / 1452 : pp = 123.9125747680664 940 / 1452 : pp = 123.95559692382812 950 / 1452 : pp = 123.93928527832031 960 / 1452 : pp = 123.94294738769531 970 / 1452 : pp = 123.95547485351562 980 / 1452 : pp = 123.8229751586914 990 / 1452 : pp = 123.73727416992188 1000 / 1452 : pp = 123.59091186523438 1010 / 1452 : pp = 123.634765625 1020 / 1452 : pp = 123.76506042480469 1030 / 1452 : pp = 123.75485229492188 1040 / 1452 : pp = 123.807861328125 1050 / 1452 : pp = 123.79156494140625 1060 / 1452 : pp = 123.73054504394531 1070 / 1452 : pp = 123.8615951538086 1080 / 1452 : pp = 123.96564483642578 1090 / 1452 : pp = 124.02104187011719 1100 / 1452 : pp = 124.012939453125 1110 / 1452 : pp = 123.87582397460938 1120 / 1452 : pp = 123.775390625 1130 / 1452 : pp = 123.63182067871094 1140 / 1452 : pp = 123.62391662597656 1150 / 1452 : pp = 123.71013641357422 1160 / 1452 : pp = 123.72423553466797 1170 / 1452 : pp = 123.71726989746094 1180 / 1452 : pp = 123.79032897949219 1190 / 1452 : pp = 123.87883758544922 1200 / 1452 : pp = 123.9125747680664 1210 / 1452 : pp = 123.90140533447266 1220 / 1452 : pp = 124.03245544433594 1230 / 1452 : pp = 124.19799041748047 1240 / 1452 : pp = 124.21469116210938 1250 / 1452 : pp = 124.34103393554688 1260 / 1452 : pp = 124.4041976928711 1270 / 1452 : pp = 124.42852020263672 1280 / 1452 : pp = 124.46656036376953 1290 / 1452 : pp = 124.4811019897461 1300 / 1452 : pp = 124.52384185791016 1310 / 1452 : pp = 124.57533264160156 1320 / 1452 : pp = 124.5398178100586 1330 / 1452 : pp = 124.52598571777344 1340 / 1452 : pp = 124.53311157226562 1350 / 1452 : pp = 124.57759094238281 1360 / 1452 : pp = 124.63385772705078 1370 / 1452 : pp = 124.58133697509766 1380 / 1452 : pp = 124.55769348144531 1390 / 1452 : pp = 124.54011535644531 1400 / 1452 : pp = 124.4884033203125 1410 / 1452 : pp = 124.51226806640625 1420 / 1452 : pp = 124.49683380126953 1430 / 1452 : pp = 124.4754638671875 1440 / 1452 : pp = 124.50164031982422 1450 / 1452 : pp = 124.50894165039062 0 / 115 : pp = 230.8488006591797 10 / 115 : pp = 209.2509002685547 20 / 115 : pp = 211.68577575683594 30 / 115 : pp = 208.44056701660156 40 / 115 : pp = 207.2039337158203 50 / 115 : pp = 202.1859588623047 60 / 115 : pp = 201.34739685058594 70 / 115 : pp = 197.4251251220703 80 / 115 : pp = 195.2623291015625 90 / 115 : pp = 192.592529296875 100 / 115 : pp = 187.39553833007812 110 / 115 : pp = 184.791259765625 Training perplexity: 124.4933853149414 Validation perplexity:184.32510375976562 Total time : 40.856229066848755 0 / 128 : pp = 184.6475067138672 10 / 128 : pp = 176.8856964111328 20 / 128 : pp = 164.3444366455078 30 / 128 : pp = 167.85472106933594 40 / 128 : pp = 169.25367736816406 50 / 128 : pp = 168.86561584472656 60 / 128 : pp = 168.11801147460938 70 / 128 : pp = 165.4105224609375 80 / 128 : pp = 162.91146850585938 90 / 128 : pp = 161.29742431640625 100 / 128 : pp = 162.45989990234375 110 / 128 : pp = 162.6834716796875 120 / 128 : pp = 164.3359832763672 =-==-==-==-==-= Test perplexity: 164.0149383544922 =-==-==-==-==-=
更详细的内容请参考下面链接
https://github.com/weizhenzhao/cs224d_nlp_problem_set2
今天将的还是cs224d 的problem set2 的第三部分习题，
原来国外大学的系统难度真的如此之大，相比之下还是默默地再天朝继续搬砖吧
下面讲述一下RNN语言建模的数学公式:
给出一串连续的词x1,x2...xt关于预测其后面紧跟的词xt+1的建模方式是:
vj是词库中的某个词。实现一个循环神经网络，此网络利用隐层中的反馈信息对"历史记录"x1,x2...xt进行建模:
$h^{(0)}=h_{0}\epsilon R^{D_{h}}$是隐藏层的初始化向量
$x^{(t)}L$是以$x^{(t)}$one-hot行向量与嵌入矩阵L的乘积
这个one-hot行向量就是当前处理词汇的索引
是词嵌入矩阵,
$L$是词嵌入矩阵
$I$是输入词表征矩阵
$H$是隐藏转换矩阵
$U$是输出词表征矩阵
$b_{1}$ $b_{2}$是偏置值
$d$是词嵌入的维数
|V|代表词库的规模
$D_{h}$是隐层的维数
输出向量
是面向整个词库的概率分布，我们需要最优化交叉熵（非正则化的）的损失率：
使用困惑度来评估语言模型的性能，其定义形式如下：
梯度:
该模型中各个变量进行最优化迭代的时候的梯度如下所示:
初始化所有的上面这些需要训练的参数的值
然后通过对每一个词进行训练，安装上述公司求出每个参数的导数值
然后使用梯度下降方法对其进行更新
将新得到的参数代入到模型中，如果损失的值小于初始设定的值则停止迭代，否则继续进行迭代
下面是一张RNNLM的结构图
上面这张是第二层RNN节点的结构图
上面这张是在RNN的变量上面应用Dropout的结构，降低模型过拟合的误差，第一层RNN的dropout结构
上面这张是第一层RNN的结构图
（注意前方高能，一大批天书即将来袭）
''' Created on 2017年9月26日 @author: weizhen ''' import getpass import sys import time import numpy as np from copy import deepcopy from utils import calculate_perplexity, get_ptb_dataset, Vocab from utils import ptb_iterator, sample import tensorflow as tf from model import LanguageModel from tensorflow.contrib.legacy_seq2seq.python.ops.seq2seq import sequence_loss class Config(object): """储存超参数和数据信息""" batch_size = 64 embed_size = 50 hidden_size = 100 num_steps = 10 max_epochs = 16 early_stopping = 2 dropout = 0.9 lr = 0.001 class RNNLM_Model(LanguageModel): def load_data(self, debug=False): """加载词向量并且训练 train/dev/test 数据""" self.vocab = Vocab() self.vocab.construct(get_ptb_dataset('train')) self.encoded_train = np.array([self.vocab.encode(word) for word in get_ptb_dataset('train')], dtype=np.int32) self.encoded_valid = np.array([self.vocab.encode(word) for word in get_ptb_dataset('valid')], dtype=np.int32) self.encoded_test = np.array([self.vocab.encode(word) for word in get_ptb_dataset('test')]) if debug: num_debug = 1024 self.encoded_train = self.encoded_train[:num_debug] self.encoded_valid = self.encoded_valid[:num_debug] self.encoded_test = self.encoded_test[:num_debug] def add_placeholders(self): """生成placeholder 变量来表示输入的 tensors 这些placeholder 被用来在模型的其他地方被填充 并且在训练的过程中会被填充 input_placeholder:Input placeholder shape (None,num_steps),type tf.int32 labels_placeholder:label placeholder shape (None,num_steps) type tf.float32 dropout_placeholder:dropput value placeholder (scalar), type tf.float32 """ self.input_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Input') self.labels_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Target') self.dropout_placeholder = tf.placeholder(tf.float32, name='Dropout') def add_embedding(self): """添加词嵌入层 Hint : 这一层应该用input_placeholder 来索引词嵌入 Hint : 你或许能发现tf.nn.embedding_lookup 是有用的 Hint : 你或许能发现tf.split , tf.squeeze 是有用的在构造tensor 的输入的时候 Hint : 下面是你需要创建的变量的维度 L:(len(self.vocab),embed_size) Returns: inputs:一个训练次数的列表,每一个元素应该是 一个张量 大小是 (batch_size,embed_size) tf.split(dimension,num_split,input) dimension表示输入张量的哪一个维度, 如果是0就表示对第0维度进行切割, num_split就是切割的数量, 如果是2就表示输入张量被切成2份, 每一份是一个列表 tf.squeeze(input,squeeze_dims=None,name=None) 从tensor中删除所有大小是1的维度 example: t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t))==>[2,3] t is a tensor of shape [1,2,1,3,1,1] shape(squeeze(t,[2,4]))==>[1,2,3,1] tf.nn.embedding_lookup 将词的索引映射到词的向量 """ with tf.device('/cpu:0'): embedding = tf.get_variable('Embedding', [len(self.vocab), self.config.embed_size], trainable=True) inputs = tf.nn.embedding_lookup(embedding, self.input_placeholder) inputs = [tf.squeeze(x, [1]) for x in tf.split(inputs, self.config.num_steps, 1)] return inputs def add_projection(self, rnn_outputs): """添加一个投影层 投影层将隐藏层的表示变换到整个词向量上的分布式表示 Hint:下面是你需要去创建的维度 U(hidden_size,len(vocab)) b_2:(len(vocab),) 参数: rnn_outputs:一个训练次数的列表，每一个元素应该是一个张量 大小是(batch_size,embed_size) Returns: outputs:一个长度的列表，每一个元素是一个张量(batch_size,len(vocab)) """ with tf.variable_scope('Projection'): U = tf.get_variable('Matrix', [self.config.hidden_size, len(self.vocab)]) proj_b = tf.get_variable('Bias', [len(self.vocab)]) outputs = [tf.matmul(o, U) + proj_b for o in rnn_outputs] return outputs def add_loss_op(self, output): """将损失添加到目标函数上面 Hint:使用tensorflow.python.ops.seq2seq.sequence_loss 来实现序列损失 参数: 输出:一个张量 大小是 (None,self.vocab) 返回: 损失：一个0-d大小的张量 """ all_ones = [tf.ones([self.config.batch_size * self.config.num_steps])] cross_entropy = sequence_loss([output], [tf.reshape(self.labels_placeholder, [-1])], all_ones, len(self.vocab)) tf.add_to_collection('total_loss', cross_entropy) loss = tf.add_n(tf.get_collection('total_loss')) return loss def add_training_op(self, loss): """将目标损失添加到计算图上 创建一个优化器并且应用梯度下降到所有的训练变量上面 Hint:使用tf.train.AdamOptimizer 对于这个模型 使用optimizer.minimize() 会返回一个train_op的对象 参数: loss: 损失张量，来自于cross_entropy_loss 交叉熵损失 返回: train_op:训练的目标 """ with tf.variable_scope("Optimizer") as scope: train_op = tf.train.AdamOptimizer(self.config.lr).minimize(loss) return train_op def __init__(self, config): self.config = config self.load_data(debug=False) self.add_placeholders() self.inputs = self.add_embedding() self.rnn_outputs = self.add_model(self.inputs) self.outputs = self.add_projection(self.rnn_outputs) # 我们想去检验下一个词预测得多好 # 我们把o转变成float64 位 因为如果不这样就会有数值问题 # sum(output of softmax) = 1.00000298179 并且不是 1 self.predictions = [tf.nn.softmax(tf.cast(o, 'float64')) for o in self.outputs] # 将输出值转变成 len(vocab) 的大小 output = tf.reshape(tf.concat(self.outputs, 1), [-1, len(self.vocab)]) self.calculate_loss = self.add_loss_op(output) self.train_step = self.add_training_op(self.calculate_loss) def add_model(self, inputs): """创建RNN LM 模型 在下面的实现里面你需要去实现RNN LM 模型的等式 Hint: 使用一个零向量 大小是 (batch_size,hidden_size) 作为初始的RNN的状态 Hint: 将最后RNN输出 作为实例变量 self.final_state Hint : 确保将dropout应用到 输入和输出的 变量上面 Hint : 使用变量域 RNN 来定义 RNN变量 Hint : 表现一个明显的 for-loop 在输入上面 你可以使用scope.reuse_variable() 来确定权重 在每一次迭代都是相同的 确保不会在第一次循环的时候调用这个，因为没有变量会被初始化 Hint : 下面变量的不同的维度 ， 你需要去创建的 H: (hidden_size,hidden_size) I: (embed_size,hidden_size) b_1:(hidden_size,) Args: inputs:一个记录num_steps的列表，里边的每一个元素应该是一个张量 大小是(batch_size,embed_size)的大小 Returns:返回 outputs:一个记录num_steps的列表，里面每一个元素应该是一个张量 大小是(batch_size,hidden_size) """ with tf.variable_scope('InputDropout'): inputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in inputs] with tf.variable_scope('RNN') as scope: self.initial_state = tf.zeros([self.config.batch_size, self.config.hidden_size]) state = self.initial_state rnn_outputs = [] for tstep, current_input in enumerate(inputs): if tstep > 0: scope.reuse_variables() RNN_H = tf.get_variable('HMatrix', [self.config.hidden_size, self.config.hidden_size]) RNN_I = tf.get_variable('IMatrix', [self.config.embed_size, self.config.hidden_size]) RNN_b = tf.get_variable('B', [self.config.hidden_size]) state = tf.nn.sigmoid(tf.matmul(state, RNN_H) + tf.matmul(current_input, RNN_I) + RNN_b) rnn_outputs.append(state) self.final_state = rnn_outputs[-1] with tf.variable_scope('RNNDropout'): rnn_outputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in rnn_outputs] return rnn_outputs def run_epoch(self, session, data, train_op=None, verbose=10): config = self.config dp = config.dropout if not train_op: train_op = tf.no_op() dp = 1 total_steps = sum(1 for x in ptb_iterator(data, config.batch_size, config.num_steps)) total_loss = [] state = self.initial_state.eval() for step, (x, y) in enumerate(ptb_iterator(data, config.batch_size, config.num_steps)): # 我们需要通过初始状态，并且从最终状态中抽取数据来进行填充 # RNN 合适的 历史 feed = {self.input_placeholder: x, self.labels_placeholder: y, self.initial_state: state, self.dropout_placeholder: dp } loss, state, _ = session.run([self.calculate_loss, self.final_state, train_op], feed_dict=feed) total_loss.append(loss) if verbose and step % verbose == 0: sys.stdout.write('\r{} / {} : pp = {} '.format(step, total_steps, np.exp(np.mean(total_loss)))) sys.stdout.flush() if verbose: sys.stdout.write('\r') return np.exp(np.mean(total_loss)) def generate_text(session, model, config, starting_text='<eos>', stop_length=100, stop_tokens=None, temp=1.0): """从模型自动生成文字 Hint:创建一个feed-dictionary 并且使用sess.run()方法去执行这个模型 你会需要使用model.initial_state 作为一个键传递给feed_dict Hint:得到model.final_state 和 model.predictions[-1]. 在add_model()方法中设置model.final_state 。 model.predictions 是在 __init__方法中设置的 Hint:在模型的训练中存储输出的参数值，和预测的y_pred的值 参数: Args: session : tf.Session() object model : Object of type RNNLM Model config : A Config() object starting_text:Initial text passed to model Returns: output : List of word idxs """ state = model.initial_state.eval() # Imagine tokens as a batch size of one, length of len(tokens[0]) tokens = [model.vocab.encode(word) for word in starting_text.split()] for i in range(stop_length): feed = {model.input_placeholder: [tokens[-1:]], model.initial_state: state, model.dropout_placeholder: 1} state, y_pred = session.run([model.final_state, model.predictions[-1]], feed_dict=feed) next_word_idx = sample(y_pred[0], temperature=temp) tokens.append(next_word_idx) if stop_tokens and model.vocab.decode(tokens[-1]) in stop_tokens: break output = [model.vocab.decode(word_idx) for word_idx in tokens] return output def generate_sentence(session, model, config, *args, **kwargs): """方便从模型来生成句子""" return generate_text(session, model, config, *args, stop_tokens=['<eos>'], **kwargs) def test_RNNLM(): config = Config() gen_config = deepcopy(config) gen_config.batch_size = gen_config.num_steps = 1 # 创建训练模型，并且生成模型 with tf.variable_scope('RNNLM',reuse=None) as scope: model = RNNLM_Model(config) # 这个指示gen_model来重新使用相同的变量作为以上的模型 scope.reuse_variables() gen_model = RNNLM_Model(gen_config) init = tf.global_variables_initializer() saver = tf.train.Saver() with tf.Session() as session: best_val_pp = float('inf') best_val_epoch = 0 session.run(init) for epoch in range(config.max_epochs): print('Epoch {0}'.format(epoch)) start = time.time() train_pp = model.run_epoch(session, model.encoded_train, train_op=model.train_step) valid_pp = model.run_epoch(session, model.encoded_valid) print('Training perplexity: {0}'.format(train_pp)) print('Validation perplexity:{0}'.format(valid_pp)) if valid_pp < best_val_pp: best_val_pp = valid_pp best_val_epoch = epoch saver.save(session, './ptb_rnnlm.weights') if epoch - best_val_epoch > config.early_stopping: break print('Total time : {0}'.format(time.time() - start)) saver.restore(session, 'ptb_rnnlm.weights') test_pp = model.run_epoch(session, model.encoded_test) print('=-=' * 5) print('Test perplexity: {0} '.format(test_pp)) print('=-=' * 5) starting_text = 'in palo alto' while starting_text: print(' '.join(generate_sentence(session, gen_model, gen_config, starting_text=starting_text, temp=1.0))) #starting_text = raw_input('>') if __name__ == "__main__": test_RNNLM()
（其实也不算是天书啦，比高数简单多啦，比数学分析那是简单了好几十万倍了呀）
下面是训练的Log
1380 / 1452 : pp = 266.20892333984375 1390 / 1452 : pp = 265.94439697265625 1400 / 1452 : pp = 265.66845703125 1410 / 1452 : pp = 265.5393981933594 1420 / 1452 : pp = 265.32489013671875 1430 / 1452 : pp = 265.2019348144531 1440 / 1452 : pp = 265.13720703125 1450 / 1452 : pp = 264.954833984375 0 / 115 : pp = 296.9217224121094 10 / 115 : pp = 282.02130126953125 20 / 115 : pp = 279.76800537109375 30 / 115 : pp = 276.4101257324219 40 / 115 : pp = 276.2939147949219 50 / 115 : pp = 270.73565673828125 60 / 115 : pp = 269.88134765625 70 / 115 : pp = 266.8675231933594 80 / 115 : pp = 263.6731872558594 90 / 115 : pp = 260.8569030761719 100 / 115 : pp = 256.3356628417969 110 / 115 : pp = 255.1026611328125 Training perplexity: 264.9092102050781 Validation perplexity:254.84902954101562 Total time : 41.65332388877869 Epoch 3 0 / 1452 : pp = 327.0847473144531 10 / 1452 : pp = 273.9620056152344 20 / 1452 : pp = 270.22943115234375 30 / 1452 : pp = 263.5213317871094 40 / 1452 : pp = 264.0644836425781 50 / 1452 : pp = 258.6029968261719 60 / 1452 : pp = 257.04290771484375 70 / 1452 : pp = 257.59161376953125 80 / 1452 : pp = 256.7600402832031 90 / 1452 : pp = 254.5120391845703 100 / 1452 : pp = 252.44725036621094 110 / 1452 : pp = 250.13954162597656 120 / 1452 : pp = 249.91647338867188 130 / 1452 : pp = 249.50460815429688 140 / 1452 : pp = 247.67440795898438 150 / 1452 : pp = 247.19090270996094 160 / 1452 : pp = 247.8919219970703 170 / 1452 : pp = 247.54322814941406 180 / 1452 : pp = 246.17623901367188 190 / 1452 : pp = 245.78330993652344 200 / 1452 : pp = 246.80552673339844 210 / 1452 : pp = 246.3059844970703 220 / 1452 : pp = 246.19021606445312 230 / 1452 : pp = 246.70140075683594 240 / 1452 : pp = 246.3099822998047 250 / 1452 : pp = 245.1745147705078 260 / 1452 : pp = 244.17384338378906 270 / 1452 : pp = 242.57363891601562 280 / 1452 : pp = 242.8500213623047 290 / 1452 : pp = 243.0492706298828 300 / 1452 : pp = 243.1466522216797 310 / 1452 : pp = 242.89044189453125 320 / 1452 : pp = 243.08045959472656 330 / 1452 : pp = 243.32235717773438 340 / 1452 : pp = 242.34715270996094 350 / 1452 : pp = 242.80972290039062 360 / 1452 : pp = 242.5345458984375 370 / 1452 : pp = 242.0083465576172 380 / 1452 : pp = 241.22708129882812 390 / 1452 : pp = 241.24398803710938 400 / 1452 : pp = 240.63473510742188 410 / 1452 : pp = 240.94094848632812 420 / 1452 : pp = 241.19717407226562 430 / 1452 : pp = 240.8896026611328 440 / 1452 : pp = 240.7772979736328 450 / 1452 : pp = 240.45913696289062 460 / 1452 : pp = 240.06674194335938 470 / 1452 : pp = 239.42198181152344 480 / 1452 : pp = 238.39271545410156 490 / 1452 : pp = 238.0517120361328 500 / 1452 : pp = 237.31752014160156 510 / 1452 : pp = 237.1197967529297 520 / 1452 : pp = 236.64865112304688 530 / 1452 : pp = 236.004638671875 540 / 1452 : pp = 235.192626953125 550 / 1452 : pp = 234.6700439453125 560 / 1452 : pp = 234.1914825439453 570 / 1452 : pp = 233.80899047851562 580 / 1452 : pp = 233.3753662109375 590 / 1452 : pp = 232.8699188232422 600 / 1452 : pp = 232.2629852294922 610 / 1452 : pp = 231.8668212890625 620 / 1452 : pp = 231.478515625 630 / 1452 : pp = 231.0444793701172 640 / 1452 : pp = 231.2737579345703 650 / 1452 : pp = 231.28114318847656 660 / 1452 : pp = 231.4324951171875 670 / 1452 : pp = 231.48513793945312 680 / 1452 : pp = 231.45932006835938 690 / 1452 : pp = 231.17738342285156 700 / 1452 : pp = 231.00570678710938 710 / 1452 : pp = 231.03810119628906 720 / 1452 : pp = 230.96131896972656 730 / 1452 : pp = 230.91110229492188 740 / 1452 : pp = 231.13539123535156 750 / 1452 : pp = 231.04393005371094 760 / 1452 : pp = 231.03489685058594 770 / 1452 : pp = 231.19744873046875 780 / 1452 : pp = 231.26625061035156 790 / 1452 : pp = 231.38714599609375 800 / 1452 : pp = 231.24441528320312 810 / 1452 : pp = 231.16824340820312 820 / 1452 : pp = 231.11831665039062 830 / 1452 : pp = 231.34886169433594 840 / 1452 : pp = 231.221923828125 850 / 1452 : pp = 231.2562255859375 860 / 1452 : pp = 231.26492309570312 870 / 1452 : pp = 231.1961212158203 880 / 1452 : pp = 231.30506896972656 890 / 1452 : pp = 231.24728393554688 900 / 1452 : pp = 231.15744018554688 910 / 1452 : pp = 231.20175170898438 920 / 1452 : pp = 231.25534057617188 930 / 1452 : pp = 231.09461975097656 940 / 1452 : pp = 231.12612915039062 950 / 1452 : pp = 231.0475616455078 960 / 1452 : pp = 230.86056518554688 970 / 1452 : pp = 230.80377197265625 980 / 1452 : pp = 230.4598846435547 990 / 1452 : pp = 230.24559020996094 1000 / 1452 : pp = 229.91030883789062 1010 / 1452 : pp = 229.9349822998047 1020 / 1452 : pp = 230.01470947265625 1030 / 1452 : pp = 229.8909149169922 1040 / 1452 : pp = 229.9403533935547 1050 / 1452 : pp = 229.84815979003906 1060 / 1452 : pp = 229.60377502441406 1070 / 1452 : pp = 229.74647521972656 1080 / 1452 : pp = 229.80410766601562 1090 / 1452 : pp = 229.78733825683594 1100 / 1452 : pp = 229.64549255371094 1110 / 1452 : pp = 229.26255798339844 1120 / 1452 : pp = 229.00262451171875 1130 / 1452 : pp = 228.6716766357422 1140 / 1452 : pp = 228.55067443847656 1150 / 1452 : pp = 228.61563110351562 1160 / 1452 : pp = 228.50958251953125 1170 / 1452 : pp = 228.3498992919922 1180 / 1452 : pp = 228.29786682128906 1190 / 1452 : pp = 228.33204650878906 1200 / 1452 : pp = 228.27369689941406 1210 / 1452 : pp = 228.11831665039062 1220 / 1452 : pp = 228.21775817871094 1230 / 1452 : pp = 228.3170166015625 1240 / 1452 : pp = 228.22134399414062 1250 / 1452 : pp = 228.3769073486328 1260 / 1452 : pp = 228.37527465820312 1270 / 1452 : pp = 228.33694458007812 1280 / 1452 : pp = 228.27108764648438 1290 / 1452 : pp = 228.1731414794922 1300 / 1452 : pp = 228.12200927734375 1310 / 1452 : pp = 228.10275268554688 1320 / 1452 : pp = 227.9289093017578 1330 / 1452 : pp = 227.77723693847656 1340 / 1452 : pp = 227.79623413085938 1350 / 1452 : pp = 227.7408447265625 1360 / 1452 : pp = 227.72586059570312 1370 / 1452 : pp = 227.49728393554688 1380 / 1452 : pp = 227.37940979003906 1390 / 1452 : pp = 227.20166015625 1400 / 1452 : pp = 227.018310546875 1410 / 1452 : pp = 226.95651245117188 1420 / 1452 : pp = 226.8065643310547 1430 / 1452 : pp = 226.7261199951172 1440 / 1452 : pp = 226.7193145751953 1450 / 1452 : pp = 226.61068725585938 0 / 115 : pp = 269.342041015625 10 / 115 : pp = 255.03016662597656 20 / 115 : pp = 253.8992919921875 30 / 115 : pp = 251.04025268554688 40 / 115 : pp = 250.51756286621094 50 / 115 : pp = 245.3595428466797 60 / 115 : pp = 244.4713897705078 70 / 115 : pp = 241.2674560546875 80 / 115 : pp = 238.3473663330078 90 / 115 : pp = 235.56423950195312 100 / 115 : pp = 231.2281036376953 110 / 115 : pp = 229.8423614501953 Training perplexity: 226.5760040283203 Validation perplexity:229.59939575195312 Total time : 42.202677726745605 Epoch 4 0 / 1452 : pp = 282.2423095703125 10 / 1452 : pp = 240.16258239746094 20 / 1452 : pp = 236.12203979492188 30 / 1452 : pp = 230.3953857421875 40 / 1452 : pp = 231.8789825439453 50 / 1452 : pp = 227.26612854003906 60 / 1452 : pp = 226.22061157226562 70 / 1452 : pp = 227.01885986328125 80 / 1452 : pp = 226.2459716796875 90 / 1452 : pp = 224.3211669921875 100 / 1452 : pp = 222.65615844726562 110 / 1452 : pp = 220.70326232910156 120 / 1452 : pp = 220.42288208007812 130 / 1452 : pp = 219.8100128173828 140 / 1452 : pp = 218.04432678222656 150 / 1452 : pp = 217.31639099121094 160 / 1452 : pp = 217.86349487304688 170 / 1452 : pp = 217.46597290039062 180 / 1452 : pp = 216.3349151611328 190 / 1452 : pp = 216.12240600585938 200 / 1452 : pp = 216.97842407226562 210 / 1452 : pp = 216.51014709472656 220 / 1452 : pp = 216.46751403808594 230 / 1452 : pp = 216.80126953125 240 / 1452 : pp = 216.45965576171875 250 / 1452 : pp = 215.5008544921875 260 / 1452 : pp = 214.62210083007812 270 / 1452 : pp = 213.29183959960938 280 / 1452 : pp = 213.5621337890625 290 / 1452 : pp = 213.80657958984375 300 / 1452 : pp = 213.8963165283203 310 / 1452 : pp = 213.60653686523438 320 / 1452 : pp = 213.85877990722656 330 / 1452 : pp = 214.07345581054688 340 / 1452 : pp = 213.25421142578125 350 / 1452 : pp = 213.68019104003906 360 / 1452 : pp = 213.41717529296875 370 / 1452 : pp = 213.04920959472656 380 / 1452 : pp = 212.39019775390625 390 / 1452 : pp = 212.4908905029297 400 / 1452 : pp = 212.01914978027344 410 / 1452 : pp = 212.36903381347656 420 / 1452 : pp = 212.6802520751953 430 / 1452 : pp = 212.42697143554688 440 / 1452 : pp = 212.42990112304688 450 / 1452 : pp = 212.14524841308594 460 / 1452 : pp = 211.7836151123047 470 / 1452 : pp = 211.17282104492188 480 / 1452 : pp = 210.27903747558594 490 / 1452 : pp = 209.95211791992188 500 / 1452 : pp = 209.28302001953125 510 / 1452 : pp = 209.1029815673828 520 / 1452 : pp = 208.73855590820312 530 / 1452 : pp = 208.19700622558594 540 / 1452 : pp = 207.4554443359375 550 / 1452 : pp = 207.0062255859375 560 / 1452 : pp = 206.59739685058594 570 / 1452 : pp = 206.27874755859375 580 / 1452 : pp = 205.87144470214844 590 / 1452 : pp = 205.43545532226562 600 / 1452 : pp = 204.90940856933594 610 / 1452 : pp = 204.5686798095703 620 / 1452 : pp = 204.22862243652344 630 / 1452 : pp = 203.8448028564453 640 / 1452 : pp = 204.06576538085938 650 / 1452 : pp = 204.0941925048828 660 / 1452 : pp = 204.22103881835938 670 / 1452 : pp = 204.289794921875 680 / 1452 : pp = 204.3115234375 690 / 1452 : pp = 204.10284423828125 700 / 1452 : pp = 203.99757385253906 710 / 1452 : pp = 204.04971313476562 720 / 1452 : pp = 204.03152465820312 730 / 1452 : pp = 203.99046325683594 740 / 1452 : pp = 204.19786071777344 750 / 1452 : pp = 204.1642608642578 760 / 1452 : pp = 204.19435119628906 770 / 1452 : pp = 204.37786865234375 780 / 1452 : pp = 204.4965057373047 790 / 1452 : pp = 204.6479034423828 800 / 1452 : pp = 204.56117248535156 810 / 1452 : pp = 204.52284240722656 820 / 1452 : pp = 204.50978088378906 830 / 1452 : pp = 204.7531280517578 840 / 1452 : pp = 204.64468383789062 850 / 1452 : pp = 204.71348571777344 860 / 1452 : pp = 204.7399444580078 870 / 1452 : pp = 204.69406127929688 880 / 1452 : pp = 204.7965850830078 890 / 1452 : pp = 204.7594757080078 900 / 1452 : pp = 204.71446228027344 910 / 1452 : pp = 204.7590789794922 920 / 1452 : pp = 204.85772705078125 930 / 1452 : pp = 204.7428741455078 940 / 1452 : pp = 204.8068389892578 950 / 1452 : pp = 204.75791931152344 960 / 1452 : pp = 204.63815307617188 970 / 1452 : pp = 204.60760498046875 980 / 1452 : pp = 204.34347534179688 990 / 1452 : pp = 204.151611328125 1000 / 1452 : pp = 203.8665771484375 1010 / 1452 : pp = 203.9164581298828 1020 / 1452 : pp = 204.0184783935547 1030 / 1452 : pp = 203.95166015625 1040 / 1452 : pp = 204.03045654296875 1050 / 1452 : pp = 203.95846557617188 1060 / 1452 : pp = 203.77114868164062 1070 / 1452 : pp = 203.93260192871094 1080 / 1452 : pp = 204.00048828125 1090 / 1452 : pp = 204.00233459472656 1100 / 1452 : pp = 203.8960418701172 1110 / 1452 : pp = 203.5987548828125 1120 / 1452 : pp = 203.38392639160156 1130 / 1452 : pp = 203.08872985839844 1140 / 1452 : pp = 203.01272583007812 1150 / 1452 : pp = 203.0865936279297 1160 / 1452 : pp = 203.02308654785156 1170 / 1452 : pp = 202.9125518798828 1180 / 1452 : pp = 202.9097442626953 1190 / 1452 : pp = 202.98252868652344 1200 / 1452 : pp = 202.95387268066406 1210 / 1452 : pp = 202.851318359375 1220 / 1452 : pp = 202.97671508789062 1230 / 1452 : pp = 203.1051025390625 1240 / 1452 : pp = 203.0526123046875 1250 / 1452 : pp = 203.21417236328125 1260 / 1452 : pp = 203.23617553710938 1270 / 1452 : pp = 203.22802734375 1280 / 1452 : pp = 203.20846557617188 1290 / 1452 : pp = 203.15362548828125 1300 / 1452 : pp = 203.14315795898438 1310 / 1452 : pp = 203.15264892578125 1320 / 1452 : pp = 203.02801513671875 1330 / 1452 : pp = 202.92977905273438 1340 / 1452 : pp = 202.95484924316406 1350 / 1452 : pp = 202.9335479736328 1360 / 1452 : pp = 202.955322265625 1370 / 1452 : pp = 202.7740478515625 1380 / 1452 : pp = 202.68569946289062 1390 / 1452 : pp = 202.55816650390625 1400 / 1452 : pp = 202.41651916503906 1410 / 1452 : pp = 202.38494873046875 1420 / 1452 : pp = 202.27593994140625 1430 / 1452 : pp = 202.21826171875 1440 / 1452 : pp = 202.23272705078125 1450 / 1452 : pp = 202.16099548339844 0 / 115 : pp = 253.23211669921875 10 / 115 : pp = 237.62506103515625 20 / 115 : pp = 237.60557556152344 30 / 115 : pp = 234.9273223876953 40 / 115 : pp = 234.30519104003906 50 / 115 : pp = 229.43960571289062 60 / 115 : pp = 228.6050567626953 70 / 115 : pp = 225.2646484375 80 / 115 : pp = 222.55935668945312 90 / 115 : pp = 219.83255004882812 100 / 115 : pp = 215.5491485595703 110 / 115 : pp = 214.07937622070312 Training perplexity: 202.1349639892578 Validation perplexity:213.85256958007812 Total time : 42.10724234580994 Epoch 5 0 / 1452 : pp = 255.92384338378906 10 / 1452 : pp = 219.5322265625 20 / 1452 : pp = 214.36212158203125 30 / 1452 : pp = 209.12620544433594 40 / 1452 : pp = 210.04193115234375 50 / 1452 : pp = 205.77398681640625 60 / 1452 : pp = 204.8201141357422 70 / 1452 : pp = 205.3955841064453 80 / 1452 : pp = 204.8386688232422 90 / 1452 : pp = 203.21194458007812 100 / 1452 : pp = 201.87643432617188 110 / 1452 : pp = 200.10122680664062 120 / 1452 : pp = 199.82012939453125 130 / 1452 : pp = 199.11192321777344 140 / 1452 : pp = 197.51919555664062 150 / 1452 : pp = 197.03567504882812 160 / 1452 : pp = 197.4231414794922 170 / 1452 : pp = 197.09571838378906 180 / 1452 : pp = 196.17665100097656 190 / 1452 : pp = 196.0064697265625 200 / 1452 : pp = 196.7347869873047 210 / 1452 : pp = 196.3063507080078 220 / 1452 : pp = 196.21388244628906 230 / 1452 : pp = 196.5252227783203 240 / 1452 : pp = 196.203125 250 / 1452 : pp = 195.3251953125 260 / 1452 : pp = 194.53335571289062 270 / 1452 : pp = 193.3546142578125 280 / 1452 : pp = 193.59420776367188 290 / 1452 : pp = 193.83297729492188 300 / 1452 : pp = 193.98489379882812 310 / 1452 : pp = 193.68414306640625 320 / 1452 : pp = 193.89065551757812 330 / 1452 : pp = 194.0518798828125 340 / 1452 : pp = 193.32888793945312 350 / 1452 : pp = 193.76219177246094 360 / 1452 : pp = 193.56106567382812 370 / 1452 : pp = 193.28179931640625 380 / 1452 : pp = 192.7037811279297 390 / 1452 : pp = 192.8145294189453 400 / 1452 : pp = 192.43325805664062 410 / 1452 : pp = 192.81527709960938 420 / 1452 : pp = 193.13760375976562 430 / 1452 : pp = 192.9148712158203 440 / 1452 : pp = 192.92526245117188 450 / 1452 : pp = 192.70083618164062 460 / 1452 : pp = 192.36647033691406 470 / 1452 : pp = 191.85394287109375 480 / 1452 : pp = 191.07244873046875 490 / 1452 : pp = 190.75401306152344 500 / 1452 : pp = 190.1843719482422 510 / 1452 : pp = 190.03334045410156 520 / 1452 : pp = 189.72938537597656 530 / 1452 : pp = 189.25889587402344 540 / 1452 : pp = 188.59315490722656 550 / 1452 : pp = 188.19313049316406 560 / 1452 : pp = 187.80621337890625 570 / 1452 : pp = 187.5229034423828 580 / 1452 : pp = 187.1091766357422 590 / 1452 : pp = 186.72592163085938 600 / 1452 : pp = 186.2238006591797 610 / 1452 : pp = 185.89695739746094 620 / 1452 : pp = 185.60989379882812 630 / 1452 : pp = 185.2689208984375 640 / 1452 : pp = 185.47567749023438 650 / 1452 : pp = 185.5127410888672 660 / 1452 : pp = 185.64627075195312 670 / 1452 : pp = 185.71311950683594 680 / 1452 : pp = 185.72569274902344 690 / 1452 : pp = 185.56459045410156 700 / 1452 : pp = 185.48681640625 710 / 1452 : pp = 185.5458221435547 720 / 1452 : pp = 185.5598907470703 730 / 1452 : pp = 185.5335235595703 740 / 1452 : pp = 185.73995971679688 750 / 1452 : pp = 185.744384765625 760 / 1452 : pp = 185.81268310546875 770 / 1452 : pp = 186.00088500976562 780 / 1452 : pp = 186.14443969726562 790 / 1452 : pp = 186.30764770507812 800 / 1452 : pp = 186.2595977783203 810 / 1452 : pp = 186.23028564453125 820 / 1452 : pp = 186.23997497558594 830 / 1452 : pp = 186.49057006835938 840 / 1452 : pp = 186.43331909179688 850 / 1452 : pp = 186.48887634277344 860 / 1452 : pp = 186.51502990722656 870 / 1452 : pp = 186.5167999267578 880 / 1452 : pp = 186.62400817871094 890 / 1452 : pp = 186.6103973388672 900 / 1452 : pp = 186.58111572265625 910 / 1452 : pp = 186.64126586914062 920 / 1452 : pp = 186.7366180419922 930 / 1452 : pp = 186.65719604492188 940 / 1452 : pp = 186.71755981445312 950 / 1452 : pp = 186.6977996826172 960 / 1452 : pp = 186.62774658203125 970 / 1452 : pp = 186.62115478515625 980 / 1452 : pp = 186.3773193359375 990 / 1452 : pp = 186.23109436035156 1000 / 1452 : pp = 185.99227905273438 1010 / 1452 : pp = 186.0488739013672 1020 / 1452 : pp = 186.1744384765625 1030 / 1452 : pp = 186.1162109375 1040 / 1452 : pp = 186.18899536132812 1050 / 1452 : pp = 186.1549072265625 1060 / 1452 : pp = 186.01419067382812 1070 / 1452 : pp = 186.17364501953125 1080 / 1452 : pp = 186.27061462402344 1090 / 1452 : pp = 186.28428649902344 1100 / 1452 : pp = 186.2150115966797 1110 / 1452 : pp = 185.95103454589844 1120 / 1452 : pp = 185.77423095703125 1130 / 1452 : pp = 185.5232696533203 1140 / 1452 : pp = 185.4607391357422 1150 / 1452 : pp = 185.56077575683594 1160 / 1452 : pp = 185.53343200683594 1170 / 1452 : pp = 185.46453857421875 1180 / 1452 : pp = 185.4741668701172 1190 / 1452 : pp = 185.5594482421875 1200 / 1452 : pp = 185.53785705566406 1210 / 1452 : pp = 185.4576416015625 1220 / 1452 : pp = 185.5943145751953 1230 / 1452 : pp = 185.7483673095703 1240 / 1452 : pp = 185.70762634277344 1250 / 1452 : pp = 185.8568115234375 1260 / 1452 : pp = 185.90635681152344 1270 / 1452 : pp = 185.8961639404297 1280 / 1452 : pp = 185.89199829101562 1290 / 1452 : pp = 185.85911560058594 1300 / 1452 : pp = 185.86097717285156 1310 / 1452 : pp = 185.88739013671875 1320 / 1452 : pp = 185.79248046875 1330 / 1452 : pp = 185.69700622558594 1340 / 1452 : pp = 185.7310028076172 1350 / 1452 : pp = 185.72613525390625 1360 / 1452 : pp = 185.76829528808594 1370 / 1452 : pp = 185.6322021484375 1380 / 1452 : pp = 185.56378173828125 1390 / 1452 : pp = 185.4654998779297 1400 / 1452 : pp = 185.35110473632812 1410 / 1452 : pp = 185.33917236328125 1420 / 1452 : pp = 185.2509002685547 1430 / 1452 : pp = 185.20436096191406 1440 / 1452 : pp = 185.2254638671875 1450 / 1452 : pp = 185.16542053222656 0 / 115 : pp = 242.26800537109375 10 / 115 : pp = 226.12258911132812 20 / 115 : pp = 226.4702606201172 30 / 115 : pp = 223.982666015625 40 / 115 : pp = 223.376953125 50 / 115 : pp = 218.65716552734375 60 / 115 : pp = 217.95306396484375 70 / 115 : pp = 214.5392303466797 80 / 115 : pp = 212.07525634765625 90 / 115 : pp = 209.40631103515625 100 / 115 : pp = 205.1455078125 110 / 115 : pp = 203.6289520263672 Training perplexity: 185.14476013183594 Validation perplexity:203.3822784423828 Total time : 42.47052240371704 Epoch 6 0 / 1452 : pp = 233.56707763671875 10 / 1452 : pp = 202.6468505859375 20 / 1452 : pp = 198.2734375 30 / 1452 : pp = 193.47442626953125 40 / 1452 : pp = 195.17147827148438 50 / 1452 : pp = 191.5596923828125 60 / 1452 : pp = 190.4825897216797 70 / 1452 : pp = 191.07681274414062 80 / 1452 : pp = 190.339599609375 90 / 1452 : pp = 188.98277282714844 100 / 1452 : pp = 187.74757385253906 110 / 1452 : pp = 186.10104370117188 120 / 1452 : pp = 185.7500457763672 130 / 1452 : pp = 184.90707397460938 140 / 1452 : pp = 183.340087890625 150 / 1452 : pp = 182.70840454101562 160 / 1452 : pp = 183.1043701171875 170 / 1452 : pp = 182.69776916503906 180 / 1452 : pp = 181.88400268554688 190 / 1452 : pp = 181.8062286376953 200 / 1452 : pp = 182.4969940185547 210 / 1452 : pp = 182.10572814941406 220 / 1452 : pp = 181.9981689453125 230 / 1452 : pp = 182.3802490234375 240 / 1452 : pp = 182.03636169433594 250 / 1452 : pp = 181.23712158203125 260 / 1452 : pp = 180.53726196289062 270 / 1452 : pp = 179.53567504882812 280 / 1452 : pp = 179.70208740234375 290 / 1452 : pp = 179.977783203125 300 / 1452 : pp = 180.16600036621094 310 / 1452 : pp = 179.87294006347656 320 / 1452 : pp = 180.11849975585938 330 / 1452 : pp = 180.31838989257812 340 / 1452 : pp = 179.56759643554688 350 / 1452 : pp = 179.97134399414062 360 / 1452 : pp = 179.80030822753906 370 / 1452 : pp = 179.52085876464844 380 / 1452 : pp = 178.98228454589844 390 / 1452 : pp = 179.0868682861328 400 / 1452 : pp = 178.74569702148438 410 / 1452 : pp = 179.1776580810547 420 / 1452 : pp = 179.5055389404297 430 / 1452 : pp = 179.3883056640625 440 / 1452 : pp = 179.42279052734375 450 / 1452 : pp = 179.2106475830078 460 / 1452 : pp = 178.85311889648438 470 / 1452 : pp = 178.33840942382812 480 / 1452 : pp = 177.60350036621094 490 / 1452 : pp = 177.30335998535156 500 / 1452 : pp = 176.72222900390625 510 / 1452 : pp = 176.6067352294922 520 / 1452 : pp = 176.33998107910156 530 / 1452 : pp = 175.93162536621094 540 / 1452 : pp = 175.30657958984375 550 / 1452 : pp = 174.9462432861328 560 / 1452 : pp = 174.5836639404297 570 / 1452 : pp = 174.31431579589844 580 / 1452 : pp = 173.92300415039062 590 / 1452 : pp = 173.55856323242188 600 / 1452 : pp = 173.08277893066406 610 / 1452 : pp = 172.75930786132812 620 / 1452 : pp = 172.53192138671875 630 / 1452 : pp = 172.20652770996094 640 / 1452 : pp = 172.37454223632812 650 / 1452 : pp = 172.39845275878906 660 / 1452 : pp = 172.52255249023438 670 / 1452 : pp = 172.60935974121094 680 / 1452 : pp = 172.6611328125 690 / 1452 : pp = 172.53118896484375 700 / 1452 : pp = 172.4709014892578 710 / 1452 : pp = 172.5406494140625 720 / 1452 : pp = 172.55447387695312 730 / 1452 : pp = 172.5330047607422 740 / 1452 : pp = 172.7061767578125 750 / 1452 : pp = 172.71054077148438 760 / 1452 : pp = 172.77743530273438 770 / 1452 : pp = 172.95481872558594 780 / 1452 : pp = 173.11265563964844 790 / 1452 : pp = 173.2832794189453 800 / 1452 : pp = 173.2537841796875 810 / 1452 : pp = 173.22164916992188 820 / 1452 : pp = 173.24148559570312 830 / 1452 : pp = 173.48228454589844 840 / 1452 : pp = 173.43753051757812 850 / 1452 : pp = 173.505615234375 860 / 1452 : pp = 173.5214080810547 870 / 1452 : pp = 173.5009002685547 880 / 1452 : pp = 173.6202392578125 890 / 1452 : pp = 173.622802734375 900 / 1452 : pp = 173.5987091064453 910 / 1452 : pp = 173.68316650390625 920 / 1452 : pp = 173.77330017089844 930 / 1452 : pp = 173.72018432617188 940 / 1452 : pp = 173.79351806640625 950 / 1452 : pp = 173.7653350830078 960 / 1452 : pp = 173.7102508544922 970 / 1452 : pp = 173.69766235351562 980 / 1452 : pp = 173.4836883544922 990 / 1452 : pp = 173.3550262451172 1000 / 1452 : pp = 173.14816284179688 1010 / 1452 : pp = 173.20777893066406 1020 / 1452 : pp = 173.3390655517578 1030 / 1452 : pp = 173.2884063720703 1040 / 1452 : pp = 173.38015747070312 1050 / 1452 : pp = 173.35592651367188 1060 / 1452 : pp = 173.2260284423828 1070 / 1452 : pp = 173.39321899414062 1080 / 1452 : pp = 173.4879913330078 1090 / 1452 : pp = 173.5231475830078 1100 / 1452 : pp = 173.47177124023438 1110 / 1452 : pp = 173.24453735351562 1120 / 1452 : pp = 173.09408569335938 1130 / 1452 : pp = 172.86627197265625 1140 / 1452 : pp = 172.8234100341797 1150 / 1452 : pp = 172.92843627929688 1160 / 1452 : pp = 172.90065002441406 1170 / 1452 : pp = 172.8550567626953 1180 / 1452 : pp = 172.8810272216797 1190 / 1452 : pp = 172.97312927246094 1200 / 1452 : pp = 172.9776611328125 1210 / 1452 : pp = 172.89413452148438 1220 / 1452 : pp = 173.0257568359375 1230 / 1452 : pp = 173.1847381591797 1240 / 1452 : pp = 173.1756591796875 1250 / 1452 : pp = 173.32138061523438 1260 / 1452 : pp = 173.37229919433594 1270 / 1452 : pp = 173.36891174316406 1280 / 1452 : pp = 173.36337280273438 1290 / 1452 : pp = 173.3444366455078 1300 / 1452 : pp = 173.36138916015625 1310 / 1452 : pp = 173.4015655517578 1320 / 1452 : pp = 173.31790161132812 1330 / 1452 : pp = 173.24710083007812 1340 / 1452 : pp = 173.27212524414062 1350 / 1452 : pp = 173.27674865722656 1360 / 1452 : pp = 173.32749938964844 1370 / 1452 : pp = 173.20472717285156 1380 / 1452 : pp = 173.14889526367188 1390 / 1452 : pp = 173.0755157470703 1400 / 1452 : pp = 172.9678497314453 1410 / 1452 : pp = 172.9612579345703 1420 / 1452 : pp = 172.8872833251953 1430 / 1452 : pp = 172.84805297851562 1440 / 1452 : pp = 172.87252807617188 1450 / 1452 : pp = 172.82505798339844 0 / 115 : pp = 236.35635375976562 10 / 115 : pp = 219.06166076660156 20 / 115 : pp = 219.7670440673828 30 / 115 : pp = 217.33587646484375 40 / 115 : pp = 216.6626739501953 50 / 115 : pp = 212.04734802246094 60 / 115 : pp = 211.42068481445312 70 / 115 : pp = 207.9592742919922 80 / 115 : pp = 205.6216583251953 90 / 115 : pp = 202.93597412109375 100 / 115 : pp = 198.62583923339844 110 / 115 : pp = 196.97216796875 Training perplexity: 172.80404663085938 Validation perplexity:196.6871337890625 Total time : 41.52522921562195 Epoch 7 0 / 1452 : pp = 219.23231506347656 10 / 1452 : pp = 192.07225036621094 20 / 1452 : pp = 187.48464965820312 30 / 1452 : pp = 182.9149932861328 40 / 1452 : pp = 184.2945098876953 50 / 1452 : pp = 180.78492736816406 60 / 1452 : pp = 179.377197265625 70 / 1452 : pp = 180.0273895263672 80 / 1452 : pp = 179.2517547607422 90 / 1452 : pp = 177.77540588378906 100 / 1452 : pp = 176.6474151611328 110 / 1452 : pp = 174.84066772460938 120 / 1452 : pp = 174.46890258789062 130 / 1452 : pp = 173.64573669433594 140 / 1452 : pp = 172.17483520507812 150 / 1452 : pp = 171.57041931152344 160 / 1452 : pp = 171.92059326171875 170 / 1452 : pp = 171.5497283935547 180 / 1452 : pp = 170.77249145507812 190 / 1452 : pp = 170.72103881835938 200 / 1452 : pp = 171.336181640625 210 / 1452 : pp = 170.98524475097656 220 / 1452 : pp = 170.99771118164062 230 / 1452 : pp = 171.39918518066406 240 / 1452 : pp = 171.09925842285156 250 / 1452 : pp = 170.39962768554688 260 / 1452 : pp = 169.7328643798828 270 / 1452 : pp = 168.72225952148438 280 / 1452 : pp = 168.92552185058594 290 / 1452 : pp = 169.20147705078125 300 / 1452 : pp = 169.40338134765625 310 / 1452 : pp = 169.12057495117188 320 / 1452 : pp = 169.31236267089844 330 / 1452 : pp = 169.49945068359375 340 / 1452 : pp = 168.8396759033203 350 / 1452 : pp = 169.25917053222656 360 / 1452 : pp = 169.09388732910156 370 / 1452 : pp = 168.84323120117188 380 / 1452 : pp = 168.3832550048828 390 / 1452 : pp = 168.48275756835938 400 / 1452 : pp = 168.19972229003906 410 / 1452 : pp = 168.5838623046875 420 / 1452 : pp = 168.91119384765625 430 / 1452 : pp = 168.80836486816406 440 / 1452 : pp = 168.90264892578125 450 / 1452 : pp = 168.68589782714844 460 / 1452 : pp = 168.3704071044922 470 / 1452 : pp = 167.90394592285156 480 / 1452 : pp = 167.23373413085938 490 / 1452 : pp = 166.9560546875 500 / 1452 : pp = 166.43161010742188 510 / 1452 : pp = 166.320068359375 520 / 1452 : pp = 166.05902099609375 530 / 1452 : pp = 165.71714782714844 540 / 1452 : pp = 165.10398864746094 550 / 1452 : pp = 164.80430603027344 560 / 1452 : pp = 164.4687042236328 570 / 1452 : pp = 164.2272491455078 580 / 1452 : pp = 163.84312438964844 590 / 1452 : pp = 163.46035766601562 600 / 1452 : pp = 163.01559448242188 610 / 1452 : pp = 162.74134826660156 620 / 1452 : pp = 162.50267028808594 630 / 1452 : pp = 162.2018280029297 640 / 1452 : pp = 162.37130737304688 650 / 1452 : pp = 162.3895721435547 660 / 1452 : pp = 162.51351928710938 670 / 1452 : pp = 162.57684326171875 680 / 1452 : pp = 162.6346893310547 690 / 1452 : pp = 162.5135955810547 700 / 1452 : pp = 162.47052001953125 710 / 1452 : pp = 162.539794921875 720 / 1452 : pp = 162.55381774902344 730 / 1452 : pp = 162.5297088623047 740 / 1452 : pp = 162.71652221679688 750 / 1452 : pp = 162.740966796875 760 / 1452 : pp = 162.79754638671875 770 / 1452 : pp = 162.9949951171875 780 / 1452 : pp = 163.17868041992188 790 / 1452 : pp = 163.33055114746094 800 / 1452 : pp = 163.31591796875 810 / 1452 : pp = 163.2859344482422 820 / 1452 : pp = 163.2958984375 830 / 1452 : pp = 163.528564453125 840 / 1452 : pp = 163.47610473632812 850 / 1452 : pp = 163.5260772705078 860 / 1452 : pp = 163.55352783203125 870 / 1452 : pp = 163.55718994140625 880 / 1452 : pp = 163.67523193359375 890 / 1452 : pp = 163.6920166015625 900 / 1452 : pp = 163.67710876464844 910 / 1452 : pp = 163.7476806640625 920 / 1452 : pp = 163.84803771972656 930 / 1452 : pp = 163.8114013671875 940 / 1452 : pp = 163.86663818359375 950 / 1452 : pp = 163.83531188964844 960 / 1452 : pp = 163.79945373535156 970 / 1452 : pp = 163.80320739746094 980 / 1452 : pp = 163.5953369140625 990 / 1452 : pp = 163.48382568359375 1000 / 1452 : pp = 163.2642822265625 1010 / 1452 : pp = 163.32113647460938 1020 / 1452 : pp = 163.44204711914062 1030 / 1452 : pp = 163.40206909179688 1040 / 1452 : pp = 163.4915313720703 1050 / 1452 : pp = 163.47096252441406 1060 / 1452 : pp = 163.3601531982422 1070 / 1452 : pp = 163.5138397216797 1080 / 1452 : pp = 163.6189727783203 1090 / 1452 : pp = 163.6471405029297 1100 / 1452 : pp = 163.60406494140625 1110 / 1452 : pp = 163.40736389160156 1120 / 1452 : pp = 163.26841735839844 1130 / 1452 : pp = 163.0680694580078 1140 / 1452 : pp = 163.04591369628906 1150 / 1452 : pp = 163.15478515625 1160 / 1452 : pp = 163.1380615234375 1170 / 1452 : pp = 163.09303283691406 1180 / 1452 : pp = 163.14149475097656 1190 / 1452 : pp = 163.2374267578125 1200 / 1452 : pp = 163.2394561767578 1210 / 1452 : pp = 163.17835998535156 1220 / 1452 : pp = 163.32347106933594 1230 / 1452 : pp = 163.4639434814453 1240 / 1452 : pp = 163.4611358642578 1250 / 1452 : pp = 163.60687255859375 1260 / 1452 : pp = 163.67227172851562 1270 / 1452 : pp = 163.67515563964844 1280 / 1452 : pp = 163.6881103515625 1290 / 1452 : pp = 163.66648864746094 1300 / 1452 : pp = 163.69287109375 1310 / 1452 : pp = 163.7276153564453 1320 / 1452 : pp = 163.6551055908203 1330 / 1452 : pp = 163.58901977539062 1340 / 1452 : pp = 163.6205291748047 1350 / 1452 : pp = 163.63824462890625 1360 / 1452 : pp = 163.69334411621094 1370 / 1452 : pp = 163.5885467529297 1380 / 1452 : pp = 163.54049682617188 1390 / 1452 : pp = 163.4760284423828 1400 / 1452 : pp = 163.38897705078125 1410 / 1452 : pp = 163.3974609375 1420 / 1452 : pp = 163.35009765625 1430 / 1452 : pp = 163.32191467285156 1440 / 1452 : pp = 163.35220336914062 1450 / 1452 : pp = 163.3201904296875 0 / 115 : pp = 232.2108154296875 10 / 115 : pp = 214.35496520996094 20 / 115 : pp = 215.20510864257812 30 / 115 : pp = 212.82754516601562 40 / 115 : pp = 212.0598907470703 50 / 115 : pp = 207.5095672607422 60 / 115 : pp = 206.86976623535156 70 / 115 : pp = 203.36016845703125 80 / 115 : pp = 201.11538696289062 90 / 115 : pp = 198.52120971679688 100 / 115 : pp = 194.1772003173828 110 / 115 : pp = 192.41224670410156 Training perplexity: 163.29916381835938 Validation perplexity:192.09552001953125 Total time : 41.78096055984497 Epoch 8 0 / 1452 : pp = 201.77548217773438 10 / 1452 : pp = 180.4141082763672 20 / 1452 : pp = 176.41432189941406 30 / 1452 : pp = 172.7764434814453 40 / 1452 : pp = 174.69166564941406 50 / 1452 : pp = 171.2933807373047 60 / 1452 : pp = 170.08010864257812 70 / 1452 : pp = 170.6719512939453 80 / 1452 : pp = 170.07589721679688 90 / 1452 : pp = 168.7478485107422 100 / 1452 : pp = 167.57081604003906 110 / 1452 : pp = 166.06971740722656 120 / 1452 : pp = 165.73374938964844 130 / 1452 : pp = 164.80674743652344 140 / 1452 : pp = 163.32821655273438 150 / 1452 : pp = 162.6752471923828 160 / 1452 : pp = 163.02049255371094 170 / 1452 : pp = 162.64120483398438 180 / 1452 : pp = 161.95529174804688 190 / 1452 : pp = 161.91954040527344 200 / 1452 : pp = 162.5446014404297 210 / 1452 : pp = 162.2645721435547 220 / 1452 : pp = 162.3128662109375 230 / 1452 : pp = 162.65872192382812 240 / 1452 : pp = 162.40948486328125 250 / 1452 : pp = 161.75787353515625 260 / 1452 : pp = 161.15213012695312 270 / 1452 : pp = 160.22256469726562 280 / 1452 : pp = 160.3651123046875 290 / 1452 : pp = 160.63780212402344 300 / 1452 : pp = 160.80026245117188 310 / 1452 : pp = 160.54383850097656 320 / 1452 : pp = 160.7539520263672 330 / 1452 : pp = 160.94317626953125 340 / 1452 : pp = 160.3373565673828 350 / 1452 : pp = 160.71763610839844 360 / 1452 : pp = 160.60960388183594 370 / 1452 : pp = 160.37527465820312 380 / 1452 : pp = 159.92990112304688 390 / 1452 : pp = 160.0165557861328 400 / 1452 : pp = 159.75697326660156 410 / 1452 : pp = 160.15274047851562 420 / 1452 : pp = 160.48390197753906 430 / 1452 : pp = 160.4031982421875 440 / 1452 : pp = 160.4693603515625 450 / 1452 : pp = 160.28016662597656 460 / 1452 : pp = 159.94004821777344 470 / 1452 : pp = 159.48257446289062 480 / 1452 : pp = 158.87998962402344 490 / 1452 : pp = 158.59765625 500 / 1452 : pp = 158.10865783691406 510 / 1452 : pp = 157.96795654296875 520 / 1452 : pp = 157.7591552734375 530 / 1452 : pp = 157.42648315429688 540 / 1452 : pp = 156.85348510742188 550 / 1452 : pp = 156.5618438720703 560 / 1452 : pp = 156.24905395507812 570 / 1452 : pp = 155.9994354248047 580 / 1452 : pp = 155.612060546875 590 / 1452 : pp = 155.25830078125 600 / 1452 : pp = 154.8464813232422 610 / 1452 : pp = 154.5833282470703 620 / 1452 : pp = 154.38040161132812 630 / 1452 : pp = 154.0767364501953 640 / 1452 : pp = 154.2534637451172 650 / 1452 : pp = 154.25875854492188 660 / 1452 : pp = 154.35874938964844 670 / 1452 : pp = 154.4289093017578 680 / 1452 : pp = 154.51412963867188 690 / 1452 : pp = 154.41676330566406 700 / 1452 : pp = 154.37892150878906 710 / 1452 : pp = 154.4234619140625 720 / 1452 : pp = 154.4586639404297 730 / 1452 : pp = 154.4351806640625 740 / 1452 : pp = 154.6002197265625 750 / 1452 : pp = 154.65684509277344 760 / 1452 : pp = 154.73318481445312 770 / 1452 : pp = 154.92935180664062 780 / 1452 : pp = 155.1021728515625 790 / 1452 : pp = 155.24757385253906 800 / 1452 : pp = 155.223876953125 810 / 1452 : pp = 155.2095184326172 820 / 1452 : pp = 155.24009704589844 830 / 1452 : pp = 155.4519500732422 840 / 1452 : pp = 155.3947296142578 850 / 1452 : pp = 155.45306396484375 860 / 1452 : pp = 155.4661102294922 870 / 1452 : pp = 155.45765686035156 880 / 1452 : pp = 155.58758544921875 890 / 1452 : pp = 155.59373474121094 900 / 1452 : pp = 155.59254455566406 910 / 1452 : pp = 155.66854858398438 920 / 1452 : pp = 155.75942993164062 930 / 1452 : pp = 155.73350524902344 940 / 1452 : pp = 155.80740356445312 950 / 1452 : pp = 155.7733917236328 960 / 1452 : pp = 155.73565673828125 970 / 1452 : pp = 155.74404907226562 980 / 1452 : pp = 155.55902099609375 990 / 1452 : pp = 155.45675659179688 1000 / 1452 : pp = 155.2649688720703 1010 / 1452 : pp = 155.31332397460938 1020 / 1452 : pp = 155.44979858398438 1030 / 1452 : pp = 155.4137725830078 1040 / 1452 : pp = 155.49012756347656 1050 / 1452 : pp = 155.46054077148438 1060 / 1452 : pp = 155.3616943359375 1070 / 1452 : pp = 155.5286865234375 1080 / 1452 : pp = 155.63743591308594 1090 / 1452 : pp = 155.6842803955078 1100 / 1452 : pp = 155.65599060058594 1110 / 1452 : pp = 155.4827880859375 1120 / 1452 : pp = 155.35450744628906 1130 / 1452 : pp = 155.1777801513672 1140 / 1452 : pp = 155.15994262695312 1150 / 1452 : pp = 155.26193237304688 1160 / 1452 : pp = 155.26214599609375 1170 / 1452 : pp = 155.23231506347656 1180 / 1452 : pp = 155.29266357421875 1190 / 1452 : pp = 155.37680053710938 1200 / 1452 : pp = 155.3736114501953 1210 / 1452 : pp = 155.3380584716797 1220 / 1452 : pp = 155.474853515625 1230 / 1452 : pp = 155.62986755371094 1240 / 1452 : pp = 155.62831115722656 1250 / 1452 : pp = 155.77101135253906 1260 / 1452 : pp = 155.83445739746094 1270 / 1452 : pp = 155.845458984375 1280 / 1452 : pp = 155.8556365966797 1290 / 1452 : pp = 155.8556365966797 1300 / 1452 : pp = 155.8843994140625 1310 / 1452 : pp = 155.92417907714844 1320 / 1452 : pp = 155.8560791015625 1330 / 1452 : pp = 155.80636596679688 1340 / 1452 : pp = 155.84344482421875 1350 / 1452 : pp = 155.8706512451172 1360 / 1452 : pp = 155.9273681640625 1370 / 1452 : pp = 155.83140563964844 1380 / 1452 : pp = 155.7911376953125 1390 / 1452 : pp = 155.7401885986328 1400 / 1452 : pp = 155.6622314453125 1410 / 1452 : pp = 155.68531799316406 1420 / 1452 : pp = 155.64041137695312 1430 / 1452 : pp = 155.62216186523438 1440 / 1452 : pp = 155.6437530517578 1450 / 1452 : pp = 155.62757873535156 0 / 115 : pp = 228.70111083984375 10 / 115 : pp = 211.03330993652344 20 / 115 : pp = 212.24957275390625 30 / 115 : pp = 209.8839569091797 40 / 115 : pp = 209.11045837402344 50 / 115 : pp = 204.66351318359375 60 / 115 : pp = 204.03366088867188 70 / 115 : pp = 200.46681213378906 80 / 115 : pp = 198.24404907226562 90 / 115 : pp = 195.63223266601562 100 / 115 : pp = 191.18345642089844 110 / 115 : pp = 189.31134033203125 Training perplexity: 155.61154174804688 Validation perplexity:188.94537353515625 Total time : 42.13483738899231 Epoch 9 0 / 1452 : pp = 197.80628967285156 10 / 1452 : pp = 172.6316680908203 20 / 1452 : pp = 168.6739959716797 30 / 1452 : pp = 164.4781036376953 40 / 1452 : pp = 166.1627960205078 50 / 1452 : pp = 163.05197143554688 60 / 1452 : pp = 161.87924194335938 70 / 1452 : pp = 162.5297088623047 80 / 1452 : pp = 161.7450714111328 90 / 1452 : pp = 160.6148223876953 100 / 1452 : pp = 159.73289489746094 110 / 1452 : pp = 158.4092254638672 120 / 1452 : pp = 158.04653930664062 130 / 1452 : pp = 157.13563537597656 140 / 1452 : pp = 155.71798706054688 150 / 1452 : pp = 155.19161987304688 160 / 1452 : pp = 155.42718505859375 170 / 1452 : pp = 155.0531463623047 180 / 1452 : pp = 154.46897888183594 190 / 1452 : pp = 154.4127197265625 200 / 1452 : pp = 154.97154235839844 210 / 1452 : pp = 154.70169067382812 220 / 1452 : pp = 154.72816467285156 230 / 1452 : pp = 155.03799438476562 240 / 1452 : pp = 154.85601806640625 250 / 1452 : pp = 154.28016662597656 260 / 1452 : pp = 153.7699432373047 270 / 1452 : pp = 152.90948486328125 280 / 1452 : pp = 153.0459747314453 290 / 1452 : pp = 153.298095703125 300 / 1452 : pp = 153.45716857910156 310 / 1452 : pp = 153.22195434570312 320 / 1452 : pp = 153.41664123535156 330 / 1452 : pp = 153.66542053222656 340 / 1452 : pp = 153.06378173828125 350 / 1452 : pp = 153.43923950195312 360 / 1452 : pp = 153.31381225585938 370 / 1452 : pp = 153.13473510742188 380 / 1452 : pp = 152.75267028808594 390 / 1452 : pp = 152.85504150390625 400 / 1452 : pp = 152.62342834472656 410 / 1452 : pp = 153.03152465820312 420 / 1452 : pp = 153.39161682128906 430 / 1452 : pp = 153.30364990234375 440 / 1452 : pp = 153.37896728515625 450 / 1452 : pp = 153.18988037109375 460 / 1452 : pp = 152.88478088378906 470 / 1452 : pp = 152.4380340576172 480 / 1452 : pp = 151.86618041992188 490 / 1452 : pp = 151.5962371826172 500 / 1452 : pp = 151.11614990234375 510 / 1452 : pp = 150.99830627441406 520 / 1452 : pp = 150.8135986328125 530 / 1452 : pp = 150.500732421875 540 / 1452 : pp = 149.9623260498047 550 / 1452 : pp = 149.68028259277344 560 / 1452 : pp = 149.3885040283203 570 / 1452 : pp = 149.140380859375 580 / 1452 : pp = 148.76876831054688 590 / 1452 : pp = 148.43368530273438 600 / 1452 : pp = 148.02598571777344 610 / 1452 : pp = 147.7869110107422 620 / 1452 : pp = 147.59796142578125 630 / 1452 : pp = 147.30068969726562 640 / 1452 : pp = 147.45240783691406 650 / 1452 : pp = 147.4651336669922 660 / 1452 : pp = 147.5808563232422 670 / 1452 : pp = 147.65582275390625 680 / 1452 : pp = 147.7360382080078 690 / 1452 : pp = 147.63075256347656 700 / 1452 : pp = 147.6066131591797 710 / 1452 : pp = 147.7024383544922 720 / 1452 : pp = 147.7445526123047 730 / 1452 : pp = 147.72279357910156 740 / 1452 : pp = 147.87107849121094 750 / 1452 : pp = 147.91436767578125 760 / 1452 : pp = 147.9857635498047 770 / 1452 : pp = 148.18206787109375 780 / 1452 : pp = 148.3845672607422 790 / 1452 : pp = 148.5517120361328 800 / 1452 : pp = 148.54002380371094 810 / 1452 : pp = 148.51119995117188 820 / 1452 : pp = 148.5664520263672 830 / 1452 : pp = 148.7821044921875 840 / 1452 : pp = 148.72486877441406 850 / 1452 : pp = 148.77452087402344 860 / 1452 : pp = 148.80076599121094 870 / 1452 : pp = 148.79701232910156 880 / 1452 : pp = 148.9181671142578 890 / 1452 : pp = 148.94537353515625 900 / 1452 : pp = 148.9435272216797 910 / 1452 : pp = 149.02102661132812 920 / 1452 : pp = 149.1085968017578 930 / 1452 : pp = 149.06893920898438 940 / 1452 : pp = 149.1317138671875 950 / 1452 : pp = 149.1232452392578 960 / 1452 : pp = 149.10354614257812 970 / 1452 : pp = 149.11656188964844 980 / 1452 : pp = 148.94259643554688 990 / 1452 : pp = 148.8236846923828 1000 / 1452 : pp = 148.633056640625 1010 / 1452 : pp = 148.6830291748047 1020 / 1452 : pp = 148.8126220703125 1030 / 1452 : pp = 148.78089904785156 1040 / 1452 : pp = 148.8600311279297 1050 / 1452 : pp = 148.8486785888672 1060 / 1452 : pp = 148.7664337158203 1070 / 1452 : pp = 148.9337921142578 1080 / 1452 : pp = 149.04441833496094 1090 / 1452 : pp = 149.07284545898438 1100 / 1452 : pp = 149.03318786621094 1110 / 1452 : pp = 148.86428833007812 1120 / 1452 : pp = 148.7332305908203 1130 / 1452 : pp = 148.5670166015625 1140 / 1452 : pp = 148.54661560058594 1150 / 1452 : pp = 148.64219665527344 1160 / 1452 : pp = 148.6490020751953 1170 / 1452 : pp = 148.62420654296875 1180 / 1452 : pp = 148.67665100097656 1190 / 1452 : pp = 148.7633056640625 1200 / 1452 : pp = 148.7782745361328 1210 / 1452 : pp = 148.72500610351562 1220 / 1452 : pp = 148.87493896484375 1230 / 1452 : pp = 149.039794921875 1240 / 1452 : pp = 149.04000854492188 1250 / 1452 : pp = 149.17054748535156 1260 / 1452 : pp = 149.23863220214844 1270 / 1452 : pp = 149.2436065673828 1280 / 1452 : pp = 149.25086975097656 1290 / 1452 : pp = 149.24147033691406 1300 / 1452 : pp = 149.27413940429688 1310 / 1452 : pp = 149.32077026367188 1320 / 1452 : pp = 149.27301025390625 1330 / 1452 : pp = 149.23080444335938 1340 / 1452 : pp = 149.25791931152344 1350 / 1452 : pp = 149.2841033935547 1360 / 1452 : pp = 149.337158203125 1370 / 1452 : pp = 149.2467498779297 1380 / 1452 : pp = 149.21351623535156 1390 / 1452 : pp = 149.15403747558594 1400 / 1452 : pp = 149.0877685546875 1410 / 1452 : pp = 149.110595703125 1420 / 1452 : pp = 149.07241821289062 1430 / 1452 : pp = 149.05166625976562 1440 / 1452 : pp = 149.0776824951172 1450 / 1452 : pp = 149.06771850585938 0 / 115 : pp = 227.0559844970703 10 / 115 : pp = 208.7002410888672 20 / 115 : pp = 210.38775634765625 30 / 115 : pp = 207.9513397216797 40 / 115 : pp = 207.12994384765625 50 / 115 : pp = 202.70811462402344 60 / 115 : pp = 202.05787658691406 70 / 115 : pp = 198.3761444091797 80 / 115 : pp = 196.17637634277344 90 / 115 : pp = 193.5880126953125 100 / 115 : pp = 189.0758819580078 110 / 115 : pp = 187.07528686523438 Training perplexity: 149.0502471923828 Validation perplexity:186.6911163330078 Total time : 47.274805545806885 Epoch 10 0 / 1452 : pp = 181.8408203125 10 / 1452 : pp = 164.99664306640625 20 / 1452 : pp = 161.8847198486328 30 / 1452 : pp = 158.30064392089844 40 / 1452 : pp = 160.13914489746094 50 / 1452 : pp = 157.58743286132812 60 / 1452 : pp = 156.11871337890625 70 / 1452 : pp = 156.82948303222656 80 / 1452 : pp = 156.2889862060547 90 / 1452 : pp = 155.04833984375 100 / 1452 : pp = 154.09327697753906 110 / 1452 : pp = 152.5070343017578 120 / 1452 : pp = 152.20750427246094 130 / 1452 : pp = 151.3399200439453 140 / 1452 : pp = 149.90740966796875 150 / 1452 : pp = 149.345703125 160 / 1452 : pp = 149.59814453125 170 / 1452 : pp = 149.26539611816406 180 / 1452 : pp = 148.624267578125 190 / 1452 : pp = 148.58819580078125 200 / 1452 : pp = 149.09552001953125 210 / 1452 : pp = 148.8439178466797 220 / 1452 : pp = 148.86605834960938 230 / 1452 : pp = 149.1971435546875 240 / 1452 : pp = 148.96533203125 250 / 1452 : pp = 148.4253387451172 260 / 1452 : pp = 147.9200897216797 270 / 1452 : pp = 147.08816528320312 280 / 1452 : pp = 147.24366760253906 290 / 1452 : pp = 147.52182006835938 300 / 1452 : pp = 147.72222900390625 310 / 1452 : pp = 147.50486755371094 320 / 1452 : pp = 147.73892211914062 330 / 1452 : pp = 147.9404754638672 340 / 1452 : pp = 147.37803649902344 350 / 1452 : pp = 147.6969451904297 360 / 1452 : pp = 147.5704345703125 370 / 1452 : pp = 147.38674926757812 380 / 1452 : pp = 147.03970336914062 390 / 1452 : pp = 147.14231872558594 400 / 1452 : pp = 146.91656494140625 410 / 1452 : pp = 147.34059143066406 420 / 1452 : pp = 147.68496704101562 430 / 1452 : pp = 147.61195373535156 440 / 1452 : pp = 147.68405151367188 450 / 1452 : pp = 147.4711151123047 460 / 1452 : pp = 147.1927032470703 470 / 1452 : pp = 146.72970581054688 480 / 1452 : pp = 146.17173767089844 490 / 1452 : pp = 145.9028778076172 500 / 1452 : pp = 145.42721557617188 510 / 1452 : pp = 145.3111114501953 520 / 1452 : pp = 145.11460876464844 530 / 1452 : pp = 144.81488037109375 540 / 1452 : pp = 144.263916015625 550 / 1452 : pp = 143.997802734375 560 / 1452 : pp = 143.71766662597656 570 / 1452 : pp = 143.47451782226562 580 / 1452 : pp = 143.08474731445312 590 / 1452 : pp = 142.77920532226562 600 / 1452 : pp = 142.39573669433594 610 / 1452 : pp = 142.14906311035156 620 / 1452 : pp = 141.9574432373047 630 / 1452 : pp = 141.67369079589844 640 / 1452 : pp = 141.81556701660156 650 / 1452 : pp = 141.81759643554688 660 / 1452 : pp = 141.9339599609375 670 / 1452 : pp = 142.01248168945312 680 / 1452 : pp = 142.08773803710938 690 / 1452 : pp = 142.00328063964844 700 / 1452 : pp = 141.98086547851562 710 / 1452 : pp = 142.0632781982422 720 / 1452 : pp = 142.10372924804688 730 / 1452 : pp = 142.08055114746094 740 / 1452 : pp = 142.23619079589844 750 / 1452 : pp = 142.2660369873047 760 / 1452 : pp = 142.34678649902344 770 / 1452 : pp = 142.5257568359375 780 / 1452 : pp = 142.70025634765625 790 / 1452 : pp = 142.8614044189453 800 / 1452 : pp = 142.84573364257812 810 / 1452 : pp = 142.8250274658203 820 / 1452 : pp = 142.8540496826172 830 / 1452 : pp = 143.06053161621094 840 / 1452 : pp = 143.0423126220703 850 / 1452 : pp = 143.09634399414062 860 / 1452 : pp = 143.10487365722656 870 / 1452 : pp = 143.0884246826172 880 / 1452 : pp = 143.19387817382812 890 / 1452 : pp = 143.236083984375 900 / 1452 : pp = 143.23390197753906 910 / 1452 : pp = 143.29537963867188 920 / 1452 : pp = 143.3722686767578 930 / 1452 : pp = 143.33795166015625 940 / 1452 : pp = 143.40618896484375 950 / 1452 : pp = 143.3929901123047 960 / 1452 : pp = 143.3693389892578 970 / 1452 : pp = 143.39736938476562 980 / 1452 : pp = 143.2371063232422 990 / 1452 : pp = 143.13893127441406 1000 / 1452 : pp = 142.9658660888672 1010 / 1452 : pp = 143.01544189453125 1020 / 1452 : pp = 143.152587890625 1030 / 1452 : pp = 143.11334228515625 1040 / 1452 : pp = 143.19020080566406 1050 / 1452 : pp = 143.18234252929688 1060 / 1452 : pp = 143.092041015625 1070 / 1452 : pp = 143.24449157714844 1080 / 1452 : pp = 143.34828186035156 1090 / 1452 : pp = 143.38739013671875 1100 / 1452 : pp = 143.37432861328125 1110 / 1452 : pp = 143.20596313476562 1120 / 1452 : pp = 143.07969665527344 1130 / 1452 : pp = 142.92041015625 1140 / 1452 : pp = 142.90902709960938 1150 / 1452 : pp = 143.00732421875 1160 / 1452 : pp = 143.01182556152344 1170 / 1452 : pp = 142.9925994873047 1180 / 1452 : pp = 143.06080627441406 1190 / 1452 : pp = 143.14337158203125 1200 / 1452 : pp = 143.16644287109375 1210 / 1452 : pp = 143.1259002685547 1220 / 1452 : pp = 143.2671661376953 1230 / 1452 : pp = 143.4210968017578 1240 / 1452 : pp = 143.4327850341797 1250 / 1452 : pp = 143.5699920654297 1260 / 1452 : pp = 143.63771057128906 1270 / 1452 : pp = 143.65798950195312 1280 / 1452 : pp = 143.68251037597656 1290 / 1452 : pp = 143.68045043945312 1300 / 1452 : pp = 143.72293090820312 1310 / 1452 : pp = 143.77015686035156 1320 / 1452 : pp = 143.71910095214844 1330 / 1452 : pp = 143.68792724609375 1340 / 1452 : pp = 143.7241668701172 1350 / 1452 : pp = 143.7570037841797 1360 / 1452 : pp = 143.81829833984375 1370 / 1452 : pp = 143.7487030029297 1380 / 1452 : pp = 143.7196502685547 1390 / 1452 : pp = 143.67359924316406 1400 / 1452 : pp = 143.60592651367188 1410 / 1452 : pp = 143.62620544433594 1420 / 1452 : pp = 143.5905303955078 1430 / 1452 : pp = 143.55799865722656 1440 / 1452 : pp = 143.5891571044922 1450 / 1452 : pp = 143.5869598388672 0 / 115 : pp = 226.9864959716797 10 / 115 : pp = 207.8067169189453 20 / 115 : pp = 209.68667602539062 30 / 115 : pp = 207.1610565185547 40 / 115 : pp = 206.3247833251953 50 / 115 : pp = 201.77403259277344 60 / 115 : pp = 201.07098388671875 70 / 115 : pp = 197.33335876464844 80 / 115 : pp = 195.12513732910156 90 / 115 : pp = 192.5349578857422 100 / 115 : pp = 187.90072631835938 110 / 115 : pp = 185.81240844726562 Training perplexity: 143.57354736328125 Validation perplexity:185.40573120117188 Total time : 46.14846849441528 Epoch 11 0 / 1452 : pp = 181.93162536621094 10 / 1452 : pp = 159.94607543945312 20 / 1452 : pp = 156.83673095703125 30 / 1452 : pp = 153.75843811035156 40 / 1452 : pp = 155.18362426757812 50 / 1452 : pp = 152.39529418945312 60 / 1452 : pp = 151.18772888183594 70 / 1452 : pp = 151.9004364013672 80 / 1452 : pp = 151.30239868164062 90 / 1452 : pp = 150.1591033935547 100 / 1452 : pp = 149.18618774414062 110 / 1452 : pp = 147.72653198242188 120 / 1452 : pp = 147.4357452392578 130 / 1452 : pp = 146.41372680664062 140 / 1452 : pp = 145.0057373046875 150 / 1452 : pp = 144.39447021484375 160 / 1452 : pp = 144.5330047607422 170 / 1452 : pp = 144.23593139648438 180 / 1452 : pp = 143.63990783691406 190 / 1452 : pp = 143.63812255859375 200 / 1452 : pp = 144.1143798828125 210 / 1452 : pp = 143.88278198242188 220 / 1452 : pp = 143.92518615722656 230 / 1452 : pp = 144.24032592773438 240 / 1452 : pp = 143.94110107421875 250 / 1452 : pp = 143.3688507080078 260 / 1452 : pp = 142.8829345703125 270 / 1452 : pp = 142.11952209472656 280 / 1452 : pp = 142.19415283203125 290 / 1452 : pp = 142.51889038085938 300 / 1452 : pp = 142.70494079589844 310 / 1452 : pp = 142.51426696777344 320 / 1452 : pp = 142.70106506347656 330 / 1452 : pp = 142.88014221191406 340 / 1452 : pp = 142.3287353515625 350 / 1452 : pp = 142.6169891357422 360 / 1452 : pp = 142.51971435546875 370 / 1452 : pp = 142.33566284179688 380 / 1452 : pp = 142.04161071777344 390 / 1452 : pp = 142.13551330566406 400 / 1452 : pp = 141.9499969482422 410 / 1452 : pp = 142.3361358642578 420 / 1452 : pp = 142.64065551757812 430 / 1452 : pp = 142.5511016845703 440 / 1452 : pp = 142.6728973388672 450 / 1452 : pp = 142.47030639648438 460 / 1452 : pp = 142.1704864501953 470 / 1452 : pp = 141.73390197753906 480 / 1452 : pp = 141.23020935058594 490 / 1452 : pp = 140.9759521484375 500 / 1452 : pp = 140.51609802246094 510 / 1452 : pp = 140.40545654296875 520 / 1452 : pp = 140.1936492919922 530 / 1452 : pp = 139.8929443359375 540 / 1452 : pp = 139.3696746826172 550 / 1452 : pp = 139.13217163085938 560 / 1452 : pp = 138.85247802734375 570 / 1452 : pp = 138.6092987060547 580 / 1452 : pp = 138.2471160888672 590 / 1452 : pp = 137.9485626220703 600 / 1452 : pp = 137.57379150390625 610 / 1452 : pp = 137.31576538085938 620 / 1452 : pp = 137.14230346679688 630 / 1452 : pp = 136.87405395507812 640 / 1452 : pp = 137.02928161621094 650 / 1452 : pp = 137.0481719970703 660 / 1452 : pp = 137.1595001220703 670 / 1452 : pp = 137.21124267578125 680 / 1452 : pp = 137.2671356201172 690 / 1452 : pp = 137.19410705566406 700 / 1452 : pp = 137.1850128173828 710 / 1452 : pp = 137.26058959960938 720 / 1452 : pp = 137.30726623535156 730 / 1452 : pp = 137.28048706054688 740 / 1452 : pp = 137.4352569580078 750 / 1452 : pp = 137.4680938720703 760 / 1452 : pp = 137.5524139404297 770 / 1452 : pp = 137.73829650878906 780 / 1452 : pp = 137.90882873535156 790 / 1452 : pp = 138.05865478515625 800 / 1452 : pp = 138.0673370361328 810 / 1452 : pp = 138.03909301757812 820 / 1452 : pp = 138.084716796875 830 / 1452 : pp = 138.27989196777344 840 / 1452 : pp = 138.23545837402344 850 / 1452 : pp = 138.30343627929688 860 / 1452 : pp = 138.3339080810547 870 / 1452 : pp = 138.32835388183594 880 / 1452 : pp = 138.4450225830078 890 / 1452 : pp = 138.47157287597656 900 / 1452 : pp = 138.46304321289062 910 / 1452 : pp = 138.55618286132812 920 / 1452 : pp = 138.64512634277344 930 / 1452 : pp = 138.6160430908203 940 / 1452 : pp = 138.66932678222656 950 / 1452 : pp = 138.6573028564453 960 / 1452 : pp = 138.6463165283203 970 / 1452 : pp = 138.67059326171875 980 / 1452 : pp = 138.50999450683594 990 / 1452 : pp = 138.42430114746094 1000 / 1452 : pp = 138.25344848632812 1010 / 1452 : pp = 138.3004608154297 1020 / 1452 : pp = 138.4243621826172 1030 / 1452 : pp = 138.40713500976562 1040 / 1452 : pp = 138.47129821777344 1050 / 1452 : pp = 138.45928955078125 1060 / 1452 : pp = 138.3919677734375 1070 / 1452 : pp = 138.5287628173828 1080 / 1452 : pp = 138.62298583984375 1090 / 1452 : pp = 138.6699981689453 1100 / 1452 : pp = 138.64849853515625 1110 / 1452 : pp = 138.49191284179688 1120 / 1452 : pp = 138.37355041503906 1130 / 1452 : pp = 138.2216796875 1140 / 1452 : pp = 138.21534729003906 1150 / 1452 : pp = 138.30963134765625 1160 / 1452 : pp = 138.316162109375 1170 / 1452 : pp = 138.3023681640625 1180 / 1452 : pp = 138.36932373046875 1190 / 1452 : pp = 138.45960998535156 1200 / 1452 : pp = 138.4866180419922 1210 / 1452 : pp = 138.45730590820312 1220 / 1452 : pp = 138.60031127929688 1230 / 1452 : pp = 138.75485229492188 1240 / 1452 : pp = 138.7751007080078 1250 / 1452 : pp = 138.91221618652344 1260 / 1452 : pp = 138.9815216064453 1270 / 1452 : pp = 138.9919891357422 1280 / 1452 : pp = 139.0243377685547 1290 / 1452 : pp = 139.02725219726562 1300 / 1452 : pp = 139.0701446533203 1310 / 1452 : pp = 139.1090850830078 1320 / 1452 : pp = 139.06027221679688 1330 / 1452 : pp = 139.0338134765625 1340 / 1452 : pp = 139.06385803222656 1350 / 1452 : pp = 139.09608459472656 1360 / 1452 : pp = 139.1609649658203 1370 / 1452 : pp = 139.0869903564453 1380 / 1452 : pp = 139.0604705810547 1390 / 1452 : pp = 139.01670837402344 1400 / 1452 : pp = 138.94393920898438 1410 / 1452 : pp = 138.97323608398438 1420 / 1452 : pp = 138.9404296875 1430 / 1452 : pp = 138.90943908691406 1440 / 1452 : pp = 138.94268798828125 1450 / 1452 : pp = 138.93991088867188 0 / 115 : pp = 225.55990600585938 10 / 115 : pp = 207.0504608154297 20 / 115 : pp = 208.98306274414062 30 / 115 : pp = 206.28396606445312 40 / 115 : pp = 205.35386657714844 50 / 115 : pp = 200.7255401611328 60 / 115 : pp = 200.0526580810547 70 / 115 : pp = 196.33087158203125 80 / 115 : pp = 194.12110900878906 90 / 115 : pp = 191.52816772460938 100 / 115 : pp = 186.7974395751953 110 / 115 : pp = 184.59829711914062 Training perplexity: 138.9222869873047 Validation perplexity:184.18101501464844 Total time : 43.92928600311279 Epoch 12 0 / 1452 : pp = 173.0251007080078 10 / 1452 : pp = 152.98446655273438 20 / 1452 : pp = 150.43128967285156 30 / 1452 : pp = 147.5819854736328 40 / 1452 : pp = 149.4164276123047 50 / 1452 : pp = 146.70816040039062 60 / 1452 : pp = 145.557861328125 70 / 1452 : pp = 146.50473022460938 80 / 1452 : pp = 145.83200073242188 90 / 1452 : pp = 144.84402465820312 100 / 1452 : pp = 144.0390167236328 110 / 1452 : pp = 142.66514587402344 120 / 1452 : pp = 142.3549346923828 130 / 1452 : pp = 141.4630126953125 140 / 1452 : pp = 140.2266082763672 150 / 1452 : pp = 139.67518615722656 160 / 1452 : pp = 139.90414428710938 170 / 1452 : pp = 139.5490264892578 180 / 1452 : pp = 138.91969299316406 190 / 1452 : pp = 138.89234924316406 200 / 1452 : pp = 139.40908813476562 210 / 1452 : pp = 139.19068908691406 220 / 1452 : pp = 139.35513305664062 230 / 1452 : pp = 139.5464324951172 240 / 1452 : pp = 139.3047637939453 250 / 1452 : pp = 138.7708740234375 260 / 1452 : pp = 138.29188537597656 270 / 1452 : pp = 137.4787139892578 280 / 1452 : pp = 137.6367950439453 290 / 1452 : pp = 137.98513793945312 300 / 1452 : pp = 138.17819213867188 310 / 1452 : pp = 137.943359375 320 / 1452 : pp = 138.12060546875 330 / 1452 : pp = 138.29037475585938 340 / 1452 : pp = 137.77606201171875 350 / 1452 : pp = 138.06378173828125 360 / 1452 : pp = 137.99000549316406 370 / 1452 : pp = 137.81922912597656 380 / 1452 : pp = 137.52159118652344 390 / 1452 : pp = 137.61782836914062 400 / 1452 : pp = 137.4178924560547 410 / 1452 : pp = 137.82632446289062 420 / 1452 : pp = 138.17567443847656 430 / 1452 : pp = 138.11863708496094 440 / 1452 : pp = 138.215087890625 450 / 1452 : pp = 137.9976348876953 460 / 1452 : pp = 137.6929168701172 470 / 1452 : pp = 137.25416564941406 480 / 1452 : pp = 136.75140380859375 490 / 1452 : pp = 136.51712036132812 500 / 1452 : pp = 136.0896453857422 510 / 1452 : pp = 135.97048950195312 520 / 1452 : pp = 135.7760009765625 530 / 1452 : pp = 135.50389099121094 540 / 1452 : pp = 135.01437377929688 550 / 1452 : pp = 134.7666015625 560 / 1452 : pp = 134.48973083496094 570 / 1452 : pp = 134.22853088378906 580 / 1452 : pp = 133.88455200195312 590 / 1452 : pp = 133.5808868408203 600 / 1452 : pp = 133.22975158691406 610 / 1452 : pp = 132.99591064453125 620 / 1452 : pp = 132.79502868652344 630 / 1452 : pp = 132.5094451904297 640 / 1452 : pp = 132.62892150878906 650 / 1452 : pp = 132.63499450683594 660 / 1452 : pp = 132.7379913330078 670 / 1452 : pp = 132.79046630859375 680 / 1452 : pp = 132.85842895507812 690 / 1452 : pp = 132.80364990234375 700 / 1452 : pp = 132.80477905273438 710 / 1452 : pp = 132.90170288085938 720 / 1452 : pp = 132.92971801757812 730 / 1452 : pp = 132.9019012451172 740 / 1452 : pp = 133.04811096191406 750 / 1452 : pp = 133.10877990722656 760 / 1452 : pp = 133.19189453125 770 / 1452 : pp = 133.3564910888672 780 / 1452 : pp = 133.54000854492188 790 / 1452 : pp = 133.69239807128906 800 / 1452 : pp = 133.68495178222656 810 / 1452 : pp = 133.67971801757812 820 / 1452 : pp = 133.7035675048828 830 / 1452 : pp = 133.89329528808594 840 / 1452 : pp = 133.850341796875 850 / 1452 : pp = 133.90390014648438 860 / 1452 : pp = 133.9090118408203 870 / 1452 : pp = 133.89974975585938 880 / 1452 : pp = 134.0077667236328 890 / 1452 : pp = 134.03485107421875 900 / 1452 : pp = 134.0261688232422 910 / 1452 : pp = 134.10255432128906 920 / 1452 : pp = 134.17291259765625 930 / 1452 : pp = 134.14796447753906 940 / 1452 : pp = 134.20925903320312 950 / 1452 : pp = 134.19281005859375 960 / 1452 : pp = 134.17745971679688 970 / 1452 : pp = 134.18653869628906 980 / 1452 : pp = 134.03192138671875 990 / 1452 : pp = 133.94349670410156 1000 / 1452 : pp = 133.79685974121094 1010 / 1452 : pp = 133.8438262939453 1020 / 1452 : pp = 133.9608612060547 1030 / 1452 : pp = 133.93934631347656 1040 / 1452 : pp = 134.02833557128906 1050 / 1452 : pp = 134.01734924316406 1060 / 1452 : pp = 133.95346069335938 1070 / 1452 : pp = 134.10205078125 1080 / 1452 : pp = 134.2030487060547 1090 / 1452 : pp = 134.23696899414062 1100 / 1452 : pp = 134.2230224609375 1110 / 1452 : pp = 134.0829315185547 1120 / 1452 : pp = 133.980224609375 1130 / 1452 : pp = 133.83815002441406 1140 / 1452 : pp = 133.8366241455078 1150 / 1452 : pp = 133.92108154296875 1160 / 1452 : pp = 133.94375610351562 1170 / 1452 : pp = 133.9360809326172 1180 / 1452 : pp = 133.99684143066406 1190 / 1452 : pp = 134.0944366455078 1200 / 1452 : pp = 134.11676025390625 1210 / 1452 : pp = 134.0911102294922 1220 / 1452 : pp = 134.22763061523438 1230 / 1452 : pp = 134.38043212890625 1240 / 1452 : pp = 134.39817810058594 1250 / 1452 : pp = 134.5367431640625 1260 / 1452 : pp = 134.593017578125 1270 / 1452 : pp = 134.61497497558594 1280 / 1452 : pp = 134.6423797607422 1290 / 1452 : pp = 134.64340209960938 1300 / 1452 : pp = 134.68026733398438 1310 / 1452 : pp = 134.73556518554688 1320 / 1452 : pp = 134.69021606445312 1330 / 1452 : pp = 134.66131591796875 1340 / 1452 : pp = 134.69393920898438 1350 / 1452 : pp = 134.7328643798828 1360 / 1452 : pp = 134.79405212402344 1370 / 1452 : pp = 134.71237182617188 1380 / 1452 : pp = 134.6885528564453 1390 / 1452 : pp = 134.65110778808594 1400 / 1452 : pp = 134.59584045410156 1410 / 1452 : pp = 134.6193389892578 1420 / 1452 : pp = 134.58338928222656 1430 / 1452 : pp = 134.559326171875 1440 / 1452 : pp = 134.59507751464844 1450 / 1452 : pp = 134.59365844726562 0 / 115 : pp = 226.0741729736328 10 / 115 : pp = 207.00494384765625 20 / 115 : pp = 209.26976013183594 30 / 115 : pp = 206.44662475585938 40 / 115 : pp = 205.47268676757812 50 / 115 : pp = 200.7876739501953 60 / 115 : pp = 200.13414001464844 70 / 115 : pp = 196.35549926757812 80 / 115 : pp = 194.10777282714844 90 / 115 : pp = 191.47467041015625 100 / 115 : pp = 186.61351013183594 110 / 115 : pp = 184.30374145507812 Training perplexity: 134.57826232910156 Validation perplexity:183.8900146484375 Total time : 45.410256147384644 Epoch 13 0 / 1452 : pp = 169.39393615722656 10 / 1452 : pp = 150.13232421875 20 / 1452 : pp = 147.60450744628906 30 / 1452 : pp = 144.64317321777344 40 / 1452 : pp = 146.47427368164062 50 / 1452 : pp = 143.929443359375 60 / 1452 : pp = 142.8344268798828 70 / 1452 : pp = 143.45248413085938 80 / 1452 : pp = 142.5418701171875 90 / 1452 : pp = 141.6178436279297 100 / 1452 : pp = 140.70127868652344 110 / 1452 : pp = 139.2852325439453 120 / 1452 : pp = 138.8017120361328 130 / 1452 : pp = 137.85629272460938 140 / 1452 : pp = 136.51718139648438 150 / 1452 : pp = 136.03619384765625 160 / 1452 : pp = 136.154296875 170 / 1452 : pp = 135.67037963867188 180 / 1452 : pp = 135.0376739501953 190 / 1452 : pp = 134.9230499267578 200 / 1452 : pp = 135.4241180419922 210 / 1452 : pp = 135.24581909179688 220 / 1452 : pp = 135.37957763671875 230 / 1452 : pp = 135.67652893066406 240 / 1452 : pp = 135.4161834716797 250 / 1452 : pp = 134.90895080566406 260 / 1452 : pp = 134.46754455566406 270 / 1452 : pp = 133.68577575683594 280 / 1452 : pp = 133.86770629882812 290 / 1452 : pp = 134.18475341796875 300 / 1452 : pp = 134.39132690429688 310 / 1452 : pp = 134.19985961914062 320 / 1452 : pp = 134.37998962402344 330 / 1452 : pp = 134.5557403564453 340 / 1452 : pp = 134.00686645507812 350 / 1452 : pp = 134.27749633789062 360 / 1452 : pp = 134.20286560058594 370 / 1452 : pp = 134.042724609375 380 / 1452 : pp = 133.74398803710938 390 / 1452 : pp = 133.83584594726562 400 / 1452 : pp = 133.64382934570312 410 / 1452 : pp = 134.02366638183594 420 / 1452 : pp = 134.35415649414062 430 / 1452 : pp = 134.310546875 440 / 1452 : pp = 134.3634490966797 450 / 1452 : pp = 134.15602111816406 460 / 1452 : pp = 133.86578369140625 470 / 1452 : pp = 133.43414306640625 480 / 1452 : pp = 132.90310668945312 490 / 1452 : pp = 132.646240234375 500 / 1452 : pp = 132.1982421875 510 / 1452 : pp = 132.04200744628906 520 / 1452 : pp = 131.86940002441406 530 / 1452 : pp = 131.59841918945312 540 / 1452 : pp = 131.12356567382812 550 / 1452 : pp = 130.887939453125 560 / 1452 : pp = 130.6210174560547 570 / 1452 : pp = 130.37826538085938 580 / 1452 : pp = 130.0374755859375 590 / 1452 : pp = 129.75979614257812 600 / 1452 : pp = 129.38308715820312 610 / 1452 : pp = 129.16685485839844 620 / 1452 : pp = 129.0115509033203 630 / 1452 : pp = 128.75152587890625 640 / 1452 : pp = 128.87295532226562 650 / 1452 : pp = 128.88734436035156 660 / 1452 : pp = 128.98275756835938 670 / 1452 : pp = 129.0487060546875 680 / 1452 : pp = 129.11013793945312 690 / 1452 : pp = 129.0646514892578 700 / 1452 : pp = 129.06280517578125 710 / 1452 : pp = 129.1343994140625 720 / 1452 : pp = 129.18582153320312 730 / 1452 : pp = 129.15138244628906 740 / 1452 : pp = 129.29811096191406 750 / 1452 : pp = 129.339599609375 760 / 1452 : pp = 129.4257354736328 770 / 1452 : pp = 129.61631774902344 780 / 1452 : pp = 129.802734375 790 / 1452 : pp = 129.96804809570312 800 / 1452 : pp = 129.95187377929688 810 / 1452 : pp = 129.92417907714844 820 / 1452 : pp = 129.9774627685547 830 / 1452 : pp = 130.1638946533203 840 / 1452 : pp = 130.13095092773438 850 / 1452 : pp = 130.16595458984375 860 / 1452 : pp = 130.173828125 870 / 1452 : pp = 130.170166015625 880 / 1452 : pp = 130.27032470703125 890 / 1452 : pp = 130.3022003173828 900 / 1452 : pp = 130.3071746826172 910 / 1452 : pp = 130.37939453125 920 / 1452 : pp = 130.46229553222656 930 / 1452 : pp = 130.43846130371094 940 / 1452 : pp = 130.50889587402344 950 / 1452 : pp = 130.50086975097656 960 / 1452 : pp = 130.4833221435547 970 / 1452 : pp = 130.50814819335938 980 / 1452 : pp = 130.35577392578125 990 / 1452 : pp = 130.26759338378906 1000 / 1452 : pp = 130.1064453125 1010 / 1452 : pp = 130.1472625732422 1020 / 1452 : pp = 130.27169799804688 1030 / 1452 : pp = 130.25100708007812 1040 / 1452 : pp = 130.30816650390625 1050 / 1452 : pp = 130.29803466796875 1060 / 1452 : pp = 130.2242431640625 1070 / 1452 : pp = 130.35906982421875 1080 / 1452 : pp = 130.45103454589844 1090 / 1452 : pp = 130.49838256835938 1100 / 1452 : pp = 130.484130859375 1110 / 1452 : pp = 130.35316467285156 1120 / 1452 : pp = 130.24697875976562 1130 / 1452 : pp = 130.10804748535156 1140 / 1452 : pp = 130.1076202392578 1150 / 1452 : pp = 130.195068359375 1160 / 1452 : pp = 130.19674682617188 1170 / 1452 : pp = 130.18321228027344 1180 / 1452 : pp = 130.24623107910156 1190 / 1452 : pp = 130.33905029296875 1200 / 1452 : pp = 130.3650360107422 1210 / 1452 : pp = 130.34588623046875 1220 / 1452 : pp = 130.4850616455078 1230 / 1452 : pp = 130.63160705566406 1240 / 1452 : pp = 130.64674377441406 1250 / 1452 : pp = 130.77078247070312 1260 / 1452 : pp = 130.8397674560547 1270 / 1452 : pp = 130.8511199951172 1280 / 1452 : pp = 130.88967895507812 1290 / 1452 : pp = 130.9040985107422 1300 / 1452 : pp = 130.93511962890625 1310 / 1452 : pp = 130.9759063720703 1320 / 1452 : pp = 130.92800903320312 1330 / 1452 : pp = 130.9105224609375 1340 / 1452 : pp = 130.929443359375 1350 / 1452 : pp = 130.96153259277344 1360 / 1452 : pp = 131.02381896972656 1370 / 1452 : pp = 130.9545440673828 1380 / 1452 : pp = 130.9344940185547 1390 / 1452 : pp = 130.9055938720703 1400 / 1452 : pp = 130.85386657714844 1410 / 1452 : pp = 130.8874969482422 1420 / 1452 : pp = 130.85928344726562 1430 / 1452 : pp = 130.83995056152344 1440 / 1452 : pp = 130.86659240722656 1450 / 1452 : pp = 130.86839294433594 0 / 115 : pp = 227.78428649902344 10 / 115 : pp = 207.609619140625 20 / 115 : pp = 209.92459106445312 30 / 115 : pp = 206.96240234375 40 / 115 : pp = 205.9295654296875 50 / 115 : pp = 201.0296630859375 60 / 115 : pp = 200.38059997558594 70 / 115 : pp = 196.55764770507812 80 / 115 : pp = 194.31735229492188 90 / 115 : pp = 191.66146850585938 100 / 115 : pp = 186.70437622070312 110 / 115 : pp = 184.3171844482422 Training perplexity: 130.85043334960938 Validation perplexity:183.88186645507812 Total time : 45.345656394958496 Epoch 14 0 / 1452 : pp = 164.82191467285156 10 / 1452 : pp = 146.39089965820312 20 / 1452 : pp = 142.93240356445312 30 / 1452 : pp = 140.3113555908203 40 / 1452 : pp = 142.39939880371094 50 / 1452 : pp = 139.70162963867188 60 / 1452 : pp = 138.73023986816406 70 / 1452 : pp = 139.2675018310547 80 / 1452 : pp = 138.47824096679688 90 / 1452 : pp = 137.40432739257812 100 / 1452 : pp = 136.47793579101562 110 / 1452 : pp = 135.2294464111328 120 / 1452 : pp = 134.80728149414062 130 / 1452 : pp = 133.89822387695312 140 / 1452 : pp = 132.54141235351562 150 / 1452 : pp = 132.10025024414062 160 / 1452 : pp = 132.21829223632812 170 / 1452 : pp = 131.8765106201172 180 / 1452 : pp = 131.37515258789062 190 / 1452 : pp = 131.31622314453125 200 / 1452 : pp = 131.78297424316406 210 / 1452 : pp = 131.5507354736328 220 / 1452 : pp = 131.7002410888672 230 / 1452 : pp = 131.9277801513672 240 / 1452 : pp = 131.72166442871094 250 / 1452 : pp = 131.225830078125 260 / 1452 : pp = 130.7496337890625 270 / 1452 : pp = 129.9896697998047 280 / 1452 : pp = 130.10594177246094 290 / 1452 : pp = 130.41644287109375 300 / 1452 : pp = 130.5982208251953 310 / 1452 : pp = 130.36329650878906 320 / 1452 : pp = 130.5633544921875 330 / 1452 : pp = 130.77252197265625 340 / 1452 : pp = 130.273193359375 350 / 1452 : pp = 130.47889709472656 360 / 1452 : pp = 130.4348602294922 370 / 1452 : pp = 130.28126525878906 380 / 1452 : pp = 130.02786254882812 390 / 1452 : pp = 130.1564483642578 400 / 1452 : pp = 129.98440551757812 410 / 1452 : pp = 130.37721252441406 420 / 1452 : pp = 130.71859741210938 430 / 1452 : pp = 130.65939331054688 440 / 1452 : pp = 130.72987365722656 450 / 1452 : pp = 130.56272888183594 460 / 1452 : pp = 130.28195190429688 470 / 1452 : pp = 129.90936279296875 480 / 1452 : pp = 129.42857360839844 490 / 1452 : pp = 129.18077087402344 500 / 1452 : pp = 128.7588348388672 510 / 1452 : pp = 128.6303253173828 520 / 1452 : pp = 128.47616577148438 530 / 1452 : pp = 128.21148681640625 540 / 1452 : pp = 127.7218017578125 550 / 1452 : pp = 127.50067138671875 560 / 1452 : pp = 127.27574157714844 570 / 1452 : pp = 127.05399322509766 580 / 1452 : pp = 126.73983001708984 590 / 1452 : pp = 126.43692779541016 600 / 1452 : pp = 126.06050109863281 610 / 1452 : pp = 125.82952880859375 620 / 1452 : pp = 125.66295623779297 630 / 1452 : pp = 125.39354705810547 640 / 1452 : pp = 125.49463653564453 650 / 1452 : pp = 125.48816680908203 660 / 1452 : pp = 125.58712005615234 670 / 1452 : pp = 125.65978240966797 680 / 1452 : pp = 125.71456146240234 690 / 1452 : pp = 125.66937255859375 700 / 1452 : pp = 125.65900421142578 710 / 1452 : pp = 125.7271499633789 720 / 1452 : pp = 125.77758026123047 730 / 1452 : pp = 125.74129486083984 740 / 1452 : pp = 125.8759765625 750 / 1452 : pp = 125.91793823242188 760 / 1452 : pp = 125.99595642089844 770 / 1452 : pp = 126.18113708496094 780 / 1452 : pp = 126.35147094726562 790 / 1452 : pp = 126.50797271728516 800 / 1452 : pp = 126.49759674072266 810 / 1452 : pp = 126.48113250732422 820 / 1452 : pp = 126.52528381347656 830 / 1452 : pp = 126.705810546875 840 / 1452 : pp = 126.67517852783203 850 / 1452 : pp = 126.74176025390625 860 / 1452 : pp = 126.74151611328125 870 / 1452 : pp = 126.73414611816406 880 / 1452 : pp = 126.83026885986328 890 / 1452 : pp = 126.88519287109375 900 / 1452 : pp = 126.88053894042969 910 / 1452 : pp = 126.97138214111328 920 / 1452 : pp = 127.04660034179688 930 / 1452 : pp = 127.03763580322266 940 / 1452 : pp = 127.1126480102539 950 / 1452 : pp = 127.09610748291016 960 / 1452 : pp = 127.0873794555664 970 / 1452 : pp = 127.10343933105469 980 / 1452 : pp = 126.96441650390625 990 / 1452 : pp = 126.88519287109375 1000 / 1452 : pp = 126.7336654663086 1010 / 1452 : pp = 126.77796936035156 1020 / 1452 : pp = 126.89826202392578 1030 / 1452 : pp = 126.88761138916016 1040 / 1452 : pp = 126.95309448242188 1050 / 1452 : pp = 126.96478271484375 1060 / 1452 : pp = 126.89324188232422 1070 / 1452 : pp = 127.03242492675781 1080 / 1452 : pp = 127.13228607177734 1090 / 1452 : pp = 127.173095703125 1100 / 1452 : pp = 127.15975189208984 1110 / 1452 : pp = 127.0392074584961 1120 / 1452 : pp = 126.94032287597656 1130 / 1452 : pp = 126.80693054199219 1140 / 1452 : pp = 126.81315612792969 1150 / 1452 : pp = 126.90467834472656 1160 / 1452 : pp = 126.91236114501953 1170 / 1452 : pp = 126.90897369384766 1180 / 1452 : pp = 126.98052215576172 1190 / 1452 : pp = 127.07483673095703 1200 / 1452 : pp = 127.10216522216797 1210 / 1452 : pp = 127.08258819580078 1220 / 1452 : pp = 127.22943878173828 1230 / 1452 : pp = 127.38563537597656 1240 / 1452 : pp = 127.40538024902344 1250 / 1452 : pp = 127.53369140625 1260 / 1452 : pp = 127.59293365478516 1270 / 1452 : pp = 127.61489868164062 1280 / 1452 : pp = 127.6484375 1290 / 1452 : pp = 127.65257263183594 1300 / 1452 : pp = 127.69329833984375 1310 / 1452 : pp = 127.74549102783203 1320 / 1452 : pp = 127.7043228149414 1330 / 1452 : pp = 127.6866683959961 1340 / 1452 : pp = 127.70913696289062 1350 / 1452 : pp = 127.73233795166016 1360 / 1452 : pp = 127.7855224609375 1370 / 1452 : pp = 127.71918487548828 1380 / 1452 : pp = 127.69987487792969 1390 / 1452 : pp = 127.6697998046875 1400 / 1452 : pp = 127.61137390136719 1410 / 1452 : pp = 127.6404037475586 1420 / 1452 : pp = 127.61094665527344 1430 / 1452 : pp = 127.58216857910156 1440 / 1452 : pp = 127.61477661132812 1450 / 1452 : pp = 127.61964416503906 0 / 115 : pp = 228.21578979492188 10 / 115 : pp = 208.11244201660156 20 / 115 : pp = 210.688232421875 30 / 115 : pp = 207.62408447265625 40 / 115 : pp = 206.45184326171875 50 / 115 : pp = 201.52760314941406 60 / 115 : pp = 200.7784881591797 70 / 115 : pp = 196.83067321777344 80 / 115 : pp = 194.6357879638672 90 / 115 : pp = 191.9783935546875 100 / 115 : pp = 186.8787841796875 110 / 115 : pp = 184.35252380371094 Training perplexity: 127.60413360595703 Validation perplexity:183.8877410888672 Total time : 41.6636528968811 Epoch 15 0 / 1452 : pp = 156.81654357910156 10 / 1452 : pp = 142.1070556640625 20 / 1452 : pp = 139.55076599121094 30 / 1452 : pp = 136.63551330566406 40 / 1452 : pp = 138.5840606689453 50 / 1452 : pp = 136.052734375 60 / 1452 : pp = 134.93019104003906 70 / 1452 : pp = 135.65206909179688 80 / 1452 : pp = 135.2620086669922 90 / 1452 : pp = 134.314697265625 100 / 1452 : pp = 133.4916229248047 110 / 1452 : pp = 132.26052856445312 120 / 1452 : pp = 131.7714080810547 130 / 1452 : pp = 130.77365112304688 140 / 1452 : pp = 129.5411834716797 150 / 1452 : pp = 129.0791778564453 160 / 1452 : pp = 129.21920776367188 170 / 1452 : pp = 128.7528839111328 180 / 1452 : pp = 128.22279357910156 190 / 1452 : pp = 128.18177795410156 200 / 1452 : pp = 128.58758544921875 210 / 1452 : pp = 128.3906707763672 220 / 1452 : pp = 128.5266571044922 230 / 1452 : pp = 128.80563354492188 240 / 1452 : pp = 128.61886596679688 250 / 1452 : pp = 128.13172912597656 260 / 1452 : pp = 127.69220733642578 270 / 1452 : pp = 126.96150970458984 280 / 1452 : pp = 127.04702758789062 290 / 1452 : pp = 127.33565521240234 300 / 1452 : pp = 127.55929565429688 310 / 1452 : pp = 127.38514709472656 320 / 1452 : pp = 127.52171325683594 330 / 1452 : pp = 127.68690490722656 340 / 1452 : pp = 127.18340301513672 350 / 1452 : pp = 127.4073257446289 360 / 1452 : pp = 127.30432891845703 370 / 1452 : pp = 127.17618560791016 380 / 1452 : pp = 126.92579650878906 390 / 1452 : pp = 127.02473449707031 400 / 1452 : pp = 126.8515625 410 / 1452 : pp = 127.211669921875 420 / 1452 : pp = 127.51788330078125 430 / 1452 : pp = 127.47386169433594 440 / 1452 : pp = 127.57164001464844 450 / 1452 : pp = 127.3601303100586 460 / 1452 : pp = 127.09434509277344 470 / 1452 : pp = 126.71922302246094 480 / 1452 : pp = 126.24349212646484 490 / 1452 : pp = 125.98778533935547 500 / 1452 : pp = 125.59526824951172 510 / 1452 : pp = 125.4450912475586 520 / 1452 : pp = 125.29247283935547 530 / 1452 : pp = 125.03536224365234 540 / 1452 : pp = 124.5813980102539 550 / 1452 : pp = 124.33724212646484 560 / 1452 : pp = 124.08995819091797 570 / 1452 : pp = 123.86637878417969 580 / 1452 : pp = 123.53152465820312 590 / 1452 : pp = 123.20321655273438 600 / 1452 : pp = 122.85673522949219 610 / 1452 : pp = 122.64250946044922 620 / 1452 : pp = 122.4958724975586 630 / 1452 : pp = 122.22386169433594 640 / 1452 : pp = 122.31143188476562 650 / 1452 : pp = 122.30093383789062 660 / 1452 : pp = 122.39427947998047 670 / 1452 : pp = 122.45440673828125 680 / 1452 : pp = 122.51146697998047 690 / 1452 : pp = 122.4854736328125 700 / 1452 : pp = 122.48600006103516 710 / 1452 : pp = 122.56084442138672 720 / 1452 : pp = 122.59059143066406 730 / 1452 : pp = 122.55529022216797 740 / 1452 : pp = 122.69409942626953 750 / 1452 : pp = 122.76456451416016 760 / 1452 : pp = 122.84437561035156 770 / 1452 : pp = 123.02527618408203 780 / 1452 : pp = 123.20509338378906 790 / 1452 : pp = 123.36305236816406 800 / 1452 : pp = 123.36852264404297 810 / 1452 : pp = 123.36799621582031 820 / 1452 : pp = 123.39976501464844 830 / 1452 : pp = 123.59362030029297 840 / 1452 : pp = 123.56946563720703 850 / 1452 : pp = 123.63800811767578 860 / 1452 : pp = 123.63983917236328 870 / 1452 : pp = 123.64148712158203 880 / 1452 : pp = 123.7568588256836 890 / 1452 : pp = 123.7885513305664 900 / 1452 : pp = 123.79640197753906 910 / 1452 : pp = 123.86153411865234 920 / 1452 : pp = 123.92941284179688 930 / 1452 : pp = 123.9125747680664 940 / 1452 : pp = 123.95559692382812 950 / 1452 : pp = 123.93928527832031 960 / 1452 : pp = 123.94294738769531 970 / 1452 : pp = 123.95547485351562 980 / 1452 : pp = 123.8229751586914 990 / 1452 : pp = 123.73727416992188 1000 / 1452 : pp = 123.59091186523438 1010 / 1452 : pp = 123.634765625 1020 / 1452 : pp = 123.76506042480469 1030 / 1452 : pp = 123.75485229492188 1040 / 1452 : pp = 123.807861328125 1050 / 1452 : pp = 123.79156494140625 1060 / 1452 : pp = 123.73054504394531 1070 / 1452 : pp = 123.8615951538086 1080 / 1452 : pp = 123.96564483642578 1090 / 1452 : pp = 124.02104187011719 1100 / 1452 : pp = 124.012939453125 1110 / 1452 : pp = 123.87582397460938 1120 / 1452 : pp = 123.775390625 1130 / 1452 : pp = 123.63182067871094 1140 / 1452 : pp = 123.62391662597656 1150 / 1452 : pp = 123.71013641357422 1160 / 1452 : pp = 123.72423553466797 1170 / 1452 : pp = 123.71726989746094 1180 / 1452 : pp = 123.79032897949219 1190 / 1452 : pp = 123.87883758544922 1200 / 1452 : pp = 123.9125747680664 1210 / 1452 : pp = 123.90140533447266 1220 / 1452 : pp = 124.03245544433594 1230 / 1452 : pp = 124.19799041748047 1240 / 1452 : pp = 124.21469116210938 1250 / 1452 : pp = 124.34103393554688 1260 / 1452 : pp = 124.4041976928711 1270 / 1452 : pp = 124.42852020263672 1280 / 1452 : pp = 124.46656036376953 1290 / 1452 : pp = 124.4811019897461 1300 / 1452 : pp = 124.52384185791016 1310 / 1452 : pp = 124.57533264160156 1320 / 1452 : pp = 124.5398178100586 1330 / 1452 : pp = 124.52598571777344 1340 / 1452 : pp = 124.53311157226562 1350 / 1452 : pp = 124.57759094238281 1360 / 1452 : pp = 124.63385772705078 1370 / 1452 : pp = 124.58133697509766 1380 / 1452 : pp = 124.55769348144531 1390 / 1452 : pp = 124.54011535644531 1400 / 1452 : pp = 124.4884033203125 1410 / 1452 : pp = 124.51226806640625 1420 / 1452 : pp = 124.49683380126953 1430 / 1452 : pp = 124.4754638671875 1440 / 1452 : pp = 124.50164031982422 1450 / 1452 : pp = 124.50894165039062 0 / 115 : pp = 230.8488006591797 10 / 115 : pp = 209.2509002685547 20 / 115 : pp = 211.68577575683594 30 / 115 : pp = 208.44056701660156 40 / 115 : pp = 207.2039337158203 50 / 115 : pp = 202.1859588623047 60 / 115 : pp = 201.34739685058594 70 / 115 : pp = 197.4251251220703 80 / 115 : pp = 195.2623291015625 90 / 115 : pp = 192.592529296875 100 / 115 : pp = 187.39553833007812 110 / 115 : pp = 184.791259765625 Training perplexity: 124.4933853149414 Validation perplexity:184.32510375976562 Total time : 40.856229066848755 0 / 128 : pp = 184.6475067138672 10 / 128 : pp = 176.8856964111328 20 / 128 : pp = 164.3444366455078 30 / 128 : pp = 167.85472106933594 40 / 128 : pp = 169.25367736816406 50 / 128 : pp = 168.86561584472656 60 / 128 : pp = 168.11801147460938 70 / 128 : pp = 165.4105224609375 80 / 128 : pp = 162.91146850585938 90 / 128 : pp = 161.29742431640625 100 / 128 : pp = 162.45989990234375 110 / 128 : pp = 162.6834716796875 120 / 128 : pp = 164.3359832763672 =-==-==-==-==-= Test perplexity: 164.0149383544922 =-==-==-==-==-=
更详细的内容请参考下面链接
https://github.com/weizhenzhao/cs224d_nlp_problem_set2
267
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
