一、系统设计思路和框架
本次系统全部使用 Python 编写，在系统设计上遵循着配置灵活、代码模块化的思路，分为数据预处理器、数据处理器、执行器、深度学习模型、可视化展示五个模块。
模块间的逻辑关系大致为：
1）数据预处理是将原始语料进行初步的处理以满足于数据处理模块的要求；
2）执行器是整个系统引擎分别在运转的时候调用数据处理器、深度学习模型进行数据处理、模型训练、模型运作等工作；
3）深度学习模型是一个基于 TF 的 seq2seq 模型，用于定义神经网络并进行模型计算；
4）可视化展示是一个用 Flask 前端框架写的简单的人机交互程序，在运行时调用执行器进行人机对话。
整体的框架图如下：
本系统利用 seq2seq 模型的特点，结合 word2vec 的思路（当然这样做有点简单粗暴，没有进行分词），将训练语料分为 Ask 语料集和 Replay 语料集，并根据一定的比例分为训练语料集和验证语料集，然后就是 word2vec。
这些步骤都是在数据预处理器和数据处理器中完成的，关于训练语料集与验证语料集的数量都是做成可外部配置的，这也是这个系统我认为设计的最合理的地方（可惜不是我设计的）。
在完成数据的处理后，执行器就会根据训练模式（这也是外部可配置的）来调用 seq2seq 进行神经网络的创建，神经网络的超参数同样也是可以外部进行配置的。
在进行训练过程中，使用 perprelixy 来计算模型的 loss，通过自动的调整 learning rate 来逐步的取得最优值，当 learning rate 减少为 0 达到最优值。
最后，就是可视化展示模块启动一个进程调用执行器来实时在线提供聊天服务，在语句输入和输出利用 seq2seq 的特点，直接将输入 seq 转换成 vec 作为已经训练好的神经网络，然后神经网络会生成一个 seq 向量，然后通过查询词典的方式将生成的向量替换成中文句子。
在神经网络模型这里，TF 有 GRU 和 LSTM 模型可供选择，我比较了 GRU 和 LSTM 的训练效果，发现还是 GRU 比较适合对话语句场景的训练，这点在后面的代码分析环节会详细解释。
二、源码结构
- data_utls.py（数据预处理器）包含的函数 convert_seq2seq_files，将主程序分好的 ask 语料集和
response 语料集转换成 seq2seq
文件，以便数据处理器进行进一步的数据处理。对于整个系统，一般该数据预处理器只需要运行一次即可。 - prepareData.py（数据处理器）包含的函数：create_vocabulary、convert_to_vector、prepare_custom_data、
basic_tokenizer、initialize_vocabulary，这些函数的作用分别是创建字典、句子转向量、根据超参定制化训练数据、基础数据标记、初始化词典。 - execute.py（执行器）包含的函数：get_config、read_data、create_model、train、
self_test、init_session、decode_line，这些函数的作用分别是获取配置参数和超参、读取数据、创建模型、训练模型、模式测试、初始化会话、在线对话。 - seq2seqmodel.py（深度机器学习模型）包含的函数：_init、sampled_loss、seq2seq_f、step、get_batch，这些函数的作用分别是程序初始化、loss
函数、seq2seq 函数、拟合模型、获取批量数据。 - app.py（可视化展示模块）包含的函数： heartbeat、reply、index，这些函数的作用分别是心跳、在线对话、主页入口。
三、源码详解
*1. 数据预处理器（data_utls.py)
# coding=utf-8
import math
import os
import random
import getConfig
import jieba
gConfig = {}
gConfig=getConfig.get_config()
conv_path = gConfig['resource_data']
if not os.path.exists(conv_path):
exit()
convs = [] # 用于存储对话集合
with open(conv_path,encoding='utf-8') as f:
one_conv = [] # 存储一次完整对话
for line in f:
line = line.strip('\n').replace('/', '')#去除换行符，并在字符间添加空格符，原因是用于区分 123 与1 2 3.
if line == '':
continue
if line[0] == gConfig['e']:
if one_conv:
convs.append(one_conv)
one_conv = []
elif line[0] == gConfig['m']:
one_conv.append(line.split(' ')[1])#将一次完整的对话存储下来
# 把对话分成问与答两个部分
ask = [] # 问
response = [] # 答
for conv in convs:
if len(conv) == 1:
continue
if len(conv) % 2 != 0: # 因为默认是一问一答的，所以需要进行数据的粗裁剪，对话行数要是偶数的
conv = conv[:-1]
for i in range(len(conv)):
if i % 2 == 0:
conv[i]=" ".join(jieba.cut(conv[i]))#使用jieba分词器进行分词
ask.append(conv[i])#因为i是从0开始的，因此偶数行为发问的语句，奇数行为回答的语句
else:
conv[i]=" ".join(jieba.cut(conv[i]))
response.append(conv[i])
def convert_seq2seq_files(questions, answers, TESTSET_SIZE):
# 创建文件
train_enc = open(gConfig['train_enc'],'w',encoding="utf-8") # 问
train_dec = open(gConfig['train_dec'],'w',encoding="utf-8") # 答
test_enc = open(gConfig['test_enc'], 'w',encoding="utf-8") # 问
test_dec = open(gConfig['test_dec'], 'w',encoding="utf-8") # 答
test_index = random.sample([i for i in range(len(questions))],TESTSET_SIZE) #可以从指定的序列中，随机的截取指定长度的片断，不作原地修改
for i in range(len(questions)):
if i in test_index:
test_enc.write(questions[i]+'\n')
test_dec.write(answers[i]+ '\n' )
else:
train_enc.write(questions[i]+'\n')
train_dec.write(answers[i]+ '\n' )
if i % 1000 == 0:
print(len(range(len(questions))), '处理进度：', i)
train_enc.close()
train_dec.close()
test_enc.close()
test_dec.close()
convert_seq2seq_files(ask, response,10000)
# 生成的*.enc文件保存了问题
# 生成的*.dec文件保存了回答
- 数据处理器（prepareData.py）
# -*- coding:utf-8 -*-
import math
import os
import random
import getConfig
from tensorflow.python.platform import gfile
import re
# 特殊标记，用来填充标记对话
PAD = "__PAD__"
GO = "__GO__"
EOS = "__EOS__" # 对话结束
UNK = "__UNK__" # 标记未出现在词汇表中的字符
START_VOCABULART = [PAD, GO, EOS, UNK]
PAD_ID = 0
GO_ID = 1
EOS_ID = 2
UNK_ID = 3
# 定义字典生成函数
def create_vocabulary(input_file, vocabulary_size, output_file):
vocabulary = {}
k = int(vocabulary_size)
with open(input_file, 'r', encoding="utf-8") as f:
counter = 0
for line in f:
counter += 1
tokens = [word for word in line.split()]
for word in tokens:
if word in vocabulary:
vocabulary[word] += 1
else:
vocabulary[word] = 1
vocabulary_list = START_VOCABULART + sorted(vocabulary, key=vocabulary.get, reverse=True)
# 取前20000个常用汉字
if len(vocabulary_list) > k:
vocabulary_list = vocabulary_list[:k]
print(input_file + " 词汇表大小:", len(vocabulary_list))
with open(output_file, 'w', encoding="utf-8") as ff:
for word in vocabulary_list:
ff.write(word + "\n")
# 把对话字符串转为向量形式
def convert_to_vector(input_file, vocabulary_file, output_file):
print('对话转向量...')
tmp_vocab = []
with open(vocabulary_file, "r", encoding="utf-8") as f: # 读取字典文件的数据，生成一个dict，也就是键值对的字典
tmp_vocab.extend(f.readlines())
tmp_vocab = [line.strip() for line in tmp_vocab]
vocab = dict([(x, y) for (y, x) in
enumerate(tmp_vocab)]) # 将vocabulary_file中的键值对互换，因为在字典文件里是按照{123：好}这种格式存储的，我们需要换成{好：123}格式
output_f = open(output_file, 'w', encoding="utf-8")
with open(input_file, 'r', encoding="utf-8") as f:
for line in f:
line_vec = []
for words in line.split():
line_vec.append(vocab.get(words, UNK_ID))
output_f.write(
" ".join([str(num) for num in line_vec]) + "\n") # 将input_file里的中文字符通过查字典的方式，替换成对应的key，并保存在output_file
output_f.close()
def prepare_custom_data(working_directory, train_enc, train_dec, test_enc, test_dec, enc_vocabulary_size,
dec_vocabulary_size, tokenizer=None):
# Create vocabularies of the appropriate sizes.
enc_vocab_path = os.path.join(working_directory, "vocab%d.enc" % enc_vocabulary_size)
dec_vocab_path = os.path.join(working_directory, "vocab%d.dec" % dec_vocabulary_size)
create_vocabulary(train_enc, enc_vocabulary_size, enc_vocab_path)
create_vocabulary(train_dec, dec_vocabulary_size, dec_vocab_path)
# Create token ids for the training data.
enc_train_ids_path = train_enc + (".ids%d" % enc_vocabulary_size)
dec_train_ids_path = train_dec + (".ids%d" % dec_vocabulary_size)
convert_to_vector(train_enc, enc_vocab_path, enc_train_ids_path)
convert_to_vector(train_dec, dec_vocab_path, dec_train_ids_path)
# Create token ids for the development data.
enc_dev_ids_path = test_enc + (".ids%d" % enc_vocabulary_size)
dec_dev_ids_path = test_dec + (".ids%d" % dec_vocabulary_size)
convert_to_vector(test_enc, enc_vocab_path, enc_dev_ids_path)
convert_to_vector(test_dec, dec_vocab_path, dec_dev_ids_path)
return (enc_train_ids_path, dec_train_ids_path, enc_dev_ids_path, dec_dev_ids_path, enc_vocab_path, dec_vocab_path)
# 用于语句切割的正则表达
_WORD_SPLIT = re.compile(b"([.,!?\"':;)(])")
_DIGIT_RE = re.compile(br"\d")
def basic_tokenizer(sentence):
# 将一个语句中的字符切割成一个list，这样是为了下一步进行向量化训练
words = []
for space_separated_fragment in sentence.strip().split():
words.extend(re.split(_WORD_SPLIT, space_separated_fragment))
return [w for w in words if w]
def sentence_to_token_ids(sentence, vocabulary, normalize_digits=True): # 将输入语句从中文字符转换成数字符号
words = basic_tokenizer(sentence)
if not normalize_digits:
return [vocabulary.get(w, UNK_ID) for w in words]
# Normalize digits by 0 before looking words up in the vocabulary.
return [vocabulary.get(re.sub(_DIGIT_RE, b"0", w), UNK_ID) for w in words]
def initialize_vocabulary(vocabulary_path): # 初始化字典，这里的操作与上面的48行的的作用是一样的，是对调字典中的key-value
if gfile.Exists(vocabulary_path):
rev_vocab = []
with open(vocabulary_path, "r", encoding="utf-8") as f:
rev_vocab.extend(f.readlines())
rev_vocab = [line.strip() for line in rev_vocab]
vocab = dict([(x, y) for (y, x) in enumerate(rev_vocab)])
return vocab, rev_vocab
else:
raise ValueError("Vocabulary file %s not found.", vocabulary_path)
3.seq2seq 模型（seq2seq_model.py）
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Sequence-to-sequence model with an attention mechanism."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import random
import numpy as np
from six.moves import xrange # pylint: disable=redefined-builtin
import tensorflow as tf
import prepareData
import copy
class Seq2SeqModel(object):
"""Sequence-to-sequence model with attention and for multiple buckets.
This class implements a multi-layer recurrent neural network as encoder,
and an attention-based decoder. This is the same as the model described in
this paper: http://arxiv.org/abs/1412.7449 - please look there for details,
or into the seq2seq library for complete model implementation.
This class also allows to use GRU cells in addition to LSTM cells, and
sampled softmax to handle large output vocabulary size. A single-layer
version of this model, but with bi-directional encoder, was presented in
http://arxiv.org/abs/1409.0473
and sampled softmax is described in Section 3 of the following paper.
http://arxiv.org/abs/1412.2007
"""
def __init__(self, source_vocab_size, target_vocab_size, buckets, size,
num_layers, max_gradient_norm, batch_size, learning_rate,
learning_rate_decay_factor, use_lstm=False,
num_samples=512, forward_only=False):
"""Create the model.
Args:
source_vocab_size: size of the source vocabulary.
target_vocab_size: size of the target vocabulary.
buckets: a list of pairs (I, O), where I specifies maximum input length
that will be processed in that bucket, and O specifies maximum output
length. Training instances that have inputs longer than I or outputs
longer than O will be pushed to the next bucket and padded accordingly.
We assume that the list is sorted, e.g., [(2, 4), (8, 16)].
size: number of units in each layer of the model.
num_layers: number of layers in the model.
max_gradient_norm: gradients will be clipped to maximally this norm.
batch_size: the size of the batches used during training;
the model construction is independent of batch_size, so it can be
changed after initialization if this is convenient, e.g., for decoding.
learning_rate: learning rate to start with.
learning_rate_decay_factor: decay learning rate by this much when needed.
use_lstm: if true, we use LSTM cells instead of GRU cells.
num_samples: number of samples for sampled softmax.
forward_only: if set, we do not construct the backward pass in the model.
"""
self.source_vocab_size = source_vocab_size
self.target_vocab_size = target_vocab_size
self.buckets = buckets
self.batch_size = batch_size
self.learning_rate = tf.Variable(float(learning_rate), trainable=False)
self.learning_rate_decay_op = self.learning_rate.assign(self.learning_rate * learning_rate_decay_factor)
self.global_step = tf.Variable(0, trainable=False)
# If we use sampled softmax, we need an output projection.
output_projection = None
softmax_loss_function = None
# Sampled softmax only makes sense if we sample less than vocabulary size.
if num_samples > 0 and num_samples < self.target_vocab_size:
w = tf.get_variable("proj_w", [size, self.target_vocab_size])
w_t = tf.transpose(w)
b = tf.get_variable("proj_b", [self.target_vocab_size])
output_projection = (w, b)
def sampled_loss(labels,logits):
labels = tf.reshape(labels, [-1, 1])
return tf.nn.sampled_softmax_loss(w_t, b, labels,logits,num_samples,self.target_vocab_size)
softmax_loss_function = sampled_loss
# Create the internal multi-layer cell for our RNN.
single_cell = tf.contrib.rnn.GRUCell(size)
if use_lstm:
single_cell = tf.contrib.rnn.BasicLSTMCell(size)
cell = single_cell
if num_layers > 1:
cell = tf.contrib.rnn.MultiRNNCell([single_cell] * num_layers)
# The seq2seq function: we use embedding for the input and attention.
def seq2seq_f(encoder_inputs, decoder_inputs, do_decode):
tem_cell=copy.deepcopy(cell)
return tf.contrib.legacy_seq2seq.embedding_attention_seq2seq(
encoder_inputs, decoder_inputs, tem_cell,
num_encoder_symbols=source_vocab_size,
num_decoder_symbols=target_vocab_size,
embedding_size=size,
output_projection=output_projection,
feed_previous=do_decode)
# Feeds for inputs.
self.encoder_inputs = []
self.decoder_inputs = []
self.target_weights = []
for i in xrange(buckets[-1][0]): # Last bucket is the biggest one.
self.encoder_inputs.append(tf.placeholder(tf.int32, shape=[None],
name="encoder{0}".format(i)))
for i in xrange(buckets[-1][1] + 1):
self.decoder_inputs.append(tf.placeholder(tf.int32, shape=[None],
name="decoder{0}".format(i)))
self.target_weights.append(tf.placeholder(tf.float32, shape=[None],
name="weight{0}".format(i)))
# Our targets are decoder inputs shifted by one.
targets = [self.decoder_inputs[i + 1]
for i in xrange(len(self.decoder_inputs) - 1)]
# Training outputs and losses.
if forward_only:
self.outputs, self.losses = tf.contrib.legacy_seq2seq.model_with_buckets(
self.encoder_inputs, self.decoder_inputs, targets,
self.target_weights, buckets, lambda x, y: seq2seq_f(x, y, True),
softmax_loss_function=softmax_loss_function)
# If we use output projection, we need to project outputs for decoding.
if output_projection is not None:
for b in xrange(len(buckets)):
self.outputs[b] = [
tf.matmul(output, output_projection[0]) + output_projection[1]
for output in self.outputs[b]
]
else:
self.outputs, self.losses = tf.contrib.legacy_seq2seq.model_with_buckets(
self.encoder_inputs, self.decoder_inputs, targets,
self.target_weights, buckets,
lambda x, y: seq2seq_f(x, y, False),
softmax_loss_function=softmax_loss_function)
# Gradients and SGD update operation for training the model.
params = tf.trainable_variables()
if not forward_only:
self.gradient_norms = []
self.updates = []
opt = tf.train.GradientDescentOptimizer(self.learning_rate)
for b in xrange(len(buckets)):
gradients = tf.gradients(self.losses[b], params)
clipped_gradients, norm = tf.clip_by_global_norm(gradients,
max_gradient_norm)
self.gradient_norms.append(norm)
self.updates.append(opt.apply_gradients(
zip(clipped_gradients, params), global_step=self.global_step))
self.saver = tf.train.Saver(tf.all_variables())
def step(self, session, encoder_inputs, decoder_inputs, target_weights,
bucket_id, forward_only):
"""Run a step of the model feeding the given inputs.
Args:
session: tensorflow session to use.
encoder_inputs: list of numpy int vectors to feed as encoder inputs.
decoder_inputs: list of numpy int vectors to feed as decoder inputs.
target_weights: list of numpy float vectors to feed as target weights.
bucket_id: which bucket of the model to use.
forward_only: whether to do the backward step or only forward.
Returns:
A triple consisting of gradient norm (or None if we did not do backward),
average perplexity, and the outputs.
Raises:
ValueError: if length of encoder_inputs, decoder_inputs, or
target_weights disagrees with bucket size for the specified bucket_id.
"""
# Check if the sizes match.
encoder_size, decoder_size = self.buckets[bucket_id]
if len(encoder_inputs) != encoder_size:
raise ValueError("Encoder length must be equal to the one in bucket,"
" %d != %d." % (len(encoder_inputs), encoder_size))
if len(decoder_inputs) != decoder_size:
raise ValueError("Decoder length must be equal to the one in bucket,"
" %d != %d." % (len(decoder_inputs), decoder_size))
if len(target_weights) != decoder_size:
raise ValueError("Weights length must be equal to the one in bucket,"
" %d != %d." % (len(target_weights), decoder_size))
# Input feed: encoder inputs, decoder inputs, target_weights, as provided.
input_feed = {}
for l in xrange(encoder_size):
input_feed[self.encoder_inputs[l].name] = encoder_inputs[l]
for l in xrange(decoder_size):
input_feed[self.decoder_inputs[l].name] = decoder_inputs[l]
input_feed[self.target_weights[l].name] = target_weights[l]
# Since our targets are decoder inputs shifted by one, we need one more.
last_target = self.decoder_inputs[decoder_size].name
input_feed[last_target] = np.zeros([self.batch_size], dtype=np.int32)
# Output feed: depends on whether we do a backward step or not.
if not forward_only:
output_feed = [self.updates[bucket_id], # Update Op that does SGD.
self.gradient_norms[bucket_id], # Gradient norm.
self.losses[bucket_id]] # Loss for this batch.
else:
output_feed = [self.losses[bucket_id]] # Loss for this batch.
for l in xrange(decoder_size): # Output logits.
output_feed.append(self.outputs[bucket_id][l])
outputs = session.run(output_feed, input_feed)
if not forward_only:
return outputs[1], outputs[2], None # Gradient norm, loss, no outputs.
else:
return None, outputs[0], outputs[1:] # No gradient norm, loss, outputs.
def get_batch(self, data, bucket_id):
"""Get a random batch of data from the specified bucket, prepare for step.
To feed data in step(..) it must be a list of batch-major vectors, while
data here contains single length-major cases. So the main logic of this
function is to re-index data cases to be in the proper format for feeding.
Args:
data: a tuple of size len(self.buckets) in which each element contains
lists of pairs of input and output data that we use to create a batch.
bucket_id: integer, which bucket to get the batch for.
Returns:
The triple (encoder_inputs, decoder_inputs, target_weights) for
the constructed batch that has the proper format to call step(...) later.
"""
encoder_size, decoder_size = self.buckets[bucket_id]
encoder_inputs, decoder_inputs = [], []
# Get a random batch of encoder and decoder inputs from data,
# pad them if needed, reverse encoder inputs and add GO to decoder.
for _ in xrange(self.batch_size):
encoder_input, decoder_input = random.choice(data[bucket_id])
# Encoder inputs are padded and then reversed.
encoder_pad = [prepareData.PAD_ID] * (encoder_size - len(encoder_input))
encoder_inputs.append(list(reversed(encoder_input + encoder_pad)))
# Decoder inputs get an extra "GO" symbol, and are padded then.
decoder_pad_size = decoder_size - len(decoder_input) - 1
decoder_inputs.append([prepareData.GO_ID] + decoder_input +
[prepareData.PAD_ID] * decoder_pad_size)
# Now we create batch-major vectors from the data selected above.
batch_encoder_inputs, batch_decoder_inputs, batch_weights = [], [], []
# Batch encoder inputs are just re-indexed encoder_inputs.
for length_idx in xrange(encoder_size):
batch_encoder_inputs.append(
np.array([encoder_inputs[batch_idx][length_idx]
for batch_idx in xrange(self.batch_size)], dtype=np.int32))
# Batch decoder inputs are re-indexed decoder_inputs, we create weights.
for length_idx in xrange(decoder_size):
batch_decoder_inputs.append(
np.array([decoder_inputs[batch_idx][length_idx]
for batch_idx in xrange(self.batch_size)], dtype=np.int32))
# Create target_weights to be 0 for targets that are padding.
batch_weight = np.ones(self.batch_size, dtype=np.float32)
for batch_idx in xrange(self.batch_size):
# We set weight to 0 if the corresponding target is a PAD symbol.
# The corresponding target is decoder_input shifted by 1 forward.
if length_idx < decoder_size - 1:
target = decoder_inputs[batch_idx][length_idx + 1]
if length_idx == decoder_size - 1 or target == prepareData.PAD_ID:
batch_weight[batch_idx] = 0.0
batch_weights.append(batch_weight)
return batch_encoder_inputs, batch_decoder_inputs, batch_weights
- 执行器（execute.py）
# -*- coding:utf-8 -*-
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import math
import os
import random
import sys
import time
import numpy as np
from six.moves import xrange
import tensorflow as tf
from configparser import ConfigParser
import prepareData
import seq2seq_model
gConfig = {}
def get_config(config_file='seq2seq.ini'):
parser = ConfigParser()
parser.read(config_file, encoding="utf-8")
# get the ints, floats and strings
_conf_ints = [(key, int(value)) for key, value in parser.items('ints')]
_conf_floats = [(key, float(value)) for key, value in parser.items('floats')]
_conf_strings = [(key, str(value)) for key, value in parser.items('strings')]
return dict(_conf_ints + _conf_floats + _conf_strings)
# 设置不同的桶以及桶的长度，原则两个：1、尽量覆盖所有训练语料语句的长度，2、尽量保持桶里语料的平衡
_buckets = [(1, 10), (10, 15), (20, 25), (40, 50)]
def read_data(source_path, target_path, max_size=None):
"""Read data from source and target files and put into buckets.
Args:
source_path: path to the files with token-ids for the source language.
target_path: path to the file with token-ids for the target language;
it must be aligned with the source file: n-th line contains the desired
output for n-th line from the source_path.
max_size: maximum number of lines to read, all other will be ignored;
if 0 or None, data files will be read completely (no limit).
Returns:
data_set: a list of length len(_buckets); data_set[n] contains a list of
(source, target) pairs read from the provided data files that fit
into the n-th bucket, i.e., such that len(source) < _buckets[n][0] and
len(target) < _buckets[n][1]; source and target are lists of token-ids.
"""
data_set = [[] for _ in _buckets]
with tf.gfile.GFile(source_path, mode="r") as source_file:
with tf.gfile.GFile(target_path, mode="r") as target_file:
source, target = source_file.readline(), target_file.readline()
counter = 0
while source and target and (not max_size or counter < max_size):
counter += 1
if counter % 100000 == 0:
print(" reading data line %d" % counter)
sys.stdout.flush()
source_ids = [int(x) for x in source.split()]
target_ids = [int(x) for x in target.split()]
target_ids.append(prepareData.EOS_ID)
for bucket_id, (source_size, target_size) in enumerate(_buckets):
if len(source_ids) < source_size and len(target_ids) < target_size:
data_set[bucket_id].append([source_ids, target_ids])
break
source, target = source_file.readline(), target_file.readline()
return data_set
def create_model(session, forward_only):
"""Create model and initialize or load parameters"""
model = seq2seq_model.Seq2SeqModel(gConfig['enc_vocab_size'], gConfig['dec_vocab_size'], _buckets,
gConfig['layer_size'], gConfig['num_layers'], gConfig['max_gradient_norm'],
gConfig['batch_size'], gConfig['learning_rate'],
gConfig['learning_rate_decay_factor'], forward_only=forward_only)
if 'pretrained_model' in gConfig:
model.saver.restore(session, gConfig['pretrained_model'])
return model
ckpt = tf.train.get_checkpoint_state(gConfig['working_directory'])
"""if ckpt and tf.gfile.Exists(ckpt.model_checkpoint_path):"""
if ckpt and ckpt.model_checkpoint_path:
print("Reading model parameters from %s" % ckpt.model_checkpoint_path)
model.saver.restore(session, ckpt.model_checkpoint_path)
else:
print("Created model with fresh parameters.")
session.run(tf.global_variables_initializer())
return model
def train():
# prepare dataset
print("Preparing data in %s" % gConfig['working_directory'])
enc_train, dec_train, enc_dev, dec_dev, _, _ = prepareData.prepare_custom_data(gConfig['working_directory'],
gConfig['train_enc'],
gConfig['train_dec'],
gConfig['test_enc'],
gConfig['test_dec'],
gConfig['enc_vocab_size'],
gConfig['dec_vocab_size'])
# setup config to use BFC allocator
config = tf.ConfigProto() # tf.ConfigProto()主要的作用是配置tf.Session的运算方式，比如gpu运算或者cpu运算
config.gpu_options.allocator_type = 'BFC' # 使用BFC算法
with tf.Session(config=config) as sess:
# Create model.
print("Creating %d layers of %d units." % (gConfig['num_layers'], gConfig['layer_size']))
model = create_model(sess, False)
# 读取数据并计算其长度，将其放入对应的桶中
print("Reading development and training data (limit: %d)." % gConfig['max_train_data_size'])
dev_set = read_data(enc_dev, dec_dev)
train_set = read_data(enc_train, dec_train, gConfig['max_train_data_size'])
train_bucket_sizes = [len(train_set[b]) for b in xrange(len(_buckets))]
train_total_size = float(sum(train_bucket_sizes))
# A bucket scale is a list of increasing numbers from 0 to 1 that we'll use
# to select a bucket. Length of [scale[i], scale[i+1]] is proportional to
# the size if i-th training bucket, as used later.
train_buckets_scale = [sum(train_bucket_sizes[:i + 1]) / train_total_size
for i in xrange(len(train_bucket_sizes))]
# This is the training loop.
step_time, loss = 0.0, 0.0
current_step = 0
previous_losses = []
while True:
# Choose a bucket according to data distribution. We pick a random number
# in [0, 1] and use the corresponding interval in train_buckets_scale.
random_number_01 = np.random.random_sample()
bucket_id = min([i for i in xrange(len(train_buckets_scale))
if train_buckets_scale[i] > random_number_01])
# Get a batch and make a step.
start_time = time.time()
encoder_inputs, decoder_inputs, target_weights = model.get_batch(
train_set, bucket_id)
_, step_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
target_weights, bucket_id, False)
step_time += (time.time() - start_time) / gConfig['steps_per_checkpoint']
loss += step_loss / gConfig['steps_per_checkpoint']
current_step += 1
# Once in a while, we save checkpoint, print statistics, and run evals.
if current_step % gConfig['steps_per_checkpoint'] == 0:
# jian
perplexity = math.exp(loss) if loss < 300 else float('inf')
print("global step %d learning rate %.4f step-time %.2f perplexity "
"%.2f" % (model.global_step.eval(), model.learning_rate.eval(),
step_time, perplexity))
# Decrease learning rate if no improvement was seen over last 3 times.
if len(previous_losses) > 2 and loss > max(previous_losses[-3:]):
sess.run(model.learning_rate_decay_op)
previous_losses.append(loss)
# Save checkpoint and zero timer and loss.
checkpoint_path = os.path.join(gConfig['working_directory'], "seq2seq.ckpt")
model.saver.save(sess, checkpoint_path, global_step=model.global_step)
step_time, loss = 0.0, 0.0
# Run evals on development set and print their perplexity.
for bucket_id in xrange(len(_buckets)):
if len(dev_set[bucket_id]) == 0:
print(" eval: empty bucket %d" % (bucket_id))
continue
encoder_inputs, decoder_inputs, target_weights = model.get_batch(
dev_set, bucket_id)
_, eval_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
target_weights, bucket_id, True)
eval_ppx = math.exp(eval_loss) if eval_loss < 300 else float('inf')
print(" eval: bucket %d perplexity %.2f" % (bucket_id, eval_ppx))
sys.stdout.flush()
def self_test():
"""Test the translation model."""
with tf.Session() as sess:
print("Self-test for neural translation model.")
# Create model with vocabularies of 10, 2 small buckets, 2 layers of 32.
model = seq2seq_model.Seq2SeqModel(10, 10, [(3, 3), (6, 6)], 32, 2,
5.0, 32, 0.3, 0.99, num_samples=8)
sess.run(tf.initialize_all_variables())
# Fake data set for both the (3, 3) and (6, 6) bucket.
data_set = ([([1, 1], [2, 2]), ([3, 3], [4]), ([5], [6])],
[([1, 1, 1, 1, 1], [2, 2, 2, 2, 2]), ([3, 3, 3], [5, 6])])
for _ in xrange(5): # Train the fake model for 5 steps.
bucket_id = random.choice([0, 1])
encoder_inputs, decoder_inputs, target_weights = model.get_batch(
data_set, bucket_id)
model.step(sess, encoder_inputs, decoder_inputs, target_weights,
bucket_id, False)
def init_session(sess, conf='seq2seq.ini'):
global gConfig
gConfig = get_config(conf)
# Create model and load parameters.
model = create_model(sess, True)
model.batch_size = 1 # We decode one sentence at a time.
# Load vocabularies.
enc_vocab_path = os.path.join(gConfig['working_directory'], "vocab%d.enc" % gConfig['enc_vocab_size'])
dec_vocab_path = os.path.join(gConfig['working_directory'], "vocab%d.dec" % gConfig['dec_vocab_size'])
enc_vocab, _ = prepareData.initialize_vocabulary(enc_vocab_path)
_, rev_dec_vocab = prepareData.initialize_vocabulary(dec_vocab_path)
return sess, model, enc_vocab, rev_dec_vocab
def decode_line(sess, model, enc_vocab, rev_dec_vocab, sentence):
# Get token-ids for the input sentence.
token_ids = prepareData.sentence_to_token_ids(tf.compat.as_bytes(sentence), enc_vocab)
# Which bucket does it belong to?
bucket_id = min([b for b in xrange(len(_buckets)) if _buckets[b][0] > len(token_ids)])
# Get a 1-element batch to feed the sentence to the model.
encoder_inputs, decoder_inputs, target_weights = model.get_batch({bucket_id: [(token_ids, [])]}, bucket_id)
# Get output logits for the sentence.
_, _, output_logits = model.step(sess, encoder_inputs, decoder_inputs, target_weights, bucket_id, True)
# This is a greedy decoder - outputs are just argmaxes of output_logits.
outputs = [int(np.argmax(logit, axis=1)) for logit in output_logits]
# If there is an EOS symbol in outputs, cut them at that point.
if prepareData.EOS_ID in outputs:
outputs = outputs[:outputs.index(prepareData.EOS_ID)]
return " ".join([tf.compat.as_str(rev_dec_vocab[output]) for output in outputs])
if __name__ == '__main__':
if len(sys.argv) - 1:
gConfig = get_config(sys.argv[1])
else:
# get configuration from seq2seq.ini
gConfig = get_config()
print('\n>> Mode : %s\n' % (gConfig['mode']))
if gConfig['mode'] == 'train':
# start training
train()
elif gConfig['mode'] == 'server':
print('Serve Usage : >> python3 webui/app.py')
print('# uses seq2seq_serve.ini as conf file')
6. 可视化展示模块（app.py）*
#!/usr/bin/env Python
# coding=utf-8
from flask import Flask, render_template, request, make_response
from flask import jsonify
import sys
import time
import hashlib
import threading
def heartbeat():
print (time.strftime('%Y-%m-%d %H:%M:%S - heartbeat', time.localtime(time.time())))
timer = threading.Timer(60, heartbeat)
timer.start()
timer = threading.Timer(60, heartbeat)
timer.start()
try:
import xml.etree.cElementTree as ET
except ImportError:
import xml.etree.ElementTree as ET
import re
zhPattern = re.compile(u'[\u4e00-\u9fa5]+')
app = Flask(__name__,static_url_path="/static")
@app.route('/message', methods=['POST'])
def reply():
req_msg = request.form['msg']
res_msg = '^_^'
print(req_msg)
print(''.join([f+' ' for fh in req_msg for f in fh]))
req_msg=''.join([f+' ' for fh in req_msg for f in fh])
print(req_msg)
res_msg = execute.decode_line(sess, model, enc_vocab, rev_dec_vocab, req_msg )
res_msg = res_msg.replace('_UNK', '^_^')
res_msg=res_msg.strip()
# 如果接受到的内容为空，则给出相应的恢复
if res_msg == '':
res_msg = '请与我聊聊天吧'
return jsonify( { 'text': res_msg } )
@app.route("/")
def index():
return render_template("index.html")
#
'''
初始化seq2seqModel，并进行动作
1. 调用执行器的主程序
2. 生成一个在线decode进程，来提供在线聊天服务
'''
#_________________________________________________________________
import tensorflow as tf
import execute
sess = tf.Session()
sess, model, enc_vocab, rev_dec_vocab = execute.init_session(sess, conf='seq2seq_serve.ini')
#_________________________________________________________________
# 启动APP
if (__name__ == "__main__"):
app.run(host = '127.0.0.1', port = 8808)
扫一扫
774
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
