seq2seq+attention=>评论摘要

Summarizing Text with Amazon Reviews
数据集：Amazon 500000评论

本节内容:

• 数据预处理
• 构建Seq2Seq模型
• 训练网络
• 测试效果

seq2seq教程: https://github.com/j-min/tf_tutorial_plus/tree/master/RNN_seq2seq/contrib_seq2seq
https://www.jianshu.com/p/c0c5f1bdbb88

浏览数据集

将少量的含有缺失值的记录和其他属性dropout

1、数据预处理

• 全部转换成小写
• 连词转换
• 去停用词（只在描述中去掉）
• 去除特殊字符https?:\/\/.*[\r\n]*\<a href[_"\-;%()|+&=*%.,!?:#$@\[\]/]

re.sub(r'https?:\/\/.*[\r\n]*\<a href[_"\-;%()|+&=*%.,!?:#$@\[\]/]`', ' ', text)

• 连词转换：
  “can’t’ve”: “cannot have”,
  “'cause”: “because”,
  “could’ve”: “could have”,

# 统计不重复的词的个数
def count_words(count_dict, text):
    '''Count the number of occurrences of each word in a set of text'''
    for sentence in text:
        for word in sentence.split():
            if word not in count_dict:
                count_dict[word] = 1
            else:
                count_dict[word] += 1

# Find the number of times each word was used and the size of the vocabulary
word_counts = {}

count_words(word_counts, clean_summaries)
print("Size of Vocabulary:", len(word_counts))
count_words(word_counts, clean_texts)
print("Size of Vocabulary:", len(word_counts))

OUT:
Size of Vocabulary: 33809
Size of Vocabulary: 132884

我们的任务通过seq2seq完成encode-decod的操作，需要词向量作为输入，这里的词向量也可以用别人训练好的词向量喂入模型。github上有训练好的词向量

格式：[0]为单词 ,[1：]为向量
/c/en/absolute_value -0.0847 -0.1316 -0.0800 -0.0708 -0.2514 -0.1687 -…

现将单词和其向量转换为字典的格式

embeddings_index = {}
with open('numberbatch-en-17.04b.txt', encoding='utf-8') as f:
    for line in f:
        values = line.split(' ')
        word = values[0]
        embedding = np.asarray(values[1:], dtype='float32')
        embeddings_index[word] = embedding

词库中的词不一样全都在词向量文件中，所以需要统计词库不在词向量文件中的单词，如果是词频大于20，且词向量文件中没有，就直接建立一个相同维度的向量训练即可。

统计miss单词的占比：

Number of words missing from CN: 3866
Percent of words that are missing from vocabulary: 2.91%

现将词频大于20的构建数字和单词的映射，方便计算机理解。
将标识符也加入词汇中，开始、PAD、unknow和结束符号

codes = ["<UNK>","<PAD>","<EOS>","<GO>"]   

int_to_vocab：

vocab_to_int：

做好词语和数字的映射之后，再将每一个语句中的单词转换为数字形式：有些词如词频小于20、还没有过滤掉的则替换为’unknown’

def convert_to_ints(text, word_count, unk_count, eos=False):
    '''Convert words in text to an integer.
       If word is not in vocab_to_int, use UNK's integer.  ’unknown‘
       Total the number of words and UNKs.
       Add EOS token to the end of texts'''
    
    ints = []
    for sentence in text:
        sentence_ints = []
        for word in sentence.split():
            word_count += 1
            if word in vocab_to_int:
                sentence_ints.append(vocab_to_int[word])
            else:
                sentence_ints.append(vocab_to_int["<UNK>"])
                unk_count += 1
        if eos:
            sentence_ints.append(vocab_to_int["<EOS>"])
        ints.append(sentence_ints)
    return ints, word_count, unk_count

得到数字表示的句子：

为了更能清晰的观察数据，将每个句子的长度转换DataFrame，再用describe函数查看具体情况

查看%90、95、99分位的数值

现在再对句子进行一次过滤，

• 1、对summaries和text过滤：
  一句话中unk太多，设定阈值过滤一些句子
• 2、对summaries和text的长度排序，只取%90分位以下的长度值即可

sorted_summaries = []
sorted_texts = []
max_text_length = 84
max_summary_length = 13
min_length = 2
unk_text_limit = 1
unk_summary_limit = 0

for length in range(min(lengths_texts.counts), max_text_length): 
    for count, words in enumerate(int_summaries):
        if (len(int_summaries[count]) >= min_length and
            len(int_summaries[count]) <= max_summary_length and
            len(int_texts[count]) >= min_length and
            unk_counter(int_summaries[count]) <= unk_summary_limit and
            unk_counter(int_texts[count]) <= unk_text_limit and
            length == len(int_texts[count])
           ):
            sorted_summaries.append(int_summaries[count])

到这里，就完成了对数据的筛选，处理部分，接下来构造Seq2Seq模型

2、构造Seq2Seq模型

Bidirectional RNNs(双向网络)的改进之处便是，假设当前的输出(第t步的输出)不仅仅与前面的序列有关，并且还与后面的序列有关。

例如：预测一个语句中缺失的词语那么就需要根据上下文来进行预测。Bidirectional RNNs是一个相对较简单的RNNs，是由两个RNNs上下叠加在一起组成的。输出由这两个RNNs的隐藏层的状态决定的

• 输入xt乘以权值W+记忆乘以权重V+b
  

（1）数据输入

将数据切分，在每个句子前加‘’标识符

def process_encoding_input(target_data, vocab_to_int, batch_size):
    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''
    
    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])
    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)

    return dec_input

strded_slicc(起始，结束，步长)，举例：

（2）encoding_layer

动态双向RNN，输出enc_output, enc_state

def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):
    '''Create the encoding layer'''
    
    for layer in range(num_layers):
        with tf.variable_scope('encoder_{}'.format(layer)):
            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, 
                                                    input_keep_prob = keep_prob)

            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, 
                                                    input_keep_prob = keep_prob)

            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, 
                                                                    cell_bw, 
                                                                    rnn_inputs,
                                                                    sequence_length,# 一个batchSize的大小
                                                                    dtype=tf.float32)
    # Join outputs since we are using a bidirectional RNN    
    enc_output = tf.concat(enc_output,2)
    
    return enc_output, enc_state

(3)训练，辅助层

1、训练：training_decoding_layer：
tf.contrib.seq2seq.    
			TrainingHelper
			BasicDecoder
			dynamic_decode

2、测试：inference_decoding_layer
tf.contrib.seq2seq.
			GreedyEmbeddingHelper
			BasicDecoder
			dynamic_decode

(4)decoding_layer

LSTMCell
BahdanauAttention
AttentionWrapper

with tf.variable_scope("decode"):
        training_logits = training_decoding_layer(dec_embed_input, 
                                                  summary_length, 
                                                  dec_cell, 
                                                  initial_state,
                                                  output_layer,
                                                  vocab_size, 
                                                  max_summary_length)
    with tf.variable_scope("decode", reuse=True):
        inference_logits = inference_decoding_layer(embeddings,  
                                                    vocab_to_int['<GO>'], 
                                                    vocab_to_int['<EOS>'],
                                                    dec_cell, 
                                                    initial_state, 
                                                    output_layer,
                                                    max_summary_length,
                                                    batch_size)

(5)seq2seq_model

将上面建立的操作结合

(6)Graph built

get_batches 得到batch
pad_sentence_batch 将长度不够的句子用’PAD‘补齐
tf.contrib.seq2seq.sequence_loss
tf.train.AdamOptimizer