基于seq2seq+attention的中文语音识别

好久没有写博客码字了

本人的中文语音识别跟小米的语音识别作者有过沟通（即参考论文1的作者），期望能够实现一个完整版的中文语音识别模型，那么这就开始啦

提纲如下：

1.数据准备

2.seq2seq介绍

3.Attention介绍

4.bilstm介绍

5.bilstm + seq2seq+Attention

1.数据准备

2.seq2seq介绍

介绍seq2seq的博客满天飞（英语好的同学可以看tensorflow的官方教程：https://www.tensorflow.org/tutorials/seq2seq），这里简单介绍一下，熟悉seq2seq模型的同学直接跳过

seq2seq就是把一个序列翻译成另一个序列的模型，实质就是两个rnn，一个是encoder，另一个是decoder，encoder负责将source序列编码成固定长度的表达，decoder负责将该固定长度的表达解码成target序列，刚开始是在机器翻译上使用的，其实这个模型应用非常广泛，凡是变长之间的映射关系都可以做，常见是机器翻译，语音识别，摘要提取等等；还可以把encoder和decoder拆开使用。

seq2seq当前比较流行的结构如下（https://arxiv.org/pdf/1409.3215.pdf）

seq2seq模型的encoder非常简单（上图中ABC对应的部分），就是rnn，可以是多层（simple-rnn，GRU，LSTM），decoder在训练和测试的时候，稍微有点不同

decoder训练的时候输入由两部分组成，一部分是encoder的last state，另一部分是target序列，如上图中的<GO> WXYZ;其中<GO>和<EOS>表示的是序列开始符和结束符;

decoder 测试的时候输入也是由两部分组成，一部分是encoder的last state，另一部分是来自于上一个时刻的输出（上一个时刻的输出作为下一个时刻的输入），直到某个时刻的输出遇到结束符<EOS>为止,网络结构如下：

注意⚠️：decoder测试的时候需要重用 cell的参数！

对于encoder 和 decoder 的理论内容也就这么多啦，下面说一下decoder部分，tensorflow是如何来实现训练和测试的不同操作的

tensorflow在decoder进行train test的时候，使用的目录下的各种tf.contrib.seq2seq目录下的各种helper方法来区分不同的输入

Helper

常用的Helper：

• TrainingHelper：适用于训练的helper。
• InferenceHelper：适用于测试的helper。
• GreedyEmbeddingHelper：适用于测试中采用Greedy策略sample的helper。
• CustomHelper：用户自定义的helper。

以下贴出我常用的decoder train和test的函数

def decoding_layer_train(encoder_state, dec_cell, dec_embed_input,
                         target_sequence_length, max_summary_length,
                         output_layer, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_summary_length: The length of the longest sequence in the batch
    :param output_layer: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing training logits and sample_id
    """
    helper = tf.contrib.seq2seq.TrainingHelper(inputs = dec_embed_input, sequence_length = target_sequence_length,time_major = False)
    decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, helper, encoder_state, output_layer=output_layer)
    dec_outputs, dec_state,dec_sequence_length = tf.contrib.seq2seq.dynamic_decode(decoder, impute_finished=True, maximum_iterations=max_summary_length)
    return dec_outputs

def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,
                         end_of_sequence_id, max_target_sequence_length,
                         vocab_size, output_layer, batch_size, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param max_target_sequence_length: Maximum length of target sequences
    :param vocab_size: Size of decoder/target vocabulary
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_layer: Function to apply the output layer
    :param batch_size: Batch size
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing inference logits and sample_id
    """
    # TODO: Implement Function
    start_tokens = tf.tile(tf.constant([start_of_sequence_id], dtype=tf.int32), [batch_size], name='start_tokens')
    helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(dec_embeddings,
        start_tokens, end_of_sequence_id)
    decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, helper, encoder_state, output_layer=output_layer)
    dec_outputs, dec_state,dec_sequence_length = tf.contrib.seq2seq.dynamic_decode(decoder,impute_finished=True,
                                        maximum_iterations=max_target_sequence_length)
    return dec_outputs

def decoding_layer(dec_input, encoder_state,
                   target_sequence_length, max_target_sequence_length,
                   rnn_size,
                   num_layers, target_vocab_to_int, target_vocab_size,
                   batch_size, keep_prob, decoding_embedding_size):
    """
    Create decoding layer
    :param dec_input: Decoder input
    :param encoder_state: Encoder state
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_target_sequence_length: Maximum length of target sequences
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param target_vocab_size: Size of target vocabulary
    :param batch_size: The size of the batch
    :param keep_prob: Dropout keep probability
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # embedding target sequence
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)
    # construct decoder lstm cell
    dec_cell = tf.contrib.rnn.MultiRNNCell([
        tf.contrib.rnn.LSTMCell(rnn_size) \
        for _ in range(num_layers) ])
    # create output layer to map the outputs of the decoder to the elements of our vocabulary
    output_layer = layers_core.Dense(target_vocab_size,
                                    kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))
    # decoder train
    with tf.variable_scope("decoding") as decoding_scope:
        dec_outputs_train = decoding_layer_train(encoder_state, dec_cell, dec_embed_input,
                             target_sequence_length, max_target_sequence_length,
                             output_layer, keep_prob)
    # decoder inference
    start_of_sequence_id = target_vocab_to_int["<GO>"]
    end_of_sequence_id = target_vocab_to_int["<EOS>"]
    with tf.variable_scope("decoding", reuse=True) as decoding_scope:
        dec_outputs_infer = decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,
                             end_of_sequence_id, max_target_sequence_length,
                             target_vocab_size, output_layer, batch_size, keep_prob)
    # rerturn
    return dec_outputs_train, dec_outputs_infer

3.Attention介绍：

Attention的原理介绍，网上也是满天飞，这里简单介绍一下，熟悉Attention模型的同学直接跳过（英语好的同学依旧可以从tensorflow的官方教程找到：https://www.tensorflow.org/tutorials/seq2seq）

首先说一下为什么要有Attention模型

Attention模型的出现是上述的seq2seq模型存在缺陷，即无论之前的encoder的context有多长，包含多少信息量，最终都要被压缩成一个几百维的vector。这意味着context越大，decoder的输入之一的last state 会丢失越多的信息。对于机器翻译问题，意味着输入sentence长度增加后，最终decoder翻译的结果会显著变差。

Attention实质上是一种content-based addressing的机制，即从网络中某些状态集合中选取与给定状态较为相似的状态，进而做后续的信息抽取；说人话就是：首先根据Encoder和Decoder的特征计算权值，然后对Encoder的特征进行加权求和，作为Decoder的输入，其作用是将Encoder的特征以更好的方式呈献给Decoder，即：并不是所有context都对下一个状态的生成产生影响，Attention就是选择恰当的context用它生成下一个状态。

下面使用李宏毅老师经典的一页PPT来表达Attention算法的核心内容

上图已经很形象概括了Attention的处理过程，下面以公式来表达一下：

其中hj是encoder的对应输出，公式中的初始值的s0（对应图中z0） 是encoder的last state，上图的Match对应公式中的Score，这个Score计算方法有很多种，常用的是多层感知机（MLP），这就是常说的attention context，表达式如下：

实际上attention的计算过程比较简单，attention的原理部分就介绍完啦，下面看看tensorflow是如何实现attention的。tensorflow对attention在不同版本改动比较大，我们只看当前tf1.4.1版本中tensorflow是如何使用attention的

tensorflow把attention单独分出来，引入了attention_wrapper文件，定义的几种attention机制（BahdanauAttention、 LuongAttention、 BahdanauMonotonicAttention、 LuongMonotonicAttention），将attention机制封装到RNNCell上面的方法AttentionWrapper。其实很简单，就跟dropoutwrapper、outputwrapper一样，我们只需要在原本RNNCell的基础上在封装一层attention即可。

下面贴一下我常用的attention函数：

def decoder_attn(batch_size,encoder_hidden_size,attention_size,enc_seq_len,encoder_output,encoder_state,dec_cell):
    attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(num_units=encoder_hidden_size,
                                                               memory=encoder_output,
                                                               memory_sequence_length=enc_seq_len,
                                                               name="BahdanauAttention")
    attention_cell = tf.contrib.seq2seq.AttentionWrapper(dec_cell, attention_mechanism,attention_size,
                                                         name="attention_wrapper")
    init_state = attention_cell.zero_state(batch_size, tf.float32).clone(cell_state=encoder_state)
    return attention_cell,init_state

def decoding_layer(dec_input, encoder_state,encoder_output,source_sequence_length,
                   target_sequence_length, max_target_sequence_length,
                   encoder_rnn_hidden_unit,
                   decode_rnn_hidden_unit,
                   attention_size,
                   target_vocab_to_int, target_vocab_size,
                   batch_size, keep_prob, decoding_embedding_size):
    # embedding target sequence
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)
    # Todo : 这里一定不能用 MultiRNNCell,否则出错: 'Tensor' object is not iterable.
    dec_cell = tf.contrib.rnn.LSTMCell(decode_rnn_hidden_unit)
    # create output layer to map the outputs of the decoder to the elements of our vocabulary
    output_layer = layers_core.Dense(target_vocab_size,
                                    kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))
    attention_cell,init_state = decoder_attn(batch_size,encoder_rnn_hidden_unit, attention_size,
                                             source_sequence_length, encoder_output, encoder_state, dec_cell)
    # decoder train
    with tf.variable_scope("decoding") as decoding_scope:
        dec_outputs_train = decoding_layer_train(init_state, attention_cell, dec_embed_input,
                             target_sequence_length, max_target_sequence_length,
                             output_layer, keep_prob)
    # decoder inference
    start_of_sequence_id = target_vocab_to_int["<GO>"]
    end_of_sequence_id = target_vocab_to_int["<EOS>"]
    with tf.variable_scope("decoding", reuse=True) as decoding_scope:
        dec_outputs_infer = decoding_layer_infer(init_state, attention_cell, dec_embeddings,
                            start_of_sequence_id,
                            end_of_sequence_id, max_target_sequence_length,
                            output_layer, batch_size, keep_prob)
    # rerturn
    return dec_outputs_train, dec_outputs_infer

这里的decoding_layer函数要重新定义，但是修改非常小，可自行比较。

4.bilstm介绍

其实bilstm没有啥好介绍的，就是把单向的变成双向而已，可以获取完整信息，对于文本或者不要求实时的语音识别比较适用。

直接上自己常用的encoder的代码：

def encoding_layer(emb_encoder_inputs, rnn_size, encoder_num_layers, keep_prob,
                   source_sequence_length,encode_type = "lstm"):
    # time_major=False
    if encode_type == "lstm":
        # construcll rnn cell
        cell = tf.contrib.rnn.MultiRNNCell([
            tf.contrib.rnn.LSTMCell(rnn_size) \
            for _ in range(encoder_num_layers) ])
        # rnn forward
        cell = DropoutWrapper(cell, output_keep_prob=keep_prob)
        enc_output, enc_state = tf.nn.dynamic_rnn(cell, emb_encoder_inputs, sequence_length=source_sequence_length, dtype=tf.float32)
        return enc_output,enc_state if encoder_num_layers <=1 else enc_state[-1]
    elif encode_type == "bilstm":
        cell_fw = tf.nn.rnn_cell.LSTMCell(
            rnn_size,
            initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=123),
            state_is_tuple=True)
        cell_bw = tf.nn.rnn_cell.LSTMCell(
            rnn_size,
            initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=113),
            state_is_tuple=True)
        encoder_f_cell = DropoutWrapper(cell_fw, output_keep_prob=keep_prob)
        encoder_b_cell = DropoutWrapper(cell_bw, output_keep_prob=keep_prob)
        (encoder_fw_outputs, encoder_bw_outputs), (encoder_fw_final_state, encoder_bw_final_state) = \
            tf.nn.bidirectional_dynamic_rnn(
                encoder_f_cell, encoder_b_cell, inputs=emb_encoder_inputs, dtype=tf.float32, sequence_length=source_sequence_length)
        emb_encoder_outputs = tf.concat((encoder_fw_outputs, encoder_bw_outputs), 2)
        encoder_final_state_c = tf.concat((encoder_fw_final_state.c, encoder_bw_final_state.c), 1)
        encoder_final_state_h = tf.concat((encoder_fw_final_state.h, encoder_bw_final_state.h), 1)
        encoder_final_state = LSTMStateTuple(
            c=encoder_final_state_c,
            h=encoder_final_state_h
        )
        return emb_encoder_outputs, encoder_final_state

5.

完整代码见我的github

本文参考论文如下：

1.ATTENTION-BASED END-TO-END SPEECH RECOGNITION ON VOICE SEARCH

2.Listen, Attend and Spell

3.STATE-OF-THE-ART SPEECH RECOGNITION WITH SEQUENCE-TO-SEQUENCE MODELS

4.VERY DEEP CONVOLUTIONAL NETWORKS FOR END-TO-END SPEECH RECOGNITION