神经网络机器翻译Neural Machine Translation(1): Encoder-Decoder Architecture

本文深入探讨了神经网络机器翻译(NMT)中的Encoder-Decoder架构,引用了Kyunghyun Cho等人的研究以及Ilya Sutskever等人的工作,这些工作为统计机器翻译提供了新的视角,利用RNN来学习短语表示,并通过序列到序列学习进行建模。
端到端的神经网络机器翻译(End-to-End Neural Machine Translation)是近几年兴起的一种全新的机器翻译方法。本文首先将简要介绍传统的统计机器翻译方法以及神经网络在机器翻译中的应用,然后介绍NMT中基本的“编码-解码”框架(Encoder-Decoder)。

转载请注明出处:http://blog.csdn.net/u011414416/article/details/51048994

本文中,详细介绍的工作有:
Kyunghyun Cho, Bart van Merrenboer, Caglar Gulcehre Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine. In Proceedings of the 2014 Conference on EMNLP, 1724-1734.

Ilya Sutskever, Oriol Vinyals, Quoc V. Le. 2014. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 4:3104-3112.

1
2

The code you provided defines a named tuple `Hypothesis` with two fields, `value` and `score`. This is a convenient way to store and manipulate hypotheses in the context of sequence-to-sequence models. The `NMT` class is a PyTorch module that implements a simple neural machine translation model. It consists of a bidirectional LSTM encoder, a unidirectional LSTM decoder, and a global attention mechanism based on Luong et al. (2015). Here's a breakdown of the code: ```python from collections import namedtuple import torch import torch.nn as nn import torch.nn.functional as F Hypothesis = namedtuple('Hypothesis', ['value', 'score']) class NMT(nn.Module): def __init__(self, src_vocab_size, tgt_vocab_size, emb_size, hidden_size): super(NMT, self).__init__() self.src_embed = nn.Embedding(src_vocab_size, emb_size) self.tgt_embed = nn.Embedding(tgt_vocab_size, emb_size) self.encoder = nn.LSTM(emb_size, hidden_size, bidirectional=True) self.decoder = nn.LSTMCell(emb_size + hidden_size, hidden_size) self.attention = nn.Linear(hidden_size * 2, hidden_size) self.out = nn.Linear(hidden_size, tgt_vocab_size) self.hidden_size = hidden_size def forward(self, src, tgt): batch_size = src.size(0) src_len = src.size(1) tgt_len = tgt.size(1) # Encode the source sentence src_embedded = self.src_embed(src) encoder_outputs, (last_hidden, last_cell) = self.encoder(src_embedded) # Initialize the decoder states decoder_hidden = last_hidden.view(batch_size, self.hidden_size) decoder_cell = last_cell.view(batch_size, self.hidden_size) # Initialize the attention context vector context = torch.zeros(batch_size, self.hidden_size, device=src.device) # Initialize the output scores outputs = torch.zeros(batch_size, tgt_len, self.hidden_size, device=src.device) # Decode the target sentence for t in range(tgt_len): tgt_embedded = self.tgt_embed(tgt[:, t]) decoder_input = torch.cat([tgt_embedded, context], dim=1) decoder_hidden, decoder_cell = self.decoder(decoder_input, (decoder_hidden, decoder_cell)) attention_scores = self.attention(encoder_outputs) attention_weights = F.softmax(torch.bmm(attention_scores, decoder_hidden.unsqueeze(2)).squeeze(2), dim=1) context = torch.bmm(attention_weights.unsqueeze(1), encoder_outputs).squeeze(1) output = self.out(decoder_hidden) outputs[:, t] = output return outputs ``` The `__init__` method initializes the model parameters and layers. It takes four arguments: - `src_vocab_size`: the size of the source vocabulary - `tgt_vocab_size`: the size of the target vocabulary - `emb_size`: the size of the word embeddings - `hidden_size`: the size of the encoder and decoder hidden states The model has four main components: - `src_embed`: an embedding layer for the source sentence - `tgt_embed`: an embedding layer for the target sentence - `encoder`: a bidirectional LSTM encoder that encodes the source sentence - `decoder`: a unidirectional LSTM decoder that generates the target sentence The attention mechanism is implemented in the `forward` method. It takes two arguments: - `src`: the source sentence tensor of shape `(batch_size, src_len)` - `tgt`: the target sentence tensor of shape `(batch_size, tgt_len)` The method first encodes the source sentence using the bidirectional LSTM encoder. The encoder outputs and final hidden and cell states are stored in `encoder_outputs`, `last_hidden`, and `last_cell`, respectively. The decoder is initialized with the final hidden and cell states of the encoder. At each time step, the decoder takes as input the embedded target word and the context vector, which is a weighted sum of the encoder outputs based on the attention scores. The decoder output and hidden and cell states are updated using the LSTMCell module. The attention scores are calculated by applying a linear transform to the concatenated decoder hidden state and encoder outputs, followed by a softmax activation. The attention weights are used to compute the context vector as a weighted sum of the encoder outputs. Finally, the decoder hidden state is passed through a linear layer to produce the output scores for each target word in the sequence. The output scores are stored in the `outputs` tensor and returned by the method.
评论 3
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值