深度学习Task4

最新推荐文章于 2023-02-25 11:50:23 发布

Mr_Right_only

最新推荐文章于 2023-02-25 11:50:23 发布

阅读量236

点赞数

本文链接：https://blog.csdn.net/Mr_Right_only/article/details/104401073

版权

本文介绍了神经机器翻译（NMT）的基本概念，并通过实例展示了数据预处理、分词、建立词典、载入数据集以及Sequence to Sequence模型在机器翻译中的应用。在数据预处理阶段，对原始文本进行了清洗和转换。接着，使用编码器-解码器模型搭建了Sequence to Sequence架构，并进行了训练和测试。

摘要由CSDN通过智能技术生成

机器翻译和数据集

机器翻译（MT）：将一段文本从一种语言自动翻译为另一种语言，用神经网络解决这个问题通常称为神经机器翻译（NMT）。主要特征：输出是单词序列而不是单个单词。输出序列的长度可能与源序列的长度不同。

import os
os.listdir(’/home/kesci/input/’)
[‘fraeng6506’, ‘d2l9528’, ‘d2l6239’]
import sys
sys.path.append(’/home/kesci/input/d2l9528/’)
import collections
import d2l
import zipfile
from d2l.data.base import Vocab
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils import data
from torch import optim

数据预处理

将数据集清洗、转化为神经网络的输入minbatch

with open(’/home/kesci/input/fraeng6506/fra.txt’, ‘r’) as f:
raw_text = f.read()
print(raw_text[0:1000])
Go. Va ! CC-BY 2.0 (France) Attribution: tatoeba.org #2877272 (CM) & #1158250 (Wittydev)
Hi. Salut ! CC-BY 2.0 (France) Attribution: tatoeba.org #538123 (CM) & #509819 (Aiji)
Hi. Salut. CC-BY 2.0 (France) Attribution: tatoeba.org #538123 (CM) & #4320462 (gillux)
Run! Cours ! CC-BY 2.0 (France) Attribution: tatoeba.org #906328 (papabear) & #906331 (sacredceltic)
Run! Courez ! CC-BY 2.0 (France) Attribution: tatoeba.org #906328 (papabear) & #906332 (sacredceltic)
Who? Qui ? CC-BY 2.0 (France) Attribution: tatoeba.org #2083030 (CK) & #4366796 (gillux)
Wow! Ça alors ! CC-BY 2.0 (France) Attribution: tatoeba.org #52027 (Zifre) & #374631 (zmoo)
Fire! Au feu ! CC-BY 2.0 (France) Attribution: tatoeba.org #1829639 (Spamster) & #4627939 (sacredceltic)
Help! À l’aide ! CC-BY 2.0 (France) Attribution: tatoeba.org #435084 (lukaszpp) & #128430 (sysko)
Jump. Saute. CC-BY 2.0 (France) Attribution: tatoeba.org #631038 (Shishir) & #2416938 (Phoenix)
Stop! Ça suffit ! CC-BY 2.0 (France) Attribution: tato
def preprocess_raw(text):
text = text.replace(’\u202f’, ’ ‘).replace(’\xa0’, ’ ‘)
out = ‘’
for i, char in enumerate(text.lower()):
if char in (’,’, ‘!’, ‘.’) and i > 0 and text[i-1] != ’ ':
out += ’ ’
out += char
return out

text = preprocess_raw(raw_text)
print(text[0:1000])
go . va ! cc-by 2 .0 (france) attribution: tatoeba .org #2877272 (cm) & #1158250 (wittydev)
hi . salut ! cc-by 2 .0 (france) attribution: tatoeba .org #538123 (cm) & #509819 (aiji)
hi . salut . cc-by 2 .0 (france) attribution: tatoeba .org #538123 (cm) & #4320462 (gillux)
run ! cours ! cc-by 2 .0 (france) attribution: tatoeba .org #906328 (papabear) & #906331 (sacredceltic)
run ! courez ! cc-by 2 .0 (france) attribution: tatoeba .org #906328 (papabear) & #906332 (sacredceltic)
who? qui ? cc-by 2 .0 (france) attribution: tatoeba .org #2083030 (ck) & #4366796 (gillux)
wow ! ça alors ! cc-by 2 .0 (france) attribution: tatoeba .org #52027 (zifre) & #374631 (zmoo)
fire ! au feu ! cc-by 2 .0 (france) attribution: tatoeba .org #18