Facebook 开源 M2M-100，不依赖英语互译百种语言

Facebook 10 月 19 日开源了 M2M-100 模型的源代码，并称该算法是第一个能够在不依赖英语数据的情况下，在 100 种语言之间进行翻译的算法。也就是这说，无需通过英文做中介，该模型就可对这 100 种语言直接互译。

M2M-100 基于 Facebook 的多语言模型 XLM-R，使用 ccAligned，ccMatrix 和 LASER 等开源数据挖掘工具收集了包含 100 多种语言的超过 75 亿个句子，根据语言分类、地理和文化相似性等参数分为 14 种不同的语言组。在 14 个语言组中，每个语言会被标识一到三种“过渡语言”，然后将其作为翻译成不同语言组的基础。

生成数据

在应用 SPM 进行数据预处理之前，需要标记所有数据集。如果使用这些评估数据集，请引用其相关论文。

# WMT - use sacrebleu, example here:
sacrebleu -t wmt14 -l fr-en --echo src > wmt.test.fr-en.fr
sacrebleu -t wmt14 -l fr-en --echo ref > wmt.test.fr-en.en

# WAT
wget http://lotus.kuee.kyoto-u.ac.jp/WAT/my-en-data/wat2019.my-en.zip
unzip wat2019.my-en.zip

# FLORES
# download from: https://github.com/facebookresearch/flores

# TED - need to detokenize with Moses!
# from: https://github.com/neulab/word-embeddings-for-nmt
wget http://phontron.com/data/ted_talks.tar.gz

# Autshumato
# request to download: https://repo.sadilar.org/handle/20.500.12185/397

# Tatoeba Challenge
# available here: https://github.com/Helsinki-NLP/Tatoeba-Challenge

训练数据

为了产生训练数据，结合使用 CCMatrix 和 CCAligned。查看此处的说明下载原始数据。

数据预处理

下载原始数据后，需要对数据进行处理，然后应用 SPM，然后进行二值化。

# preprocess data

# remove sentences with more than 50% punctuation
python /path/to/fairseq/examples/m2m_100/process_data/remove_too_much_punc.py 

# deduplicate training data
paste /path/to/datadir/train.$src /path/to/datadir/train.$tgt | awk '!x[$0]++' > /path/to/datadir/train.dedup
echo "keeping $(wc -l /path/to/datadir/train.dedup) bitext out of $(wc -l /path/to/datadir/train.$src)"
cut -f1 /path/to/datadir/train.dedup > /path/to/datadir/train.$src
cut -f2 /path/to/datadir/train.dedup > /path/to/datadir/train.$tgt

# remove all instances of evaluation data from the training data
python /path/to/fairseq/examples/m2m_100/process_data/dedup_data.py 

# frequency cleaning
wget https://dl.fbaipublicfiles.com/m2m_100/histograms.tar.gz 
tar -xvzf histograms.tar.gz
python /path/to/fairseq/examples/m2m_100/process_data/clean_histogram.py --src $src --tgt $tgt --src-file /path/to/source/file --tgt-file /path/to/output/file --src-output-file source_output.$src --tgt-output-file target_output.$tgt --histograms /path/to/histograms

# apply SPM
wget https://dl.fbaipublicfiles.com/m2m_100/spm.128k.model
python /path/to/fairseq/scripts/spm_encode.py \
    --model spm.128k.model \
    --output_format=piece \
    --inputs=/path/to/input/file/here \
    --outputs=/path/to/output/file/here

# length ratio cleaning
perl mosesdecoder/scripts/training/clean-corpus-n.perl --ratio 3 /path/to/training/data/train.spm.$src-$tgt $src $tgt /path/to/output/directory/train.spm.$src-$tgt 1 250

# binarize data
wget https://dl.fbaipublicfiles.com/m2m_100/data_dict.128k.txt
fairseq-preprocess \
    --source-lang $src --target-lang $tgt \
    --testpref spm.$src.$tgt \
    --thresholdsrc 0 --thresholdtgt 0 \
    --destdir data_bin \
    --srcdict data_dict.128k.txt --tgtdict data_dict.128k.txt

训练脚本

为了重现模型训练，使用 fairseq-py 的多语言翻译任务进行训练。

生成

从该模型中生成，需遵循以下生成部分中的命令。

@article{fan2020beyond,
  title={Beyond English-Centric Multilingual Machine Translation},
  author={Fan, Angela and Bhosale, Shruti and Schwenk, Holger and Ma, Zhiyi and El-Kishky, Ahmed and Goyal, Siddharth and Baines, Mandeep and Celebi, Onur and Wenzek, Guillaume and Chaudhary, Vishrav and Goyal, Naman and Birch, Tom and Liptchinsky, Vitaliy and Edunov, Sergey and Grave, Edouard and Auli, Michael and Joulin, Armand},
  journal={arXiv preprint},
  year={2020}
}

@article{schwenk2019ccmatrix,
  title={Ccmatrix: Mining billions of high-quality parallel sentences on the web},
  author={Schwenk, Holger and Wenzek, Guillaume and Edunov, Sergey and Grave, Edouard and Joulin, Armand},
  journal={arXiv preprint arXiv:1911.04944},
  year={2019}
}

@article{el2019massive,
  title={A Massive Collection of Cross-Lingual Web-Document Pairs},
  author={El-Kishky, Ahmed and Chaudhary, Vishrav and Guzman, Francisco and Koehn, Philipp},
  journal={arXiv preprint arXiv:1911.06154},
  year={2019}
}

参考链接：

https://venturebeat.com/2020/10/19/facebooks-open-source-m2m-100-model-can-translate-between-100-different-languages/

https://siliconangle.com/2020/10/19/facebook-ai-open-sources-m2m-100-multilingual-model-improve-translation-accuracy/