Unsupervised Quality Estimation for Neural Machine Translation

1）Set up 定义变量

SRC_LANG=en
TGT_LANG=de
INPUT=/home/huyu1/tools/mlqe/data/en-de/train
OUTPUT_DIR=/home/huyu1/datas/en-de/output-en-de-train
TMP=/home/huyu1/datas/en-de/tmp-en-de-train
BPE=/home/huyu1/tools/en-de/bpecodes
MOSES_DECODER=/home/huyu1/tools/mosesdecoder
BPE_ROOT=/home/huyu1/tools/subword-nmt/subword_nmt
MODEL_DIR=/home/huyu1/tools/en-de
GPU=0
DROPOUT_N=10
SCRIPTS=/home/huyu1/fairseq/fairseq/examples/unsupervised_quality_estimation
METEOR=/home/huyu1/tools/meteor-1.5/meteor-1.5.jar

• BPE（直接定义在了命令行中）: path to BPE model（学习bpe模型）

2）Translate the data using standard decoding（模型已经预训练好/标准解码，没有dropout）

Preprocess the input data:

for LANG in $SRC_LANG $TGT_LANG; do
  perl $MOSES_DECODER/scripts/tokenizer/tokenizer.perl -threads 80 -a -l $LANG < $INPUT.$LANG > $TMP/preprocessed.tok.$LANG
  python $BPE_ROOT/apply_bpe.py -c ${BPE} < $TMP/preprocessed.tok.$LANG > $TMP/preprocessed.tok.bpe.$LANG
done

单个运行

  perl $MOSES_DECODER/scripts/tokenizer/tokenizer.perl -threads 80 -a -l en < $INPUT.en > $TMP/preprocessed.tok.en
  python $BPE_ROOT/learn_bpe.py -s 10000 <$TMP/preprocessed.tok.en >$TMP/codes.en
  python $BPE_ROOT/apply_bpe.py -c $TMP/codes.en < $TMP/preprocessed.tok.en > $TMP/preprocessed.tok.bpe.en

• 分词器tokenizer，将一个完整的句子拆分成Token
• bpe将原有的单词拆解成了更小单元的高频词进行翻译，有效的解决了未登录词的问题：(处理后的语料中@-@ 是对原本语料中-的替代)
  首先学习bpe模型
  从$ TMP/preprocessed.tok.en两个文件中学习BPE codes并输出到codes文件（$ TMP/codes.$LANG）中，10000为BPE codes的个数（取排名前10000的子词）。
  再将生成的codes文件应用到文件中进行分词。

Binarize the data for faster translation:

fairseq-preprocess --srcdict $MODEL_DIR/dict.$SRC_LANG.txt --tgtdict $MODEL_DIR/dict.$TGT_LANG.txt --source-lang ${SRC_LANG} --target-lang ${TGT_LANG} --testpref $TMP/preprocessed.tok.bpe --destdir $TMP/bin --workers 4

–fairseq-preprocess:数据预处理：构建词汇表和训练数据的二进制化
–srcdict：重用给定的源字典
–tgtdict：重用给定目标字典
–testpref：逗号分隔，测试文件前缀(从训练集合中缺失的单词被替换为 < unk> )
–destdir：目标目录 默认值：“data-bin”
–workers:number of parallel workers Default: 1

Translate

CUDA_VISIBLE_DEVICES=$GPU fairseq-generate $TMP/bin --path ${MODEL_DIR}/${SRC_LANG}-${TGT_LANG}.pt --beam 5 --source-lang $SRC_LANG --target-lang $TGT_LANG --no-progress-bar --unkpen 5 > $TMP/fairseq.out
grep ^H $TMP/fairseq.out | cut -d- -f2- | sort -n | cut -f3- > $TMP/mt.out

fairseq-generate：用经过训练的模型转换预处理后的数据
–path：模型路径
–beam：beam size Default: 5
–no-progress-bar：禁用进度条 Default: False
–unkpen：未知单词惩罚：<0产生更多的unks，>0生成更少的 Default：0

Post-process

sed -r 's/(@@ )| (@@ ?$)//g' < $TMP/mt.out | perl $MOSES_DECODER/scripts/tokenizer/detokenizer.perl -l $TGT_LANG > $OUTPUT_DIR/mt.out

翻译的时候使用的数据均是经过bpe编码的，因此翻译得到的result也是bpe后的，因此需要对result进行还原，还原回bpe之前的（即norm、tok、clean、tc处理后的）状态

3）Produce uncertainty estimates

Scoring

Make temporary files to store the translations repeated N times.（创造临时文件存储重复N次的翻译。）

python ${SCRIPTS}/repeat_lines.py -i $TMP/preprocessed.tok.bpe.$SRC_LANG -n $DROPOUT_N -o $TMP/repeated.$SRC_LANG
python ${SCRIPTS}/repeat_lines.py -i $TMP/mt.out -n $DROPOUT_N -o $TMP/repeated.$TGT_LANG
fairseq-preprocess --srcdict ${MODEL_DIR}/dict.${SRC_LANG}.txt --tgtdict $MODEL_DIR/dict.$TGT_LANG.txt --source-lang ${SRC_LANG} --target-lang ${TGT_LANG} --testpref ${TMP}/repeated --destdir ${TMP}/bin-repeated

将预处理好的源数据和译文重复DROPOUT_N遍
然后重复之前的数据二值化操作

Produce model scores for the generated translations using –retain-dropout option to apply dropout at inference time（使用dropout）:

CUDA_VISIBLE_DEVICES=${GPU} fairseq-generate ${TMP}/bin-repeated --path ${MODEL_DIR}/${SRC_LANG}-${TGT_LANG}.pt --beam 5 --source-lang $SRC_LANG --target-lang $TGT_LANG --no-progress-bar --unkpen 5 --score-reference --retain-dropout --retain-dropout-modules '["TransformerModel","TransformerEncoder","TransformerDecoder","TransformerEncoderLayer","TransformerDecoderLayer"]'  --seed 46 > $TMP/dropout.scoring.out
grep ^H $TMP/dropout.scoring.out | cut -d- -f2- | sort -n | cut -f2 > $TMP/dropout.scores

–score-reference：just score the reference translation，Default: False
–retain-dropout：Use dropout at inference time Default: False
–retain-dropout-modules： if set, only retain dropout for the specified modules; if not set, then dropout will be retained for all modules

python $SCRIPTS/aggregate_scores.py -i $TMP/dropout.scores -o $OUTPUT_DIR/dropout.scores.mean -n $DROPOUT_N

Generation

Produce multiple translation hypotheses for the same source using –retain-dropout option:

CUDA_VISIBLE_DEVICES=${GPU} fairseq-generate ${TMP}/bin-repeated --path ${MODEL_DIR}/${SRC_LANG}-${TGT_LANG}.pt --beam 5 --source-lang $SRC_LANG --target-lang $TGT_LANG --no-progress-bar --retain-dropout --unkpen 5 --retain-dropout-modules '["TransformerModel","TransformerEncoder","TransformerDecoder","TransformerEncoderLayer","TransformerDecoderLayer"]' --seed 46 > $TMP/dropout.generation.out
grep ^H $TMP/dropout.generation.out | cut -d- -f2- | sort -n | cut -f3- > $TMP/dropout.hypotheses_
sed -r 's/(@@ )| (@@ ?$)//g' < $TMP/dropout.hypotheses_ | perl $MOSES_DECODER/scripts/tokenizer/detokenizer.perl -l $TGT_LANG > $TMP/dropout.hypotheses

python ${SCRIPTS}/meteor.py -i $TMP/dropout.hypotheses -m $METEOR -n $DROPOUT_N -o $OUTPUT_DIR/dropout.gen.sim.meteor