Ubuntu10.10 32位系统下moses统计机器翻译系统使用命令记录

准备语料

/disknew/zw/moses/work/corpus translated.ench 1035224句已分词

数据预处理

过滤掉长句子

/tools/moses-scripts/scripts-20131121-1744/training/clean-corpus-n.perl/disknew/zw/moses/work/corpus/translated en chdisknew/zw/moses/work/corpus/translated.clean 1 100

单词小写处理

tools/scripts/lowercase.perl <disknew/zw/moses/work/corpus/translated.clean.en >disknew/zw/moses/work/corpus/translated.lowercased.en

tools/scripts/lowercase.perl <disknew/zw/moses/work/corpus/translated.clean.ch >disknew/zw/moses/work/corpus/translated.lowercased.ch

创建语言模型

准备训练数据

/tools/scripts/lowercase.perl <disknew/zw/moses/work/corpus/translated.ch >disknew/zw/moses/work/lm/translated.lowercased.ch

创建语言模型

/tools/Srilm/bin/i686-m64/ngram-count-order 5 -interpolate -wbdiscount -unk -textdisknew/zw/moses/work/lm/translated.lowercased.ch -lmdisknew/zw/moses/work/lm/translated.lm

训练短语模型

nohup nice/tools/moses-scripts/scripts-20131121-1744/training/train-model.perl-scripts-root-dir /tools/moses-scripts/scripts-20131121-1744 -root-dir/disknew/zw/moses/work -corpus/disknew/zw/moses/work/corpus/translated.lowercased -f en -e ch -alignmentgrow-diag-final-and -reordering msd-bidirectional-fe -lm 0:5:/disknew/zw/moses/work/lm/translated.lm>& disknew/zw/moses/work/training.out &

数据预处理

开发集/disknew/zw/moses/work/tuning/dev.ch dev.en

最小错误率训练

nohup nice/tools/moses-scripts/scripts-20131121-1744/training/mert-moses.pldisknew/zw/moses/work/tuning/dev.en /disknew/zw/moses/work/tuning/dev.en/tools/moses/moses-cmd/src/moses /disknew/zw/moses/work/model/moses.ini--working-dir /disknew/zw/moses/work/tuning/mert --rootdir/tools/moses-scripts/scripts-20131121-1744 --mertdir /tools/moses/mert--decoder-flags "-v 0" >&/disknew/zw/moses/work/tuning/mert.out &

过滤短语表

/tools/moses-scripts/scripts-20131121-1744/training/filter-model-given-input.pldisknew/zw/moses/work/evaluation/filterdisknew/zw/moses/work/tuning/mert/moses.inidisknew/zw/moses/work/evaluation/test.en

使用调参后的解码器进行解码

nohup nice /tools/moses/moses-cmd/src/moses-config disknew/zw/moses/work/evaluation/filter/moses.ini -input-filedisknew/zw/moses/work/evaluation/test.en 1>disknew/zw/moses/work/evaluation/filter/output 2>disknew/zw/moses/work/evaluation/filter/decode.out &

获取BLEU值

/disknew/sh2013/tools/BLEU./evaluation/test.en ./evaluation/test.cn ./evaluation/filter/output bleu-out