Pre-Training with Whole Word Masking for Chinese BERT

Abstract

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks. Recently, an upgraded version of BERT has been released with Whole Word Masking (WWM), which mitigate the drawbacks of masking partial WordPiece tokens in pre-training BERT.
BERT在各种NLP任务中显示了惊人的改进。最近，使用全词掩盖 (WWM)的BERT的升级版已经发布，这减轻了在训练前屏蔽部分WordPiece标志的缺点。

In this technical report, we adapt whole word masking in Chinese text, that masking the whole word instead of masking Chinese characters, which could bring another challenge in Masked Language Model (MLM) pre-training task.
在这个技术报告中，我们采用了全字掩码的方法，将汉字掩码改为全字掩码。

The proposed models are verified on various NLP tasks, across sentence-level to document-level, including machine reading
comprehension (CMRC 2018, DRCD, CJRC), natural language inference (XNLI), sentiment classification (ChnSentiCorp), sentence pair
matching (LCQMC, BQ Corpus), and document classification (THUCNews). Experimental results on these datasets show that the whole word masking could bring another significant gain. Moreover, we also examine the effectiveness of the Chinese pre-trained models: BERT, ERNIE, BERT-wwm, BERT-wwm-ext, RoBERTa-wwm-ext, and RoBERTa-wwm-ext-large.
本文提出的模型在机器阅读理解(CMRC 2018, DRCD, CJRC)、自然语言推理(XNLI)、情感分类(ChnSentiCorp)、句子对匹配(LCQMC, BQ Corpus)和文档分类(THUCNews)等多个NLP任务上得到验证。在这些数据集上的实验结果表明，全词掩盖可以带来另一个显著的增益。此外，我们还检验了中文预训练模型：BERT、ERNIE、BERTwwm、BERT-wwm-ext、RoBERTa-wwm-ext和RoBERTa-wwm-ext-large的有效性。