【文献阅读】ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

Abstract

最近的中文预训练模型忽略了中文特有的两个重要方面：字形和拼音，它们携带重要的句法和语义信息，用于语言理解

我们提出了 ChineseBERT，它将汉字的字形和拼音信息结合到语言模型预训练中

字形：汉字的不同字体
字音：汉语拼音（处理多音字）

Introduction

由于预训练模型最初是为英语设计的,因此在当前的大规模预训练中缺少两个特定于中文的重要方面：基于字形的信息和基于拼音的信息

对于前者，使中文与英语、德语等语言区分开来的一个关键方面是汉语是一种表意语言。 字符的语标对语义信息进行编码。 例如，“液（液）”、“河（河）”、“湖（湖）”都有部首“氵（水）”，表示它们在语义上都与水有关。 直观地说，汉字字形背后丰富的语义应该增强中文 NLP 模型的表达能力。

关于字形

这个想法激发了学习和将汉字字形信息整合到神经模型中的各种工作

Y aming Sun, Lei Lin, Nan Y ang, Zhenzhou Ji, and Xiaolong Wang. 2014. Radical-enhanced chinese character embedding. In International Conference on Neural Information Processing, pages 279–286.Springer.

Xinlei Shi, Junjie Zhai, Xudong Y ang, Zehua Xie, and Chao Liu. 2015. Radical embedding: Delving deeper to Chinese radicals. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (V olume 2: Short Papers), pages 594–598, Beijing, China. Association for Computational Linguistics.

Frederick Liu, Han Lu, Chieh Lo, and Graham Neubig. 2017. Learning character-level compositionality with visual features. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, V ancouver , Canada, July 30 - August 4, V olume 1: Long Papers, pages 2059–2068.

Falcon Z Dai and Zheng Cai. 2017. Glyph-aware embedding of chinese characters. In Proceedings of the First Workshop on Subword and Character Level Models in NLP , Copenhagen, Denmark, September 7, 2017, pages 64–69.

Tzu-Ray Su and Hung-Yi Lee. 2017. Learning chinese word representations from glyphs of characters. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 911, 2017, pages 264–273.

Y uxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. In Advances in Neural Information Processing Systems, volume 32, pages 2746–2757. Curran Associates, Inc.

关于字音

多音字问题