只需一个API,直接调用BERT, GPT, GPT-2, Transfo-XL, XLNet, XLM等6大框架,包含了27个预训练模型。
简单易用,功能强大。目前已经包含了PyTorch实现、预训练模型权重、运行脚本和以下模型的转换工具:
- BERT,论文:“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”,论文作者:Jacob Devlin, Ming-Wei Chang, Kenton Lee,Kristina Toutanova
- OpenAI 的GPT,论文:“Improving Language Understanding by Generative Pre-Training”,论文作者:Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever
- OpenAI的GPT-2,论文:“Language Models are Unsupervised Multitask Learners”,论文作者:Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei,Ilya Sutskever
- 谷歌和CMU的Transformer-XL,论文:“Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”,论文作者:Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
- 谷歌和CMU的XLNet,论文:“XLNet: Generalized Autoregressive Pretraining for Language Understanding”,论文作者:Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le
- Facebook的XLM,论文:“Cross-lingual Language Model Pretraining”,论文作者:Guillaume Lample,Alexis Conneau
这些实现都在几个数据集(参见示例脚本)上进行了测试,性能与原始实现相当,例如BERT中文全词覆盖在SQuAD数据集上的F1分数为93;OpenAI GPT 在RocStories上的F1分数为88;Transformer-XL在WikiText 103上的困惑度为18.3;XLNet在STS-B的皮尔逊相关系数为0.916。
该项目是在Python 2.7和3.5+上测试(例子只在python 3.5+上测试)和PyTorch 0.4.1到1.1.0测试。