1.加载bert模型及分词
from transformers import AutoModelForMaskedLM, AutoTokenizer
model = "bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model, use_fast=True)
model = AutoModelForMaskedLM.from_pretrained(model)
2.分词演示
- 这里是对COVID hospitalization分词
print(tokenizer.tokenize('COVID'))
print(tokenizer.tokenize('hospitalization'))
['CO', '##VI', '##D']
['hospital', '##ization']
- 如果让着两个词都能保持完整,而不被拆分,进行如下操作
# Let's increase the vocabulary of Bert model and tokenizer
new_tokens = ['COVID', 'hospitalization']
num_added_toks = tokenizer.add_tokens(new_tokens)
# Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.
model.resize_token_embeddings(len(tokenizer))
print(tokenizer.tokenize('COVID'))
print(tokenizer.tokenize('hospitalization'))
# with each new word added, a new vector of embeddings
# with random values was added as well thanks to the
# model.resize_token_embeddings(len(tokenizer)) method.
tokenizer.savepretrained("modle_dir")
3.自定义bert词表
-
bert有自己的vocab配置可以人为修改以适应下游任务,参考看这篇https://blog.csdn.net/kyle1314608/article/details/10661204
-
bert词表相关配置文件介绍
https://www.jianshu.com/p/a6170d3408f5
参考链接
https://zhuanlan.zhihu.com/p/391814780