AutoTokenizer的用法

最新推荐文章于 2024-07-29 14:37:54 发布

hehui0921

最新推荐文章于 2024-07-29 14:37:54 发布

阅读量208

点赞数 2

分类专栏： huggingface 文章标签： python 前端开发语言

本文链接：https://blog.csdn.net/oHeHui1/article/details/136132895

版权

huggingface 专栏收录该内容

33 篇文章 0 订阅

订阅专栏

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
encoded_input = tokenizer("Then pass your text to the tokenizer")
print(encoded_input)
input_ids = encoded_input["input_ids"]
print(encoded_input["input_ids"])
print(encoded_input["token_type_ids"])
print(encoded_input["attention_mask"])
print(tokenizer.decode(encoded_input["input_ids"],skip_special_tokens=True))

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
encoded_input = tokenizer(["pass your text to the tokenizer","tell me","The beautiful woman is shopping on the street"],padding= True, return_tensors="pt")
print(encoded_input)