ValueError: Couldn‘t instantiate the backend tokenizer from one of: (1) a `tokenizers` library seria

E.K.江湖念书人

于 2023-07-18 11:24:57 发布

阅读量2.4k

点赞数 4

文章标签： python 开发语言

本文链接：https://blog.csdn.net/weixin_53704902/article/details/131783428

版权

文章讲述了在使用transformers库时遇到的ValueError，原因是缺少sentencepiece组件。解决方案是通过运行!pipinstalltransformers[sentencepiece]来安装必要的依赖，从而能实例化tokenizer并转换慢速tokenizer。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

ValueError: Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

使用以下代码解决该报错问题

!pip install transformers[sentencepiece]