LlamaTokenizer.from_pretrained 报错 TypeError: not a string

---

1. 报错

调用 LlamaTokenizer.from_pretrained 时报错：

from transformers import LlamaTokenizer
tokenizer = LlamaTokenizer.from_pretrained(model_dir)

报错如下：

File ~\venvs\envllama\lib\site-packages\transformers\tokenization_utils_base.py:2089 in from_pretrained
return cls._from_pretrained(

File ~\venvs\envllama\lib\site-packages\transformers\tokenization_utils_base.py:2311 in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)

File ~\venvs\envllama\lib\site-packages\transformers\models\llama\tokenization_llama.py:169 in init
self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))

File ~\venvs\envllama\lib\site-packages\transformers\models\llama\tokenization_llama.py:196 in get_spm_processor
tokenizer.Load(self.vocab_file)

File ~\venvs\envllama\lib\site-packages\sentencepiece_init_.py:961 in Load
return self.LoadFromFile(model_file)

File ~\venvs\envllama\lib\site-packages\sentencepiece_init_.py:316 in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)

TypeError: not a string

2. 解决办法

2.1 检查路径

检查模型路径是否有问题

2.2 切换 AutoTokenizer

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_dir)

3. 参考

https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/discussions/37