【大模型】解决报错ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.

运行Qwen1.5-7B-Chat大模型时报错：

2024-04-29 17:50:54,192 - INFO - intel_extension_for_pytorch auto imported
Traceback (most recent call last):
  File "/root/xx/llm/ipex/test-xpu.py", line 16, in <module>
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/xx/miniconda3/envs/llm-ipex/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 699, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.

环境

• ubuntu 22.04
• python 3.11

错误信息

2024-04-29 17:50:54,192 - INFO - intel_extension_for_pytorch auto imported
Traceback (most recent call last):
  File "/root/xx/llm/ipex/test-xpu.py", line 16, in <module>
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/xx/miniconda3/envs/llm-ipex/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 699, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.

重点错误：

ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.

解决方法

• 查看 transformers 版本

pip list|grep transformer

输出信息：

transformers                4.31.0

• 错误分析
  transformers 的版本太低导致的imported错误

ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.

• 解决方法
  升级 transformers 版本，如下：

pip install  transformers==4.40  -i https://mirrors.aliyun.com/pypi/simple/

查看transformers 的版本可以在 https://pypi.org/project/transformers/ 查看，推荐用最新的版本！！

测试

运行推理，正常执行！！！