Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects分析

报错信息

Building wheels for collected packages: tokenizers
error: subprocess-exited-with-error

× Building wheel for tokenizers (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Building wheel for tokenizers (pyproject.toml) … error
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

分析

我的基本环境

• python -V 我是 python 3.10
• transformers==4.6.0

报错原因

huggingface transformer 这个包，依赖 tokenizers包， tokenizers 使用了 Rust 多线程加速处理数据集。
这个是Rust的Bug。

解决办法

以下几种解决方案任选一种即可：

第一种：降低python版本， 比如使用 python3.8

第二种：修改源码再编译，在源码上修改依赖的 tokenizers的版本

然后使用 python setup.py install装包

第三种：使用最新版的 transformers包，修改咱们自己项目的代码。修改咱们项目内报错位置的代码，使其支持最新版的 transformers。

参考资料

1. https://github.com/huggingface/transformers/issues/2831
2. https://stackoverflow.com/questions/69595700/could-not-build-wheels-for-tokenizers-which-is-required-to-install-pyproject-to