Failed to build tokenizers ERROR: Could not build wheels for tokenizers, which is required to instal

1.在google colab 安装transformer时遇见错误tokenizers无法编译。
python版本为3.10; transformers版本为4.15;tokenizers需要<0.11, >=0.10.0 ; 直接安装tokenizers是不可行的。
错误展示:

Building wheels for collected packages: tokenizers
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

2.安装rust,命令如下:

!curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

遇见错误如下:

info: downloading installer
sh: 172: cannot open /dev/tty: No such device or address

免费版本是不存在终端的,所以将命令修改为:

!curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

安装成功:

info: installing component 'rustfmt'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.72.0 (5680fa18f 2023-08-23)


Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source "$HOME/.cargo/env"

运行命令 source “$HOME/.cargo/env”

source "$HOME/.cargo/env"
colab 里面是:source "/root/.cargo/env"

继续安装tranformer =4.15.0 仍然没有用遇见错误如上。
此时选择重启代码程序框。发现错误也一样,也无法解决。
此时只剩下最后一条路了,降低python版本为3.8.

改变colab中的python版本为3.8,参考链接为添加链接描述
1)安装 miniconda 改变 colab python 版本

%%bash
MINICONDA_INSTALLER_SCRIPT=Miniconda3-4.5.4-Linux-x86_64.sh
MINICONDA_PREFIX=/usr/local
wget https://repo.continuum.io/miniconda/$MINICONDA_INSTALLER_SCRIPT
chmod +x $MINICONDA_INSTALLER_SCRIPT
./$MINICONDA_INSTALLER_SCRIPT -b -f -p $MINICONDA_PREFIX

conda install --channel defaults conda python=3.8 --yes
conda update --channel defaults --all --yes

2)python 版本改变后的新安装包路径加入系统路径

import sys
sys.path.append("/usr/local/lib/python3.8/site-packages")
  1. 查看python版本
!python -V
python 3.8.17

继续安装环境,安装成功!

Successfully built horovod sacremoses
Installing collected packages: wcwidth, tokenizers, safetensors, pytz, typing-extensions, tqdm, threadpoolctl, tabulate, six, setuptools, regex, pyyaml, psutil, Pillow, numpy, networkx, joblib, ftfy, cloudpickle, click, torch, tifffile, scipy, sacremoses, PyWavelets, python-dateutil, opencv_python, imageio, horovod, filelock, torchvision, scikit_learn, scikit_image, pandas, huggingface_hub, transformers, timm
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.65.0
    Uninstalling tqdm-4.65.0:
      Successfully uninstalled tqdm-4.65.0
  Attempting uninstall: setuptools
    Found existing installation: setuptools 68.0.0
    Uninstalling setuptools-68.0.0:
      Successfully uninstalled setuptools-68.0.0
Successfully installed Pillow-9.2.0 PyWavelets-1.4.1 click-8.1.7 cloudpickle-2.2.1 filelock-3.12.3 ftfy-6.1.1 horovod-0.28.1 huggingface_hub-0.13.4 imageio-2.31.3 joblib-1.3.2 networkx-3.1 numpy-1.21.6 opencv_python-4.6.0.66 pandas-1.3.5 psutil-5.9.5 python-dateutil-2.8.2 pytz-2023.3.post1 pyyaml-6.0.1 regex-2022.10.31 sacremoses-0.0.53 safetensors-0.3.3 scikit_image-0.19.3 scikit_learn-1.0.2 scipy-1.10.1 setuptools-63.4.1 six-1.16.0 tabulate-0.9.0 threadpoolctl-3.2.0 tifffile-2023.7.10 timm-0.8.15.dev0 tokenizers-0.10.3 torch-1.12.1 torchvision-0.13.1 tqdm-4.64.1 transformers-4.15.0 typing-extensions-4.7.1 wcwidth-0.2.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: The following packages were previously imported in this runtime:
  [PIL,_distutils_hack,dateutil,numpy,pkg_resources,psutil,setuptools,six,wcwidth]
You must restart the runtime in order to use newly installed versions.

此外,如果你安装transformer 4.15之类的碰到安装不了,方便的情况下降低python为3.8版本,这是最好的选择,网上有人试了3.10,和3.6都会报编译错误;其次的话安装4.16的版本。如果要是遇见冲突再处理。

  • 4
    点赞
  • 8
    收藏
    觉得还不错? 一键收藏
  • 2
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值