whisper-large-v3 双卡 v100 64G 训练 问题总结

运行环境

pip install matplotlib

pip3.8 install jupyterlab

安装python; 在jupyterlab上面运行whisper训练脚本的时候报错 ModuleNotFoundError: No module named '_lzma';

卸载python3.8

重新安装

tar -xf Python-3.8.0.tgz

cd python3.8

./configure --enable-optimizations --with-lzma

make -j $(nproc)

sudo make altinstall

python3.8 -c "import _lzma"

更新pip

sudo pip3.8 install --upgrade pip

设置镜像取消镜像

​
pip config set global.index-url Simple Index

pip config unset global.index-url

​

安装pytorch

pip3.8 install torch torchvision

查看GPU 是否可用

nvidia-smi

bale@sxl1-0001:~$ python3.8

Python 3.8.1 (default, Dec 13 2023, 10:43:13)

[GCC 7.5.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import torch

>>> torch.cuda.is_available()

结果是否为true

当为false的时候, 尝试运行命令

sudo apt-get install nvidia-modprobe

ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`

安装jupyter 启动 停止进程。

pip3.8 install jupyterlab

nohup jupyter-lab --ip=0.0.0.0 --port=8888 > jupyter.log 2>&1 &

pkill -f "jupyter-lab"

可以写一个.sh脚本

Jupyter 无法调用huggingface接口。

启动之前执行

set HTTP_PROXY=http://127.0.0.1:7890

set HTTPS_PROXY=http://127.0.0.1:7890

切换数据集需要删除缓存

rm -rf ~/.cache/huggingface/datasets/

启动Tensorboard

nohup tensorboard --logdir=/home/bale/Tensorboard/loggingdir --port=8887 --bind_all > tensorboard.log 2>&1 &

报错 AttributeError: module 'distutils' has no attribute 'version'

pip uninstall setuptools
pip install setuptools==41.2.0

相关参考文档

https://huggingface.co/blog/zh/fine-tune-whisper

[Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files · Issue #5659 · huggingface/datasets · GitHub

Ability to fine-tune whisper large on a GPU with 24 gb of ram · Issue #20348 · huggingface/transformers · GitHubhttps://github.com/huggingface/transformers/issues/20348

Gradio Audio Docs