在ESPnet使用Makefile安装PyTorch和相关依赖的详细教程
介绍
在数据科学和机器学习项目中,经常需要安装和配置多个软件包和库。手动安装这些依赖项可能会非常繁琐和耗时。为了简化这个过程,我们可以使用Makefile来自动化安装过程。本教程将详细介绍如何使用Makefile来安装PyTorch和其他相关依赖项。
环境要求
- 操作系统:Linux 或 macOS
- Python 版本:3.7 及以上
- Conda 或 pip(如果你使用的是Anaconda)
项目结构
假设你的espnet项目结构如下:
espnet/
│
tools/
│
├── installers/
│ ├── install_ffmpeg.sh
│ ├── install_sctk.sh
│ ├── install_sph2pipe.sh
│ ├── install_torch.sh
│ ├── install_warp-transducer.sh
│ ├── install_chainer.sh
│ ├── install_kenlm.sh
│ └── ...
├── activate_python.sh
├── Makefile
└── ...
Makefile 详解
Makefile 是一个自动化构建工具,通过定义目标(targets)和依赖关系,可以方便地管理构建任务。
基本语法
在Makefile中,每个目标都有一组依赖项和一组命令。例如:
target: dependencies
command
主要目标
在我们的Makefile中,主要目标包括:
- 安装基本工具和库(如
cmake
、ffmpeg
、sox
等) - 安装 Python 依赖项(如
numpy
、torch
、espnet
等) - 安装特定工具(如
sctk
、sph2pipe
等)
示例Makefile
以下是一个示例Makefile,包含所有主要目标和命令:
TH_VERSION := 2.3.0
# Use pip for pytorch installation even if you have anaconda
ifneq ($(shell test -f ./activate_python.sh && grep 'conda activate' ./activate_python.sh),)
USE_CONDA := 1
else
USE_CONDA :=
endif
# Set if install binaries on CPU mode e.g. make CPU_ONLY=0
# If you don't have nvcc, this value will be set automatically
ifneq ($(shell which nvcc 2>/dev/null),)
CPU_ONLY :=
# Derive CUDA version from nvcc
CUDA_VERSION := $(shell nvcc --version | grep "Cuda compilation tools" | cut -d" " -f5 | sed s/,//)
CUDA_VERSION_WITHOUT_DOT := $(strip $(subst .,,$(CUDA_VERSION)))
else
CPU_ONLY := 0
CUDA_VERSION :=
CUDA_VERSION_WITHOUT_DOT :=
endif
WITH_OMP=ON
.PHONY: all clean
all: showenv conda_packages.done python ffmpeg.done sctk check_install
python: activate_python.sh setuptools.done packaging.done espnet.done pytorch.done chainer.done fairscale.done torch_optimizer.done flash_attn.done
activate_python.sh:
test -f activate_python.sh || { echo "Error: Run ./setup_python.sh or ./setup_anaconda.sh"; exit 1; }
################ Logging ################
showenv: activate_python.sh
ifeq ($(strip $(CPU_ONLY)),)
@echo CUDA_VERSION=$(CUDA_VERSION)
else
@echo Perform on CPU mode: CPU_ONLY=$(CPU_ONLY)
endif
@echo PYTHON=$(shell . ./activate_python.sh && command -v python3)
@echo PYTHON_VERSION=$(shell . ./activate_python.sh && python3 --version)
@echo USE_CONDA=$(USE_CONDA)
@echo TH_VERSION=$(TH_VERSION)
@echo WITH_OMP=$(WITH_OMP)
#########################################
bc.done: activate_python.sh
. ./activate_python.sh && { command -v bc || conda install -y bc -c conda-forge; }
touch bc.done
cmake.done: activate_python.sh
. ./activate_python.sh && { command -v cmake || conda install -y cmake; }
touch cmake.done
flac.done: activate_python.sh
. ./activate_python.sh && { command -v flac || conda install -y libflac -c conda-forge; }
touch flac.done
sox.done: activate_python.sh
. ./activate_python.sh && { command -v sox || conda install -y sox -c conda-forge; }
touch sox.done
sndfile.done: activate_python.sh
. ./activate_python.sh && { conda install -y libsndfile -c conda-forge; }
touch sndfile.done
ifneq ($(strip $(USE_CONDA)),)
conda_packages.done: bc.done cmake.done flac.done sox.done sndfile.done
else
conda_packages.done:
endif
touch conda_packages.done
ffmpeg.done: activate_python.sh
ifneq ($(strip $(USE_CONDA)),)
. ./activate_python.sh && { command -v ffmpeg || conda install -y ffmpeg -c conda-forge; }
else
. ./activate_python.sh && { command -v ffmpeg || ./installers/install_ffmpeg.sh; }
endif
touch ffmpeg.done
sctk: sctk/bin/sclite
sctk/bin/sclite:
./installers/install_sctk.sh
sph2pipe: sph2pipe/sph2pipe
sph2pipe/sph2pipe:
./installers/install_sph2pipe.sh
setuptools.done: activate_python.sh
. ./activate_python.sh && { python -m ensurepip --upgrade; pip install setuptools; }
touch setuptools.done
packaging.done: setuptools.done
. ./activate_python.sh && python3 -m pip install packaging
touch packaging.done
numpy.done: activate_python.sh
ifeq ($(strip $(USE_CONDA)),)
. ./activate_python.sh && python3 -m pip install numpy
else
. ./activate_python.sh && conda install -y numpy
endif
touch numpy.done
numba.done: numpy.done
. ./activate_python.sh && python3 -m pip install -U numba
touch numba.done
pytorch.done: packaging.done numba.done
ifeq ($(strip $(USE_CONDA)),)
. ./activate_python.sh && ./installers/install_torch.sh "false" "${TH_VERSION}" "${CUDA_VERSION}"
else
. ./activate_python.sh && ./installers/install_torch.sh "true" "${TH_VERSION}" "${CUDA_VERSION}"
endif
touch pytorch.done
espnet.done: pytorch.done conda_packages.done
. ./activate_python.sh && python3 -m pip install -e "..[train, recipe]"
. ./activate_python.sh && python -m nltk.downloader averaged_perceptron_tagger_eng
touch espnet.done
sounfile_test: espnet.done
. ./activate_python.sh && python3 ./test_soundfile.py
chainer.done: espnet.done
. ./activate_python.sh && ./installers/install_chainer.sh "${CUDA_VERSION}"
touch chainer.done
warp-transducer.done: pytorch.done conda_packages.done
ifeq ($(strip $(CPU_ONLY)),)
[ -n "${CUDA_HOME}" ] || { echo "Error: CUDA_HOME is not set."; exit 1; }
endif
. ./activate_python.sh && ./installers/install_warp-transducer.sh ${WITH_OMP}
touch warp-transducer.done
chainer_ctc.done: chainer.done
ifeq ($(strip $(CPU_ONLY)),)
[ -n "${CUDA_HOME}" ] || { echo "Error: CUDA_HOME is not set."; exit 1; }
endif
. ./activate_python.sh && ./installers/install_chainer_ctc.sh ${WITH_OMP}
touch chainer_ctc.done
nkf.done:
./installers/install_nkf.sh
touch nkf.done
pyopenjtalk.done: espnet.done conda_packages.done
. ./activate_python.sh && ./installers/install_pyopenjtalk.sh
touch pyopenjtalk.done
phonemizer.done: espnet.done conda_packages.done
ifeq ($(WITH_OMP),ON)
. ./activate_python.sh && ./installers/install_phonemizer.sh
touch phonemizer.done
else
echo "Warning: ./installers/install_phonemizer.sh requires WITH_OMP=ON"
endif
moses.done:
git clone --depth 1 https://github.com/moses-smt/mosesdecoder.git moses
touch moses.done
mwerSegmenter.done:
./installers/install_mwerSegmenter.sh
touch mwerSegmenter.done
kenlm.done: espnet.done conda_packages.done
. ./activate_python.sh && ./installers/install_kenlm.sh
touch kenlm.done
pesq: PESQ/P862_annex_A_2005_CD/source/PESQ
PESQ/P862_annex_A_2005_CD/source/PESQ:
./installers/install_pesq.sh
py3mmseg.done: espnet.done
. ./activate_python.sh && ./installers/install_py3mmseg.sh
touch py3mmseg.done
beamformit.done:
./installers/install_beamformit.sh
touch beamformit.done
torch_optimizer.done: espnet.done
. ./activate_python.sh && ./installers/install_torch_optimizer.sh
touch torch_optimizer.done
fairscale.done: espnet.done
. ./activate_python.sh && ./installers/install_fairscale.sh
touch fairscale.done
fairseq.done: espnet.done
. ./activate_python.sh && ./installers/install_fairseq.sh
touch fairseq.done
s3prl.done: espnet.done
. ./activate_python.sh && ./installers/install_s3prl.sh
touch s3prl.done
whisper.done: espnet.done
. ./activate_python.sh && ./installers/install_whisper.sh
touch whisper.done
lora.done: espnet.done
. ./activate_python.sh && ./installers/install_lora.sh
touch lora.done
k2.done: espnet.done
. ./activate_python.sh && ./installers/install_k2.sh
touch k2.done
transformers.done: espnet.done
. ./activate_python.sh && ./installers/install_transformers.sh
touch transformers.done
parallel-wavegan.done: espnet.done
. ./activate_python.sh && ./installers/install_parallel-wavegan.sh
touch parallel-wavegan.done
longformer.done: espnet.done
. ./activate_python.sh && ./installers/install_longformer.sh
touch longformer.done
flash_attn.done: espnet.done
. ./activate_python.sh && ./installers/install_flash_attn.sh
touch flash_attn.done
muskits.done: espnet.done
. ./activate_python.sh && ./installers/install_muskits.sh
touch muskits.done
check_install: python
. ./activate_python.sh; . ./extra_path.sh; python3 check_install.py
clean: clean_extra
rm -rf warp-transducer chainer_ctc
rm -rf *.done
find . -iname "*.pyc" -delete
clean_python:
rm -rf warp-transducer chainer_ctc
rm -f chainer_ctc.done espnet.done chainer.done pytorch.done warp-transducer.done
find . -iname "*.pyc" -delete
clean_extra:
rm -rf nkf.done swig.done moses.done mwerSegmenter.done
rm -rf hts_engine_API.done open_jtalk.done pyopenjtalk.done
rm -rf muskits.done
rm -rf rvad_fast.done
rm -rf chainer espeak-ng festival MBROLA ParallelWaveGAN versa
rm -rf py3mmseg sctk* speech_tools sph2pipe* ._mwerSegmenter
rm -rf nkf mecab swig moses mwerSegmenter
rm -rf PESQ PESQ.zip
安装和使用步骤
-
准备环境
- 安装
conda
或者确保有pip
。 - 克隆或下载你的项目。
- 安装
-
激活Python环境
- 创建并激活Python环境:
./setup_anaconda.sh # 或者 ./setup_python.sh
- 创建并激活Python环境:
-
运行Makefile
- 安装所有依赖项:
make all
- 你可以使用特定目标来安装某些依赖项,例如:
make pytorch.done
- 安装所有依赖项:
-
检查安装
- 确保所有依赖项都已正确安装:
make check_install
- 确保所有依赖项都已正确安装:
常见问题
-
问题:安装过程中遇到权限问题。
- 解决方案:确保你有足够的权限,或者在命令前添加
sudo
。
- 解决方案:确保你有足够的权限,或者在命令前添加
-
问题:某些依赖项未能安装。
- 解决方案:查看具体的错误信息,并根据提示手动安装依赖项。
清理
- 可以使用以下命令来清理安装的文件和依赖项:
make clean
结论
通过使用Makefile,可以大大简化复杂项目的安装过程。希望本教程能帮助你更好地理解和使用Makefile来管理项目依赖。如果有任何问题或建议,请随时留言。