算法面试经典100题，Transformers之环境安装(1)，Python高级工程师必备知识

在transformers的docker目录中提供了各个版本的dockerfile文件。可以直接docker命令创建镜像。

docker build --build-arg --network host -t ai_suite:cuda12.1-torch2.1-22.04 .

启动容器。

xhost + && docker run --gpus all --env NVIDIA_DISABLE_REQUIRE=1 -it --network=host --name transformers-container --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home:/home -v /tmp:/tmp -v /mnt:/mnt --ipc=host -e DISPLAY=${DISPLAY} -e GIT_INDEX_FILE ai_suite:cuda12.1-torch2.1-22.04 bash

自定义dockerfile

自定义安装，主要是为了在docker中使用conda虚拟环境。

https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-23-11.html#rel-23-11

FROM nvcr.io/nvidia/pytorch:23.11-py3
LABEL maintainer=“transformers”

ARG DEBIAN_FRONTEND=noninteractive

ARG PYTORCH=‘2.1.0’

Example: cu102, cu113, etc.

ARG CUDA=‘cu121’

RUN apt-get update &&
apt-get install -y libaio-dev wget bzip2 ca-certificates curl git git-lfs unzip mlocate usbutils
vim tmux g++ gcc build-essential cmake checkinstall lsb-release &&
rm -rf /var/lib/apt/lists/* &&
apt-get clean

RUN python3 -m pip uninstall -y torch torchvision torchaudio torch-tensorrt transformer-engine apex

SHELL [“/bin/bash”, “–login”, “-c”]

RUN cd / && wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /miniconda.sh &&
/bin/bash /miniconda.sh -b -p /opt/conda &&
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh &&
echo “. /opt/conda/etc/profile.d/conda.sh” >> ~/.bashrc &&
/bin/bash -c “source ~/.bashrc” &&
/opt/conda/bin/conda update -n base -c defaults conda -y &&
/opt/conda/bin/conda config --set ssl_verify no &&
/opt/conda/bin/conda config --add channels conda-forge &&
/opt/conda/bin/conda create -n ai python=3.10

ENV PATH $PATH:/opt/conda/envs/ai/bin

RUN conda init bash &&
echo “conda activate ai” >> ~/.bashrc &&
conda activate ai &&
pip install --upgrade pip -i https://mirror.baidu.com/pypi/simple &&\
pip config set global.index-url https://mirror.baidu.com/pypi/simple &&\

Install latest release PyTorch

(PyTorch must be installed before pre-compiling any DeepSpeed c++/cuda ops.)

(https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops)

pip install --no-cache-dir -U torch== P Y T O R C H t o r c h v i s i o n