上海交大超算容器制作

1.登录PI2.0的login节点

2.转到PI2.0的容器制作节点

ssh build@container-x86

3.编写docker的容器定义文件siyuan1.DockerFile，siyuan1是名字可以随便起

我的siyuan1.DockerFile文件内容

# siyuan1.DockerFile
FROM centos:8 
# make essential
RUN sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-* && \
sed -i -e "s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g" /etc/yum.repos.d/CentOS-* 
# basic tools
RUN yum install -y \
        tree \
        vim \
        wget && \
    rm -rf /var/cache/yum/*
RUN echo "basic tools completed !!" 
# gmsh basic libs
RUN yum install -y \
        mesa-libGLU \
        libXrender-0.9.10-7.el8.x86_64 \
        libXcursor-1.1.15-3.el8.x86_64 \
        libXft-2.3.3-1.el8.x86_64 \
        libXinerama-1.1.4-1.el8.x86_64 \
        libgomp-8.5.0-4.el8_5.x86_64 && \
    rm -rf /var/cache/yum/*
RUN echo "gmsh basic libs completed !!" 
# install gmsh
RUN wget http://www.gmsh.info/bin/Linux/gmsh-4.4.0-Linux64.tgz && \
    tar -xzf gmsh-4.4.0-Linux64.tgz && \
    rm -f gmsh-4.4.0-Linux64.tgz 
RUN echo "gmsh 4.4.0 completed !!" 
# openfoam06 basic tools
RUN yum install -y \
        epel-release \
        cmake3 \
        make \
        git \
        wget && \
    # ln -s /usr/bin/cmake3 /usr/bin/cmake && \
    rm -rf /var/cache/yum/* 
RUN echo "openfoam06 basic tools completed !!" 
# OpenFOAM 6.0 tools 2
RUN yum install -y \
        patch \
        flex \
        bison \
        zlib-devel \
        boost-system \
        boost-thread \
        readline-devel \
        ncurses-devel \
        openssl && \
    rm -rf /var/cache/yum/*
RUN echo "OpenFOAM 6.0 tools 2 completed !!" 
# OpenFOAM 6.0 download
RUN mkdir -p /opt && \
    export BUILD_DIR=/opt/OpenFOAM && \
    mkdir -p $BUILD_DIR && \
    cd $BUILD_DIR && \
    git clone https://gitee.com/diyboy/OpenFOAM-6 && \
    git clone https://gitee.com/diyboy/ThirdParty-6 && \
    # git clone https://github.com/OpenFOAM/OpenFOAM-6 && \
    # git clone https://github.com/OpenFOAM/ThirdParty-6 && \
    pwd 
RUN echo "OpenFOAM 6.0 download completed !!" 
# OpenFOAM 6.0 libs
RUN yum install -y \
        gcc-8.5.0-4.el8_5.x86_64 \
        gcc-c++-8.5.0-4.el8_5.x86_64 \
        openmpi-4.1.1-2.el8.x86_64 && \
    rm -rf /var/cache/yum/* 
RUN echo "OpenFOAM 6.0 libs completed !!" 
# OpenFOAM 6.0 make
RUN export PATH=$PATH:/usr/lib64/openmpi/bin && \
    cd /opt/OpenFOAM/OpenFOAM-6 && \
    sed -i '47d' etc/bashrc && \
    sed -i '46a export FOAM_INST_DIR=/opt/\$WM_PROJECT' etc/bashrc && \
    source etc/bashrc && \
    ./Allwmake -j 
RUN echo "OpenFOAM 6.0 make completed !! " 
RUN echo "export PATH="$PATH":/usr/lib64/openmpi/bin:/usr/lib64/openmpi/bin/mpicc " >> /etc/bashrc && \
    echo "source /opt/OpenFOAM/OpenFOAM-6/etc/bashrc " >> /etc/bashrc 
# ThirdParty libs
RUN yum groupinstall -y 'Development Tools' && \
    yum install -y \
        qt5-qtbase-devel \
        libXt-devel \
        zlib-devel \
        libXext-devel \
        libGLU-devel \
        libXrender-devel \
        libXinerama-devel \
        libpng-devel \
        libXrandr-devel \
        libXi-devel \
        libXft-devel \
        libjpeg-turbo-devel \
        libXcursor-devel \
        readline-devel \
        ncurses-devel \
        python38 \
        cmake \
        mpfr-devel \
        gmp \
        gmp-devel && \
    rm -rf /var/cache/yum/* 
RUN echo "ThirdParty libs completed !! " 
# ThirdParty-6 make 
RUN source /etc/bashrc && \
    cd $WM_THIRD_PARTY_DIR && \
    export PATH=$PATH:/usr/lib64/qt5/bin 
    # ./makeParaView && \
    # wMRefresh && \
    # cd $FOAM_UTILITIES/postProcessing/graphics/PVReaders && \
    # ./Allwclean && \
    # ./Allwmake -j 
RUN echo "make ThirdParty-6 completed !! " 

RUN echo "----end-----" 

CMD /bin/bash 

注意：我是从centos8开始制作的，很多库不成熟，建议大家从centos7开始制作容器，我优先安装了gmsh这款软件，这是不需要的，可以直接安装OpenFoam，我OpenFoam-6编译成功了，但是还没有编译ThirdParty-6，因为centos8这个系统太新了，不太熟

注意这几行：

RUN echo "export PATH="$PATH":/usr/lib64/openmpi/bin:/usr/lib64/openmpi/bin/mpicc " >> /etc/bashrc && \
    echo "source /opt/OpenFOAM/OpenFOAM-6/etc/bashrc " >> /etc/bashrc 

代表把mpicc和OF6的bashrc添加进容器系统的bashrc，我只安装了OF6如果安装更多的OF版本，需要

RUN echo "export PATH="$PATH":/usr/lib64/openmpi/bin:/usr/lib64/openmpi/bin/mpicc " >> /etc/bashrc && \
    echo "alias of60=\"/opt/OpenFOAM/OpenFOAM-6/etc/bashrc\" " >> /etc/bashrc 

然后在使用容器的时候，记得先加载/etc/bashrc

注意一下这里：

RUN export PATH=$PATH:/usr/lib64/openmpi/bin && \
    cd /opt/OpenFOAM/OpenFOAM-6 && \
    sed -i '47d' etc/bashrc && \
    sed -i '46a export FOAM_INST_DIR=/opt/\$WM_PROJECT' etc/bashrc && \
    source etc/bashrc && \
    ./Allwmake -j 

这里通过sed命令更改一下OF6的安装路径，把他安装在/opt/里面

4.制作siyuan1的docker镜像

docker build -f ./container/siyuan1/siyuan1.DockerFile -t siyuan1 . 

这里需要等他自己下载源代码，然后安装编译啥的，首次大概俩小时

5.把制作好的docker容器转换成sif文件

singularity build --force siyuan1.sif docker-daemon://siyuan1:latest 

6.把sif文件通过scp命令传到思源一号上

scp -o StrictHostKeyChecking=no ./siyuan1.sif naoxb@sylogin1.hpc.sjtu.edu.cn:~/tongxin/ 

注意：修改成你自己的路径

7.退出，然后登录思源一号，给大家看一下我的slurm文件

#!/bin/bash

#SBATCH --job-name=myCase
#SBATCH --partition=64c512g
#SBATCH -N 1
#SBATCH --ntasks-per-node=default
#SBATCH --output=%j.out
#SBATCH --error=%j.err

module purge &&
module --ignore-cache load "openmpi/4.1.1-gcc-9.3.0" &&

export IMAGE_NAME=../../../siyuan1.sif &&
export myScript=". /etc/bashrc " &&
export myScript=$myScript"&&"$myAppBin" -parallel " &&
echo "begin mpi running " &&
mpirun -n $SLURM_NTASKS singularity exec $IMAGE_NAME /bin/bash -c "$myScript" &&
echo "end mpi running " 

注意：先加载一下openmpi模组

运行容器的时候先source /etc/bashrc 再加你的命令，当然，你需要先在容器里编译你的求解器

export script=". /etc/bashrc " &&
export script=$script"&&wclean "$mySolverPath" " &&
export script=$script"&&wmake "$mySolverPath" " &&
export script=$script"&&wclean "$mySolverPath" " &&
singularity exec ../../../siyuan1.sif /bin/bash -c "$script" &&
export FOAM_USER_APPBIN='/dssg/home/acct-naoxb/naoxb/OpenFOAM/naoxb-6/platforms/linux64GccDPInt32Opt/bin/' &&
cp $FOAM_USER_APPBIN/$mySolverName ./dict/$mySolverName 

大致流程就是这样，思源一号比pi2.0快了三倍，不需要排队，还是比较好用的，编译求解器需要先申请一个计算节点

srun -p 64c512g -n 4 --pty /bin/bash 

最后，稍微讲一下容器的工作原理，容器里的文件系统如下

/etc
/opt
/dssg/home/acct-naoxb/naoxb/**
/lib
等等

其中/dssg/home/acct-naoxb/naoxb/**这个数据盘是自动挂载的，和宿主机同步的。容器的本质就是除了数据盘以外的文件系统，我是把OF6安装在了/opt/里面。进入容器就相当于是把除了数据盘以外的文件系统替换成了容器自己的文件系统，你可以在这个文件系统里安装任意需要的软件。

本文结束，谢谢阅读