make ex9
/thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0 -I/thfs1/home/***/petsc-3.18.1/include -I/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/include -I/thfs1/software/mpich/mpi-n-gcc9.3.0/include -export-dynamic ex9.c -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -L/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/lib/aarch64-linux-gnu -L/lib/aarch64-linux-gnu -Wl,-rpath,/usr/lib/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -lpetsc -lflapack -lfblas -lpthread -lm -lX11 -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lstdc++ -ldl -o ex9
1) 下载PETSC
https://petsc.org/release/install/download/
目前最新版是3.18.1
https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.18.1.tar.gz
2)PETSC依赖lapack,blas和mpi
下载
在https://bitbucket.org/petsc/pkg-fblaslapack/downloads/中下载
https://bitbucket.org/petsc/pkg-fblaslapack/get/e8a03f57d64c.zip
具体名字可能会变
下载得到:petsc-pkg-fblaslapack-e8a03f57d64c.zip
MPI天河超算已经有,使用
$ module avail | grep mpi-n 得到
mpich-3.4.2-gcc-7.5.0-jndkedx
mpich-3.4.2-gcc-9.3.0-fy2diix
py-mpi4py-3.1.2-gcc-9.3.0-6zi3t5f
$ module load module load mpich/mpi-n-gcc9.3.0
$ which mpif90
/thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpif90
得到具体地址,会自动把库和头文件路径加到当前界面的系统环境中
3)
把petsc-3.18.1.tar.gz上传到登录节点
gunzip petsc-3.18.1.tar.gz
tar -xf petsc-3.18.1
得到解压后的目录petsc-3.18.1
把petsc-pkg-fblaslapack-e8a03f57d64c.zip上传到登录节点
改名成:petsc-pkg-fblaslapack.zip
unzip petsc-pkg-fblaslapack.zip
解压后移动到目录petsc-3.18.1
3)
cd到目录 petsc-3.18.1
export PETSC_DIR=$PWD
mkdir arch-linux-c-debug
export PETSC_ARCH=arch-linux-c-debug
./configure --with-mpi-dir=/thfs1/software/mpich/mpi-n-gcc9.3.0/ --download-fblaslapack=/thfs1/home/***/petsc-3.18.1/petsc-pkg-fblaslapack
***指特定用户目录
这个过程挺慢的,等了好长时间。
这是什么玩意? Trying to download ./petsc-pkg-fblaslapack for FBLASLAPACK
不是已经下载了吗?
是这个编译器吗?
Found environment variable:
F77=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gfortran.
这是全部的输出:
==========================================================================================
Configuring PETSc to compile on your system
==========================================================================================
==========================================================================================
***** WARNING *****
Found environment variable:
CC=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gcc.
Ignoring it! Use "./configure CC=$CC" if you really want to use this value
==========================================================================================
==========================================================================================
***** WARNING *****
Found environment variable:
CXX=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/g++.
Ignoring it! Use "./configure CXX=$CXX" if you really want to use this value
==========================================================================================
==========================================================================================
***** WARNING *****
Found environment variable:
FC=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gfortran.
Ignoring it! Use "./configure FC=$FC" if you really want to use this value
==========================================================================================
==========================================================================================
***** WARNING *****
Found environment variable:
F77=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gfortran.
Ignoring it! Use "./configure F77=$F77" if you really want to use this value
==========================================================================================
==========================================================================================
Trying to download ./petsc-pkg-fblaslapack for FBLASLAPACK
==========================================================================================
==========================================================================================
Compiling FBLASLAPACK; this may take several minutes
==========================================================================================
Compilers:
C Compiler: /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
Version: gcc (Spack GCC) 9.3.0
C++ Compiler: /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g -O0 -std=gnu++17
Version: g++ (Spack GCC) 9.3.0
Fortran Compiler: /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpif90 -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0
Version: GNU Fortran (Spack GCC) 9.3.0
Linkers:
Shared linker: /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc -shared -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
Dynamic linker: /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc -shared -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
Libraries linked against: -lstdc++ -ldl
BlasLapack:
Libraries: -Wl,-rpath,arch-linux-c-debug/lib -Larch-linux-c-debug/lib -lflapack -lfblas
uses 4 byte integers
MPI:
Version: 3
Includes: -I/thfs1/software/mpich/mpi-n-gcc9.3.0/include
mpiexec: /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpiexec
Implementation: mpich3
MPICH_NUMVERSION: 30402300
X:
Libraries: -lX11
pthread:
Libraries: -lpthread
fblaslapack:
cmake:
Version: 3.23.1
Executable: /thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-9.3.0/cmake-3.23.1-2yky63z/bin/cmake
python:
Version: 3.9.12
Executable: /thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-9.3.0/python-3.9.12-6onwaq6/bin/python3
regex:
bison:
Version: 3.5
Executable: /usr/bin/bison
PETSc:
Language used to compile PETSc: C
PETSC_ARCH: arch-linux-c-debug
PETSC_DIR: /thfs1/home/***/petsc-3.18.1
Prefix: <inplace installation>
Scalar type: real
Precision: double
Support for __fp16
Integer size: 4 bytes
Single library: yes
Shared libraries: yes
Memory alignment from malloc(): 16 bytes
Using GNU make: /usr/bin/gmake
xxx=========================================================================xxx
Configure stage complete. Now build PETSc libraries with:
make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug all
xxx====================================================================
make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug all
==========================================
FC arch-linux-c-debug/obj/sys/fsrc/somefort.o
FC arch-linux-c-debug/obj/sys/f90-src/fsrc/f90_fwrap.o
CC arch-linux-c-debug/obj/sys/classes/random/interface/dlregisrand.o
CC arch-linux-c-debug/obj/sys/classes/random/interface/ftn-auto/randomcf.o
CC arch-linux-c-debug/obj/sys/classes/random/interface/ftn-auto/randomf.o
CC arch-linux-c-debug/obj/sys/classes/random/interface/randreg.o
CC arch-linux-c-debug/obj/sys/classes/viewer/interface/flush.o
CC arch-linux-c-debug/obj/sys/classes/random/impls/rand/rand.o
CC arch-linux-c-debug/obj/sys/classes/random/impls/rander48/rander48.o
。。。。。。
FC arch-linux-c-debug/obj/snes/f90-mod/petscsnesmod.o
FC arch-linux-c-debug/obj/ts/f90-mod/petsctsmod.o
FC arch-linux-c-debug/obj/tao/f90-mod/petsctaomod.o
CLINKER arch-linux-c-debug/lib/libpetsc.so.3.18.1
=========================================
Now to check if the libraries are working do:
make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug check
=========================================
最后 告诉我们怎么测试,但是输出真是太长了,总共1400多行。
测试是否安装成功
make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug check
结果如下:
Running check examples to verify correct installation
Using PETSC_DIR=/thfs1/home/***/petsc-3.18.1 and PETSC_ARCH=arch-linux-c-debug
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
测试成功。
4)案例测试
cd 到src/ksp/ksp/tutorials
make ex9
得到 如下
make ex9
/thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0 -I/thfs1/home/***/petsc-3.18.1/include -I/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/include -I/thfs1/software/mpich/mpi-n-gcc9.3.0/include -export-dynamic ex9.c -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -L/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/lib/aarch64-linux-gnu -L/lib/aarch64-linux-gnu -Wl,-rpath,/usr/lib/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -lpetsc -lflapack -lfblas -lpthread -lm -lX11 -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lstdc++ -ldl -o ex9
生成可执行文件 ex9
因为ex9没有输出, 同样可以make ex1
yhrun -n 1 -p thcp1 ex1
KSP Object: 1 MPI process
type: gmres
restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
happy breakdown tolerance 1e-30
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI process
type: jacobi
type DIAGONAL
linear system matrix = precond matrix:
Mat Object: 1 MPI process
type: seqaij
rows=10, cols=10
total: nonzeros=28, allocated nonzeros=50
total number of mallocs used during MatSetValues calls=0
not using I-node routines
Norm of error 2.41202e-15, Iterations 5
5)yhrun多进程运行及出现的问题
好像是MPI不匹配
yhrun -n 2 -p thcp1 ex1
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Example/application run with number of MPI ranks it does not support
[0]PETSC ERROR: This is a uniprocessor example only!
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.1, Oct 26, 2022
[0]PETSC ERROR: ex1 on a arch-linux-c-debug named cn309 by monkeycode Sat Nov 19 10:57:32 2022
[0]PETSC ERROR: Configure options --with-mpi-dir=/***/mpich/mpi-n-gcc9.3.0/ --download-fblaslapack=/thfs1/home/***/petsc-3.18.1/petsc-pkg-fblaslapack
[0]PETSC ERROR: #1 main() at ex1.c:30
[0]PETSC ERROR: No PETSc Option Table entries
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 94) - process 0
slurmstepd: error: *** STEP 1706171.0 ON cn309 CANCELLED AT 2022-11-19T10:57:32 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: cn309: tasks 0-1: Killed
感觉是这个地方 不对:--with-mpi-dir
rm arch-linux-c-debug/* -rf
把上面的过程重新来一遍
可能是问题本身的问题:
cd petsc-3.18.1/src/snes/tutorials
make ex19
yhrun -n 4 -p thcp1 ./ex19
srun: job 1706249 queued and waiting for resources
srun: job 1706249 has been allocated resources
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2
但是yhrun -n 5 -p thcp1 ./ex19
就会出错:
yhrun -n 5 -p thcp1 ./ex19
srun: job 1706250 queued and waiting for resources
srun: job 1706250 has been allocated resources
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Partition in y direction is too fine! 4 5
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.1, Oct 26, 2022
[0]PETSC ERROR: /thfs1/home/***/petsc-3.18.1/src/snes/tutorials/./ex19 on a arch-linux-c-debug named cn291 by monkeycode Sat Nov 19 11:54:26 2022
[0]PETSC ERROR: Configure options --with-mpi-dir=/thfs1/software/mpich/mpi-n-gcc9.3.0/ --download-fblaslapack=/thfs1/home/***/petsc-3.18.1/petsc-pkg-fblaslapack
[0]PETSC ERROR: #1 DMSetUp_DA_2D() at /thfs1/home/***/petsc-3.18.1/src/dm/impls/da/da2.c:258
[0]PETSC ERROR: #2 DMSetUp_DA() at /thfs1/home/***/petsc-3.18.1/src/dm/impls/da/dareg.c:21
[0]PETSC ERROR: #3 DMSetUp() at /thfs1/home/***/petsc-3.18.1/src/dm/interface/dm.c:824
[0]PETSC ERROR: #4 main() at ex19.c:110
[0]PETSC ERROR: No PETSc Option Table entries
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_SELF, 63) - process 0
slurmstepd: error: *** STEP 1706250.0 ON cn291 CANCELLED AT 2022-11-19T11:54:26 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: cn291: tasks 0-4: Killed
应该是正确 的了
6)运行时间比较久的示例:
./ex45 -ksp_monitor_short -da_grid_x 81 -da_grid_y 81 -da_grid_z 81 -pc_type mg -pc_mg_levels 3 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi
-da_grid_x 81 -da_grid_y 81 -da_grid_z 81,这个81可以是21,41等,具体自己试一试