世上最全PETSC Linux安装攻略：天河新一代超算上PETSC安装，运行

 make ex9
/thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0  -I/thfs1/home/***/petsc-3.18.1/include -I/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/include -I/thfs1/software/mpich/mpi-n-gcc9.3.0/include     -export-dynamic ex9.c  -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -L/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/lib/aarch64-linux-gnu -L/lib/aarch64-linux-gnu -Wl,-rpath,/usr/lib/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -lpetsc -lflapack -lfblas -lpthread -lm -lX11 -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lstdc++ -ldl -o ex9

1) 下载PETSC

https://petsc.org/release/install/download/

目前最新版是3.18.1

https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.18.1.tar.gz

2）PETSC依赖lapack，blas和mpi

下载

在https://bitbucket.org/petsc/pkg-fblaslapack/downloads/中下载 

https://bitbucket.org/petsc/pkg-fblaslapack/get/e8a03f57d64c.zip  

具体名字可能会变

下载得到：petsc-pkg-fblaslapack-e8a03f57d64c.zip

MPI天河超算已经有，使用

$ module avail | grep mpi-n 得到 
mpich-3.4.2-gcc-7.5.0-jndkedx
mpich-3.4.2-gcc-9.3.0-fy2diix
py-mpi4py-3.1.2-gcc-9.3.0-6zi3t5f

$ module load module load mpich/mpi-n-gcc9.3.0

$ which mpif90

/thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpif90

得到具体地址，会自动把库和头文件路径加到当前界面的系统环境中

3）

把petsc-3.18.1.tar.gz上传到登录节点

gunzip petsc-3.18.1.tar.gz
tar -xf petsc-3.18.1

得到解压后的目录petsc-3.18.1

把petsc-pkg-fblaslapack-e8a03f57d64c.zip上传到登录节点

改名成：petsc-pkg-fblaslapack.zip

unzip petsc-pkg-fblaslapack.zip

解压后移动到目录petsc-3.18.1

3）

cd到目录 petsc-3.18.1

export PETSC_DIR=$PWD
mkdir arch-linux-c-debug
export PETSC_ARCH=arch-linux-c-debug
./configure --with-mpi-dir=/thfs1/software/mpich/mpi-n-gcc9.3.0/ --download-fblaslapack=/thfs1/home/***/petsc-3.18.1/petsc-pkg-fblaslapack

***指特定用户目录

这个过程挺慢的，等了好长时间。

这是什么玩意? Trying to download ./petsc-pkg-fblaslapack for FBLASLAPACK

不是已经下载了吗？

是这个编译器吗？

Found environment variable:
  F77=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gfortran.

这是全部的输出：

==========================================================================================
                         Configuring PETSc to compile on your system
==========================================================================================
==========================================================================================
                                     ***** WARNING *****
  Found environment variable:
  CC=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gcc.
  Ignoring it! Use "./configure CC=$CC" if you really want to use this value
==========================================================================================
==========================================================================================
                                     ***** WARNING *****
  Found environment variable:
  CXX=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/g++.
  Ignoring it! Use "./configure CXX=$CXX" if you really want to use this value
==========================================================================================
==========================================================================================
                                     ***** WARNING *****
  Found environment variable:
  FC=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gfortran.
  Ignoring it! Use "./configure FC=$FC" if you really want to use this value
==========================================================================================
==========================================================================================
                                     ***** WARNING *****
  Found environment variable:
  F77=/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/bin/gfortran.
  Ignoring it! Use "./configure F77=$F77" if you really want to use this value
==========================================================================================
==========================================================================================
                  Trying to download ./petsc-pkg-fblaslapack for FBLASLAPACK
==========================================================================================
==========================================================================================
                     Compiling FBLASLAPACK; this may take several minutes
==========================================================================================
Compilers:
  C Compiler:         /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc  -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
    Version: gcc (Spack GCC) 9.3.0
  C++ Compiler:         /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicxx  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g -O0  -std=gnu++17
    Version: g++ (Spack GCC) 9.3.0
  Fortran Compiler:         /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpif90  -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0
    Version: GNU Fortran (Spack GCC) 9.3.0
Linkers:
  Shared linker:   /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc  -shared  -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
  Dynamic linker:   /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc  -shared  -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0
  Libraries linked against:   -lstdc++ -ldl
BlasLapack:
  Libraries:  -Wl,-rpath,arch-linux-c-debug/lib -Larch-linux-c-debug/lib -lflapack -lfblas
  uses 4 byte integers
MPI:
  Version:    3
  Includes:   -I/thfs1/software/mpich/mpi-n-gcc9.3.0/include
  mpiexec: /thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpiexec
  Implementation: mpich3
  MPICH_NUMVERSION: 30402300
X:
  Libraries:  -lX11
pthread:
  Libraries:  -lpthread
fblaslapack:
cmake:
  Version:    3.23.1
  Executable: /thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-9.3.0/cmake-3.23.1-2yky63z/bin/cmake
python:
  Version:    3.9.12
  Executable: /thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-9.3.0/python-3.9.12-6onwaq6/bin/python3
regex:
bison:
  Version:    3.5
  Executable: /usr/bin/bison
PETSc:
  Language used to compile PETSc: C
  PETSC_ARCH: arch-linux-c-debug
  PETSC_DIR: /thfs1/home/***/petsc-3.18.1
  Prefix: <inplace installation>
  Scalar type: real
  Precision: double
  Support for __fp16
  Integer size: 4 bytes
  Single library: yes
  Shared libraries: yes
  Memory alignment from malloc(): 16 bytes
  Using GNU make: /usr/bin/gmake
xxx=========================================================================xxx
 Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug all
xxx====================================================================

make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug all

==========================================
          FC arch-linux-c-debug/obj/sys/fsrc/somefort.o
          FC arch-linux-c-debug/obj/sys/f90-src/fsrc/f90_fwrap.o
          CC arch-linux-c-debug/obj/sys/classes/random/interface/dlregisrand.o
          CC arch-linux-c-debug/obj/sys/classes/random/interface/ftn-auto/randomcf.o
          CC arch-linux-c-debug/obj/sys/classes/random/interface/ftn-auto/randomf.o
          CC arch-linux-c-debug/obj/sys/classes/random/interface/randreg.o
          CC arch-linux-c-debug/obj/sys/classes/viewer/interface/flush.o
          CC arch-linux-c-debug/obj/sys/classes/random/impls/rand/rand.o
          CC arch-linux-c-debug/obj/sys/classes/random/impls/rander48/rander48.o
		  。。。。。。
		  
          FC arch-linux-c-debug/obj/snes/f90-mod/petscsnesmod.o
          FC arch-linux-c-debug/obj/ts/f90-mod/petsctsmod.o
          FC arch-linux-c-debug/obj/tao/f90-mod/petsctaomod.o
     CLINKER arch-linux-c-debug/lib/libpetsc.so.3.18.1
=========================================
Now to check if the libraries are working do:
make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug check
=========================================

最后 告诉我们怎么测试，但是输出真是太长了，总共1400多行。

测试是否安装成功

make PETSC_DIR=/thfs1/home/***/petsc-3.18.1 PETSC_ARCH=arch-linux-c-debug check

结果如下：

Running check examples to verify correct installation
Using PETSC_DIR=/thfs1/home/***/petsc-3.18.1 and PETSC_ARCH=arch-linux-c-debug
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process
Completed test examples

测试成功。

4）案例测试

cd 到src/ksp/ksp/tutorials

make ex9

得到 如下 

 make ex9
/thfs1/software/mpich/mpi-n-gcc9.3.0/bin/mpicc -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g3 -O0  -I/thfs1/home/***/petsc-3.18.1/include -I/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/include -I/thfs1/software/mpich/mpi-n-gcc9.3.0/include     -export-dynamic ex9.c  -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -L/thfs1/home/***/petsc-3.18.1/arch-linux-c-debug/lib -Wl,-rpath,/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -L/thfs1/software/mpich/mpi-n-gcc9.3.0/lib -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib/gcc/aarch64-unknown-linux-gnu/9.3.0 -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib64 -Wl,-rpath,/lib/aarch64-linux-gnu -L/lib/aarch64-linux-gnu -Wl,-rpath,/usr/lib/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -Wl,-rpath,/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -L/thfs1/software/spack/optv018/linux-ubuntu20.04-aarch64/gcc-8.5.0/gcc-9.3.0-g3g7tub/lib -lpetsc -lflapack -lfblas -lpthread -lm -lX11 -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lstdc++ -ldl -o ex9

生成可执行文件 ex9

因为ex9没有输出， 同样可以make ex1

yhrun -n 1 -p thcp1 ex1

KSP Object: 1 MPI process
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI process
  type: jacobi
    type DIAGONAL
  linear system matrix = precond matrix:
  Mat Object: 1 MPI process
    type: seqaij
    rows=10, cols=10
    total: nonzeros=28, allocated nonzeros=50
    total number of mallocs used during MatSetValues calls=0
      not using I-node routines
Norm of error 2.41202e-15, Iterations 5

5）yhrun多进程运行及出现的问题

好像是MPI不匹配

 yhrun -n 2 -p thcp1 ex1
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Example/application run with number of MPI ranks it does not support
[0]PETSC ERROR: This is a uniprocessor example only!
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.1, Oct 26, 2022
[0]PETSC ERROR: ex1 on a arch-linux-c-debug named cn309 by monkeycode Sat Nov 19 10:57:32 2022
[0]PETSC ERROR: Configure options --with-mpi-dir=/***/mpich/mpi-n-gcc9.3.0/ --download-fblaslapack=/thfs1/home/***/petsc-3.18.1/petsc-pkg-fblaslapack
[0]PETSC ERROR: #1 main() at ex1.c:30
[0]PETSC ERROR: No PETSc Option Table entries
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 94) - process 0
slurmstepd: error: *** STEP 1706171.0 ON cn309 CANCELLED AT 2022-11-19T10:57:32 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: cn309: tasks 0-1: Killed

感觉是这个地方 不对：--with-mpi-dir

 rm arch-linux-c-debug/* -rf

把上面的过程重新来一遍

可能是问题本身的问题：

cd petsc-3.18.1/src/snes/tutorials

make ex19

 yhrun -n 4 -p thcp1 ./ex19

srun: job 1706249 queued and waiting for resources
srun: job 1706249 has been allocated resources
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2

但是yhrun -n 5 -p thcp1 ./ex19

就会出错：

yhrun -n 5 -p thcp1 ./ex19
srun: job 1706250 queued and waiting for resources
srun: job 1706250 has been allocated resources
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Partition in y direction is too fine! 4 5
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.1, Oct 26, 2022
[0]PETSC ERROR: /thfs1/home/***/petsc-3.18.1/src/snes/tutorials/./ex19 on a arch-linux-c-debug named cn291 by monkeycode Sat Nov 19 11:54:26 2022
[0]PETSC ERROR: Configure options --with-mpi-dir=/thfs1/software/mpich/mpi-n-gcc9.3.0/ --download-fblaslapack=/thfs1/home/***/petsc-3.18.1/petsc-pkg-fblaslapack
[0]PETSC ERROR: #1 DMSetUp_DA_2D() at /thfs1/home/***/petsc-3.18.1/src/dm/impls/da/da2.c:258
[0]PETSC ERROR: #2 DMSetUp_DA() at /thfs1/home/***/petsc-3.18.1/src/dm/impls/da/dareg.c:21
[0]PETSC ERROR: #3 DMSetUp() at /thfs1/home/***/petsc-3.18.1/src/dm/interface/dm.c:824
[0]PETSC ERROR: #4 main() at ex19.c:110
[0]PETSC ERROR: No PETSc Option Table entries
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_SELF, 63) - process 0
slurmstepd: error: *** STEP 1706250.0 ON cn291 CANCELLED AT 2022-11-19T11:54:26 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: cn291: tasks 0-4: Killed

应该是正确 的了

6)运行时间比较久的示例：

./ex45 -ksp_monitor_short -da_grid_x 81 -da_grid_y 81 -da_grid_z 81 -pc_type mg -pc_mg_levels 3 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi

-da_grid_x 81 -da_grid_y 81 -da_grid_z 81，这个81可以是21，41等，具体自己试一试