【教程】在天河上安装、移植并验证CESM2.2.0

基本情况

超算中心：国家超级计算天津中心
软件名称：CESM
软件版本：2.2.0
集群：TH-1A
编译节点：TH-1A-LN9
编译器版本：Intel_compiler/17.0.4
MPI版本：MPI/mpich/intel2017

环境配置

所需环境 CESM2 Software/Operating System Prerequisites

参考：https://escomp.github.io/CESM/versions/cesm2.2/html/introduction.html#cesm2-software-operating-system-prerequisites

• UNIX style operating system such as CNL, AIX or Linux
• python >= 2.7
• perl 5
• subversion client (version 1.8 or greater but less than v1.11) for downloading CAM, POP, and WW3
• git client (1.8 or greater)
• Fortran compiler with support for Fortran 2003
• C compiler
• MPI (although CESM does not absolutely require it for running on one processor)
• NetCDF 4.3 or newer.
• ESMF 5.2.0 or newer (optional).
• pnetcdf 1.7.0 is required and 1.8.1 is optional but recommended
• Trilinos may be required for certain configurations
• LAPACK and BLAS
• CMake 2.8.6 or newer

若perl>=5.6
需要把<CESM源码目录>/components/clm/bld/CLMBuildNamelist.pm里面的bsd_glob改成glob
"Since v5.6.0, Perl’s CORE::glob() is implemented in terms of bsd_glob()."

libxml2问题
libreadline.so.5
'/vol-th/software/libxml2-291/bin/xmllint: error while loading shared libraries: libreadline.so.5: cannot open shared object file: No such file or directory'
ERROR: Command: '/vol-th/software/libxml2-291/bin/xmllint --xinclude --noout --schema /vol-th/home/laiyq/cesm/cesm220/cime/config/xml_schemas/env_entry_id.xsd /vol-th/home/laiyq/cesm/scratch/F2000climo/env_case.xml' failed with error '/vol-th/software/libxml2-291/bin/xmllint: error while loading shared libraries: libreadline.so.5: cannot open shared object file: No such file or directory' from dir '/vol-th/home/laiyq/cesm/scratch/F2000climo
在TH-1A里，不导入perl和libxml2，用原本自带的反而正常运行了！可能和版本有关？

ld: cannot find -lmkl_cdft_core
在TH-1A使用Intel_complier2018会出现该问题，因为mkl版本不是cluster版本的
Interl_complier2017是对的！

Interl_complier2017使用impi在yhrun/yhbatch/sbatch时会出现错误！用mpich

使用source脚本配置模块和环境变量

source cesm_env_init

echo ""
echo "CESM Env Init"
echo ""

# This is Module load part (DO NOT OVERWRITE IT) 
source ~/../../software/modules/3.2.10/Modules/3.2.10/init/bash

module purge

module add Intel_compiler/17.0.4
module add MPI/mpich/intel2017
#import IMPI/17.0.4 manualy
#export PATH=/vol-th/software/intel2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin:$PATH
#export LIBRARY_PATH=/vol-th/software/intel2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib:$LIBRARY_PATH
#export LD_LIBRARY_PATH=/vol-th/software/intel2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib:$LD_LIBRARY_PATH

module add cmake/3.20.3
module add netcdf/4.4
module add pnetcdf/1.6.1
module add python/3.7_anaconda

module add lapack/3.8.0
module add blas/200703

module add mkl/17.0.4
module add loginnode/ln9

#module add hdf5/1.8.11

export PERL5LIB=/vol-th/software/loginnode/ln9/usr/lib64/perl5/vendor_perl:$PERL5LIB

#export LD_LIBRARY_PATH=/vol-th/software/io_tools/hdf5/mpi/1.8.11/lib:/vol-th/software/io_tools/netcdf/mpi/4.4/lib:/vol-th/software/io_tools/pnetcdf/1.6.1/lib:$LD_LIBRARY_PATH

export CESMDATAROOT=/vol-th/home/laiyq/cesm
export CESMROOT=/vol-th/home/laiyq/cesm
export CIME_MACHINE=tianhe-1A
export CIMEROOT=/vol-th/home/laiyq/cesm/cesm220/cime

MPI测试

It is usually very helpful to assure that you can run a basic mpi parallel program on your machine prior to attempting a CIME port. Understanding how to compile and run the program fhello_world_mpi.F90 shown here could potentially save many hours of frustration.

program fhello_world_mpi
  use mpi
  implicit none
  integer ( kind = 4 ) error
  integer ( kind = 4 ) id
  integer p
  character(len=MPI_MAX_PROCESSOR_NAME) :: name
  integer clen
  integer, allocatable :: mype(:)
  real ( kind = 8 ) wtime

  call MPI_Init ( error )
  call MPI_Comm_size ( MPI_COMM_WORLD, p, error )
  call MPI_Comm_rank ( MPI_COMM_WORLD, id, error )
  if ( id == 0 ) then
     wtime = MPI_Wtime ( )

     write ( *, '(a)' ) ' '
     write ( *, '(a)' ) 'HELLO_MPI - Master process:'
     write ( *, '(a)' ) '  FORTRAN90/MPI version'
     write ( *, '(a)' ) ' '
     write ( *, '(a)' ) '  An MPI test program.'
     write ( *, '(a)' ) ' '
     write ( *, '(a,i8)' ) '  The number of processes is ', p
     write ( *, '(a)' ) ' '
  end if
  call MPI_GET_PROCESSOR_NAME(NAME, CLEN, ERROR)
  write ( *, '(a)' ) ' '
  write ( *, '(a,i8,a,a)' ) '  Process ', id, ' says "Hello, world!" ',name(1:clen)

  call MPI_Finalize ( error )
end program

在登录节点上进行测试：

mpif90 fhello_world_mpi.F90 -o hello_world
yhrun -n 24 --ntasks-per-node 12 -p debug ./hello_world

CESM安装及移植

CESM源码下载

由于天河环境无法连接外网，需在本地下载好代码（并且完成“./manage_externals/checkout_externals”）后上传至天河。

源码地址及下载教程：https://github.com/ESCOMP/CESM

inputdata同理。

/vol6/software/CESM2里面有inputdata！还有LUH2部分数据。
但是case.submit以后会有问题，经测试计算节点无法访问/vol6
/vol-th/software/inputdata里也有，但不完整！

/vol6/software/CESM2的du结果：
11T	./inputdata/atm
979G	./inputdata/cesm2_init
387M	./inputdata/data_licom
4.5G	./inputdata/dx7
741M	./inputdata/glc
44G	./inputdata/ice
4.0K	./inputdata/import
8.2T	./inputdata/lnd
3.7T	./inputdata/ocn
7.8G	./inputdata/rof
33G	./inputdata/share
18G	./inputdata/validation