Build netCDF, IO/API and CMAQ

Build netCDF, IO/API and CMAQ

Following Ch. 3, 4, and 5 in the CMAQ manual , build and install the following packages in your UNIX account (e.g., ~/projects/cmaq/):

• netcdf 4.1.2
• ioapi 3.1
• CMAQ 4.7.1

Notes:

1. Introduction
   Air quality simulation models for use at regional, state, and local scales within the United States. The models have been applied to estimate the ability of various control strategies to improve air quality and ensure cost-effective results.

   1. First-generation air quality models simulated air quality using simple chemistry at local scales, and Gaussian plume formulation was the basis for prediction.

   2. Second-generation models covered a broader range of scales (local, urban, regional) and pollutants, addressing each scale with a separate model that often focused on a single pollutant or issue (e.g., ozone, acid deposition).

   3. Third-generation models were needed that could treat multiple pollutants simultaneously and at scales up to continental or larger.

The EPACommunity Multiscale Air Quality (CMAQ) modeling system is a third-generation air quality model. It is available online at http://www.cmaq-model.org. CMAQ is designed for applications ranging from regulatory and policy analysis to understanding the complex interactions of atmospheric chemistry and physics. It is a three-dimensional Eulerian (i.e., gridded) atmospheric chemistry and transport modeling system that simulates ozone, particulate matter (PM), toxic airborne pollutants, visibility, and acidic and nutrient pollutant species throughout the troposphere.

• • CMAQ uses coupled mathematical representations of actual chemical and physical processes to simulate air quality.

  • CMAQ belongs to the Eulerian class of mathematical models that calculate a mass balance within each grid cell by solving the transport across each cell boundary and chemical transformations within each cell during a given time period.

  • CMAQ thus requires two primary types of inputs: meteorological information, and emission rates from sources of emissions that affect air quality.

The five main CMAQ programs are

• • The initial conditions processor (ICON)

The configuration options for ICON include selecting the chemical mechanism to model, defining the horizontal and vertical grids, and choosing whether the initial conditions are generated from an ASCII profile or from an existing CCTM output file.

• • The boundary conditions processor (BCON)

The configuration options for BCON include selecting the chemical mechanism to model, defining the horizontal and vertical grids, and choosing whether the boundary conditions are generated from an ASCII profile or from an existing CCTM output file.

• • The clear-sky photolysis rate calculator (JPROC)

The only configuration option required for JPROC is the selection of the chemical mechanism to use in the modeling. Output from JPROC is an ASCII look-up table of photolysis rates that CCTM uses to calculate gas-phase chemical transformations and pollutant concentrations.

• • The Meteorology-Chemistry Interface Processor (MCIP)

Configuration options for MCIP include the time periods over which to extract data from the meteorological model output files, horizontal and vertical grid definitions, and selections for calculating dry deposition velocities and integrating satellite cloud observations into MCIP output.

• • The CMAQ Chemistry-Transport Model (CCTM)

CCTM integrates the output from the preprocessing programs described above (JPROC, BCON, ICON, and MCIP), as well as CMAQ-ready emissions inputs (e.g., output from SMOKE), to simulate continuous atmospheric chemical conditions.

Configuration options for CCTM include the temporal coverage of the simulation, the chemical mechanism to use in the modeling, the physics scheme to use for modeling pollutant transport, heterogeneous and aqueous chemistry options, and diagnostic options (such as process analysis, discussed in the next paragraph).

CCTM has the largest number of configuration options of all the CMAQ programs.

Ancillary support programs distributed with CMAQ include

• • The code builder/manager (M3BLD)

It is the first program that needs to compiled after installing the CMAQ source code.

• • The chemical mechanism compiler (CHEMMECH)

Chemical mechanisms are represented in CMAQ through a series of INCLUDE files that contain mechanistic and kinetic parameters that describe a photochemical mechanism. CHEMMECH creates these INCLUDE files from an ASCII mechanism-definition file that represents the chemistry as sequential equations of reactants, products, and reaction rate information.

• • The process analysis preprocessor (PROCAN)

This preprocessor uses an input configuration file to select the process analysis options desired, and outputs three INCLUDE files that are used to compile a version of CCTM that is instrumented for process analysis.

2. Overview of the Science in the CMAQ Modeling System

3. CMAQ System Requirements and Installation

After benchmarking has been successfully completed, the CMAQ system can be configured for

other simulations.

Before installing any components of CMAQ, users must install CVS, netCDF, and I/O API.

The relation between the 'icc' command and 'icpc' command is the same as between 'gcc' and 'g++'. While both 'icc' and 'gcc' can compile C++ source, the don't know about the runtime libraries needed.

For gcc you can in fact use CXX=gcc when compiling the MySQL server sources, because work has been done to avoid C++ constructs that adds C++ runtime library dependencies. But this has not been done for icc, so you need to set CXX=icpc.

Setup Environmental Variables

# Setting compilers variables export CC="/opt/intel/Compiler/11.1/046/bin/intel64/icc" #"/usr/bin/gcc" export CXX="/opt/intel/Compiler/11.1/046/bin/intel64/icpc" #"/usr/bin/g++" export FC=ifort export F90=ifort export F77=ifort export PERL=/usr/bin/perl export BIN=Linux2_x86_64ifort # Setting code generation options export CPPFLAGS="-O2 -DNDEBUG -DpgiFortran" export CFLAGS="-g -O2" export CXXFLAGS="-g -O2" export FFLAGS='-g -O2 -mp -recursive' # Setting CMAQ environmental variables export M3HOME=~/CMAQ export M3DATA=$M3HOME/data export M3MODEL=$M3HOME/models export M3LIB=$M3HOME/lib export M3WORK=$M3HOME/work # Setting ifort environmental variables source /opt/intel/Compiler/11.1/046/bin/iccvars.sh intel64 source /opt/intel/Compiler/11.1/046/bin/ifortvars.sh intel64

安装 netcdf-4.1.2.tar.gz

1. 1. 在netcdf的安装文件（tar.gz）的目录中，提示符中输入 tar -xvf netcdf-4.1.2
   2. 解压完毕后，输入

./configure --prefix=/home/michaelchen/CMAQ/NetCDF4.1.2 --disable-separate-fortran --disable-netcdf4 --disable-dap --enable-f77 --disable-compiler-recover --disable-cxx --disable-shared
这一步里的参数很重要，会影响到ioapi的编译。

3. 3. make
   4. make install
      You should see netCDF in /home/michaelchen/CMAQ/NetCDF4.1.2 as well as new files in /include, /lib, /share and /man.

CMAQ仅需要库文件libnetcdf.a，因此只要将编译得到的libnetcdf.a链接到CMAQ的适当目录下

libnetcdf.a 信息：

size: 2.44 MB (2,561,780 bytes) size on disk: 2.44 MB (2,562,048 bytes)

安装 ioapi-3.1.tar.gz

1. 1. 在ioapi的安装文件（tar.gz）的目录中，提示符中输入 tar -xvf ioapi-3.1.tar.gz
   2. 进入ioapi-3.1目录，进行如下操作
   3. vim编辑Makefile （粗体是修改后的结果）

      BASEDIR = ${HOME}/CUDA_exercise/ex2/ioapi-3.1 INSTALL = ${HOME}/CUDA_exercise/ex2/ioapi-3.1/ioapi LIBINST = $(INSTALL)/$(BIN) BININST = $(INSTALL)/$(BIN) CPLMODE = nocpl IOAPIDEFS = PVMINCL = /dev/null #dev$(PVM_ROOT)/conf/$(PVM_ARCH).def

      1. 1. 1. 完成Makefile编辑后进行
      
      

      make configure

      make 在make的过程中会遇到这样的一些问题，每次遇到修改后重新make编译

      • • • /ioapi-3.1/ioapi/iobin3.c(1104) (col. 29): internal error: 0_1529
            这是因为Makeinclude.Linux2_x86_64ifort中用的是O3编译，替换对应的为O2即可
            COPTFLAGS = -O2 ${MFLAGS}
            FOPTFLAGS = -O2 -no-fp-port -unroll -stack_temps -safe_cray_ptr ${MFLAGS}
          • /ioapi-3.1/Linux2_x86_64ifort; ifort -auto -cm -w90 -w95 -warn notruncated_source -Bstatic airs2m3.o -L/home/ckcheng/projects/gpu/epd/ioapi-3.1/Linux2_x86_64ifort -lioapi -lnetcdf -lpvm3 -openmp -Vaxlib -Bstatic -o airs2m3
            ld: cannot find -lnetcdf
            这是因为lnetcdf.a没有在Linux2_x86_64ifort中。
            在/home/michaelchen/CMAQ/NetCDF4.1.2的lib目录下找到lentcdf.a拷贝到Linux2_x86_64ifort目录中即可。亦可以用ln命令链接过去。
          • 在编译m3tools中文件时，出现“cannot find -lpvm3”
            这是因为m3tools目录中Makefile有参数${PVMLIBS}的缘故。删除（反灰处）即可：
            LIBS = -L${OBJDIR} -lioapi -lnetcdf $(PVMLIBS) $(OMPLIBS) \
            $(ARCHLIB) $(ARCHLIBS)
            （有点奇怪的是，貌似编译完毕后，这个参数会重新出现在删除的位置）
          • 会有类似下面的现象出现多次。
            writeprofile.o: In function `main':
            writeprofile.f:(.text+0x0): multiple definition of `main'
            这是因为重复定义，删除对应的.o文件即可。大概会出现6次左右的.o文件冲突
      
      

      Linux2_x86_64ifort目录中会产生所需文件：libioapi.a

      CMAQ仅需要库文件libioapi.a，因此原则上只要将该文件编译出，并置于CMAQ的适当目录下即可

      libioapi.a 信息：

      size: 5.74 MB (6,019,448 bytes) size on disk: 5.74 MB (6,020,096 bytes)

      
      

      链接libnetcdf.a和libioapi.a（也可以不做这步，用拷贝替代）

      libnetcdf.a: /home/michaelchen/CMAQ/NetCDF4.1.2/lib/libnetcdf.a

      libioapi.a: /home/michaelchen/CUDA_exercise/ex2/ioapi-3.1/Linux2_x86_64ifort/libioapi.a

      
      

      编译所有模块前，确定在CMAQ的lib目录结构应至少有如下文件（如果没有可以手动建立）

      lib/: ioapi netCDF lib/ioapi: fixed_src Linux2_x86_64ifort lib/ioapi/fixed_src: ATDSC3.EXT CONST3.EXT FDESC3.EXT IODECL3.EXT NETCDF.EXT NOTICE.EXT PARMS3.EXT STATE3.EXT lib/ioapi/Linux2_x86_64ifort: libioapi.a lib/netCDF: Linux2_x86_64ifort lib/netCDF/Linux2_x86_64ifort: libnetcdf.a

      
      
      
      

      编译所有模块时，记住模块目录里bldit.XXX文件中的FC和MPICH要修改成自己机器上的ifor命令和MPICH及include文件所在位置

      set CC = /opt/intel/Compiler/11.1/046/bin/intel64/icc

      set FC = /opt/intel/Compiler/1/1.1/046/bin/intel64/ifort

      set MPICH = /opt/openmpi-1.4.3_intel

      set INCL = /opt/openmpi-1.4.3_intel/include (stenex目录中)

      set FC = /opt/openmpi-1.4.3_intel/bin/mpif90 (pario目录中，cctm的bldit.cctm)

      
      

      cctm目录中的machine8文件改成自己机器的名字xi03（8行同名）

      
      

      Suppose you are running in multiprocessor mode, please make the following changes.

      1. Edit cctm/bldit.cctm and re-run bldit.cctm

      < # set ParOpt # set for multiple PE's; comment out for single PE
      > set ParOpt # set for multiple PE's; comment out for single PE
      

      2. Edit cctm/run.cctm and re-run run.cctm. You may need to set the path for MPIRUN

      < setenv NPCOL_NPROW "1 1"; set NPROCS = 1 # single processor setting
      < # setenv NPCOL_NPROW "4 2"; set NPROCS = 8
      > # setenv NPCOL_NPROW "1 1"; set NPROCS = 1 # single processor setting
      > setenv NPCOL_NPROW "4 2"; set NPROCS = 8
      
      < time $BASE/$EXEC
      < # time $MPIRUN -v -machinefile $TASKMAP -np $NPROCS $BASE/$EXEC
      > # time $BASE/$EXEC
      > time $MPIRUN -v -machinefile $TASKMAP -np $NPROCS $BASE/$EXEC
      

      3. After a while, you may encounter error:

      ...
      The curr_str buffer is not big enough!
      The curr_str buffer is not big enough!
      The curr_str buffer is not big enough!
      The curr_str buffer is not big enough!
      ...
      

      4. Go to the directory cctm/BLD_ela and change the mode of file cctm/BLD_ela/distr_env.c

      cd BLD_ela
      chmod 666 distr_env.c
      

      5. To remove this error, edit cctm/BLD_ela/distr_env.c

      < #define TEMP_BUF_SIZE 102400
      < #define CURR_STR_SIZE 1024
      > #define TEMP_BUF_SIZE 204800
      > #define CURR_STR_SIZE 2048
      

      6. Remove all .o files in the directory cctm/BLD_ela

      rm *.o
      

      7. Re-do steps 1 and 2.

      If the buffer is not big enough again, try doubling the value in the step 4. (If the problem still exists after doubling, try larger value. For example, 10 times value)

      
      
      

      Links:

      • http://www.meto.umd.edu/~drb/

      • http://dorn1984.blog.163.com/blog/static/6418608720100215718225/

      • http://bbs.lasg.ac.cn/?action-viewthread-tid-45276

      • http://hi.baidu.com/wybai/blog/item/7ef2ec60f71047e4f73654f3.html

      • file:/fs0/michaelchen/CUDA_exercise/ex2/ioapi-3.1/HTML/ERRORS.html

      • http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install/Running-the-configure-Script.html

      • http://www.linuxsir.org/main/node/192?q=node/192

      
      
      

      Notes:

      • pwd 显示当前路径

      • mkdir 创建文件夹

      • rmdir 删除文件夹
        rm -rf 删除非空文件夹
      • History of Shells (http://www.dartmouth.edu/~rc/classes/ksh/history.html)
      
      

      sh

      aka "Bourne" shell, written by Steve Bourne at AT&T Bell Labs for Unix V7 (1979). Small, simple, and (originally) very few internal commands, so it called external programs for even the simplest of tasks. It is always available on everything that looks vaguely like Unix.

      csh

      The "C" shell. (Bill Joy, at Berkeley). Many things in common with the Bourne shell, but many enhancements to improve interactive use. The internal commands used only in scripts are very different from "sh", and similar (by design) to the "C" language syntax.

      tcsh

      The "TC" shell. Freely available and based on "csh". It has many additional features to make interactive use more convenient.

      ksh

      The "Korn" shell, written by David Korn of AT&T Bell Labs (now AT&T Research). Written as a major upgrade to "sh" and backwards compatible with it, but has many internal commands for the most frequently used functions. It also incorporates many of the features from tcsh which enhance interactive use (command line history recall etc.).

      POSIX 1003.2 Shell Standard

      Standards committees worked over the Bourne shell and added many features of the Korn shell (ksh88) and C shell to define a standard set of features which all compliant shells must have.

      bash

      The "Bourne again" shell. Written as part of the GNU/Linux Open Source effort, and the default shell for Linux and Mac OS-X. It is a functional clone of sh, with additional features to enhance interactive use, add POSIX compliance, and partial ksh compatability.

      zsh

      A freeware functional clone of sh, with parts of ksh, bash and full POSIX compliance, and many new interactive command-line editing features. It was installed as the default shell on early MacOSX systems.

      • setenv/unsetenv are commands that are built into the csh family of shells.
        Under bash the equivalent built ins are export and unset. For example:
        CSH: setenv name value
        BASH: export name=value
        CSH: unsetenv name
        BASH: unset name
      • openmpi 对 icc
        mpich 对 gcc
      • ls -l 显示更多属性