Running NWCHEM

NOTE:
NWChem was built using a different compiler then from the system version and uses special libraries as well. As such you must include the new GCC into your environment to use the new version of NWChem. This can be done by running:source /share/apps/simulations/nwchem-6.1.1/nwchem-source.sh

This can also be included into your default environment by adding that line to your~/.bash_profile file that is read every time when one opens a shell. Thus if this is already part of one's environment, one can use the-V option in their job script to pass the current environment to the job.

The preferred way would be to add this to the job script by including the following snippet in the job script before it launches the program.
#$ -S /bin/bash
source /share/apps/simulations/nwchem-6.1.1/nwchem-source.sh


First Run
For first time users, you need to ensure the following file is in your home directory. The following command is the best way:
ln -s /share/apps/simulations/nwchem-6.1.1/default.nwchemrc ~/.nwchemrc

For simple calculations, the defaults are the best to go with. Only in very special circumstances should one specify memory in one's input file

The orte parallel environment is set to schedule jobs of size 4 or 8 such that all processes stay on a single node. This improves inter-process messaging as they don't incure a network overhead.

Here is an example job script:


#!/bin/bash
#$ -N nwchemJob
#$ -cwd
#$ -pe orte 4
#$ -S /bin/bash
source /share/apps/simulations/nwchem-6.1.1/nwchem-source.sh
/share/apps/openmpi/1.4.3-gcc-4.6.1/bin/mpirun -n $NSLOTS /share/apps/simulations/nwchem-6.1.1/bin/nwchem inputfile.nw
exit 0


Tips for improved performance: If a job runs slow due to disk caching, try the following:

Local Scratch Space. Note that nwchem creates local files to cache calculations and these files can grow to significant size, eating into your storage quota. Each node does have a local scratch space at /state/partition1/tmp where one can create a folder for use. Examine this example:


#!/bin/bash
#$ -N nwchemJob
#$ -cwd
#$ -pe orte 4
#$ -S /bin/bash
source /share/apps/simulations/nwchem-6.1.1/nwchem-source.sh
# Create scratch directory
mkdir /state/partition1/tmp/myjob
/share/apps/openmpi/1.4.3-gcc-4.6.1/bin/mpirun -n $NSLOTS /share/apps/simulations/nwchem-6.1.1/bin/nwchem inputfile.nw
# Cleanup scratch directory
rm -rf /state/partition1/tmp/myjob
exit 0


Inside input file, include the following as a top level directive:

scratch_dir /state/partition1/tmp/myjob

Please note the cleanup section of the job script. It is important to cleanup those files as they would take up space in the tmp drive from everyone else.

Running completely within scratch space: It may be desirable to run completely within the scratch drive if running off of the network filesystem is an issue for your job. This is an advanced use case as it requires you to be aware of what files need to be copied back. Would be best to debug your script with a short job before moving on to larger jobs to ensure correct working of your script. Here is an example(an example only, you'll need to test for your particular situation):


#!/bin/bash
#$ -N nwchemJob
#$ -cwd
#$ -pe orte 4
#$ -S /bin/bash
source /share/apps/simulations/nwchem-6.1.1/nwchem-source.sh
# Set directories
export OWD = `pwd`
export SCR = /state/partition1/tmp/myjob
# Create scratch directory
mkdir $SCR
cp inputfile.nw $SCR
cd $SCR
# Run nwchem
/share/apps/openmpi/1.4.3-gcc-4.6.1/bin/mpirun -n $NSLOTS /share/apps/simulations/nwchem-6.1.1/bin/nwchem inputfile.nw
# Cleanup
# Copy back important files
cp -a fileName $OWD
# Remove scratch directory
rm -rf $SCR
exit 0


Advanced tips: If you use the following modules: SCF, DFT, MP2, CCSD:

For larger jobs that do not fit within a single node, the use of disk caching becomes more complicated. It is possible to use multiple scratch disks on multiple nodes, but would require some advanced scripting to modify your input file before running. More information on this advanced usage of the scratch_dir directive can be found here.

Its possible to force nwchem to either cache only in memory or to calculate needed integrals on the fly with the following directives:


scf
semidirect memsize 200000000 filesize 0 # this uses 1.6 GB of memory and no disk for caching integrals
end

dft
direct # all integrals are computed on the fly
end

mp2
scratchdisk 1024 # use 1 GB of disk per process for intermediates
end

ccsd
nodisk # most integrals are computed on-the-fly
end


Large Job Allocation: Currently jobs are restricted to only 8 slots or less, so that all slots will be on a single node. This minimizes network overhead for inter-process communication. In the case you need more slots then can fit in a node, this is a special case. Understand that more slots does not mean better performance, usually only in extreme cases where the input is a very very large system. In that case, please contact CBI staff so that we may work with you.

More information here: nwchem-sw.org

http://www.cbi.utsa.edu/nwchem