Batch Jobs

Batch Jobs

 

• commands: qsub, qstat, qdel
  • qsub
  • qstat
  • qdel
• examples
  • serial programme
  • parallel: MPI
  • parallel: OpenMP

 

commands: qsub, qstat, qdel

Within the alibaba cluster, the batch queing system torque is used. torque an open source resource manager providing control over batch jobs running on the compute nodes.

The most important commands are qsub for submitting a job, qstat for monitoring its status, and qdel for deleting a job.

For the description of these and related other commands:

qalter, pbs_alterjob, pbs_statjob, pbs_statque, pbs_statserver, pbs_submit, pbs_job_attributes, pbs_queue_attributes, pbs_server_attributes, pbs_resources_*

see http://www.clusterresources.com/wiki/doku.php?id=torque:torque_wiki or the corresponding man-pages.

 

qsub

The qsub command usually is called with the filename of a script as a parameter. That script holds job parameters as well as the call of the actual programme. Parameters are placed as a comment ("#") in the first lines of the script and start with the command prefix "PBS" followed by the parameter setting, eg

...
# PBS -l walltime=6:10:00
...

to set the maximum amount of real time during which the job can be in the running state. Parameters can also be specified can as command-line arguments. eg

 > qsub -l nodes=12  

to request 12 nodes. Command line arguments take precedence over parameters set in the script.

Important options are:

Option	Description
-N name	job name
-q queue	destination queue
-S shell	path to the shell that interprets the job script
-j oe	join stdout and stderr streams
-o filename	file or directory to which the (joined) output is written
-m a	send an e-mail notification in case of a job abort
-M e-mail	e-mail address to which the notification is sent
-l resource(s)	see below

The important resource parameters are:

Ressource	Format	Description
nodes	number-of-nodes [:ppn= processes-per-node ]	Number and type of nodes to be reserved for exclusive use by the job.
walltime	seconds or [[HH:]MM:]SS	Maximum amount of real time during which the job can be in the running state

At run time the following environment variables are set:

Variablename	Used for
PBS_JOBNAME	user specified jobname
PBS_O_WORKDIR	job's submission directory
PBS_TASKNUM	number of tasks requested
PBS_O_HOME	home directory of submitting user
PBS_O_LOGNAME	name of submitting user
PBS_JOBCOOKIE	job cookie
PBS_NODENUM	node offset number
PBS_O_SHELL	script shell
PBS_O_JOBID	unique pbs job id
PBS_O_HOST	host on which job script is currently running
PBS_QUEUE	job queue
PBS_NODEFILE	file containing line delimited list on nodes allocated to the job
PBS_O_PATH	path variable used to locate executables within job script

 

 

qstat

To monitor submitted jobs, the qstat command is used. Though not all job-information will be presented to a normal user, one can get information like job-ID, name, queing status etc. The output can be given in different formats and verbosity.

To get an overview in table form, type qstat without any argument.

Job id	Name	User	Time Use	S	Queue
<ID>	<Jobname> (given by user)	<Username>	Used CPU time	Status (see below)	<Jobque>

Status can be

C	Job is completed after having run
E	Job is exiting after having run
H	Job is held
Q	job is queued, eligible to run or routed
R	job is running
T	job is being moved to new location
W	job is waiting for its execution time (-a option) to be reached

More detailed information can be requested by using the "-f" option:

> qstat -f [job_id]

For more information, see the man-page or the online man page of qstat at torque.

 

qdel

The qdel command is used to delete a job, which has to be specified by its job-identifier, that is, type

> qdel <job_id>

to delete the job with the id <id>. After submission of the command, a "Delete Job batch request" will be sent to the batch server that owns the job. See the man-page for more information.

 

 

examples

Simple examples are given for serial and parallel batch jobs.

 

serial programme

 

#PBS -N test1
#PBS -j oe
#PBS -o /home/user/test/test1.log
#PBS -l walltime=100
#
set -x
cd /work/user
/home/user/test/a.out
exit

The batch jobs executes /home/user/test/a.out in directory /work/user and writes a log file to /home/user/test/test1.log

 

parallel: MPI

 

#PBS -N test2
#PBS -j oe
#PBS -o /home/user/test/test2.log
#PBS -l nodes=3:ppn=8
#PBS -l walltime=100
#
set -x
cd /work/user
/home/user/test/a.out -np 24
exit

The job executes on 3 nodes using all 8 cores of each node. On has to make sure that the "-l nodes=...:ppa=..." and "-np ..." specifications match. In parallel jobs one should always request 8 cores per node (ie ppn=8). Otherwise one would share nodes with other users what should be avoided.

A special case is ppn=4 (or smaller). In that case one should also specify the interconnect one wants to use. This can be done by adding #PBS -q gbe for gigabit-ethernet or #PBS -q ib for inifiniband. (For ppn larger than 4 the system will automatically use the large nodes and infiniband.)

 

 

parallel: OpenMP

 

#PBS -N test3
#PBS -j oe
#PBS -o /home/user/test/test3.log
#PBS -l nodes=1:ppn=T
#PBS -l walltime=100
#
set -x
cd /work/user
export OMP_NUM_THREADS=T
/home/user/test/a.out
exit

Where T stands for the number of threads. In parallel jobs one should always request T = 8 cores per node (ie ppn=8). Otherwise one would share nodes with other users what should be avoided. As a consequence OMP_NUM_THREADS should be set to 8.

An alterative is requesting 4 threads. Then one should explicitly request small nodes by adding #PBS -q gbe.