http://www.schedmd.com/#index
ub link submit job using SLURM
Slurm Architecture
slurmd daemon running on each node and a central slurmctld daemon running on a management node
SLURM ITEMS
node, compute resource in Slurm
partition, group of nodes in a logical(possibly overlapping) sets
job, allocation of resources assigned to a user for a specified amout of time
job steps, sets of tasks within a job
SLUM COMMANDS
slurm --help / man
sacct, report job/job step accouting information about active or completed jobs
salloc, allocate resources for a job in real time ( usually allocate resource ans spawn a shell, which is then to execute srun to launch parallel taskes)
sattach, attach std I/O Error plus signal capabilities to a currently running job/jobstep, usually can attach/deattach multi-times
sbatch, submit a job script for later execution, usually contian one or more srun commands to launch parallel tasks
sbcast, transfer a file from local disk to local disk on the nodes allocated to one job, usually for shared file system
scancel, cancel a pending or running job
scontrol, administrator tool to view/modify Slurm state, usually need root user
sinfo, reports state partion and nodes managed by Slurm, with variety of filtering, sorting, formatting options
smap, report state information for jobs, partitions, node by Slum graphically
squeue, report state of job(samilar as sinfo), with running jobs in priority order
srun, sumbit job for execution or initiate jobsteps in real time, options e.g. max/min node cout, processor count, specific node to use or not use, specific node characteristics( memory, disk space, centain required features)
strigger, job monitor
sview, GUI to get / update state informaton for jobs, partions, nodes managed by Slurm
SRUN
a simple way: a single commond line e.g. srum -N3 -l ../mpi/example
/* running example on three nodes with output(-l)
a common way: submit a script:
/*
* options in script can be supplied by using a prefix of "#SBATCH" followed by the option at the beginning of the script(before any commands to be excuted in the script), Options supplied on the command line would override any options specified within the script
*/
another way: create a resource allocation and spawn job steps within that allocation
/*
* salloc used to create a resource allocation and typically start a shell within that allocation. one or more job can excute.
Slurm doesn't automactically migrate executable or data files to the nodes allocated to a job. Either the files must exist on local disk or in some global file system(NFS). sbcast can be used to transfer files to local storage on allocated nodes using Slurm's hierarchical communications
*/
CPU Management Steps (重要)
step1: selection of nodes in slurm.conf
NodeName
PartitionName
FastSchedule
SelectType selecc/linear | select/cons_res
SelectTypeParameters CR_CPU | CR_CPU_Memory | CR_Core | CR_Core_Memory |CR_Socket | CR_Socket_Memory
srun/salloc/sbatch command line options ::
-B --extra-node-info <sockets[:cores[:threads]]> , restricts node selection to nodes with a specified layout of sockets, cores and threads
-C --constraint <list> , restrict node selection to nodes with specified attributes
--contiguous N/A , restrict node selection to contiguous nodes
--cores-per-socket <cores> restrict node selection to nodes with at least the specified number of cores per socket
-c, --cpus-per-task <ncpus> , control the number of CPUs allocated per tast
--exclusive N/A , pervent sharing of allocated nodes with other jobs, suballocates CPUs to job steps
-F, --nodefile <node file> File containing a list of specific nodes to be selected for the job(salloc sbatch)
--hint comput_bound | memory_hound | [no] multithread, additional controls on allocation CPU resoures
--minicpus <n> controls the minimum # of CPUs allocated per node
-N, --nodes <minnodes[-maxnodes]>
-n, --ntasks <numbers> number of tasks to be created for the job
--ntasks-per-core <number> maximum number of tasks per allocated core
--ntasks-per-socket <number>
--ntasks-per-node <number>
-O, --overcommit N/A, allows fewer CPUs to be allocated than the number of tasks
-p, --partition <partition_names>, which partition is used for the job
-s, --share N/A, allow sharing of allocated nodes with other jobs
--sockets-per-node <sockets>
--threads-per-core <threads>
-w, --nodelist <host1,host2, ... or filename>, list of specific nodes to be allocated to the job
-x, --exclude <host1, host2, .. filename>, list of specific nodes to be excluded from allocated to the job
-Z --no-allocate N/A
step2: allocation CPUs from the selected nodes in slurm.conf(same as in step1)
step3: distribution of tasks to the selected nodes
each task is distributed to only one node, but more than one task may be distributed to each node.
除非cpu资源巨多,一个节点上分配的任务数取决于所有分配的cpu总数和每个任务需要占用的cpu数目。
in slurm.conf MaxTasksPerNode <number>, max number of tasks that a job step can spawn on a single node
srun/salloc/sbatch options in this step:
--distribution, -m block | cyclic | arbitrary | plane=<options> [:block|cyclic]
--ntasks-per-core <number>
--ntasks-per-socket <number>
--ntasks-per-node <number>
-r, --relative N/A which node is used for a job step
step 4: opt/binding CPU
SLURM distributed and bind each task to a specified subset of the allocated CPUs on the node
举例:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --constraint=CPU-L5520
#SBATCH --partition=debug
#SBATCH --time=00:00:10
#SBATCH --mail-type=END
#SBATCH --mail-user=xx.gmail.com
#SBATCH --output=core8.out
#
echo "SLUM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_CPUS_PER_NODE="$SLURM_JOB_CPUS_PER_NODE
echo "SLURM_CPUS_PER_TASK="$SLURM_CPUS_PER_TASK
echo "SLURM_TASKS_PER_NODE="$SLURM_TASKS_PER_NODE
echo "SLURM_NTASKS="$SLURM_NTASKS