本文描述Linux集群的基本Slurm用法。
1. 一个简单的Slurm脚本
$ cat slurm-job.sh
#!/usr/bin/env bash
#SBATCH -o slurm.sh.out
#SBATCH -p defq
echo "In the directory: `pwd`"
echo "As the user: `whoami`"
echo "write this is a file" > analysis.output
sleep 60
2. 提交作业
$ module load slurm
$ sbatch slurm-job.sh
Submitted batch job 106
3. 列出作业
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
106 defq slurm-jo rstober R 0:04 1 atom01
4. 获取作业细节
$ scontrol show job 106
JobId=106 Name=slurm-job.sh
UserId=rstober(1001) GroupId=rstober(1001)
Priority=4294901717 Account=(null) QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=00:00:07 TimeLimit=UNLIMITED TimeMin=N/A
SubmitTime=2013-01-26T12:55:02 EligibleTime=2013-01-26T12:55:02
StartTime=2013-01-26T12:55:02 EndTime=Unknown
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=defq AllocNode:Sid=atom-head1:3526
ReqNodeList=(null) ExcNodeList=(null)
NodeList=atom01
BatchHost=atom01
NumNodes=1 NumCPUs=2 CPUs/Task=1 ReqS:C:T=*:*:*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=0 Contiguous=0 Licenses=(null) Network=(null)
Command=/home/rstober/slurm/local/slurm-job.sh
WorkDir=/home/rstober/slurm/local
5. Suspend a job (root only)
# scontrol suspend 135
# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
135 defq simple.s rstober S 0:10 1 atom01
6. Resume a job (root only)
# scontrol resume 135
# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
135 defq simple.s rstober R 0:13 1 atom01
7. Kill a job
用户可以杀死自己的作业,root可以杀死任何作业。
$ scancel 135
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
8. Hold a job
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
139 defq simple rstober PD 0:00 1 (Dependency)
138 defq simple rstober R 0:16 1 atom01
$ scontrol hold 139
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
139 defq simple rstober PD 0:00 1 (JobHeldUser)
138 defq simple rstober R 0:32 1 atom01
9. Release a job
$ scontrol release 139
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
139 defq simple rstober PD 0:00 1 (Dependency)
138 defq simple rstober R 0:46 1 atom01
10. List partitions
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
defq* up infinite 1 down* atom04
defq* up infinite 3 idle atom[01-03]
cloud up infinite 2 down* cnode1,cnodegpu1
cloudtran up infinite 1 idle atom-head1
11. 作业依赖
首先提交一个简单的作业:
#!/usr/bin/env bash
#SBATCH -p defq
#SBATCH -J simple
sleep 60
Submit the job
$ sbatch simple.sh
Submitted batch job 149
现在,我们将提交另一个依赖于先前作业的作业。 有许多方法可以指定依赖条件,但是“singleton ”是最简单的。 Slurm -d singleton 参数告诉Slurm在之前所有具有相同名称的作业完成之前不要调度此作业。
$ sbatch -d singleton simple.sh
Submitted batch job 150
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
150 defq simple rstober PD 0:00 1 (Dependency)
149 defq simple rstober R 0:17 1 atom01
前提作业完成后,将调度从属作业。
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
150 defq simple rstober R 0:31 1 atom01