SLURM FAQ

1. Exporting environment variables to a script

At times, you want to use your script as a template with an environment variable and supply that varaible's value in job submission time. This would make it easy to use the same script and submit it n times just changing the environment variable for each submission.

It is possible to do this using the --export option of the sbatch command.

Importantly, if you would like to keep the cluster wide user environment variables(E.g java -> /opt/ghpc/jdk-current/bin/java), then use the syntax as --export=ALL,whatever so that all default environment variables are retained.

Example,

sbatch --export ALL,bam=1_10.bam bam_processor.sh

and you use the variable $bam in your script - bam_processor.sh

2. Representing job dependancies in Slurm

Often times, you have a pipeline of jobs to do. Example, you need to complete job A before starting job B. Specifying such dependancies to Slurm is easy. sbatch command can be used to delare dependencies using the syntax below.

sbatch --dependency=<type:job_id[:job_id][,type:job_id[:job_id]]> ...

Dependency types:

A simple example of job dependency, stating that this job needs to be started only after successful completion of a certain prior job is below.

sbatch --dependency=afterok:11254323 job2.sh

Where, 11254323 is the job ID of the job which needs to be completed successfully prior to this job starting to run.

Using this technique, one can build pipelines of jobs. Bash is the best tool for this task.

#! /bin/bash

# first job - no dependencies
jid1=$(sbatch  --mem=12g job1.sh)

# multiple jobs can depend on a single job
jid2=$(sbatch  --dependency=afterok:$jid1 --mem=20g job2.sh)
jid3=$(sbatch  --dependency=afterok:$jid1 --mem=20g job3.sh)

# a single job can depend on multiple jobs
jid4=$(sbatch  --dependency=afterok:$jid2:$jid3 job4.sh)

3. How do I get email notifications for Slurm jobs?

So, you have an important job that you want to be notified about when it is complete. Follow the simple steps below to send yourself an email from within your job.

mailghpc -s 'subject of my email goes here' -t 'myemailaddress@email.com' 'Body of my email - write whatever text you need here'

You can place such a command above with modified subject, to address, and body message anywhere in your job script to get appropriate notification.

For example, place this command at the beginning of the job script to know when your job has started. Place it at the end of the job script and you can be notified when the job script has finished.

You can also get crafty and compose the subject and body fields of this command within your bash script to get relevant notifications wherever in your job script.