Slurm Job Array R Example
R Example Slurm Script
The simplest R script looks like:
Save this line to a new file called
hello.R. This one-liner script can be run with
However, we will run it via the Slurm scheduler on the Yen10 server.
Here is an example slurm script that loads R module and runs the hello world R script.
#!/bin/bash # Example of running R script in a batch mode #SBATCH -J hello #SBATCH -p normal #SBATCH -c 1 # one CPU core #SBATCH -t 10:00 #SBATCH -o hello-%j.out #SBATCH --mail-type=ALL #SBATCH --email@example.com # Load software module load R # Run R script srun Rscript hello.R
Note: to use the scheduler, you prepend
Rscript hello.R with
srun command. Save this slurm script to
Then run it by submitting the job to the slurm scheduler with:
We will take this slurm job script and modify it to run as a job array. Each task in a job array will run the same R script and print ‘Hello!’ and the job array task ID.
We are going to pass a job array task ID as a command line argument to the R script. The R script accepts the command line argument and prints ‘Hello!’ and the task ID passed to it.
Let’s modify our hello world R script to look like:
#!/usr/bin/env Rscript # accept command line arguments and save them in a list called args args = commandArgs(trailingOnly=TRUE) # print task number print(paste0('Hello! I am a task number: ', args) )
Then we modify the slurm file to look like below (save this to
#!/bin/bash # Example of running R script with a job array #SBATCH -J hello #SBATCH -p normal #SBATCH --array=1-10 # how many tasks in the array #SBATCH -c 1 # one CPU core per task #SBATCH -t 10:00 #SBATCH -o hello-%j-%a.out #SBATCH --mail-type=ALL #SBATCH --firstname.lastname@example.org # Load software module load R # Run R script with a command line argument srun Rscript hello-parallel.R $SLURM_ARRAY_TASK_ID
Note that in this case, we specified slurm option
#SBATCH --array=1-10 to run ten independent tasks in parallel.
The maximum job array size is set to 512 on yen10. Each task will generate a unique log file
so we can look at those and see if any of the tasks failed.
Submit Job Array to Scheduler
We can now submit our
hello-parallel.slurm script to the slurm scheduler to run the job array on the Yen10 server.
It will launch all 10 tasks at the same time (some might sit in the queue while others are going to run right away).
To submit, run:
Monitor your jobs with
watch squeue -u USER where
USER is your SUNet ID. Check which job array tasks failed.
Rerun those by setting
--array= only to failed indices.
Connect with us