Interactive R or Python or Julia session on cluster node
R
Sometimes, you just need an interactive session of R to get your work done without submitting a cluster batch job. If you run R directly on console[1,2] and use a lot of memory or CPU, you are making the console server slow for other users who just want to submit jobs to the cluster. Additionally your (usually nice) sysadmin will yell at you for causing headache to other users for your convenience. So, the solution is to get an interactive R session on one of the cluster nodes.
As a side benefit, you are now running R on a server dedicated for HPC use that is much faster than console[1,2] and has vast resources than console.
But, how?
It is as easy as running srun --pty R. <<<- This will only get you 1 CPU core, and 10GiB of memory on ghpc_v1 queue with a time limit of 12 hours. It is sufficient for most usecases. If you would need more, read further to learn about how to customise your request.
asampath@console1:[~] > srun --pty R
R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> Sys.info()
sysname release
"Linux" "3.10.0-862.14.4.el7.x86_64"
version nodename
"#1 SMP Wed Sep 26 15:12:11 UTC 2018" "c09b02.ghpc.au.dk"
machine login
"x86_64" "unknown"
user effective_user
"asampath" "asampath"
> q()
Save workspace image? [y/n/c]: n
asampath@console1:[~] >
Psssh, I'm an R expert, and I want more resources than 2 CPUs and 10GB of memory..
Internally your interactive session gets submitted as a job to the Slurm cluster, and giving you resources from the same limits you're already bound to. You can customise the srun command to specify your resource requirments exactly like you would if you were running SBATCH.
For example,
asampath@console1:[~] > srun -N 1 -n 4 --mem=40g -t 24:00:00 -J testing_R_for_science --pty R
R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
>
Where, I requested for 1 node, 4 cpus, 40 GiB of memory, max time of 24 hours, and a label of "testing_R_for_science", and to be dropped into a R session directly.
To understand your request, if you open another SSH session, and use the myst command to see your own jobs, you will see as below.
asampath@console1:[~] > myst
JOBID PARTITION NAME USER ST TIME_LIMIT TIME NODES CPUS MIN_MEMORY NODELIST(REASON)
3180 ghpc_v1 testing_ asampath R 1-00:00:00 0:10 1 4 40G c09b02.ghpc.au.dk
showing that Slurm treated your Interactive session as yet another job, and shows how much resources you reserved for it.
Python and Julia
The same technique used for R in the above section is useful for getting interactive sessions of other languages as well. For example:
asampath@console1:[~] > srun --pty julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: https://docs.julialang.org
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.7.0 (2018-08-08 06:46 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release
|__/ | x86_64-pc-linux-gnu
julia>
asampath@console1:[~] > srun --pty python3
Python 3.7.1 (default, Oct 27 2018, 15:53:24)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
SAS
Interactively running SAS means you are running in "line-mode" according to SAS terminology. Reference: http://support.sas.com/documentation/cdl/en/hostunx/61879/HTML/default/a002460896.htm
Use the following command to get such an interactive session.
srun --pty sas -nodms
Example:
asampath@console1:[~] > srun --pty sas -nodms
NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software 9.4 (TS1M0)
Licensed to CAMPUS - AARHUS UNI, Site 50800154.
NOTE: This session is executing on the Linux 3.10.0-862.el7.x86_64 (LIN X64)
platform.
NOTE: Updated analytical products:
SAS/STAT 12.3 (maintenance)
SAS/ETS 12.3 (maintenance)
SAS/OR 12.3 (maintenance)
SAS/IML 12.3 (maintenance)
SAS/QC 12.3 (maintenance)
NOTE: Additional host information:
Linux LIN X64 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018
x86_64 CentOS Linux release 7.5.1804 (Core)
You are running SAS 9. Some SAS 8 files will be automatically converted
by the V9 engine; others are incompatible. Please see
http://support.sas.com/rnd/migration/planning/platform/64bit.html
PROC MIGRATE will preserve current SAS file attributes and is
recommended for converting all your SAS libraries from any
SAS 8 release to SAS 9. For details and examples, please see
http://support.sas.com/rnd/migration/index.html
This message is contained in the SAS news file, and is presented upon
initialization. Edit the file "news" in the "misc/base" directory to
display site-specific news and information in the program log.
The command line option "-nonews" will prevent this display.
NOTE: SAS initialization used:
real time 0.28 seconds
cpu time 0.02 seconds
1?