Palmetto Cluster and deal.II
This guide is provided by Timo Heister.
Access palmetto
Log into login node (replace USERNAME with your clemson id):
ssh USERNAME@user.palmetto.clemson.edu
More info at https://www.palmetto.clemson.edu/palmetto/basic/started/
Interactive running
For compiling and running interactively, you NEED to be on one of the compute nodes:
qsub -I -l select=1:ncpus=24:mpiprocs=24:mem=120gb:interconnect=fdr,walltime=2:00:00
See the user guide for more options (for example "chip_type" or "phase").
The modules I compiled require an e5-2680v3 or newer, meaning phase 12 or newer. This will automatically be the case if you request 24 MPI ranks. For running jobs with more than one MPI rank, I would recommend specifying the "phase" directly.
Every time you log into an interactive node, you need to load the modules I prepared before you can compile or run:
source /home/heister/shared-dealii/modules_9.2.0-v1
Older versions are not available at the moment.
Job submission
Example job file run.pbs (see /home/heister/shared-dealii/run.pbs) needs to be copied into the directory where you want to run. You probably only need to edit:
- The number of nodes to run on ("select=X" in line 2)
- The total walltime if you need to run longer jobs
- Select a phase (12 or newer)
- The name of the program to run and optionally parameters (the "cmd=…" line)
Commands for job submission:
- Submit a job (needs to be done from the login node and not a compute node!):
qsub run.pbs
- List jobs (running and queued) belonging to USERNAME:
qstat -u USERNAME
- Cancel job (jobid is the first column in qstat):
qdel JOBID
File systems
The following filesystems are available for you:
/home/USERNAME/
- backed up, slow (do not use for job output)/scratch1/USERNAME/
- not backed up, deleted every 30 days, parallel filesystem/scratch2/USERNAME/
- similar to 2 but not suitable for large files and not useful for parallel I/O$TMPDIR
- local to specific node you are running on, wiped after your job finishes (also known as/local_scratch/
)
Advice:
- Keep your source code in
/home/
- Produce output on
/scratch1/
and copy things you want to keep to/home/
Misc
- Check the available hardware using
cat /etc/hardware-table
(only available on login node) - Check what is free using
whatsfree
(only available on login node) - Check information about nodes using
pbsnodes -a | less