Palmetto Cluster and deal.II

This guide is provided by Timo Heister.

Access palmetto

Log into login node (replace USERNAME with your clemson id):

ssh USERNAME@user.palmetto.clemson.edu

More info at https://www.palmetto.clemson.edu/palmetto/basic/started/

Interactive running

For compiling and running interactively, you NEED to be on one of the compute nodes:

qsub -I -l select=1:ncpus=24:mpiprocs=24:mem=120gb:interconnect=fdr,walltime=2:00:00

See the user guide for more options (for example "chip_type" or "phase").

The modules I compiled require an e5-2680v3 or newer, meaning phase 12 or newer. This will automatically be the case if you request 24 MPI ranks. For running jobs with more than one MPI rank, I would recommend specifying the "phase" directly.

Every time you log into an interactive node, you need to load the modules I prepared before you can compile or run:

source /home/heister/shared-dealii/modules_9.2.0-v1

Older versions are not available at the moment.

Job submission

Example job file run.pbs (see /home/heister/shared-dealii/run.pbs) needs to be copied into the directory where you want to run. You probably only need to edit:

  1. The number of nodes to run on ("select=X" in line 2)
  2. The total walltime if you need to run longer jobs
  3. Select a phase (12 or newer)
  4. The name of the program to run and optionally parameters (the "cmd=…" line)

Commands for job submission:

  1. Submit a job (needs to be done from the login node and not a compute node!):
qsub run.pbs
  1. List jobs (running and queued) belonging to USERNAME:
qstat -u USERNAME
  1. Cancel job (jobid is the first column in qstat):
qdel JOBID

File systems

The following filesystems are available for you:

  1. /home/USERNAME/ - backed up, slow (do not use for job output)
  2. /scratch1/USERNAME/ - not backed up, deleted every 30 days, parallel filesystem
  3. /scratch2/USERNAME/ - similar to 2 but not suitable for large files and not useful for parallel I/O
  4. $TMPDIR - local to specific node you are running on, wiped after your job finishes (also known as /local_scratch/)

Advice:

  • Keep your source code in /home/
  • Produce output on /scratch1/ and copy things you want to keep to /home/

Misc

  • Check the available hardware using cat /etc/hardware-table (only available on login node)
  • Check what is free using whatsfree (only available on login node)
  • Check information about nodes using pbsnodes -a | less

Created: 2020-12-07 Mon 10:01

Validate