Comet Technical Summary

Comet is a dedicated XSEDE cluster designed by Dell and SDSC delivering ~2.0 petaflops, featuring Intel next-gen processors with AVX2, Mellanox FDR InfiniBand interconnects and Aeon storage.

The standard compute nodes consist of Intel Xeon E5-2680v3 (formerly codenamed Haswell) processors, 128 GB DDR4 DRAM (64 GB per socket), and 320 GB of SSD local scratch memory. The GPU nodes contain four NVIDIA GPUs each. The large memory nodes contain 1.5 TB of DRAM and four Haswell processors each. The network topology is 56 Gbps FDR InfiniBand with rack-level full bisection bandwidth and 4:1 oversubscription cross-rack bandwidth. Comet has 7 petabytes of 200 GB/second performance storage and 6 petabytes of 100 GB/second durable storage. It also has dedicated gateway hosting nodes and a Virtual Machine repository. External connectivity to Internet2 and ESNet is 100 Gbps.

Serving the Long Tail

Comet was designed and is operated on the principle that the majority of computational research is performed at modest scale. Comet also supports science gateways, which are web-based applications that simplify access to HPC resources on behalf of a diverse range of research communities and domains, typically with hundreds to thousands of users. Comet is an NSF-funded system operated by the San Diego Supercomputer Center at UC San Diego. Comet is available through the Extreme Science and Discovery Environment (XSEDE) program.

Logging in to Comet

Comet supports several access methods:

• Single Sign On through the XSEDE User Portal
• Command-line SSH login using SDSC username and XSEDE User Portal Password

To login to Comet from the command line, use the hostname:

comet.sdsc.xsede.org

The following are examples of Secure Shell (ssh) commands that may be used to log in to Comet:

ssh sdscusername@comet.sdsc.xsede.org
ssh -l sdscusername comet.sdsc.xsede.org

Notes and hints

• When you log in to comet.sdsc.edu, you will be assigned one of the four login nodes: comet-ln[1-4].sdsc.edu. These nodes are identical in both architecture and software environment. Users should normally log in through comet.sdsc.xsede.org, but may specify one of the four nodes directly if they see poor performance.
• Please feel free to append your public RSA key to your "~/.ssh/authorized_keys" file to enable access from authorized hosts without having to enter your password. Make sure you have a password on the private key on your local machine. You can use ssh-agent or keychain to avoid repeatedly typing the private key password.

Do NOT use the login nodes for computationally intensive processes. These nodes are meant for compilation, file editing, simple data analysis, and other tasks that use minimal compute resources. All computationally demanding jobs should be submitted and run through the batch queuing system.

Modules

For example, if the Intel module and mvapich2_ib module are loaded and the user compiles with mpif90, the generated code is compiled with the Intel Fortran 90 compiler and linked with the mvapich2_ib MPI libraries.

Several modules that determine the default Comet environment are loaded at login time. These include the MVAPICH implementation of the MPI library and the Intel compilers. We strongly suggest that you use this combination whenever possible to get the best performance.

Table 3. Useful Module Commands

Loading and Unloading Modules

You must remove some modules before loading others. Some modules depend on others, so they may be loaded or unloaded as a consequence of another module command. For example, if intel and mvapich are both loaded, running the command module unload intel will automatically unload mvapich. Subsequently issuing the module load intel command does not automatically reload mvapich.

If you find yourself regularly using a set of module commands, you may want to add these to your configuration files (".bashrc" for bash users, ".cshrc" for C shell users). Complete documentation is available in the module(1) and modulefile(4) manpages.

login1$ man 1 module
login1$ man 4 modulefile

module: command not found

The error message "module: command not found" is sometimes encountered when switching from one shell to another or attempting to run the module command from within a shell script or batch job. The reason that the module command may not be inherited as expected is that it is defined as a function for your login shell. If you encounter this error execute the following from the command line (interactive shells) or add to your shell script (including Slurm batch scripts)

login1$ source /etc/profile.d/modules.sh

Useful Commands

The show_accounts command lists the accounts that you are authorized to use, together with a summary of the used and remaining time.

[user@comet-login1 ~]$ show_accounts
ID name project used available used_by_proj
user project SUs by user SUs available SUs by proj

#SBATCH -A project

Adding Users to an Account

Project PIs and co-PIs can add or remove users from an account. To do this, log in to your XSEDE portal account and go to theAdd User page.

Charging

The charge unit for all SDSC machines, including Comet, is the Service Unit (SU). This corresponds to the use of one compute core for one hour. Keep in mind that your charges are based on the resources that are tied up by your job and don't necessarily reflect how the resources are used. Charges are based on either the number of cores or the fraction of the memory requested, whichever is larger. The minimum charge for any job longer than 10 seconds is 1 SU.

Job Charge Considerations

• A node-exclusive job that runs on a compute node for one hour will be charged 24 SUs (24 cores x 1 hour)
• A serial job in the shared queue that uses less than 5 GB memory and runs for one hour will be charged 1 SU (1 core x 1 hour)
• A P100 gpu/gpu-shared job will be charged a premium of 1.5x; P100 GPUs are substantially faster than the K80, achieving more than twice the performance for some applications. Accordingly, users will incur a 1.5x premium when running on the P100 vs the K80. (Please note that the SUs on the GPU resource are measured in terms of K80 GPU hours).
• A GPU is equivalent to 1/4th of a node which equals 6 cores on the K80 GPUs and 7 cores on the P100 GPUs
• Multicore jobs will scale according to resource utilization
• Each standard compute node has ~128 GB of memory and 24 cores
  • Each standard node core has 5 GB of memory (1/24th of the total memory on a standard compute node)
• Each large memory node has ~1.5 TB of memory and 64 cores
  • Each large memory core has 24 GB of memory (1/64 of total memory on a large memory node)

Compiling

Comet provides the Intel, Portland Group (PGI), and GNU compilers along with multiple MPI implementations (MVAPICH2, MPICH2, OpenMPI). Most applications will achieve the best performance on Comet using the Intel compilers and MVAPICH2 and the majority of libraries installed on Gordon have been built using this combination. Although other compilers and MPI implementations are available, we suggest using these only for compatibility purposes.

All three compilers now support the Advanced Vector Extensions 2 (AVX2). Using AVX2, up to eight floating point operations can be executed per cycle per core, potentially doubling the performance relative to non-AVX2 processors running at the same clock speed. Note that AVX2 support is not enabled by default and compiler flags must be set as described below.

Using the Intel Compilers (Default/Suggested)

The Intel compilers and the MVAPICH2 MPI implementation will be loaded by default. If you have modified your environment, you can reload by executing the following commands at the Linux prompt or placing in your startup file ("~/.cshrc" or "~/.bashrc")

login1$ module purge
login1$ module load gnutools
login1$ module load intel mvapich2_ib

For AVX2 support, compile with the "-xHOST" option. Note that the "-xHOST" option alone does not enable aggressive optimization, so compilation with "-O3" is also suggested. The "-fast" flag invokes "-xHOST", but should be avoided since it also turns on interprocedural optimization ("-ipo"), which may cause problems in some instances.

Intel MKL libraries are available as part of the "intel" modules on Comet. Once this module is loaded, the environment variable MKL_ROOT points to the location of the mkl libraries. TheMKL link advisor can be used to ascertain the link line (change the MKL_ROOT aspect appropriately).

For example to compile a C program statically linking 64 bit scalapack libraries on Comet:

mpicc -o pdpttr.exe pdpttr.c \
    -I$MKL_ROOT/include ${MKL_ROOT}/lib/intel64/libmkl_scalapack_lp64.a \
    -Wl,--start-group ${MKL_ROOT}/lib/intel64/libmkl_intel_lp64.a \
    ${MKL_ROOT}/lib/intel64/libmkl_core.a ${MKL_ROOT}/lib/intel64/libmkl_sequential.a \
    -Wl,--end-group ${MKL_ROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm 

login1$ icc -help
login1$ icpc -help
login1$ ifort -help

Table 4. Intel Compilers

Using the PGI Compilers

The PGI compilers can be loaded by executing the following commands at the Linux prompt or placing in your startup file (~/.cshrc or ~/.bashrc)

login1$ module purge
login1$ module load gnutools
login1$ module load pgi mvapich2_ib

login1$ man pgf90 
login1$ man pgcc
login1$ man pgCC

Table 5. PGI Compilers

Using the GNU Compilers

The GNU compilers can be loaded by executing the following commands at the Linux prompt or placing in your startup files (~/.cshrc or ~/.bashrc)

login1$ module purge
login1$ module load gnutools
login1$ module load gnu openmpi_ib

login1$ man gfortran
login1$ man g++
login1$ man gfortran | gcc | g++

Table 6. GNU Compilers

MVAPICH2-GDR on Comet GPU Nodes

The GPU nodes on Comet have MVAPICH2-GDR available. MVAPICH2-GDR is based on the standard MVAPICH2 software stack. It incorporates designs that take advantage of the new GPUDirect RDMA technology for inter-node data movement on NVIDIA GPUs clusters with Mellanox InfiniBand interconnect. The "mvapich2-gdr" modules are also available on the login nodes for compiling purposes. An example compile and run script is provided in "/share/apps/examples/MVAPICH2GDR".

Notes and hints

• The "mpif90", "mpicc", and "mpicxx" commands are actually wrappers that call the appropriate serial compilers and load the correct MPI libraries. While the same names are used for the Intel, PGI and GNU compilers, keep in mind that these are completely independent scripts.
• If you use the PGI or GNU compilers or switch between compilers for different applications, make sure that you load the appropriate modules before running your executables.
• When building OpenMP applications and moving between different compilers, one of the most common errors is to use the wrong flag to enable handling of OpenMP directives. Note that Intel, PGI, and GNU compilers use the "-openmp", "-mp", and "-fopenmp" flags, respectively.
• Explicitly set the optimization level in your makefiles or compilation scripts. Most well written codes can safely use the highest optimization level ("-O3"), but many compilers set lower default levels (e.g. GNU compilers use the default "-O0", which turns off all optimizations).
• Turn off debugging, profiling, and bounds checking when building executables intended for production runs as these can seriously impact performance. These options are all disabled by default. The flag used for bounds checking is compiler dependent, but the debugging ("-g") and profiling ("-pg") flags tend to be the same for all major compilers.

Running Jobs on Regular Compute Nodes

Comet uses the Simple Linux Utility for Resource Management (SLURM) batch environment. When you run in batch mode, you submit jobs to be run on the compute nodes using the sbatch command as described below. Remember that computationally intensive jobs should be run only on the compute nodes and not the login nodes.

Comet places limits on the number of jobs queued and running on a per group (allocation) and partition basis. Please note that submitting a large number of jobs (especially very short ones) can impact the overall scheduler response for all users. If you are anticipating submitting a lot of jobs, please contact the before you submit them. We can work to check if there are bundling options that make your workflow more efficient and reduce the impact on the scheduler.

The limits for each partition are given in the below table:

Queues on Comet

Table 7. Queues on Comet

Requesting interactive resources using srun

You can request an interactive session using the srun command. The following example will request one node, all 24 cores, in the debug partition for 30 minutes:

[user@comet-ln1]$ srun --partition=debug --pty --nodes=1 --ntasks-per-node=24 -t 00:30:00 --wait=0 --export=ALL /bin/bash

Submitting Jobs Using sbatch

Jobs can be submitted to the SLURM queues using the sbatch command as follows:

[user@comet-ln1]$ sbatch jobscriptfile

Sample Job Scripts

Basic MPI Job

#!/bin/bash
#SBATCH --job-name="hellompi"
#SBATCH --output="hellompi.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --export=ALL
#SBATCH -t 01:30:00

#This job runs with 2 nodes, 24 cores per node for a total of 48 cores.
#ibrun in verbose mode will give binding detail

ibrun -v ../hello_mpi

Basic OpenMP Job

#!/bin/bash
#SBATCH --job-name="hello_openmp"
#SBATCH --output="hello_openmp.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH --export=ALL
#SBATCH -t 01:30:00

#Set the number of openmp threads
export OMP_NUM_THREADS=24

#Run the job using mpirun_rsh
hello_openmp

Hybrid MPI-OpenMP Job

#!/bin/bash
#SBATCH --job-name="hellohybrid"
#SBATCH --output="hellohybrid.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --export=ALL
#SBATCH -t 01:30:00

#This job runs with 2 nodes, 24 cores per node for a total of 48 cores.
#We use 8 MPI tasks and 6 OpenMP threads per MPI task
export OMP_NUM_THREADS=6
ibrun --npernode 4 ./hello_hybrid

Basic mpirun_rsh Job

#!/bin/bash
#SBATCH --job-name="hellompirunrsh"
#SBATCH --output="hellompirunrsh.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --export=ALL
#SBATCH -t 01:30:00

#Generate a hostfile from the slurm node list
export SLURM_NODEFILE=`generate_pbs_nodefile`

#Run the job using mpirun_rsh
mpirun_rsh -hostfile $SLURM_NODEFILE -np 48 ../hello_mpi

Using the Shared Partition

#!/bin/bash
#SBATCH -p shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --mem=40G
#SBATCH -t 01:00:00
#SBATCH -J HPL.8
#SBATCH -o HPL.8.%j.%N.out
#SBATCH -e HPL.8.%j.%N.err
#SBATCH --export=ALL

export MV2_SHOW_CPU_BINDING=1
ibrun -np 8 ./xhpl.exe

The above script will run using 8 cores and 40 GB of memory. Please note that the performance in the shared partition may vary depending on how sensitive your application is to memory locality and the cores you are assigned by the scheduler. It is possible the 8 cores will span two sockets for example.

SLURM No-Requeue Option

SLURM will requeue jobs if there is a node failure. However, in some cases this might be detrimental if files get overwritten. If users wish to avoid automatic requeue, the following line should be added to their script:

#SBATCH --no-requeue

License Scheduling

MATLAB MDCS, IDL, and Abaqus have restricted licenses with limits on the number of cores used at any given time. Please ensure that you add a license request to your run script so that they are not oversubscribed and cause job failures. For example the following Abaqus job requests 24 licenses:

#!/bin/bash
#SBATCH --job-name="abaqus"
#SBATCH --output="abaqus.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=1
#SBATCH --export=ALL
#SBATCH --ntasks-per-node=24
#SBATCH -L abaqus:24
#SBATCH -t 01:00:00
module load abaqus/6.14-1
export EXE=`which abq6141`
$EXE job=s4b input=s4b scratch=/scratch/$USER/$SLURM_JOBID cpus=24 mp_mode=threads memory=120000mb interactive

Example Scripts for Applications

SDSC User Services staff have developed sample run scripts for common applications. They are available in the /share/apps/examples directory on Comet.

Job Dependencies

There are several scenarios (e.g. splitting long running jobs, workflows) where users may require jobs with dependencies on successful completions of other jobs. In such cases, SLURM's "--dependency" option can be used. The syntax is as follows:

[user@comet-ln1 ~]$ sbatch --dependency=afterok:jobid jobscriptfile

Job Monitoring and Management

Users can monitor jobs using the squeue command.

[user@comet-ln1 ~]$ squeue -u user1
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
256556 compute raxml_na user1 R 2:03:57 4 comet-14-[11-14]
256555 compute raxml_na user1 R 2:14:44 4 comet-02-[06-09]

Table 8. squeue options

Help with ibrun

The options and arguments for ibrun are as follows:

Using Globus Endpoints, Data Movers and Mount Points

All of Comet's NFS and Lustre filesystems are acccessible via the Globus endpoint "xsede#comet". The servers also mount Gordon's filesystems, so the mount points are different for each system. The following table shows the mount points on the data mover nodes (that are the backend for "xsede#comet" and "xsede#gordon").

SSD Scratch Space

The compute nodes on Comet have access to fast flash storage. There is 250GB of SSD space available for use on each compute node. The latency to the SSDs is several orders of magnitude lower than that for spinning disk (> 100 microseconds vs. milliseconds) making them ideal for user-level check pointing and applications that need fast random I/O to large scratch files. Users can access the SSDs only during job execution under the following directories local to each compute node:

/scratch/$USER/$SLURM_JOB_ID

Parallel Lustre Filesystems

In addition to the local scratch storage, users will have access to global parallel filesystems on Comet. Overall, Comet has 7 petabytes of 200 GB/second performance storage and 6 petabytes of 100 GB/second durable storage.

Users can now access /oasis/projects from Comet. The two Lustre filesystems available on Comet are:

Lustre Comet scratch filesystem: /oasis/scratch/comet/$USER/temp_project
Lustre NSF projects filesystem: /oasis/projects/nsf

Software Packages

Comet supports the XSEDE core software stack, which includes remote login, remote computation, data movement, science workflow support, and science gateway support toolkits.

Table 9. Software on Comet

Software Package Descriptions