Bridges User Guide
Last update: November 2, 2017

System Overview

Compute Nodes

Bridges comprises login nodes and three types of computational nodes:

  • Regular Shared Memory (RSM) with 128GB RAM each. This includes GPU nodes.
  • Large Shared Memory (LSM) with 3TB RAM each
  • Extreme Shared Memory (ESM) with 12TB RAM each

Currently, Bridges computational nodes supply 0.8946 Pf/s and 144 TiB RAM.

Auxiliary Nodes

In addition, Bridges contains database, web server, data transfer and login nodes. Current Bridges resources include:

  • 752 RSM nodes: HPE Apollo 2000s, with 2 Intel Xeon E5-2695 v3 CPUs (14 cores per CPU), 128GB RAM and 8TB on-node storage
  • 16 RSM GPU nodes: HPE Apollo 2000s, each with 2 NVIDIA K80 GPUs, 2 Intel Xeon E5-2695 v3 CPUs (14 cores per CPU), 128GB RAM, and 8TB on-node storage
  • 32 RSM GPU nodes: HPE Apollo 2000s, each with 2 NVIDIA P100 GPUs, 2 Intel Xeon E5-2683 v4 CPUs (16 cores per CPU), 128GB RAM, and 8TB on-node storage
  • 8 LSM nodes: HPE ProLiant DL580s, each with 4 Intel Xeon E7-8860 v3 CPUs (16 cores per CPU), 3TB RAM, and 16TB on-node storage
  • 34 LSM nodes: HPE ProLiant DL580s, each with 4 Intel Xeon E7-8870 v4 CPUs (20 cores per CPU), 3TB RAM, and 16TB on-node storage
  • 2 ESM nodes: HPE Integrity Superdome Xs, each with 16 Intel Xeon E7-8880 v3 CPUs (18 cores per CPU), 12TB RAM, and 64TB on-node storage
  • 2 ESM nodes: HPE Integrity Superdome Xs, each with 16 Intel Xeon E7-8880 v4 CPUs (22 cores per CPU), 12TB RAM, and 64TB on-node storage
  • Database, web server, data transfer, and login nodes: HPE ProLiant DL360s and HPE ProLiant DL380s, each with 2 Intel Xeon E5-2695 v3 CPUs (14 cores per CPU) and 128GB RAM. Database nodes have SSDs or HDDs. ** Details: Bridges login, database, web server and data transfer nodes contain 2 Intel Xeon E5-2695 v3 CPUs; each CPU has 14 cores, 2.3 GHz base frequency, 3.3 GHz max turbo frequency, 35MB LLC; each node also holds 128GB RAM, DDR4 @ 2133 MHz; database nodes have SSDs or additional HDDs.

System Access

Before the first time you connect to Bridges, you must create your PSC password. Depending on your preferences, you may want to change your login shell once you are logged in.

If you have questions at any time, you can send email to bridges@psc.edu.

Your PSC password

If you do not already have an active PSC account, you must create a PSC password (also called a PSC Kerberos password) before you can connect to Bridges. Your PSC password is the same on all PSC systems, so if you have an active account on another PSC system, you do not need to reset it before connecting to Bridges.

Your PSC password is separate from your XSEDE Portal password. Resetting one password does not change the other password.

Changing your password

There are two ways to change or reset your PSC password:

When you change your PSC password, whether you do it via the online utility or via the kpasswd command on one PSC system, you change it on all PSC systems.

Connect to Bridges

When you connect to Bridges, you are connecting to one of its login nodes. The login nodes are used for managing files, submitting batch jobs and launching interactive sessions. They are not suited for production computing.

See the Running Jobs section of this User Guide for information on production computing on Bridges.

There are several methods you can use to connect to Bridges.

  • ssh, using either XSEDE or PSC credentials. If you are registered with XSEDE for DUO Multi-Factor Authentication (MFA), you can use this security feature in connecting to Bridges.
    See the XSEDE instructions to set up DUO for MFA.
  • gsissh, if you have the Globus toolkit installed
  • XSEDE Single Sign On Hub, including using Multi-Factor authentication if you are an XSEDE user

SSH

You can use an SSH client from your local machine to connect to Bridges using either your PSC or XSEDE credentials.

SSH is a program that enables secure logins over an unsecure network. It encrypts the data passing both ways so that if it is intercepted it cannot be read.

SSH is client-server software, which means that both the user's local computer and the remote computer must have it installed. SSH server software is installed on all the PSC machines. You must install SSH client software on your local machine.

Read more about SSH in the Getting Started with HPC document.

Once you have an SSH client installed, you can use either your PSC credentials or XSEDE credentials (optionally with DUO MFA) to connect to Bridges. Note that you must have created your PSC password before you can use SSH to connect to Bridges.

Use ssh to connect to Bridges using XSEDE credentials and (optionally) DUO MFA

  1. Using your SSH client, connect to hostname "bridges.psc.xsede.org" or "bridges.psc.edu" using port 2222.
    Either hostname will connect you to Bridges, but you must specify port 2222.
  2. Enter your XSEDE username and password when prompted.
  3. (Optional) If you are registered with XSEDE DUO, you will receive a prompt on your phone. Once you have answered it, you will be logged in.

Use ssh to connect to Bridges using PSC credentials

  1. Using your SSH client, connect to hostname "bridges.psc.xsede.org" or "bridges.psc.edu" using the default port (22).
    Either hostname will connect you to Bridges. You do not have to specify the port.
  2. Enter your PSC username and password when prompted.

Read more about using SSH to connect to PSC systems.

gsissh

If you have installed the Globus toolkit you can use gsissh to connect to Bridges. GsiSSH is a version of SSH which uses certificate authentication. Use the "myproxy-logon" command to get a suitable certificate. The Globus toolkit includes a man page for myproxy-logon. See more information on the Globus Toolkit website.

Public-private keys

You can also use the public-private key pair method to connect to Bridges. To do so you must first fill out our form for this type of access.

XSEDE Single Sign On Hub

XSEDE users can use their XSEDE usernames and passwords in the XSEDE User Portal Single Sign On Login Hub (SSO Hub) to access bridges.psc.xsede.org or bridges.psc.edu.

You must use DUO Multi-Factor Authentication in the SSO Hub.

See the XSEDE instructions to set up DUO for Multi-Factor Authentication.

Change your default shell

The "change_shell" command allows you to change your default shell. This command is only available on the login nodes.

To see which shells are available, type:

login1$ change_shell -l

To change your default shell, type:

login1$ change_shell newshell

where newshell is one of the choices output by the "change_shell -l" command. You must use the entire path output by change_shell -l, e.g. /usr/psc/shells/bash. You must log out and back in again for the new shell to take effect.

Account Administration

Charging

Until June 1, 2017, allocations for Bridges were given for "Bridges regular" or "Bridges large" and "Bridges regular" included use of GPU nodes. After that date, the GPU nodes are allocated and charged separately.

  • Bridges regular: The RSM nodes are allocated as "Bridges regular". Prior to June 1, 2017, this included Bridges GPU nodes. Service Units are defined in terms of compute resources:

    1 SU = 1 core-hour

  • Bridges large: The LSM and ESM nodes are allocated as "Bridges large". Service Units are defined in terms of memory requested:

    1 SU = 1 TB-hour

  • Bridges GPU

    Until June 1, 2017, Bridges GPU nodes were part of "Bridges regular" allocations. They were charged at the same rate as RSM nodes. As of June 1, 2017, Bridges GPU nodes are allocated and charged for separately from the RSM nodes.

    Bridges contains two kinds of NVIDIA Tesla GPU nodes: K80s and P100s. Because of the difference in node performance, charges are different for each node type.

    • K80 node charges

    The K80 nodes hold 4 GPU units each, which can be allocated separately. Service units (SUs) are defined in terms of GPU-hours:

    1 GPU-hour = 1 SU

    Note that the use of an entire K80 GPU node for one hour would be charged 4 SUs.

    • P100 node charges

    The P100 nodes hold 2 GPU units each, which can be allocated separately. Service units (SUs) are defined in terms of GPU-hours:

    1 GPU-hour = 2.5 SUs

    Note that the use of an entire P100 node for one hour would be charged 5 SUs.

Transferring Regular Memory SUs to GPU SUs

You can transfer some of your Regular Memory allocation to a GPU allocation. To do this, submit a transfer request through the XSEDE User Portal. See instructions for submitting a transfer request.

Managing multiple grants

If you have more than one grant, be sure to charge your usage to the correct one. Usage is tracked by group name.

  • Find your group names

    To find your group names, use the "id" command.

      id -Gn
    

    will list all the groups you belong to.

  • Find your current group

      id -gn
    

    will list the group you associated with your current session.

  • Change your default group

    Your primary group is charged with all usage by default. To change your primary group, the group to which your SLURM jobs are charged by default, use the "change_primary_group" command. Type:

      change_primary_group -l
    

    to see all your groups. Then type:

      change_primary_group groupname
    

    to set groupname as your default group.

Charging to a different group

Batch jobs and interactive sessions are charged to your primary group by default. To charge your usage to a different group, you must specify the appropriate group with the -A groupname option to the SLURM "sbatch" command. See the Running Jobs section of this Guide for more information on batch jobs, interactive sessions and SLURM.

Tracking your usage

The "projects" command will help you keep tabs on your usage. It shows grant information, including usage and the scratch directory associated with the grant.

Type:

projects

For more detailed accounting data you can use the Grant Management System (authorization required).

File Spaces

There are several distinct file spaces available on Bridges, each serving a different function.

  • Home ($HOME), your home directory on Bridges
  • pylon2, persistent file storage
  • pylon5, temporary file storage. Pylon5 is replacing pylon1.
  • Node-local storage ($LOCAL), scratch storage in the local memory associated with a running job
  • Memory storage ($RAMDISK), scratch storage on the local disk associated with a running job

File expiration

Three months after your grant expires all of your Bridges files associated with that grant will be deleted, no matter which file space they are in. You will be able to login during this 3-month period to transfer files, but you will not be able to run jobs or create new files.

Home ($HOME)

This is your Bridges home directory. It is the usual location for your batch scripts, source code and parameter files. Its path is /home/username, where username is your PSC userid. You can refer to your home directory with the environment variable $HOME. Your home directory is visible to all of the Bridges nodes.

Your home directory is backed up daily, although it is still a good idea to store copies of your important files in another location, such as the pylon2 file system or on a local file system at your site. If you need to recover a home directory file from backup send email to remarks@psc.edu. The process of recovery will take 3 to 4 days.

$Home quota

Your home directory has a 10GB quota. You can check your home directory usage using the "quota" command or the command "du -sh". To improve the access speed to your home directory files you should stay as far below your home directory quota as you can.

Grant expiration

Three months after a grant expires, the files in your home directory associated with that grant will be deleted.

pylon2 ($PROJECT)

The pylon2 file system is a persistent file system. You should use it for long-term storage of your files and not for working space for running jobs. The pylon2 file system is shared across all of Bridges nodes.

pylon2 directories

The path of your pylon2 home directory is /pylon2/groupname/username, where groupname is the name for the PSC group associated with your grant. The id command can be used to find your group name.

The command "id -Gn" will list all the groups you belong to.

The command "id -gn" will list the group associated with your current session.

The environment variable $PROJECT is defined at the pylon2 home directory for the group you are currently working under. If you have more than one grant, you will have a pylon2 home directory for each grant. Be sure to use the appropriate directory when working with multiple grants.

The definition of $PROJECT will change automatically when you change your group with the newgrp command, so that it is always pointing to the correct place for the group that your usage is being charged to.

The pylon2 file system is not backed up. You should therefore store copies of your important pylon2 files in another location.

pylon2 quota

Your usage quota for each of your grants is the Pylon storage allocation you received when your proposal was approved. Files stored under a grant in both pylon5 and pylon2 count towards this storage allocation. If your total use in pylon5 and pylon2 exceeds this quota your access to the partitions on Bridges will be shut off until you are under quota.

Use the "du -sh" or "projects" command to check your total pylon usage (pylon5 plus pylon2). You can also check your usage on the XSEDE User Portal.

If you have multiple grants, it is very important that you store your files in the correct pylon2 directory.

Grant expiration

Three months after a grant expires, the files in the pylon2 directories associated with that grant will be deleted.

pylon5 ($SCRATCH)

On March 7, 2017, an upgraded scratch file system named pylon5 was made available on Bridges to replace pylon1. Be aware that any job scripts which specifically reference /pylon1 must be edited to reference /pylon5 instead. The pylon1 file system was decommissioned on April 4, 2017.

The pylon5 file system is temporary storage, to be used as working space for your running jobs. It provides fast, temporary file space for data read or written by running jobs. I/O to pylon5 is much faster IO than your home directory.

The pylon5 file system is shared across all of the Bridges nodes. It is available on Bridges compute nodes as $SCRATCH.

Because it is temporary storage, files on pylon5:

  • Are wiped after 30 days
  • Are not backed up

pylon5 directories

The path of your pylon5 home directory is /pylon5/groupname/username, where groupname is the name for the PSC group associated with your grant. Use the "id" command to find your group name.

The command "id -Gn" will list all the groups you belong to.

The command "id -gn" will list the group associated with your current session.

If you have more than one grant, you will have a pylon5 directory for each grant. Be sure to use the appropriate directory when working with multiple grants.

pylon5 wiper

The pylon5 file system is not a persistent file system. Files are wiped after 30 days. It is also not backed up. Be sure to move copies of your important files to another location as soon as you can after you create them.

If you have a compelling reason that your files should not be wiped after 30 days, email bridges@psc.edu and request an exemption. Be sure to explain the need for the exemption and supply a deadline after which the files can be safely wiped.

pylon5 quota

Your usage quota for each of your grants is the Pylon storage allocation you received when your proposal was approved. Files stored under a grant in both pylon5 and pylon2 count together towards this storage allocation. If your total use in pylon5 and pylon2 exceeds this quota your access to pylon5 and pylon2 will be shut off until you are under quota.

Use the "du -sh" or "projects" command to check your total pylon usage (pylon5 plus pylon2). You can also check your usage on the XSEDE User Portal.

If you have multiple grants, it is very important that you store your files in the correct pylon5 directory.

Grant expiration

Three months after a grant expires, the files in any pylon5 directories associated with that grant will be deleted, although the wiper will probably have deleted them already.

Node-local ($LOCAL)

Each Bridges node has a local file system attached to it. This local file system is only visible to the node to which it is attached. The local file system provides fast access to local storage

This file space is available on all nodes as $LOCAL.

$LOCAL is only available when your job is running, and can only be used as working space for a running job. Once your job finishes your local files are inaccessible and deleted. To use local space, copy files to $LOCAL at the beginning of your job and back out to a persistent file space before your job ends.

If a node crashes all the $LOCAL files are lost. Therefore, you should checkpoint your $LOCAL files by copying them to pylon5 during long runs.

Multi-node jobs

If you are running a multi-node job the variable "$LOCAL" points to the local file space on your node that is running your rank 0 process.

You can use the "srun" command to copy files between $LOCAL on the nodes in a multi-node job. See the MPI job script in the Running Jobs section of this User Guide for details.

$LOCAL size

The maximum amount of local space varies by node type. The RSM (128GB) nodes have a maximum of 3.7TB. The LSM (3TB) nodes have a maximum of 14TB and the ESM (12TB) nodes have a maximum of 49TB.

To check on your local file space usage type:

du -sh

There is no charge for the use of $LOCAL.

Memory files ($RAMDISK)

You can also use the memory allocated for your job for IO rather than using disk space. This will offer the fastest IO on Bridges.

In a running job the environment variable $RAMDISK will refer to the memory associated with the nodes in use.

The amount of memory space available to you depends on the size of the memory on the nodes and the number of nodes you are using. You can only perform IO to the memory of nodes assigned to your job.

If you do not use all of the cores on a node, you are allocated memory in proportion to the number of cores you are using. Note that you cannot use 100% of a node's memory for IO; some is needed for program and data usage.

$RAMDISK is only available to you while your job is running, and can only be used as working space for a running job. Once your job ends this space is inaccessible. To use memory files, copy files to $RAMDISK at the beginning of your job and back out to a permanent space before your job ends. If your job terminates abnormally your memory files are lost.

Within your job you can cd to $RAMDISK, copy files to and from it, and use it to open files. Use the command du -sh to see how much space you are using.

If you are running a multi-node job the $RAMDISK variable points to the memory space on your node that is running your rank 0 process.

Transferring Files

Overview

There are a variety of file transfer methods available for Bridges:

If you are required to use Two-factor Authentication to access Bridges filesystems, see Two-factor Authentication for instructions on file transfer.

Paths for Bridges file spaces

For all file transfer methods other than cp, you must always use the full path for your Bridges files. The start of the full paths for your Bridges directories are:

Home directory /home/username
Pylon2 directory /pylon2/groupname/username
Pylon5 directory /pylon5/groupname/username

The command id -Gn will show all of your valid groupnames. You have a pylon2 and pylon5 directory for each grant you have.

Transfers into your Bridges home directory

Your Bridges home directory quota is 10 Gbytes, so large files cannot be stored there; they should be copied into one of your pylon file spaces instead. Exceeding your home directory quota will prevent you from writing more data into your home directory and will adversely impact other operations you might want to perform.

Globus

Globus can be used for any file transfer to Bridges. It tracks the progress of the transfer and retries when there is a failure; this makes it especially useful for transfers involving large files or many files. This includes transfers between the pylon2 and pylon5 filesystems and file transfers between the Data Supercell and pylon2 or pylon5.

To use Globus to transfer files you must authenticate either via a Globus account or with InCommon credentials. You can set up a Globus account at the Globus site.

Once you have the proper authentication you can initiate file transfers from the Globus site. A Globus transfer requires a Globus endpoint, a file path and a file name for both the source and destination. The endpoints for Bridges are:

  • psc#bridges-xsede if you are using a Globus account for authentication
  • psc#bridges-cilogon if you are using InCommon for authentication

You must always specify a full path for the Bridges file systems. See Paths for Bridges file spaces for details.

Globus-url-copy

The globus-url-copy command can be used if you have access to Globus client software. Both the globus-url-copy and myproxy-logon commands are available on Bridges, and can be used for file transfers internal to the PSC.

To use globus-url-copy you must have a current user proxy certificate. The command grid-proxy-info will tell you if you have current user proxy certificate and if so, what the remaining life of your certificate is.

Use the myproxy-logon command to get a valid user proxy certificate if any one of these applies:

  • you get an error from the grid-proxy-info command
  • you do not have a current user proxy certificate
  • the remaining life of your certificate is not sufficient for your planned file transfer

When prompted for your MyProxy passphrase enter your XSEDE Portal password.

To use globus-url-copy for transfers to a machine, you must know the Grid FTP server address. The Grid FTP server address for Bridges is

gsiftp://gridftp.bridges.psc.edu

The use of globus-url-copy always requires full paths. See Paths for Bridges file spaces for details.

SCP

To use scp for a file transfer, you must specify a source and destination for your transfer. The format for either source or destination is

username@machine-name:path/filename

For transfers involving Bridges, username is your PSC username. The machine-name should be given as data.bridges.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using scp involving Bridges. Using it prevents file transfers from disrupting interactive use on Bridges login nodes. File transfers using scp must specify full paths for Bridges file systems. See Paths for Bridges file spaces for details.

SFTP

To use sftp, first connect to the remote machine:

sftp username@machine-name

When Bridges is the remote machine, use your PSC userid as username. The Bridges machine-name should be specified as data.bridges.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using sftp involving Bridges. Using it prevents file transfers from disrupting interactive use on Bridges login nodes.

You will be prompted for your password on the remote machine. If Bridges is the remote machine enter your PSC password.

You can then enter sftp subcommands, like put to copy a file from the local system to the remote system, or get to copy a file from the remote system to the local system.

To copy files into Bridges you must either cd to the proper directory or use full pathnames in your file transfer commands. See Paths for Bridges file spaces for details.

Two-factor Authentication

If you are required to use two-factor authentication (TFA) to access Bridges filesystems, you must enroll in XSEDE DUO. Once that is complete, use scp or sftp to transfer files to/from Bridges.

TFA users must use port 2222 and XSEDE Portal usernames and passwords. The machine name for these transfers is data.bridges.psc.edu.

In the examples below, myfile is the local filename, XSEDE-username is your XSEDE Portal username and /path/to/file is the full path to the file on a Bridges filesystem. Note that -P ( capital P) is necessary.

scp

Transfer a file from a local machine to Bridges:

scp -P 2222 myfile XSEDE-username@data.bridges.psc.edu:/path/to/file

Transfer a file from Bridges to a local machine:

scp -P 2222 XSEDE-username@data.bridges.psc.edu:/path/to/file myfile

sftp

Use sftp interactively:

sftp -P 2222 XSEDE-username@data.bridges.psc.edu

Then use the put command to copy a file from the local machine to Bridges, or the get command to transfer a file from Bridges to the local machine.

Graphical SSH client

If you are using a graphical SSH client, configure it to connect to data.bridges.psc.edu on port 2222/TCP. Login using your XSEDE Portal username and password.

Transfer rates

PSC maintains a Web page at http://speedpage.psc.edu that lists average data transfer rates between all XSEDE resources. If your data transfer rates are lower than these average rates or you believe that your file transfer performance is subpar, send email to bridges@psc.edu. We will examine approaches for improving your file transfer performance.

Programming Environment

Bridges provides a rich programming environment for the development of applications.

C, C++ and Fortran

Intel, Gnu and PGI compilers for C, C++ and Fortan are available on Bridges. The compilers are:

  C C++ Fortran
Intel icc icpc ifort
Gnu gcc g++ gfortran
PGI pgcc pgc++ pgfortran

The Intel and Gnu compilers are loaded for you automatically.

To run the PGI compilers you must first issue the command

module load pgi

There are man pages for each of the compilers.

OpenMP programming

To compile OpenMP programs you must add an option to your compile command:

Intel -qopenmp for example: icc -qopenmp myprog.c
Gnu -fopenmp for example: gcc -fopenmp myprog.c
PGI -mp for example: pgcc -mp myprog.c

MPI programming

Three types of MPI are supported on Bridges: MVAPICH2, OpenMPI and Intel MPI.

There are two steps to compile an MPI program:

  1. Load the correct module for the compiler and MPI type you want to use, unless you are using Intel MPI. The Intel MPI module is loaded for you on login.
  2. Issue the appropriate MPI wrapper command to compile your program

The three MPI types may perform differently on different problems or in different programming environments. If you are having trouble with one type of MPI, please try using another type. Contact bridges@psc.edu for more help.

INTEL COMPILERS

To use the Intel compilers with Load this module Compile C with this command Compile C++ with this command Compile Fortran with this command
Intel MPI none, this is loaded by default mpiicc mpiicpc mpiifort
OpenMPI mpi/intel_openmpi mpicc mpicxx mpifort
MVAPICH2 mpi/intel_mvapich mpicc code.c -lifcore mpicxx code.cpp -lifcore mpifort code.f90 -lifcore

For proper Intel MPI behavior, you must set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT to 0. Otherwise the mpirun task placement settings you give will be ignored. For example:

BASH

export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0

CSH:

setenv I_MPI_JOB_RESPECT_PROCESS_PLACEMENT 0

GNU COMPILERS

To use the Gnu compilers with Load this module Compile C with this command Compile C++ with this command Compile Fortran with this command
Intel MPI none, this is loaded by default mpicc mpicxx mpifort
OpenMPI mpi/gcc_openmpi
MVAPICH2 mpi/gcc_mvapich

PGI COMPILERS

To use the PGI compilers with Load this module Compile C with this command Compile C++ with this command Compile Fortran with this command
OpenMPI mpi/pgi_openmpi mpicc mpicxx mpifort
MVAPICH2 mpi/pgi_mvapich

Other languages

Other languages, including Java, Python, R, and MATLAB, are available. See the software section for information.

Debugging and performance analysis

DDT

DDT is a debugging tool for C, C++ and Fortran 90 threaded and parallel codes. It is client-server software. Install the client on your local machine and then you can access the GUI on Bridges to debug your code.

See the PSC DDT page for more information.

VTune

VTune is a performance analysis tool from Intel for serial, multithreaded and MPI applications. Install the client on your local machine and then you can access the GUI on Bridges. See the PSC VTune page for more information.

Software available on Bridges

The Module package

The environment management package Module is essential for running software on PSC systems. Be sure to check if there is a module for the software you want to use by typing module avail software-name

The module help command lists any additional modules that must also be loaded. Note that in some cases the order in which these additional modules are loaded matters.

View the module package documentation.

You may request that additional software be installed by sending mail to remarks@psc.edu.

PSC provides an up-to-date list of available software.

Running Jobs

The SLURM scheduler (Simple Linux Utility for Resource Management) manages and allocates all of the Bridges compute nodes. All of your production computing must be done on compute nodes. You cannot use Bridges login nodes to do your work on.

Several partitions have been set up in SLURM to allocate resources efficiently. Partitions can be considered job queues. Different partitions control different types of Bridges resources; they are configured by the type of node and other job requirements. You will choose the appropriate partition to run your jobs in based on the resources you need.

Regardless of which partition you use, you can work on Bridges in either interactive mode - where you type commands and receive output back to your screen as the commands complete - or batch mode - where you first create a batch (or job) script which contains the commands to be run, then submit the job to be run as soon as resources are available.

Partitions

Each SLURM partition manages a subset of Bridges resources. Each partition allocates resources to both interactive sessions and batch jobs that request resources from it. There are five partitions organized by the type of resource they control:

  • RM, for jobs that will run on Bridges RSM (128GB) nodes
  • RM-shared, for jobs that will run on the RSM nodes, but share a node with other jobs
  • GPU, for jobs that will run on Bridges GPU nodes
  • GPU-shared, for jobs that will run on Bridges GPU nodes, but share a node with other jobs
  • LM, for jobs that will run on Bridges LSM and ESM (3TB and 12TB) nodes

All the partitions use FIFO scheduling. If the top job in the partition will not fit on the machine, SLURM will skip that job and try to schedule the next job in the partition.

Use your allocation wisely: To make the most of your allocation, use the shared partitions whenever possible. Jobs in the RM and GPU partitions are charged for the use of all cores on a node. Jobs in the RM-shared and GPU-shared partitions share nodes, and are only charged for the cores they are allocated. The RM partition is the default for the sbatch command, while RM-shared is the default for the interact command. The interact and sbatch commands are discussed below.

Know which partitions are open to you: Your Bridges allocations determine which partitions you can submit jobs to. For example, to run jobs in the GPU partition, you must have a "Bridges GPU" allocation. If you have more than one grant, be sure to use the correct one when submitting jobs.

This table summarizes the resources available and limits on Bridges partitions. More information on each partition follows.

Partition Node type Nodes shared? Node default Node max Core default Core max GPU default GPU max Walltime default Walltime max Memory
RM 128GB 28 cores 8TB on-node storage No 1 168 If your research needs more than 168 nodes, contact bridges@psc.edu to make special arrangements. 28/node 28/node N/A N/A 30 min 48 hrs 128GB/node
RM-shared 128GB 28 cores 8TB on-node storage Yes 1 1 1 28 N/A N/A 30 min 48 hrs 4.5GB/node
GPU P100 nodes 2 GPUs 2 16-core CPUs 8 TB on-node storage No 1 8 There is a limit of 16 GPUs per job. Because there are 2 GPUs on each P100 node, you can request at most 8 nodes. 32/node 32/node 2 per node 2 per node 30 min 48 hrs 128GB/node
GPU K80 nodes 128GB 4 GPUs 2 14-core CPUS 8TB on-node storage No 1 4 There is a limit of 16 GPUs per job. Because there are 4 GPUs on each K80 node, you can request at most 4 nodes. 28/node 28/node 4 per node 4 per node 30 min 48 hrs 128GB/node
GPU-shared P100 nodes 2 GPUs 2 16-core CPUs 8 TB on-node storage Yes 1 1 16/GPU 16/GPU No default 2 30 min 48 hrs 7GB/GPU
GPU-shared K80 nodes 128GB 4 GPUs 2 14-core CPUS 8TB on-node storage Yes 1 1 7/GPU 7/GPU No default 4 30 min 48 hrs 7GB/GPU
LM LSM nodes: 3TB RAM 16TB on-node storage
ESM nodes: 12TB RAM 64TB on-node storage
Yes 1 42 for 3 TB nodes
4 for 12 TB nodes
Jobs in LM are allocated 1 core/48GB of memory requested N/A N/A 30 min 14 days 12000 GB

RM partition

When submitting a job to the RM partition, you must specify the number of nodes, the number of cores and the walltime that it needs.

Jobs in the RM partition do not share nodes, so jobs are allocated all the cores associated with the nodes assigned to them. Your job will be charged for all the cores associated with your assigned nodes. However, although RM jobs can use more than one node, the memory space of all the nodes is not an integrated memory space. The cores within a node access a shared memory space. Cores in different nodes do not.

Internode communication performance of jobs in the RM partition is best when using 42 or fewer nodes.

RM Parameters
Default walltime 30 minutes
Max walltime 48 hours
Default # nodes 1

RM-shared partition

Jobs in the RM-shared partition will share nodes, but not cores. By using less than 28 cores your job will be charged less. It could also start running sooner.

When submitting a job to the RM-shared partition, you must specify the number of cores, the number of nodes, and the walltime it needs. The number of nodes requested must always be 1.

Jobs are assigned memory in proportion to their number of requested cores. You get the fraction of the node's total memory in proportion to the fraction of cores you requested. If your job exceeds this amount of memory it will be killed.

RM-shared parameters
Default walltime 30 minutes
Max walltime 48 hours
Number of nodes Always 1
Default # cores 1
Max # cores 28

LM partition

Jobs in the LM partition always share nodes. They never span nodes.

When submitting a job to the LM partition, you must specify the amount of memory in GB and the walltime that it needs.

Any value up to 12000GB can be requested.

SLURM will place a job on a 3TB or a 12TB node based on the memory request.

Jobs asking for 3000GB or less will run on a 3TB node, unless there are none available and there is a 12TB node available; then the job will run on a 12TB node.

The number of cores assigned to jobs in the LM partition is proportional to the amount of memory requested. For every 48 GB of memory you get 1 core.

LM parameters
Default walltime 30 minutes
Max walltime 96 hours
Default memory There is no default value
Max memory 12000 GB

GPU Partition

Jobs in the GPU partition use Bridges GPU nodes. Note that Bridges has 2 types of GPU nodes: K80s and P100s. See the System Configuration section of this User Guide for the details of each type.

Jobs in the GPU partition do not share nodes, so jobs are allocated all the cores associated with the nodes assigned to them and all of the GPUs. Your job will be charged for all the cores associated with your assigned nodes.

However, the memory space across nodes is not integrated. The cores within a node access a shared memory space, but cores in different nodes do not.

When submitting a job to the GPU partition, you must specify the number of GPUs.

You should also specify:

  • the type of node you want, K80or P100, with the --gres=type option to the interact or sbatch commands. K80 is the default if no type is specified. See the sbatch command options below for more details.
  • the number of nodes
  • the walltime limit

For information on requesting resources and submitting a job to the RM partition see the section below on the interact or the sbatch commands.

GPU parameters
Default walltime 30 minutes
Max walltime 48 hours
Default # nodes 1
Max # nodes 16

GPU-Shared Partition

Jobs in the GPU-shared partition run on Bridges GPU nodes. Note that Bridges has 2 types of GPU nodes: K80s and P100s. See the System Configuration section of this User Guide for the details of each type.

Jobs in the GPU-shared partition share nodes, but not cores. By sharing nodes your job will be charged less. It could also start running sooner.

You will always run on (part of) one node in the GPU-shared partition.

Your jobs will be allocated memory in proportion to the number of requested GPUs. You get the fraction of the node's total memory in proportion to the fraction of GPUs you requested. If your job exceeds this amount of memory it will be killed.

When submitting a job to the GPU-shared partition, you must specify the number of GPUs.

You should also specify:

  • the type of node you want, K80or P100, with the --gres=type option to the interact or sbatch commands. K80 is the default if no type is specified. See the sbatch command options below for more details.
  • the walltime limit

For information on requesting resources and submitting a job to the RM partition see the section below on the interact or the sbatch commands.

GPU-shared parameters
Default walltime 30 minutes
Max walltime 48 hours
Number of nodes Always 1
Default # of GPUs No default
Max # of GPUs 4

Interactive sessions

You must be allocated the use of one or more Bridges compute nodes by SLURM to work interactively on Bridges. You cannot use the Bridges login nodes for your work.

You can run an interactive session in any of the SLURM partitions. You will need to specify which partition you want, so that the proper resources are allocated for your use.

Resources are set aside for interactive use. If those resources are all in use, your request will wait until the resources you need are available. Using a shared partition (RM-shared, GPU-shared) will probably allow your job to start sooner.

The interact command

To start an interactive session, use the command interact. The format is

interact -options

The simplest interact command is

interact

This command will start an interactive job using the defaults for interact, which are:

  • Partition: RM-shared
  • Cores: 1
  • Time limit: 60 minutes

The simplest interact command to start a GPU job is

interact -gpu

This command will start an interactive job on a P100 node in the GPU-shared partition with 1 GPU and for 60 minutes.

Once the interact command returns with a command prompt you can enter your commands. The shell will be your default shell. When you are finished with your job type CTRL-D.

Note

  • Be sure to charge your job to the correct group if you have more than one grant. See the -A option below to change the charging group for an interact job. Information on how to determine your valid groups and change your default group is in the Account adminstration section of the Bridges User Guide.
  • You are charged for your resource usage from the prompt appears until you type CTRL-D, so be sure to type CTRL-D as soon as you are done.
  • The maximum time you can request is 8 hours. Inactive interact jobs are logged out after 30 minutes of idle time.

Options for interact

If you want to run in a different partition, use more than one core or set a different time limit, you will need to use options to the interact command.

The available options are:

Option Description Default value
-p partition Partition requested RM-shared
-t HH:MM:SS Walltime requested (maximum 8 hours) 60:00 (1 hour)
-N n Number of nodes requested 1
--egress Note the "--" for this option Allows your compute nodes to communicate with sites external to Bridges. N/A
-A groupname Group to charge the job to %a(href=account:multiple) Find your default group Your default group
-R reservation-name Reservation name, if you have one
Use of -R does not automatically set any other interact options. You still need to specify the other options (partition, walltime, number of nodes) to override the defaults for the interact command. If your reservation is not assigned to your default account, then you will need to use the -A option when you issue your interact command.
No default
--mem=nGB Note the "--" for this option Amount of memory requested in GB. This option should only be used for the LM partition. No default
--gres=gpu:type:n Note the "--" for this option type is either p100 or k80; default is k80
n is the number of GPUs (1-4 when type=k80; 1-2 when type=p100)
No default
-gpu Runs your job on 1 P100 GPU in the GPU-shared partition No default
--ntasks-per-node Note the "--" for this option Number of cores to allocate per node 1
-h Help, lists all the available command options

Sample interact commands

Run in the RM-shared partition using 4 cores

interact --ntasks-per-node=4

Run in the LM partition and request 2TB of memory

interact -p LM --mem=2000GB

Run in the GPU-shared partition. This command requests 2 GPUs. The job will be allocated 14 cores (7 per GPU).

interact -p GPU-shared --gres=gpu:p100:2

If you want more complex control over your interactive job you can use the srun command instead of the interact command.

See the srun man page.

Batch jobs

A batch script is a file that consists of SBATCH directives, executable commands and comments.

SBATCH directives specify your resource requests and other job options in your batch script. You can also specify resource requests and options on the sbatch command line. Any options on the command line take precedence over those given in the batch script. The SBATCH directives must start in column 1 (that is, be the first text on a line, with no leading spaces) with '#SBATCH'.

Comments begin with a '#' character.

The first line of any batch script must indicate the shell to use for your batch job.

Sample batch scripts

Some sample scripts are given here. Note that:

Each script uses the bash shell, indicated by the first line #!/bin/bash. Some Unix commands will differ if you use another shell.

For username and groupname you must substitute your username and your appropriate group.

Sample batch script for OpenMP job

#!/bin/bash
#SBATCH -N 1
#SBATCH -p RM
#SBATCH --ntasks-per-node 28
#SBATCH -t 5:00:00
# echo commands to stdout 
set -x

# move to working directory
cd $SCRATCH
# copy input file from your pylon2 directory
#   to the working directory
cp $PROJECT/input.data .

# run OpenMP program
export OMP_NUM_THREADS=28
./myopenmp

# copy output file to persistent space
cp output.data $PROJECT

Notes

  • The --ntasks-per-node option indicates that you will use all 28 cores.

  • For username and groupname you must substitute your username and your appropriate group.

Sample batch script for MPI job

#!/bin/bash
#SBATCH -p RM
#SBATCH -t 5:00:00
#SBATCH -N 2
#SBATCH --ntasks-per-node 28

# echo commands to stdout
set -x

# move to working directory
cd $SCRATCH

# set variable so that task placement works as expected
export  I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0

# copy input files to LOCAL file storage
srun -N $SLURM_NNODES --ntasks-per-node=1 \
    sh -c 'cp $PROJECT/input.${SLURM_PROCID} $LOCAL'

# run MPI program
mpirun -np $SLURM_NTASKS ./mympi

# copy output files to persistent space
srun -N $SLURM_NNODES --ntasks-per-node=1 \ 
    sh -c 'cp $LOCAL/output.* $PROJECT'

Notes

  • The variable $SLURM_NTASKS gives the total number of cores requested in a job. In this example the value of $SLURM_NTASKS will be 56 because the -N option requested 2 nodes and the --ntasks-per-node option requested all 28 cores on each node.

  • The export command sets I_MPI_JOB_RESPECT_PROCESS_PLACEMENT so that your task placement settings are effective. Otherwise, the SLURM defaults are in effect.

  • The srun commands are used to copy files between pylon2 and the $LOCAL file systems on each of your nodes.

    • The first srun command assumes you have two files named input.0 and input.1 in your pylon2 file space. It will copy input.0 and input.1 to, respectively, the $LOCAL file systems on the first and second nodes allocated to your job.
    • The second srun command will copy files named output.* back from your $LOCAL file systems to your pylon2 file space before your job ends. In this command * functions as the usual Unix wildcard.
  • For username and groupname you must substitute your username and your appropriate group.

Sample batch script for Hybrid OpenMP/MPI job

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=14
#SBATCH --time=00:10:00
#SBATCH --job-name=hybrid
cd $SLURM_SUBMIT_DIR

mpiifort -xHOST -O3 -qopenmp -mt_mpi hello_hybrid.f90 -o hello_hybrid.exe

# set variable so task placement works as expected
export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0

mpirun -print-rank-map -n $SLURM_NTASKS -genv \
 OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK -genv I_MPI_PIN_DOMAIN=omp \
 ./hello_hybrid.exe

Notes

  • This example asks for 2 nodes, 4 MPI tasks and 14 OpenMP threads per MPI task.

  • The export command sets I_MPI_JOB_RESPECT_PROCESS_PLACEMENT so that your task placement settings are effective. Otherwise, the SLURM defaults are in effect.

Sample batch script for RM-shared partition

#!/bin/bash
#SBATCH -N 1
#SBATCH -p RM-shared
#SBATCH -t 5:00:00
#SBATCH --ntasks-per-node 2

# echo commands to stdout
set -x

# move to working directory
cd $SCRATCH

# copy input file from your pylon2 directory
#   to the working directory
cp $PROJECT/input.data .

# run OpenMP program
export OMP_NUM_THREADS 2
./myopenmp

# copy output file to persistent space
cp output.data $PROJECT

Notes

  • When using the RM-shared partition the number of nodes requested with the "-N" option must always be 1. The --ntasks-per-node option indicates how many cores you want.

  • For username and groupname you must substitute your username and your appropriate group.

Sample batch script for GPU partition

#!/bin/bash
#SBATCH -N 2
#SBATCH -p GPU
#SBATCH --ntasks-per-node 28
#SBATCH -t 5:00:00
#SBATCH --gres=gpu:p100:2

# echo commands to stdout
set -x

# move to working directory
cd $SCRATCH

# copy input to working directory
cp $PROJECT/input.data .

# run GPU program
./mygpu

# copy output file to persistent storage
cp output.data $PROJECT

Notes

  • The option --gres-gpu indicates the type and number of GPUs you want.

  • For username and groupname you must substitute your username and your appropriate group.

Sample batch script for GPU-shared partition

#!/bin/bash
#SBATCH -N 1
#SBATCH -p GPU-shared
#SBATCH --ntasks-per-node 7
#SBATCH --gres=gpu:p100:1
#SBATCH -t 5:00:00

# echo commands to stdout
set -x

# move to working directory
cd $SCRATCH

# copy input file to working directory
cp $PROJECT/input.data .

# run GPU program
./mygpu

# copy output file to persistent storage
cp output.data $PROJECT

Notes

  • The option --gres-gpu indicates the type and number of GPUs you want.

  • For username and groupname you must substitute your username and your appropriate group.

Sample batch script for job array job

#!/bin/bash
#SBATCH -t 05:00:00
#SBATCH -p RM-shared
#SBATCH -N 1
#SBATCH --ntasks-per-node 5
#SBATCH --array=1-5

set -x

./myexecutable $SLURM_ARRAY_TASK_ID

Note

  • The above job will generate five jobs that will each run on a separate core on the same node. The value of the variable SLURM_ARRAY_TASK_ID is the core number, which, in this example, will range from 1 to 5. Good candidates for job array jobs are jobs that can use only this core index to determine the different processing path for each job. For more information about job array jobs see the sbatch man page and the online SLURM documentation.

Sample batch script for bundling single-core jobs

#!/bin/bash
#SBATCH -N 1
#SBATCH -p RM-shared
#SBATCH -t 05:00:00
#SBATCH --ntasks-per-node 14

echo SLURM NTASKS: $SLURM_NTASK
i=0
while [ $i -lt $SLURM_NTASKS ]
do
     numactl -C +$i ./run.sh &
     let i=i+1
done

wait # IMPORTANT: wait for all to finish or get killed

Note

  • Bundling or packing multiple jobs in a single job can improve your turnaround and improve the performance of the SLURM scheduler.

Sample batch script for bundling multi-core jobs

#!/bin/bash
#SBATCH -N 1
#SBATCH -p RM-shared
#SBATCH -t 05:00:00
#SBATCH --ntasks-per-node 14
#SBATCH --cpus-per-task 2

echo SLURM NTASKS: $SLURM_NTASKS
i=0
while [ $i -lt $SLURM_NTASKS ]
do
     numactl -C +$i ./run.sh &
     let i=i+1
done
wait # IMPORTANT: wait for all to finish or get killed

Note

  • Bundling or packing multiple jobs in a single job can improve your turnaround and improve the performance of the SLURM scheduler.

sbatch command

To submit a batch job, use the sbatch command. The format is

sbatch -options batch-script

The options to sbatch can either be in your batch script or on your sbatch command line. Options in the command line override those in the batch script.

Notes

  • Be sure to charge your job to the correct group if you have more than one grant. See the -A option below to change the charging group for an interact job. Information on how to determine your valid groups and change your default group is in the Account adminstration section of the Bridges User Guide.

  • In some cases, the options for sbatch differ from the options for interact or srun.

Examples of the sbatch command

RM partition

An example of a sbatch command to submit a job to the RM partition is

sbatch -p RM -t 5:00:00 -N 1 myscript.job

where:

  • -p indicates the intended partition
  • -t is the walltime requested in the format HH:MM:SS
  • -N is the number of nodes requested
  • myscript.job is the name of your batch script
LM partition

Jobs submitted to the LM partition must request the amount of memory they need rather than the number of cores. Each core on the 3TB and 12TB nodes is associated with a fixed amount of memory, so the amount of memory you request determines the number of cores assigned to your job. The environment variable SLURM_NTASKS tells you the number of cores assigned to your job. Since there is no default memory value you must always include the --mem option for the LM partition.

A sample sbatch command for the LM partition is:

sbatch -p LM - t 10:00:00 --mem 2000GB myscript.job

where:

  • -p indicates the intended partition (LM)
  • -t is the walltime requested in the format HH:MM:SS
  • --mem is the amount of memory requested
  • myscript.job is the name of your batch script

Jobs in the LM partition do share nodes. They cannot span nodes. Your memory space for an LM job is an integrated, shared memory space.

Useful sbatch options

For more information about these options and other useful sbatch options see the sbatch man page.

Option Description
-p partition Partition requested. Defaults to the RM partition
-t HH:MM:SS Walltime requested in HH:MM:SS
-N n Number of nodes requested
-A groupname Group to charge the job to; if not specified, your default group is charged; [find your default group](https://psc.edu/component/content/article?id=2129&Itemid=889#group-name)
--res reservation-name Use the reservation that has been set up for you. Use of i`--res` does not automatically set any other options. You still need to specify the other options (partition, walltime, number of nodes) that you would in any sbatch command. Note the "--" for this option. If your reservation is not assigned to your default account then you will need to use the `-A` option to sbatch to specify the account.
--mem nGB Memory in GB. Note the "--" for this option. This option should only be used for the LM partition.
-C EGRESS Allows your compute nodes to communicate with sites external to Bridges.
--gres=gpu:type:n Specifies the type and number of GPUs requested. 'type' is either p100 or k80. The default is k80.
'n' is the number of requested GPUs. Valid choices are 1-4, when type is k80 and 1-2 when type is p100.
Note the "--" for this option.
--ntasks-per-node=n Request n cores be allocated per node. Note the "--" for this option.
--mail-type=type Send email when job events occur, where type can be BEGIN, END, FAIL or ALL.
--mail-user=user User to send email to as specified by -mail-type. Default is the user who submits the job.
-d=dependency-list Set up dependencies between jobs, where dependency-list can be:
after:job_id[:jobid...]
This job can begin execution after the specified jobs have begun execution.
afterany:job_id[:jobid...]
This job can begin execution after the specified jobs have terminated.
aftercorr:job_id[:jobid...]
A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully (ran to completion with an exit code of zero).
afternotok:job_id[:jobid...]
This job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc).
afterok:job_id[:jobid...]
This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero).
singleton
This job can begin execution after any previously launched jobs sharing the same job name and user have terminated.
--no-requeue Specifies that your job will be not be requeued under any circumstances. If your job is running on a node that fails it will not be restarted. Note the "--" for this option.
--time-min=HH:MM:SS Specifies a minimum walltime for your job in HH:MM:SS format.
SLURM considers the walltime requested when deciding which job to start next. Free slots on the machine are defined by the number of nodes and how long those nodes are free until they will be needed by another job. By specifying a minimum walltime you allow the scheduler to reduce your walltime request to your specified minimum time when deciding whether to schedule your job. This could allow your job to start sooner.

If you use this option your actual walltime assignment can vary between your minimum time and the time you specified with the -t option. If your job hits its actual walltime limit, it will be killed. When you use this option you should checkpoint your job frequently to save the results obtained to that point.
--switches=1
--switches=1@HH:MM:SSHH:MM:SS
Requests that the nodes your job runs on all be on one switch, which is a hardware grouping of 42 nodes. If you are asking for more than 1 and fewer than 42 nodes, your job will run more efficiently if it runs on one switch. Normally switches are shared across jobs, so using the switches option means your job may wait longer in the queue before it starts.
The optional time parameter gives a maximum time that your job will wait for a switch to be available. If it has waited this maximum time, the request for your job to be run on a switch will be cancelled.
-C=constraints Specifies features which the nodes allocated to this job must have. Some examples are:

-C LM
Ensures that a job in the LM partition uses only the 3TB nodes. This option is required for any jobs in the LM partition which use /pylon5.
-C PH1Ensures that the job will run on LM nodes which have 16 cores and 48GB/core
-C PH2
Ensures that the job will run on LM nodes which have 20 cores and 38.5GB/core

Multiple constraints can be specified with AND, OR, etc. For example, -C "LM&PH2" constrains the nodes to 3TB nodes with 20 cores and 38.5GB/core. See the [sbatch man page](https://slurm.schedmd.com/sbatch.html) for further details.
-h Help, lists all the available command options

Other SLURM commands

sinfo command

The sinfo command displays information about the state of Bridges nodes. The nodes can have several states:

alloc Allocated to a job
down Down
drain Not available for scheduling
idle Free
resv Reserved
squeue command

The squeue command displays information about the jobs in the partitions. Some useful options are:

-j jobid Displays the information for the specified jobid
-u username Restricts information to jobs belonging to the specified username
-p partition Restricts information to the specified partition
-l (long) Displays information including: time requested, time used, number of requested nodes, the nodes on which a job is running, job state and the reason why a job is waiting to run.

See the squeue man page for more options, for a discussion of the codes for job state and for why a job is waiting to run.

scancel command

The scancel command is used to kill a job in a partition, whether it is running or still waiting to run. Specify the jobid for the job you want to kill. For example,

scancel 12345

kills job # 12345.

sacct command

The sacct command can be used to display detailed information about jobs. It is especially useful in investigating why one of your jobs failed. The general format of the command is

sacct -X -j XXXXXX -S MMDDYY --format parameter1,parameter2, ...

For 'XXXXXX' substitute the jobid of the job you are investigating. The date given for the -S option is the date at which sacct begins searching for information about your job.

The --format option determines what information to display about a job. Useful paramaters are JobID, Partition, Account, ExitCode, State, Start, End, Elapsed, NodeList, NNodes, MaxRSS and AllocCPUs. The ExitCode and State parameters are especially useful in determining why a job failed. NNodes displays how many nodes your job used, while AllocCPUs displays how many cores your job used. MaxRSS displays how much memory your job used. The commas between the parameters in the --format option cannot be followed by spaces.

See the man page for sacct for more information.

srun command
Monitoring memory usage

It can be useful to find the memory usage of your jobs. For example, you may want to find out if memory usage was a reason a job failed or if you need to move your job from the 3-TB nodes to the 12-TB nodes.

There are two cases. You can determine a job's memory usage if it is still running or when it it has finished.

If your job is still running, which you can determine with the squeue command, you can issue the command:

srun --jobid=XXXXXX top -b -n 1 | grep userid

For 'XXXXXX' substitute the jobid of your job. For 'userid' substitute your userid. The RES field in the top output shows the actual amount of memory used by a process. The top man page can be used to identify the fields of top output.

The other method to use for a running job is to issue the command

sstat -j XXXXXX.batch --format=JobID,MaxRss

For 'XXXXXX' substitute your jobid.

If your job has finished there are two methods to use to find out its memory usage. If you are checking within a day or two after your job has finished you can issue the command:

sacct -j XXXXXX --format=JobID,MaxRss

If this command no longer shows a value for MaxRss then you must use the command:

job_info XXXXXX | grep max_rss

Again, substitute your jobid for 'XXXXXX' in both of these commands.

There are man pages for top, srun, sstat and sacct if you need more information.

More help

There are man pages for all the SLURM commands. SLURM also has extensive online documentation.

Using Bridges GPU nodes

The NVIDIA Tesla K80 and P100 GPUs on Bridges provide substantial, complementary computational power for deep learning, simulations and other applications.

A standard NVIDIA accelerator environment is installed on Bridges GPU nodes. If you have programmed using GPUs before, you should find this familiar. Please contact bridges@psc.edu for more help.

GPU Nodes

There are two types of GPU nodes on Bridges: 16 nodes with NVIDIA K80 GPUs and 32 nodes with NVIDIA P100 GPUs.

K80 Nodes

The 16 K80 GPU nodes each contain 2 NVIDIA K80 GPU cards, and each card contains two GPUs that can be individually scheduled. Ideally, the GPUs are shared in a single application to ensure that the expected amount of on-board memory is available and that the GPUs are used to their maximum capacity. This makes the K80 GPU nodes optimal for applications that scale effectively to 2, 4 or more GPUs. Some examples are GROMACS, NAMD and VASP. Applications using a multiple of 4 K80 GPUs will maximize system throughput.

The nodes are HPE Apollo 2000s, each with 2 Intel Xeon E5-2695 v3 CPUs (14 cores per CPU) and 128GB RAM.

Each Bridges Phase 1 RSM GPU node contains:

  • 2 NVIDIA K80 GPUs
  • 2 Intel Xeon E5-2695 v3 CPUs, each with
    • 14 cores, 2.3 GHz base frequency, 3.3 GHz max turbo frequency
    • 35MB LLC
  • 128GB RAM, DDR4 @ 2133 MHz

P100 Nodes

The 32 P100 GPU nodes contain 2 NVIDIA P100 GPU cards, and each card holds one very powerful GPU, optimally suited for single-GPU applications that require maximum acceleration. The most prominent example of this is deep learning training using frameworks that do not use multiple GPUs.

The nodes are HPE Apollo 2000s, each with 2 Intel Xeon E5-2683 v4 CPUs (16 cores per CPU) and 128GB RAM.

Each Bridges Phase 2 RSM GPU node contains:

  • 2 NVIDIA P100 GPUs
  • 2 Intel Xeon E5-2683 v4 CPUs, each with
    • 16 cores, 2.1 GHz base frequency, 3.0 GHz max turbo frequency
    • 40MB cache
  • 128GB RAM, DDR4 @ 2400 MHz

File Systems

The /home, pylon2 and pylon5 file systems are available on all of these nodes. See the File Spaces section of the User Guide for more information on these file systems.

Compiling and Running Jobs

Use the GPU queue, either in batch or interactively, to compile your code and run your jobs. See the Running Jobs section of the User Guide for more information on Bridges queues and how to run jobs.

CUDA

To use CUDA, first you must load the CUDA module. To see all versions of CUDA that are available, type:

module avail cuda

Then choose the version that you need and load the module for it.

module load cuda

loads the default CUDA. To load a different version, use the full module name.

module load cuda/8.0

CUDA 8 codes should run on both types of Bridges GPU nodes with no issues. CUDA 7 should only be used on the K80 GPUs (Phase 1). Performance may suffer with CUDA 7 on the P100 nodes (Phase 2).

OpenACC

Our primary GPU programming environment is OpenACC.

The PGI compilers are available on all GPU nodes. To set up the appropriate environment for the PGI compilers, use the module command:

module load pgi

Read more about the module command at PSC

If you will be using these compilers often, it will be useful to add this command to your shell initialization script.

There are many options available with these compilers. See the online man pages

  • man pgf90
  • man pgcc
  • man pgCC

for detailed information.

You may find these basic OpenACC options a good place to start:

pgcc -acc yourcode.c
pgf90 -acc yourcode.f90

P100 node users should add the -ta=tesla,cuda8.0 option to the compile command, for example:

pgcc -acc -ta=tesla,cuda8.0 yourcode.c

Adding the -Minfo=accel flag to the compile command (whether pgf90, pgcc or pgCC) will provide useful feedback regarding compiler errors or success with your OpenACC commands.

pgf90 -acc yourcode.f90 -Minfo=accel

Hybrid MPI/GPU Jobs

To run a hybrid MPI/GPU job, use the following commands for compiling your program:

module load cuda
module load mpi/pgi_openmpi
mpicc -acc yourcode.c

When you execute your program you must first issue the above two module load commands.

Profiling and Debugging

The environment variables PGI_ACC_TIME, PGI_ACC_NOTIFY and PGI_ACC_DEBUG can provide profiling and debugging information for your job. Specific commands depend on the shell you are using.

A Unix shell is a command-line interpreter that provides a traditional user interface for Unix and Unix-like systems.

The two major shell types are the Bourne shell and the C shell. Each type has its own commands and syntax.

The default shell on Bridges is bash, a Bourne-type shell. Other shells, including some C-type shells, are available for you to use if you prefer.

Send email to bridges@psc.edu to request that additional CUDA-oriented debugging tools be installed.

Performance profiling

Enable runtime GPU performance profiling by setting the PGI_ACC_TIME environment variable.

export PGI_ACC_TIME=1 [bash]
setenv PGI_ACC_TIME 1 [csh]

Debugging

Basic debugging can be accomplished by setting the PGI_ACC_NOTIFY environment variable. For more detail, set PGI_ACC_NOTIFY to 3.

export PGI_ACC_NOTIFY=1 [bash]
setenv PGI_ACC_NOTIFY 1 [csh]

A further level of debugging is available with the PGI_ACC_DEBUG environment variable.

export PGI_ACC_DEBUG=1 [bash]
setenv PGI_ACC_DEBUG 1 [csh]

Hadoop and Spark

If you want to run Hadoop or Spark on Bridges, you should note that when you apply for your account.

/home

The /home file system, which contains your home directory, is available on all Bridges Hadoop nodes.

HDFS

The Hadoop filesystem, HDFS, is available from all Hadoop nodes. There is no explicit quota for the HDFS, but it uses your $SCRATCH disk space and will be deleted when your job has ended. Files must reside in HDFS to be used in Hadoop jobs. Putting files into HDFS requires these steps:

  1. Transfer the files to the namenode with scp or sftp
  2. Format them for ingestion into HDFS
  3. Use the hadoop fs -put command to copy the files into HDFS. This command distributes your data files across the cluster's datanodes.

The hadoop fs command should be in your command path by default.

Documentation for the "hadoop fs" command lists other options. These options can be used to list your files in HDFS, delete HDFS files, copy files out of HDFS and other file operations.

To request the installation of data ingestion tools on the Hadoop cluster send email to bridges@psc.edu.

Accessing the Hadoop/Spark cluster

To start using Hadoop and Spark with Yarn and HDFS on Bridges, connect to the login node and issue the following commands:

interact -N 3 # you will need to wait until resources are allocated to you before continuing
module load hadoop
start-hadoop.sh

Your cluster will be set up and you'll be able to run hadoop and spark jobs. The cluster requires a minimum of three nodes (-N 3). Larger jobs may require a reservation. Please contact bridges@psc.edu if you would like to use more than 8 nodes or run for longer than 8 hours.

Please note that when your job ends, your HDFS will be unavailable so be sure to retrieve any data you need before your job finishes.

Web interfaces are currently not available for interactive jobs but can be made available for reservations.

Spark

The Spark data framework is available on Bridges. Spark, built on the HDFS filesystem, extends the Hadoop MapReduce paradigm in several directions. It supports a wider variety of workflows than MapReduce. Most importantly, it allows you to process some or all of your data in memory if you choose. This enables very fast parallel processing of your data.

Python, Java and Scala are available for Spark applications. The pyspark interpreter is especially effective for interactive, exploratory tasks in Spark. To use Spark you must first load your data into Spark's highly efficient file structure called Resilient Distributed Dataset (RDD).

Extensive online documentation is available at the Spark website. If you have questions about or encounter problems using Spark, send email to bridges@psc.edu.

Spark example using Yarn

Here is an example command to run a Spark job using yarn. This example calculates pi using 10 iterations.

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster $SPARK_HOME/examples/jars/spark-examples_2.11-2.1.0.jar 10

To view the full output:

yarn logs -applicationId yarnapplicationId

where yarnapplicationId is the yarn applicationId assigned by the cluster.

A Simple Hadoop Example

This section demonstrates how to run a MapReduce Java program on the Hadoop cluster. This is the standard paradigm for Hadoop jobs. If you want to run jobs using another framework or in other languages besides Java send email to bridges@psc.edu for assistance.

Follow these steps to run a job on the Hadoop cluster. All the commands listed below should be in your command path by default. The variable HADOOP_HOME should be set for you also.

  1. Compile your Java MapReduce program with a command similar to:
    hadoop com.sun.tools.javac.Main WordCount WordCount.java

    where:

    • WordCount is the name of the output directory where you want your class file to be put
    • WordCount.java is the name of your source file
  2. Create a jar file out of your class file with a command similar to:
    jar -cvf WordCount.jar -C WordCount/ .

    where:

    • WordCount.jar is the name of your output jar file
    • WordCount is the name of the directory which contains your class file
  3. Launch your Hadoop job with the "hadoop" command

    Once you have your jar file you can run the hadoop command to launch your Hadoop job. Your hadoop command will be similar to

    hadoop jar WordCount.jar org.myorg.WordCount \/datasets/compleat.txt $MYOUTPUT

    where:

    • Wordcount.jar is the name of your jar file
    • org.myorg.WordCount specifies the folder hierarchy inside your jar file. Substitute the appropriate hierarchy for your jar file.
    • /datasets/compleat.txt is the path to your input file in the HDFS file system. This file must already exist in HDFS.
    • $MYOUTPUT is the path to your output file, which will be saved in the HDFS file system. You must set this variable to the output file path before you issue the hadoop command.

After you issue the hadoop command your job is controlled by the Hadoop scheduler to run on the datanodes. The scheduler is currently a stricty FIFO scheduler. If your job turnaround is not meeting your needs send email to bridges@psc.edu.

When your job finishes, the hadoop command will end and you will be returned to the system prompt.

Other Hadoop Technologies

An entire ecosystem of technologies has grown up around Hadoop, such as HBase and Hive. To request the installation of a different package send email to bridges@psc.edu.

Virtual Machines

A Virtual Machine (VM) is a portion of a physical machine that is partitioned off through software so that it acts as an independent physical machine.

You should indicate that you want a VM when you apply for time on Bridges.

When you have an active Bridges grant, use the VM Request form to request a VM. You will be contacted in one business day about your request.

Why use a VM?

A VM provides you with control over your environment, but you will have access to the computing power, memory capacity and file spaces of Bridges.

Common uses of VMs include hosting database and web servers. These servers can be restricted just to you or you can open them up to outside user communities to share your work. You can also connect your database and web servers and other processing components in a complex workflow.

VMs provide several other benefits. Since the computing power behind the VM is a supercomputer, sufficient resources are available to support multiple users. Since each VM acts like an independent machine, user security is heightened. No outside users can violate the security of your independent VM. However, you can allow other users to access your VM if you choose.

A VM can be customized to meet your requirements. PSC will set up the VM and give you access to your database and web server at a level that matches your requirements.

To discuss whether a VM would be appropriate for your research project, send email to bridges@psc.edu.

Downtime

VMs are affected by system downtime, and will not be available during an outage. Scheduled downtimes are announced in advance.

Data backups

It is your responsiblity to backup any important data to another location outside of the VM. PSC will make infrequent snapshots of VMs for recovery from system failure, but cannot be responsible for managing your data.

Grant expiration

When your grant expires, your VM will be suspended. You have a 3-month grace period to request via email to bridges@psc.edu that it be reactivated so that you can move data from the VM. Three months after your grant expires, the VM will be removed. Please notify bridges@psc.edu if you need help moving your data during the grace period.

Data Collections

Bridges hosts both public and private datasets, providing rapid access for individuals, collaborations and communities with appropriate protections.

Data collections are stored on pylon2, Bridges persistent file system. The space they use counts toward the Bridges storage allocation for the grant hosting them.

If you would like to store a large data collection on Bridges, submit the Community Dataset Request form.

Publicly available datasets

Some data collections are available to anyone with a Bridges account. They include:

Natural Languge Tool Kit Data

NLTK comes with many corpora, toy grammars, trained models, etc. A complete list of the available data is posted at: http://www.nltk.org/nltk_data/

NLTK is available on Bridges at /pylon2/datasets/community/nltk.

Help

To report a problem on Bridges, please email bridges@psc.edu. Please report only one problem per email; it will help us to track and solve any issues more quickly and efficiently.

Be sure to include

  • the JobID
  • the error message you received
  • the date and time the job ran
  • any other pertinent information
  • a screen shot of the error or the output file showing the error, if possible