Create or change your PSC password

If you do not already have an active PSC account, you must create a PSC password (also called a PSC Kerberos password) before you can connect to Bridges. Your PSC password is the same on all PSC systems, so if you have an active account on another PSC system, you do not need to reset it before connecting to Bridges.

Your PSC password is separate from your XSEDE User Portal password. Resetting one password does not change the other password.

Setting your initial PSC password

To set your initial PSC password, use the web-based PSC password change utility.

See PSC password policies.

Changing your PSC password

There are two ways to change or reset your PSC password:

• Use the web-based PSC password change utility.
• Use the kpasswd command when logged into a PSC system. Do not use the passwd command.

When you change your PSC password, whether you do it via the online utility or via the kpasswd command on one PSC system, you change it on all PSC systems.

Connect to Bridges

When you connect to Bridges, you are connecting to one of its login nodes. The login nodes are used for managing files, submitting batch jobs and launching interactive sessions. They are not suited for production computing.

See the Running Jobs section of this User Guide for information on production computing on Bridges.

There are several methods you can use to connect to Bridges.

1. You can access Bridges through a web browser by using the OnDemand software. You will still need to understand Bridges' partition structure and the options which specify job limits like time and memory use, but OnDemand provides a more modern, graphical interface to Bridges.

   See the OnDemand section of this User Guide for more information.

2. You can connect to a traditional command line interface by logging in via one of these:

   • ssh, using either XSEDE or PSC credentials. If you are registered with XSEDE for DUO Multi-Factor Authentication (MFA), you can use this security feature in connecting to Bridges.
     See the XSEDE instructions to set up DUO for MFA.
   • XSEDE Single Sign On Hub, including using Multi-Factor authentication if you are an XSEDE user

This section explains how to use ssh or XSEDE Single Sign On to access Bridges.

SSH

You can use an SSH client from your local machine to connect to Bridges using either your PSC or XSEDE credentials.

SSH is a program that enables secure logins over an unsecure network. It encrypts the data passing both ways so that if it is intercepted it cannot be read.

SSH is client-server software, which means that both the user's local computer and the remote computer must have it installed. SSH server software is installed on all the PSC machines. You must install SSH client software on your local machine.

Free ssh clients for Macs, Windows machines and many versions of Unix are available. Popular ssh clients (GUI) include PuTTY for Windows and Cyberduck for Macs. A command line version of ssh is installed on Macs by default; if you prefer that, you can use it in the Terminal application. You can also check with your university to see if there is an ssh client that they recommend.

Once you have an SSH client installed, you can use either your PSC credentials or XSEDE credentials (optionally with DUO MFA) to connect to Bridges. Note that you must have created your PSC password before you can use SSH to connect to Bridges.

Use ssh to connect to Bridges using XSEDE credentials and (optionally) DUO MFA

1. Using your SSH client, connect to hostname bridges.psc.xsede.org or bridges.psc.edu using port 2222.
   Either hostname will connect you to Bridges, but you must specify port 2222.
2. Enter your XSEDE username and password when prompted.
3. (Optional) If you are registered with XSEDE DUO, you will receive a prompt on your phone. Once you have answered it, you will be logged in.

Use ssh to connect to Bridges using PSC credentials

1. Using your SSH client, connect to hostname bridges.psc.xsede.org or bridges.psc.edu using the default port (22).
   Either hostname will connect you to Bridges. You do not have to specify the port.
2. Enter your PSC username and password when prompted.

Read more about using SSH to connect to PSC systems.

Public-private keys

You can also use public-private key pairs to connect to Bridges. To do so, you must first fill out this form to register your keys with PSC.

XSEDE Single Sign On

XSEDE users can use their XSEDE usernames and passwords in the XSEDE User Portal Single Sign On Login Hub (SSO Hub) to access bridges.psc.xsede.org or bridges.psc.edu.

You must use DUO Multi-Factor Authentication in the SSO Hub.

See the XSEDE instructions to set up DUO for Multi-Factor Authentication.

The projects command

The projects command will help you monitor your allocation on Bridges. You can determine what Bridges resources you have been allocated, your remaining balance, your account id (used to track usage), and more. The output below shows that this user has an allocation on Bridges AI and Bridges Large resources for computing and Bridges Pylon for file storage.

[userid@login018 ~]$ projects
 Your default charging project charge id is ABC0123456. If you would like to change the default charging project 
use the command change_primary_group ~charge_id~. Use the charge id listed below for the project you would like 
to make the default in place of ~charge_id~


Project: XYZ654321D
PI: My Principal Investigator
Title: Important Research

    Resource: BRIDGES AI
  Allocation: 10,000.00
     Balance: 680.49
    End Date: 2030-07-15
Award Active: Yes
 User Active: Yes
   Charge ID: ABC0123456
   *** Default charging project ***
 Directories:
     HOME /home/username

    Resource: BRIDGES LARGE MEMORY
  Allocation: 200,000.00
     Balance: 84,597.05
    End Date: 2030-07-15
Award Active: Yes
 User Active: Yes
   Charge ID: ABC0123456
   *** Default charging project ***
 Directories:
     HOME /home/username

    Resource: BRIDGES PYLON STORAGE
  Allocation: 100,000.00
     Balance: 21,937.62
    End Date: 2030-07-15
Award Active: Yes
 User Active: Yes
   Charge ID: ABC0123456
 Directories:
     Lustre Project Storage /pylon5/ABC0123456 
     Lustre Storage /pylon5/ABC0123456/username

Accounting for Bridges use

Accounting for Bridges use varies with the type of node used, which is determined by the type of allocation you have: "Bridges Regular", for Bridges' RSM (128GB) nodes); "Bridges Large", for Bridges LSM and ESM (3TB and 12TB) nodes; "Bridges GPU", for Bridges' K80 and P100 GPU nodes; and "Bridges AI" for Bridges' Volta GPU nodes and DGX-2 system.

Usage is defined in terms of "Service Units" or SUs. The definition of an SU varies with the type of node being used.

Bridges regular

The RSM nodes are allocated as "Bridges regular". This does not include Bridges' GPU nodes. Each RM node holds 28 cores, each of which can be allocated separately. Service Units are defined in terms of "core hours": the use of one core for 1 hour.

1 core-hour = 1 SU

Because the RM nodes each hold 28 cores, if you use one entire RM node for one hour, 28 SUs will be deducted from your allocation.

28 cores x 1 hour = 28 core-hours = 28 SUs

If you use 2 cores on a node for 30 minutes, 1 SU will be deducted from your allocation.

2 cores x 0.5 hours = 1 core-hour = 1 SU

Bridges large

The LSM and ESM nodes are allocated as "Bridges large". Accounting for the LM and ESM nodes is done by the memory requested for the job. Service Units (SUs) are defined in terms of "TB-hours": the use of 1TB of memory for one hour. Note that because the memory requested for a job is set aside for your use when the job begins, SU usage is calculated based on memory requested, not on how much memory is actually used.

1 SU = 1 TB-hour

If your job requests 3TB of memory and runs for 1 hour, 3 SUs will be deducted from your allocation.

3TB x 1 hour = 3TB-hours = 3 SUs

If your job requests 8TB and runs for 6 hours, 48 SUs will be deducted from your allocation.

8TB x 6 hours = 48 TB-hours = 48 SUs

Bridges GPU

Bridges contains two kinds of GPU nodes: NVIDIA Tesla K80s and NVIDIA Tesla P100s. Service Units (SUs) for GPU nodes are defined in terms of "gpu-hours": the use of one GPU Unit for one hour.

Because of the difference in the performance of the nodes, SUs are calculated differently for the two types of nodes.

K80 nodes

The K80 nodes hold 4 GPU units each, which can be allocated separately. Service units (SUs) are defined in terms of gpu-hours:

For K80 GPU nodes, 1 gpu-hour = 1 SU

If you use 2 entire K80 nodes for 1 hour, 8 SUs will be deducted from your allocation.

4 GPU units/node x 2 nodes x 1 hour = 8 gpu-hours = 8 SUs

If you use 2 GPU units for 3 hours, 6 SUs will be deducted from your allocation.

2 GPU units x 3 hours = 6 gpu-hours = 6 SUs

P100 nodes

The P100 nodes hold 2 GPU units each, which can be allocated separately. Service Units (SUs) are defined in terms of gpu-hours. Because the P100s are more powerful than the K80 nodes, the SU definition is different.

For P100 GPU nodes, 1 gpu-hour = 2.5 SUs

If you use an entire P100 node for one hour, 5 SUs will be deducted from your allocation.

2 GPU units/node x 1 node x 1 hour = 2 gpu-hours

2 gpu-hours x 2.5 SUs/gpu-hour = 5 SUs

If you use 1 GPU unit on a P100 for 8 hours, 20 SUs will be deducted from your allocation.

1 GPU unit x 8 hours = 8 gpu-hours

8 gpu-hours x 2.5 SU/gpu-hours = 20 SUs

Bridges AI

Bridges AI comprises two kinds of GPU nodes: an NVIDIA DGX-2 enterprise research AI system ("Volta 32"), and 9 HPE Apollo 6500 servers ("Volta 16"). The DGX-2 tightly couples 16 NVIDIA Tesla V100 (Volta) GPUs, each with 32 GB of GPU memory, connected by NVLink and NVSwitch. Each Volta 16 node has 8 NVIDIA Tesla V100 GPUs, each with 16 GB of GPU memory, and the GPUs are connected by NVLink 2.0.

Service Units (SUs) for AI nodes are defined in terms of "gpu-hours": the use of one GPU Unit for one hour.

DGX-2 node

The DGX-2 node holds 16 GPU units, each of which can be allocated separately. Service Units (SUs) are defined in terms of gpu-hours:

For the DGX-2 node, 1 GPU-hour = 1 SU

If you use 2 GPUs on the DGX-2 node for 1 hour, 2 SUs will be deducted from your allocation.

2 GPU units x 1 hour = 2 gpu-hours = 2 SUs

If you use the entire DGX-2 for 3 hours, 48 SUs will be deducted from your allocation.

16 GPU units x 3 hours = 48 gpu-hours = 48 SUs

Volta 16 nodes

The Volta 16 nodes hold 8 GPU units each, each of which can be allocated separately. Service Units (SUs) are defined in terms of gpu-hours.

For Volta 16 GPU nodes, 1 gpu-hour = 1 SU

If you use an entire Volta 16 node for one hour, 8 SUs will be deducted from your allocation.

8 GPU units/node x 1 node x 1 hour = 8 gpu-hours = 8 SUs

If you use 4 GPU units on a Volta 16 for 48 hours, 196 SUs will be deducted from your allocation.

4 GPU units x 48 hours = 196 gpu-hours = 196 SUs

Accounting for file space

Every Bridges grant has a pylon storage allocation associated with it. There are no SUs deducted from your allocation for the space you use, but if you exceed your storage quota, you will not be able to submit jobs to Bridges.

Each grant has a Unix group associated with it. Every file is "owned" by a Unix group, and that file ownership determines which grant is charged for the file space. See Managing multiple grants for a further explanation of Unix groups, and how to manage file ownership if you have more than one grant.

You can check your pylon usage with the projects command.

[username@br018]$ projects
.
.
.

Resource: BRIDGES PYLON STORAGE
    Allocation: 500.00
       Balance: 412.75
      End Date: 2019-02-15
  Award Active: Yes
   User Active: Yes
     Charge ID: account-1
   Directories:
       Lustre Project Storage /pylon5/account-1
       Lustre Storage /pylon5/account-1/username

Managing multiple grants

If you have multiple grants on Bridges, you should ensure that the work you do under each grant is assigned correctly to that grant. The files created under or associated with that grant should belong to it, to make them easily available to others on the same grant.

There are two fields associated with each grant for these purposes: a SLURM account id and a Unix group.

SLURM account ids determine which grant your Bridges use is deducted from.

Unix groups determine which pylon5 allocation the storage space for files is deducted from, and who owns and can access a file or directory.

For a given grant, the SLURM account id and the Unix group are identical strings.

One of your grants has been designated as your default grant, and the account id and Unix group associated with the grant are your default account id and default Unix group.

When a Bridges job runs, any SUs it uses are deducted from the default grant. Any files created by that job are owned by the default Unix group.

Find your default account id and Unix group

To find your SLURM account ids, use the projects command. It will display all the grants you belong to. It will also list your default account id (called charge id in the projects output) at the top. Your default Unix group is the same.

In this example, the user has two grants with account ids account-1 and account-2. The default account id is account-2.

[myusername@br006]$ projects
Your default charging project charge id is account-2. If you would like to change the default charging project use the command change_primary_group charge_id. Use the charge id listed below for the project you would like to make the default in place of charge_id.

Project: AAA000000A
     PI: My Principal Investigator
  Title: Important Research

      Resource: BRIDGES GPU
    Allocation: 37,830.00
       Balance: 17,457.19
      End Date: 2030-07-15
  Award Active: Yes
   User Active: Yes
     Charge ID: account-1
   Directories:
       HOME /home/myusername
.
.
.
Project: AAA111111A
     PI: My Other PI
  Title: More Important Research

      Resource: BRIDGES PYLON STORAGE
    Allocation: 57,500.00
       Balance: 12,474.99
      End Date: 2019-06-15
  Award Active: Yes
   User Active: Yes
     Charge ID: account-2
   *** Default charging project ***
   Directories:
       Lustre Project Storage /pylon5/account-2
       Lustre Storage /pylon5/account-2/myusername

Use a secondary (non-default) grant

To use a grant other than your default grant on Bridges, you must specify the appropriate account id with the -A option to the SLURM sbatch command. See the Running Jobs section of this Guide for more information on batch jobs, interactive sessions and SLURM.

Note that using the -A option does not change your default Unix group. Any files created during a job are owned by your default Unix group, no matter which account id is used for the job, and the space they use will be deducted from the pylon allocation for the default Unix group.

Change your Unix group for a login session

To temporarily change your Unix group, use the newgrp command. Any files created subsequently during this login session will be owned by the new group you have specified. Their storage will be deducted from the pylon allocation of the new group. After logging out of the session, your default Unix group will be in effect again.

newgrp unix_group

Note that the newgrp command has no effect on the account id in effect. Any Bridges usage will be deducted from the default account id or the one specified with the -A option to sbatch.

Change your default account id and Unix group permanently

You can permanently change your default account id and your default Unix group with the change_primary_group command. Type:

change_primary_group -l

to see all your groups. Then type

change_primary_group account-id

to set account-id as your default.

Your default account id changes immediately. Bridges use by any batch jobs or interactive sessions following this command are deducted from the new account by default.

Your default Unix group does not change immediately. It takes about an hour for the change to take effect. You must log out and log back in after that window for the new Unix group to be the default.

Tracking your usage

There are several ways to track your Bridges usage: the xdusage command, the projects command, and the Grant Management System.

The xdusage command displays project and user account usage information for XSEDE projects. Type man xdusage on Bridges for information.

The projects command shows information on all Bridges grants, including usage and the pylon directories associated with the grant.

Managing your XSEDE allocation

Most account management functions for your XSEDE grant are handled through the XSEDE User Portal. You can search the Knowledge Base to get help. Some common questions:

• How do I add a user to my grant?
• How can I transfer SUs from Bridges Large to Bridges Regular (or vice versa)?
• How can I request more SUs or storage space on Bridges?

Change your default shell

The change_shell command allows you to change your default shell. This command is only available on the login nodes.

To see which shells are available, type:

change_shell -l

To change your default shell, type:

change_shell newshell

where newshell is one of the choices output by the change_shell -l command. You must use the entire path output by change_shell -l, e.g. /usr/psc/shells/bash. You must log out and back in again for the new shell to take effect.

File expiration

Three months after your grant expires all of your Bridges files associated with that grant will be deleted, no matter which file space they are in. You will be able to login during this 3-month period to transfer files, but you will not be able to run jobs or create new files.

File permissions

Access to files in any Bridges space is governed by Unix file permissions. If your data has additional security or compliance requirements, please contact compliance@psc.edu.

Unix file permissions

For detailed information on Unix file protections, see the man page for the chmod command.

To share files with your group, give the group read and execute access for each directory from your top-level directory down to the directory that contains the files you want to share:

chmod g+rx directory-name

Then give the group read and execute access to each file you want to share:

chmod g+rx filename

To give the group the ability to edit or change a file, add write access to the group:

chmod g+rwx filename

Access Control Lists

If you want more fine-grained control than Unix file permissions allow, e.g. if you want to give only certain members of a group access to a file, but not all members, then you need to use Access Control Lists (ACLs). Suppose, for example, that you want to give janeuser access to a file in a directory, but no one else in the group.

Use the setfacl (set file acl) command:

setfacl -m user:janeuser:rx directory-name

for each directory from your top-level directory down to the directory that contains the file you want to share. Then give janeuser access to the specific file with:

setfacl -m user:janeuser:r filename

User janeuser will now be able to read this file, but no one else in the group will have access to it.

To see what ACLs are set on a file, use the getfacl command.

There are man pages for chmod, setfacl and getfacl.

Home ($HOME)

This is your Bridges home directory. It is the usual location for your batch scripts, source code and parameter files. Its path is /home/username, where username is your PSC userid. You can refer to your home directory with the environment variable $HOME. Your home directory is visible to all of the Bridges nodes.

Your home directory is backed up daily, although it is still a good idea to store copies of your important files in another location, such as the pylon5 file system or on a local file system at your site. If you need to recover a home directory file from backup send email to remarks@psc.edu. The process of recovery will take 3 to 4 days.

$HOME quota

Your home directory has a 10GB quota. You can check your home directory usage using the quota command or the command du -sh. To improve the access speed to your home directory files you should stay as far below your home directory quota as you can.

Grant expiration

Three months after a grant expires, the files in your home directory associated with that grant will be deleted.

pylon5 ($SCRATCH)

Pylon5 is a Lustre file system shared across all of Bridges' nodes. It is available on Bridges compute nodes as $SCRATCH.

The pylon5 file system is persistent storage, and can be used as working space for your running jobs. It provides fast access for data read or written by running jobs. I/O to pylon5 is much faster than to your home directory.

Files on pylon5 are not backed up, so you should store copies of important pylon5 files in another location.

pylon5 directories

The path of your pylon5 home directory is /pylon5/groupname/username, where groupname is the Unix group associated with your grant. Use the id command to find your group name.

The command id -Gn will list all the groups you belong to.

The command id -gn will list the group associated with your current session.

If you have more than one grant, you will have a pylon5 directory for each grant. Be sure to use the appropriate directory when working with multiple grants.

pylon5 quota

Your usage quota for each of your grants is the Pylon storage allocation you received when your proposal was approved. If your total use in pylon5 exceeds this quota your access to the partitions on Bridges will be shut off until you are under quota.

Use the du -sh or projects command to check your pylon5 usage. You can also check your usage on the XSEDE User Portal.

If you have multiple grants, it is very important that you store your files in the correct pylon5 directory.

Grant expiration

Three months after a grant expires, the files in any pylon5 directories associated with that grant will be deleted.

Node-local ($LOCAL)

Each Bridges node has a local file system attached to it. This local file system is only visible to the node to which it is attached. The local file system provides fast access to local storage.

This file space is available on all nodes as $LOCAL.

If your application performs a lot of small reads and writes, then you could benefit from using $LOCAL.

$LOCAL is only available when your job is running, and can only be used as working space for a running job. Once your job finishes, your local files are inaccessible and deleted. To use local space, copy files to $LOCAL at the beginning of your job and back out to a persistent file space before your job ends.

If a node crashes, all the $LOCAL files are lost. Therefore, you should checkpoint your $LOCAL files by copying them to pylon5 during long runs.

Multi-node jobs

If you are running a multi-node job, the variable $LOCAL points to the local file space on the node that is running your rank 0 process.

You can use the srun command to copy files between $LOCAL on the nodes in a multi-node job. See the Sample script for MPI in the Sample Batch Scripts section of this User Guide for details.

$LOCAL size

The maximum amount of local space varies by node type. The RSM (128GB) and GPU nodes have a maximum of 3.7TB. The LSM (3TB) nodes have a maximum of 14TB and the ESM (12TB) nodes have a maximum of 49TB.

To check on your local file space usage:

du -sh

No Service Units accrue for the use of $LOCAL.

Using $LOCAL

To use $LOCAL you must first copy your files to $LOCAL at the beginning of your script, before your executable runs. The following script is an example of how to do this:

RC=1
n=0
while [[ $RC -ne 0 && $n -lt 20 ]]; do
    rsync -aP $sourcedir $LOCAL/
    RC=$?
    let n = n + 1
    sleep 10
done

Set $sourcedir to point to the directory that contains the files to be copied before you execute your program. This code will try at most 20 times to copy your files. If it succeeds, the loop will exit. If an invocation of rsync was unsuccessful, the loop will try again and pick up where it left off.

At the end of your job you must copy your results back from $LOCAL or they will be lost. The following script will do this:

mkdir $SCRATCH/results
RC=1
n=0
while [[ $RC -ne 0 && $n -lt 20 ]]; do
    rsync -aP $LOCAL/ $SCRATCH/results
    RC=$?
    let n = n + 1
    sleep 10
done

This code fragment copies your files to a directory in your pylon5 file space named results, which you must have created previously with the mkdir command. Again it will loop at most 20 times and stop if it is successful.

Memory files ($RAMDISK)

You can also use the memory allocated for your job for I/O rather than using disk space. This will offer the fastest I/O on Bridges.

In a running job the environment variable $RAMDISK will refer to the memory associated with the nodes in use.

The amount of memory space available to you depends on the size of the memory on the nodes and the number of nodes you are using. You can only perform I/O to the memory of nodes assigned to your job.

If you do not use all of the cores on a node, you are allocated memory in proportion to the number of cores you are using. Note that you cannot use 100% of a node's memory for I/O; some is needed for program and data usage.

$RAMDISK is only available to you while your job is running, and can only be used as working space for a running job. Once your job ends this space is inaccessible. To use memory files, copy files to $RAMDISK at the beginning of your job and back out to a permanent space before your job ends. If your job terminates abnormally, your memory files are lost.

Within your job you can cd to $RAMDISK, copy files to and from it, and use it to open files. Use the command du -sh to see how much space you are using.

If you are running a multi-node job the $RAMDISK variable points to the memory space on the node that is running your rank 0 process.

Paths for Bridges file spaces

For all file transfer methods other than cp, you must always use the full path for your Bridges files. The start of the full paths for your Bridges directories are:

Home directory  /home/username
Pylon5 directory  /pylon5/Unix-group/username

The command id -Gn will show all of your valid Unix-groups. You have a pylon5 directory for each grant you have.

Transfers into your Bridges home directory

Your home directory quota is 10GB, so large files cannot be stored there; they should be copied into one of your pylon file spaces instead. Exceeding your home directory quota will prevent you from writing more data into your home directory and will adversely impact other operations you might want to perform.

File transfer methods

rsync

You can use the rsync command to copy files to and from Bridges. A sample rsync command to copy to a Bridges directory is

rsync -rltpDvp -e 'ssh -l joeuser' source_directory data.bridges.psc.edu:target_directory

Substitute your userid for 'joeuser'. Make sure you use the correct group name in your target directory. By default, rsync will not copy older files with the same name in place of newer files in the target directory. It will overwrite older files.

We recommend the rsync options -rltDvp. See the rsync man page for information on these options and other options you might want to use. We also recommend the option

-oMACS=umac-64@openssh.com

If you use this option your transfer will use a faster data validation algorithm.

You may to want to put your rsync command in a loop to ensure that it completes. A sample loop is

RC=1
n=0
while [[ $RC -ne 0 && $n -lt 20 ]] do
    rsync ...
    RC = $?
    let n = n + 1
    sleep 10
done

This loop will try your rsync command 20 times. If it succeeds it will exit. If an rsync invocation is unsuccessful the system will try again and pick up where it left off. It will copy only those files that have not already been transferred. You can put this loop, with your rsync command, into a batch script and run it with sbatch.

scp

To use scp for a file transfer, you must specify a source and destination for your transfer. The format for either source or destination is

username@machine-name:path/filename

For transfers involving Bridges, username is your PSC username. The machine-name should be given as data.bridges.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using scp involving Bridges. Using it prevents file transfers from disrupting interactive use on Bridges login nodes.

File transfers using scp must specify full paths for Bridges file systems. See Paths for Bridges file spaces for details.

sftp

To use sftp, first connect to the remote machine:

sftp username@machine-name

When Bridges is the remote machine, use your PSC userid as username. The Bridges machine-name should be specified as data.bridges.psc.edu. This is the name for a high-speed data connector at PSC. We recommend using it for all file transfers using sftp involving Bridges. Using it prevents file transfers from disrupting interactive use on Bridges login nodes.

You will be prompted for your password on the remote machine. If Bridges is the remote machine enter your PSC password.

You can then enter sftp subcommands, like put to copy a file from the local system to the remote system, or get to copy a file from the remote system to the local system.

To copy files into Bridges you must either cd to the proper directory or use full pathnames in your file transfer commands. See Paths for Bridges file spaces for details.

Two-factor Authentication

If you are required to use two-factor authentication (TFA) to access Bridges filesystems, you must enroll in XSEDE DUO. Once that is complete, use scp or sftp to transfer files to/from Bridges.

TFA users must use port 2222 and XSEDE User Portal usernames and passwords. The machine name for these transfers is data.bridges.psc.edu.

In the examples below, myfile is the local filename, XSEDE-username is your XSEDE User Portal username and /path/to/file is the full path to the file on a Bridges filesystem. Note that -P (capital P) is necessary.

scp

Transfer a file from a local machine to Bridges:

scp -P 2222 myfile XSEDE-username@data.bridges.psc.edu:/path/to/file

Transfer a file from Bridges to a local machine:

scp -P 2222 XSEDE-username@data.bridges.psc.edu:/path/to/file myfile

sftp

Use sftp interactively:

sftp -P 2222 XSEDE-username@data.bridges.psc.edu

Then use the put command to copy a file from the local machine to Bridges, or the get command to transfer a file from Bridges to the local machine.

Graphical SSH client

If you are using a graphical SSH client, configure it to connect to data.bridges.psc.edu on port 2222/TCP. Login using your XSEDE User Portal username and password.

Globus

Globus can be used for any file transfer to Bridges. It tracks the progress of the transfer and retries when there is a failure; this makes it especially useful for transfers involving large files or many files.

To use Globus to transfer files you must authenticate either via a Globus account or with InCommon credentials.

Globus account

You can set up a Globus account at the Globus site.

InCommon credentials

If you wish to use InCommon credentials to transfer files to/from Bridges, you must first provide your CILogon Certificate Subject information to PSC. Follow these steps:

1. Find your Certificate Subject string

   1. Navigate your web browser to https://cilogon.org.
   2. Select your institution from the 'Select an Identity Provider' list.
   3. Click the 'Log On' button. You will be taken to the web login page for your institution.
   4. Login with your username and password for your institution.
      1. If your institution has an additional login requirement (e.g., Duo), authenticate to that as well. After successfully authenticating to your institution's web login interface, you will be returned to the CILogon webpage. Note the boxed section near the top that lists a field named 'Certificate Subject'.

2. Send your Certificate Subject string to PSC

   1. In the CILogon webpage, select and copy the Certificate Subject text. Take care to get the entire text string if it is broken up onto multiple lines.
   2. Send email to support@psc.edu. Paste your Certificate Subject field into the message, asking that it be mapped to your PSC username.

Your CILogon Certificate Subject information will be added within one business day, and you will be able to begin transferring files to and from Bridges.

Globus endpoints

Once you have the proper authentication you can initiate file transfers from the Globus site. A Globus transfer requires a Globus endpoint, a file path and a file name for both the source and destination. The endpoints for Bridges are:

• psc#bridges-xsede if you are using an XSEDE User Portal account for authentication
• psc#bridges-cilogon if you are using InCommon for authentication

These endpoints are owned by psc@globusid.org. If you use DUO MFA for your XSEDE authentication, you do not need to because you cannot use it with Globus. You must always specify a full path for the Bridges file systems. See Paths for Bridges file spaces for details.

Globus-url-copy

The globus-url-copy command can be used if you have access to Globus client software. Both the globus-url-copy and myproxy-logon commands are available on Bridges, and can be used for file transfers internal to the PSC.

To use globus-url-copy you must have a current user proxy certificate. The command grid-proxy-info will tell you if you have current user proxy certificate and if so, what the remaining life of your certificate is.

Use the myproxy-logon command to get a valid user proxy certificate if any one of these applies:

• you get an error from the grid-proxy-info command
• you do not have a current user proxy certificate
• the remaining life of your certificate is not sufficient for your planned file transfer

When prompted for your MyProxy passphrase enter your XSEDE User Portal password.

To use globus-url-copy for transfers to a machine, you must know the Grid FTP server address. The Grid FTP server address for Bridges is

gsiftp://gridftp.bridges.psc.edu

The use of globus-url-copy always requires full paths. See Paths for Bridges file spaces for details.

Transfer rates

PSC maintains a Web page at http://speedpage.psc.edu that lists average data transfer rates between all XSEDE resources. If your data transfer rates are lower than these average rates or you believe that your file transfer performance is subpar, send email to bridges@psc.edu. We will examine approaches for improving your file transfer performance.

C, C++ and Fortran

Intel, Gnu and PGI compilers for C, C++ and Fortan are available on Bridges. The compilers are:

OpenMP programming

To compile OpenMP programs you must add an option to your compile command:

MPI programming

Three types of MPI are supported on Bridges: MVAPICH2, OpenMPI and Intel MPI.

There are two steps to compile an MPI program:

1. If you are using MVAPICH2 or OpenMPI, load the correct module for the compiler and MPI type you want to use. The Intel MPI module is loaded for you on login.
2. Issue the appropriate MPI wrapper command to compile your program.

The three MPI types may perform differently on different problems or in different programming environments. If you are having trouble with one type of MPI, please try using another type. Contact bridges@psc.edu for more help.

INTEL COMPILERS

For proper Intel MPI behavior, you must set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT to 0. Otherwise the mpirun task placement settings you give will be ignored.

BASH

export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0

CSH:

setenv I_MPI_JOB_RESPECT_PROCESS_PLACEMENT 0

GNU COMPILERS

PGI COMPILERS

See also:

• Intel MPI website
• MVAPICH2 website
• OpenMPI website
• Module documentation for information on what modules are available and how to use them.

Other languages

Other languages, including Java, Python, R, and MATLAB, are available. See the software section for information.

Debugging and performance analysis

DDT

DDT is a debugging tool for C, C++ and Fortran 90 threaded and parallel codes. It is client-server software. Install the client on your local machine and then you can access the GUI on Bridges to debug your code.

See the PSC DDT and MAP page for more information.

MAP

MAP is a profiling tool for C, C++ and Fortran applications. It is client-server software. Install the client on your local machine and then you can access the GUI on Bridges to debug your code.

See the PSC DDT and MAP page for more information.

VTune

VTune is a performance analysis tool from Intel for serial, multithreaded and MPI applications. Install the client on your local machine and then you can access the GUI on Bridges. See the PSC VTune page for more information.

The Module package

The environment management package Module is essential for running software on most PSC systems. Be sure to check if there is a module for the software you want to use by typing module avail software-name.

The module help software-name command lists any additional modules that must also be loaded. Note that in some cases the order in which these additional modules are loaded matters. See documentation on the module command for more information.

Interactive sessions

You can do your production work interactively on Bridges, typing commands on the command line, and getting responses back in real time. But you must be allocated the use of one or more Bridges' compute nodes by SLURM to work interactively on Bridges. You cannot use the Bridges login nodes for your work.

You can run an interactive session in any of the SLURM partitions. You will need to specify which partition you want, so that the proper resources are allocated for your use.

If all of the resources set aside for interactive use are in use, your request will wait until the resources you need are available. Using a shared partition (RM-shared, GPU-shared) will probably allow your job to start sooner.

interact -options

$ interact

[bridgesuser@br006 ~]$ interact

[bridgesuser@r004 ~]$

See also:

• Bridges partitions
• How to determine your valid SLURM account ids and Unix groups and change your default in the Account adminstration section
• Charging with multiple grants
• The srun command, for more complex control over your interactive job

Batch jobs

To run a batch job, you must first create a batch (or job) script, and then submit the script using the sbatch command.

A batch script is a file that consists of SBATCH directives, executable commands, and comments.

SBATCH directives specify your resource requests and other job options in your batch script. You can also specify resource requests and options on the sbatch command line. Any options on the command line take precedence over those given in the batch script. The SBATCH directives must start with "#SBATCH" as the first text on a line, with no leading spaces.

Comments begin with a '#' character.

The first line of any batch script must indicate the shell to use for your batch job.

Sample script for OpenMP #!/bin/bash #SBATCH -N 1 #SBATCH -p RM #SBATCH --ntasks-per-node 28 #SBATCH -t 5:00:00 # echo commands to stdout set -x # move to your appropriate pylon5 directory # this job assumes: # - all input data is stored in this directory # - all output should be stored in this directory cd /pylon5/groupname/username/path-to-directory # run OpenMP program export OMP_NUM_THREADS=28 ./myopenmp Notes: The "--ntasks-per-node" option indicates that you will use all 28 cores. For groupname, username, and path-to-directory you must substitute your Unix group, username, and appropriate directory path.

Sample script for MPI #!/bin/bash #SBATCH -p RM #SBATCH -t 5:00:00 #SBATCH -N 2 #SBATCH --ntasks-per-node 28 #echo commands to stdout set -x #move to your appropriate pylon5 directory cd /pylon5/groupname/username/path-to-directory #set variable so that task placement works as expected export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0 #copy input files to `$LOCAL` file storage srun -N $SLURM_JOB_NUM_NODES --ntasks-per-node=1 \ sh -c 'cp path-to-directory/input.${SLURM_PROCID} $LOCAL' #run MPI program mpirun -np $SLURM_NTASKS ./mympi # Copy output files to pylon5 srun -N $SLURM_JOB_NUM_NODES --ntasks-per-node=1 \ sh -c 'cp $LOCAL/output.* /pylon5/groupname/username/path-to-directory' Notes: The variable $SLURM_NTASKS gives the total number of cores requested in a job. In this example $SLURM_NTASKS will be 56 because the "-N" option requested 2 nodes and the "--ntasks-per-node" option requested all 28 cores on each node. The export command sets "I_MPI_JOB_RESPECT_PROCESS_PLACEMENT" so that your task placement settings are effective. Otherwise, the SLURM defaults are in effect. The srun commands are used to copy files between pylon5 and the $LOCAL file systems on each of your nodes. The first srun command assumes you have two files named input.0 and input.1 in your pylon5 file space. It will copy input.0 and input.1 to, respectively, the $LOCAL file systems on the first and second nodes allocated to your job. The second srun command will copy files named "output.*" back from your $LOCAL file systems to your pylon5 file space before your job ends. In this command '*' functions as the usual Unix wildcard. For groupname, username, and path-to-directory you must substitute your Unix group, username, and appropriate directory path.

Sample script for hybrid OpenMP-MPI #!/bin/bash #SBATCH --nodes=2 #SBATCH --ntasks=4 #SBATCH --cpus-per-task=14 #SBATCH --time=00:10:00 #SBATCH --job-name=hybrid cd $SLURM_SUBMIT_DIR mpiifort -xHOST -O3 -qopenmp -mt_mpi hello_hybrid.f90 -o hello_hybrid.exe # set variable so task placement works as expected export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0 mpirun -print-rank-map -n $SLURM_NTASKS -genv \ OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK -genv I_MPI_PIN_DOMAIN=omp \ ./hello_hybrid.exe Notes: This example asks for 2 nodes, 4 MPI tasks and 14 OpenMP threads per MPI task. The export command sets I_MPI_JOB_RESPECT_PROCESS_PLACEMENT so that your task placement settings are effective. Otherwise, the SLURM defaults are in effect.

Sample script for Job Array #!/bin/bash #SBATCH -t 05:00:00 #SBATCH -p RM-shared #SBATCH -N 1 #SBATCH --ntasks-per-node 5 #SBATCH --array=1-5 set -x ./myexecutable $SLURM_ARRAY_TASK_ID Notes: This script will generate five jobs that will each run on a separate core on the same node. The value of the variable SLURM_ARRAY_TASK_ID is the core number, which, in this example, will range from 1 to 5. Good candidates for job array jobs are jobs that can use only this core index to determine the different processing path for each job. For more information about job array jobs see the sbatch man page and the online SLURM documentation.

Sample script for bundling single-core jobs #!/bin/bash #SBATCH -N 1 #SBATCH -p RM-shared #SBATCH -t 05:00:00 #SBATCH --ntasks-per-node 14 echo SLURM NTASKS: $SLURM_NTASK i=0 while [ $i -lt $SLURM_NTASKS ] do numactl -C +$i ./run.sh & let i=i+1 done wait # IMPORTANT: wait for all to finish or get killed Notes: Bundling or packing multiple jobs in a single job can improve your turnaround and improve the performance of the SLURM scheduler.

Sample script for bundling multi-core jobs #!/bin/bash #SBATCH -N 1 #SBATCH -p RM-shared #SBATCH -t 05:00:00 #SBATCH --ntasks-per-node 14 #SBATCH --cpus-per-task 2 echo SLURM NTASKS: $SLURM_NTASKS i=0 while [ $i -lt $SLURM_NTASKS ] do numactl -C +$i ./run.sh & let i=i+1 done wait # IMPORTANT: wait for all to finish or get killed Notes: Bundling or packing multiple jobs in a single job can improve your turnaround and improve the performance of the SLURM scheduler.

Sample script for GPU #!/bin/bash #SBATCH -N 2 #SBATCH -p GPU #SBATCH --ntasks-per-node 28 #SBATCH -t 5:00:00 #SBATCH --gres=gpu:p100:2 # echo commands to stdout set -x # move to working directory # this job assumes: # - all input data is stored in this directory # - all output should be stored in this directory cd /pylon5/groupname/username/path-to-directory # run GPU program ./mygpu Notes: The option --gres-gpu indicates the number and type of GPUs you want. For groupname, username, and path-to-directory you must substitute your Unix group, username, and appropriate directory path.

Sample script for GPU-shared partition #!/bin/bash #SBATCH -N 1 #SBATCH -p GPU-shared #SBATCH --ntasks-per-node 7 #SBATCH --gres=gpu:p100:1 #SBATCH -t 5:00:00 # echo commands to stdout set -x # move to working directory # this job assumes: # - all input data is stored in this directory # - all output should be stored in this directory cd /pylon5/groupname/username/path-to-directory # run GPU program ./mygpu Notes: The option --gres-gpu indicates the number and type of GPUs you want. For groupname, username, and path-to-directory you must substitute your Unix group, username, and appropriate directory path.

Sample script for the GPU-AI partition #!/bin/bash #SBATCH --partition=GPU-AI #SBATCH --nodes=1 #SBATCH --gres=gpu:volta16:8 #SBATCH --time=1:00:00 source /etc/profile.d/modules.sh cd $SCRATCH/ngc module load singularity singularity exec --nv \ /pylon5/containers/ngc/directory-name/tensorflow-image.simg \ /pylon5/groupname/username/path-to-executable Notes: The bash shell is used and the source command loads the module command. The --nv switch to the singularity exec command is necessary to include CUDA support. For directory-name and tensorflow-image, substitute the appropriate directory and image names. For groupname, username, and path-to-executable you must substitute your Unix group, username, and appropriate executable path.

The sbatch command

To submit a batch job, use the sbatch command. The format is

sbatch -options batch-script

The options to sbatch can either be in your batch script or on the sbatch command line. Options in the command line override those in the batch script.

Note:

• Be sure to use the correct account id for your job if you have more than one grant. See the -A option for sbatch to change the SLURM account id for a job. Information on how to determine your valid account ids and change your default account id is in the Account adminstration section.
• In some cases, the options for sbatch differ from the options for interact or srun.
• By default, sbatch submits jobs to the RM partition. Use the -p option for sbatch to direct your job to a different partition.

Charging with multiple grants

If you have more than one grant, be sure to use the correct SLURM account id and Unix group when running jobs.

See Managing multiple grants in the Account Administration section to see how to find your account ids and Unix groups and determine or change your defaults.

Permanently change your default SLURM account id and Unix group

See the change_primary_group command in the Account Administration section to permanently change your default SLURM account id and Unix group.

Temporarily change your SLURM account id or Unix group

See the -A option to the sbatch or interact commands to set the SLURM account id for a specific job.

The newgrp command will change your Unix group for that login session only. Note that any files created by a job are owned by the Unix group in effect when the job is submitted, which is not necessarily the same as the account id used for the job. See the newgrp command in the Account Administration section to see how to change the Unix group currently in effect.

Bridges partitions

Each SLURM partition manages a subset of Bridges' resources. Each partition allocates resources to interactive sessions, batch jobs, and OnDemand sessions that request resources from it.

Know which partitions are open to you: Your Bridges allocations determine which partitions you can submit jobs to.

• A "Bridges regular memory" allocation allows you to use Bridges' RSM (128GB) nodes. Partitions available to "Bridges regular memory" allocations are
  

  • RM, for jobs that will run on Bridges' RSM (128GB) nodes, and use one or more full nodes
  • RM-shared, for jobs that will run on Bridges' RSM (128GB) nodes, but share a node with other jobs
  • RM-small, for short jobs needing 2 full nodes or less, that will run on Bridges RSM (128GB) nodes

• A "Bridges large memory" allocation allows you to use Bridges LSM and ESM (3TB and 12TB) nodes. There is one partition available to "Bridges large memory" allocations:

  • LM, for jobs that will run on Bridges' LSM and ESM (3TB and 12TB) nodes

• A "Bridges GPU" allocation allows you to use Bridges' GPU nodes. Partitions available to "Bridges GPU" allocations are:

  • GPU, for jobs that will run on Bridges' GPU nodes, and use one or more full nodes
  • GPU-shared, for jobs that will run on Bridges' GPU nodes, but share a node with other jobs
  • GPU-small, for jobs that will use only one of Bridges' GPU nodes and 8 hours or less of wall time.

• A "Bridges-AI" allocation allows you to you Bridges' Volta GPU nodes. There is one partition available to "Bridges-AI" allocations:

  • GPU-AI, for jobs that will run on Bridges' Volta 16 nodes or the DGX-2.

All the partitions use FIFO scheduling. If the top job in the partition will not fit, SLURM will try to schedule the next job in the partition. The scheduler follows policies to ensure that one user does not dominate the machine. There are also limits to the number of nodes and cores a user can simultaneously use. Scheduling policies are always under review to ensure best turnaround for users.

Partitions for "Bridges regular memory" allocations

There are three partitions available for "Bridges regular memory allocations": RM, RM-shared and RM-small.

Use your allocation wisely: To make the most of your allocation, use the shared partitions whenever possible. Jobs in the RM partition are charged for the use of all cores on a node, and incur Service Units (SU) for all 28 cores. Jobs in the RM-shared partition share nodes, and SUs accrue only for the number of cores they are allocated. The RM partition is the default for the sbatch command, while RM-small is the default for the interact command. See the discussion of the interact and sbatch commands in this document for more information.

Use the appropriate account id for your jobs: If you have more than one Bridges grant, be sure to use the correct SLURM account id for each job. See Charging with multiple grants.

For information on requesting resources and submitting jobs see the discussion of the interact and sbatch commands.

interact -p RM -N 2 -t 30:00

sbatch -p RM -t 5:00:00 -N 1 myscript.job

interact -p RM-shared --ntasks-per-node=4 -t 1:00:00

sbatch -p RM-shared --ntasks-per-node 2 -t 5:00:00 myscript.job

#!/bin/bash
#SBATCH -N 1
#SBATCH -p RM-shared
#SBATCH -t 5:00:00
#SBATCH --ntasks-per-node 2

#echo commands to stdout
set -x

# move to working directory
# this job assumes:
# - all input data is stored in this directory 
# - all output should be stored in this directory

cd /pylon5/groupname/username/path-to-directory

#run OpenMP program
export OMP_NUM_THREADS 2
./myopenmp

interact -p RM-small -N 1 --ntasks-per-node=8 -t 45:00

sbatch -p RM-small -N 2 -t 6:00:00 myscript.job

Partitions for "Bridges large memory" allocations

There is one partition available for Bridges large memory allocations: LM.

Charge your jobs to the correct group: If you have more than one Bridges grant, be sure to charge your usage to the correct one. See Charging with multiple grants.

For information on requesting resources and submitting jobs see the interact or sbatch commands.

interact -p LM --mem=2000GB

sbatch -p LM - t 10:00:00 --mem 6000GB myscript.job

Partitions for "Bridges GPU" allocations

There are three partitions available for "Bridges GPU" allocations: GPU, GPU-shared and GPU-small.

Use your allocation wisely: To make the most of your allocation, use the shared partitions whenever possible. Jobs in the GPU partition use of all cores on a node, and accrue SU costs for every core. Jobs in the GPU-shared partition share nodes, and are only charged for the cores they are allocated.

Charge your jobs to the correct group: If you have more than one Bridges grant, be sure to charge your usage to the correct one. See Charging for multiple grants

For information on requesting resources and submitting jobs see the interact or sbatch commands.

 interact -p GPU --gres=gpu:p100:2 -N 4 -t 30:00

sbatch -p GPU --gres=gpu:k80:4 -N 1 -t 45:00 myscript.job

#!/bin/bash
#SBATCH -N 2
#SBATCH -p GPU
#SBATCH --ntasks-per-node 28
#SBATCH -t 5:00:00
#SBATCH --gres=gpu:p100:2

#echo commands to stdout
set -x

#move to working directory
# this job assumes:
# - all input data is stored in this directory 
# - all output should be stored in this directory 
cd /pylon5/groupname/username/path-to-directory


#run GPU program
./mygpu

interact -p GPU-shared --gres=gpu:k80:4 -t 8:00:00

sbatch -p GPU-shared --gres=gpu:p100:2 -t 1:00:00 myscript.job

#!/bin/bash
#SBATCH -N 1
#SBATCH -p GPU-shared
#SBATCH --ntasks-per-node 7
#SBATCH --gres=gpu:p100:1
#SBATCH -t 5:00:00

#echo commands to stdout
set -x

#move to working directory
# this job assumes:
# - all input data is stored in this directory 
# - all output should be stored in this directory 
cd /pylon5/groupname/username/path-to-directory

#run GPU program
./mygpu

interact -p GPU-small --gres=gpu:p100:2 -t 2:00:00

sbatch -p GPU-small --gres=gpu:p100:2 -t 1:00:00 myscript.job

Partition for "Bridges-AI" allocations

There is one partition available for "Bridges-AI" allocations: GPU-AI. There are two node types available:

• "Volta 16" - nine HPE Apollo 6500 servers, each with 8 NVIDIA Tesla V100 GPUs with 16 GB of GPU memory each, connected by NVLink 2.0
• "Volta 32" - NVIDIA DGX-2 enterprise research AI system tightly coupling 16 NVIDIA Tesla V100 (Volta) GPUs with 32 GB of GPU memory each, connected by NVLink and NVSwitch

We strongly recommend the use of Singularity containers on the AI nodes, especially on the DGX-2. We have installed containers for many popular AI packages on Bridges for you to use, but you can create your own if you like.

For more information on Singularity and the containers available on Bridges:

• The PSC documentation on Singularity
• The Containers section of this guide

Use the appropriate account id for your jobs: If you have more than one Bridges grant, be sure to use the correct SLURM account id for each job. See Charging with multiple grants.

For information on requesting resources and submitting jobs see the interact or sbatch commands.

Using module files on Bridges-AI

The Module package provides for the dynamic modification of a users's environment via module files. Module files manage necessary changes to the environment, such as adding to the default path or defining environment variables, so that you do not have to manage those definitions and paths manually. Before you can use module files in a batch job on Bridges-AI, you must issue the following command:

If you are using the bash or ksh:

source /etc/profile.d/modules.sh

If you are using csh or tcsh:

source /etc/profile.d/modules.csh

See the PSC Module documentation for information on the module command.

GPU-AI partition

When submitting a job to the GPU-AI partition, you must use the --gres=gpu:type:n parameter to specify the type and number of Volta GPUs you will use. Valid options are

• For the Volta 16 nodes, with 16GB of GPU memory, type is volta16; n can be 1-8.
• For the DGX-2, with 32GB of GPU memory, type is volta32; n can be 1-16.

See the sbatch command for an explanation of the --gres option.

Sample interact command for GPU-AI

To run in an interactive session on Bridges-AI, use the interact command and specify the GPU-AI partition. An example interact command to request 1 GPU on a Volta 16 node is:

interact -p GPU-AI --gres=gpu:volta16:1

Where:

• -p indicates the intended partition

• --gres=gpu:volta16:1 requests the use of 1 V100 GPU on an Apollo node

Sample sbatch command for the GPU-AI partition

A sample sbatch command to submit a job to run on one of the Volta 16 nodes and use all eight GPUs would be

sbatch -p GPU-AI -N 1 --gres=gpu:volta16:8 -t 1:00:00 myscript.job

where

• -p GPU-AI requests the GPU-AI partition

• -N 1 requests one node

• --gres=gpu:volta16:8 requests an Apollo server with Volta 100 GPUs , and specifies you will use all 8 GPUs on that node

• -t 1:00:00 requests one hour of running time

myscript.job is your batch script.

Summary of the partition for Bridges' "AI" GPU nodes

See also:

• The interact command
• Batch scripts
• The sbatch command
• OnDemand

Node, partition, and job status information

scancel

The scancel command is used to kill a job in a partition, whether it is running or still waiting to run. Specify the jobid for the job you want to kill. For example,

scancel 12345

kills job # 12345.

See also: scancel man page

sacct

The sacct command can be used to display detailed information about jobs. It is especially useful in investigating why one of your jobs failed. The general format of the command is

sacct -X -j nnnnnn -S MMDDYY --format parameter1,parameter2, ...

• For 'nnnnnn' substitute the jobid of the job you are investigating.
• The date given for the -S option is the date at which sacct begins searching for information about your job.
• The commas between the parameters in the --format option cannot be followed by spaces.

The --format option determines what information to display about a job. Useful parameters are

• JobID
• Partition
• Account - the account charged
• ExitCode - useful in determining why a job failed
• State - useful in determining why a job failed
• Start, End, Elapsed - start, end and elapsed time of the job
• NodeList - list of nodes used in the job
• NNodes - how many nodes the job was allocated
• MaxRSS - how much memory the job used
• AllocCPUs - how many cores the job was allocated

See also: sacct man page

Monitoring memory usage

It can be useful to find the memory usage of your jobs. For example, you may want to find out if memory usage was a reason a job failed.

You can determine a job's memory usage whether it is still running or has finished. To determine if your job is still running, use the squeue command.

squeue -j nnnnnn -O state

where nnnnnn is the jobid.

For running jobs: srun and top or sstat

You can use the srun and top commands to determine the amount of memory being used.

srun --jobid=nnnnnn top -b -n 1 | grep userid

For nnnnnn substitute the jobid of your job. For 'userid' substitute your userid. The RES field in the output from top shows the actual amount of memory used by a process. The top man page can be used to identify the fields in the output of the top command.

• See the man pages for srun and top for more information.

You can also use the sstat command to determine the amount of memory being used in a running job

sstat -j nnnnnn.batch --format=JobID,MaxRss

where nnnnnn is your jobid.

• See the man page for sstat for more information.

For jobs that are finished: sacct or job_info

If you are checking within a day or two after your job has finished you can issue the command

sacct -j nnnnnn --format=JobID,MaxRss

If this command no longer shows a value for MaxRss, use the job_info command

job_info nnnnnn | grep max_rss

Substitute your jobid for nnnnnn in both of these commands.

• See the man page for sacct for more information.

See also: documentation for SLURM, including man pages for all the SLURM commands

Start OnDemand

To connect via OnDemand, point your browser to https://ondemand.bridges.psc.edu.

• You will be prompted for a username and password. Enter your PSC username and password.
• The OnDemand Dashboard will open. From this page, you can use the menus across the top of the page to manage files and submit jobs to Bridges.

To end your OnDemand session, choose Log Out at the top right of the Dashboard window and close your browser.

Manage files

To create, edit or move files, click on the Files menu from the Dashboard window. A dropdown menu will appear, listing all your file spaces on Bridges: your home directory and the pylon directories for each of your Bridges' grants.

Choosing one of the file spaces opens the File Explorer in a new browser tab. The files in the selected directory are listed. No matter which directory you are in, your home directory is displayed in a panel on the left.

There are two sets of buttons in the File Explorer.

Buttons on the top left just below the name of the current directory allow you to View, Edit, Rename, Download, Copy or Paste (after you have moved to a different directory) a file, or you can toggle the file selection with (Un)Select All.

Buttons on the top of the window on the right perform these functions:

• Go To Navigate to another directory or file system
• Open in Terminal Open a terminal window on Bridges in a new browser tab
• New File Creates a new empty file
• New Dir Create a new subdirectory
• Upload Copies a file from your local machine to Bridges
• Show Dotfiles Toggles the display of dotfiles
• Show Owner/Mode Toggles the display of owner and permisson settings

Create and edit jobs

You can create new job scripts and edit existing scripts, and submit those scripts to Bridges through OnDemand.

From the top menus in the Dashboard window, choose Jobs > Job Composer. A Job Composer window will open.

There are two tabs at the top: Jobs and Templates.

In the Jobs tab, a listing of your jobs is gven.

Create a new job script

To create a new job script:

1. Select a template to begin with
2. Edit the job script
3. Edit the job options

Select a template

1. Go to the Jobs tab in the Jobs Composer window. You have been given a default template, named Simple Sequential Job.
2. To create a new job script, click the blue New Job > From Default Template button in the upper left. You will see a green message at the top of the window, "Job was successfully created".

At the right of the Jobs window, you will see the Job Details, including the location of the script and the script name (by default, main_job.sh). Under that, you will see the contents of the job script in a section titled Submit Script.

Edit the job script

Edit the job script so that it has the commands and workflow that you need.

If you do not want the default settings for a job, you must include options to change them in the job script. For example, you may need more time or more than one node. For the GPU partitions, you must specify the type and number of GPUs you want. For the LM partition, you must specify how much memory you need. Use an SBATCH directive in the job script to set these options.

• See the default settings for all partitions
• See how to use SBATCH directives

You can edit the script in several ways.

• Click the blue Edit Files button at the top of the Jobs tab in the Jobs Composer window
• In the Jobs tab in the Jobs Composer window, find the Submit Script section at the bottom right. Click the blue Open Editor button.

After you save the file, the editor window remains open, but if you return to the Jobs Composer window, you will see that the content of your script has changed.

Edit the job options

In the Jobs tab in the Jobs Composer window, click the blue Job Options button. The options for the selected job such as name, the job script to run, and the account it will run under are displayed and can be edited. Click Save or Cancel to return to the job listing.

Submit jobs to Bridges

Select a job in the Jobs tab in the Jobs Composer window. Click the green Submit button to submit the selected job. A message at the top of the window shows whether the job submission was successful or not. If it is not, you can edit the job script or options and resubmit. When the job submits successfully, the status of the job in the Jobs Composer window will change to Queued or Running. When the job completes, the status will change to Completed.

JupyterHub and IJulia

You can run JupyterHub, and IJulia notebooks, through OnDemand. You must do some setup before the first time you run IJulia through OnDemand.

Setup IJulia for OnDemand use

Note: You only need to do this once.

While logged in to Bridges, request an interactive session with access to sites external to Bridges by typing:

interact --egress

Once the sesstion starts, type these commands:

module load anaconda3
module load julia
julia

When Julia starts, type

Pkg.add("IJulia")

When you see the message that IJulia has been installed, you can end your interactive session.

1. Select Interactive Apps > Jupyter Notebooks from the top menu in the Dashboard window.

2. In the screen that opens, specify the time limit, number of nodes, and partition to use. If you have more than one grant on Bridges, you can also designate the account to deduct this usage from. If you will use the LM or one of the GPU partitions, you must add a flag in the Extra Args field for the amount of memory or the number and type of GPUs you want:

   --mem=numberGB

   --gres=gpu:type:number

   See the Running jobs section of this User Guide for more information on Bridges' partitions and the options available.

3. Click the blue Launch button to start your JupyterHub session. You may have to wait in the queue for resources to be available.

4. When your session starts, click the blue Connect to Jupyter button. The Dashboard window now displays information about your JupyterHub session including which node it is running on, when it began, and how much time remains.

   A new window running JupyterHub also opens. Note the three tabs: Files, Running and Clusters.

   Files

   By default you are in the Files tab, and it displays the contents of your Bridges home directory. You can navigate through your home directory tree.

   Running

   Under the Running tab, you will see listed any notebooks or terminal sessions that you are currently running.

5. Now you can start a Jupyter or IJulia notebook:

   a. To start a Jupyter notebook which is stored in your home directory space, in the Files tab, click on its name. A new window running the notebook opens.

   b. To start a Jupyter notebook which is stored in your pylon5 directory, you must first create a symbolic link to it from your home directory. While in your home directory, use a command like:

    ln -s /pylon5/yourgroup/youruserid PYLONDIR
   

   When you enter JuypterHub, you will see the entry PYLONDIR in your list of files under the Files tab. Click on this to be moved to your pylon5 directory.

   c. To start IJulia, in the Files tab, click on the New button at the top right of the file listing. Choose IJulia from the drop down.

Errors

If you get an "Internal Server Error" when starting a JupyterHub session, you may be over your home directory quota. Check the Details section of the error for a line like:

ActionView::Template::Error: Disk quota exceeded @ dir_s_mkdir ...

You can confirm that you are over quota by opening a Bridges shell access window and typing:

du -sh

This command shows the amount of storage in your home directory. Home directory quotas are 10GB. If du -sh shows you are near 10GB, you should delete or move some files out of your home directory. You can do this in OnDemand in the File Explorer window or in a shell access window.

When you are under quota, you can try starting a JupyterHub session again.

Stopping your JupyterHub session

In the Dashboard window, click the red Delete button.

RStudio

You can run RStudio through OnDemand.

1. Select Interactive Apps > RStudio Server from the top menu in the Dashboard window.
2. In the screen that opens, specify the time limit, number of nodes, and partition to use. You can also designate the account to apply this usage to if you have more than one grant on Bridges.
   Use the Extra Args field to ask for specific resources. If you will use the LM partition, you must add a flag in the Extra Args field for the amount of memory you need:
   

   --mem=numberGB

   If you will use one of the GPU partitions, you must add a flag in the Extra Args field for the number and type of GPUs you want:

   --gres=gpu:type:number

   If you want to add additional external packages to your User Library, you must use the `-C EGRESS` flag in the __Extra Args__ field to allow access to external sites:

   -C EGRESS

   See the Running jobs section of this User Guide for more information on Bridges' partitions and the options available.
3. Click the blue Launch button to start your RStudio session. You may have to wait in the queue for resources to be available.
4. When your session starts, click the blue Connect to RStudio Server button. A new window opens with the RStudio interface.

Installed Packages

The Packages tab in the lower right pane of the RStudio interface lists all the packages currently installed in your User Library and in the System Library. To install additional packages into your user library location, click the Install link under the Packages tab. A pop-up window will open asking where to find the package (either the CRAN repository or a Package Archive file).

If you choose CRAN repository, you can type the names of the package(s) you want in the Packages field.

If you choose Package Archive File, you can browse for the file you want. By default your home directory space on Bridges will be shown. To browse through your pylon5 space, click on "..." at the right end of the Home row and enter pylon5/yourgroup in the Path to Folder field in the pop-up window.

Errors

If you exceed the time limit you requested when setting up your RStudio session, you will see this error:

Error: Status code 503 returned

To continue using RStudio, go to Interactive Apps > RStudio from the top menu in the Dashboard window and start a new session.

Stopping your RStudio session

To end your RStudio session, either select File > Quit Session or click the red icon in the upper right of your RStudio window. NOTE that this only closes your RStudio session; it does not close your interactive Bridges session. You are still accruing Service Units. If you like, you can start another RStudio session.

To end your interactive Bridges session so that you are no longer using Service Units, return to the Dashboard window and click the red Delete button.

Shell access

You can get shell access to Bridges by choosing Clusters >> Bridges Shell Access from the top menus in the Dashboard window. In the window that opens, you are logged in to one of Bridges' login nodes as if you used ssh to connect to Bridges.

Miscellaneous

Accessing Bridges documentation

In the Dashboard window, under the Help menu, choose Online Documentation to be taken to the Bridges User Guide.

Change your PSC password

In the Dashboard window, under the Help menu, choose Change HPC Password to be taken to the PSC password change utility.

GPU Nodes

Bridges AI

The Bridges AI resource consists of ten nodes.

• An NVIDIA DGX-2 enterprise research AI system which tightly couples 16 NVIDIA Tesla V100 (Volta) GPUs with 32GB of GPU memory each (512GB/node). The DGX-2 also holds two Intel Xeon Platinum 8168 CPUs with 24 cores/CPU (48 cores total) and 1.5TB RAM. The GPUs are connected by NVLink and NVSwitch, to provide maximum capability for the most demanding of AI challenges.

• 9 NVIDIA Tesla V100 GPU nodes, each with 8 GPUs with 16GB of GPU memory each (128GB/node), on HPE Apollo 6500 servers with 2 Intel Xeon Gold 6148, 20 cores/CPU (40 cores total) and 192GB RAM. The GPUs are connected by NVLink 2.0, to balance great AI capability and capacity.

Bridges GPU

The Bridges GPU resource consists of 48 nodes.

• 16 NVIDIA Tesla K80 GPU nodes on HPE Apollo servers. Each holds two NVIDIA K80 GPU cards with 24GB of GPU memory (48GB/node), and each card contains two GPUs that can be individually scheduled. Each node also has 2 Intel Xeon E5-2695 v3 CPUs (14 cores per CPU) and 128GB RAM.

Ideally, the GPUs are shared in a single application to ensure that the expected amount of on-board memory is available and that the GPUs are used to their maximum capacity. This makes the K80 GPU nodes optimal for applications that scale effectively to 2, 4 or more GPUs. Some examples are GROMACS, NAMD and VASP. Applications using a multiple of 4 K80 GPUs will maximize system throughput.

• 32 NVIDIA Tesla P100 GPU nodes on HPE Apollo 2000 servers. Each node contains 2 NVIDIA P100 GPU cards, and each card holds one very powerful GPU and 16GB of GPU memory (32GB/node). Each nodes also has 2 Intel Xeon E5-2683 v4 CPUs (16 cores per CPU) and 128GB RAM.

These nodes are optimally suited for single-GPU applications that require maximum acceleration. The most prominent example of this is deep learning training using frameworks that do not use multiple GPUs.

See the System configuration section for hardware details for all GPU node types.

File Systems

The /home and pylon5 file systems are available on all of these nodes. See the File Spaces section for more information on these file systems.

Compiling and Running Jobs

Use the GPU partition, either in batch or interactively, to compile your code and run your jobs. See the Running Jobs section for more information on Bridges partitions and how to run jobs.

CUDA

To use CUDA, first you must load the CUDA module. To see all versions of CUDA that are available, type:

module avail cuda

Then choose the version that you need and load the module for it.

module load cuda

loads the default CUDA. To load a different version, use the full module name.

module load cuda/8.0

CUDA 8 codes should run on both types of Bridges GPU nodes with no issues. CUDA 7 should only be used on the K80 GPUs (Phase 1). Performance may suffer with CUDA 7 on the P100 nodes (Phase 2).

OpenACC

Our primary GPU programming environment is OpenACC.

The PGI compilers are available on all GPU nodes. To set up the appropriate environment for the PGI compilers, use the module command:

module load pgi

Read more about the module command at PSC.

If you will be using these compilers often, it will be useful to add this command to your shell initialization script.

There are many options available with these compilers. See the online man pages

• man pgf90
• man pgcc
• man pgCC

for detailed information.

You may find these basic OpenACC options a good place to start:

pgcc -acc yourcode.c
pgf90 -acc yourcode.f90

P100 node users should add the -ta=tesla,cuda8.0 option to the compile command, for example:

pgcc -acc -ta=tesla,cuda8.0 yourcode.c

Adding the -Minfo=accel flag to the compile command (whether pgf90, pgcc or pgCC) will provide useful feedback regarding compiler errors or success with your OpenACC commands.

pgf90 -acc -Minfo=accel yourcode.f90

Hybrid MPI/GPU Jobs

To run a hybrid MPI/GPU job, use the following commands for compiling your program:

module load cuda
module load mpi/pgi_openmpi
mpicc -acc yourcode.c

When you execute your program you must first issue the above two module load commands.

Profiling and Debugging

For CUDA codes, use the command line profiler nvprof. See the CUDA document for more information.

For OpenACC codes, the environment variables PGI_ACC_TIME, PGI_ACC_NOTIFY and PGI_ACC_DEBUG can provide profiling and debugging information for your job. Specific commands depend on the shell you are using.

A Unix shell is a command-line interpreter that provides a traditional user interface for Unix and Unix-like systems.

The two major shell types are the Bourne shell and the C shell. Each type has its own commands and syntax.

The default shell on Bridges is bash, a Bourne-type shell. Other shells, including some C-type shells, are available for you to use if you prefer.

Performance profiling

Enable runtime GPU performance profiling:

export PGI_ACC_TIME=1 [bash]
setenv PGI_ACC_TIME 1 [csh]

Debugging

Basic debugging; For data transfer information, set PGI_ACC_NOTIFY to 3:

export PGI_ACC_NOTIFY=1 [bash]
setenv PGI_ACC_NOTIFY 1 [csh]

More detailed debugging:

export PGI_ACC_DEBUG=1 [bash]
setenv PGI_ACC_DEBUG 1 [csh]

File Systems

/home

The /home file system, which contains your home directory, is available on all Bridges Hadoop nodes.

HDFS

The Hadoop filesystem, HDFS, is available from all Hadoop nodes. There is no explicit quota for the HDFS, but it uses your $SCRATCH disk space. Please delete any files you don't need when your job has ended.

Files must reside in HDFS to be used in Hadoop jobs. Putting files into HDFS requires these steps:

1. Transfer the files to the namenode with scp or sftp
2. Format them for ingestion into HDFS
3. Use the hadoop fs -put command to copy the files into HDFS. This command distributes your data files across the cluster's datanodes.

The hadoop fs command should be in your command path by default.

Documentation for the hadoop fs command lists other options. These options can be used to list your files in HDFS, delete HDFS files, copy files out of HDFS and other file operations.

To request the installation of data ingestion tools on the Hadoop cluster send email to bridges@psc.edu.

Accessing the Hadoop/Spark cluster

To start using Hadoop and Spark with Yarn and HDFS on Bridges, connect to the login node and issue the following commands:

interact -N 3 # you will need to wait until resources are allocated to you before continuing
module load hadoop
start-hadoop.sh

Your cluster will be set up and you'll be able to run hadoop and spark jobs. The cluster requires a minimum of three nodes (-N 3). Larger jobs may require a reservation. Please contact bridges@psc.edu if you would like to use more than 8 nodes or run for longer than 8 hours.

Please note that when your job ends, your HDFS will be unavailable so be sure to retrieve any data you need before your job finishes.

Web interfaces are currently not available for interactive jobs but can be made available for reservations.

Spark

The Spark data framework is available on Bridges. Spark, built on the HDFS filesystem, extends the Hadoop MapReduce paradigm in several directions. It supports a wider variety of workflows than MapReduce. Most importantly, it allows you to process some or all of your data in memory if you choose. This enables very fast parallel processing of your data.

Python, Java and Scala are available for Spark applications. The pyspark interpreter is especially effective for interactive, exploratory tasks in Spark. To use Spark you must first load your data into Spark's highly efficient file structure called Resilient Distributed Dataset (RDD).

Extensive online documentation is available at the Spark website. If you have questions about or encounter problems using Spark, send email to bridges@psc.edu.

Spark example using Yarn

Here is an example command to run a Spark job using yarn. This example calculates pi using 10 iterations.

spark-submit --class org.apache.spark.examples.SparkPi --master yarn \
    --deploy-mode cluster $SPARK_HOME/examples/jars/spark-examples_2.11-2.1.0.jar 10

To view the full output:

yarn logs -applicationId yarnapplicationId

where yarnapplicationId is the yarn applicationId assigned by the cluster.

A Simple Hadoop Example

This section demonstrates how to run a MapReduce Java program on the Hadoop cluster. This is the standard paradigm for Hadoop jobs. If you want to run jobs using another framework or in other languages besides Java send email to bridges@psc.edu for assistance.

Follow these steps to run a job on the Hadoop cluster. All the commands listed below should be in your command path by default. The variable HADOOP_HOME should be set for you also.

1. Compile your Java MapReduce program with a command similar to:

   hadoop com.sun.tools.javac.Main WordCount WordCount.java

   where:

   • WordCount is the name of the output directory where you want your class file to be put
   • WordCount.java is the name of your source file
2. Create a jar file out of your class file with a command similar to:

   jar -cvf WordCount.jar -C WordCount/ .

   where:

   • WordCount.jar is the name of your output jar file
   • WordCount is the name of the directory which contains your class file
3. Launch your Hadoop job with the "hadoop" command

   Once you have your jar file you can run the hadoop command to launch your Hadoop job. Your hadoop command will be similar to

   hadoop jar WordCount.jar org.myorg.WordCount \/datasets/compleat.txt $MYOUTPUT

   where:

   • Wordcount.jar is the name of your jar file
   • org.myorg.WordCount specifies the folder hierarchy inside your jar file. Substitute the appropriate hierarchy for your jar file.
   • /datasets/compleat.txt is the path to your input file in the HDFS file system. This file must already exist in HDFS.
   • $MYOUTPUT is the path to your output file, which will be saved in the HDFS file system. You must set this variable to the output file path before you issue the hadoop command.

After you issue the hadoop command your job is controlled by the Hadoop scheduler to run on the datanodes. The scheduler is currently a stricty FIFO scheduler. If your job turnaround is not meeting your needs send email to bridges@psc.edu.

When your job finishes, the hadoop command will end and you will be returned to the system prompt.

Other Hadoop Technologies

An entire ecosystem of technologies has grown up around Hadoop, such as HBase and Hive. To request the installation of a different package send email to bridges@psc.edu.

Containers available on Bridges

We have installed many containers from the NVIDIA GPU Cloud (NGC) on Bridges. These containers are fully optimized, GPU-accelerated environments for AI, machine learning and HPC. They can be used on the Bridges-AI (Volta 16 and DGX-2) nodes and on some RM-GPU nodes (P100 GPU only).

See the PSC documentation on Singularity for more details on Singularity use on Bridges.

Creating a container

Singularity is the only container software supported on Bridges. You can create a Singularity container, copy it to Bridges and then execute your container on Bridges, where it can use Bridges' compute nodes and filesystems. In your container you can use any software required by your application: a different version of CentOS, a different Unix operating system, any software in any specific version needed. You can install your Singularity container without any intervention from PSC staff.

See the PSC documentation on Singularity for more details on producing your own container and Singularity use on Bridges.

Singularity images

We have installed many of the NVIDIA GPU Cloud (NGC) containers as Singularity images on Bridges. These containers have been optimized for Volta and Pascal architectures by NVIDIA, including rigorous quality assurance.

These containers can all be found on Bridges in the directory /pylon5/containers/ngc/package-name, e.g., /pylon5/containers/ngc/caffe. A link to further documentation is included below the table for each package. The naming convention for these containers includes the year of the release as the first two digits and month as the next two, so release 18.10 was created in October 2018.

See also:

• Additional information about the NVIDIA NGC Registry
• Much more detail on the NGC containers

Why use a VM?

If you need a persistent environment you need to use a virtual machine (VM). Examples of a need for a persistent environment are a Web server with a database backend or just a persistent database.

If you can use a Singularity container rather than a VM you should use the container. You can set up your Singularity container yourself without any intervention by PSC staff. Also, a VM is charged for usage the entire time the VM is set up, whether or not it is being actively used. Containers are only charged for the time during which they are executing, because they are not persistent computing environments. A Singularity environment only exists while you are executing it. Thus, VMs are much more expensive in terms of SUs used than Singularity containers.

A VM provides you with control over your environment, but you will have access to the computing power, memory capacity and file spaces of Bridges.

Common uses of VMs include hosting database and web servers. These servers can be restricted just to you or you can open them up to outside user communities to share your work. You can also connect your database and web servers and other processing components in a complex workflow.

VMs provide several other benefits. Since the computing power behind the VM is a supercomputer, sufficient resources are available to support multiple users. Since each VM acts like an independent machine, user security is heightened. No outside users can violate the security of your independent VM. However, you can allow other users to access your VM if you choose.

A VM can be customized to meet your requirements. PSC will set up the VM and give you access to your database and web server at a level that matches your requirements.

To discuss whether a VM would be appropriate for your research project, send email to bridges@psc.edu.

Downtime

VMs are affected by system downtime, and will not be available during an outage. Scheduled downtimes are announced in advance.

Data backups

It is your responsiblity to backup any important data to another location outside of the VM. PSC will make infrequent snapshots of VMs for recovery from system failure, but cannot be responsible for managing your data.

Grant expiration

When your grant expires, your VM will be suspended. You have a 3-month grace period to request via email to bridges@psc.edu that it be reactivated so that you can move data from the VM. Three months after your grant expires, the VM will be removed. Please notify bridges@psc.edu if you need help moving your data during the grace period.

You can request a VM by submitting the Virtual Machine Request Form.

Publicly available datasets

Some data collections are available to anyone with a Bridges account. They include:

ImageNet

ImageNet is an image dataset organized according to WordNet hierarchy. See the ImageNet website for complete information.

Available on Bridges at /pylon5/datasets/community/imagenet

Natural Languge Tool Kit Data

NLTK comes with many corpora, toy grammars, trained models, etc. A complete list of the available data is posted at: https://www.nltk.org/nltk_data/

NLTK is available on Bridges at /pylon2/datasets/community/nltk.

MNIST

Dataset of handwritten digits used to train image processing systems.

Available on Bridges at /pylon5/datasets/community/mnist

Genomics Data

Several genomics datasets are publicly available.

BLAST

The BLAST databases can be accessed through the environment variable $BLASTDB after loading the BLAST module.

CAMI

CAMI (Critical Assessment of Metagenome Interpretation) is a community-led initiative designed to help tackle challenges in metagenome assembly and analysis by aiming for an independent, comprehensive and bias-free evaluation of methods. Data from the first CAMI challenge is available at /pylon5/datasets/community/genomics/cami.

RepBase

RepBase is the most commonly used database of repetitive DNA elements. You must register with RepBase and send proof of registration in order to use the RepBase database.

UCSC

The University of California Santa Cruz reference genomes are available at /pylon5/datasets/community/genomics/UCSC. The collection includes human, mouse and drosophila genomes.

Other genomics datasets

Other available datasets are typically used with a particular genomics package. These include:

XSEDE supported research on Bridges

We ask that you use the following text:

This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system, which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC).

Please include these citations:

Towns, J., Cockerill, T., Dahan, M., Foster, I., Gaither, K., Grimshaw, A., Hazlewood, V., Lathrop, S., Lifka, D., Peterson, G.D., Roskies, R., Scott, J.R. and Wilkens-Diehr, N. 2014. XSEDE: Accelerating Scientific Discovery. Computing in Science & Engineering. 16(5):62-74. IEEE Computer Society.

Nystrom, N. A., Levine, M. J., Roskies, R. Z., and Scott, J. R. 2015. Bridges: A Uniquely Flexible HPC Resource for New Communities and Data Analytics. In Proceedings of the 2015 Annual Conference on Extreme Science and Engineering Discovery Environment (St. Louis, MO, July 26-30, 2015). XSEDE15. ACM, New York, NY, USA. ACM Digital Library.

Download Bibtex entry

Download EndNote entry

Additional Support

Please also acknowledge support provided through XSEDE's Extended Collaborative Support Services (ECSS) and/or by PSC staff.

Other research on Bridges

For research on Bridges supported by programs other than XSEDE, such as PRCI, we ask that you use the following text:

This work used the Bridges system, which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC).

Please include this citation:

Nystrom, N. A., Levine, M. J., Roskies, R. Z., and Scott, J. R. 2015. Bridges: A Uniquely Flexible HPC Resource for New Communities and Data Analytics. In Proceedings of the 2015 Annual Conference on Extreme Science and Engineering Discovery Environment (St. Louis, MO, July 26-30, 2015). XSEDE15. ACM, New York, NY, USA. ACM Digital Library.

Download Bibtex entry

Download EndNote entry

Additional support

Please also acknowledge any support provided by PSC staff.

ECSS Support

To acknowledge support provided through XSEDE's Extended Collaborative Support Services (ECSS), please use this text:

We thank [consultant name(s)] for [his/her/their] assistance with [describe tasks such as porting, optimization, visualization, etc.], which was made possible through the XSEDE Extended Collaborative Support Service (ECSS) program.

Please include this citation:

Wilkins-Diehr, N and S Sanielevici, J Alameda, J Cazes, L Crosby, M Pierce, R Roskies. High Performance Computer Applications 6th International Conference, ISUM 2015, Mexico City, Mexico, March 9-13, 2015, Revised Selected Papers Gitler, Isidoro, Klapp, Jaime (Eds.) Springer International Publishing. ISBN 978-3-319-32243-8, 3-13, 2016. 10.1007/978-3-319-32243-8.

PSC Support

If PSC staff contributed substantially to software development, optimization, or other aspects of the research, they should be considered as coauthors.

When PSC staff contributions do not warrant coauthorship, please acknowledge their support with the following text:

We thank consultant name(s) for his/her/their assistance with describe tasks such as porting, optimization, visualization, etc.

Security Measures

Security is very important to PSC. These policies are intended to ensure that our machines are not misused and that your data is secure.

What You Can Do

You play a significant role in security! To keep your account and PSC resources secure, please:

• Be aware of and comply with PSC policies on security, use and privacy found in this document
• Choose strong passwords and don't share them between accounts or with others. More information can be found in the PSC password policies.
• Utilize your local security team for advice and assistance
• Take the online XSEDE Cybersecurity Tutorial. Go to Online Training and click on "XSEDE Cybersecurity (CI-Tutor)"
• Keep your computer properly patched and protected
• Report any security concerns to the PSC help desk ASAP by calling the PSC hotline at: 412-268-6350 or email remarks@psc.edu

What We Will Never Do

• PSC will never send you unsolicited emails requesting confidential information.
• We will also never ask you for your password via an unsolicited email or phone call.

Remember that the PSC help desk is always a phone call away to confirm any correspondence at 412-267-6350.

If you have replied to an email appearing to be from PSC and supplied your password or other sensitive information, please contact the help desk immediately.

What You Can Expect

• We will send you email when we need to communicate with you about service outages, new HPC resources, and the like.
• We will send you email when your password is about to expire and ask you to change it by using the web-based PSC password change utility.

Other Security Policies

• PSC password policies
• Users must connect to PSC machines using ssh in order to avoid remote logins with clear text passwords
• We vigilantly monitor our computer systems and network connections for security violations
• We are in close contact with the CERT Coordination Project with regard to possible Internet security violations

Reporting Security Incidents

To report a security incident you should contact our Hotline at 412-268-6350. To report non-emergency security incidents you can send email to remarks@psc.edu.

PSC resource abuse policy

The Pittsburgh Supercomputing Center's computing resources are vital to the scientific community, and we have a responsibility to ensure that these resources are utilized in a responsible manner. The term computing resources in this case refers to all computers owned or operated by PSC and all hardware, data, software, storage systems and communications networks associated with these computers. Improper use of PSC systems is generally referred to as resource abuse.

Users of PSC's systems are subject to applicable Commonwealth of Pennsylvania and federal laws. Abuse of resources will be referred to the PSC User Services manager and/or the appropriate local, state and federal authorities, at PSC's discretion. Furthermore, PSC may terminate or restrict any user's access to its systems, without prior notice, if such action is necessary to maintain computing availability and security for other users of the systems.

Resource abuse includes, but is not limited to:

• using, or attempting to use, the center's computing resources without prior authorization or for unauthorized purposes
• tampering with or obstructing the operation of the computing resources, or attempting to do so
• inspecting, modifying, distributing, or copying privileged data or software without proper authorization, or attempting to do so
• supplying, or attempting to supply, false or misleading information or identification in order to access PSC's computing resources.

If there is any doubt regarding the legitimacy or authorization of any action, contact PSC User Services.

Privacy

Pittsburgh Supercomputing Center is committed to preserving your privacy. This privacy policy explains exactly what information is collected when you visit our site and how it is used.

This policy may be modified as new features are added to the site. Any changes to the policy will be posted on this page.

• Any data automatically collected from our site visitors - domain name, browser types, etc. - are used only in aggregate to help us better meet site visitors' needs.
• There is no identification of individuals from our aggregate data. Therefore, unless you choose otherwise, you are totally anonymous when visiting our site.
• We do not share data with anyone for commercial purposes.
• If you choose to submit personally identifiable information to us electronically via the PSC feedback page, email, etc., we will treat it with the same respect for privacy afforded to mailed submissions. Submission of such information is always optional.

PSC respects individual privacy and takes great effort in supporting web site privacy policy outlined above. Please be aware, however, that we publish URLs of other sites on our web site that may not adhere to the same policy.

Applying for a Bridges account

How to prepare a request, what to request, when requests can be submitted, examples of successful requests

Is Bridges a good fit for my research?

Bridges was built to facilitate research ranging from traditional HPC areas like astronomy and physics through emerging fields like genomics to decision science, natural language processing and digital humanities. Bridges could be a good fit for you if:

You want to scale your research beyond the limits of your laptop, using familiar software and user environments.

You want to collaborate with other researchers with complementary expertise.

Your research can take advantage of any of the following:

• Rich data collections – Rapid access to data collections will support their use by individuals, collaborations and communities.
• Cross-domain analyses_ – Concurrent access to datasets from different sources, along with tools for their integration and fusion, will enable new kinds of questions.
• Gateways and workflows_ – Web portals will provide intuitive access to complex workflows that run "behind the scenes".
• Large coherent memory_ – Bridges' 3TB and 12TB nodes will be ideal for memory-intensive applications, such as genomics and machine learning.
• In-memory databases_ – Bridges' large-memory nodes will be valuable for in-memory databases, which are important due to their performance advantages.
• Graph analytics_ – Bridges' hardware-enabled shared memory nodes will execute algorithms for large, nonpartitionable graphs and complex data very efficiently.
• Optimization and parameter sweeps_ – Bridges is designed to run large numbers of small to moderate jobs extremely well, making it ideal for large-scale optimization problems.
• Rich software environments_ – Robust collections of applications and tools, for example in statistics, machine learning and natural language processing, will allow researchers to focus on analysis rather than coding.
• Data-intensive workflows_ – Bridges' filesystems and high bandwidth will provide strong support for applications that are typically I/O bandwidth-bound. One example is an analysis that runs best with steps expressed in different programming models, such as data cleaning and summarization with Hadoop-based tools, followed by graph algorithms that run more efficiently with shared memory.
• Contemporary applications_ – Applications written in Java, Python, R, MATLAB, SQL, C++, C, Fortran, MPI, OpenACC, CUDA and other popular languages will run naturally on Bridges.

What is Bridges' hardware configuration?

The Bridges virtual tour depicts Bridges architecture and illustrates features which could be used in various research models.

View the Bridges virtual tour

Bridges comprises 5 classes of compute nodes with additional dedicated nodes for databases, webservers and data transfer. Several types of filesystems with different functions are available. Bridges components are interconnected by the Intel Omni-Path Fabric.

Bridges has 5 classes of compute nodes:

• 752 Regular Shared Memory (RSM) nodes, each with 2 Intel Xeon EP-series CPUs and 128GB of RAM
• 48 "RSM-GPU" nodes: 16 nodes with 2 NVIDIA Tesla K80 GPUs, 2 Intel Xeon EP-series CPUs, and 128GB of RAM each, and 32 nodes with 2 NVIDIA Tesla P100 GPUs, 2 Intel Xeon EP-series CPUs, and 128GB of RAM each
• 10 "GPU-AI" nodes: 9 nodes with 8 Volta V100 GPUs each, 16GB of GPU memory per GPU and 192 GB of RAM per node, and a DGX-2, with 16 Volta V100 GPUs, 32GB of GPU memory per GPU and 1.5TB of RAM
• 42 Large Shared Memory (LSM) nodes, HP DL580 servers with 4 Intel Xeon EX-series CPUs and 3TB of RAM
• 4 Extreme Shared Memory (ESM) nodes, HP Integrity Superdome X servers with 16 Intel Xeon EX-series CPUs and 12TB of RAM

Bridges' database nodes are dual-socket Xeon servers with 128GB of RAM. Some contain solid-state disks to deliver high IOPs for latency-sensitive workloads and others contain banks of hard disk drives for capacity-oriented workloads.

Bridges' webserver nodes are dual-socket Xeon servers with 128GB of RAM and are connected to PSC's wide-area network, including XSEDE and commodity Internet. They are implemented in virtual machines to provide security, allow maximum use of Bridges' resources and grant project-specific customization of the web server configuration.

Bridges' data transfer nodes are dual-socket Xeon servers with 128Gb of RAM and 10 GigE connections to PSC's wide-area network, enabling high-performance data transfers between Bridges and XSEDE, campuses, instruments and other advanced cyberinfrastructure.

Bridges supports a shared parallel filesystem for persistent storage, node-local filesystems and memory filesystems.

The shared parallel filesystem, named Pylon, is a high-bandwidth, high-capacity centralized, parallel system cross-mounted across Bridges' nodes. Pylon is modeled on other PSC production filesystems, including the one on the Data Exacell. It is entirely disk-based with high-level RAID providing data safety. Pylon will have approximately 10PB of storage and bandwidth to the system of approximately 180GB/s.

Node-local filesystems are available on each compute node. They provide natural support for Hadoop and software layers that need it, applications, frameworks, etc. that distribute data or are written to shard, and applications that benefit from local "scratch" storage. Node-local filesystems also improve bandwidth and performance consistency to Pylon.

Memory filesystems are supported on Bridges' compute nodes, especially the ESM and LSM nodes. They provide maximum IOPs and bandwidth to improve the performance of applications such as pipelined workflows, genome sequence assembly and in-memory instantiations of otherwise disk-based databases.

Bridges' components are interconnected by the Intel Omni-Path Fabric, which delivers 100Gbps line speed, low latency, excellent scalability and improved tolerance to data errors. A unique two-level "island" topology, designed by PSC, maximizes performance for the intended workloads. Compute islands provide full bi-section bandwidth communication performance to applications spanning up to 42 nodes. Storage islands take advantage of the Intel Omni-Path Fabric to implement mutiple paths and provide optimal bandwidth to the Pylon filesystem. Storage switches are cross-linked to all other storage switches and connect management nodes, database nodes, web server nodes and data transfer nodes.

Who is eligible to apply for an allocation on Bridges?

Allocations for Bridges are open to faculty members, postdocs, and researchers at a U.S.-based institution. Eligible institutions include federal research labs or commercial organizations; however, special rules may apply if your institution is not a university or a two- or four-year college.

The Principal Investigator (PI) for an allocation may not be a high school or undergraduate student; a qualified advisor, e.g., a high school teacher or faculty member, must serve in this capacity. In most cases, a graduate student may not be a PI. A post-doctoral researcher is eligible to be a PI.

Note that a streamlined application process exists for faculty, staff, and postdocs at Carnegie Mellon University and the University of Pittsburgh. See the Pittsburgh Research Computing Inititative for details.

For all others, full details on eligibility are available here:

https://portal.xsede.org/allocations/policies#22
https://portal.xsede.org/allocations/policies#80

What types of allocations are available on Bridges?

Four types of allocations for non-proprietary research are available on Bridges. We recommend all new investigators first get a startup allocation before submitting a full research proposal.

• Startup grants are appropriate for groups that are just getting familiarized with XSEDE resources. They are used to prepare a group to submit a full research proposal.
• Research grants are for full production research on Bridges.
• Educational allocations are to be used by students in classroom settings.
• Campus Champion allocations are for dedicated personnel who assist researchers at their campus to utilize XSEDE's HPC resources for their respective research. Read more information on the Campus Champions program.

Allocations for propietary research are available through the PSC Corporate Affiliates Program.

How can I apply for a Bridges account?

If you are:

• Faculty at Carnegie Mellon University or the University of Pittsburgh, you can get access to Bridges through the Pittsburgh Research Computing Initiative.
• A researcher or educator at a US academic or non-profit research institution, you can apply for Bridges access through the NSF's XSEDE program. To apply through XSEDE, request an allocation on Bridges through the XSEDE User Portal(XUP. You will need to create an XSEDE portal account before you can apply if you don't already have one.
  • Navigate to https://portal.xsede.org
  • Click on the "Sign In" button on the top-right of the screen
  • At the next screen, you will see a link under the sign in button to "Create Account"
  • Once you have created the account, login
  • Navigate to the "Allocations" tab
  • Click on "Submit/Review Request"
  • On the next page, you will see the option to begin a submission. We recommend all new investigators first get a startup allocation before submitting a full research proposal
• Doing proprietary research, you can get access to Bridges through the PSC Corporate Affiliates Program.

When can I apply?

Four types of applications are accepted at any time:

• Pittsburgh Research Computing Initiative accounts, for faculty at Carnegie Mellon University and the University of Pittsburgh
• Startup XSEDE allocations, the easiest way to get started with XSEDE resources and recommended for all new XSEDE users. See https://portal.xsede.org/allocations/startup for more information.
• Campus Champion accounts
• Corporate Affiliate accounts

Requests for Research XSEDE allocations are accepted four times a year:

• Mar 15 - Apr 15

• Jun 15 - Jul 15

• Sep 15 - Oct 15

• Dec 15 - Jan 15

See https://portal.xsede.org/allocations/research for more information.

How do I prepare an XSEDE allocation request?

There are detailed instructions on the XSEDE User Portal(XUP) explaining how to prepare an allocation request. These resources may be helpful:

A video on how to write and submit a successful XSEDE allocation proposal

A sample resource request for Bridges

Detailed instructions and examples of requests for:

• A Startup grant
• A Research grant
• An Education grant

What do I ask for in a Bridges allocation request?

At a minimum, you will request computing time and a storage allocation on Pylon, Bridges' persistent storage system. If you want to use GPU nodes, you must request a GPU allocation.

See an example of an XSEDE resource request for Bridges. The "Resource Justification" section may be particularly helpful in quantifying your request.

Computing time : You will request computing time on "Bridges" for Regular memory nodes, "Bridges GPU" nodes, Bridges "AI-GPU" nodes, "Bridges Large" nodes (3TB - 12TB RAM), or any combination, depending on what your application needs.

Bridges "Regular memory" nodes are appropriate for applications needing up to 128GB of cache-coherent shared memory. Bridges "GPU" nodes are for applications which want to take advantage of GPU processors, and the "GPU-AI" nodes can support highly complex deep learning models with extremely large data sets. "Large memory" nodes can accommodate applications requiring up to 12TB of cache-coherent shared memory.

Computing time allocations are given in terms of Service Units (SU).

For Bridges' "Regular Memory" nodes, SUs are defined in terms of core-hours:

The use of one core for 1 hour = 1 SU

For "Bridges GPU" nodes, SUs are defined in terms of GPU-hours. Because of the difference in the performance between the K80 and P100 nodes, the charges are different for the two types of nodes.

For Bridges' K80 GPU nodes, the use of 1 GPU for 1 hour = 1 SU

For Bridges' P100 GPU nodes, the use of 1 GPU for 1 hour = 2.5 SUs

For Bridges' "GPU-AI" nodes, SUs are defined in terms of GPU-hours:

The use of one GPU for 1 hour = 1 SU

For "Bridges' Large" nodes, SUs are defined in terms of memory-hours:

The use of 1TB of memory for 1 hour = 1 SU

For more information on accounting for Bridges' use and examples of calculating SUs, see the Account Administration section.

Storage allocation: You must also request a storage allocation on Pylon, a parallel filesystem shared across all of Bridges' nodes. The default Pylon allocation is 512GB. If you need more than that, you should justify it in your allocation request.

Other: You must also note in your request if you need big data frameworks like Hadoop, or virtual machines (if your research requires persistent databases or webservers, e.g., for gateways or community datasets).

How should I estimate the resources I will need for an XSEDE allocation request?

If you need experience with large memory HPC to help you estimate the resources your research requires, you can request a Start-up allocation on Bridges for benchmarking. You can also ask for help from XSEDE's Extended Collaborative Support Services.

If you have questions about resources for persistent databases, gateways or other types of distributed applications, please contact PSC User Services.

See an example of an XSEDE resource request for Bridges. The "Resource Justification" section may be particularly helpful in quantifying your request.

Do I need to make any special requests in addition to computing time on Bridges?

If your research requires any of the following, be sure to specifically ask for (and justify) them in your allocation request:

• Big data frameworks like Spark and Hadoop
• Virtual machines
• More than 512GB of storage

Can I see some examples of successful proposals?

Example proposals can be found here:

• For research example proposals, see https://portal.xsede.org/allocations/research#examples
• For startup example proposals, see https://portal.xsede.org/allocations/startup#examples
• For educational example proposals, see https://portal.xsede.org/allocations/education#examples

What's the maximum allocation I can request?

For startup allocations, the limits are:

• Bridges Regular Memory: 50,000 core-hours
• Bridges GPU: 2,500 GPU-hours
• Bridges GPU-AI: 1,500 GPU-hours
• Bridges Large Memory: 1,000 TB-hours

For research allocations, there are no limits for the size of the request. Note, however, that all requests must be extremely detailed and justified.

How long does it take for a proposal to be reviewed?

Startups and educational proposals take about 2 weeks to be reviewed and processed, depending on the size of the request.

Research allocation proposals are reviewed quarterly. They must be submitted about 3 months prior to the start date. See "When can I apply?" for the submission deadlines. Decisions are announced a few weeks before the start date. For example, if a proposal is submitted by January 1 for a start date of April 1, a notification regarding the outcome will be sent around the week of March 15.

Where can I get more information?

If you would like to hear more about Bridges' capabilities or discuss how you can take advantage of Bridges in your research, call us at 412-268-4960 or email bridges@psc.edu.

Account administration

Handling multiple grants, tracking usage, adding/removing users from a grant, renewing or extending grants

How can I add (or remove) users to my grant?

Adding or removing users from an XSEDE grant is done through the XSEDE User Portal. For instructions, see https://portal.xsede.org/allocations/managing.

How do I designate an allocation manager for my project?

Currently there are two ways to add an allocation manager, and also two separate "definitions" of an allocation manager.

The first allocation manager is designated when the proposal is submitted. This allocation manager will have the same permissions as the PI/co-PI: they can submit supplements, transfers, extensions, renewals, etc. and also add or remove users on the allocation.

The second type of allocation manager can be designated at any time.

1. Log in to the XSEDE User Portal
2. Choose "Allocations/Usage" from the top menu, then the "Manage Users" button.

This allocation manager will only be able to add or remove users. They will not have the permissions to make actions (supplements, transfers, extensions, etc.) on the allocation.

How do I reset my (PSC or XSEDE) password?

To reset your PSC password:

1. Navigate to https://apr.psc.edu.
2. Enter your PSC username, and the email linked to your PSC account into the form.
   1. A security code will be sent to the associated email address.
3. Once you receive the code, copy and paste the code into the web form.
4. Reset your password on the next screen.

You have 15 minutes to complete the process or you will have to start over.

To reset your XSEDE password:

1. Navigate to the XSEDE User Portal.
2. Click the "Sign In" button on the top-right of the screen.
3. Enter your XSEDE username and the email you have linked with your XSEDE account into the form.
   1. A security code will be sent to the associated email address.
4. Once you receive the code, copy and paste the code into the web form.
5. Reset your password on the next screen.

How do I find my allocation/proposal number?

Your proposal number can be found on the confirmation email you received after you submitted a proposal. If you cannot locate this email, you can login to the XSEDE User Portal and click on "Allocations/Usage" under the "My XSEDE" tab. The proposal number will be listed next to the title of the allocation. Please note that if your allocation is still under review, you will not be able to find this information here.

How can I check the balance on my allocation?

To check the balance of your allocation, login to the XSEDE User Portal and click on "Allocations/Usage" under the "My XSEDE" tab. The next screen will display the allocations you have access to, along with current usage and end dates.

What happens to my files when my grant expires?

Your files are retained for 3 months after your grant expires. Once the 3 month grace period is over, they are deleted.

You will be able to log in during the grace period so that you can transfer your files elsewhere, but you will not be able to submit jobs or create new files.

How do I request additional SUs for my project?

If you would like more SUs to complete your project, you can request a supplement.

To submit a supplement request to your startup, research, or educational allocation,:

1. Login to the XSEDE User Portal.
2. Click on "Submit/Review Request" under the "Allocations" tab. The next screen will display your approved allocation.
3. Choose the "Supplement" action.

Please note that the option for submitting a supplement will not be available if less than 30 days remain on the allocation.

How do I request a calendar extension for my allocation?

If you need more time to complete your project, you can request a calendar extension.

To submit an extension request to your startup, research, or educational allocation:

1. Login to the XSEDE User Portal.
2. Click on "Submit/Review Request" under the "Allocations" tab. The next screen will display your approved allocation.
3. Choose the "Extension" action.

Please note this option is only available when 90 days or less remains on the allocation. You will have the option to submit an extension of 3 or 6 months.

How do I renew my Bridges allocation?

To submit a renewal for a Bridges startup allocation:

1. Login to the XSEDE User Portal.
2. Click on "Submit/Review Request" under the “Allocations” tab.
3. On the next screen, use the link on the left side of the page to begin your renewal request.

Please note the option for startup renewal will not be available until 30 days or less remains on the allocation. A progress report will be required, in addition to the other mandatory documents.

Renewal proposals for research allocations are only accepted during quarterly submission windows (see the submission schedule). Your renewal proposal is due during the same window in which you submitted in the previous year.

To start the renewal submission:

1. Login to the XSEDE User Portal.
2. Click on "Submit/Review Request" under the "Allocations" tab.
3. On the next screen use the link on the left side of the page to begin your renewal request.

Please note that a progress report will be required, in addition to the other mandatory documents. You must also attach at least one publication to the submission.

How do I transfer my allocation to another resource?

To submit a transfer request:

1. Login to the XSEDE User Portal.
2. Click on "Submit/Review Request" under the "Allocations" tab. The next screen will display your approved allocation.
3. Choose the "Transfer" action.
   1. Be sure to use the XSEDE SU Converter when preparing your transfer request since all resources are not equal.
   2. Also be sure to enter a positive number for the resource(s) you would like to transfer to and a negative number for the resource(s) you would like to transfer from.

I have multiple grants. How do I charge the correct one for a job?

There are two parameters to consider: the SLURM account id, which determines which grant the SUs used by a job are deducted from, and the Unix group, which determines which group owns any files created by the job.

See Managing multiple grants in the Account Admininstration section of the Bridges User Guide for information on determing your default SLURM account id and Unix group, and changing them either permanently or temporarily, for one job or login session.

Connecting to Bridges

Connection problems

Why can't I connect?

1. If you are receiving a "Permission denied" error message, you might need to reset your password. You can do that through the form at PSC.
2. If your password reset fails, you may be using a different userid or the email address than what we have for you in our user database. To check, contact bridges@psc.edu for verification.
3. If you think your password is correct, your account may have expired more than 3 months ago. Contact bridges@psc.edu to check on the status of your account.
4. Another possibility is that your IP address may have been been banned from connecting because you typed your password incorrectly several times or some network anomaly occurred when you tried to connect. To check if your IP address was banned send email to bridges@psc.edu with your public IP address. You can determine the latter at WhatIsMyIp.Org.

Containers

Using Singularity containers on Bridges

Do I need a container?

Containers are stand-alone packages holding the the software needed to create a very specific computing environment.

If you need a very specialized computing environment, you can create a container as your work space on Bridges. Currently, Singularity is the only type of container supported on Bridges. Docker is not supported.

However, in most cases, Bridges has all the software you will need. Before creating a container for your work, check the extensive list of software that has been installed on Bridges. While logged in to Bridges, you can also get a list of installed packages by typing

module avail

If you need a package that is not available on Bridges, you can request that it be installed by emailing bridges@psc.edu. You can also install software packages in your own file spaces and, in some cases, we can provide assistance if you encounter difficulties.

Can I use a Docker container on Bridges?

No. Singularity is the only container environment supported on Bridges.

See the PSC documentation on Singularity.

Are there containers installed on Bridges that I can use?

Yes.

We have installed many containers from the NVIDIA GPU Cloud for use on the V100 and P100 GPUs. These containers have been optimized for these architectures and provide GPU-accelerated software for AI, machine learning and HPC.

See the Singularity images installed on Bridges.

See the PSC documentation on Singularity.

Filesystems

Quotas, what happens when a grant expires, common errors

I just got an account but I don't have a pylon5 directory. Why not?

You probably don't have a pylon allocation.

Access to pylon storage is not automatically given with allocations. Typically this occurs when you have been added to an existing allocation. The PI must specifically request that additional users being added to a grant also get a pylon allocation.

Ask your PI to double check that a pylon allocation was requested for you.

What happens to my files when my grant expires?

Three months after your grant expires all of your Bridges files associated with that grant will be deleted, no matter which file space they are in. You will be able to login during this 3-month period to transfer files, but you will not be able to run jobs or create new files.

Running jobs

Common error messages, why isn't my job running

My job failed with "Invalid qos specification". What does that mean?

"Invalid qos specification" means that you asked for a resource that you do not have access to. For example:

• submitting a job to one of the GPU partitions when you don't have a Bridges_GPU allocation
• submitting a job to one of the LM partitions when you don't have a Bridges_Large allocation

Please check your grants to see what you have been allocated. The projects command will list your grants and the resources allocated to each one.

This error can also occur when you have more than one active grant. It's important to run jobs under the correct SLURM account id. Your jobs will run under your default SLURM account id unless you specify a different account id to use for a job. If the default grant does not have access to the resources you are requesting, you will get this error.

Use the projects command to find which grant is your default if you have more than one.

See

The Account administration section of the Bridges user Guide for information on finding your SLURM account ids, setting a nondefault account id for a specific job, and changing your default account id permanently.

When will my job run?

The scheduler on Bridges is largely FIFO. The squeue command lists the running and queued jobs on Bridges in FIFO order. However, jobs can move up in the queue if a slot becomes available on the machine and the job will fit in the open slot without disrupting the FIFO ordering of the other jobs in the queue. In addition, jobs can finish before their requested walltime for a variety of reasons. It is not possible to predict with any accuracy when your job will run.

Why won't my tensorflow program run?

For help running Tensorflow on Bridges:

• See the tensorflow page for detailed information
• Check the example batch scripts on Bridges in the directory /opt/packages/examples/tensorflow. For more information and instructions, see the README file in the appropriate subdirectory.

In general, you must load a tensorflow module and then activate your Python virtual environment.

To see all the available tensorflow modules, type

module avail tensorflow

Load the appropriate module with

module load tensorflow/<version>

After the module is loaded, activate the Python virtual environment. Typically this is done with the source activate command.

How do I improve my turnaround?

There are several techniques you can use to try to improve your turnaround, although, since the scheduler is basically FIFO with backfill, you may just have to wait your turn.

1. Estimate your walltime as closely as possible without underestimating.
2. Use flexible walltime by using the --time-min option to the sbatch command. The use of this option is described in the Bridges User Guide.
3. Since only the first N jobs in the queue---and N varies based on the load on the system---are considered for scheduling, use job packing to combine your jobs. The use of job packing is described in the Bridges User Guide.
4. If you can conveniently complete a computing task on your local system, you can run the task there.
5. Since Bridges runs a fairshare scheduler, if you have run a lot of jobs recently the priority of your current jobs in the queue will be reduced slightly until they have accumulated some waiting time in the queue. This fact may influence how you spread out running your jobs.

Can I reserve Bridges nodes for my exclusive use?

Reservations on Bridges are set-asides for a specific grant of a specific set of nodes for a specific time period. They may be granted under special circumstances, typically for jobs that require real-time or interactive processing (e.g. of streaming external data, student exercises etc.).

If you feel that you have such a use case, you may apply for a reservation using the Reservation Request form at least 48 hours before the proposed start time. A user consultant will contact you about your request.

Note that it may not be possible to honor all reservation requests. In addition, modifications to your request may be necessary, depending on the overall processing demands on the system.

If your reservation is accepted, your Bridges grant will be charged for all specified nodes for the entire specified time period, regardless of the actual usage your jobs incur.

Example: A reservation for 1024 RM cores that starts Monday 8:00 am EST and ends Wednesday 8:00 am EST will be charged 102448 SUs even if you run no jobs on Tuesday.*

Why hasn't my job run yet?

There can be many reasons that a job is waiting in the queue.

• There are jobs in front of you in the queue. If the status field in the output from the command squeue -l has 'Priority' in the status field, this is the case.
• There is a maximum amount of nodes your RM jobs can cummulatively request. The maximum limit varies based on the load on the system. If the status field in the squeue -l output says 'QOSMaxCPUPerUserLimit', this is the case.
• There is a maximum amount of cores your RM-shared jobs can cummulatively request. This maximum limit varies based on the load on the system. If the status field in the squeue -l output says 'QOSMaxCPUPerUserLimit', this is the case.
• There is a maximum number of GPU nodes your GPU jobs can simultaneously use. This limit applies to the p100 and k80 nodes combined. If the status field in the squeue -l output says 'QOSMaxGRESPerUser', this is the case.
• The partition you want to use is down. If the status field in the squeue -l output says 'PartitionDown', this is the case.
• There are currently no nodes available to run your job. This can be because
  • They are already running jobs
  • There are a lot of reserved nodes. The sinfo command shows reserved nodes.
  • There are a lot of down nodes. The sinfo command shows down nodes.
  • They are being reserved for a job with a reservation that is about to start
  • They are being reserved for a job at which a system drain is targeted. If the status field in the squeue -l output says 'ReqNodeNotAvailable', this is the case. The output is somewhat misleading because it will list all nodes on the machine which are unavailable to run jobs, even nodes on which your job could not run because they are in a different partition.

How do I set an environment variable?

Many applications require environment variables to be set before they will run as you intend, or even at all. The PSC-supplied modules set many of the necessary environment variables for a package, but in some cases you need to set additional environment variables. The command to use depends on the shell type you are using.

If you are using a shell in the bash family of shells, set an an environment variable with a command similar to

export VAR1 = value1

If you are using a shell in the C-shell family of shells, set an environment variable with a command similar to

setenv VAR1 value1

I have multiple grants. How do I charge the correct one for a job?

There are two parameters to consider: the SLURM account id, which determines which grant the SUs used by a job are deducted from, and the Unix group, which determines which group owns any files created by the job.

See Managing multiple grants in the Account Admininstration section for information on determing your default SLURM account id and Unix group, and changing them either permanently or temporarily, for one job or login session.

Software

What software is available on Bridges; common errors with a given package

Why can't I run Gaussian?

You must get permission before you can use Gaussian on PSC systems.

Complete the PSC Gaussian permission form.

We will notify you when you have been granted access to Gaussian.

What software is installed on Bridges?

Check the PSC software pagefor the list of software installed on Bridges.

How many CPU tasks should I request for a Gromacs job that uses both CPUs and GPUs?

To run Gromacs on both GPUs and CPUs, use the same number of CPU tasks as GPUs. Thus, no matter how many nodes you use, set the value of the SBATCH option ntasks-per-node to 4 if you are using the K80 GPU nodes and to 2 if you are using the P100 GPU nodes.

You must also use the correct Gromacs module to insure that your compilation will work.

To use GPUs and CPUs, load a gromacs module with "gpu" in its name, similar to

module load gromacs/2018_gpu

To use just CPUs, load a gromacs module with "cpu" in its name, similar to

module load gromacs/2018_cpu

There are complete sample scripts for both cases in directory /opt/packages/examples/gromacs on Bridges.