System Configuration

Data Oasis consists of 64 Object Storage Servers (OSSs) each contributing 72 TB of capacity and connected to two Arista 7508 switches via 2×10GigE links.

On Gordon, the Data Oasis Project Storage is mounted via 64 I/O nodes which act also as routers. The Gordon compute nodes are connected using QDR Infiniband to the I/O nodes, and each I/O node is connected to Data Oasis's Arista 7508 switches via 2×10GigE links.

Allocations

The default allocation is 500 GB of Project Storage to be shared among all users of a project. Projects that require more than 500 GB of Project Storage must request additional space by sending an email to help@xsede.org. This e-mail from the project PI should provide a 500 words-or-less justification that includes:

1. Amount of storage being requested
2. How the storage will be used
3. How this augments other non-SDSC storage resources available to the project

The projects storage requests will be reviewed by SDSC staff, and a decision will be made within 5 business days.

Methods of Access

The Data Oasis Project Storage space is mounted on both Gordon and Comet and can be accessed as a filesystem on all login and compute nodes. Each user's personal space can be found in

/oasis/projects/nsf/allocationname/username

where allocationname is the project's six-character allocation name (found by running the show_accounts command) and username is the user's local login name.

Since Data Oasis is mounted as a standard filesystem, UNIX file transfer utilities such as scp, sftp, and rsync can be used for transfers of modest size or scale. To enable the efficient transfer of larger amounts of data, Data Oasis is also mounted on the SDSC data mover servers:

• oasis-dm.sdsc.edu (for users with accounts on Comet)

These data movers can be used in conjunction with Globus Online and GridFTP, which are discussed in the Data Transfer Methods section below and the XSEDE Data Transfers & Management page.

Data Transfer Methods

While using the standard UNIX file transfer tools (scp, sftp, rsync) is acceptable for simple and small file transfers (< 1 GB) to and from Data Oasis, they cannot realize the maximum performance of the Data Oasis storage resource because of their limited internal buffers and inability to stripe transfers across multiple data mover servers. The preferred method for transferring big data (both large file sizes and large numbers of files) is using GridFTP (a part of the Globus Toolkit). Keep in mind that attempting to transfer large numbers of small files will result in poor performance. Whenever possible, create archives of directories with large file counts before initiating the data transfers.

The XSEDE Data Transfers & Management page provides a detailed explanation of how to use GridFTP and its associated GUI- and terminal-based tools in XSEDE. To facilitate GridFTP with SDSC Data Oasis, the following data mover has Data Oasis mounted under /oasis/projects/nsf:

• Comet: gsiftp://oasis-dm.sdsc.edu:2811/ (XUP File Manager/globus-url-copy) or xsede#comet (Globus)

The data mover is load-balanced in a round-robin fashion, but advanced users may wish to access the individual data movers explicitly via oasis-dm, oasis-dm1, oasis-dm2, oasis-dm3, and oasis-dm4.

Examples

globus-url-copy provides the greatest flexibility for optimizing transfers between XSEDE resources. To transfer a file from another XSEDE resource (e.g., TACC Stampede) to SDSC Gordon,

    $ module load globus
    $ myproxy-login -l xsedeusername

This will load the commands to use GridFTP and generate the GSI credential needed to access xsedeusername's accounts across XSEDE Resources. Then,

    $ globus-url-copy -vb -stripe -tcp-bs 8m -p 4 \
        gsiftp://data1.stampede.tacc.utexas.edu:2811///home1/02255/username/somefile.bin \
        gsiftp://oasis-dm.sdsc.xsede.org:2811///oasis/projects/nsf/allocation/username/somefile.bin

where

• "-vb" enables verbosity (report transfer rate, among other things)
• "-stripe" enables striped transfers
• "-tcp-bs 8m" specifies a 8 megabyte TCP buffer. The optimal value for this will vary; Globus provides a way to estimate the optimal tcp-bs value in its documentation
• "-p 4" indicates that four parallel data connections should be used

By comparison, the equivalent transfer using scp would be:

    $ scp login1.stampede.tacc.utexas.edu:/home1/02255/username/somefile.bin \
        /oasis/projects/nsf/allocation/username/

In the case of a 341 MB file transfer test case, GridFTP achieved an average 171 MB/s while scp achieved only 34.1 MB/s. When transferring terabytes of data, GridFTP is clearly preferable.

Caveats to Users

This resource is based on a Lustre filesystem which has some limitations. A comprehensive list of Lustre best-practices is beyond the scope of this guide, but it is important to minimize unnecessary access of file metadata. For example,

• avoid performing many small file operations: opens/closes, random reads/writes
• avoid putting too many (e.g., more than several hundred) files in one directory
• avoid using "ls -l" unnecessarily, and consider using "ls --color=no -U" when navigating Data Oasis
• limit unnecessary use of wildcards on the command line
• avoid using the "find" and "du" commands. Use "lfs find" and "lfs du" instead

The "lfs" command is available by default on Gordon and can be loaded using "module load lustre" on Comet.

Troubleshooting / Common errors

• Problem: Any attempts to access files on Data Oasis just hang OR access is extremely sluggish/unresponsive
  Solution: This can occur on both login nodes and compute nodes and typically results from Data Oasis being overloaded. These conditions typically "un-hang" within a few minutes; if they persist for longer, contact help@xsede.org with the system (or specific compute nodes) on which this is occurring.
• Problem: /oasis/projects/nsf exists but is empty
  Solution: This problem is infrequent and should be reported to the XSEDE helpdesk with the system (or specific compute nodes) on which this is occurring.