Globus at XSEDE

XSEDE provides the Globus service for file transfer and sharing. Designed specifically for researchers, Globus provides fast, reliable, and secure file transfer among XSEDE resources or between an XSEDE resource and another system (such as a campus cluster, lab server, or personal computer). Beyond file transfer, Globus allows researchers to securely share data with collaborators, and to publish data for broader access as required by many data management plans.

Use Globus to:

  • Move data between any two systems: in XSEDE, at your campus or laboratory, or at your home. Easily move small numbers of very large files (even terabyte-sized) as well as large numbers of small files (thousands or more at a time).
  • Share data with your colleagues and collaborators, whether the data is stored at your laboratory on campus, your office, your home, or on selected XSEDE resources.

With Globus:

  • You don't have to move your data to a public cloud service to share it.
  • You have full control over what data is shared and with whom.
  • You can share data with colleagues who aren't registered with XSEDE.
Figure 1. The Globus website

Globus services are available to everyone with an XSEDE User Portal account.

  • Use the Globus interfaces described below, including the Web interface (used by most people), the command-line interface (used by advanced users and developers), and the REST APIs (used by application developers).
  • move files to and from any XSEDE resource on which you have a current allocation (permission to use).
  • download and install the Globus Connect Personal software on your personal computer to transfer files between it and XSEDE or other systems.
  • Enable sharing on your Globus Connect Personal endpoints. (See below.)
  • Download and install the Globus Connect Server software on any of your personal or campus systems and use it to provide multi-user file transfer access on those systems. File sharing on multi-user systems is limited to XSEDE service providers and is otherwise available with a Globus subscription.

Use XSEDE to access Globus

Make sure that Globus knows you are registered with XSEDE. This will allow you to use any of the features listed above. First, open a web browser and navigate to the Globus website, https://www.globus.org.

  1. Click the "Login" button at the top-right of the page.

    • If you've recently used Globus, you'll be automatically logged in. See Add XSEDE to your Globus Profile.
    • If you haven't used Globus recently (or ever), you will be prompted to pick an organization where you already have an account. Type or select "XSEDE" to specify that you want to use your XSEDE identity.
  2. When you click "Continue," you will be asked to enter your XSEDE username and password. The web page should have the familiar XSEDE interface and should have an address beginning with "weblogin.xsede.org/".

  3. Once you enter your XSEDE username and password, you will be logged into Globus and you should see the main file transfer page shown in Figure 4.

Add XSEDE to your Globus Profile

If you've used Globus before, then you already have a Globus profile. Add your XSEDE identity to it to enable the XSEDE features described here.

After you are logged into Globus, click the "Account" button in the upper-right of the web page. You will see a list of the identities that Globus has associated with your profile. If XSEDE isn't already listed, click "Add Linked Identity" in the upper-right of the web page. You will be asked to select an organization. Type or select "XSEDE" to specify that you want to use your XSEDE identity.

Figure 5. Link your XSEDE identity to an existing Globus profile

You should see the XSEDE login page shown in Figure 3, where you can enter your XSEDE username and password and then give Globus permission to access your XSEDE identity information. Once you successfully authenticate to XSEDE, you will have access to all of XSEDE's Globus features as described below.

Move files

Once you are logged in to Globus, the Transfer Files web interface (see Figure 4) is self-explanatory and you will probably understand how to use it without further help. If you need some hints or further details, the Globus website provides an excellent Getting Started guide. The next few sections explain how to find XSEDE's systems within Globus so you can move data to and from them, and how to access your local systems (personal or campus computers).

Find XSEDE's Resources in Globus

As you can see in Figure 4, Globus lets you move files to and from endpoints. All XSEDE resources have Globus endpoints that allow you to move files to and from them. All of XSEDE's endpoints have the word "XSEDE" in their names, making them easy to look up. Just click the "start here" box in Globus, and a panel will appear where you can type the endpoint's name. Type "XSEDE," and all of the XSEDE endpoints will appear, as shown in Figure 6. XSEDE keeps its endpoints up-to-date in Globus on a routine basis. If the resource you are looking for is especially new or is no longer online and you can't find it in the list, use the XSEDE Help Desk to inquire about its availability. Select the endpoint you want to move files to or from, and you will be connected to that system and shown the files and directories there.

Find your local computers in Globus

You can easily create a Globus endpoint on your own computer so that you can upload or download files. To do this, download and install the Globus Connect Personal software. (Globus Connect Personal is available for Windows, Mac, and Linux systems.) You can set Globus Connect Personal up as a background task (so that it is always running in the background) or you can start and stop it like any other application. While Globus Connect Personal is running, your system will be available in Globus's transfer interface (but only to you!) and you will be able to start transfers between it and any other Globus endpoint, including XSEDE resources.

Note that you can start and stop Globus Connect Personal on your system whenever you need to, even shutting your system down, hibernating, and moving from one network to another. Globus will automatically find your system and continue any active transfers whenever you are connected to a network and have the tool running. Globus Connect Personal is designed to work with most firewalls and NAT devices. It does not require administrative privileges to run on your system.

Find other multi-user research systems in Globus

Many colleges, universities, and national research institutions and laboratories offer Globus endpoints for their systems. To find an endpoint associated with an organization, try typing the name of the organization in the "start here" endpoint box in Globus. You may be surprised how many organizations are already listed and available. You will typically need to have an account with the organization to access its endpoints.

Any multi-user HPC or shared storage system can be configured as a Globus endpoint using Globus Connect Server. On most servers, installing Globus Connect Server requires just a few commands. Once the software is installed, any user with a local account on that system can move and share files between it and an XSEDE resource. Information on Globus Connect Server is available on the Globus documentation web site.

Share data

You can share files from a Globus endpoint with your colleagues and collaborators. Your collaborators will receive an email message from you inviting them to login to Globus and access the files you've shared with them. You can give them read-only or both read and write access to your files. They can log into Globus using their campus credentials (with InCommon-participating campuses), their XSEDE account (if they have one), or a new Globus account.

Share your personal servers

To enable sharing on your own system running Globus Connect Personal, join XSEDE's "XSEDE Globus Plus Users" group. When logged in to Globus, click the "Groups" button in the top bar and select "Search for Groups" as shown in Figure 7.

Figure 7. Search for the XSEDE Globus Plus Users group

You will be taken to a search page. Type "XSEDE Globus Plus" in the search box and press Enter. Click on the "XSEDE Globus Plus Users" group in the search results box, and you will be shown brief information about the group and given a button labeled "Join Group" on the right side of the page, as shown in Figure 8.

Figure 8. Join the XSEDE Globus Plus Users group

When you click the "Join Group" button, you will be asked for some very brief information about yourself (name, email organization, etc.). The first question on the form is, "Select which username to join as." Click the pop-up list and select your XSEDE identity if it is available in the list. (If your XSEDE identity isn't in the list, you may use another identity to join.) Click the button "Submit Application to Join." You will receive an email message shortly (during normally staffed hours) letting you know that your request has been approved.

Once Globus Plus has been enabled for your account, follow Globus's instructions for enabling sharing on your system and creating a Shared Endpoint.

Share files across XSEDE ecosystem

As of January 2017, only the XSEDE resources managed by the San Diego Supercomputing Center and the Wrangler system at TACC and Indiana University allow sharing on their endpoints. If you would like to use sharing on any other XSEDE endpoint, please contact the XSEDE Help Desk.

Share files on other systems

To enable sharing on any other multi-user system running Globus Connect Server (such as a shared campus system), you or your campus will need a Globus subscription. Many campuses have subscriptions already; enquire with your local staff to find out if your systems include Globus and sharing.

Globus Interfaces

Globus provides a user command-line interface as well as the REST API programmatic interface.

Command Line Interface

Globus provides a command line interface (CLI) that may be accessed using any standard SSH terminal client. Prior to running shell commands, you must upload your public SSH key to your Globus account. See Getting started with the Command Line Interface for more details.

Once your key is uploaded, launch a secure Globus shell:

$ ssh globususername@cli.globusonline.org

You will see the Globus command prompt:

Welcome to globusonline.org, .  Type "help" for help.
$ _

Commonly used commands include:

  • Find endpoints matching a keyword with endpoint-search "keyword"
  • Start a transfer with scp
  • Display the transfer tasks status with status

Information on using Globus commands is available in the Using the Command Line Interface guide. A complete list of Globus commands, along with with detailed descriptions is available in the CLI command reference.

When using the CLI, you may find it helpful to turn off email notifications. Issue the following Globus CLI command:

$ profile -n off

This command will also disable alert emails, such as a notification that your credentials have expired, requiring you to keep a slightly closer watch on long-running transfer requests.

REST API

Globus also provides programmatic access to the file transfer mechanism via a full-featured REST API. The API is a great way to integrate Globus services into your custom portal or science gateway. A Python SDK is available, along with many examples of open source code. A good starting point for working with the API is the Globus Jupyter notebook.

You are also encouraged to use the developer mailing list to engage directly with the Globus engineering team and other developers.

References

The Globus service is quite powerful and includes many features not mentioned above (e.g., group management, monitoring, endpoint management, etc.) You can learn more using the following resources:

Last update: February 14, 2017

No comments yet. Be the first.