XSEDE provides the Globus service for file transfer and sharing. Designed specifically for researchers, Globus provides fast, reliable, and secure file transfer among XSEDE resources or between an XSEDE resource and another system (such as a campus cluster, lab server, or personal computer). Beyond file transfer, Globus allows researchers to securely share data with collaborators, and to publish data for broader access as required by many data management plans.
Use Globus to:
- Move data between any two systems: in XSEDE, at your campus or laboratory, or at your home. Easily move small numbers of very large files (even terabyte-sized) as well as large numbers of small files (thousands or more at a time).
- Share data with your colleagues and collaborators, whether the data is stored at your laboratory on campus, your office, your home, or on selected XSEDE resources.
- You don't have to move your data to a public cloud service to share it.
- You have full control over what data is shared and with whom.
- You can share data with colleagues who aren't registered with XSEDE.
Globus services are available to everyone with an XSEDE User Portal account.
- Use the Globus interfaces described below, including the Web interface (used by most people), the command-line interface (used by advanced users and developers), and the REST APIs (used by application developers).
- move files to and from any XSEDE resource on which you have a current allocation (permission to use).
- download and install the Globus Connect Personal software on your personal computer to transfer files between it and XSEDE or other systems.
- Enable sharing on your Globus Connect Personal endpoints. (See below.)
- Download and install the Globus Connect Server software on any of your personal or campus systems and use it to provide multi-user file transfer access on those systems. File sharing on multi-user systems is limited to XSEDE service providers and is otherwise available with a Globus subscription.
Make sure that Globus knows you are registered with XSEDE. This will allow you to use any of the features listed above. First, open a web browser and navigate to the Globus website, https://www.globus.org.
Click the "Login" button at the top-right of the page.
- If you've recently used Globus, you'll be automatically logged in. See Add XSEDE to your Globus Profile.
- If you haven't used Globus recently (or ever), you will be prompted to pick an organization where you already have an account. Type or select "XSEDE" to specify that you want to use your XSEDE identity.
When you click "Continue," you will be asked to enter your XSEDE username and password. The web page should have the familiar XSEDE interface and should have an address beginning with "weblogin.xsede.org/".
Once you enter your XSEDE username and password, you will be logged into Globus and you should see the main file transfer page shown in Figure 4.
If you've used Globus before, then you already have a Globus profile. Add your XSEDE identity to it to enable the XSEDE features described here.
After you are logged into Globus, click the "Account" button in the upper-right of the web page. You will see a list of the identities that Globus has associated with your profile. If XSEDE isn't already listed, click "Add Linked Identity" in the upper-right of the web page. You will be asked to select an organization. Type or select "XSEDE" to specify that you want to use your XSEDE identity.
You should see the XSEDE login page shown in Figure 3, where you can enter your XSEDE username and password and then give Globus permission to access your XSEDE identity information. Once you successfully authenticate to XSEDE, you will have access to all of XSEDE's Globus features as described below.
Once you are logged in to Globus, the Transfer Files web interface (see Figure 4) is self-explanatory and you will probably understand how to use it without further help. If you need some hints or further details, the Globus website provides an excellent Getting Started guide. The next few sections explain how to find XSEDE's systems within Globus so you can move data to and from them, and how to access your local systems (personal or campus computers).
As you can see in Figure 4, Globus lets you move files to and from endpoints. All XSEDE resources have Globus endpoints that allow you to move files to and from them. All of XSEDE's endpoints have the word "XSEDE" in their names, making them easy to look up. Just click the "start here" box in Globus, and a panel will appear where you can type the endpoint's name. Type "XSEDE," and all of the XSEDE endpoints will appear, as shown in Figure 6. XSEDE keeps its endpoints up-to-date in Globus on a routine basis. If the resource you are looking for is especially new or is no longer online and you can't find it in the list, use the XSEDE Help Desk to inquire about its availability. Select the endpoint you want to move files to or from, and you will be connected to that system and shown the files and directories there.
You can easily create a Globus endpoint on your own computer so that you can upload or download files. To do this, download and install the Globus Connect Personal software. (Globus Connect Personal is available for Windows, Mac, and Linux systems.) You can set Globus Connect Personal up as a background task (so that it is always running in the background) or you can start and stop it like any other application. While Globus Connect Personal is running, your system will be available in Globus's transfer interface (but only to you!) and you will be able to start transfers between it and any other Globus endpoint, including XSEDE resources.
Note that you can start and stop Globus Connect Personal on your system whenever you need to, even shutting your system down, hibernating, and moving from one network to another. Globus will automatically find your system and continue any active transfers whenever you are connected to a network and have the tool running. Globus Connect Personal is designed to work with most firewalls and NAT devices. It does not require administrative privileges to run on your system.
Many colleges, universities, and national research institutions and laboratories offer Globus endpoints for their systems. To find an endpoint associated with an organization, try typing the name of the organization in the "start here" endpoint box in Globus. You may be surprised how many organizations are already listed and available. You will typically need to have an account with the organization to access its endpoints.
Any multi-user HPC or shared storage system can be configured as a Globus endpoint using Globus Connect Server. On most servers, installing Globus Connect Server requires just a few commands. Once the software is installed, any user with a local account on that system can move and share files between it and an XSEDE resource. Information on Globus Connect Server is available on the Globus documentation web site.
Globus provides a user command-line interface as well as the REST API programmatic interface.
Globus provides a command line interface (CLI) that may be accessed using any standard SSH terminal client. Prior to running shell commands, you must upload your public SSH key to your Globus account. See Getting started with the Command Line Interface for more details.
Once your key is uploaded, launch a secure Globus shell:
$ ssh firstname.lastname@example.org
You will see the Globus command prompt:
Welcome to globusonline.org, . Type "help" for help. $ _
Commonly used commands include:
- Find endpoints matching a keyword with endpoint-search "keyword"
- Start a transfer with scp
- Display the transfer tasks status with status
Information on using Globus commands is available in the Using the Command Line Interface guide. A complete list of Globus commands, along with with detailed descriptions is available in the CLI command reference.
When using the CLI, you may find it helpful to turn off email notifications. Issue the following Globus CLI command:
$ profile -n off
This command will also disable alert emails, such as a notification that your credentials have expired, requiring you to keep a slightly closer watch on long-running transfer requests.
Globus also provides programmatic access to the file transfer mechanism via a full-featured REST API. The API is a great way to integrate Globus services into your custom portal or science gateway. A Python SDK is available, along with many examples of open source code. A good starting point for working with the API is the Globus Jupyter notebook.
You are also encouraged to use the developer mailing list to engage directly with the Globus engineering team and other developers.
The Globus service is quite powerful and includes many features not mentioned above (e.g., group management, monitoring, endpoint management, etc.) You can learn more using the following resources:
- Getting started with Globus file transfer
- Frequently Asked Questions
- Sharing data using Globus
- Getting started with the Globus command line interface (CLI)
- Globus REST API
Last update: February 14, 2017