Jetstream User Guide
Last update: September 10, 2019

System Overview

Jetstream is an NSF-funded (NSF-1445604), user-friendly cloud environment designed to give researchers access to interactive computing and data analysis resources on demand, whenever and wherever they want to analyze their data.

It provides a library of virtual machines designed to do discipline-specific scientific analysis. Software creators and researchers will also be able to create their own customized virtual machines (VM) or their own private computing system within Jetstream.

Jetstream features a web-based user interface based on the popular Atmosphere cloud computing environment developed by CyVerse (formerly known as the iPlant Collaborative) and extended to support science and engineering research generally. The operational software environment is based on OpenStack.

Accessing Jetstream

Jetstream is accessible through a web interface using XSEDE credentials via Globus Auth.
Jetstream is NOT accessible via the XSEDE Single Sign-On Login Hub.
Newly created XSEDE accounts must be added to a specific allocation by the PI or Resource manager in order to access Jetstream.

Jetstream is meant primarily for interactive research, small scale processing on demand, or as the backend to science gateways to send research jobs to other HPC or HTC resources.

Jetstream is different in that you can work with GUIs that you couldn't otherwise use on most HPC systems.

Jetstream may be used for prototyping, for creating tailored workflows to either use at smaller scale with a handful of CPUs or to port to larger environments after doing your proof of concept work at a smaller level.

Jetstream is not a typical High Performance Computing (HPC) or High Throughput Computing (HTC) environment and won't be used for large scale parallel processing or high-throughput computing.

Figure 1: System overview of Jetstream environment

Key Features

Jetstream utilizes Atmosphere, an easy to use web application, on-demand environment that is designed to accommodate computationally and data-intense research tasks, including Infrastructure as a Service (IaaS) with advanced APIs; Platform as a Service (PaaS) for developing and deploying software to the science community; and Software as a Service (SaaS).

Some of the key features include:

  • Access virtual machine images preconfigured with an operating system and software to help you do scientific computations in domain-specific tasks
  • Find and use tools with the intuitive self-service portal
  • Easily manage virtual machines
  • Publish your own software suites, create your own work environments, and run the software for community use
  • Integrate with existing infrastructure components using API services
  • Easily generate and manage statistical reporting of user resources for total CPU hours and memory usages, total instances and applications launched by user, cloud monitoring, and on-demand intelligence resource allocation

Within Atmosphere, you launch an instance (a launched image of a virtual machine), selecting from the list of available images (a template of a virtual machine containing an installed operating system, software, and configuration). It is recommended that you familiarize yourself with the Linux command line as some actions require some degree of knowledge of the Command Line Interface (CLI).

System Configuration

The computing environment consists of two homogenous clusters at Indiana University and Texas Advanced Computing Center (TACC) with a test environment at the University of Arizona. The system provides over 1/2 a PetaFLOPS of computational capacity and 2 petabytes of block and object storage. The individual nodes contain two Intel "Haswell" processors, 128 GB of RAM, 2 terabytes of local storage, and 10 gigabit Ethernet networking. The system leverages 40 gigabit Ethernet for network aggregation and each of the production clusters connect to Internet2 at 100 Gbps. The physically distributed system allows Jetstream to be highly available and resilient. GlobusAuth is used for large-scale file transfer and authentication.

Figure 2: System configuration of Jetstream environment

System Configuration Aggregate information Per Node (Compute Node)
Machine type Dell (Various) Dell M630 Nodes
Operating system RHEL/CentOS/Ubuntu CentOS
Memory model N/A N/A
Processor cores 7,680 per site 24
CPUs 640 per site 2
Nodes 320 per site -
RAM 40TB 128GB
Network 100gbps to Internet2
40gbps x 4 to local infrastructure
10gbps to XSEDE
10gbps
Local Storage 640TB 2TB
Storage information Aggregate information Per Node
File systems Block storage for VMs, other storage options forthcoming Varies
Total disk space 640TB Local
960TB Attached Storage
2TB local
Total scratch space N/A N/A

Hardware Configuration

The underlying cloud providers have hardware with

  • 10 Gbps network connectivity from the compute hosts to the cloud's internal network infrastructure
  • 4-fold 40 Gbps uplinks from the cloud infrastructure to the site's enterprise infrastructure
  • 100 Gbps connectivity from the site infrastructure to the Internet2 backbone
  • 10 Gbps redundant connectivity to the XSEDE research network
  • Individual instances have full access to this infrastructure with no added speed limits.

Security Policies

Atmosphere

Public IP: For instances managed by the Atmosphere user interface, all instances are currently provided access to an optional public IP address.

Open ports: Current Atmosphere security policy only opens ports 22, 80, and 443 or ports > 1024 by default.

All other ports are closed by default.

Default open ports may be closed through local instance based security controls, though care should be taken to not disrupt essential Jetstream services.

API

Public IP: All API instances have only private network IP addresses by default. Public IP addresses can be requested.

Open ports: All API instances have no open security by default.

Control of networking and security for API instances is described in the API (OpenStack command line) documentation.

XSEDE Service Units

Jetstream allocations are measured in XSEDE Service Units (SUs). On Jetstream, SUs are consumed at a rate of 1 SU per vCPU_core-hour (use of one virtual core of a CPU per hour). For more information on XSEDE Service Units, see the XSEDE KB document "On XSEDE, how are compute jobs charged?"

SU cost per hour for each Jetstream VM size is outlined in Virtual Machine (VM) Sizes and Configurations.

Currently, only instances that have an Active green status light decrement SUs at the full rate.

Starting October 1, 2018:

SUSPENDED instances are decremented only 0.75 SUs per vCPU_core-hour (75%).
STOPPED instances are decremented only 0.50 SUs per vCPU_core-hour (50%).
SHELVED instances are not decremented SUs. (0%)
ACTIVE-DEPLOYING instances are not decremented SUs. (0%)

Figure 3: Jetstream instance with Active status

Note: To use the system you must have a valid Jetstream allocation. Allocations are processed through the eXtreme Science and Engineering Discovery Environment (XSEDE). For information on how to apply see Jetstream Allocations.

Image name Description

BioLinux 8

CentOS 6 MATLAB

  • Tags: Featured, CentOS, desktop, development, gui, m1_medium, VNC
  • Created: 8/2/2016
  • Requires a Medium VM size or larger

CentOS 6 Rstudio

  • Tags: Featured, ...
  • Tags: Featured, CentOS, desktop, development, gui, R, Rstudio, VNC
  • Created: 8/30/2016
  • Based on CentOS 6.8 - Microsoft R Open, MKL (Rblas), and Rstudio
  • Requires a Small VM size or larger

CentOS 6 (6.8) Development

  • Tags: Featured, CentOS, development
  • Created: 7/22/2016
  • Based on Centos 6 Cloud 1601 Image
  • Development tools added
  • Patched up to date as of 6/29/16 -- upgraded to CentOS 6.8

CentOS 6 (6.8) Development GUI

  • Tags: Featured, CentOS, desktop, development, gui, VNC
  • Created: 7/15/2016
  • Based on CentOS 6 (6.7) Development
  • Patched up to date as of 5/31/16 -- updated from 6.7 to 6.8
  • Turned off the following from default startup in GNOME:MAKER 2.31.8 with CCTools 5.4
  • Bluetooth Power Manager Volume Control PulseAudio Package Manager

CentOS 7.2 Development

  • Tags: Featured, CentOS, development
  • Created 7/22/2016
  • Based on Centos 7 Cloud 1601 Image, development tools added
  • Patched up to date as of 6/29/16
  • Development tools added

CentOS 7 R with Intel compilers

  • Tags: Featured, CentOS, desktop, development, gui, Intel, vnc
  • Created 9/06/2016
  • R-3.3.1 with Intel compilers built on CentOS 7 (7.2)

CentOS 7 (7.2) Development GUI

  • Tags: Featured, CentOS, desktop, development, gui, x2go
  • x2go installed -- Instructions for x2go
  • Based on Centos 7 Cloud 1601 Image, Patched up to date as of 8/11/16.
  • Development tools added
  • X Window System and Xfce Groups added
  • Added yum-plugin-changelog, irods-icommands-4.1.9. firefox . thunderbird, xfce4-terminal, nm-connection-editor, network-manager-applet, net-tools, xterm, Lmod
  • NOTE: Ongoing issues with VNC installation on this image - Do not use

Galaxy 16.01 Standalone

  • Tags: Featured, community-contributed, Ubuntu
  • Created: 3/30/2016Galaxy 16.01 Standalone
  • Based on Ubuntu 14.04.4 LTS
  • This is a standalone Galaxy server that comes preconfigured with hundreds of tools and commonly used reference datasets: just launch and use.
  • Requires a Large VM size or larger

Intel Development (CentOS 7)

  • Tags: Featured, CentOS, desktop, development, gui, Intel, vnc
  • Created: 09/01/2016
  • Intel compilers and development environment
  • Based on Centos 7 Cloud 1601 Image, Development tools + GUI + iRODs access added
  • Patched up to date as of 8/23/2016
  • REQUIRES a m1.small or larger VM to launch

Ubuntu 14.04.3 Development

  • Tags: Featured, development, Ubuntu
  • Created: 6/24/2016
  • Ubuntu 14.04.3
  • Trusty Tahr v1.1
  • Patched up to date as of 5/12/16

Ubuntu 14.04.3 Development GUI

  • Tags: Featured, desktop, development, gui, Ubuntu, VNC
  • Created: 7/22/2016
  • Based on Ubuntu 14.04.3 Development
  • Patched up to date as of 6/24/16
  • Base Ubuntu 14.04.3 + Xfce + Xfce-goodies, firefox, icon sets and themes, added Emacs and cmake

Wrangler iRODS - CentOS 6.7

Software licenses

Software licenses are generally the to obtain, coordinate, implement, and maintain.

Some licenses may require users to coordinate with Jetstream staff to allow for proper configuration. Such coordination is handled on a case-by-base basis.

Licensed software maintained by staff but available to users are listed below.

Compilers and Parallel Libraries

  • The Intel compiler is covered at IU and TACC by a site license and has been configured to point to the proper license server upon boot.
  • All components of the Intel Parallel Studio are covered by the IU and TACC site licenses, including the MKL libraries.

Specialty software

  • MATLAB licenses are also available at both providers. A pref-configured MATLAB-specific image is available within Jetstream.

If you need licenses for any other software then you will have to provide your own license.

Please be aware that if you take an image or software from Jetstream and run it somewhere else then the license will not work. Jetstream license servers restrict access to only valid Jetstream IP addresses.

Known Issues

Number Issue Date Reported Date Resolved Mitigation
  Active Issues      

3

Some Jetstream virtual machines remain stuck in the Networking stage when they launch. This prevents users from logging into or using the virtual machine.

8/30/2016

 

Try redeploying the virtual machine as described here, Instance management actions.

4

X2GO on the Biolinux8 image is currently experiencing keyboard mapping issues on first login.

9/28/2016

 

Using the x2goclient software on your remote machine

  • suspend the x2go session
  • restore the x2go session by re-authenticating

This restored session will have the correct keyboard map.

7

Laptops with touchscreens are unable to use the mouse in the web desktop. There is a bug in the web browser code in both Chrome and Firefox for using a touchscreen PC with the web desktop of Jetstream. This is not a bug we can easily work around since it's in the browser code itself.

4/17/2017

 

While we generally don't recommend using Internet Explorer, we've found it does not have the same issue. Using a VNC viewer also is a workaround.

8

Cannot login with external ssh-client using DSA ssh-keys in my settings.

5/12/2017

Never

DSA keys should not be used with Jetstream. Please use another protocol like RSA

9

Users with a brace " { " in their passwords may experience errors with openstack command line commands.

5/23/2017

 

Use a password that does not contain a brace. Also may try enclosing passwords in single quote mark (e.g. export OS_PASSWORD='password{with-brace}' in your openrc file

  Resolved Issues      

1

Jetstream is not currently recognizing all XSEDE staff allocations, such as TG-STA060015, as being valid for some users. This is in part because these allocations are roaming allocations. XSEDE users whose only Jetstream allocation is one of these staff allocations may not be able to log into the system or manage VMs.

8/30/2016

9/12/2016

XSEDE users can resolve this by requesting a non-roaming or startup allocation on Jetstream. See Jetstream Allocations for details or email help@xsede.org.

2

Jetstream may not recognize XSEDE-associated usernames that differ from TACC usernames due to unresolved issues with identity mapping. This primarily affects long-time users of XSEDE or Teragrid.

8/30/2016

3/1/2017

This can be mitigated by filing a ticket at help@xsede.org requesting that your TACC and XSEDE usernames be brought into alignment.

5

If there is a dot/period within the last three characters of the instance name, that instance will fail to launch. This is not considered a bug by the OpenStack community but rather a feature that ties into a dynamic DNS feature.

9/30/2016

7/11/2017

Code now changes all dots to underscores to prevent this

6

A power issue resulted in some issue being inaccessible. Staff are restoring these instances.

4/12/2017

4/17/2017

Retry your instance. Contact help@xsede.org if it continues to not respond.

Allocations

Jetstream allocations are awarded exclusively through the eXtreme Science and Engineering Discovery Environment (XSEDE). XSEDE provides XSEDE User Portal (XUP) accounts free of charge. XSEDE allocations require that the PI be a US-based researcher.

You can read the XSEDE Allocations Guide for more detail on Jetstream allocations. Examples of successful requests are available there as well.

Create a portal account

  1. Browse to the XSEDE User Portal
  2. Click Create Account on the left side of your screen.
  3. Fill out the form and click Submit.
  4. Upon receipt of the email notification click the link in the email to verify your account and set your username and password. If the link doesn't work, go to the XSEDE User Portal, click Sign In and then select Verify Account under the Other Sign In Options.
  5. Following account verification, if not already logged in, go to the XSEDE User Portal, click Sign In and sign in with the username and password set in the verification step.
  6. You will be asked to read and accept the User Responsibilities form. This outlines acceptable use to protect shared resources and intellectual property.

Apply for an allocation

  1. Read the XSEDE Allocations Overview. There are sample allocation requests in the overview that you may find helpful.
  2. Go to XRAS, the XSEDE Resource Allocation System. On the Available Opportunities page, click Start a New Submission under Startup. If you are not familiar with the process, select Begin Guided Submission for step-by-step instructions.
  3. Before submitting an allocation request have the following information available:
    1. PI, Co-PIs, and Allocation Managers (username)
    2. Add additional users that will be able to use your allocation time and resources (optional)
    3. Title
    4. Abstract (typically a paragraph or two for a Startup request will suffice)
    5. Keywords
    6. Field of science (secondary areas of science may be also be added)
    7. Resources
      1. Select Jetstream IU/TACC from the list.
      2. A startup allocation is typically 50,000 SUs for Jetstream.
      3. Fill in number of Virtual Machines needed
      4. Fill in number of public IP addresses needed
      5. Select Jetstream Storage. For Startup allocations, the block storage provided will be limited. Read Virtual Machine Sizes and Configurations for a list of the various VM sizes and the RAM and associated storage. If additional storage for your VM is not needed, enter 1 in the Amount Requested box. Note: The maximum allowed is 500GB. A justification will be needed in the comments for any significant storage request in the Comments box.
    8. Supporting documents - PDF format required
      1. PI CV (2 page limit)
      2. CoPI CV required for every CoPI added to request (2 page limit)
    9. Supporting Grants (Optional)
    10. Publications of previous/supporting work (optional)
  4. Submit allocation request. At this point, all entered information is validated, errors or omissions are flagged.

Allow 1-2 business days for your application to go through the approval process.

You can view detailed information, with screenshots, about the allocation request process.

XSEDE also offers Trial Access Allocations to Jetstream. Trial Access Allocations are designed to give potential users faster but limited access to Jetstream in order to try the features of cloud-computing.

Request additional resources

I have a current XSEDE allocation but need additional Service Units (SUs):

My limits on the number of cores, number of instances, or memory are too low:

  • Additional resources, such as CPUs and memory, may be requested via Atmosphere.
  • After logging in, click on Change Your Settings:
Screenshot: Change Settings

Figure 1: The Jetstream Dashboard, with access to the Change Your Settings button

  • Click on request more resources in the Allocation section:
Screenshot: Request More

Figure 2: The Jetstream Change Your Settings UI, showing the link to use when requesting more resources

On the popup form:

  1. Determine if you need more Service Units or Quota
  2. Click on Next.

NOTE: if you select Service Units, you will be redirected to the XSEDE User Portal to complete the request.

On the popup form:

  1. List the resources needed, e.g., 4 CPUs and 8GB memory, running 4 cores for 1 week
  2. Enter a justification for the resource request, describing how the additional resources will be used.

Click on Complete Request to submit the form.

FAQ

1. Is there an overview of the types of allocations available as well as any restrictions those allocations have?

The Getting Started Guide describes the process of getting onto XSEDE, applying for allocations and using XSEDE resources. To review the types of allocations XSEDE and the process to get an allocation, here are some links you might find useful:

2. Is there a example or demonstration of how to get an allocation that I could follow?
There is a Cornell Virtual Workshop (CVW) on Getting a Research Allocation for Jetstream.
3. How do I let other XSEDE accounts use my allocation?
Here are instructions to add or remove users from an active XSEDE allocation which the PI, co-PI, or allocation manager can follow.
4. How often can I get a startup allocation?
Applications for startup allocations will only be accepted once. After the startup runs out it is best to apply for a research allocation.
5. How do I request additional SUs?
If you already have an XSEDE allocation and need to request additional service units (SUs) the PI, co-PI, or a delegate may submit a request via the XSEDE User Portal. For instructions on how to submit the request, see How do I request supplemental service units for an XSEDE allocation?
6. How do I request additional resources such as CPUs and memory?
Additional CPUs and memory may be requested via Atmosphere. See How to request additional resources on Jetstream.

Supplemental Allocation

Requesting additional SUs or storage

Login to the XUP

Screenshot: XUP login screen

Go to the XSEDE Resource Allocation System (XRAS)

(You can also mouse over the Allocations tab and then select Submit/Review Request)

Screenshot: XUP Go to request

Find your allocation. Under the Action menu for your allocation, select Supplement

Screenshot: XUP Supplement action

Choose Start Supplement

Screenshot: XUP Start Supplement button

Choose IU/TACC Jetstream and/or IU/TACC Jetstream Storage depending on whether you need SUs, additional storage, or both.

Screenshot: XUP Resources selection

Once you have selected the resources you need, you'll need to fill in the values of your request. The SU request has the same options as when you got your allocation. You'll fill in the SUs needed, the number VMs and IPs you expect to use (for forecasting purposes), and any comments.

Screenshot: XUP SU select

For storage requests, you'll fill in the value in gigabytes - e.g. 1000gb = 1TB.

Screenshot: XUP storage select

You will have to include a PDF "Progress Report" basically a paragraph or two on the need for the storage or SU request.

After you have added that, you may submit the request.

Screenshot: XUP support docs submit

Requests are typically reviewed within 1 to 2 business days. You'll receive notification from XSEDE Allocations on the status of your request once it has been reviewed by all service providers on the request.

Getting an Education Allocation

Jetstream allocations are awarded exclusively through the eXtreme Science and Engineering Discovery Environment (XSEDE). XSEDE provides XSEDE User Portal (XUP) accounts free of charge. XSEDE allocations require that the Principal Investigator (PI) be a US-based researcher.

The XSEDE website has a guide with more detail on Jetstream allocations.

You'll need a copy of your CV, the course syllabus, and a resource justification. The latter is basically figuring the size of the VM your students will need, the number of weeks they'll need it, and the number of students. An example of the resource selection is below.

You'll first need to create an XSEDE User Portal (XUP) account if you do not already have one.

Create an XSEDE Portal Account

  1. Go to the XSEDE User Portal (XUP)
  2. Click Create Account on the left side of your screen.
  3. Fill out the form and click Submit.
  4. Upon receipt of the email notification click the link in the email to verify your account and set your username and password. If the link doesn't work, go to the XUP, click Sign In and then select Verify Account under the Other Sign In Options.
  5. Following account verification, if not already logged in, go to the XUP, click Sign In and sign in with the username and password set in the verification step.
  6. You will be asked to read and accept the User Responsibilities form. This form outlines acceptable use to protect shared resources and intellectual property.

Apply for an XSEDE Education Allocation

  1. Read the Allocations Overview. There are sample allocation requests in the overview that you may find helpful.
  2. Go to the XSEDE Resource Allocation Request page. On Available Opportunities, click Start a New Submission under Educational. If you are not familiar with the process, select Begin Guided Submission for step-by-step instructions.
  3. Before submitting an allocation request, have the following information available:

    • PI, Co-PIs, and Allocation Managers (username)
    • Add additional users that will be able to use your allocation time and resources (optional)
    • Title
    • Abstract (typically a paragraph or two for an Educational request will suffice)
    • Keywords
    • Field of science (secondary areas of science may be also be added)
    • Resources
      • Select Jetstream IU/TACC from the list.
      • Enter the number of SUs you'll need. The Virtual Machine Sizes and Configurations page can help you figure VM sizes needed. See the box below for a walkthrough of creating your resource justification.
      • Fill in number of Virtual Machines needed (this is an estimate for planning purposes – there's no wrong answer – try to best guess at how many instances you'll run at one time)
      • Fill in number of public IP addresses needed (same as above)
      • Select Jetstream Storage. For Startup allocations, the block storage provided will be limited. By default, all Atmosphere users get 100 GB of volume space in addition to any VM root disks, so if that's enough, you need not request Jetstream Storage and do not need to select it.
    • Supporting documents — PDF format required
      • PI CV (2 page limit)
      • CoPI CV required for every CoPI added to request (2 page limit)
      • Syllabus
      • Resource Justification (see below)
    • Supporting Grants (Optional)
    • Publications of previous/supporting work (optional)
  4. Submit allocation request. At this point, all entered information is validated, errors or omissions are flagged.

Allow 3-5 business days for your application to go through the approval process.

Detailed information about the allocation request process, with screenshots, can be found in the XRAS Submit Allocation Request Step-by-Step Guide.

Writing your resource justification

A resource justification is required for an Educational allocation. You'll need to explain how the resources will be used and for how long. A sample calculation for your SU request might be:

  • 16 week course
  • 20 students
  • m1.medium instance size (6 vCPU)

Planning for 24 hour/day usage:

6 vCPU (SUs) x 24 hours/day x 7 days x 16 weeks = 16,128 SUs/student x 20 students = 322,560 SUs

It would be standard practice to round that to 350,000 SUs for development time for the instructors plus instructor VMs during the duration of the course.

You'll also need to briefly explain why you chose the VM sizes that you did – what you expect each student to be running on those VMs. The Virtual Machine Sizes and Configurations page can help you figure VM sizes needed. A paragraph or two should suffice to cover this.

If you have questions about the resource justification, please contact Jetstream Help via the XSEDE Help Desk and we can assist you.

Trial Access allocation

Jetstream is an NSF/XSEDE resource designed to promote and provide configurable cyberinfrastructure in the form of cloud computing to both novice and experienced users.

XSEDE offers Trial Access allocations to Jetstream. Trial Access allocations are designed to give potential users faster but limited access to Jetstream.

Since there is no formal allocation proposal request, within one business day users will be able to access and evaluate Jetstream prior to requesting a more involved larger startup or research allocation.

Trial Access allocations are limited to:

  • 1000 Service Units
  • 1 small (2-core) Virtual Machine (VM) instance per cloud at a time
  • 1 VM backup snapshot per instance,
  • 1 small 10 GB disk external storage volume.

This is enough power to give new users an experience with virtual computing and try some "cloud-native" work, but is not a substantial usage of the system.

The Trial Access (aka Easy Button) is for exploratory purposes only and currently the limits are hard-wired into the allocation. The additional limits placed on an Easy Button account are not present for Startup and Full allocations.

XSEDE currently has quite a rapid turn around on Startup allocations, often less than a week (excluding holidays).

You can read all about allocations.

Request a Trial Access allocation

To enroll in the Trial Access allocation:

  1. Browse to the XSEDE User Portal.
  2. Click "Create Account" on the left side of your screen.
  3. Fill out the form and click Submit.
  4. Upon receipt of the email notification click the link in the email to verify your account and set your username and password. If the link doesn't work, go to the XSEDE User Portal, click "Sign In" and then select "Verify Account" under the "Other Sign In Options".
  5. Following account verification, if not already logged in, go to the XSEDE User Portal, click "Sign In" and sign in with the username and password set in the verification step.
  6. You will be asked to read and accept the User Responsibilities form. This outlines acceptable use to protect shared resources and intellectual property.
  7. Now logged in to the portal, click on "MyXSEDE" tab.
  8. In the lower left bar, Click "Enroll".
  9. This will take you to a page describing Trial Access and allowing you to Enroll or Unenroll.
  10. Click the "Enroll Trial Access" button.
  11. A Successful Enrollment will show a message like "Your request is being processed. You will receive an immediate ticket message from our helpdesk system confirming your request. Please expect a follow-up email in 24 hours confirming your request was completed.
  12. Wait approximately 4 hours for your allocation to propagate through the authentication system.
  13. Continue with the Quick Start Guide instructions.
    1. Please note that the Trial Access allocation has ID ASC160018.
  14. You may unenroll at any time on the same page by clicking the "Unenroll Trial Access" button.

Account Conflicts

If you have a deactivated or otherwise disabled account with the Texas Advanced Computing Center (TACC), you will need to reactivate that account by changing your password before your Trial allocation can be finalized.

VM Sizes and Configurations

Jetstream can be used in several different virtual machine (VM) sizes which are charged in service units (SUs) based on how much of the total system resource is used. The table below outlines the VM sizes created for Jetstream.

VM Size vCPUs RAM (GB) Local Storage (GB) SU cost per hour Can be saved as an image?
m1.tiny 1 2 8 1 yes
m1.small 2 4 20 2 yes
m1.quad 4 10 20 4 yes
m1.medium 6 16 60 6 yes
m1.large 10 30 60 10 yes
m1.xlarge 24 60 60 24 yes
m1.xxlarge 44 120 60 44 yes
s1.large 10 30 120 10 no
s1.xlarge 24 60 240 24 no
s1.xxlarge 44 120 480 44 no

This allocation information may be subject to changes in the future.

Please note that s1.* based customized instances will NOT be able to be used to create images in Atmosphere.

If your work requires 24 GB of RAM and 60 GB of local storage, then you would request 10 SUs per hour to cover a single Large VM instance.

If your work requires 10 GB of local storage in 1 thread using 3 GB of RAM, then you would request 2 SUs per hour for a Small VM instance. You would then multiply by the number of hours you will use that size VM in the next year and multiply by the number of VMs you will need.

To calculate the number of SUs you will need in the next year, first estimate the number of hours you expect to work on a particular project. For example, if you typically work 40 hours per week and expect to spend 25% of your time on this project that would be 10 hours per week. Next, calculate the total number of hours per year for this project:

Total hours = 10 hours per week * 52 weeks per year 
Total hours = 520

Finally, calculate the total SUs for the year for a single instance medium VM:

Total SUs = 520 hours per year * vCPUs
Total SUs = 520 hours per year * 6vCPUs Total SUs = 3120

If project requires more than 1 medium size VM multiply total SUs by the number of VMs that you will need:

Total SUs needed for 3 medium size VMs = 3 * 3120
Total SUs = 9360

Note: SU cost per hour is described in the XSEDE KB document On XSEDE, how are compute jobs charged?.

Shutdown your VM properly
The calculations above assume that your VM is shutdown properly. For instructions see Shutting down, suspending, stopping instances.

For information on submitting a Research Allocation Request, please see https://portal.xsede.org/allocations/research#examples. Note that all allocations above the startup level require a strong justification for the time being requested.

Quick Start Guide

Accessing Jetstream

Accessing Jetstream

Jetstream is accessible through a web interface using XSEDE credentials via Globus Auth. Jetstream is not accessible via the XSEDE Single Sign-On Login Hub. Newly created XSEDE accounts must be added to a specific allocation by the PI or Resource manager in order to access Jetstream.

Jetstream allocations are awarded exclusively through the eXtreme Science and Engineering Discovery Environment (XSEDE). XSEDE provides XSEDE User Portal (XUP) accounts free of charge.

To get a Jetstream Trial Access account, please see Get a Jetstream Trial Access account. These accounts have limited resources available but allow you to try Jetstream.

To get a more full-featured account, you'll request a startup allocation, please see Jetstream Allocations for step-by-step instructions.

For complete details and screenshots, see the System Access section of this guide.

Getting started

  1. To start the VM provisioning process, navigate to https://use.jetstream-cloud.org.

  2. Click Login in the top right to authenticate using your XSEDE credentials.

  3. On the Globus Auth screen, click Continue.
  4. Enter your XSEDE credentials; confirm whether you will allow your credentials to be used to access Jetstream. Best results will occur if you treat your username as if it were case-sensitive when using Jetstream.
  5. To proceed, click Allow and the web interface to Jetstream will load.
  6. Once you are authenticated via Globus Auth, you will end up on the Jetstream landing page, also called the Dashboard. On this page you will be able to:
    1. launch a new instance
    2. browse help resources
    3. change your settings
    4. see your resources and usage history
    5. view a Jetstream Community Activity feed
  7. As SSH is a primary method to access Jetstream resources, we recommend you add SSH keys for each host machine from which you will connect to Jestream.

Launch an instance

To launch an instance, you will need to:

  1. open up the Jetstream Atmosphere page, https://use.jetstream-cloud.org
  2. follow the instructions below

PLEASE NOTE: It can take 5-10 minutes for an instance to fully deploy, so you may wish to plan accordingly.

  1. Click Launch New Instance from the Dashboard screen
  2. On the Featured Images screen, search for the image you would like to use or to scroll through the images that you have permission to use
  3. Click on the image you would like to use then click Launch to begin the process of creating an instance
  4. On the 'Launch an Instance / Basic Options' page:
    1. Give your instance a name
    2. Select the version if there are multiple versions available
    3. Select a project for your instance or create a new project
    4. Select the allocation source
    5. Choose which provider you want to run on, Indiana or TACC
    6. Select the instance size
    7. Click Continue
  5. Review the information entered for provisioning your instance
    1. If everything is OK click Launch instance to start the build process
    2. To make changes, click Back to return to previous screens
    3. To add or create a deployment script, click Advanced Configuration
  6. When the status shows as Active, click on the name of the instance to display the instance details
  7. To log in to the instance, click on Open Web Shell, located on the lower right hand side of the screen under Links. If this is unavailable, try refreshing your window. If the link is still not enabled, log in to your instance via SSH for your operating system
  8. If you are using Open Web Shell and you did not install your SSH key before provisioning and launching the instance, you will be prompted to enter your XSEDE username and password

System Access

Step-by-step guide

  1. To start the VM provisioning process, navigate to https://use.jetstream-cloud.org.
  2. Click Login in the top right to authenticate using your XSEDE credentials.

    Figure 1: The Jetstream Login access for XSEDE credentials

  3. On the Globus Auth screen click Continue.

    Figure 2: The Globus Auth screen for Jetstream Web App with XSEDE credentials

  4. Enter your XSEDE credentials. Best results will occur if you treat your username is if it wre case-sensitive when using Jetstream.

    Figure 3: The XSEDE credentials screen

  5. After you type in your XSEDE username and password, confirm whether you will allow your credentials to be used to access Jetstream. You may wish to review the terms of service and privacy policies linked on that page. Generally, this screen will only be displayed the first time you log into Jetstream, however, changes to Globus Auth might cause this screen to be displayed on a later login to Jetstream. To use Jetstream, click Allow.

    Figure 4: Allow Jetstream access with XSEDE credentials

  6. To proceed, click Allow and the web interface to Jetstream will load.

    Figure 5: The Jetstream interface loading

  7. Once authenticated via Globus Auth, the Jetstream Dashboard will be displayed. On this page you will be able to:

    • launch a new instance
    • browse help resources
    • change settings
    • view resources and usage history
    • view a Jetstream Community Activity feed

    Figure 6: The Jetstream UI Dashboard

    After provisioning a VM it can be accessed a number of ways, including:

    See Logging in to your VM for more details.

Find the public IP assigned to your instance from the VM command line

If your VM has a public IP and you need to find that IP (and don't have ready access to the Jetstream interface, use wget or curl from the command line to get your public IP:
  wget http://169.254.169.254/latest/meta-data/public-ipv4 -qO -
  wget http://ipinfo.io/ip -qO -
  curl http://169.254.169.254/latest/meta-data/public-ipv4
  curl http://ipinfo.io/ip
  

Note: The third option may work even in conditions in which external DNS servers are not accessible.

Note: The only way to acquire a static IP address is to use the Jetstream API. See Using the Jetstream API for details.

SSH keys

Adding SSH keys to the Jetstream Atmosphere environment

While Jetstream provides a web-based terminal for accessing your VM once it has been deployed, you might find that you wish to access your VM via SSH if you've provisioned it with a routable IP number. Please note that during early operations, all IP numbers offered will be routable - this will change in production.

If you need assistance creating SSH keys, please refer to the XSEDE KB article "How do I set up SSH public-key authentication to connect to a remote system?" Please do note that to get your keys on Jetstream, you will not follow the instructions In that article on placing your keys on a remote system.

Step-by-step guide

  1. To add your ssh key(s) to Jetstream, click on your username in the upper right hand corner and then click Settings.

    Figure 1: The Jetstream dashboard, showing how to access user settings

  2. On the Settings screen, under Advanced, click Show More, to expand the section for adding your SSH key. Check the box that says "Enable ssh access into launched instances" and then click the green plus sign to actually add your key.

    Figure 2: Accessing SSH Key advanced settings in the Jetstream dashboard

  3. On the next screen give the key a descriptive name and then paste the contents of your PUBLIC ssh key into the dialog box.

    Figure 3: The public SSH key dialog in the Jetstream UI dashboard

  4. After you have pasted in your SSH key, click Confirm. You will then be back at the Settings screen with your key shown in the SSH Configuration section.

Launching your VM

  1. To get started using a Jetstream virtual machine, click Launch New Instance from the Dashboard screen. On the list of images page, scroll through the the list of images or enter an image name, tag or description in the search box. For instance, to locate images named or tagged with "CentOS", enter that text in the search bar. The search is not case sensitive.

    Figure 1: The Jetstream List of Images page, showing Images available for launch

  2. After selecting an image, details about that image will be displayed. From here, add the selected image to a project, star it as a favorite, or edit image information. Click Edit details to:

    • Update the name of the image
    • Set a date to hide the image from public view
    • Modify the description
    • Add or delete tags
    • Click Save
    • Click Launch

    Figure 2: The Instance Details screen

  3. On the Launch an Instance / Basic Options screen:

    • Enter a name for the instance
    • Select the image version if there are multiple versions available
    • Select or create a project to hold this instance. If you have any existing projects, they will be shown here and you can select one. If you don't have any existing projects, click Create New Project and fill in Project Name and Description. A detailed description is optional, but it is recommended to include any grant names or other easily identifying details so others working with you may easily find it. Click Create to create the new project.
    • Indicate the allocation source
    • Choose the provider to run on, Indiana or TACC
    • Choose the instance size. This indicates the vCPUs, memory, and disk size for the VM. See the Virtual Machine Sizes table to show the available options and the SUs consumed per hour. Check projected resource usage: Allocation Used and Resources Instance will Use.
    • Click Launch Instance to start the initialization and build of the instance.

    Figure 3: The Launch Instance Options screen

  4. Below are several screens that typically display during the provisioning process.

    Figure 4a: Instance creation process: Build - networking

    Figure 4b: Instance creation process: Active - initializing

    Figure 4c: Instance creation process: Active - deploying

  5. The instance will be ready for use when you see a green dot and "Active" in the Status column.

    Figure 5: Instance ready for use: Green Dot means Active

Please note that it may take some time for instances to become active, 5-10 minutes on average. The start up time also depends on how busy the system is and on the size of the VM you requested.

Once Active, Instance Management Actions, may be performed on the instance.

Find the public IP assigned to your instance from the VM command line

If your VM has a public IP and you need to find that IP (and don't have ready access to the Jetstream interface, use wget or curl from the command line to get your public IP:
  wget http://169.254.169.254/latest/meta-data/public-ipv4 -qO -
  wget http://ipinfo.io/ip -qO -
  curl http://169.254.169.254/latest/meta-data/public-ipv4
  curl http://ipinfo.io/ip
  

Note: The third option may work even in conditions in which external DNS servers are not accessible.

Note: The only way to acquire a static IP address is to use the Jetstream API. See Using the Jetstream API for details.

Instance Management Actions

After launching an instance, several options are available under the Actions menu located on the right hand side of your screen.

Shutting down, suspending, stopping and shelving instances

Figure 6: Management Actions available on a launched instance

Definitions

  • Suspend: Instance in-memory state and disk are preserved, such that the instance can quickly be resumed. Network IP addresses will not be maintained. Applications and other processes running at the time of suspend will resume when the instance boots. Note: Some processes may not resume naturally, such as processes involving network connections. This state is comparable to "hibernate" or "sleep" modes in a laptop.

SUSPENDing an instance continues to consume system compute and storage resources. (75% normal usage rate)

  • Shutdown: Instance operating system is shut down, in-memory states are not stored, and only disk is preserved for a quick start. This state is comparable to "shutdown" modes in a laptop.

While the system storage footprint is smaller than that of SUSPEND, SHUTDOWN continues to consume system storage resources (50% normal usage rate).

  • Shelve: Much like SHUTDOWN, the instance operating system is shut down and instance in-memory states are not stored. Unlike SHUTDOWN, a disk snapshot is stored to long-term storage and thus will be relatively slower to re-start. HOWEVER, shelving BENEFITS the cloud by no longer consuming computation resources and using far less disk resources. This state does not currently consume service units (0% normal usage rate).

Actions

Report an instance if these situations occur:

  • you can't connect via SSH or VNC
  • the status never changes from pending to running
  • you receive errors when running or installing software
  • the metrics for the instance do not display
  • the instance exhibits any other unexpected behavior

Image request - see Customizing and Saving a VM

Suspend an instance to:

  • stop active computing while still consuming system resources
  • safely preserve the state of your instance (though cannot image)
  • preserve your allocation (75% normal usage rate) a
  • NOTE: The IP address is likely to change when the instance is resumed.
  • It may take 1-5 minutes to resume your instance
  • status will change from "Active" to "Suspended"

Shelve an instance to:

  • free up compute resources for other users
  • preserve your allocation (0% normal usage rate) a
  • allow for imaging of the instance
  • NOTE: The IP address is likely to change when the instance is resumed.
  • We strongly recommend detaching volumes before shelving.
  • It may take 5-15 minutes to resume (Unshelve) your instance
  • status will change from "Active" to "Shelved_offloaded"

Stop an instance to:

  • stop active computing, consuming fewer resources than a suspend
  • preserve your allocation (50% normal usage rate) a
  • allow for imaging of the instance
  • NOTE: The IP address is likely to change when the instance is resumed.
  • It may take 1-5 minutes to resume your instance
  • status will change from "Active" to "Shutoff"

Reboot an instance to send:

  • an 'ACPI Restart' request to the VM that will start the reboot process for your VM
  • a 'Hard Reboot', which will forcibly restart your VM, if it doesn't respond to a 'Reboot'

Redeploy an instance:

  • to fix instances that show up as 'active - deploy_error'
  • Contact support if your VM returns to the deploy_error state after redeployment.

Delete an instance:

  • Unmount volumes within your instance before deleting the instance or risk corrupting your data and the volume.
  • Your instance will be shut down and all data will be permanently lost!
  • Your resource usage charts will not reflect changes until the instance is completely deleted and has disappeared from your list of instances.

a Currently, actively running instances consume SUs at the rate of 1 SU per cores*hours used. Suspended instances consume 75% and Stopped instances consume 50%. Policy subject to change.

Comparison Table

Instance state characteristics

Action Active Suspend Shutdown Shelve
Consumes allocation Yes Yes, 75% Yes, 50% No
Consume system resources Yes Yes Yes No b
Saves running programs Yes Yes c No No
Preserves IP address Yes No No No
Time to resume n/a 1-5 min 1-10 min 1-15 min

b Some disk resources consumed
c Some processes, particularly those tied to networking, may not resume fully.

Linux VM shutdown

Shutting down a VM from Linux

If you are going to stop your instance, shut down the VM gracefully and securely. In a GUI environment on Linux, the methods may vary.

Launching a terminal and running the shutdown command as root (or with sudo privileges) should work consistently across Linux versions:

/sbin/shutdown -h now

OR

sudo /sbin/shutdown -h now

This will stop all operations, log out other active users on your VM as the system powers down and show your VM as "Shutoff" in Atmosphere once the VM has shut down.

Note that sometimes the Atmosphere interface doesn't update in a timely manner to reflect the true status of the VM. If you have done a shutdown from the console, then do a hard refresh of your browser about a minute after shutdown. This is usually Command-R on a Mac and Ctrl-F5 on a PC. Check your browser's documentation to verify.

In general, refresh the browser to verify the VM state if in doubt.

Change Allocation Charges

Changing the Allocation your instance charges to in Atmosphere

There are a couple of reasons you might wish to change the allocation an instance is charging to in Atmosphere. You might have received a research allocation and want to move the charge from the startup allocation. You may have accidentally started it and used the wrong allocation. Either way, the process of changing which allocation an instance is charging to is quick and easy.

  1. Log in to Atmosphere by going to https://use.jetstream-cloud.org and click the Login link in the upper right hand corner. Follow the normal login procedure using XSEDE or associated Globus credentials.

  2. Go to your projects page and select the project that contains the instance you wish to change and click on the instance.

  3. In the instance information screen, look for the Allocation Source.
    You can click the drop box and see any other current allocations you are on. At this time, only the XSEDE allocation number shows (typically TG-XXXYYYYYY format). We have a feature request in to show "friendly" names here. In the meantime, though, you can see more information about your allocations at the Allocations Usage page.

  4. Select the allocation you wish to change it to. Atmosphere will then update the allocation it is charging to in about 15 minutes.

Add user to running instance

These steps will let the user that you create ssh to a running instance using a password you set. The user can reset the password once they login and/or add their ssh keys.

Step-by-step guide

All steps to be run as root or using sudo.

  1. Run 'adduser username'
  2. Run 'passwd username' and assign a temporary password
  3. Run 'usermod -a -G users username'

Notes

If you have the user's public SSH key, do the following:

  1. Run 'mkdir ~username/.ssh/'
  2. Run 'chmod 700 ~username/.ssh/'
  3. Run 'chown username:username ~username/.ssh/'
  4. Copy the user's public ssh into ~username/.ssh/ or use an editor to create the file authorized_keys in that directory and paste the contents of their public SSH into that file
  5. Run 'chown username:username ~username/.ssh/authorized_keys'

If you do not have the user's public SSH key, you will also need to do these steps. It is best, from a security standpoint, to ONLY allow public key access.

  1. Edit /etc/ssh/sshd_config and add the line PasswordAuthentication yes and then save the file
  2. Restart sshd ('service sshd restart' for CentOS 6, 'service ssh restart' for Ubuntu 14.04 systems, -OR- 'systemctl sshd restart' for CentOS 7)

Logging in to your VM

Accessing Jetstream

Jetstream is accessible through a web interface using XSEDE credentials via Globus Auth.

Jetstream is not directly accessible via the XSEDE Single Sign-On Login Hub.

It is possible to use the XSEDE SSO Hub as a jump-host to individual VMs but this does not follow normal use.

Users are required to have XSEDE accounts. Newly created XSEDE accounts must be added to a specific allocation by the PI or Resource manager in order to access Jetstream.

Please note that your XSEDE Portal password is not used for logging into VMs. If you wish to use password authentication to ssh into a VM (not recommended!) you will need to set up a password (refer to Logging in with SSH for information).

Jetstream allows multiple methods for accessing and using your VM.

  • Command line interface
    • Open Web Shell: The deprecated Old Open Web Shell and Open Web Shell links available on Jetstream Cloud are the preferred methods.
    • SSH: You may also log in to your instance directly via an SSH command line.
  • Graphical interface
    • Web Desktop - Images with VNC/GUI/Desktop enabled will display a Web Desktop icon under the LINKS items in the instance detailed view (found by clicking on the instance title). You may have to use your browser's refresh button to make it appear.
      Simply click on Web Desktop to start a browser-based VNC implementation.
    • VNC client: graphical encrypted VNC desktop from your host computer.
      • For BioLinux, please use x2go instead of VNC.

Some images support additional mechanisms for accessing your VM. Some of the additional access methods are noted in image-specific documentation.

Logging in with Web Shell

  1. Log in to Jetstream via the web interface and launch the instance.
  2. When the status shows as Active, click the Open Web Shell link found on the lower right hand side of the screen. If this link is unavailable, try refreshing your window. If the link is still not enabled, log in to your instance via SSH.
    Figure 1: Active instance with Web Shell link in lower right corner

    NOTE:
    Currently, Web Shell allows ONLY ONE Web Shell window per browser login to use.jetstream-cloud.org. Each new Web Shell will disconnect previous windows/tabs. If you need more than one Web Shell, you will need to use a tab or window in INCOGNITO mode or open a second browser. You may also wish to use direct SSH connections.

  3. Under normal circumstances Web Shell (Beta) will automatically use your USERID credentials or any SSH public keys you have uploaded to Jetstream and you will not have to enter your XSEDE username and password.
    Figure 2: Successful login resulting in a common shell interface
  4. Control + D logs you out of the Web Shell and the window can be closed.
    Figure 3: After logout via Ctrl-D

Copy and Paste in Web Shell

To Copy Text to Web Shell

  1. Shift+Control+Alt (Shift+Control+Command on MacOS) brings up the clipboard pane
  2. Figure 4: Web Shell with Clipboard Open
  3. Paste text in the clipboard
    Figure 5: Clipboard with Text Copied In
  4. Shift + Control +Alt to exit clipboard
  5. Paste text in Web Shell with Shift + Control + V
    Figure 6: Text Copied Outside of Web Shell Clipboard

To Copy Text from Web Shell

  1. Select text in Web Shell. Selected text is automatically copied to clipboard
  2. Shift + Control +Alt to bring up the clipboard pane
  3. Select text in the clipboard with Control + C
  4. Shift + Control +Alt to exit clipboard
  5. Paste text outside Web Shell with Control + V

Logging in with SSH

Regardless of whether you're logging from a Linux, Mac or Windows machine:

  1. add SSH keys to your account and
  2. copy the instance IP address, either from the confirmation email or from the IP address displayed in the My Instances list.
  3. Or if you want to use password authentication (not recommended but sometimes necessary) use the web shell and log in and type sudo passwd your_username.

MacOS & Linux/Unix

  1. Open a terminal window for Mac OS X (from Finder, go to Applications, click Utilities, and then double-click Terminal).
    For Linux, there are many terminal options, including xterm, konsole, or gnome-terminal.
  2. In the terminal window, enter the following command, using your XSEDE username and password, and the instance IP address:
    $ ssh xsede_username@instance_ip_address
  3. Press Enter.

A successful login will look similar to the following:

Figure 7: Screenshot of a successful MacOS X login shell

Windows using PuTTY

PuTTY is an SSH client for Windows. It operates a bit differently than Terminal to make the initial SSH connection. For a useful guide to using PuTTY, see PuTTY - Remote Terminal and SSH Connectivity.

  1. Download the PuTTY application.
  2. Launch PuTTY.
  3. The first time PuTTY is used for login, add your private key.
    1. Single click the 'Default Settings' session to save your private key for all future sessions.
    2. Click on the + symbol next to the 'SSH' category on the left side.
    3. Click on the 'Auth' category to bring up the PuTTY Configuration screen (see screenshot below).
    4. The key is set down at the bottom under 'Private key file for authentication'. Click on the Browse button next to the 'Private key file for authentication' field and locate your private key file on the file system. Select the file and press 'Ok'. (It is probably in your My Documents folder.)
    5. Click the 'Session' category from the left hand side.
    6. Make sure "Default Settings" is still selected.
    7. Click Save.
  4. Enter the IP address, either copied from your My Instances list or from the confirmation email, and click Connect.
  5. Enter your XSEDE username when prompted and click Enter.
From the VM command line, find the public IP assigned to your instance
If your VM has a public IP address and you need to find that IP (and don't have ready access to the Jetstream interface), you can use wget or curl from the command line to get your public IP:
wget http://169.254.169.254/latest/meta-data/public-ipv4 -qO -
wget http://ipinfo.io/ip -qO -
curl http://169.254.169.254/latest/meta-data/public-ipv4
curl http://ipinfo.io/ip

*Note: http://169.254.169.254/latest/meta-data/public-ipv4 works even in conditions in which external DNS servers are not accessible.

To become root
Enter sudo su - at the command prompt or type sudo command and replace command with the command for which you want to use sudo.

Logging in with VNC desktop

VNC is only available on certain images. Please look for the GUI or Desktop tags on the Featured images.

Web Desktop

Images with VNC/GUI/Desktop enabled will display Web Desktop under the LINKS items in the instance detailed view (found by clicking on the instance title). You may have to refresh your browser to make it appear.

Simply click on Web Desktop to start a browser-based VNC implementation.

External VNC Viewer

Alternatively, if you would like to use a different VNC client (which must support encrypted connections), for example realvnc viewer (free), please follow these instructions.

For VNC launches

  1. Once your instance comes up, login via ssh.
  2. Type 'sudo passwd your_username'.
  3. Use realvnc viewer (free) to connect to x.x.x.x:1 (your_ip:1) with your username and the password you just set.
  4. When finished with the VNC, please disconnect your VNC session via the Real VNC viewer or by closing the VNC window on your host computer.

DO NOT LOG OUT INSIDE THE DESKTOP OF THE VM: this will make your VNC unusable until you manually restart the VNC.

Note on resizing: Use xrandr -s 1920x1080 to get a larger screen; adjust the numbers to get the screen size that you need. xrandr works reliability in many Linux operation systems.

Valid screen resolutions are: 800x600,1024x700,1024x768,1200x740,1280x800,1280x960,1280x1024,1600x1000,1600x1200,1680x1050,1920x1080,1920x1200,3360x1050,3200x1000,3200x1200

Note: the default VNC session will only work for the user that launched the instance. Other user accounts on that instance that wish to have an individual desktop will have to start their own VNC manually.

Disconnecting your VNC session via the Real VNC Viewer:

Figure 8: Disconnecting a VNC instance using the Real VNC Viewer

To become root
Enter sudo su - at the command prompt or type sudo command and replace command with the command for which you want to use sudo.

Managing a VNC server

Starting a VNC server manually:

By default for images with VNC capability, a VNC is only launched for the User that launched the Instance.

If you have added additional users accounts to your Instance, and those users have set their password, they must manually start their own VNC desktop using the 'vncserver' command:

[USER@js-19-210 ~]$ vncserver :1

where :1 is a display number (1-9).

Note: If a VNC server is already running on that particular display number, you will receive an error:

Error: A VNC or X Server is already running as :1 [DisplayInUse]

If this arises, choose a different display number.

Note: If a User has logged out within the VNC desktop, this manuallly stops the VNC server.

By default, the primary owner of the instance has a guacamole-based vncserver running on display :5, as well as servers on displays :1 and :2. To restart the default guacamole-based server, use the command:

[USER@js-19-210 ~]$ vncserver -config ~/.vnc/config.guac :5

Stopping a VNC server

A VNC server may be stopped using the command:

[USER@js-19-210 ~]$ vncserver -kill :1

where :1 is the display number of a VNC server owned by that User.

Logging out of the desktop in the VNC also stops the VNC server.

Note: Killing an active VNC server before data/files have been save or icon settings stored may result in a loss of data/settings.

Restarting a VNC server

If a User has stopped their VNC server either by logging out of the desktop within the VNC or with the -kill option to the vncserver command, they may simply restart the server after SSH'ing in, as above.

XSEDE SSO As Jump Host

PLEASE NOTE: This is NOT the normal mechanism to use the XSEDE SSO Hub as described in the XSEDE SSO documentation:

To make use of XSEDE's Single Sign-On Login Hub to simplify logging in to Jetstream instances:

FIRST TIME ONLY

ssh -l <XSEDE_username> login.xsede.org

Enter XSEDE password and perform multi-factor authentication

[<XSEDE_username>@ssohub ~]$ ssh-keygen

Accept defaults and hit return for your password

[<XSEDE_username>@ssohub ~]$ cat .ssh/id_rsa.pub

Copy the entire entry all the way from "ssh-rsa" through "xsede.org"

[<XSEDE_username>@ssohub ~]$ exit

Browse to https://use.jetstream-cloud.org using your XSEDE credentials.

Follow the ssh-key instructions and add the xsede.org key from above into your settings.

NORMAL USE

From now on, you can use the following to connect to your Jetstream instances:

ssh -l <XSEDE_username> -J login.xsede.org <individual Jetstream instance IP#>

Volumes

Volumes: Small virtual filesystems that may be attached to the user's running/active Instances.

Files/data saved to a Volume are maintained across successive attachment/detachment actions to the user's Instances.

Volume actions with the Atmosphere interface:

Volumes can also be controlled directly from the OpenStack API.

Detaching Volumes
REMINDER: Volumes can only be detached if:
  1. They are not in active use by a process on the instance
    Try:
  2. fuser -m /<volume> to LIST all processes using a volume
    fuser -km /<volume> to KILL all processes using a volume
  3. The instance to which they are attached is active

Cost and Size:

While Volumes are available to facilitate research at no additional Service Unit charge, and may be requested during initial or supplemental Jetstream allocation requests, large capacity storage is beyond the scope of Jetstream.

Users who have been approved for Volumes are limited to 10 volumes with an aggregate capacity of 500 GB across all.

Project and Providers:

As with Instances, Volumes are associated/organized with User Projects and with particular Providers (e.g. IU or TACC). You cannot attach a Volume from one Provider to an instance on a different Provider.

Sharing:

Volumes may only be attached to one active Instance at a time.

Volumes may be shared using standard methods (e.g. NFS) to other active Instances within Jetstream.

Backup & Exporting:

Users should regularly backup (via ssh, rsync, tar, or the like) any critical data contained on Volumes as no automated backup functions are currently provided by Jetstream.

Create a Volume

A Volume must be created in the Jetstream Atmosphere interface before it can be attached to an active Instance.

From the Jetstream Atmosphere Dashboard:

  • Click on "Projects"
  • Select your Project
  • Click "New"
  • Select "Volume"

Figure 1: Volume creation dialog

  • A window will pop up (Figure 1) in which you can:
    • Name the Volume
    • Enter the desired volume size (subject to your allocation limit)
    • Select the Provider on which the Volume will reside
  • Click "Create volume"
  • After creation, the volume will appear in your list of available volumes (Figure 2)

Figure 2: Available volumes in a project

Attach a Volume

In order to access a Volume, it must be Attached to an active Instance.

From the Jetstream Atmosphere Dashboard:

  • Click on "Projects"
  • Select your Project
  • Select the desired Volume and click "Attach"
    • NOTE: You will only be able to select Active Instances running on the same Provider as the Volume
  • The Volume will now be automatically mounted in the selected instance, e.g. on device /dev/sdb as /vol1 NOTE: The volume name (vol_b) will match the device name (sdb).
[USER@js-169-6 ~]$ df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        59G  3.0G   54G   6% /
tmpfs           7.8G  148K  7.8G   1% /dev/shm
/dev/sdb        9.9G  151M  9.2G   2% /vol1
  • NOTE: A Volume may only be attached to one active Instance at a time. However, Volumes may be shared using standard methods (e.g. NFS) to other active Instances within Jetstream.

After attaching, the Volume status in the Atmosphere interface will change to reflect the Instance to which it is attached.

Figure 3: Volume status after attaching to an instance

Detach a Volume

In order to remove a Volume from an active Instance, it must be detached. Volumes can only be detached from ACTIVE instances.

Files and data saved to a Volume are maintained across successive attach and detach actions.

From the Jetstream Atmosphere Dashboard:

  • Click on "Projects"
  • Select your Project
  • Select the desired Volume and click "Detach"
    • WARNING: If data are being written to the volume when it is detached, the data may become corrupted. Therefore, we recommend you make sure there are no data being written to the volume before detaching it.
  • Click "Yes, detach the volume"
    • NOTE: A Volume will fail to detach if:
      • it is in active use on the active Instance to which it is attached.
      • the Instance to which it is attached is in an inactive state (shelve/shutdown/suspend) (i.e. the Instance must be active in order to detach)
  • After the Volume has successfully detached, it may be subsequently re-attached to any of the user's active Instances on the same provider cloud.
Detaching Volumes
REMINDER: Volumes can only be detached if:
  1. They are not in active use by a process on the instance
    Try:
  2. fuser -km /<volume> to KILL all processes using a volume
  3. The instance to which they are attached is active

Backup and Export a Volume

Users should regularly backup any critical data in their Instance, especially data contained on Volumes, as no automated backup functions are currently provided by Jetstream.

Listed below are examples of methods to transfer files from Volumes, though similar syntax would work from any directory of the Instance.

In the following examples, the user is assumed to be using the command line from within an Instance, thus local-user means a user id on the Instance and remote-user means a user id on a computer outside of Jetstream.

SCP, SSH, and TAR

scp: One simple way to transfer the data on your Volume is using scp:

$ scp /vol1/file remote-user@destination_server_ip:/path/

tar with ssh: Use tar with ssh to transfer an entire directory or a file:

$ tar czf - /vol1/directory-or-file | ssh remote-user@destination_server_ip 'tar xzf -'

tar with ssh: As above, but leave the files in a remote archive:

$ tar czf - /vol1/directory | ssh remote-user@destination_server_ip 'cat - > archive.tar.gz'

DD with SSH

dd with ssh: Use dd with ssh to copy your entire Volume to a remote image file:

$ df -kh
    
    $ dd if=/dev/sdb | ssh remote-user@destination_server_ip 'dd of=vol1.img'
    

RSYNC

rsync: Use rsync to transfer an entire directory without encryption:

$ rsync -avtr /vol1 remote-user@destination_server_ip:/path/

rsync with ssh: Use rsync with ssh to transfer an entire directory with encryption and between different local and remote users:

$ rsync -avtr -e 'ssh -l local-user' /vol1 remote-user@destination_server_ip:/path/

GLOBUS File Transfer Tools

Instructions are available for installing and using Globus file transfer tools.

Transferring Files

There are a number of clients and methods for transferring files to and from your VM Instances.

All involve establishing a secure connection between your instance and another computer, whether inside or outside Jetstream.

SSH/SCP/SFTP

See how to use SSH/SCP/SFTP.

Web shell/Web Desktop

  • The web shell and web desktop provide a mechanism for transferring files. See instructions below.
  • Please note that this only works on the Guacamole based web shell and web desktop. The versions marked "Old" do not have this functionality.

See how to use Web shell/Web Desktop.

RSYNC

GLOBUS File Transfer tools

See how to use GLOBUS File Transfer tools.

Transfer with SCP or SFTP

There are a number of clients and methods for transferring files to and from your VMs from a computer outside Jetstream.

Here are some examples of how to backup/export files using ssh/scp/sftp from within your instance.

Transfer with the web shell and web desktop

The web shell and web desktop allow file transfers to be easily done with a built in facility.

Files can be transferred to the remote computer by dragging and dropping the files into your browser window, or through using the file browser located in the Guacamole menu. The Guacamole menu is a sidebar which is hidden until explicitly shown. On a desktop or other device which has a hardware keyboard, you can show this menu by pressing Ctrl+Alt+Shift. If you are using a mobile or touchscreen device that lacks a keyboard, you can also show the menu by swiping right from the left edge of the screen. To hide the menu, you press Ctrl+Alt+Shift again or swipe left across the screen.

Figure 1: Accessing the Guacamole menu

You can select where to transfer files by clicking the "Devices" option in the Guacamole menu. That will give you a file manager type menu where you can select the directory to transfer files into.

Please note that you will only be able to read or write files in directories that your user account would have access to. This means unless you've created directories where you can write, by default you'll only be able to write into your home directory, /tmp, and mounted volumes (/vol_X), etc.

Figure 2: File manager type menu

Double-clicking on any directory will change the current location of the file browser to that directory, updating the list of files shown as well as the "breadcrumbs" at the top of the file browser. Clicking on any of the directory names listed in the breadcrumbs will bring you back to that directory, and clicking on the drive icon on the far left will bring you all the way back to the root level.

Downloads are initiated by double-clicking on any file shown, while uploads are initiated by clicking the "Upload Files" button. Clicking "Upload Files" will open a file browsing dialog where you can choose one or more files from your local computer, ultimately uploading the selected files to the directory currently displayed within the file browser. You can also drag and drop files into the dialog box from your computer's file windows.

The state of all file uploads can be observed within the notification dialog that appears once an upload begins, and can be cleared once completed by clicking the "Clear" button. Downloads are tracked through your browser's own download notification system.

Figure 3: In-progress and completed file transfers

When you are done browsing the filesystem and transferring files, click "Back" to return to the Guacamole menu. You can close the menu by pressing Ctrl+Alt+Shift.

Transfering files with Globus

Prerequisites

On your Virtual Machine

  1. Log into VM via either ssh or Web Shell / Web Desktop

  2. Download and install Globus connect on the VM

    wget https://s3.amazonaws.com/connect.globusonline.org/linux/stable/globusconnectpersonal-latest.tgz tar xzvf globusconnectpersonal-latest.tgz

  3. Go to Globus Connect Personal

    1. Generate a new setup key for your VM

  4. Go to the globus connect directory on your VM

    cd ~/globusconnectpersonal-{your-version}

  5. Use the setup key generated in step 3 here

    ./globusconnect -setup setup_key

  6. Activate the endpoint on your VM

    ./globusconnect -start &

To Transfer Files

  1. To transfer files between endpoints, go to https://www.globus.org/app/transfer

    1. Click in the Endpoint dialogue box to select endpoints (look under tab "Administered by Me").
  2. Here you can select from the endpoints you have set up.

To Stop Globus

  1. To stop globusconnect on your VM:

    ./globusconnect -stop

Customizing and saving a VM

An image is a type of template for a virtual machine (VM).

You can launch an instance, custom install the software and files you want to use, then request an image of the instance. This will save all of the changes and updates within Atmosphere, but also make your custom changes available to yourself and collaborators. Saving instances as images helps leverage both your time and existing resources. The saved image can be used to launch a new child instance at any time. This allows you to discard non-active instances, thus saving resources.

CAUTION!
  • THE INSTANCE MUST BE ACTIVE OR STOPPED in order to request an image. Currently, Jetstream cannot image Suspended or Shelved instances.
  • ALL VOLUMES MUST BE DETACHED FROM THE INSTANCE. Failure to detach Volumes will result in a image whose child instances fail to boot.
  • INSTANCES MUST NOT BE DELETED UNTIL AFTER THE IMAGING PROCESS HAS COMPLETED.
  • Imaging Guidelines: There are several other restrictions and notes of which you must be aware in order to achieve a successful image. For example, several directories and files are removed during image creation.
  • At this time, only the creator of an image may update an image.
  • Please note that s1.* based customized instances will NOT be able to be used to create images in Atmosphere.

After submitting the form, the Jetstream Atmosphere support staff will review and process the request. Future versions of Atmosphere will allow users to initiate the VM imaging process automatically.

Be sure to test launch any image created to validate that it behaves as expected BEFORE suspending or removing the current active instance, since only active instances can be imaged.

BEST PRACTICE: It is recommended to upgrade the operating system before imaging. For CentOS based systems, it's sudo yum update. For Ubuntu based systems, do sudo apt-get update and then sudo apt-get upgrade.

VERY IMPORTANT NOTE
Creating an image on the SMALLEST possible size VM on which it will run will allow the image to be launched on VMs of the same size and larger.
For example, an image created on a Tiny size VM can be launched on a VM of any size; an image created on a Medium VM can only be launched on a Medium or larger size VM.

WARNING: Atmosphere vs. API
We cannot recommend strongly enough not using Atmosphere images on the API side of Jetstream or vice-versa.
The Atmosphere user interface is designed to be extremely easy to use, particularly for beginning users but the trade-off is that Atmosphere enforces certain practices behind the scenes.
The API user interface is close to raw openstack, makes almost no assumptions, requires advanced knowledge of Openstack and Linux, but is therefore much more flexible.
API and Atmosphere base images provided by Jetstream staff require SUBSTANTIAL work to make them CLEANLY work in the other environment.

Imaging Guidelines

Here are some tips to help ensure a viable importable image:

  • Operating system: Base the image on
    • CentOS6, CentOS 7 and later
    • Ubuntu 14.04 and later, Long Term Support (LTS) versions of Ubuntu recommended
  • File system: Ext3, Ext4, or XFS.
  • Image format: RAW or QCOW2.
  • Software: Image must contain no licensed software that would prohibit use within a cloud or virtualized environment. It is recommended that software be installed in /opt or /usr/local/.
  • New Image Name:
    DO NOT use the name of an existing featured image. Please be mindful of re-using the name of one of your own existing images.
    DO NOT use a period(".") as any of the last 5 characters of the name.
  • Description of the Image: DO add an informative description of your image and make note of how it varies from any base images.
  • Instance status: The instance MUST be ACTIVE or STOPPED in order to image. Suspended or Shelved images cannot be imaged at this time.
  • DO NOT delete Instances until AFTER the IMAGING process has completed.
  • Volumes: All Volumes MUST be DETACHED from the instance. Failure to detach Volumes will result in a image whose child instances fail to boot.
  • Owner: At this time, only the creator of an image may update an image. Developer teams wishing to update a shared image should contact Jetstream support.
  • BEST PRACTICE: Users are recommended to upgrade the operating system before imaging.
    For CentOS based systems:
    sudo yum update
    For Ubuntu based systems: do sudo apt-get update and then sudo apt-get upgrade

Before submitting a request for an image of your instance, remove the following software from the instance:

  • Licensed software: including software not purchased by Jetstream or otherwise not in the public domain.
  • Not cloud suitable: software in which the licensing, features, or activity of the software otherwise precludes its use or inhibits the activity of other software within a cloud or virtualized environment.

Volumes and iRODS FUSE mounts are not copied as part of the image.

VERY IMPORTANT NOTE:
CHOOSE SMALLEST SIZE
Users are encouraged to test launch any image created to validate that it behaves as expected before suspending or removing their current active instance.
Creating an image on the SMALLEST possible size VM on which it will run will allow the image to be launched on VMs of the same size and larger.
For example, an image created on a Tiny size VM can be launched on a VM of any size; an image created on a Medium VM can only be launched on a Medium or larger size VM.

About your /home directory

  • All files, directories, and icons located under /home/<username>/Desktop will be deleted. To preserve them, email help@jetstream-cloud.org.
  • Any files installed in /home must be saved to your volume, to iRODS, or to another storage device or system external to your VM.

Can an image be exported?
Yes...and no. Currently an automated method for exporting images to run on other systems is not available but if you find that you do need to export an image to something other than IU Scholarworks contact the XSEDE Help Desk.

CAUTION!

The following directories are deleted as part of the imaging process:

  • /home/
  • /mnt/
  • /tmp/
  • /root/

The following system files are typically overwritten by the Jetstream imaging process for security and operational reasons:

  • /etc/fstab
  • /etc/group
  • /etc/host.allow
  • /etc/host.conf
  • /etc/host.deny
  • /etc/hosts
  • /etc/ldap.conf
  • /etc/passwd
  • /etc/resolve.conf
  • /etc/shadow
  • /etc/sshd/
  • /etc/sysconfig/iptables
  • /root/
  • /var/log

Request an image

Before you begin
Read and follow the directions in Imaging Guidelines.

You can request an image (a type of template for a virtual machine) of a running instance. This saves a complete copy of all changes and updates made to the instance since it was launched it so it can be reused at any time. It also saves resources by launching the instance only when you need it.

You also can see the list of all image requests you have made.

You can add a script before requesting the image that executes after an instance using the image is launched and active.

PLEASE NOTE: Images will get created on the IU cloud by default, no matter which cloud they originated from. They will automatically be synced nightly to the TACC cloud.

CAUTION!

The following directories are deleted as part of the imaging process:

  • /home/
  • /mnt/
  • /tmp/
  • /root/

To get started:

  1. Log in to Jetstream web interface https://use.jetstream-cloud.org/.
  2. Detach all attached volumes from the instance. Detailed instructions.
  3. Click Projects on the menu bar and open the project with the instance to use for the new image.
  4. Click the instance name. The instance can have a status of Active or Stopped but it cannot be Suspended or Shelved.
  5. In the Actions list on the right, click Image.

Image Info

The information you provide on here will help others to discover this image.

  1. Create or Update? (optional): The default is to create a new image, but if you are updating an image to a new revision, you can uncheck this box and it will be created as a new version of the same image. You can see examples of this with the Ubuntu 18.04 Featured image on Atmosphere. We recommend you update the image when possible.
  2. New Image Name (required): Enter the name, up to 30 characters, to assign to the new image. Please DO NOT use the name of an existing featured image. Please be mindful of reusing the name of one of your own existing images.
  3. Description of the Image (required): The description should include keywords that concisely describe the tools installed, the purpose of the tools (e.g., This image performs X analysis), and the initial intent of the machine image (e.g. designed for XYZ workshop). Include keywords that will help users search for this image.
  4. Image Tags (optional): Click in the field and select tags that will enhance search results for this image. You may include the operating system, installed software, or configuration information (e.g. Ubuntu, NGS Viewers, MAKER, QIIME, etc.). Tags can be added and removed later, if needed.
  5. Click Next.

Figure 1: Image Request - Image Info screen

Version Info

Versioning is an important part of the imaging process. Use this information to track how your image changes over time. This information will also be helpful to others that wish to use your image.

  1. New Version Name (required): Enter the new (unique) name or number of the image. Versioning helps users understand how your changes relate to the overall progress of the Application. Versions are alphanumeric (e.g. 2.0-stable, 2.1-beta, 2.2-testing). Limit the name to 30 characters and keep versioning consistent.
  2. Change Log (required): Concisely describe what you've changed in this specific version. This description will help users understand how your application as changed over time.
  3. Click Next.

Figure 2: Image Request - Version Info screen

Provider

  1. Select the cloud provider to use for the image. If you would like the image to be available on multiple clouds, email help@jetstream-cloud.org.
  2. Indicate minimum CPU and memory requirements (optional).
  3. Click Next.

Figure 3: Image Request - Provider screen

Privacy

  1. Select the visibility for the image:

    • Public: The image will be visible to all users and anyone will be able to launch it.
    • Private: The image will be visible only to you and only you will be able to launch it.
    • Specific Users: The image will be visible to only you and the users you specify. Only you and those specific users will be able to launch it. If you chose Specific Users, select the users who will be able to launch the image.
  2. Click Advanced Options or Submit.

    Advanced Options will allow you to:

    • Exclude files from the image
    • Add a deployment script
    • Require the user to verify understanding of any license restrictions

Figure 4: Image Request - Privacy screen

Advanced Option

Exclude Files

Note the list of directories that will automatically be excluded form the image:

  • /home/
  • /mnt/
  • /tmp/
  • /root/

In the box provided, list any additional files or directories to be excluded from the image. Write one path per line.

Figure 5: Image Request - Exclude Files screen

Advanced Option

Boot Scripts & Licenses

Deployment scripts are executed when a user launches the image and each time an instance is 'Started', 'Resumed', or 'Restarted'. These scripts should be able to handle being run multiple times without adverse effects.

Click Next to continue to the next screen without adding a new script or a software license.

  1. To add a deployment script, click in the search field and search for the title of the script.
  2. To create a new deployment script:
    1. Click Create New Script, enter a title for the script, then either click URL and enter the script URL or click Full Text and enter the deployment script.
    2. When done, click Create and Add, then click Next.
  1. To list any licensed software used in the image and require users to agree to the license agreement before launching, click in the search field and search for the license title.
  2. To create a new license:
    • Click Create New License, enter a title for a license, then either click URL and enter the license URL or click Full Text and enter the full license text.
    • When done, click Create and Add, then click Next.

    Figure 6: Image Request - Boot Scripts & Licenses screen

Review Image Request

On the Review screen, verify the information entered on the previous screens. Click Back to return to the previous screens and make corrections. When all is OK, click the checkbox certifying that the license does not contain any license-restricted software that is prohibited from being distributed within a virtual or cloud environment..

  1. Click Request Image.

You will receive an email from Support when the image is completed. Please email questions to help@jetstream-cloud.org.

Figure 7: Image Request - Review screen

Viewing your images

You can view your lists of images and image requests.

  1. Click Images on the top menu bar.
  2. Click:
    • MY IMAGES to view the list of your images.
    • MY IMAGE REQUESTS to view the list of your image requests.

Figure 8: Viewing a list of images

Request image deletion

Currently, the only way to delete (archive) an image you requested is to email help@jetstream-cloud.org. You will receive an email confirmation when your image has been archived.

However, if you are the owner of an image, you may change the "Date to hide image from public view" as well as the "Visibility" for your image.

To do so, select Images, then select My Images.

Select the image you wish to modify and click Edit Details.

Image publication

Submit your Jetstream image for storage and digital object identifier (DOI)

Images that you have customized and saved may be made available for publications or to the public, within certain limits (for example, they may not contain secure or protected information or data).

To request an image be stored, given a digital object identifier (DOI), please follow these instructions.

Please be aware that at the current time, such a process requires the manual intervention of staff and is therefore subject to scheduling limits.

Jetstream EZ Tools

Instances launched using the Atmosphere web user interface are by default installed with the EZ command line tools, designed to streamline some common tasks and functions.

There are a number of different ez commands, including:

  • ez a help menu listing available tools
  • ezj run jupyter-notebook with python detection; m1.medium or larger recommended
  • ezjh run jupyterhub (currently requires CyVerse CAS credentials) m1.medium or larger recommended
  • ezd install docker (Only supported on Ubuntu 14+ and CentOS 7)
  • ezs install singularity (Only supported on Ubuntu 14+ and CentOS 6+)
  • unban list blocked IP addresses or remove blocks for IP addresses trying to access the instance
  • myip set environment variables $JETSTREAM_PUBLIC_IP and $CYVERSE_PUBLIC_IP to the public IP Address and display the same to the screen

ez: a help menu listing available tools

Options to pass to ez:

update : perform a full update of the ez facility

~$ ez update

ezj: run jupyter-notebook with python detection

Also see details for launching Jupyter Notebook within an instance.

Options to pass to ezj:

-q : do not attempt to install, just launch jupyter!

-R : install the R kernel (-r also works)

-2 : force python 2 kernel (not compatible with -3 option)

-3 : force python 3 kernel (not compatible with -2 option)

-u : force update of anaconda (default is no update)

-p : takes a directory as an option; install in a different location other than default (/home)

NOTE: if you set this, you must pass it again for future calls

You do not need to run ezj more than once. You can start/restart a notebook with the commands:

~$ ezj -q

or

~$ jupyter-notebook

Example usage:

~$ ezj -u -R
~$ ezj -p /opt -u

ezjh: run jupyterhub with CyVerse CAS integration

Options to pass to ezjh:

none

NOTE: Only run if you have CyVerse access and are familiar with CAS.

The installation may take up to 10 minutes to complete.

You do not need to run ezjh more than once. You can start/restart a hub with the command:

~$ jupyterhub

ezd: install docker

Options to pass to ezd:

-p : enable portainer on port 9000

~$ ezd

ezs: install singularity

Options to pass to ezs:

none

~$ ezs

unban: list blocked IP addresses or remove blocks for IP addresses trying to access the instance

Usage: unban [-l] [-i ip-address] [-a]

-l : list currently banned IPs

-i : unban a single IP address

-a : unban all IP addresses

NOTE: use only one of the options at a time

myip : set environment variables $JETSTREAM_PUBLIC_IP and $CYVERSE_PUBLIC_IP to the public IP Address and display the same to the screen

~$ myip

~$ echo $JETSTREAM_PUBLIC_IP

Launching Jupyter Notebook within an instance

The following steps will easily install and start a Jupyter notebook on any active instance.

NOTE: The Jupyter install generally requires more disk-space/memory than an m1.tiny or m1.small has, so we recommend m1.medium or larger.

Step-by-step guide

  1. Login to the instance using web shell or an ssh client.
  2. Type ezj. Jupyter notebook will then be installed and configured within a couple of minutes.

    ~$ ezj

  3. After the installation is complete, a URL will be shown. You can use any browser to connect to this URL.

    CAUTION: Closing your Jupyter notebook
    If you close your web shell or your ssh session, the Jupyter notebook will shutdown.

  4. To restart your Jupyter notebook without reinstalling:

    ~$ ezj -q

If you have Python3 installed, you may select it instead using:

~$ ezj 3

If you wish to choose Python2 use:

~$ ezj 2

Other options to pass to 'ezj':

-q : do not attempt to install, just launch jupyter!

-R : install the R kernel (-r also works)

-2 : force python 2 kernel (not compatible with -3 option)

-3 : force python 3 kernel (not compatible with -2 option)

-u : force update of anaconda (default is no update)

-p : takes a directory as an option; install in a different location other than default (/home)

NOTE: if you set this, you must pass it again for future calls

Example of options:

~$ ezj -u -R
~$ ezj -p /opt -u

Interested in combining your Jupyter Notebook with R-studio? See this EZJ + R-studio installation guide from the CyVerse team.

Atmosphere CLI

"atmo", a python-based command line interface for Atmosphere based on "cliff" (the same package underlying the "openstack" commands) is available on Jetstream.

Documentation and Installation

There is documentation for the package, including installation instructions.

We encourage users to install atmo locally for each user:

pip install --user atmosphere-cli

Jetstream Setup

Add the following to your .bashrc or a file called .env in your $HOME directory:

export ATMO_BASE_URL="https://use.jetstream-cloud.org"
export ATMO_AUTH_TOKEN="fff28djdFAKETOKENfff28djdFAKETOKENfff28djd"

Token

To get a Personal Access Token:

  • Launch a browser and login to use.jetstream-cloud.org.
  • In the top right, you'll see your username with a drop-down menu.
  • Click on "Settings" .
  • Click on "Show More" under Advanced.
  • Scroll down to "Persona Access Tokens" and Click on the "+" button.
  • Give your token a name and click on "Create Token".
  • Save this Token by clicking on the "Copy" button and pasting it somewhere you can recover it. THIS IS YOUR ONLY CHANCE TO SAVE THIS TOKEN. YOU WILL HAVE TO DELETE THE TOKEN AND START OVER IF YOU DO NOT SAVE IT.
  • You will likely wish to save this token into your .env in the ATMO_AUTH_TOKEN line as above.

Atmo Usage

prompt> atmo --help
usage: atmo [--version] [-v | -q] [--log-file LOG_FILE] [-h] [--debug]
            [--atmo-base-url <atmosphere-base-url>]
            [--atmo-auth-token <atmosphere-auth-token>]
            [--atmo-api-server-timeout <atmosphere-api-server-timeout>]

Atmosphere CLI

optional arguments:
  --version             show program's version number and exit
  -v, --verbose         Increase verbosity of output. Can be repeated.
  -q, --quiet           Suppress output except warnings and errors.
  --log-file LOG_FILE   Specify a file to log output. Disabled by default.
  -h, --help            Show help message and exit.
  --debug               Show tracebacks on errors.
  --atmo-base-url <atmosphere-base-url>
                        Base URL for the Atmosphere API (Env: ATMO_BASE_URL)
  --atmo-auth-token <atmosphere-auth-token>
                        Token used to authenticate with the Atmosphere API
                        (Env: ATMO_AUTH_TOKEN)
  --atmo-api-server-timeout <atmosphere-api-server-timeout>
                        Server timeout (in seconds) when accessing Atmosphere
                        API (Env: ATMO_API_SERVER_TIMEOUT)

Commands:
  allocation source list  List allocation sources for a user.
  allocation source show  Show details for an allocation source.
  complete       print bash completion command (cliff)
  group list     List groups for a user.
  group show     Show details for a group.
  help           print detailed help for another command (cliff)
  identity list  List user identities managed by Atmosphere.
  identity show  Show details for a user identity.
  image list     List images for user.
  image search   Search images for user.
  image show     Show details for an image.
  image version list  List image versions for an image.
  image version show  Show details for an image version.
  instance actions  Show available actions for an instance.
  instance attach  Attach a volume to an instance.
  instance create  Create an instance.
  instance delete  Delete an instance.
  instance detach  Detach a volume from an instance.
  instance history  List history for instance.
  instance list  List instances for user.
  instance reboot  Reboot an instance.
  instance redeploy  Redeploy to an instance.
  instance resume  Resume an instance.
  instance shelve  Shelve an instance.
  instance show  Show details for an instance.
  instance start  Start an instance.
  instance stop  Stop an instance.
  instance suspend  Suspend an instance.
  instance unshelve  Unshelve an instance.
  maintenance record list  List maintenance records for Atmosphere.
  maintenance record show  Show details for a maintenance record.
  project create  Create a project.
  project list   List projects for a user.
  project show   Show details for a project.
  provider list  List cloud providers managed by Atmosphere.
  provider show  Show details for a cloud provider.
  size list      List sizes (instance configurations) for cloud provider.
  size show      Show details for a size (instance configuration).
  version        Show Atmosphere API version.
  volume create  Create a volume.
  volume delete  Delete a volume.
  volume list    List volumes for a user.
  volume show    Show details for a volume.

For example, to list instances for the current user:

prompt> atmo provider list
+----+--------------------------------------+--------------------------------+-----------+----------------+--------+--------+------------+
| id | uuid                                 | name                           | type      | virtualization | public | active | start_date |
+----+--------------------------------------+--------------------------------+-----------+----------------+--------+--------+------------+
|  4 | f906a5ee-34a8-499a-9185-a35feb3d6f01 | Jetstream - Indiana University | OpenStack | KVM            | True   | True   | 2016-01-28 |
|  5 | 3ff65aa8-505b-48c3-aef1-aa0ada14c756 | Jetstream - TACC               | OpenStack | KVM            | True   | True   | 2016-02-16 |
+----+--------------------------------------+--------------------------------+-----------+----------------+--------+--------+------------+

Launching an instance with ATMO

To launch an instance you'll need to know the UUIDs for:

  • cloud provider (identity)
  • flavor-size
  • image source on that cloud
  • project
  • XSEDE allocation

If we want to launch an "m1.small" of "Ubuntu 18.04 Devel and Docker" on the Indiana cloud with "my-U18-Indiana" we will need to do the following:

  1. Get UUID of cloud provider (aka IDENTITY UUID)

    prompt> atmo identity list
    +--------------------------------------+--------+--------------------------------+-----------+--------------+---------------+
     | uuid                                 | name   | provider                       | quota_cpu | quota_memory | quota_storage |
     +--------------------------------------+--------+--------------------------------+-----------+--------------+---------------+
     | 88dea875-a8d6-483a-bdb3-4446651e4a71 | beckbw | Jetstream - Indiana University |       132 |          360 |           500 |
     | 08653592-af99-4f47-8c6d-3709a6deaf02 | beckbw | Jetstream - TACC               |       132 |          360 |           500 |
     +--------------------------------------+--------+--------------------------------+-----------+--------------+---------------+

    >>>>> REMEMBER IDENTITY-UUID (e.g. 88dea875-a8d6-483a-bdb3-4446651e4a71 for Indiana)

  2. Get UUID and SIZE-ALIAS of flavor on chosen cloud provider

    prompt> atmo size list
    +--------------------------------------+------------+-------+--------------------------------+-----+--------+------+--------+------------+
     | uuid                                 | name       | alias | provider                       | cpu | memory | disk | active | start_date |
     +--------------------------------------+------------+-------+--------------------------------+-----+--------+------+--------+------------+
     ...
     | 4912d26c-db76-4095-a007-25bdb8e2def8 | m1.small   | 2     | Jetstream - TACC               |   2 |   4096 |   20 | True   | 2016-03-08 |
     | 4612070a-791b-4874-88bb-e15636805ec7 | m1.small   | 2     | Jetstream - Indiana University |   2 |   4096 |   20 | True   | 2016-01-29 |
     | f663b727-786f-47f5-9122-9b1b1c4f3529 | m1.medium  | 3     | Jetstream - TACC               |   6 |  16384 |   60 | True   | 2016-03-08 |
     | d3aab647-d81a-4842-9cdc-3a9fcbb5c588 | m1.medium  | 3     | Jetstream - Indiana University |   6 |  16384 |   60 | True   | 2016-01-29 |
     ...
     +--------------------------------------+------------+-------+--------------------------------+-----+--------+------+--------+------------+

    >>>>> REMEMBER SIZE-UUID (e.g. 4612070a-791b-4874-88bb-e15636805ec7 for m1.small on Indiana)
    >>>>> REMEMBER SIZE-ALIAS (e.g. 2 for m1.small on Indiana)

  3. Get UUIDs

    1. Get UUID of image

      prompt> atmo image list --tag-name "Featured"
      +--------------------------------------+----------------------------------------+------------+-----------+------------+
       | uuid                                 | name                                   | created_by | is_public | start_date |
       +--------------------------------------+----------------------------------------+------------+-----------+------------+
       ...
       | c45c94f9-64d1-5e49-8942-0440718ef1f2 | Matlab Minimal                         | jfischer   | True      | 2018-10-26 |
       | 4afad299-3b41-5133-8326-10900b7aa6e7 | Ubuntu 18.04 Devel and Docker          | jfischer   | True      | 2018-10-25 |
       | 977e1233-54a4-51f5-97a9-560efd53b975 | Centos 7 (7.5) Development GUI         | jfischer   | True      | 2018-10-25 |
       | c97c4d5e-fe15-5156-b519-0cdb4021492b | Ubuntu 16.04 Devel and Docker          | jfischer   | True      | 2018-10-25 |
       ...
       +--------------------------------------+----------------------------------------+------------+-----------+------------+

      >>>>> REMEMBER IMAGE-UUID (e.g. 4afad299-3b41-5133-8326-10900b7aa6e7 for Ubuntu 18.04 Devel and Docker)

    2. Get UUID of the desired VERSION of the image

      prompt> atmo image show 4afad299-3b41-5133-8326-10900b7aa6e7
       (IMAGE-UUID from above)
      +-------------+------------------------------------------------------------------------------------------------+
       | Field       | Value                                                                                          |
       +-------------+------------------------------------------------------------------------------------------------+
       | id          | 717                                                                                            |
       | uuid        | 4afad299-3b41-5133-8326-10900b7aa6e7                                                           |
       | name        | Ubuntu 18.04 Devel and Docker                                                                  |
       | description | Ubuntu 18.04 LTS Development + GUI support + Docker                                            |
       |             |                                                                                                |
       |             | Based on Ubuntu cloud image for 18.04 LTS with basic dev tools, GUI/Xfce added                 |
       |             |                                                                                                |
       |             | Installation size ~ 4.4 GB                                                                     |
       | created_by  | jfischer                                                                                       |
       | versions    | 1.0 (fe8deab2-e72f-40a5-aef4-5b908e3d5a6f)                                                     |
       |             | 1.8 (dfe3c474-0711-4ca6-bef9-5b61c5a7267b)                                                     |
       |             | 1.10 (3edba696-b13b-4661-ab80-3d1a74a2d34e)                                                    |
       |             | 1.11 (309300be-2dc8-45b8-890e-7623b272efec)                                                    |
       |             | 1.12 (9c9e9963-84e7-4340-812c-e91304c2fe41)                                                    |
       |             | 1.13 (1df21c86-c940-41d8-88d6-e231fb0d401f)                                                    |
       | tags        | desktop, base, vnc, development, Ubuntu, Featured, docker, docker-compose                      |
       | url         | https://use.jetstream-cloud.org/api/v2/images/4afad299-3b41-5133-8326-10900b7aa6e7?format=json |
       | is_public   | True                                                                                           |
       | start_date  | 2018-10-25                                                                                   
      | end_date | | +-------------+------------------------------------------------------------------------------------------------+

      >>>>> REMEMBER VERSION-UUID (e.g. 1df21c86-c940-41d8-88d6-e231fb0d401f for Ubuntu 18.04 Devel and Docker Version 1.13)

    3. Get UUID of that VERSION specific to the chosen cloud (aka. MACHINE UUID)

      prompt> atmo image version show 1df21c86-c940-41d8-88d6-e231fb0d401f
      +-------------------+--------------------------------------------------------------------------------+
       | Field             | Value                                                                          |
       +-------------------+--------------------------------------------------------------------------------+
       | id                | 1df21c86-c940-41d8-88d6-e231fb0d401f                                           |
       | name              | 1.13                                                                           |
       | image_name        | Ubuntu 18.04 Devel and Docker                                                  |
       | image_description | Ubuntu 18.04 LTS Development + GUI support + Docker                            |
       |                   |                                                                                |
       |                   | Based on Ubuntu cloud image for 18.04 LTS with basic dev tools, GUI/Xfce added |
       |                   |                                                                                |
       |                   | Installation size ~ 4.4 GB                                                     |
       | created_by        | jfischer                                                                       |
       | change_log        | v1.13 - patched up to 10-25-18                                                 |
       | machines          | Jetstream - Indiana University (8567d635-ab74-4013-9a21-30821b9a7301)          |
       |                   | Jetstream - TACC (8567d635-ab74-4013-9a21-30821b9a7301)                        |
       | allow_imaging     | True                                                                           |
       | min_mem           | None                                                                           |
       | min_cpu           | None                                                                           |
       | start_date        | 2018-10-25T14:55:21.136101Z                                                    |
       +-------------------+--------------------------------------------------------------------------------+

      >>>>> REMEMBER MACHINE-UUID (e.g. 8567d635-ab74-4013-9a21-30821b9a7301 for Ubuntu 18.04 version 1.13 on Indiana)

  4. Get UUID of the PROJECT to which you will assign the instance

    prompt> atmo project list
    +--------------------------------------+-----------+--------+------------+------------+--------+-----------+---------+-------+
     | uuid                                 | name      | owner  | created_by | start_date | images | instances | volumes | links |
     +--------------------------------------+-----------+--------+------------+------------+--------+-----------+---------+-------+
     ...
     | a9d28bb7-44d6-4914-8d1a-64673d12ebf2 | testing   | beckbw | beckbw     | 2016-02-10 |      0 |         7 |       3 |     1 |
     ...
     +--------------------------------------+-----------+--------+------------+------------+--------+-----------+---------+-------+

    >>>>> REMEMBER PROJECT-UUID (e.g. a9d28bb7-44d6-4914-8d1a-64673d12ebf2 for "testing" project)

  5. Get UUID of the ALLOCATION against which you the instance will charge service units

    prompt> atmo allocation source list
    +--------------------------------------+--------------+-----------------+--------------+------------------+------------+
     | uuid                                 | name         | compute_allowed | compute_used | global_burn_rate | start_date |
     +--------------------------------------+--------------+-----------------+--------------+------------------+------------+
     ...
     | fd7de822-6398-4bfb-b6dd-47958b21178d | TG-TRA160003 |         5000000 |  3981007.652 |            47.75 | 2017-06-13 |
     +--------------------------------------+--------------+-----------------+--------------+------------------+------------+

    >>>>> REMEMBER ALLOCATION-UUID (e.g. fd7de822-6398-4bfb-b6dd-47958b21178d for "TG-TRA160003" allocation)

  6. CREATE INSTANCE – Now we can combine all those to launch an instance using the following syntax:

    1. CREATE INSTANCE USING MACHINE UUID

      prompt> atmo instance create --identity    --size-alias  --source-alias  --project  --allocation-source-id    

      prompt> atmo instance create --identity 88dea875-a8d6-483a-bdb3-4446651e4a71  --size-alias  2 --source-alias 8567d635-ab74-4013-9a21-30821b9a7301 --project a9d28bb7-44d6-4914-8d1a-64673d12ebf2 --allocation-source-id fd7de822-6398-4bfb-b6dd-47958b21178d "my-U18-Indiana"

    2. Alternative: CREATE INSTANCE USING ONLY IMAGE UUID (merge steps 3a-c) – You can also use the "--image" option to combine the "image list", "image show", "image version show", and "--source-alias" steps into one step that automatically chooses the latest version and machine appropriate to the cloud identity you have specified:

      prompt> atmo instance create --identity    --size-alias  --image  --project  --allocation-source-id    

      prompt> atmo instance create --identity 88dea875-a8d6-483a-bdb3-4446651e4a71  --size-alias  2 --project a9d28bb7-44d6-4914-8d1a-64673d12ebf2 --allocation-source-id fd7de822-6398-4bfb-b6dd-47958b21178d --image 4afad299-3b41-5133-8326-10900b7aa6e7 "my-newVM"

Optional command line auto-completion

Auto-completion functions as in bash and supports tab-based auto-completion.

    prompt> atmo complete >> ~/.bash_aliases
    prompt> . ~/.bash_aliases
    prompt> atmo <tab>
    allocation  complete    help        identity    image       instance    project     provider    size        version     volume
    prompt> atmo instance <tab>
    actions   attach    create    delete    detach    history   list      reboot    redeploy  resume    shelve    show      start     stop      suspend   unshelve
    prompt> atmo provider <tab>
Error message or problem Solution
prompt> atmo identity list
ERROR: {'detail': 'Authentication credentials were not provided.'}
You likely have a WRONG/INVALID ATMO_AUTH_TOKEN string.
Obtain a new Personal Access Token from the Atmosphere GUI as described above and reset your ATMO_AUTH_TOKEN environment variable.
I've changed/started/deleted an instance but atmo instance list doesn't reflect the changes. Go to the GUI and do a force-refresh (cmd/ctrl-R). Refreshing the GUI refreshes the API. Currently there is no force-refresh for the CLI.
ERROR: {'detail': ' - v32 deployment: Atmosphere is down for a Scheduled Maintenance, Today between 9am - 4pm MST.'} Self-explanatory

Useful atmo-cli based scripts

These are provided as-is and are not formally supported.

atmo_launch_instance.sh

This script uses atmo commands to find all needed UUID to launch an instance.

prompt> ./atmo_launch_instance.sh -h
Usage: atmo_launch [+/-d] [ - :DEFAULT, NODEBUG | +:DEBUG]
                   -c <provider cloud; TACC|Indiana; DEFAULT=Indiana>
                   -s <flavor;  DEFAULT="m1.small">
                   -i <image search string; DEFAULT="Ubuntu 16.04">
                   -v <image version string; DEFAULT=<latest> >
                   -p <project name; DEFAULT="testing">
                   -a <allocation; DEFAULT="TG-TRA160003">
                   -n <provider cloud; TACC|Indiana; DEFAULT=Indiana>
                   [-h] [this usage message]
prompt> ./atmo_launch_instance.sh +d -v 1.13 -c Indiana -n "bootscoot-IU"
Searching...
idval 88dea875-a8d6-483a-bdb3-4446651e4a71
szval 2
imgval c97c4d5e-fe15-5156-b519-0cdb4021492b
verval 04f6c686-6b8d-4be9-b277-16bf2a4a16e6
machstr 7505ea37-2fbb-499f-b150-c18fade5ce26
machval 7505ea37-2fbb-499f-b150-c18fade5ce26
srcval 7505ea37-2fbb-499f-b150-c18fade5ce26
prjval a9d28bb7-44d6-4914-8d1a-64673d12ebf2
allocval fd7de822-6398-4bfb-b6dd-47958b21178d
nameval bootscoot-IU
atmo instance create --identity  88dea875-a8d6-483a-bdb3-4446651e4a71  --size-alias  2      --source-alias 7505ea37-2fbb-499f-b150-c18fade5ce26 --project  a9d28bb7-44d6-4914-8d1a-64673d12ebf2  --allocation-source-id  fd7de822-6398-4bfb-b6dd-47958b21178d  bootscoot-IU
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| id                | 22134                                |
| uuid              | 9c1b79ce-9fcc-443d-978d-2536f8833d94 |
| name              | bootscoot-IU                         |
| username          | beckbw                               |
| allocation_source | TG-TRA160003                         |
| image_id          | 107                                  |
| image_version     | 1.13                                 |
| launched          | 2018-04-18                           |
| image_size        | m1.small                             |
| provider          | Jetstream - Indiana University       |
+-------------------+--------------------------------------+

instance_select.sh

Select from a menu of your current instances.

prompt> atmo instance show $(./instance_select.sh |grep ^[0-9]| tail -n 1 | awk '{print $3}')
1) BECK-C74-IU       4) BECK-C74-TACC     7) beck-C74-IU
2) BECK-U16-TACC     5) beckbri-U16-TACC  8) Quit
3) BECK-U16-IU       6) bootscoot-IU
Select:3
Select:5
Select:7
Select:8

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| id                | 22135                                |
| uuid              | db4a937b-9ef2-47fc-9cce-34f6476087dd |
| name              | beck-C74-IU                          |
| username          | beckbw                               |
| identity          | Username: beckbw, Project:beckbw     |
| project           | testing                              |
| allocation_source | TG-TRA160003                         |
| compute_allowed   | 5000000                              |
| compute_used      | 1096774.11                           |
| global_burn_rate  | 24.0                                 |
| user_compute_used | 2887.29                              |
| user_burn_rate    | 2.0                                  |
| image_id          | 241                                  |
| image_version     | 1.22                                 |
| image_usage       | 1096774.11                           |
| launched          | 2018-04-18                           |
| image_size        | m1.small                             |
| image_cpu         | 2                                    |
| image_mem         | 4096                                 |
| image_disk        | 20                                   |
| status            | active                               |
| activity          |                                      |
| ip_address        | 149.165.157.11                       |
| provider          | Jetstream - Indiana University       |
| web_desktop       | True                                 |
| shell             | False                                |
| vnc               | True                                 |
+-------------------+--------------------------------------+
prompt>

Getting Scientific Software

Jupyter Notebooks

All Jetstream instances can now launch Jupyter notebooks from the command line.

Pre-packaged software

  • If you are using a CentOS/rpm based VM, you can utilize software packaged by the XSEDE Campus Bridging team for the XSEDE National Integration Toolkit (XNIT).

    • The XNIT, formerly known as the XSEDE Yum Repository, is a collection of Red Hat Package Manager (RPM) packages assembled to simplify the process of converting a "bare-bones" Linux cluster into a high-performance, parallel computing system that can be used to support scientific discovery. The packages included in the repository are specific versions and builds of scientific, mathematical, and visualization applications recommended by the Extreme Science and Engineering Discovery Environment (XSEDE) for optimal compatibility with XSEDE digital services.

    • Please see the Knowledge Base entry about XNIT for details on what software is available and how to set up the XNIT repository on your VM.

  • If you are using a Debian based system such as Ubuntu, you can use the Alien package to convert RPMs to DEB packages for installation. This is not supported and may not perform exactly as expected. For more information on Alien, see Converting .rpm Packages To Debian/Ubuntu .deb Format With Alien.

Image Specific Documentation

Featured Images

Featured images are installed, verified, and maintained by Jetstream staff and are the recommend starting points for your use or development efforts.

View the list of Featured images.

Custom Images

Some of the more commonly used custom images include:

Though able to fit into 2-cores instances, a 2-core system will naturally perform more slowly than larger flavor sizes.

WARNING: Atmosphere vs. API
We cannot recommend strongly enough not using Atmosphere images on the API side of Jetstream or vice-versa.
The Atmosphere user interface is designed to be extremely easy to use, particularly for beginning users but the trade-off is that Atmosphere enforces certain practices behind the scenes.
The API user interface is close to raw openstack, makes almost no assumptions, requires advanced knowledge of Openstack and Linux, but is therefore much more flexible.
API and Atmosphere base images provided by Jetstream staff require SUBSTANTIAL work to make them CLEANLY work in the other environment.

Images and the Intel compiler

The environment variables for using the Intel compiler are set for users in /etc/skel/.bash_profile. If you want to set them manually and/or use it for the root user, you can copy the variables below.

Please note that while we don't anticipate there being issues for the root user to have these in the default root user environment, there may be unintended consequences.

# Intel environment variables
# environment variables for Intel parallel_studio_xe_2016_updater3
export INTEL_LICENSE_FILE=/opt/intel/licenses/
export PATH=/opt/intel/compilers_and_libraries/linux/bin/intel64:/opt/intel/compilers_and_libraries/linux/mpi/intel64/bin:/opt/intel/debugger_2016/gdb/intel64_mic/bin:$PATH
export MANPATH=/opt/intel/man/common:/opt/intel/compilers_and_libraries/linux/mpi/man:/opt/intel/compilers_and_libraries/linux/man/en_US:/opt/intel/documentation_2016/en/debugger/gdb-ia/man/:/opt/intel/documentation_2016/en/debugger/gdb-mic/man/:/opt/intel/documentation_2016/en/debugger/gdb-igfx/man/:$MANPATH
export LIBRARY_PATH=/opt/intel/compilers_and_libraries/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries/linux/mkl/lib/intel64:/opt/intel/compilers_and_libraries/linux/tbb/lib/intel64/gcc4.4:/opt/intel/compilers_and_libraries/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries/linux/daal/../compiler/lib/intel64_lin:$LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries/linux/mkl/lib/intel64:/opt/intel/compilers_and_libraries/linux/tbb/lib/intel64/gcc4.4:/opt/intel/debugger_2016/libipt/intel64/lib:/opt/intel/compilers_and_libraries/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries/linux/daal/../compiler/lib/intel64_lin:$LD_LIBRARY_PATH
export CPATH=/opt/intel/compilers_and_libraries/linux/ipp/include:/opt/intel/compilers_and_libraries/linux/mkl/include:/opt/intel/compilers_and_libraries/linux/tbb/include:/opt/intel/compilers_and_libraries/linux/daal/include:$CPATH
export NLSPATH=/opt/intel/compilers_and_libraries/linux/compiler/lib/intel64/locale/en_US:/opt/intel/compilers_and_libraries/linux/mkl/lib/intel64/locale/en_US:/opt/intel/debugger_2016/gdb/intel64_mic/share/locale/en_US:/opt/intel/debugger_2016/gdb/intel64/share/locale/en_US:$NLSPATH
export INFOPATH=/opt/intel/documentation_2016/en/debugger/gdb-ia/info/:/opt/intel/documentation_2016/en/debugger/gdb-mic/info/:/opt/intel/documentation_2016/en/debugger/gdb-igfx/info/:INFOPATH
export CLASSPATH=/opt/intel/compilers_and_libraries/linux/daal/lib/daal.jar:$CLASSPATH
export IPPROOT=/opt/intel/compilers_and_libraries/linux/ipp
export MKLROOT=/opt/intel/compilers_and_libraries/linux/mkl/
export TBBROOT=/opt/intel/compilers_and_libraries/linux/tbb
export DAALROOT=/opt/intel/compilers_and_libraries/linux/daal
export I_MPI_ROOT=/opt/intel/compilers_and_libraries/linux/mpi
export INTEL_PYTHONHOME=/opt/intel/debugger_2016/python/intel64/
export GDBSERVER_MIC=/opt/intel/debugger_2016/gdb/targets/mic/bin/gdbserver
export GDB_CROSS=/opt/intel/debugger_2016/gdb/intel64_mic/bin/gdb-mic
export MPM_LAUNCHER=/opt/intel/debugger_2016/mpm/mic/bin/start_mpm.sh

BioLinux 8

Using x2go with the BioLinux 8 image

The BioLinux 8 image comes bundled by default with x2go. While x2go is not the preferred desktop sharing method for Jetstream, it is very functional and does work well. At this time, VNC is not functioning properly on the BioLinux 8 image.

To get started, download the x2go client. The client for Windows, OSX, and multiple versions of Linux is available from the x2go site, includes installation instructions (all platforms).

After installation, launch the client and set up a new connection:

  • Session name: enter a descriptive name for the connection
  • Host: Your_VM_IP_address (or DNS name)
  • Login: Your_XSEDE_username
  • To login via SSH enter your Jetstream SSH key.
    • Optional: check Try auto login (via SSH agent or default SSH key)
  • Session Type: Choose "Custom Desktop" and Command: "MATE"
    • or Session Type "MATE" and Command : "" (leave blank)
  • Click OK to save the connection.

Once saved, click on the connection to bring up the virtual machine GUI desktop. Note: it may take 1-2 minutes for the connection to come up.

To use password authentication rather than SSH, set the password on your VM.

  1. Once the instance comes up, login via ssh.
  2. Type 'sudo passwd your_username'.

TROUBLESHOOTING: x2go keyboard map wrong
If your x2go session starts with a keyboard map that doesn't match your device, use your x2goclient program on your remote host to suspend, then resume your x2go session.

Genomics Toolkit image

Tools included with this image

  • abyss
  • bamtools
  • bbmap
  • bcftools
  • bedtools
  • blast
  • bowtie
  • bowtie2
  • bwa
  • cufflinks
  • cutadapt
  • edge-pro
  • fasta3
  • fastqc
  • gatk4
  • hisat2
  • htseq
  • igv
  • jbrowse
  • manta
  • mrbayes
  • multiqc
  • picard
  • raxml
  • samtools
  • spades
  • star
  • stringtie
  • tabix
  • tcoffee
  • tophat
  • trimmomatic
  • vcftools

R packages included with this image

  • ballgown
  • bioconductor
  • deseq2
  • edgeR

Using the Jetstream API

To request OpenStack API access, email help@xsede.org with the subject "Requesting Jetstream API access for the XYZ PROJECT". In your email, answer as many of the questions listed below as possible.

  • Briefly describe what you want to do and how you plan to do it.
  • What will runtime operations look like?
  • Does this application exist already or is this primarily a development effort? If yes, what parts exist and where can they be referenced?
  • What API are you planning on using, Openstack, RADOS, Atmosphere, EC2, S3, something else?
  • Are any of these gateways? Might it spawn a gateway in the future?
  • What resources could you conceive this project consuming once in production? Please include sustained as well as peak resource demands.
  • Do you anticipate bursting into/out of Jetstream?
  • Jetstream contains more than one OpenStack cloud. Were you aware of this and do you think your implementation will be able to leverage multiple clouds initially? Eventually?
  • Are there long term storage needs, for example for reference data, that need to be accessible from the running instances? If so, how much storage is needed?
  • Are there run-time scratch/transient/workbench storage needs?
  • What is/are the XSEDE project name(s)?
  • Who is working on this project and what are their roles?
  • Is there a group account that development will occur under?
  • Is there one primary developer that will be using this account; or, will a group of individuals be developing code within this account?
Important info about the Jetstream API

Jetstream's OpenStack API is open to advanced users. Jetstream API users should understand the following:

  1. You will be expected to have familiarity with being a unix system administrator. These are advanced areas of operation and require a more advanced skill set than using Jetstream via Atmosphere.
  2. Jetstream Support may not be able to help with issues beyond VM functionality and other OpenStack issues. Software and services such as databases, local or remote applications, or other server-based components are up to the user/developer to install, deploy, and maintain.
  3. The current API environment is always under development. The base services are available and many features are added and upgraded as Jetstream matures.
  4. Jetstream staff and systems administrators will prioritize and address issues as they arise.
  5. Access may be revoked or restricted to insure the effective and efficient delivery of production services.
  6. This is an OpenStack API not an Atmosphere API. When you open up Atmosphere your OpenStack VMs will not be displayed and vice versa. We discourage manually deploying Atmosphere images in API and vice versa as there are many configurational differences that can lead to deployment problems.
  7. PIs and developers must be aware of, and conform to, any and all policies required for users of the systems owned and/or operated by the service providers, Indiana University (IU) and Texas Advanced Computing Center (TACC).

    IU

    TACC

Jetstream API developers

For policy details and information on how to request access to the Jetstream API see, Using the Jetstream API.

Also note that several Jetstream "Featured" API images are available. They will be named JS-API-Featured-xxxxxx

openstack image list | grep -iE '(JS-API)'

or

glance image-list | grep -iE '(JS-API)'

See OpenStack command line for other common commands.

  • The project PI should ensure that all accounts have been associated with their XSEDE project(s).
  • Users and group account owners should verify that they can authenticate at the XSEDE User Portal.
  • Users and group account owners should verify their TACC username and password. See the TACC password reset page.
  • Once your XSEDE and TACC credentials are in order, confirm that you can authenticate to the OpenStack Horizon portal for each cloud, in the TACC domain:
  • You will need to have the OpenStack client tools. A convenient place to host these tools is to Start up a VM instance via Atmosphere and install the OpenStack clients. For more information see Install the OpenStack command-line clients.
OpenStack clients for various Operating Systems

Using an Atmosphere instance or other local Linux installation (such as CentOS-7), install these packages:

  • Set the environment variables in openrc.sh. For instructions, see Setting up openrc.sh.
  • After the clients have been installed and the environment variables set, test your configuration with the OpenStack client command: openstack image list which should now return useful information.
  • If you have never worked with OpenStack and need help in creating your first running instance, email help@xsede.org.

Setting up openrc.sh

Jetstream has two clouds: the IU cloud and the TACC cloud; each cloud has one domain, the 'TACC' domain. API users will only be interacting with one cloud at a time. Transferring entities such as images and volumes from one domain to the other requires intervention by systems personnel. To request these services email help@xsede.org.

When using the 'TACC' domain, the username and password in the openrc.sh file are the same login credentials that would be used to access any TACC resource. For assistance with TACC login credentials, visit https://portal.tacc.utexas.edu/password-reset/-/password/request-reset.

IU cloud, TACC domain

TACC domain

export OS_PROJECT_DOMAIN_NAME=tacc
export OS_USER_DOMAIN_NAME=tacc
export OS_PROJECT_NAME=TG_your_xsede_project_name
export OS_USERNAME=your_tacc_username
export OS_PASSWORD=your_tacc_password
export OS_AUTH_URL= Contact help@xsede.org for the available endpoint URL
export OS_IDENTITY_API_VERSION=3

TACC cloud, TACC domain

TACC domain

export OS_PROJECT_DOMAIN_NAME=tacc
export OS_USER_DOMAIN_NAME=tacc
export OS_PROJECT_NAME=TG_your_xsede_project_name
export OS_USERNAME=your_tacc_username
export OS_PASSWORD=your_tacc_password
export OS_AUTH_URL= Contact help@xsede.org for the available endpoint URL
export OS_IDENTITY_API_VERSION=3

Use the Horizon dashboard to generate openrc.sh

  1. Log in to Horizon:
    1. IU: https://jblb.jetstream-cloud.org/dashboard
    2. TACC: https://tacc.jetstream-cloud.org/dashboard
    3. Domain: TACC
    4. User Name: your TACC username
    5. Password: your TACC password
  2. If you are on more than one allocation, select the project (XSEDE allocation) by clicking where it says "tacc . TG-xxxxxxxx . RegionOne" and choose the appropriate allocation.
  3. Click on your username in the upper right hand corner
  4. Click on Download OpenStack RC File v3

The password is not stored in the script. This openrc.sh file will prompt for a password when sourced.

Note: To select an active project prior to generating openrc.sh, click on Identity (left hand side) to see a list of your projects.

openrc.sh horizon dashboard screenshot
Figure API-1: Generate openrc.sh via Horizon Dashboard

Installing the Openstack clients on OS X

This will help you get the Openstack clients working on Mac OS X 10.11.x. It may work on recent older versions of Mac OS X but it has not been tested.

Follow the instructions below at your own risk.

source .bash_profile

Task Command

If necessary, install Xcode from the App Store.

 

Add these lines to .bash_profile

# Set architecture flags

export ARCHFLAGS="-arch x86_64"

# Ensure user-installed binaries take precedence

export PATH=/usr/local/bin:$PATH

Run this command

Verify that the Xcode required tools are installed and functional (This command may require root access to run.)

xcode-select --install

Set the permissions that Brew expects

sudo chflags norestricted /usr/local && sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local

Install Brew if not already installed

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

If Brew is installed, update as needed.

brew update

If prompted to agree to the Xcode license run:

xcodebuild -license

Check the Brew install with this command. If errors found doctor will provide instructions to fix them.

brew doctor

Verify that Brew is up to date

brew update

Install python

brew install python

Fix the links

brew linkapps python

  +Though not strictly necessary, we recommend using virtualenv to increase the stability of the openstack cli tools.

Create a directory for the project (cd to your preferred directory first)+

mkdir <project_name>

Change to the project directory

cd <project_name>

Install the VirtualEnvironment packages

sudo easy_install virtualenv

Start the VirtualEnvironment software

virtualenv <project_name>

Activate the VirtualEnvironment for the project

source <project_name>/bin/activate

   

Install the OpenStack clients

pip install python-keystoneclient python-novaclient python-heatclient python-swiftclient python-neutronclient python-cinderclient python-glanceclient python-openstackclient

Set up your OpenStack credentials See Setting up openrc.sh for details.

source .openrc

Test an OpenStack command

openstack image list

If you get python errors try these commands:

pip uninstall six

pip install six

Following future OpenStack updates, all installed pip modules can be updated with this command:

pip freeze --local | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip install -U

Installing the Openstack clients on Linux

This will help you get the Openstack clients working on Linux using Python.

Follow the instructions below at your own risk.

Task Command
Optional Virtualenv creation Though not strictly necessary, we recommend using virtualenv to increase the stability of the openstack cli tools.

Create a directory for the project (cd to your preferred directory first)

mkdir <project_name>

Change to the project directory

cd <project_name>

Install the VirtualEnvironment packages

sudo easy_install virtualenv

Start the VirtualEnvironment software

virtualenv <project_name>

Activate the VirtualEnvironment for the project

source <project_name>/bin/activate

   

Make sure you have Python 2 installed

This should already be installed by your operating system. Openstack CLI clients MUST be installed with Python2's pip!

Check your default Python version with: python --version

Install the OpenStack clients

pip install python-openstackclient

Additional clients that may also be useful depending on your custom needs are:

python-swiftclient, python-heatclient, python-senlinclient

For current users, clients that you likely no longer need to install are:

python-keystoneclient, python-novaclient, python-neutronclient, python-cinderclient, python-glanceclient

Set up your OpenStack credentials
See Setting up openrc.sh for details.

source .openrc

Test an Open Stack command

openstack image list

Following future OpenStack updates, all installed pip modules can be updated with this command:

pip freeze --local | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip install -U

OpenStack command line

There are many options and tools for using the OpenStack API from the command line. Follow the instructions in the table below to set up a security policy and network, launch and manage a VM and then remove the entire structure. All of the commands EXCEPT creating and removing security groups may be done from the Horizon OpenStack dashboard.

For more information, see the OpenStack command-line interface cheat sheet. Help is also available directly from the command line tools as shown in this example.

Notes:

  • You must be running the latest version of the clients. This tutorial was developed using openstack --version = openstack 3.8.1
  • It is possible to create entities with the same name; e.g. you could create two networks with the same name; however, they will have different UUIDs. When this occurs you may get a cryptic error message about that entity may not exist or some other baffling error. In this case, you must address the entity by its UUID.
  • In the examples below we often use ${OS_USERNAME}-api-whatnot to name an entity. We do this so that a first time user could more or less cut and paste the example and create a working instance that is unique to you and to distinguish from other users in your project (tenant).
  • It is important to understand that everything is owned by the project(tenant). Other users in your project can see and manipulate entities that you have created. Be careful in your naming and pay attention to the things you are manipulating.
  • In older OpenStack documentation you may come across the term tenant. The term is interchangeable with the newer term projects.
Get help from a command line tool (an example): openstack security group create

usage: openstack security group create <name>
Creates a security group
Positional argument:
<name> Name of security group

Create a security group - do this once at IU and/or TACC before launching instances. Comand line

Create a security group that will enable inbound ping and SSH. For more info see,

https://wiki.openstack.org/wiki/Neutron/SecurityGroups

Important note: On OpenStack, the default is that NO ports are open versus the traditional all ports are open unless specifically closed. For this reason, a security group must be established and the SSH port opened in order to allow login.

openstack security group create --description "ssh & icmp enabled" ${OS_USERNAME}-global-ssh

openstack security group rule create --protocol tcp --dst-port 22:22 --remote-ip 0.0.0.0/0 ${OS_USERNAME}-global-ssh

openstack security group rule create --protocol icmp ${OS_USERNAME}-global-ssh

Upload SSH key - do this once

 

If you have an SSH key upload id_rsa & id_rsa.pub to nova (note: Key filenames may vary)

cd ~/.ssh

openstack keypair create --public-key id_rsa.pub ${OS_USERNAME}-api-key

If you don't have an SSH key then create a new key and upload to nova.

ssh-keygen -b 2048 -t rsa -f ${OS_USERNAME}-api-key -P ""

openstack keypair create --public-key id_rsa.pub ${OS_USERNAME}-api-key

Setup the network - do this once OpenStack command

Create a private network

openstack network create ${OS_USERNAME}-api-net

Verify that the private network was created

openstack network list

Create a subnet within the private network space

openstack subnet create --network ${OS_USERNAME}-api-net --subnet-range 10.0.0.0/24 ${OS_USERNAME}-api-subnet1

Verify that subnet was created

openstack subnet list

Create a router

openstack router create ${OS_USERNAME}-api-router

Connect the newly created subnet to the router (use names instead of UUIDs)

openstack router add subnet ${OS_USERNAME}-api-router ${OS_USERNAME}-api-subnet1

Connect the router to the gateway named "public"

openstack router set --external-gateway public ${OS_USERNAME}-api-router

Verify that the router has been connected to the gateway

openstack router show ${OS_USERNAME}-api-router

Create and launch a VM

OpenStack commands

See what sizes (flavors) are available

openstack flavor list

See what Images are available for API use

The Jetstream team makes JS-API-Featured images available from which to build. Using images from Atmosphere WILL LIKELY NOT WORK CORRECTLY.

openstack image list --limit 1000 | grep JS-API-Featured

Create and boot an instance
Notes:

  • Make sure your SSH keyname matches.
  • ${OS_USERNAME}-api-u-1 is the name of the instance; make it something meaningful for you.

openstack server create ${OS_USERNAME}-api-U-1 --flavor m1.tiny --image IMAGE-NAME --key-name ${OS_USERNAME}-api-key --security-group ${OS_USERNAME}-global-ssh --nic net-id=${OS_USERNAME}-api-net

Create a public IP address for an instance

openstack floating ip create public

Add that public IP address to that instance

openstack server add floating ip ${OS_USERNAME}-api-U-1 your.ip.number.here

Things to do once the instance is running

You're ready to log in to the running instance via ssh to public IP

ssh -i ${)S_USERNAME_-api-key centos@your.ip.number.here
or
ssh -i ${)S_USERNAME_-api-key ubuntu@your.ip.number.here

Reboot, suspend, stop or shelve an instance

openstack server reboot ${OS_USERNAME}-api-U-1

openstack server suspend ${OS_USERNAME}-api-U-1

openstack server stop ${OS_USERNAME}-api-U-1

openstack server shelve ${OS_USERNAME}-api-U-1

Commands to clean up what was created above

OpenStack commands

Remove the public IP address from the instance

openstack server remove floating ip ${OS_USERNAME}-api-U-1 your.ip.number.here

Delete an instance

openstack server delete ${OS_USERNAME}-api-U-1

Return the public IP address to the pool of IP numbers

openstack floating ip delete your.ip.number.here

Clean up commands for other entities

Note that you often want to create infrastructure such as networks, subnets, routers, etc. only once and not delete them. These entities are reusable by all members of your project.

Disconnect the router from the gateway

openstack router unset --external-gateway ${OS_USERNAME}-api-router

Delete the subnet from the router

openstack router remove subnet ${OS_USERNAME}-api-router ${OS_USERNAME}-api-subnet1

Delete the router

openstack router delete ${OS_USERNAME}-api-router

Troubleshooting your API instances from the Openstack Command Line

In addition to the common OpenStack CLI commands like openstack server list, here are some typical commands to troubleshoot an instance from the command line.

Command What it does

openstack server show UID

shows you the state of the instance and any OpenStack errors

openstack console log show UID

shows you the running console log of the instance (same as doing "dmesg" from the command line of the instance itself)

openstack console url show UID

yields a URL you can paste into your browser to get a login console for your instance

Usually a combination of these things will let you see what's going on and what to do to fix. You can also do all of these things from Horizon.

Creating and using volumes from the command line

Assuming you have an allocation for Jetstream storage, you can create and use volumes from the command line.

To view any volumes you might have:

openstack volume list

To create a 10 GB volume, you can do:

openstack volume create --size 10 ${OS_USERNAME}-10GVolume

Then you can attach it to an instance for use:

openstack server add volume vm-uid-number volume-uid-number

From your instance, you can now create a mount point, view the device, and create the filesystem (using ext4 as the filesystem type for this example).

# mkdir /vol_b
# fdisk -l /dev/sdb
# mkfs.ext4 /dev/sdb

If you get the following warning, it's safe to hit 'y' to proceed:

/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n)
# mount /dev/sdb /vol_b

Assuming you didn't get any errors, /dev/sdb should now be mounted on /vol_b

Note: Linux has a special mount option for file systems called noatime. If this option is set for a file system in /etc/fstab, then reading accesses will no longer cause the atime information (last access time - don't mix this up with the last modified time - if a file is changed, the modification date will still be set) that is associated with a file to be updated (in reverse this means that if noatime is not set, each read access will also result in a write operation). Therefore, using noatime can lead to significant performance gains.

# touch /vol_b/foo
# ls -la /vol_b/
    total 24
    drwxr-xr-x   3 root root  4096 Jul 13 13:37 .
    dr-xr-xr-x. 18 root root  4096 Jul 13 11:50 ..
    -rw-r--r--   1 root root     0 Jul 13 13:37 foo
    drwx------   2 root root 16384 Jul 13 13:36 lost+found
    
# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sda1       8.0G  2.4G  5.7G  30% /
    devtmpfs        902M     0  902M   0% /dev
    tmpfs           920M     0  920M   0% /dev/shm
    tmpfs           920M   17M  904M   2% /run
    tmpfs           920M     0  920M   0% /sys/fs/cgroup
    tmpfs           184M     0  184M   0% /run/user/0
    /dev/sdb        9.8G   37M  9.2G   1% /vol_b
    

To make the volume mount persist, you can add an entry to /etc/fstab similar to this:

/dev/sdb /vol_b ext4 defaults,noatime 0 0

You would need to change as needed for a different device id, mount point, and file system type. We do recommend using the noatime option as shown in the example.

Once you are done with your volume or want to use it with another VM, if you are not shutting down the VM, you'll need to unmount it.

umount /vol_b

To detach it from the VM, you'll do:

openstack server remove volume vm-uid-number volume-uid-number

Doing openstack volume list now should show the volume as available:

openstack volume list
ID Display Name Status Size Attached to
af59d4fa-ced2-4049-a062-7b2a15807b7f jlf599-10GVolume available 10  

If you want to completely destroy a volume, you can do:

openstack volume delete volume-uid-number

Using the OpenStack Horizon GUI Interface

Part 1: Create Private Network

1. Login to Horizon


Figure H-1: Horizon login dialog

2. Click on Project > Network > Network Topology > +Create Network

create network screenshot
Figure H-2: Create Network menu option

3. Enter a network name, for example, username_net

After giving it a descriptive name, press the blue NEXT button.

create network screenshot
Figure H-3: Use a Descriptive Network Name

4. Enter a subnet name, e.g. username_subnet, a network address, e.g. 10.10.10.0/24 and a gateway address, e.g. 10.1.1.1

This should be a non-routable subnet. You can use 10.0.0.0 - 10.255.255.255, 172.16.0.0 - 172.31.255.255, 192.168.0.0 - 192.168.255.255.

If you're not sure what to choose, you can go with 10.1.1.0/24. This will give you 255 available addresses in the 10.1.1.0 domain.

If you choose 10.1.1.0/24 you can then set the gateway address to be 10.1.1.1.

Then press the blue NEXT button.

create network screenshot 2
Figure H-4: Choose Network Addresses

5. Click Create to create the new network

create network screenshot 3
Figure H-5: Create the Network

6. Click on +Create Router.

network topo 2 screenshot
Figure H-6: Create a Router

7. Enter a router name, e.g. username_router, under the the External Network dropdown, select public, then click Create Router.

create router screenshot
Figure H-7: Create the Router

8. You'll need to connect your private network to the router. Make sure you're in Topology mode (vs Graph mode). Your screen should look like the top image on the right. If it looks like bars it's in Graph mode, click the Topology tab under the Network Topology page heading to put it into Topology mode.

  • Click on the router you just created
  • Click on +Add Interface
connect private screenshot
Figure H-8: Add Interface for Private Network

9. Select the subnet you just created from the dropdown list. Click Submit.

add interface screenshot
Figure H-9: Connect Subnet to the Router

10. The network should now be connected to the new router. The end result will look similar to this diagram.

network diagram screenshot
Figure H-10: Network Diagram

Part 2: Create Security Group

Each project has its own firewall rules. You will need to add at least an ssh rule to allow yourself to ssh into your instances.

  • On the left side bar, click on Network > Security Groups
  • Click +Create Group on the right side of the screen and give your new group a descriptive name and optional description.
  • Click the blue Create Security Group button
create security group screenshot
Figure H-11: Create a Security Group

Part 3: Create Security Rules

  • Click the checkbox by the security group you just created and then click Manage Rules
  • Click Add Rule
  • Rule: Choose SSH from the dropdown - you can keep the defaults
  • Click Add

You should see your new SSH rule in the list.

You'll also want to add a rule to allow ICMP.

  • Click Add Rule
  • Rule: Choose All ICMP from the dropdown - you can keep the defaults
  • Click Add
create security rules screenshot
Figure H-12: Create Security Rules

Part 4: Upload SSH Key

You will need to upload at least 1 ssh public key in order to access the instances you create. This assumes you already have an SSH key pair that you use. If not, you can create a key pair via this screen.

%p Note: Jetstream recommends and only supports using RSA-based SSH keys or ed25519-based keys.

  • On the left side bar, click on Compute > Key Pairs
  • Click Import Key Pair
    • Key Pair Name: username_key
    • Public Key: Paste the contents of your public key file here – generally in your ~/.ssh/identity_rsa.pub file
    • Click Import Key Pair
upload ssh key screenshot
Figure H-13: Upload SSH Key

Launching an instance

With the network, security group and ssh key set up, you should be ready to launch an instance.

Launching an instance - Step 1

  • On the left side bar, click on Project -> Compute -> Instances
  • Click Launch Instance
  • Give your instance a descriptive name.
  • You can take the defaults for the rest of the items.
  • Click Next.
launch instance step 1 screenshot
Figure H-14: Launch Instance - Step 1

Launching an instance - Step 2

  • On the Source screen, type JS-API-Featured in the "Available" box to find the Jetstream API Featured images. You can also boot one of the images shown in the list if you'd rather do that, but we only guarantee the featured images will work for anyone.
  • Find the image you want to use and click the arrow to the right of its name. You'll see it show up under "Allocated" towards the top of the dialog box.
  • Click Next.
launch instance step 2 screenshot
Figure H-15: Launch Instance - Step 2

Launching an instance - Step 3

  • On the Flavor screen, select the size VM you want to use and click the arrow to the right of its description. Like the previous screen, you'll see it move up into the "Allocated" area towards the top of the box.
  • Click Next.
launch instance step 3 screenshot
Figure H-16: Launch Instance - Step 3

Launching an instance - Step 4

  • On the Networks screen, find the network you created earlier and click the arrow to the right of its description. Like the previous screen, you'll see it move up into the "Allocated" area towards the top of the box.
  • Click Next.
  • Click Next again to skip the Network Ports screen and proceed to Security Groups
launch instance step 4 screenshot
Figure H-17: Launch Instance - Step 4

Launching an instance - Step 5

  • On the Security Groups screen, select the security group you created earlier and click the arrow to the right of its description. Like the previous screen, you'll see it move up into the "Allocated" area towards the top of the box.
  • Click Next.
launch instance step 5 screenshot
Figure H-18: Launch Instance - Step 5

Launching an instance - Step 6

  • On the Key Pair screen, if the key pair you created earlier is NOT in the "Allocated" section, click the arrow to the right of its description. Like the previous screen, you'll see it move up into the "Allocated" area towards the top of the box.
  • At this point you can click "Launch Instance"
launch instance step 6 screenshot
Figure H-19: Launch Instance - Step 6

Add a Public IP

Add a public IP (floating IP) - Step 1

  • Your instance should now show up in your instances list.
  • Once it has entered the running state, click the dropdown at the far right and select "Associate Floating IP".
add public ip - step 1 screenshot
Figure H-20: Add Public IP - Step 1

Add a public IP (floating IP) - Step 2

  • If you have an IP allocated, you'll see it in the IP Address list. If you do not, click the + sign to allocate a new IP address.
add public ip - step 2 screenshot
Figure H-21: Add Public IP - Step 2

Add a public IP (floating IP) - Step 3

  • Make sure "Public" is selected under Pool.
  • Click Allocate IP.
add public ip - step 3 screenshot
Figure H-22: Add Public IP - Step 3

Add a public IP (floating IP) - Step 4

  • Note your new IP address and make sure it's the one you want to associate with the instance.
  • Note that the correct instance is selected under "Port to be associated".
  • If all is well, click Associate.
add public ip - step 4 screenshot
Figure H-23: Add Public IP - Step 4

Test the connection

  • Please note that sometimes it takes a few seconds or even a little more for the association to become active.
  • Try pinging your host: ping -c 5 ip_address
test connection screenshot
Figure H-24: Test Connection to Public IP

Try ssh to your instance

  • If you chose a CentOS-based instance, your default user is centos.
  • If you chose an Ubuntu-based instance, your default user is ubuntu.
  • Access your host with SSH. The example from a terminal-based ssh-client as on Linux or a Mac would be ssh centos@your_ip_number or ssh ubuntu@your_ip_number.
  • Note that the first time you connect, you'll get a request to verify that you're getting a new host key and making sure you want to connect.
try ssh screenshot
Figure H-25: Connection to Instance with SSH

Creating a snapshot of an instance in Horizon

When you have created a custom workflow or configuration on the API side, you can create a snapshot for your own use. In OpenStack, an instance snapshot is an image. The only difference between an image that has been uploaded directly to glance and an image you create by snapshot is that an image created by snapshot has additional properties in the glance database and defaults to being private.

You can create an snapshot from a running server instance, but if you want to preserve data, you must shut down the source VM and verify the instance status is SHUTOFF before creating the snapshot.

Before creating the snapshot and/or image, you'll want to make sure that cloud-init is installed on your instance as well as qemu-guest-agent. If your instance was based on one of the JS-Featured-API images, both cloud-init and qemu-guest-agent should be present unless you explicitly removed them.

  1. Log in to the Horizon dashboard and go to Project > Compute > Instances. Find the instance you want to snapshot and note the Create Snapshot button.

    create snapshot Figure S-1: Create a Snapshot
  2. Click Create Snapshot. In the dialog box that comes up, give it a descriptive name. A best practice is to give is a name that describes the image and has rudimentary version information - e.g. primary-gateway-image-v1. Click Create Snapshot once you've given it a name.

    name snapshot Figure S-2: Name the Snapshot
  3. Once you do this, it will take you to the Project > Compute > Images screen. You can find your pending snapshot by using the filter.

    You'll see it in state "Queued". This is normal. Please note that if you refresh this screen or filter/unfilter that it might take 10-15 seconds to refresh the list. It has to pull the entire image list again each time.

    queue snapshot Figure S-3: Queue the Snapshot
  4. After a few minutes (or possibly more), depending on system load, you should see it change status to Active. At this point you can click "Launch" and launch a new instance from it using the step by step launch instructions on Using the OpenStack Horizon GUI Interface.

    If you've launched instances previously on the API side with either Horizon or the CLI and left your network, security groups, and keys intact, you can start with the step Launching an instance - Step 1

    snapshot ready Figure S-4: Snapshot Ready

If your snapshot stays in Queued status for more than 45 minutes, there might be an issue with either the snapshot or the Glance image store in OpenStack. In this case, you should contact help@jetstream-cloud.org with your allocation number, the UID of snapshot (which you can get by clicking on it) and the UID of the instance you are trying to snapshot.

Creating snapshots and new Glance images from the command line

When you have created a custom workflow or configuration on the API side, you can create a snapshot for your own use. In OpenStack, an instance snapshot is an image. The only difference between an image that has been uploaded directly to glance and an image you create by snapshot is that an image created by snapshot has additional properties in the glance database and defaults to being private.

Glance is a central image repository which provides discovering, registering, retrieving for disk and server images.

You can create a snapshot from a running server instance, but if you want to preserve data, you must shut down the source VM and verify the instance status is SHUTOFF before creating the snapshot.

Before creating the snapshot and/or image, you'll want to make sure that cloud-init is installed on your instance as well as qemu-guest-agent. If your instance was based on one of the JS-Featured-API images, both should be present unless you explicitly removed them.

To create the snapshot from the command line:

openstack server image create --name snapshot-image-name instance-name

(e.g. openstack server image create --name MyCustomCentos7Image-Feb-7-2017 my-CentOS7-instance)

Snapshots won't show in Horizon (this has been submitted as a bug and may be fixed real soon now) but for now, to make it visible you'll need to export it and bring it back as a Glance image. If you just plan to use this within your project and from the command line only, the rest of the steps aren't necessary.

openstack image save --file whatever_file_name_you_like.raw UID

(e.g. openstack image save --file c7custom-image.raw 569677d8-c7b0-4606-86d8-7673a5ecd5cf)

Uploading a snapshot or new image into Glance:

openstack image-create --public --disk-format raw
--container-format bare --property skip_atmosphere=yes
--property hw_disk_bus=scsi --property hw_scsi_model=virtio-scsi
--property hw_qemu_guest_agent=yes --property os_require_quiesce=yes
--file c7custom-image.raw My-Custom-Image-Name
Note

There are a lot of metadata tags there but those are important to ensure that your instances will create properly from the stored image. You definitely want to make sure you get them all.

Boot the new image. Test it. Make sure it works. Do this before deleting. Please. Once it's gone, it's really gone. Be sure.

Delete your snapshot if you no longer need it. For example:

openstack image-delete 569677d8-c7b0-4606-86d8-7673a5ecd5cf

Container Orchestration

Please note that container orchestration on Jetstream is in early development. We will do our best to help you with issues, but it may be a mutual learning effort.

This page is a work in progress!

Basic Docker container support is available in the featured CentOS 7 and Ubuntu 16 images on both Atmosphere and API. Docker and Docker Compose are installed on those two featured images.

OpenStack provides a method of container orchestration through Magnum. Magnum is an OpenStack API service developed by the OpenStack Containers Team making container orchestration engines such as Docker Swarm, Kubernetes, and Apache Mesos available as first class resources in OpenStack. Magnum uses Heat to orchestrate an OS image which contains Docker and Kubernetes and runs that image in either virtual machines or bare metal in a cluster configuration. You can watch a brief demo of how the Magnum CLI works.

Another option for using Kubernetes (especially in relation to JupyterHub) is Kubeadm Bootstrapper.

This is not an official Jetstream project, though we are working directly with the developers and testers that have been creating it. This is part of the Zero to JupyterHub project from UC Berkeley.

Creating a cluster template from the CLI

openstack coe cluster template create --coe kubernetes \ --image Fedora-Atomic-27-20180419 --keypair my_rsa_key --external-network public \ --fixed-network my_network_name_or_UID --network-driver flannel --flavor vm_size \ --master-flavor master_vm_size --docker-volume-size size_in_gb \ --docker-storage-driver devicemapper --floating-ip-enabled My_Template_Name

Launching a cluster using the template from the CLI

openstack coe cluster create --cluster-template My_Template_Name \ --master-count x --node-count y --keypair my_rsa_key \ --master-flavor master_vm_size --flavor vm_size \ --docker-volume-size size_in_gb My_Cluster_Name

An example of each

openstack coe cluster template create --coe kubernetes \ --image Fedora-Atomic-27-20180419 --keypair myRSA --external-network public \ --fixed-network test-network --network-driver flannel \ --flavor m1.medium --master-flavor m1.small --docker-volume-size 100 \ --docker-storage-driver devicemapper --floating-ip-enabled My-K8s-Test-2

openstack coe cluster create --cluster-template My-K8s-Test-2 \ --master-count 1 --node-count 3 --keypair myRSA --master-flavor m1.small \ --flavor m1.medium --docker-volume-size 100 my-k8s-cluster1

Cloud-init scripts

Using cloud-init scripts with the Jetstream API

Cloud-init is the package that handles early initialization of a cloud instance. Cloud-init allows you to customize an instance at boot using a user-data script. This allows you to override vendor-data scripts like the forced update on initial boot, add software, add git repositories, add users, and many other actions.

To use a cloud-init script on the command line, you add

--user-data ~/scriptname

to the openstack server create command.

When doing a cloud-init script, it's crucial to start it with #cloud-config so the interpreter knows that it's not just a standard shell script.

A number of examples of cloud-init scripts are available in the Cloud-init documentation.

Note

The script below should only be used by advanced users. It prevents the installation of qemu-guest-agent (a package that is REQUIRED for proper operation on Jetstream – please ensure it is installed on your image if you opt to use this script or any script that prevents it being installed by vendor-data). It also prevents the forced update of packages on instantiation. It is a recommended security practice to keep instances up-to-date with the latest operating system updates. However, if you need to ensure consistent, fast boot times, you may wish to use it.

Please do apply all critical security patches to your instances. Failure to do so may allow your instances to be compromised. If they are compromised, they will be locked and/or deleted by Jetstream staff.

#cloud-config
packages: []
package_update: false
package_upgrade: false
package_reboot_if_required: false
final_message: "Boot completed in $UPTIME seconds"

Non-Openstack native API topics

The Jetstream team is working with ECSS and individual users that are trailblazing use cases of Jetstream's cloud functionality. While we maintain this page, we do not guarantee that the various tutorials or walkthroughs are up to date and functional.

PEARC18 Tutorial on using Slurm for virtual clusters on Jetstream

The first half deals with the OpenStack API and walks you through creating a headnode that you'll use in the second half for making a virtual cluster with Slurm: Tutorial Part 1.

The second half is the more complete Slurm portion and covers using cloud-aware compute nodes: Tutorial Part 2.

Filesystems-as-a-service

BETA!!!

Manila is the file share service project for OpenStack. Manila provides the management of file shares for example, NFS and CIFS, as a core service to OpenStack. Manila works with a variety of proprietary backend storage arrays and appliances, with open source distributed filesystems, as well as with a base Linux NFS or Samba server.

Prereqs: Make sure you have the nfs client installed on your instance: apt install nfs-common for Ubuntu and yum install nfs-utils for CentOS

Presently, Manila is in BETA mode on Jetstream and is only functional on the IU cloud.

To use Manila via Horizon, you can do the following:

  1. Log into Horizon
  2. Click on: Compute -> Shares
  3. Create a share
    1. Key settings — Protocol:nfs, Share Type: cephnfstype

In Horizon, add interface to the instance from the instance screen and attach to manila_testing network – note the IP.

On a running instance add an interface on the manila_testing network.

Note the interface that gets added: ip addr show and note the interface name.

For CentOS:

cat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOF

DEVICE="eth1"
BOOTPROTO="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
PEERDNS="no"
IPV6INIT="no"
PERSISTENT_DHCLIENT="1"

EOF

For Ubuntu:

cat >> /etc/netplan/50-cloud-init.yaml

network:
  version: 2
  ethernets:
    ens7:
      dhcp4: true

EOF
  1. Then bring the interface up: ifup eth1 or ip link set ens7 up
  2. You may have to tell it to get an IP address: dhclient interface (e.g dhclient ens7)
  3. Then do ip addr show again and get your addr on the new interface that's connected to the manila-testing network
  4. Go to Compute -> Shares and select Manage Rules – create a read/write rule for your IP
  5. Make sure you have your mount point created. Our example is /OSShare
  6. Go back to the share details and copy the path you see on the screen
  7. Mount it on your instance:
e.g. mount 10.255.0.1:/volumes/_nogroup/9453a642-6333-4a65-b8ac-37c403b5f0c7 /OSShare/

Wrangler

Using Jetstream to access Wrangler data collections with iRODS

Jetstream allows you to quickly spin up new science resources in a managed cloud. You can even leverage other XSEDE resources, such as Wrangler, to retrieve or store additional data sets.

Wrangler iRODS accounts Wrangler iRODS accounts must be requested. They are not automatically generated upon account creation. Please see the Wrangler user guide for more information.

Step-by-step guide

  1. To get started, choose one of the following featured images as source for a new Jetstream node:
    1. Ubuntu 14.04.3 Development
    2. Ubuntu 14.04.3 Development GUI
    3. CentOS 6 (6.8) Development
    4. CentOS 6 (6.8) Development GUI
    5. Centos 7 (7.2) Development
    6. Centos 7 (7.2) Development GUI
    7. Intel Development (CentOS 7)
  2. Click on the Launch button to create a new instance based on this image.
  3. After the node finishes deployment, a compatible client package for iRODS will already be installed.
  4. Run isetup.
  5. At the prompt, enter the username used by Wrangler for the collection. There is no need to match usernames within the Jetstream node and iRODS usernames. Multiple Jetstream usernames can access the same iRODS collection.
  6. Next, you will be prompted to choose either the IU or TACC iRODS instance. Although both iRODS zones are linked, they contain separate data sets. It is best to communicate directly with the instance where the data collection is stored.

The isetup script will create the initial iRODS environment file, "irods_environment.json" in the .irods folder in your home directory and ask to overwrite it, if it already exists.

atmosphere login screenshot

Figure 1: Running isetup on Jetstream Atmosphere

Once the proper environment is set, initialize a session using iinit using the Wrangler account credentials, which will use the Wrangler password for the account you specified. This creates a temporary credential which will be valid for two weeks. Once connected, the file system can be navigated and the standard iRODS commands are available.

The ihelp command lists the common iRODS commands, such as ils, ipwd, iget, iput, and imeta, which are used to transfer files to iRODS or perform tagging, queries, or other operations. Detailed instructions on using these commands can be found at the iRODS site.

Any files submitted into iRODS are given a checksum, and access is logged for reads and writes. These files will then also be available via the Wrangler data analysis system on XSEDE. It is possible to prototype program flow using Jetstream, and then move to Wrangler to perform larger calculations on the same data collection.

Policies and Best Practices

Good citizenship

  • Each instance burns SUs for the time it is in operation.
    • It is beneficial to you and other Jetstream users to shelve instances when they are not in active use.
    • This frees up resources for other users and also preserves more of your SUs for future use.

Resource recovery policies

  • Instances that have been suspended or stopped can be shelved after two (2) weeks in said state.
  • Shelved images/instances will be deleted after six (6) months of inactivity.
  • All entities owned by a project (e.g. running instances, images, objects, volumes, networks, IPs allocated etc.) will be deleted six (6) months after the allocation ends.
  • NOTE: If you are using a community-contributed image (i.e. non-Featured image) that belongs to another Atmosphere user, if that user no longer has an allocation or is removed for other reasons, their image will go away as well! If your work is dependent on someone else's image, you might consider making your own version using the instructions Customizing and saving a VM.

Security

  • As stated in the XSEDE Usage Policy, users of the Jetstream system are expected to abide by policies established by the service providers.

  • Periodically apply operating system updates to your running VMs

    • The Ubuntu 18.04, 16.04 and Centos 7.4 featured images are piloting unattended security updates. Nodes will not reboot, but they will apply any update marked as a security update. It's still a good idea to update your VM, just in case.
    • CentOS: sudo yum update
    • Ubuntu: sudo apt-get update then sudo apt-get upgrade
    • If the kernel or glibc/libc packages are being updated, rebooting is necessary to implement those updates.
    • Always run updates before requesting a new custom image. – An actively updating instance may be slow to deploy and may require a redeploy or even a reboot after updating in order to fully successfully deploy.
  • Remember to log out from the menu at the top right (where it shows your username).

    • This ensures that you do not inadvertently allow others to access your Jetstream account.
  • Update and configure software that utilizes the network to preclude unauthorized access.
    • Commonly neglected examples include MongoDB.

Image management policies

  • As operating systems near the end of their supported life cycles, they will be removed from Featured Images. In addition, custom images based on those operating systems will no longer be supported on Atmosphere. You may be notified that you must migrate your workflows to a newer version of the operating system you're using. Security policies noted above require that any operating system that is no longer receiving timely security patching may not remain in service on IU or TACC networks.

Software installation/development

  • SSH/SSHD

    • The Secure Shell client (SSH) and daemon (SSHD) come preinstalled on all instances.
    • Configurations settings appropriate and/or required for normal Jetstream operation have been put in place.
    • Under normal circumstances, SSH/SSHD should not need to be upgraded or reinstalled.
    • If you do reinstall or modify SSH/SSHD or the related configuration settings, you may disrupt essential Jetstream features and MAY LOCK YOURSELF AND ADMINISTRATORS OUT OF THE INSTANCE.
      • Fixing this may be difficult or impossible, thus forcing you to delete your instance.
  • Python

    • Python2 comes preinstalled on all instances.
    • Configurations settings appropriate and/or required for normal Jetstream operation have been put in place.
    • If you wish to reconfigure, reinstall, or upgrade Python, we recommend implementing a user-level virtual-environment technology, such as VIRTUALENV, ANACONDA, or MINICONDA to avoid conflicts with the system-level installation.
    • The default system-level Python must remain at Python2, though you may add a non-default Python3.
    • If you do reinstall or modify Python or the related configuration settings, you may disrupt essential Jetstream features and MAY PREVENT SUCCESSFUL BOOTING AND CLOUD DEPLOYMENT.
      • Fixing this may be difficult or impossible, thus forcing you to delete your instance.
    • Be aware that some PIP upgrades, even within Python2 or within Python3, particularly from pip9 to a higher version, require several additional steps to ensure continued instance operation or backward compatibility with installed software.
Title Creator Modified

Launching Jupyter Notebook within an instance

Brian Beck

Apr 25, 2018

Logging in with Old Web Shell (Deprecated)

Peg Lindenlaub

Jan 26, 2018

Adding SSH keys to the Jetstream Atmosphere environment

Peg Lindenlaub

Aug 16, 2017

Quick Start Guide

Peg Lindenlaub

Aug 14, 2017

Logging in with SSH

Peg Lindenlaub

Jun 27, 2017

Logging in with VNC desktop

Peg Lindenlaub

May 05, 2017

System Access

Peg Lindenlaub

Jan 04, 2017

Launching your VM

Peg Lindenlaub

Jan 04, 2017

Using Jetstream to access Wrangler data collections with iRODS

Peg Lindenlaub

Nov 04, 2016

How to add users to a running instance

Peg Lindenlaub

May 25, 2016

Increasing fail2ban retries

fail2ban is installed on every launched instance at boot. The default number of login failures is set to three. If you want to change that, you can:

cd /etc/fail2ban
sudo nano jail.local

and look for this section:

[ssh-iptables]
enabled = true
filter = sshd
action = iptables[name=SSH, port=22, protocol=tcp]
sendmail-whois[name=SSH, dest=root@localhost, sender=Fail2Ban, sendername="Fail2Ban"]
logpath = /var/log/secure
maxretry = 3

Change the maxretry value and save the file:

sudo systemctl restart fail2ban

to restart the service with the new config.

To remove any banned IPs, do the following:

sudo iptables -L -n

Look for your IP number in that output, if it's there, proceed with:

sudo fail2ban-client status

Should show you fail2ban is working and your jail name(s):

[js-157-95] root ~-->fail2ban-client status
Status
|- Number of jail: 1
`- Jail list: ssh-iptables

sudo fail2ban-client set ssh-iptables unbanip YOUR_IP_NUMBER

Tutorials

In addition to our "How-to" articles, a number of tutorials have been developed by both staff and external users. We list some of these below:

An Introduction to Linux Cornell Virtual Workshop (full access to the training materials requires logging in with XSEDE credentials)
Learn Linux section from Linux.com website
HowtoForge user-friendly Linux tutorials
Linux Knowledge Base tutorials, forums, and how-tos for Linux
Learning the Shell learn the Linux command line
Command-line bootcamp learn all the basic skills needed to start being productive in the UNIX terminal

Microsoft Windows on Jetstream

We presently do not support Windows on Jetstream. It can be installed via the API side but we are unable to provide any sort of support for it at all.

If you wish to explore installing Windows on Jetstream, first get a Startup Allocation if you do not have one.

Once you have the allocation you'll need to get API access. Follow these instructions to get API access.

Once you have API access, you'll want to install the command line clients for later. Instructions for that are available for Windows and for Linux and OSX.

Then you can follow these instructions or these instructions to get Windows onto Jetstream via the OpenStack API.

One key difference is you'll need to make sure the image is in RAW format to upload. There are instructions on uploading images; especially make sure you specify the right options under Uploading a snapshot or new image into Glance.

Depending on the application you are wanting to use, you might see if there is a Linux version available. It would definitely be easier to install a Linux-based software package rather than install and support your own Windows installation on Jetstream.

Problem Possible Solution

VM gets stuck trying to provision the network or in deploying

or

Previously behaving and unmodified VM develops problems.

Try these in this order:

  1. Refresh: Refresh your browser and confirm that the Atmosphere interface has synchronized with the latest Jetstream information.
  2. Wait: Please be aware that while provisioning is generally rapid, it can sometimes take several minutes during times of heavy activity. (wait 5-10 minutes)
  3. Redeploy/Reboot: Click Projects (on the top menu) and select the Project with the instance that is stuck.

    1. Then click on the instance name that's stuck.

    From the Actions menu on the right, try

    • Redeploy first (waiting about 5-10 minutes before going to the next step),

    • Reboot if that fails (waiting about 5-10 minutes before going to the next step), and

    • Hard Reboot if the first two actions fail (waiting about 5-10 minutes).

  4. If everything otherwise seems in order deploying but the instance seems to be delayed in getting an IP address or in showing the WebShell buttons, the instance may be updating system packages. You will have to wait until this finalizes, then redeploy. Jetstream staff regularly update FEATURED IMAGES to address this. For images not maintained by Jetstream you will need to contact the owner.

If none of these Actions work, DO NOT DELETE THE INSTANCE. Make note of the Alias (UID) and IP Address of the Instance, and email that information to help@jetstream-cloud.org or click the Feedback & Support button.

Do I need to apply security updates to my VM?

It's always a good practice to apply periodic updates. A good recommendation is to check for updates once a week at least. See the Jetstream Policies and Best Practices page for details.

Hostname mismatch

If you are having hostname mismatch issues with some software, try the following:

  1. Type su
  2. Type crontab -e
  3. Insert this line: @reboot /bin/hostname | awk -F'.' '{print $1}' > /etc/hostname
  4. Save and Exit

Volume fails to detach

  1. make sure the instance to which the volume is attached is actively running
  2. make sure no jobs or process running on the instance are using the volume The command fuser -vm /vol_b will show all processes using the volume namedvol_b`
  3. Kill those jobs (after verifying the don't need to be running): fuser -km /vol_b

Volume fails to attach

  1. make sure the instance to which the volume should be attached is actively running

My instance has been locked by an administrator

  1. Are you running an insecure web-service like Apache or MongoDB with no security features enabled?
    1. MongoDB: https://docs.mongodb.com/manual/administration/security-checklist
    2. Apache: https://httpd.apache.org/docs/2.4/misc/security_tips.html
  2. Have you recently requested help with the instance? If so the administrator is likely working on solving your problem.

Old Web-Shell (Gate-One) does not connect (via ssh)

First, try the new Guacamole-based Web-Shell "Open Web Shell". If your Old Web-Shell / Gate-One "Open Old Web Shell" stalls at a prompt like: Host/IP or ssh:// URL [localhost]:

  1. Close the session by clicking on the "X" in the top-right menu bar
  2. Close the browser window/tab
  3. Go back to your Jetstream Atmosphere window and re-click on the "Open Old Web Shell" link

Web Desktop does not launch

Web Desktop should work for any Jetstream Atmosphere Featured Image. If it does not, you might try doing a redeploy on the instance.

If it's not a featured image, you might wish to try using the instructions for an external VNC viewer.

API command line access to TACC fails with "requires authentication" error

Add the following two lines to the openrc.sh script if they are not there:

missing TACC openrc.sh env variables

export OS_USER_DOMAIN_NAME=tacc

export OS_IDENTITY_API_VERSION=3

btrfs in conjunction with OpenStack's ceph has known kernel bug

please use a different file system. e.g XFS, EXT4

Mouse does not work in web desktop

Laptops with touchscreens are unable to use the mouse in the web desktop using Chrome and Firefox on Windows 10.

There is a bug in the web browser code in both Chrome and Firefox for using a touchscreen PC with the web desktop of Jetstream. This is not a bug we can easily work around since it's in the browser code itself.

While we do not recommend using Internet Explorer, it does not have the issue. Using a VNC Viewer also is a viable workaround.

web-shell works for my Ubuntu 16.04 version or newer instance, but I cannot ssh to it directly from an external ssh client

  • Have you added keys to your Atmosphere settings?
  • Are you using DSA keys? DSA keys are deprecated for Ubuntu 16.04 and newer instances. Please use RSA keys.
  • Are you being asked for a password? Since password authentication is turned off by default, you may have a problem with your keys.

Users with a brace " { " or parenthesis "(" in their passwords may experience errors with OpenStack command line commands.

  • There is a known issue with braces in openstack passwords when using the OpenStack command line client. At this time, do not use a brace in your password.
  • A potential work around is to enclose your password in single quotes in your .openrc

You get an ssh "connection refused" message using a terminal-based shell client or Putty

Probable cause: fail2ban has locked you out from vnc or terminal-based shell sessions

You can use the unban utility that is part of the Jetstream Atmosphere EZ-tools. Use the web shell and do the following:

unban -l

will show you all currently banned IPs. Find your IP in the list and do:

unban -i your_ip_address

OLD Method:

  • sudo iptables -L -n
    • Look for your IP number in that output, if it's there, proceed
  • sudo fail2ban-client status

Should show you fail2ban is working and your jail name(s):

[js-157-95] root ~-->fail2ban-client status

Status

|- Number of jail: 1

`- Jail list: ssh-iptables

  • sudo fail2ban-client set ssh-iptables unbanip YOUR_IP_NUMBER

Jetstream can't be accessed from certain countries

Certain organizations and individuals are subject to trade sanctions, embargoes, and other restrictions under US law or by individual cloud provider policy. These restrictions apply to both domestic and foreign transactions.

See Table 4-1 for the list of Embargoed Countries and links to US government ITAR information. This list is subject to change based on US policy.

You get a "403 Forbidden" screen trying to access the web shell or web desktop.

Your Globus credentials have become stale. Log out of Atmosphere by clicking on your username in the upper right hand corner of the Atmosphere interface and select "Sign Out" from the menu. Then log in again and try using the web shell or desktop.

I have an Ubuntu18-based instance that has been stuck deploying for quite some time.

Click on your instance and check the version:

  • Based on Ubuntu 18.04 Devel and Docker v1.12

If the version is < 1.8, you will have a known bug that introduces a configuration conflict that prevents rebooting/redeployment.

Ubuntu versions 1.8+ have been patched to fix this networking bug.

If possible, please rebuild your instance with a more recent version, otherwise, please contact help@jetstream-cloud.org for assistance in applying the fix.

When I try to login after having selected XSEDE as my authorization provider, I get a page with the error message:

"Unknown user or wrong password"

If you can, then it's likely that your username and password combination are correct.

  • Are you treating your username and password as if they were case-sensitive?

Best results will occur if you treat your credentials as if they were case-sensitive when using Jetstream, Globus, or XSEDE.

My CentOS based instance shows active but isn't reachable via ping or SSH or my CentOS based instance is hung up at "Active Networking" in the booting phase.

Following an upgrade rebuilding virtual networks is now taking multiple hours. In the best case scenario, the failover happens successfully and no VMs are affected. If a networking node is rebooted before all of the rebuild has completed, there may be a gap where dhcp requests are not being answered. Red Hat based OSes like CentOS stop attempting to get a dhcp address after a certain period.

Rebooting the instance should bring the instance back online.

I want to use Microsoft Windows on Jetstream

Jetstream presently does not support Microsoft Windows at all. It will likely not be integrated into Atmosphere during the life of this project. If you wish to explore it with the understanding that we cannot provide support beyond normal VM operations, you can refer to Microsoft Windows on Jetstream.

I have created a custom image in the Atmosphere interface but it isn't deploying properly in the API interface. (or vice versa)

We cannot recommend strongly enough not using Atmosphere images on the API side of Jetstream or vice-versa.

The Atmosphere user interface is designed to be extremely easy to use, particularly for beginning users but the trade-off is that Atmosphere enforces certain practices behind the scenes.

The API user interface is close to raw Openstack, makes almost no assumptions, requires advanced knowledge of Openstack and Linux, but is therefore much more flexible.

API and Atmosphere base images provided by Jetstream staff require SUBSTANTIAL work to make them CLEANLY work in the other environment.

Jetstream resources may not be available from these countries

Because of restrictions places at the federal level, access to Jetstream may be limited from the following countries below.

More information is available from the following US Government websites:

DDTC

ITAR

Country Top-level Domain
Afghanistan .af
Belarus .by
Central African Republic .cf
Congo .cg
Cote D'Ivoire .ci
Cuba .cu
Cyprus .cy
Eritrea .er
Fiji .fj
Haiti .ht
Iran .ir
Iraq .iq
Lebanon .lb
Liberia .lr
Libya .ly
Myanmar .mm
North Korea .kp
Somalia .so
Sri Lanka .lk
Sudan .sd
Syria .sy
Venezuala .ve
Vietnam .vn
Zimbabwe .zw
Image template of a virtual machine containing an installed operating system, software, and configuration
Instance launched image of a virtual machine
Information as a Service (IaaS) form of cloud computing that provides virtualized computing resources over the Internet. Read more...
Platform as a Service (PaaS) cloud computing model that delivers applications over the Internet. Read more...
Software as a Service (SaaS) software distribution model in which applications are hosted by a vendor or service provider and made available to customers over a network, typically the Internet. Read more...
Volumes small virtual filesystems that may be attached to running instances. Read more...