Jetstream2 User Guide
Last update: April 14, 2022

Jetstream2's User Guide is Coming Soon

Indiana's Jetstream2, in collaboration with Arizona State, Cornell, Hawai'i and TACC, entered into early operations in February, 2022.

Preliminary User Guide information is presented below. Updates are being posted as they become available. Please see Jetstream2 Home for the most recent changes, and subscribe to XSEDE User News to receive direct announcements.

Jetstream2

Jetstream2 is a user-friendly cloud computing environment for researchers and educators running on OpenStack and featuring Exosphere as the primary user interface and Cacao (formerly known as Atmosphere) as an additional graphical user interface. It is built on the successes of Jetstream1 and continues the main features of that system while extending to a broader range of hardware and services, including GPUs, large memory nodes, virtual clustering, programmable cyberinfrastructure with OpenStack Heat and Terraform, and many other features. It is designed to provide both infrastructure for gateways and other always on services as well as giving researchers access to interactive computing and data analysis resources on demand.

For a more in-depth description please see the System Overview.

Migrating from Jetstream1 to Jetstream2

The team is actively working on a migration plan for allocations and resources on Jetstream1 but it is not publicly available at this time.

Jetstream1 will remain online through the end of Q1 2022. We will stop accepting any new allocations on Jetstream1 as soon as Jetstream2 enters operations phase. That date is still yet to be determined.

VMs on Jetstream1 will continue running as long as you have a valid allocation there and until it is decommissioned. The TACC cloud of Jetstream1 will likely be taken down in stages prior to the IU cloud, but those timelines are still up in the air, as well.

Migration plans are still under development. We will provide instructions for moving your VMs and possibly have some tools to help facilitate that. It is also highly likely to NOT be an entirely automatic process. Volumes will not persist because they are completely separate systems and do not share storage. There will be migration information for that, as well.

Please watch this space and XSEDE user news and mailing lists for announcements about Jetstream2 operations and migration. To ensure that you are receiving all updates to your inbox, login on the page above to manage your XSEDE User News subscriptions. All Jetstream2 users should be added when they are placed on an allocation, but you may wish to verify so that you don't miss any important migration updates.

Important note for CLI users

Jetstream2 uses application credentials for CLI access so CLI users can authenticate using XSEDE credentials. Please see Setting up openrc.sh for the Jetstream2 CLI for instructions on generating application credentials and an openrc for Jetstream2.

Updates will all be made here and shared via XSEDE User News.

Early Operations FAQ

Early operations has started on Jetstream2! Official start date is February 9, 2022. We will continue early operations until acceptance is complete by the National Science Foundation. We will post more complete timelines as we will post them here and all official announcements will be made via XSEDE user news.

  1. When will Jetstream2 be available for XSEDE research allocations?

Jetstream2 is presently available only via the XSEDE Resource Allocation System for research allocation submissions. The system is still not available to researchers, educators, or students at this time. All research allocations successfully receiving a Jetstream2 research award will be amongst the first groups given access to Jetstream2.

Please see the XSEDE Research Allocations page for more information.

  1. When will Jetstream2 be available for XSEDE startup and education allocations?

Until Jetstream2 completes National Science Foundation (NSF) acceptance testing, timelines remain uncertain. Once acceptance testing has been completed, documented, presented and approved by the NSF, we will be able to publish a more definitive timeline.

Please watch this space and XSEDE user news and mailing lists for announcements about Jetstream2 operations, access, and migration. To ensure that you are receiving all updates to your inbox, login on the page above to manage your XSEDE User News subscriptions. All Jetstream users should be added when they are placed on an allocation, but you may wish to verify so that you don't miss any important operations updates.

For information about migrating from Jetstream1 to Jetstream2, you may want to also monitor Migrating from Jetstream1 to Jetstream2

  1. When will Jetstream2 be available for Jetstream2 Trial allocations?

Jetstream2 Trial Allocations will be made available after Jetstream2 goes into production. They will likely not be available in early user operations.

Until Jetstream2 completes National Science Foundation (NSF) acceptance testing, timelines remain uncertain. Once acceptance testing has been completed, documented, presented and approved by the NSF, we will be able to publish a more definitive timeline.

Please watch this space and XSEDE user news and mailing lists for announcements about Jetstream2 operations, access, and migration. To ensure that you are receiving all updates to your inbox, login on the page above to manage your XSEDE User News subscriptions. All Jetstream users should be added when they are placed on an allocation, but you may wish to verify so that you don't miss any important operations updates.

  1. My research group wants to bring up a new gateway or infrastructure project, should I wait for Jetstream2?

Since the timelines for Jetstream2 are still uncertain at this time, we would recommend that your research group get a startup allocation now on Jetstream1 and get a supplement for JS2 when it becomes available. As we recommend the API side for gateways and infrastructure, migrating to JS2 will be reasonably easy.

More information about getting Jetstream1 startup allocations

More information about getting Jetstream1 API access

Jetstream2 Info

System Overview

Jetstream2 is a transformative update to the NSF's science and engineering cloud infrastructure, will provide 8 petaFLOPS of virtual supercomputing power to simplify data analysis, boost discovery, and increase availability of AI resources. It is an NSF-funded, user-friendly cloud environment designed to allow always on research infrastructure and to give researchers access to interactive computing and data analysis resources on demand, whenever and wherever they want to analyze their data. While it shares many common features and abilities with other research computing resources, it is not a traditional High Performance Computing (HPC) or High Throughput Computing (HTC) environment.

It provides a library of virtual machines and shared software designed to create research infrastructure or to perform discipline-specific scientific analysis. Software creators and researchers will be able to create their own customized virtual machines (VM) or their own private computing system within Jetstream2.

Jetstream2 features multiple user interfaces, including the following web-based user interfaces:

as well as the OpenStack Command Line Interface (CLI) and the OpenStack Software Development Kit (SDK).

The operational software environment is based on OpenStack.

Access

Jetstream2 is accessible primarily through either the Exosphere or Cacao web interfaces using XSEDE credentials via Globus Auth.

Jetstream2 is not accessible via XSEDE's Single Sign-On Login Hub.

Newly created XSEDE accounts must be added to a Jetstream2-specific allocation by the PI or Resource Manager in order to access Jetstream2.

Jetstream2 is meant primarily for:

  • interactive research
  • small-scale, on-demand processing
  • as a backend for Science Gateways distributing jobs to other HPC or HTC resources
  • for general research infrastructure or research-related development.

Jetstream2 may be used for prototyping, for creating tailored workflows to either use at smaller scale with a handful of CPUs or to port to larger environments after doing your proof of concept work at a smaller level.

About

Consisting of five computational systems, Jetstream2's primary system will be located at Indiana University, with four smaller regional systems deployed nationwide at partners Arizona State University, Cornell University, the University of Hawai'i, and the Texas Advanced Computing Center. The 8 petaFLOPS cloud computing system will use next-generation AMD EPYC processors and NVIDIA A100 GPUs, and 17.2 petabytes of storage.

Within the Pervasive Technology Institute the project will be led by UITS Research Technologies with key contributions from the Cyberinfrastructure Integration Research Center and the Center for Applied Cybersecurity Research. Additional Jetstream2 partners include the University of Arizona, Johns Hopkins University, and the University Corporation for Atmospheric Research, with Dell Inc. as the primary supplier.

For more information, please see:

Key Differences with Jetstream1

Jetstream1 utilized two primary clouds, geographically dispersed and powered by OpenStack, to provide thousands of concurrent virtual machines to researchers, educators, and students from hundreds of institutions across the United States. Jetstream1 featured Atmosphere, a user-friendly graphical user environment, as the means of access for the majority of users. Jetstream1 also allowed API access for infrastructure-driven projects like science gateways.

There will be quite a few similarities between Jetstream1 and Jetstream2. Jetstream2 will be powered by a more recent version of OpenStack and will still have a primary mission of providing virtual machine services to research and education faculty, staff, and students. Jetstream2, however, will build on what Jetstream1 delivered and provide a number of improvements and new services. This page will try to identify the major differences.

The biggest difference is that Jetstream2 will consist of one primary cloud and multiple regional clouds. As you can see by the image below, the primary cloud will be hosted at Indiana University in Bloomington, IN with regional clouds at various institutions across the United States.

Another huge distinction is that instead of different domains: default (Atmosphere) and tacc (API) domains, there is one namespace for Jetstream2. You can change between the CLI, Horizon, Exosphere, and Cacao as you see fit to manage your resources.

Allocations will only be awarded on the primary cloud by default.

Hardware

Jetstream2 will also bring multiple classes of research computing hardware. Jetstream2 will still have hundreds of CPU-based compute nodes for general purpose virtual machines/computing. Jetstream2 will also feature a small number of large memory nodes with up to 1 terabyte of RAM. Jetstream2 will also make available 90 nodes of GPU-enabled nodes with four NVIDIA A100 GPUs. These will be subdivided using NVIDIA virtual GPU (vGPU) to allow Jetstream2 allocations to utilize from 1/8th of a GPU to an entire GPU in their instances to allow everything from educational use requiring a minimal amount of GPU processing power to a full GPU for research workloads.

Interfaces

Jetstream2 will also have multiple user interfaces. Atmosphere has evolved into a new tool called Containerized Atmosphere for Continuous Analysis Orchestration (CACAO or simply Cacao), which is built on the principles of Atmosphere (abstracting complicated functions such as firewalls and virtual networking). Jetstream2 will also provide API services utilizing both the OpenStack Horizon GUI and a robust command line interface (CLI). Because Jetstream2 will no longer have separate operating domains for Cacao and API operations, those utilizing Jetstream2 can switch between interfaces easily, seeing all virtual machines and other assets created in any interface. This single namespace also allows for third-party interfaces that can manage any OpenStack created resource to be used with Jetstream2. At the time of deployment, Jetstream2 will feature one such third-party interface called Exosphere.

Containers and Orchestration

Jetstream2 will bring support for containers to the forefront of services. It will also support managing and scaling container-based workloads via the cloud-native functionality of OpenStack Magnum as has been demonstrated with Jetstream1. Users will be able to deploy Docker Swarm, Apache Mesos, or Kubernetes container orchestration engines to manage and run their container-based research workloads. In addition, the features of Cacao will provide similar functionality to individuals who have no desire to access the OpenStack API directly. Both approaches will allow researchers and educators to scale their workloads dynamically according to their needs.

Additional Services

Services such as OpenStack Heat will be available for researchers and developers, as well. OpenStack Heat is a service that allows individuals to instantiate complex resources with dependencies via a declarative YAML-based language. Similar to Magnum, other OpenStack services such as Trove and Sahara also leverage Heat to provide relational and non-relational databases and to provision data-intensive application clusters. Further, tools such as HashiCorp's Terraform programmable infrastructure, with the ability to deploy on Jetstream2, private clouds, and commercial clouds easily and consistently, will allow developers and researchers to have their environements where they need and when they need it. These capabilities build on one of the fundamental aspects of cloud computing that was demonstrated in abundance with Jetstream1: the ability of users to create, manage, and orchestrate use of tools autonomously, based on need, without involving sysadmins to install or enable new software.

Virtual Clusters

In addition to the ability for individuals to control their infrastructure programmatically, Jetstream2 will provide the capability to spin up elastic HPC virtual clusters (VCs) at the push of a button. These have been tested extensively on Jetstream1, with about thirty VCs running in production at different times. These Slurm-powered virtual clusters allow individuals to transition easily between cloud and HPC resources, acting as both a test-bed environment for custom software, and a highly-available production resource for projects with modest computational needs. The deployment process for these resources in Jetstream2 will be streamlined, allowing individuals to deploy an instance, acting as a head node, that is ready to accept jobs. Once jobs are submitted, worker instances will be automatically created and destroyed as needed. The Singularity/Apptainer container runtime environment will be built into these VCs, allowing individuals to use containerized scientific software without lengthy installation processes.

Compute Nodes (384 nodes)
System Configuration Aggregate information Per Node (Compute Node)
Machine type Dell Dell PowerEdge C6525
Operating system Ubuntu Ubuntu
Processor cores 49,152 128
CPUs 768 (AMD Milan 7713) 2
RAM 192 TiB 512 GiB
Network 100 Gbps x 4 to Internet2 100 Gpbs to switch
Storage 14 PB Total Ceph Storage 240 GB SSD
Large Memory Nodes (32 nodes)
System Configuration Aggregate information Per Node (Large Memory Node)
Machine type Dell Dell PowerEdge R7525
Operating system Ubuntu Ubuntu
Processor cores 4,096 128
CPUs 64 (AMD Milan 7713) 2
RAM 32 TiB 1024 GiB
Network 100 Gbps x 4 to Internet2 100 Gpbs to switch
Storage 14 PB Total Ceph Storage 480 GB SSD
GPU Nodes (90 nodes)
System Configuration Aggregate information Per Node (GPU Node)
Machine type Dell Dell PowerEdge XE8545
Operating system Ubuntu Ubuntu
Processor cores 11,520 128
CPUs 180 (AMD Milan 7713) 2
RAM 45 TiB 1024 GiB
GPUs 360 (NVIDIA A100 SXM4 40GB) 4
Network 100 Gbps x 4 to Internet2 100 Gpbs to switch
Storage 14 PB Total Ceph Storage 960 GB SSD

Summary of Network Configuration and Policies

Hardware Configuration

The Jetstream2 primary cloud configuation features:

  • 100 Gbps network connectivity from the compute hosts to the cloud's internal network infrastructure
  • 4x40 Gbps uplinks from the cloud infrastructure to the data center infrastructure (2x100 planned)
  • 100 Gbps connectivity from the site infrastructure to the Internet2 backbone
  • 100 Gbps connectivity to the XSEDE research network via virtualized link
  • Individual instances have full access to this infrastructure with no added speed limits.

It is important to note that connections coming from commercial/commodity internet will likely not be as fast as those coming from Internet2 sites.

Persistent IPs:

A key difference between Jetstream1 and Jetstream2 is that no special or additional access is required to get a persistent IP address. Some of the GUI interfaces like Exosphere and Cacao release IPs by default when a VM is deleted. Horizon and the Jetstream2 CLI require you to explicitly release the IP address.

We do ask that you release any unused IPs back to the public pool. There are a finite number of IPs available and allocations hoarding them may inadvertently cause issues for other Jetstream2 researchers.

The Jetstream administration team reserves the right to release any IP addresses not associated with a VM as needed.

Network Security Policies:

In general, Jetstream2 does not restrict inbound or outbound access to virtual machines. There are a handful of blocks at the instutional level that are outside of the control of the Jetstream2 staff. In general, though, the most common Unix service ports (eg. 22/ssh, 80/http, 443/https, etc) are not restricted in any way. Whether they are open by default will be dependent on which user interface you're launching your VM with.

Please refer to the Security FAQ for additional information.

In-depth Exploration of Jetstream2 Networking

This section describes the network architecture of Jetstream2's primary cloud at Indiana University. Regional sites may be configured differently.

There are three kinds of networks used on Jetstream2.

  • The Cluster Network, which carries all of the tenant/workload/data traffic. It underlays the Neutron overlay networks for cloud workloads. It also carries all storage traffic and OpenStack control plane traffic.
  • The Neutron overlay networks, an entirely software-defined (virtual) networking stack which runs on top of the cluster network.
  • The Management Network, which is fully physically separate from the cluster network. It is used for managing hosts and their iDRACs.

This document primarily looks at the Cluster Network and may delve into the Neutron overlay networks. The Management Network is not in the scope of user-facing documentation.

Cluster Network

BGP in the Data Center is recommended reading. It provides background and orientation to Clos networks and BGP the way that it is used here. It is a short, accessible read for someone with general IT networking knowledge. For a quicker version, read chapters 1, 2, 4, and 6. Skim chapters 3 and 5.

The Jetstream2 network is a two-tier spine-and-leaf network in a fat tree or Clos topology.

There are no inter-switch links (ISLs). Traffic that transits the network will generally traverse a leaf switch, a spine switch, and another leaf switch.

Two of the leaf switches connect the Cluster network to the campus WAN and internet. These happen to be the two leaf switches that the control plane nodes are also connected to.

Physical Switches

Switches use the Cumulus Linux operating system.

Switch Types in the Cluster Network
Model Topology Layer Switch Quantity Port Speed Port Qty per Switch
Mellanox SN4600 Spine 6 100 GbE 64
Mellanox SN2700 Leaf 37 100 GbE 32

Dual-attached Servers

Each host (server) is dual-attached to the Cluster network via two separate leaf switches, using two 100 GbE interfaces. Dual-attaching serves both purposes of redundancy and load-balancing (via equal-cost multipathing).

Redundancy is particularly important here, because many workloads that run on Jetstream2 will not be 'cloud-native' or tolerant of partial-cluster network connectivity failure. Many users will hand-configure their analyses and services on a single instance (virtual machine) that runs on a single host in a single cabinet. Dual-attaching every physical server means that a single network switch failure will not cause a complete or prolonged outage for these non-cloud-native workloads. The physical compute node will quickly notice that one of its network links is offline, and re-route all network traffic to use the remaining functional link.

Server cross-wiring

Throughout the cluster, a given host may be variously connected to:

  • 2 switches in the same cabinet, or
  • 2 switches in an adjacent cabinet, or
  • 1 switch in the same cabinet and 1 in an adjacent cabinet.

The benefit of this cross-wiring between cabinets is increased utilization of switch ports, and reduction of the number of switches needed. This is especially true in Jetstream2 because different types of hosts (e.g. compute, GPU, storage) are different physical sizes, i.e. different densities in a cabinet. The cost of cross-wiring is limited cabinet mobility, because hosts in one cabinet may have many connections to a switch in the next cabinet over, but cabinets are not expected to be moved over the lifetime of Jetstream2.

Border Gateway Protocol

Jetstream2 uses Border Gateway Protocol (BGP) to route and deliver packets across the entire cluster network. BGP was originally created to handle traffic between large network service providers (a.k.a. carriers), but is also adapted for use within data centers. In this application, BGP replaces use of MLAG to route traffic over redundant physical paths between switches. Jetstream1 used MLAG, but Jetstream2 does not, for the following reasons:

  • MLAG is fragile when making config changes
  • "Rigorous and conservative interface state management needed. Temporary loops or duplicates not acceptable" (source)
  • MLAG does not maximize use of aggregate throughout supported by redundant physical links.
  • MLAG requires inter-switch links (ISLs), which would require many (about 96) additional links between switches, which would require more cabling and purchasing switches with more physical ports.

Use of BGP solves all of these drawbacks with MLAG. (See also MLAG on Linux: Lessons Learned. For a comparison of BGP and other internal routing protocols like Open Shortest Path First (OSPF), see pages 22 and 24 of BGP in the Data Center (labeled as pages 14 and 16)).

Some more configuration details:

  • On the Cluster network, peering between all nodes uses eBGP (external BGP), because each host and each leaf (but not spine) switch has its own individually-assigned private autonomous system number (ASN). iBGP (internal BGP) is not used.
  • The spine switches all share a common ASN, 64512. This avoids path hunting problems as described in pages 26-29 of BGP in the Data Center (labeled as pages 18-21).
  • Equal-cost multipathing allows packet forwarding to a single destination to be load-balanced across multiple network paths.

BGP Unnumbered

In the cluster network, there is no network bridging or explicitly-defined shared subnets between connected switches (or between switches and hosts). There is only routing of packets between hosts. BGP requires underlying TCP/IP connectivity in order to create a connection, so how can that happen if we have no explicitly-defined subnets or bridges?

In very short, each physical 100 GbE interface on each node (host or switch) has only a link-local IPv6 address assigned. IPv6 Router Advertisement (RA) is a link-level protocol that periodically announces an interface's IPv6 addresses, including the link-local address. This is how each node identifies the IP of its peer. RFC 5549 allows a system to advertise IPv4 routes, and route IPv4 packets, over a pure IPv6 network. It also adds an "extended nexthop" capability to BGP, which negotiates use of RFC 5549 over a BGP peering session.

This strategy is described in greater detail in chapter 4 of BGP in the Data Center. The result is that there is no need to define a shared subnet between any nodes. There is also no need to explicitly define neighbor-specific information in a node's configuration. Two peers connected with a network cable will discover each others' link-local addresses, exchange IPv6 route advertisements, and initiate a BGP route exchange.

Routing on the Host

In traditional enterprise networking, packet routing (at layer 3 in the OSI model) is generally performed off the host, i.e., by router hardware that is physically separate from servers running the workloads. Sometimes there is purpose-dedicated router hardware, and sometimes routing is a role performed by the ethernet switches. In either case, servers do not make most packet routing decisions. Servers are bridged to routers at layer 2, on a common subnetwork. Servers determine where to send packets using ARP (Address Resolution Protocol). This is not how the Jetstream2 Cluster network operates.

Instead, the BGP routing fabric is extended all the way to each server. Each host has its own BGP autonomous system number (ASN), and runs the same FRRouting software as the switches. The host's primary IPv4 address (in private, RFC 1918 space) will appear on the loopback interface, and it is configured as a /32 in CIDR notation as its very own single-IP subnetwork, not bridged anywhere else!

A server advertises BGP routes for its own internal IPv4 address. These routes are advertised to both switches that the server is attached to, via IPv6 link-local connectivity. This overall strategy is called "Routing on the Host", and it is described in chapter 6 of BGP in the Data Center.

Again, in the cluster network there is no bridging between physical nodes, only routing.

Some more background reading (if you don't like the book): - Linux Routing On The Host With FRR (archived version) - Independence from L2 Data Centers - Border Gateway Protocol (archived version)

Conclusion

All of this BGP unnumbered architecture supports the 'Abstraction layer' of one or more IPv4 addresses on each host, and the ability to route packets from any host to any other host by its IPv4 address. At this level of abstraction and above, there is no concern with (or visibility of) the BGP routing layer or link-local IPv6 addresses. The Neutron overlay networks do not see it, or know about its complexity.

Neutron Overlay Network

Areas of future discussion:

Neutron Overlay Networks

TODO

VXLAN / Open vSwitch / Linux Bridge?

Layer 3 (Network)

Neutron Distributed Virtual Routing (DVR)

Typically with DVR, egress traffic leaves a compute node directly to the WAN, while ingress traffic passes through a network node.

See also: - https://docs.openstack.org/neutron/latest/admin/config-service-subnets.html

Jetstream2 FAQ Home

For specific FAQs, see the following pages:

I need a root disk larger than the maximum size for Jetstream2 instances. Can you create a custom flavor for me?

In OpenStack, flavors define the compute, memory, and storage capacity of instances. We may also refer to it in support tickets or documentation as the VM's size.

We won't create custom flavors, but there are ways to get larger root disks. You can review the flavors and see if moving up from one of the smaller VMs to a slightly larger one would yield a larger root disk. The other option is to use a custom sized root disk using what Openstack calls "boot from volume", or a "volume-backed" instance. What this means is that instead of an ephemeral boot disk for the instance, a volume is used to be the root disk.

There are several upsides to this:

  • You can have a root disk large enough to fit your needs
  • The disk becomes reusable as a volume if necessary
  • Shelving the instance is extremely fast compared to shelving a typical instance

The downside is that using boot from volume will count against your Jetstream2-Storage allocation whereas root disks do not.

Instructions for using boot from volume are here:

Will there be Microsoft Windows support on Jetstream2?

Microsoft Windows is not officially supported on Jetstream2. We will be making a limited number of images available for experimental use.

It is not known at this time whether GPUs will work on Microsoft-based instances. We will test this as time permits.

More information may be found on the Microsoft Windows on Jetstream2 page.

How do I share a volume between virtual machines?

You can't easily share volumes in OpenStack without deploying a Shared File System service. However, the native Openstack Manila filesystems-as-a-service option is available.

Instructions for using manila on Jetstream2 are at Manila - Filesystems-as-a-service - on Jetstream2.

Can I set the password for a user on my virtual machine?

We generally don't recommend using password authentication on Jetstream2, recommending that you use SSH keys for access. That said, if you need to set a password for console access or for some other reason, you can do it like this:

sudo passwd *username*

Troubleshooting

Jetstream2 requires that you be on a valid XSEDE allocation. If you are having issues accessing the various Jetstream2 interfaces, please first verify that you're on a valid allocation via the XSEDE User Portal.

General Troubleshooting

  • Topics will go here

Allocations FAQ

  1. Is there an overview of the types of allocations available as well as any restrictions those allocations have?

    The Getting Started guide describes the process of getting onto XSEDE, applying for allocations and using XSEDE resources. To review the types of allocations XSEDE and the process to get an allocation, here are some links you might find useful:

  2. Is there a example or demonstration of how to get an allocation that I could follow?

    There is a Cornell Virtual Workshop (CVW) on Getting a Research Allocation for Jetstream. UPDATE FOR JS2

  3. How do I let other XSEDE accounts use my allocation?

    You can add users to (or remove them from) your XSEDE allocation via the XSEDE User Portal. Users must have already created their XSEDE accounts before they can be added to an allocation.

    The principal investigator, co-principal investigator, or allocation manager can follow these instructions to add users to, or remove them from, an active Extreme Science and Engineering Discovery Environment (XSEDE) allocation.

    Please note that it can take up to four hours for users added to an allocation to become active.

  4. How often can I get a startup allocation?

    Applications for startup allocations will only be accepted once. If you have modest needs that are equal or less than startup values, you may renew your startup allocation. If you need a larger allocation, it is best to apply for a research allocation.

    Maximum Startup/Campus Champion Allocation values for each resource are:

    • Jetstream2 CPU - 200,000 SUs
    • Jetstream2 Large Memory - 400,000 SUs
    • Jetstream2 GPU - 600,000 SUs
    • Jetstream2 Storage - 1TB default*

    * Storage limits may be larger than 1TB per allocation for a startup if well-justified.

  5. I'm running out of Service Units (SUs) or storage. How do I request more?

    If you already have an XSEDE allocation and need to request additional service units (SUs), the PI, co-PI, or delegate may submit a request via the XSEDE User Portal. For instructions on how to submit the request, see Jetstream2 Allocation Supplements.

  6. I am at or exceeding the quota limits for my allocation.

    How do I request additional resources such as CPUs and memory?

    You may contact or with those requests.

    It is important to note that Jetstream Trial Allocation quotas are fixed and will NOT be increased under any circumstances.

    For other allocation types, justification will be required and will be granted at the discretion of the Jetstream2 staff based on the justification and available resources. Please note that large memory and GPU resources are limited so requests for those will require strong justification for success or partial success. We strive to make resources available to all researchers that require them, so striking a balance between the needs of one versus many is often necessary.

Security FAQ

Do I need to patch/update my VMs?

The featured images provided by the Jetstream team have unattended security updates enabled. Instances will not reboot, but they will apply any update marked as a security update. It's still a good idea to update your VM periodically.

  • CentOS 7: sudo yum update

  • Rocky 8: sudo dnf update

  • Ubuntu: sudo apt-get update ; sudo apt-get upgrade

If the kernel or glibc/libc packages are being updated, rebooting is necessary to implement those updates

Always run updates before taking snapshots so your new instances will have the latest patches in place.

Do I need to secure services running on my VMs?

If you have a public IP (also called a floating IP in OpenStack terminology), your instance may be exposed to the internet. While you can use security groups or host-based firewalls like iptables or UFW to limit access, it's best to take a posture of defense in depth and always secure any listening services.

We recommend limiting access to any service ports via security group and/or firewall in addition to making sure any non-public services like databases have secure passwords for all accounts. If your service only needs to be accessible to the VM it's running on, check the documentation to see if it can be bound only to the localhost port. (Not all services have this ability.)

Do I need to run a firewall on my VM?

Jetstream2 staff encourages a defense-in-depth approach to security. This potentially involves several methods of restricting access and securing instances.

Firewalls are not enabled by default on Jetstream2 instances. Depending on the user interface you launched your instance from, you may have different security groups established for your instance. (See What is the default security profile for Jetstream2 VMs? below for more information on that.)

We encourage keeping your instances patched, rebooting as needed for any kernel or glibc patches, limiting access to all services as much as possible, utilizing security groups if your interface allows it, and running your own host-based firewall if you're comfortable administering it.

Please refer to Jetstream2 Virtual Machine Firewalls for more information.

What is the default security profile for Jetstream2 VMs?

That depends on the interface.

The CLI and Horizon by default allow egress only. You have to apply the appropriate security groups for ingress. Please refer to the CLI or Horizon sections for more information on managing security groups in those interfaces.

For Exosphere, the default security group allows all egress and inbound access.

For CACAO, the default security group will be announced when it is available for general usage.

In Exosphere, there is a way to get a passphrase for any instance. Can I prevent other users on my allocation from accessing my instance(s)?

The way Jetstream2 is currently architected, all users on an allocation have access to all resources on the allocation. By default, Exosphere hides some resources created by other users, but this is only a convenience and it cannot assure separation of access.

It is possible to make it less straightforward for another user on the same allocation to access your running instance. You can do this by changing the password for the default exouser account. Changing the password does not prevent any access, but makes it more difficult. (Note that currently, changing the exouser account password will break Web Shell, Web Desktop, and some other Exosphere-powered instance interactions. This may change in the future.)

We note how to change a user password at Can I set the password for a user on my virtual machine?.

You can do:

sudo passwd exouser

and that solves the issue of them being able to access your instance using the credential listed on the Exosphere page. While you can use the console option still, we HIGHLY suggest utilizing ssh keys for your instances to ensure you have access. That's covered in Creating an Instance with Exosphere under the advanced options. You can also manually add your key to an already running instance.

Software FAQ

Jetstream2 provides some software packages for all researchers. The sheer number of available packages across all of the scientific disciplines makes it impossible to meet the desires of everyone in installing and maintaining software for research.

How can I see what software is maintained by the Jetstream2 staff?

See the Software Collection section.

How do I access the software in this collection?

This software store is mounted automatically when the instance is launched. Instructions for using it are in the Software Collection section.

Can I request to have other software packages added to the software collection?

You can request software using the contact form.

Please keep in mind that we cannot add and maintain all requested software packages. We will review any requests and track those requests.

Since everyone on Jetstream2 allocations potentially has root access on their VMs, you can install any software you need. You can then snapshot and save your customized images.

General Usage Info

Jetstream2 Resources

Jetstream2 consists of four distinct XSEDE-allocated resources. These resources are:

The specifications for each are linked above.

With the exception of Jetstream2 Storage, these resources may be allocated individually. Jetstream2 Storage requires the PI apply for or already have an allocation on one of the three compute resources.

Maximum Allocations

Maximum Startup/Campus Champion Allocation values for each resource

  • Jetstream2 CPU - 200,000 SUs
  • Jetstream2 Large Memory - 400,000 SUs
  • Jetstream2 GPU - 600,000 SUs
  • Jetstream2 Storage - 1TB default**

**Storage limits may be larger than 1TB per allocation for a startup if well-justified.

Limits and important notes

  • There are no request limits for Education or Research allocations. However, all requests must show the appropriate justification. Please refer to Jetstream2 Education Allocations or the XSEDE Research Allocation page for additional information on how to create an appropriate Education or Research allocation request.
  • There are no restrictions on runtime for the Jetstream2 CPU resource. As long as you have an active allocation and SUs remaining, you may run your Jetstream2 CPU VM(s) continuously or on demand.
  • Jetstream2 GPU and Jetstream2 Large Memory may have runtime restriction placed at a future date. Present policy notes that Jetstream2 Staff may limit runtime on these resources to two weeks at a time. As long as resources are not in contention, we may opt to allow continuous running of VMs/services. Any change in resource limits will be noted via XSEDE User News.

Jetstream2 Storage

Jetstream2 storage is an allocated resource. All allocations will be given a default storage amount (as noted on the Jetstream2 Resources page).

This storage is usable by all users on that allocation so the PI may want to institute per user quotas or discuss proper usage etiquette with the members of their allocation. Jetstream2 staff will not institute per user storage quotas, with the exception of the Jetstream2 Trial Allocation.

Limits on Jetstream2 Storage

  • Startup allocations are generally limited to 5-10TB max
  • Education allocations are generally limited to 10TB max
  • Research allocations are genereally limited to 40TB max

All are subject to proper justification in the allocations process. Maxmimum values may be adjusted with proper justification and if there are adequate resources available. This is entirely at the discretion of the Jetstream2 team.

Please refer to the following pages for more information on using Jetstream2 storage under the various interfaces:

XSEDE Service Units

Jetstream2 allocations are measured in XSEDE Service Units (SUs).

SUs are consumed at a rate of:

  • Jetstream2 (CPU) - 1 SU per vCPU_core-hour (use of one virtual core of a CPU per hour)
  • Jetstream2-LM (Large Memory) - 2 SUs per vCPU_core-hour
  • Jetstream2-GPU - 4 SUs per vCPU_core-hour

Please refer to VM Sizes and configurations to see available VM flavors and per hour cost on Jetstream2.


  • SUSPENDED instances will be charged .75 of their normal SU value. (75%)
  • STOPPED instances will be charge 0.50 of their normal SU value. (50%)
  • SHELVED instances will not be charged SUs. (0%)

For Large Memory and GPU allocations, the vCPU core hour cost is 2x and 4x respectively as noted above.

The reason for continuing to charge for VMs that are not in a usable state is that they still consume resorces if they are suspended or stopped. In those states, they still occupy allocable/usable space on the hypervisor, preventing other users from using those resources.

For instructions on managing instances, please see:

VM Sizes and Configurations

Jetstream2 can be used in several different virtual machine (VM) sizes which are charged in service units (SUs) based on how much of the total system resource is used.

The basic unit of VM allocation for Jetstream2 is based on a virtual CPU (vCPU) hour: 1 service unit (SU) is equivalent to 1 vCPU for 1 hour of wall clock time. The table below outlines the VM sizes created for Jetstream2.

Please note that these are all separate resources. Jetstream2 CPU is the default resource. To use Large Memory or GPU resources, you must have an allocation for those resources.

While the root disk sizes here are fixed, there is an option called "boot from volume" that will allow you to specify a larger root disk using quota from your storage allocation. Instructions for that are in the user interface sections.

Table 1. Jetstream2 CPU
VM Size vCPUs RAM (GB) Local Storage (GB) SU cost per hour
m3.tiny 1 3 10 1
m3.small 2 6 20 2
m3.quad 4 15 20 4
m3.medium 8 30 60 8
m3.large 16 60 60 16
m3.xl 32 125 60 32
m3.2xl 64 250 60 64
m3.3xl* 128 500 60 128

*m3.3xl will not be available by default. It will only be available by request and with proper justification

Table 2. Jetstream2 Large Memory
VM Size vCPUs RAM (GB) Local Storage (GB) SU cost per hour
r3.large 64 500 60 128
r3.xl 128 1000 60 256

Jetstream2 GPU

Jetstream2 GPU nodes charge 4 SUs per vCPU hour or 4 SUs per core per hour. Additionally, there are four NVIDIA A100 GPUs on each node. These GPUs are subdivided using NVIDIA Multi-Instance GPU (MIG) into up to 7 slices to allow more researchers and students to make use of the GPU resource.

7 GPU slices = 1 NVIDIA 40GB Ampere A100 GPU

Table 3. Jetstream2 GPU
VM Size vCPUs RAM (GB) Local Storage (GB) GPU Slices/GPU RAM SU cost/hr
g3.small 4 15 120 1 slice/5GB RAM 16
g3.medium 8 30 120 2 slices/10GB RAM 32
g3.large 16 60 120 3 slices/20GB RAM 64
g3.xl 32 125 120 7 slices/40GB RAM 128

This flavor information may be subject to changes in the future.

Example of SU estimation

  • First determine the VM resource appropriate to your needs (CPU only, large memory, GPU):
    • If your work requires 24 GB of RAM and 60 GB of local storage:
      • you would request 8 SUs per hour to cover a single m3.medium VM instance.
    • If your work requires 10 GB of local storage in 1 core using 3 GB of RAM:
      • you would request 2 SUs per hour for an m3.small VM instance.
    • If your work requires 1TB of RAM:
      • you would request 256 SUs per hour for an r3.xl instance on Jetstream2 Large Memory
    • If you work requires 20GB of GPU RAM:
      • you would request 64 SUs per hour for a g3.large instance on Jetstream2 GPU
  • You then would calculate for the appropriate resource (refer to the tables above):
    • For Jetstream2 CPU, you would then multiply by the number of hours you will use that size VM in the next year and multiply by the number of VMs you will need.
    • For Jetstream2 Large Memory and GPU, either refer to the SU cost per hour in the last column, or multiply hours times 2 for LM or 4 for GPU
  • To calculate the number of SUs you will need in the next year, first estimate the number of hours you expect to work on a particular project. For example, if you typically work 40 hours per week and expect to spend 25% of your time on this project that would be 10 hours per week.
  • Next, calculate the total number of hours per year for this project:
    • Total hours = 10 hours per week * 52 weeks per year
    • Total hours = 520
  • Finally, calculate the total SUs for the year for a single VM instance:
    • Total SUs = 520 hours per year * vCPUs
      • e.g. For a Medium VM instance: Total SUs = 520 hours per year * 8vCPUs
      • Total SUs = 4160
  • If your project requires more than 1 VM instance, multiply the total SUs by the number of VMs that you will need:
    • Total SUs needed for 3 medium size VMs = 3 * 4160
    • Total SUs = 12480

The calculations above assume that your VM is shutdown properly. For instructions see:

SU Estimation for Infrastructure or Always On allocations

For jobs that may need to run for extended periods or as always on infrastructure, you can take this approach:

VM cost (SUs) x 24 hours/day x 365 days = single VM cost per year

or as an example for each resource, an m3.large, r3.large, and g3.large each running for a year:

m3.large (16 cores) x 24 hours/day x 365 days = 140,160 SUs
r3.large (64 cores x 2 SUs/hour) x 24 hours/day x 365 days = 1,121,280 SUs
g3.large (16 cores x 4 SUs/hour) x 24 hours/day x 365 days = 560,640 SUs

Startup allocations are subject to limitation. Please refer to the Resources section for Startup Allocation limits.

For information on submitting a Research Allocation Request, please see Submitting a Successful Allocation Request. Note that all allocations above the startup level on Jetstream2 CPU require a strong justification for the time being requested.

Jetstream2 will have a limited set of featured images. These are images that the Jestream2 team curates and maintains.

A key difference between Jetstream1 and Jetstream2 is that there will not be application-specific featured images. We will maintain a software collection that will be made available on every virtual machine at boot. Software will be loaded via Lmod Modules.

At this time, the featured images will be:

  • Ubuntu 20.04
  • Ubuntu 18.04
  • Rocky 8
  • CentOS 7

These featured images will evolve over time. As distributions leave support (e.g Ubuntu 18 will end support in April 2023), we will replace them with newer, supported versions. NVIDIA drivers will be present on all featured images so any of the featured images will work with Jetstream2 GPUs.

Our goal is to maintain a minimum of featured images but keep them updated via automated pipeline on a weekly basis.

Instance Management Actions

Please orient yourself to the following instance management actions. These will help you use Jetstream2 effectively and conserve your allocation.

Each user interface for Jetstream2 has instance actions in a slightly different place. See guidance for Exosphere, Cacao, Horizon, and the OpenStack CLI.

Basic Actions

Shelve and Unshelve

When your instance is not performing work or otherwise in active use, please shelve it. Shelving an instance shuts it down and frees up resources on the cloud for other users. It also conserves the SUs (service units) on your allocation.

Shelving an instance shuts down its operating system. The instance's disk contents are preserved, but any running programs will exit, so please save any unfinished work before shelving.

Shelving and unshelving each take a few minutes, so shelving doesn't make sense for very short periods of inactivity. In other words, shelve your instance when you're done for the day or the week, not merely for your lunch break.

A shelved instance will not accept shell, SSH, or any other connections. So, if your instance runs a server that you want to provide others the ability to connect to at any time, you must leave it active. If this describes your instance, consider re-sizing it to the smallest flavor that will work for your server needs. This will conserve SUs on your allocation.

Lock and Unlock

Locking an instance helps prevent anyone from accidentally deleting or performing other actions on it, from Exosphere and all other Jetstream2 interfaces. If your instance is running an important job or used in "production" capacity, consider keeping it locked. You must unlock your instance again before performing other actions (such as shelving it). Locking and unlocking are non-disruptive actions; they do not affect a running instance.

Be aware that locking an instance does not prevent:

  • another user on your allocation from unlocking it.
  • modifications to the instance's filesystem(s) or running software. For example, someone with access to the instance could still log in and delete files. Locking only prevents instance actions at the cloud (OpenStack) level.
  • the instance from shutting off when your allocation expires or is exhausted.

Reboot

Rebooting an instance is just like restarting a computer. The cloud will first attempt a graceful or "soft" reboot, where all of your programs are allowed to exit. If that fails then OpenStack will perform a "hard" reboot, which will lose any work that is not yet written to disk. If you cannot connect to your instance, rebooting is a good troubleshooting step before creating a support ticket.

Advanced Actions

The following actions are for more sophisticated use cases. If you're a new cloud user, it's okay to skip reading about these for now.

Resize

Resizing allows you to choose a different flavor for your instance. When you resize, your instance will shut down and then restart with the new flavor (so please save any work in progress first). You can now resize using Exosphere, Horizon, or the CLI.

Consider resizing if you find yourself in one of these situations:

  • Your instance exhausts its compute resources, e.g. you run out of working memory (RAM) or you want it to process work faster.
  • Your instance's CPU is sitting idle most of the time, in which case a smaller flavor would burn your allocation more slowly.
    • Exosphere's instance resource usage graphs are a useful guide here.
  • You launched a GPU flavor, then later find that you no longer need the GPU, but want to keep using the instance.
  • You launched a non-GPU flavor, then later find that you want to use the same instance with a GPU.

If your software stack sometimes needs a large flavor to run a compute-intensive job, but you can develop and tune it on a smaller flavor, consider resizing down to a small flavor for development work, and back up when you're ready to run it at a larger scale. This get you best performance when you need it, while conserving your allocation when you don't.

Moving to a larger flavor is generally not appropriate in these situations:

  • The speed of your workload is limited by a process that is single-threaded (not parallelized). If this process cannot be parallelized then resizing is unlikely to speed it up.
  • The speed of your workload is limited by disk or network transfer speed. Larger instances do not have faster storage or network connectivity.
  • Your instance is running out of storage. Instead, create a volume, attach it, and move your data to the volume. If you're installing a lot of software that is not easily moved to a volume, resizing may be appropriate. Please open a ticket and ask for advice.

When resizing, you must select your desired new flavor. After the resize is complete, the instance will be in status "Resize verify". At that time, access the instance (e.g. using Web Shell or SSH) and confirm that it is working, then choose the "Confirm resize" action. If the resize process broke something and you need to return to the previous flavor, choose the "Revert resize" action.

Image

When you create an image, you capture the entire state of your instance's root disk. New instances can be launched from that image, which means that you can use an image to 'snapshot' and 'clone' an instance.

After you specify the image name, it will take several minutes for the image to finish creating and become active.

Consider creating an image in the following situations:

  • You want to create a new instance that is a clone of your existing instance. In this case, create an image of the existing instance and launch your clone(s) from that image.
  • You are about to perform a possibly destructive action on your instance (like installing, upgrading, or removing software), and you need the ability to go back and get to the prior disk state of the instance if something goes wrong.
  • You are building a software stack that other people will consume via their own instance (e.g. you are teaching a class). Providing an image can be an easy way for other people to get a new instance just like yours.

Be aware that system images quickly fall behind on operating system updates. As more time passes since an image was created, the more software will need to be updated when a new instance is created for it. This can lead to excessively long instance lanuch times and other problems. For this reason, custom images are not the right tool for sharing software or workflows more than a few months into the future. If this describes your situation, please open a support ticket and ask for advice.

Suspend and Resume

Suspending an instance is like placing a computer on standby (a.k.a. sleep). When you resume the instance, all running programs will be in the state they were in prior to entering standby. (Still, it is wise to save any work in progress before suspending.)

Please consider suspending instead of shelving only if you are running software that was complex or labor-intensive to start (not install), and you only need to leave it suspended for a relatively short time (e.g. a few days). Suspended instances still occupy resources on the cloud, and they continue to consume your allocation at a reduced rate.

Software Collection

Details on the Jetstream2 Software Collection will be forthcoming. At this time, we anticipate a shared disk that will be automatically mounted by instances. Researchers will then be able to use modules to load any combination of software they need, similar to how HPC systems do it.

This list will be ever evolving, but at this time, pre-launch, we anticipate the following software on the shared software store:

  • Matlab
  • R / R Studio / Shiny
  • Intel Compiler
  • NVIDIA (formerly PGI) Compiler
  • AOCC Compiler
  • Anaconda (may live on image itself for simplicity)
  • Databases (e.g. Mysql, Postgresql, Mongo)
  • Java
  • Jupyter/JupyterLab/JupyterHub
  • WQ-Maker
  • Genemark
  • BLAT

Items like CUDA and the NVIDIA HPC SDK are not installed on the VMs or in the software collection. We recommend using the NVIDIA containers for GPU-related software from NVIDIA as the ecosystem can often be problematic for individual local installations. The NVIDIA Container Toolkit is installed on all featured images.

This is still an ongoing effort so this list will almost certainly change in coming weeks and months.

Software Licenses

Software licenses are generally the responsibility of the user to obtain, coordinate, implement, and maintain.

Some licenses may require users to coordinate with Jetstream staff to allow for proper configuration. Such coordination is handled on a case-by-base basis.

Licensed software maintained by staff but available to users are listed below.

Compilers and Parallel Libraries

  • The Intel Compiler is covered on the primary cloud by a site license and has been configured to point to the proper license server.
  • All components of the Intel Parallel Studio are covered by the IU site license, including the MKL libraries.

Specialty software

  • MATLAB is covered on the primary cloud by a site license and has been configured to point to the proper license server.

If you need licenses for any other software that is not provided through the Jetstream2 shared software store then you will have to provide your own license. Licenses on regional clouds may vary and will need to be discussed with the regional cloud provider.

  • Please be aware that if you take an image or software from Jetstream and run it somewhere else then the license will not work. Jetstream license servers restrict access to only valid Jetstream IP addresses.

It is also the responsibility of the PI and those placed on each allocation to ensure that they are complying with the terms of any software license they are using on Jetstream2.

Policies and Best Practices

Acceptable Use of Jetstream2

Jetstream2 requires compliance with all XSEDE and Indiana University policies, including but not limited to:

Regional clouds may have additional usage and security policies. If you are using one of the regional clouds, please check with your support team there for more information.

Protected Data and Jetstream2

Jetstream2 researchers agree to:

  • Ensure that data that must be protected by Federal security or privacy laws (e.g., HIPAA, FERPA, ITAR, classified information, export control, etc.) are not stored on this system unless such storage and usage is specifically authorized by the responsible University administrator and complies with any processes for management of access to such information. For export controlled information, including ITAR information, approval of the University Export Compliance Office is required prior to use of the Jetstream2 systems for storage/processing of export controlled data. The Jetstream2 system is not intended, by default, to meet the security requirements of these laws or regulations and specific usage related controls or restrictions may be required prior to authorization of the use of the Jetstream2 system for such purposes.

  • Ensure that the project does not violate any export control end use restrictions contained in Part 744 of the EAR.
  • Follow all US government guidance on export controls for research and research data. Please see Jetstream2 Export Control Guidance for more information.

Jetstream2 and General Research Policies

In general, fundamental, publishable research is permitted on Jetstream2 from any non-EAR sanctioned countries. Fundamental research is defined as:

"Fundamental research means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from Industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reason." [National Security Decision Directive (NSDD) 189, National Policy on the Transfer of Scientific, Technical, and Engineering Information]

For anything other than fundamental, publishable research, you may wish to consult with your institution's export control office for any export/sharing restrictions.

Specialty System (GPU & Large Memory) Specific Policies

  • Only g3.* flavors should be run on the Jetstream2-GPU resource
  • Only r3.* flavors should be run on the Jetstream2-LargeMemory resource
  • Running standard compute (m3.*) flavors on the specialty resources may result in those instances being deleted without warning

Jetstream2 requires an active XSEDE allocation for access. If your allocation expires you will no longer be able to access Jetstream2 or your resources. Presently, XSEDE warns PIs monthly, starting at 3 months until allocation expiration, and at the time of expiration.

Jetstream2 policy is that we will do the following when allocations expire:

  • At expiration + 1 day - the allocation will be disabled on Jetstream2 and access is no longer possible to allocation users
  • If the allocation has not been renewed (preferred) or extended in 10 days, all VMs on the allocation will be shelved and thus no longer accessible
  • If the allocation has not been renewed (preferred) or extended in 30 days, all resources (VMs, volumes, shares, images, etc) on the allocation will be destroyed and will not be recoverable

Countries with Heightened Restrictions

Some countries have some heightened control over certain types of research and technology. Please consult with your institution's export control office to ensure that you are in full compliance with US regulations and institutional policies when working with countries under heightened controls and/or when working with research or information that may be limited by export control law.

The US Department of Commerce, Bureau of Industrial Security (BIS) publishes a list of EAR Country Group designations. Country Group D (countries of national security concern to the United States) face these heightened control restrictions and are included in this document of country groups. (Please note that Country Group D begins on page 6 of this document.)

Sanctioned Countries (not eligible to use Jetstream2)

In accordance to federal guidelines, Jetstream2 will not be accessible to IP ranges from the following countries:

  • Cuba
  • Iran
  • North Korea
  • Syria
  • Crimea Region of the Ukraine

The Bureau of Industry and Security (BIS) implements U.S. Government certain sanctions against these countries pursuant to the Export Administration Regulations (EAR), either unilaterally or to implement United Nations Security Council Resolutions.

The US Department of Commerce, Bureau of Industrial Security (BIS) publishes a list of EAR Country Group designations. Country Groups E1 and E2 identify embargoed countries subject to comprehensive restrictions and are included in this document of country groups. (Please note that Country Group E begins on page 8 of this document.)

AUPs for Hosted Gateways

Gateways hosted on Jetstream2 agree to abide by the usage policies defined on Policies: Acceptable Use and Research and Export Control Guidance. We also recommend having an Acceptable Usage Policy (AUP) on your gateway.

We recommend adding a statement to any acceptable use agreement that targets export control issues, such as the language below:

  • Ensure that data that must be protected by Federal security or privacy laws (e.g., HIPAA, FERPA, ITAR, classified information, export control, etc.) are not stored on this system unless such storage and usage is specifically authorized by the responsible University administrator and complies with any processes for management of access to such information. For export controlled information, including ITAR information, approval of the University Export Compliance Office is required prior to use of the [name of system] systems for storage/processing of export controlled data. The [name of system] system is not intended, by default, to meet the security requirements of these laws or regulations and specific usage related controls or restrictions may be required prior to authorization of the use of the [name of system] system for such purposes.

  • Ensure that the project does not violate any export control end use restrictions contained in Part 744 of the EAR.

The TrustedCI project maintains a sample AUP that you may use to model your gateway's policy on.

Best Practices

  • Update policies and tools
  • GPU 2-week policy

Windows on Jetstream2

Microsoft Windows is not officially supported on Jetstream2. We will be making a limited number of images available for experimental use.

While these images will be created by one of the Jetstream2 parters in conjunction with Jetstream2 staff, it is NOT supported nor guaranteed to work. It is unlikely that the Jetstream2 staff will be able to do more than general VM troubleshooting as we do not have Windows expertise on staff. No license will be installed so you will need to bring your own license.

It is not known at this time whether GPUs will work on Microsoft-based instances. We will test this as time permits.

Allocations

Allocations Overview

Access to Jetstream2 is available solely through Xtreme Science and Engineering Discovery Environment (XSEDE) allocations. You must be on a valid allocation or the PI of a valid allocation to have access to Jetstream2.

Jetstream allocations are awarded exclusively through XSEDE. XSEDE provides XSEDE User Portal (XUP) accounts free of charge. XSEDE allocations require that the Principal Investigator (PI) be a US-based researcher.

A guide with more detail on Jetstream allocations is available online at the XSEDE Resource Information page.

Allocation types:

  • Trial allocation: Limited in hours and cores, non-renewable, meant to explore using Jetstream2
  • Startup allocation: For smaller scale research or exploratory usage before applying for a larger research allocation. Not intended for any courses or workshops.
  • Education allocation: For workshops, tutorials, or courses. Startup limits may not apply.
  • Research allocation: Larger scale allocations intended for research. Limits are generally only in what the PI can justify requesting.

Portal Accounts

You'll need to have an XSEDE Portal account to request access to the Jetstream2 Trial Allocation, be added to a PI's allocation, or apply for an allocation.

To create an XSEDE portal account if you do not have one:

  1. Go to https://portal.xsede.org/
  2. Click "Create Account" on the left side of your screen.
  3. Fill out the form and click Submit.
  4. Upon receipt of the email notification click the link in the email to verify your account and set your username and password. If the link doesn't work, go to https://portal.xsede.org/, click "Sign In" and then select "Verify Account" under "Other Sign In Options".
  5. Following account verification, if not already logged in, go to https://portal.xsede.org/, click "Sign In" and sign in with the username and password set in the verification step.
  6. You will be asked to read and accept the User Responsibilities form. This form outlines acceptable use to protect shared resources and intellectual property.

Trial Allocation

Trial allocations will be available after Jetstream2 is available for general use.

This page will be updated with more information as we near that time.

Startup Allocations

Jetstream2 startup allocations are meant to be used for exploring the Jetstream2 system for research or research infrastructure purposes. They are not intended for teaching purposes. Please apply for an education allocation using the instructions here for any course, tutorial, or workshop needs.

You'll need a copy of your CV, abstract, and description of your intended research in PDF format.

You'll first need to create an XSEDE Portal Account if you do not already have one:

Portal Accounts

To create an XSEDE portal account if you do not have one:

  1. Go to https://portal.xsede.org/
  2. Click "Create Account" on the left side of your screen.
  3. Fill out the form and click Submit.
  4. Upon receipt of the email notification click the link in the email to verify your account and set your username and password. If the link doesn't work, go to https://portal.xsede.org/, click "Sign In" and then select "Verify Account" under the "Other Sign In Options".
  5. Following account verification, if not already logged in, go to https://portal.xsede.org/, click "Sign In" and sign in with the username and password set in the verification step.
  6. You will be asked to read and accept the User Responsibilities form. This form outlines acceptable use to protect shared resources and intellectual property.

Apply for an XSEDE Startup Allocation

  1. Read the Startup Allocations Overview. There are sample allocation requests in the overview that you may find helpful.
  2. Go to the XSEDE Resource Allocation System page. On the Available Opportunities page, click "Start a New Submission" under "Startup". If you are not familiar with the process, select "Begin Guided Submission" for step-by-step instructions.

Before submitting an allocation request have the following information available:

  • XSEDE usersnams for PI (required), Co-PIs (optional), and Allocation Managers (optional)
  • Additional XSEDE user names to add so they may use your allocation time and resources (optional)
  • Title
  • Abstract (typically a paragraph or two for an Educational request will suffice)
  • Keywords
  • Field of science (secondary areas of science may be also be added)
  1. Select your resources. You can have any combination of Jetstream2 CPU, Jetstream2 Large Memory, and Jetstream2 GPU, but you must justify Large Memory or GPU requests specifically.

  2. Select the appropriate resources(s) from the list.
    1. Enter the number of SUs you'll need. The Virtual Machine Sizes and Configurations page can help you figure VM sizes needed.
    2. Fill in number of Virtual Machines needed (this is an estimate for planning purposes (there's no wrong answer) try to best guess at how many instances you'll run at one time)
    3. Fill in number of public IP addresses needed (same as above)
  3. If you need additional storage beyond the VM's root disks, select "Jetstream2 Storage".
    1. All allocations by default will receive 1TB of storage so if that covers your needs, you do NOT need to select Jetstream2 Storage
  4. Upload your supporting documents (PDF format required)
    1. PI CV (2 page limit)
    2. CoPI CV required for every CoPI added to request (2 page limit) (optional)
    3. Syllabus
    4. Resource Justification (see below)
    5. Supporting Grants (optional)
    6. Publications of previous/supporting work (optional)
  5. Submit allocation request. At this point, all entered information is validated, errors or omissions are flagged.

Allow 1-2 business days for your application to go through the approval process.

Detailed information about the allocation request process, with screenshots, is available in the XRAS Submit Allocation Request Step-by-Step Guide.

Education Allocations

Jetstream2 Education Allocations

Jetstream2 education allocations are meant to be used for teaching courses, workshops, or tutorials. They are not intended for research.

You'll need a copy of your CV, the course syllabus, and a resource justification in PDF format. The latter is basically figuring the size of the VM your students will need, the number of weeks they'll need it, and the number of students. An example of the resource selection is below.

You'll first need to create an XSEDE Portal Account if you do not already have one:

Portal Accounts

To create an XSEDE portal account if you do not have one:

  1. Go to https://portal.xsede.org
  2. Click "Create Account" on the left side of your screen.
  3. Fill out the form and click Submit.
  4. Upon receipt of the email notification click the link in the email to verify your account and set your username and password. If the link doesn't work, go to the portal, click "Sign In" and then select "Verify Account" under the "Other Sign In Options".
  5. Following account verification, if not already logged in, go to https://portal.xsede.org/, click "Sign In" and sign in with the username and password set in the verification step.
  6. You will be asked to read and accept the User Responsibilities form. This form outlines acceptable use to protect shared resources and intellectual property.

Apply for Education Allocation

  1. Read the Education Allocations Overview. There are sample allocation requests in the overview that you may find helpful.
  2. Go to the Resource Allocation System page. On the Available Opportunities page, click "Start a New Submission" under "Educational". If you are not familiar with the process, select "Begin Guided Submission" for step-by-step instructions.

    Before submitting an allocation request have the following information available:
    
    * XSEDE usersnams for PI (required), Co-PIs (optional), and Allocation Managers (optional)
    * Additional XSEDE user names to add so they may use your allocation time and resources (optional)
    * Title
    * Abstract (typically a paragraph or two for an Educational request will suffice)
    * Keywords
    * Field of science (secondary areas of science may be also be added)
  3. Select your resources; you can have any combination of Jetstream2 CPU, Jetstream2 Large Memory, and Jetstream2 GPU; but you must justify Large Memory or GPU requests specifically
  4. Select the appropriate resources(s) from the list.
    • Enter the number of SUs you'll need. The Virtual Machine Sizes and Configurations page can help you figure VM sizes needed. See the section below for a walkthrough of creating your resource justification.
    • Fill in number of Virtual Machines needed (this is an estimate for planning purposes, there's no wrong answer, try to best guess at how many instances you'll run at one time)
    • Fill in number of public IP addresses needed (same as above)
  5. If you need additional storage beyond the VM's root disks, select "Jetstream2 Storage".
    • All allocations by default will receive 1TB of storage so if that covers your needs, you do NOT need to select Jetstream2 Storage
  6. Upload your supporting documents; PDF format required
    • PI CV (2 page limit)
    • CoPI CV required for every CoPI added to request (2 page limit), optional
    • Syllabus
    • Resource Justification (see below)
    • Supporting Grants, optional
    • Publications of previous/supporting work, optional
  7. Submit allocation request. At this point, all entered information is validated, errors or omissions are flagged.

Allow 3-5 business days for your application to go through the approval process.

Detailed information about the allocation request process, with screenshots, is available in the XRAS Submit Allocation Request Step-by-Step Guide.

Writing Your Resource Justification

A resource justification is required for an Educational allocation. You'll need to explain how the resources will be used and for how long. A sample calculation for your SU request might be:

16 week course
20 students
m1.medium instance size (8 vCPU)
Planning for 24 hour/day usage

8 vCPU (SUs) x 24 hours/day x 7 days x 16 weeks = 21,504 SUs/student x 20 students = 430,080 SUs

It would be standard practice to round that to 450,000 SUs for development time for the instructors plus instructor VMs during the duration of the course.

You'll also need to briefly explain why you chose the VM sizes that you did, what you expect each student to be running on those VMs. The Virtual Machine Sizes and Configurations page can help you figure VM sizes needed. A paragraph or two should suffice to cover this.

If you have questions about the resource justification please do contact Jetstream Help via and we can assist you.

Research Allocations

Coming Soon!

Jetstream2 Allocation Supplements

If your allocation has been depleted, or you need to add SUs or storage for a Jetstream2 resource to your existing allocation, you may request a supplement.

Detailed information about the supplement request process is available from the Manage Allocations page.

Instructions for Supplement Requests

  • Login to portal.xsede.org
  • Go to the Submit / Review Requests page
    • (You can also mouse over the "Allocations tab" and then select "Submit/Review Request")
  • Find your allocation. Under the Action menu for your allocation, select "Supplement"
    • Note: If your allocation is going to expire in 30 days or less, you will not see the Supplement option. You should see the Extension option, though. Choose a 3 month extension and once it's approved, the supplement button should appear.
  • Choose "Start Supplement"
  • Select the appropriate Jetstream2 resources(s) from the list.
  • Once you have selected the resources you need, you'll need to fill in the values of your request. The SU request has the same options as when you got your allocation. You'll fill in the SUs needed, the number VMs and IPs you expect to use (for forecasting purposes), and any comments.
  • For storage requests, you'll fill in the value in gigabytes, e.g. 1000gb = 1TB.
  • For the number of SUs you'll need. The Virtual Machine Sizes and Configurations page can help you estimage VM sizes needed and SUs that will consume.
  • For more information about the resources, please see the Jetstream2 Resources page
  • You will have to include a PDF "Progress Report", basically a paragraph or two on the need for the storage or SU request.
  • We do like to see any publications that have been published or submitted during your allocation period as part of this report.
  • Any other milestones or achievements would also be welcomed in the progress report along with user or job counts if it's a science gateway resource.
  • After you have added that, you may submit the request.

Requests are typically reviewed within 1 to 2 business days. You'll receive notification from XSEDE Allocations on the status of your request once it has been reviewed by all service providers on the request.

Allocation Extensions & Renewals

If your allocation is expired or close to expired and has adequate SUs remaining for the resource(s), you may request an extension if you need 3-6 months to complete your work or to prepare for a research allocation. If you are planning to continue your research for another year (or more), you should pursue a renewal of your allocation.

Detailed information about the extension request process is available from the Manage Allocations page.

Please note: Extensions are not a substitution for the renewal process.

If you have a startup allocation and you are planning to stay under the startup limits (as noted on the Resources page), you may continue to renew your startup and not have to enter the research allocations cycle. A renewal is ALWAYS preferred over an extension. Please see below for information on renewing a startup allocation.

  • Research allocations may be extended once with adequate justification.
  • Research allocations are renewable on a quarterly basis. More information may be found on the Research Allocations page

Instructions for Extension Request

  • Login to portal.xsede.org
  • Go to the Submit / Review Requests page
    • (You can also mouse over the "Allocations tab" and then select "Submit/Review Request")
  • Find your allocation. Under the Action menu for your allocation, select "Extension"
    • Note: You will not see the extension option until your allocation is within 90 days of expiring.
  • Choose "Start Extension"
  • Select the duration from the list.
  • Enter all comments and justifications for the extension.
  • After you have added that, you may submit the request.

Instructions for Renewal Request

  • Login to portal.xsede.org
  • You will need the same materials as a startup submission plus a progress report in PDF format with any pertinent information about the status of the project, including publications (preferably in bioliography form with DOIs or URLs), user and job counts for science gateways, and any other information you'd like to provide to XSEDE and the NSF.
  • Go to the Submit / Review Requests page
    • (You can also mouse over the "Allocations tab" and then select "Submit/Review Request")
  • Look on the LEFT side of the page and scroll down to find your allocation.
    • Note: You will not see the renewal option until your allocation is within 30 days of expiring.
  • Choose "Start Renewal for [Your allocation number]"
  • Select the duration from the list.
  • Enter all comments and justifications for the extension.
  • After you have added that, you may submit the request.

Requests are typically reviewed within 1 to 2 business days. You'll receive notification from XSEDE Allocations on the status of your request once it has been reviewed by all service providers on the request.

1. Is there an overview of the types of allocations available as well as any restrictions those allocations have?

The Getting Started Guide describes the process of getting onto XSEDE, applying for allocations and using XSEDE resources. To review the types of allocations XSEDE and the process to get an allocation, here are some links you might find useful:

2. Is there a example or demonstration of how to get an allocation that I could follow?

There is a Cornell Virtual Workshop (CVW) on getting a Research Allocation for Jetstream here: https://cvw.cac.cornell.edu/JetstreamReq UPDATE FOR JS2

3. How do I let other XSEDE accounts use my allocation?

You can add users to (or remove them from) your XSEDE allocation via the XSEDE User Portal. Users must have already created their XSEDE accounts before they can be added to an allocation.

To add users to, or remove them from, an active Extreme Science and Engineering Discovery Environment (XSEDE) allocation, the principal investigator, co-principal investigator, or allocation manager can follow the instructions for:

Please note that it can take up to four hours for users added to an allocation to become active.

4. How often can I get a startup allocation?

Applications for startup allocations will only be accepted once. If you have modest needs that are equal or less than startup values, you may renew your startup allocation. If you need a larger allocation, it is best to apply for a research allocation.

Maximum Startup/Campus Champion Allocation values for each resource are:

  • Jetstream2 CPU: 200,000 SUs
  • Jetstream2 Large Memory: 400,000 SUs
  • Jetstream2 GPU: 600,000 SUs
  • Jetstream2 Storage: 1TB default*

*Storage limits may be larger than 1TB per allocation for a startup if well-justified.

5. I'm running out of Service Units (SUs) or storage. How do I request more?

If you already have an XSEDE allocation and need to request additional service units (SUs), the PI, co-PI, or delegate may submit a request via the XSEDE User Portal. For instructions on how to submit the request, see Requesting additional SUs, other Jetstream resources, or storage for Jetstream: Supplemental Allocations.

6. I am at or exceeding the quota limits for my allocation.

How do I request additional resources such as CPUs and memory?

You may contact or with those requests.

It's important to note that Jetstream Trial Allocation quotas are fixed and will NOT be increased under any circumstances.

For other allocation types, justification will be required and will be granted at the discretion of the Jetstream2 staff based on the justification and available resources. Please note that large memory and GPU resources are limited so requests for those will require strong justification for success or partial success. We strive to make resources available to all researchers that require them, so striking a balance between the needs of one versus many is often necessary.

7. Can you extend my allocation for me or give me access to my allocation for just a few days/weeks/months more?

If your allocation is expired or out of SUs, you may request an extension, renewal, or supplement. Please see one of the following links:

Jetstream2 staff are unable to take these actions on your behalf.

Exosphere

Overview

Exosphere strives to be the most user-friendly interface for research clouds. If you are new to Jetstream2 and unsure which interface to use, Exosphere is a great place to start. Whether you are exploring new software tools, running compute-intensive jobs, teaching a course/workshop, or building a science gateway, Exosphere can likely help you reach your goals. Use the Exosphere interface for Jetstream2 in your web browser.

Right now, Exosphere supports creating and managing instances. Instances are virtual computers (a.k.a. servers) that run your code, containers, software, and services. When you create an instance, you get helpful features like:

  • A one-click web shell (terminal) in your browser
  • Optionally, a one-click desktop environment for running graphical software
  • A browser-based file upload/download tool
  • Resource usage graphs that show how hard your instance is working
  • Easy passphrase-based SSH access, if you want it

You can also use volumes to store large data sets, and manage persistent IP addresses for servers and science gateways. More powerful features, like data science workbenches and workflow sharing, are in experimental status now.

With Exosphere, there is no requirement to learn about advanced cloud technologies like virtual networks or SSH keypairs. If your use of Jetstream2 becomes more sophisticated, and you need to reach for more complex tools like the OpenStack CLI or APIs, Exosphere does not get in your way.

How Exosphere compares with the Horizon dashboard, OpenStack CLI, and APIs

Exosphere supports users who wish to mix their use of Exosphere with other OpenStack interfaces like Horizon dashboard, the OpenStack command-line interface, and the APIs. Generally (and with a few limitations), resources that you create in one interface will show up in other interfaces. They are merely different ways to manage the same infrastructure.

The other OpenStack interfaces support more features of OpenStack that Exosphere doesn't (yet), like Heat for cluster orchestration and Swift for object storage. So, they may better support some advanced cloud use cases than Exosphere, but they are generally less accessible to newer users. The Horizon dashboard, the OpenStack CLI, and the APIs were all built for use by IT engineers, not by researchers and data scientists. For example, in any of these tools you must create a network, subnet, router, security group, and SSH keypair before you can create an instance and connect to it (using an SSH client program and your private SSH key). If your use cases grow sophisticated enough, you may need this lower-level control, but using the Horizon dashboard is sort of like driving a car with a manual transmission. Using the CLI feels somewhat like using Horizon, but now you're shifting gears by typing shell commands instead of clicking buttons. Using the OpenStack APIs directly is like building your own transmission for the car.

Instances created via these other tools do not get a one-click shell, desktop, data upload/download tool, or any of the other interactions that Exosphere sets up for you. If you want these with the other OpenStack interfaces, you must set them up yourself with varying degrees of difficulty.

How Exosphere compares with Atmosphere2

This section will be populated once Atmosphere2 is ready enough to explore and compare.

Getting Help

In addition to the XSEDE ticketing system, there is an #exosphere-user-support channel in the Jetstream Slack workspace. To request access, please open a ticket from the XSEDE user portal. This support option includes no promise of immediate, real-time assistance, but the Exosphere core developers monitor it and help when they can. Sometimes it's easier to chat with them than wait for ticket notifications.

Contributing

If you'd like to help build Exosphere or request a new feature, visit the project on GitLab. The Exosphere maintainers strive to provide a welcoming experience for new contributors. At a broader level than Jetstream, the Exosphere project has a chat room on Matrix/Element which is used to coordinate development work, but community members are also welcome to join. Further, the Exosphere team discusses project progress and priorities on a weekly video call on Mondays at 16:00 UTC. You can join the call online or dial in at +1.512.647.1431, PIN: 3037 7824 88#. (The agenda and notes from previous meetings are available.)

Using XSEDE Account (single sign-on)

The default login method uses your XSEDE account credentials. Note that this may require multi-factor (Duo) authentication. (If you need help setting up or changing your multi-factor authentication method, please open an XSEDE ticket, as Jetstream2 staff cannot fix this for you directly.)

Choosing Allocations and Regions

After you log in with your XSEDE credentials, Exosphere will prompt you to select from the allocations that you are a member of. Any un-selected allocations will not be added the Exosphere interface, so select all that you may want to use. If you are granted access to a Jetstream2 regional cloud, you will be logged into those allocations in both the main (Indiana University) region and any other regions.

Logging into non-Jetstream Clouds in Exosphere

The Exosphere interface for Jetstream2 also allows you to manage resources on other OpenStack-based research clouds alongside Jetstream2. In order for this to work, these third-party clouds must expose their OpenStack APIs publicly, and you must have OpenStack credentials (or an OpenRC file) to provide. To add other clouds, choose "Add allocation", select "Other login methods", and pick the "OpenStack" login method.

If you encounter difficulty adding non-Jetstream2 clouds to Exosphere, Jetstream2 staff will have limited ability to troubleshoot and help, so this capability is not guaranteed to work.

Creating an Instance with Exosphere

Once you have logged in and selected an allocation, select "Create" and then "Instance".

Choosing an Instance Source

Next, choose an instance source. If you are a new user, select your desired operating system "By Type".

Choose by Type

Select your preferred operating system, or if you don't have a preference, pick the newest Ubuntu version. Exosphere selects the latest official image for your chosen type automatically.

Choose by Image

Alternatively, if you want to specify a particular image to create an instance from, select the "By Image" tab. Here, you can browse the entire Jetstream2 image catalog.

Configure Instance

Next, you can select options your new instance. If you're unsure what to choose for any of these, you can leave it at the default. When you're done choosing options, click the "Create" button at the bottom.

Choose a Name

If you will use the instance for anything important, resist the urge to accept the randomly-generated name. Give it a descriptive name to distinguish it from other instances later.

Choose a Flavor

Exosphere selects the smallest flavor by default. This is good for exploring and development work, because it conserves your Jetstream2 allocation.

Select a larger-size flavor if the smallest flavor proves too small for your explorations, or if you are ready to scale a workload up to many CPU cores / many GB of RAM. Note that larger flavors will consume your allocation (SUs) more quickly. Select a GPU flavor if you require a GPU (and your Jetstream2 allocation provides GPU access). You can always resize your instance to a different flavor later.

Choose a Root Disk Size

If the default size for the selected flavor is too small for your needs, you can specify a larger custom disk size. This will create a volume-backed instance.

Choose a Quantity

You can create multiple instances simultaneously, up to the maximum quantity your quota limits support.

When you create multiple instances at a time, and "X of Y" will be appended to the name of each. Otherwise, they will all will receive the same configuration.

Decide Whether to Enable Web Desktop

Enable Web Desktop if you need to use graphical software on your instance, or if you prefer working in a graphical environment instead of a text-based shell (terminal). Note that the graphical desktop environment consumes slightly more resources (CPU and RAM) than a shell, so consider using at least the m3.small flavor for a more responsive experience.

Advanced Options

Most people can skip these advanced options, and just click the "Create" button at the bottom. Advanced options are intended for power users and less-common use cases.

Install Operating System Updates

By default, new instances install operating system security and bug-fix updates when they first power on. This takes a couple of minutes. So, only skip these if you really want the fastest possible setup and you do not care about the security of your instance. This option does not disable unattended upgrades, which may still run in the background shortly after your instance powers on.

Deploy Guacamole for Easy Remote Access

By default, Exosphere instances provide easy remote access via a Guacamole-based terminal (Web Shell), and optionally a graphical environment (Web Desktop). If you don't want these for some reason, and you are comfortable using native SSH to connect to your instance, then you can disable setup of the Guacamole components.

Choose a Network

By default, Exosphere will use the network topology that OpenStack automatically creates for your allocation (and ask OpenStack to create it if needed). If you want your instance connected to a different OpenStack network, you can choose that here. If you change this without knowing what you're doing, your instance may end up with broken network connectivity.

Assign a Public IP Address

By default, Exosphere assigns a public (floating) IP address to your instances. Right now, a public IP is required for the Guacamole-based interactions (Web Shell and Desktop) to work. It is also required for you to make a direct native SSH connection to your instance from your workstation (or from anywhere else outside Jetstream2).

Still, you may be creating a cluster of instances, and you may not want all of the cluster nodes to have public IP addresses. In this and similar cases, you can disable the assignment of a public IP address here. If you disable the public IP address for an instance, the only way to connect to that instance will be via native SSH, from one of your other instances that does have a public IP address.

SSH Public Key

If you want to make a native SSH connection to your instance using public key-based authentication, you can select your public SSH key here, and upload a new one if needed.

Note that you do not need to use key-based authentication to connect to an instance via native SSH. Regardless of your choice here, Exosphere will also set a strong passphrase for the exouser user. You can view this passphrase (and use it to SSH in) after instance setup is complete.

Boot Script

Here you can see how the sausage is made! The term "boot script" is a slight over-simplification: this text area contains cloud-init configuration represented in YAML format, which Exosphere passes to the new instance when it is first powered on.

It is generally best to leave this text area alone, create the instance, then log into it and make further configuration changes as needed (e.g. installing software and downloading data).

Most of what you see here is important for Exosphere's interactions and other features to work with the new instance. So, if you make changes without knowing what you're doing, the instance setup may not complete. This will leave the instance in a partially working, partially broken state from Exosphere's perspective. In this case, Jetstream2 support staff would likely advise you to delete it and create another instance with this field left un-modified.

Still, if you are not afraid of editing YAML, you can modify this configuration before clicking the "Create" button. Note that Exosphere templates out a few important variables which are enclosed in single curly braces ({ and }) in this configuration data.

Finally, click the "Create" button at the bottom, and Exosphere will set up your instance.

Instance Management

In Exosphere, to see available management actions for an instance and choose one if desired, click the "Actions" drop-down menu on the instance details page.

Volumes

In Exosphere, you can create volumes and attach them to instances. Exosphere will display the volume mount point in the user interfaces.

Please note that attached volumes are mounted the first time they are accessed inside the instance (e.g. with a cd command).

File Shares

Support for shared filesystems (using OpenStack Manila) is coming soon.

Push-button Clusters using Exosphere

This guide assumes that you are comfortable creating instances using Exosphere. If you are new to Exosphere, see Exosphere: Overview.

Exosphere makes it simple to create a virtual Slurm cluster, complete with automatic, elastic scaling of compute nodes. This helps you get started with parallel computing, and to scale up your cluster as your needs grow.

To learn more about virtual clusters, as well as other ways to create them, see Advanced Capabilities: Virtual Clusters on Jetstream2.

Enable Experimental Features in Settings

Open the Settings page (top menu). Ensure that the "Experimental Features" option is enabled.

Create a Cluster Head Node Instance

Select "Create" and then "Instance".

For instance source type select Rocky Linux 8.

Choose a unique name (preferably a short name with no spaces or special characters) for your instance. Ensure there is no existing instance with the same name by looking at the Exosphere instance list page and pressing the "Clear filters" button.

This will be the base name for all resources created for the cluster, including networks, SSH keys, compute nodes, etc.

For flavor choose "m1.small".

Expand "Advanced Options".

Select an SSH key (highly recommended).

You should see an option "Create your own Slurm cluster with this instance as the head node". Select "Yes" for this option.

Click the "Create" button underneath the Boot Script text input area.

Wait until the instance is ready; this could take up to half-an-hour.

Run a Job on the Cluster

SSH into the new instance as exouser. (Alternatively use the web shell.)

Switch to the rocky user:

sudo su - rocky

Go to the cluster repository directory:

cd ~/CRI_Jetstream_Cluster/

Submit a test batch job:

sbatch slurm_test.job

View the live SLURM logs:

sudo tail -f /var/log/slurm/slurm_elastic.log /var/log/slurm/slurmctld.log

You should see messages indicating a new compute node being created as an OpenStack instance.

Confirm the new compute node instance exists by refreshing Exosphere's instance list page. Its name should begin with the same name as your head node instance, and end with -compute-0.

Once the job is finished confirm that the compute node is gone by refreshing Exosphere's instance list page.

Check the output of the test batch job by finding a new file ending with .out in the directory where you ran sbatch and viewing its contents. The file should contain two copies of the hostname of the compute node. For example:

ls -alt *.out
cat nodes_1.out

You should seem something like this:

$ ls -alt *.out
-rw-rw-r--. 1 exouser exouser 72 Jan 28 19:10 nodes_1.out
$ cat nodes_1.out
yourclustername-compute-0.novalocal
yourclustername-compute-0.novalocal

You now have your very own working Slurm cluster. Congratulations!

Clean-up steps

Once you're done with your cluster, and you want to get rid of the head node instance as well as all the OpenStack resources created for the cluster, run the following commands on the head node instance (in an SSH session or web shell):

sudo su - rocky
cd ~/CRI_Jetstream_Cluster
./cluster_destroy_local.sh -d

Your SSH or web shell session on the head node will be terminated, and you will be disconnected.

Confirm that the head node instance is gone by refreshing Exosphere's instance list page.

Troubleshooting

Coming soon!

Cacao (fka Atmosphere2)

Overview

Everything you wanted to know about Cacao to do more research in Jetstream2

What is Cacao?

What is Cacao?

Cacao is a multi-cloud orchestration service for researchers and educators that eliminates the complexity of using multiple clouds. By focusing on getting stuff done, Cacao helps transform research and education in a multi-cloud world.

Cacao is built and maintained by CyVerse, the NSF research project that created Atmosphere.

Why Cacao?

Cacao helps you use clouds with ease of use, flexibility, and collaboration.

Ease of use

Cacao makes it convenient to use your XSEDE Jetstream 2 allocation on any Jetstream cloud. You can organize your Openstack resources -- servers, volumes, containers, and more -- into workspaces. At the basic level, you can use Cacao just as you did with Atmosphere on Jetstream 1: launch a vm with the tools you use, do your work, and shutdown when you're done.

Flexibility (coming soon!)

Cacao helps adapt the cloud to fit your needs. Under the hood, your cloud resources are created using pre-defined Hashicorp Terraform templates. If you don't know Terraform, don't worry -- you don't need to know Terraform to use Cacao. However, you can create your own Openstack Terraform templates if you wish to level-up your Cacao workflow game. Future features in Cacao will include support for AWS/GCP/Azure+Terraform as well as non-Terraform-based templates, such as Kubernetes and Argo templates.

Cacao will also allow you to activate your resources when your data or workflow code changes and shutdown your resources after executing your workflow, in a Continuous Analysis way.

Collaboration (coming soon!)

Cacao makes collaboration in the cloud, less cloudy. You will be able share resources and workflows with the people you want.

What next?

If you are a new user to Cacao, see get started in using Cacao.

Glossary

Cacao Glossary

This glossary defines common terms you might see in Cacao.

Application credential

An application credential, specifically an Openstack application credential, is a credential that allows a user to login to an Openstack cloud without the user's password.

Continuous analysis

Continuous analysis is the ability to execute workflows triggered by events, such as data changes or code changes.

Deployment

A deployment is one or more resources, such as instances, created on one or more clouds.

Instance

An instance is another name for a virtual machine or server (in Openstack).

Meta cloud orchestration

The ability to provision resources in different types of clouds

Provider

A provider is another name for cloud or cloud provider, where cloud resources are created.

Resource

A resource or cloud resource is a general term used for any object -- instance, volume, container -- in a cloud.

Template

A template refers to a set of instructions or description of what cloud resources to create on a cloud. A template in Cacao requires a template engine to process the hte instructions or description. Currently, Cacao's primary templates for OpenStack use Hashicorp Terraform.

Workspace

A workspace is a way to group one or more deployments.

Cacao UI basics

Welcome to Cacao, a multi-cloud orchestration service. Cacao helps transform research and education in a multi-cloud world. This guide will cover how to navigate the user interface of Cacao.

The sidebar is the main navigation for the Cacao user interface.

Home

The Home view provides a dashboard for your account, including a summary of your providers, allocation, and resources used.

Deployments

Deployments displays your workspaces and deployemnts.

Credentials

Credentials provides access to your the various credentials used to access clouds and resources, including ssh key and openstack credentials.

Storage (still baking)

Storage displays your storage resources

Providers (still baking)

Providers displays details about your clouds.

Templates (still baking)

Templates provides advanced users access to import or share templates they create.

Help

Help provides access to help resources.

Your Account

Your Account menu shows your account information and access to your settings and preferences.

Getting Started for New Users

Cacao is a space for you to organize and manage your Jetstream cloud resources. Once you obtain your Jetstream allocation, this guide will help you get started.

  1. Login to Cacao

    Cacao uses Globus XSEDE credentials for identity.

    1. In your browser, connect to https://cacao.jetstream-cloud.org
    2. Click "Sign-In".
    3. If you are not currently logged into globus, you should select XSEDE
    4. Enter your XSEDE login credentials
    5. You may need to authenticate with your two-factor

  2. Add your Jetstream 2 credentials

    Access to any Jetstream 2 cloud will begin with adding your Openstack credentials, which may be different from your XSEDE credentials.

    1. Click on the Credentials menu
    2. Click on Add Credential button
    3. Select Jetstream Password Credential
    4. Select Cloud
    5. Enter Openstack Project Name; this is the default project to create resources
    6. Enter your Openstack Username
    7. Enter your Openstack Password
    8. Click on the Add button

  3. Add a Public SSH Key

    SSH Keys are used to login to virtual machines (sometimes called 'servers' or 'instances') after they are launched.

    1. Click on the Credentials menu
    2. Click on the Add Credential button
    3. Select Public SSH Key
    4. Enter a name for your public ssh key
    5. Paste in your public ssh key
    6. Click on the Add button

  4. Create your first workspace

    You can use Workspaces to organize your deployments.

    1. Click on the Deployments menu
    2. Click on the Add Workspace button
    3. Enter your Workspace name
    4. Enter a Description
    5. Enter a Default Cloud, which is used to deploy your resources
    6. Click on the Create Workspace Button

  5. Create your first deployment

    1. Click on the Deployments menu
    2. Click on the Add Deployment button
    3. Select Workspace, then Next button
    4. Select the "openstack-single-image-new" Template, then Next Button
    5. Select your Openstack credential, then wait a few seconds for the project list to be retrieved
    6. Select your Openstack project to launch your deployment
    7. Enter your deployment values A. Deployment name B. Select your image C. Number of instances D. Size (also called 'flavor')
    8. Click Next button
    9. Review the deployment settings, then click Submit button

  6. Learn about other Cacao features

Credentials

Coming soon!

Cacao allows you to manage various types of cloud and access credentials. Some credentials are required before using a cloud, such as the Jetstream Password Credential or the Jetstream.

Workspaces

Coming soon!

Choosing Your Allocation

Your account should have an Openstack project for every active XSEDE project-allocation. Jetstream2 uses the Openstack project to launch and recognize utilization against your XSEDE project-allocation.

Within Cacao, you can assign the XSEDE project-allocation in the deployment wizard when you select the appropriate credentials. Note, a Jetstream2 application credential will automatically assign the Openstack project, which is tied to the application credential. Jetstream password credentials allow you to change the XSEDE project-allocation.

Now that you know how to choose your allocation, you can learn more about Credentials and Deployments.

Logging into an Instance

Cacao provides easy access to your instance after it is launched as part of a deployment: * Command line access via SSH * Command line access using Web Shell * Web Desktop.

Instance Management

Coming soon!

Storage

Coming soon!

Troubleshooting

Coming soon!

Horizon

Overview

Horizon is a web-based GUI for interacting with the Openstack API. It is likely the most complete GUI for working with Openstack but isn't the fastest or most user-friendly. Almost all functionality that is available from the CLI is available in Horizon. Exosphere and Cacao both provide subsets of Horizon functionality with more focus on ease of use.

That said, there are times you may need the more complete features of Horizon instead of the other GUI interfaces. This documentation will cover the basics of launching, using, and managing instances, using various storage options in Horizon, using container orchestration engines, and other aspects of using Openstack.

Logging into Horizon

To login to Horizon and view the Dashboard, follow the steps in Using the Horizon Dashboard to Generate openrc.sh.

If you are on multiple XSEDE allocations, you'll want to verify you're using the correct one and change to the correct one if you are not.

You do that by clicking at the top left next to the Jetstream2 logo where it has "XSEDE * TG-XXXXXXXXX * IU". That will show allocations under "Projects".

Security Group Management

Security Group Management in Horizon

Security groups can be thought of like firewalls. They ultimately control inbound and outbound traffic to your virtual machines. Under the CLI and Horizon, access defaults to all outbound allowed and NO inbound allowed.

To allow access to your VM for things like SSH, you will need to create a security group and add rules to it.

You can reuse a security group many times, so a best practice is to create groups by related services. For instance, you might create a basic group for ssh and icmp (which is what we will show as an example) and then a separate security group for http and https access if you're running a web service on your instance.

Creating Security Groups and Rules

This will walk you through creating a basic security group in Horizon and adding a couple of simple access rules.

Login to the Horizon dashboard and make sure you've selected the correct allocation. Select the "Network" tab on the sidebar and click "Security Groups".

Once you're on the security group page, you'll need to click the "Create Security Group" button (noted with a red arrow on the screenshot)

In the popup box that comes up, you'll give your new security group a name (we suggest something like my-username-ssh-and-icmp) and optional description. We recommend giving a meaningful name and noting in the description what your intended purpose is.

When the creation is successful, it will bring you back to the security group page and note the success in the corner with a green status message. You'll see your new group name at the top where it says Manage Security Group Rules: your-rule-name.

You'll then want to click "Add Rule" (noted with a red arrow on the screenshot)

This will bring up a new dialog box where you can select the parameters for your security group rule.

If you click the "Rule" dropdown at the top, you'll see a list of common rule types as well as the option for custom rules. For this example, we'll select "SSH" to allow inbound port 22/SSH access.

We'll fill in the other details needed. We do recommend putting in a description with what the rule does so it's easy to see at a glance. We'll also select CIDR as the remote type and then set 0.0.0.0/0 as the CIDR. This allows all traffic to the SSH port.

You can make that be a single IP or a specific CIDR block.

In general, limiting access to specific CIDR blocks or IPs is best.

When the creation is successful, it will bring you back to the security group page and note the success in the corner with a green status message. You'll see your new rule now on the page.

You'll need to click the "Create Security Group" button and second time and we'll create a second rule for "All ICMP". This will allow things like incoming ping to check the status of your virtual machine. You'll select Ingress, CIDR, and set the CIDR to 0.0.0.0/0 to allow all hosts to ping your virtual machine.

As before, when the creation is successful, it will bring you back to the security group page and note the success in the corner with a green status message. You'll see your new rule now on the page.

You can then add additional rules or additional security groups. This will allow the most basic of access to your VMs.

We do recommend limiting access as much as possible for best security practices.

Manila via Horizon

To use Manila via Horizon

Create the Share

  1. Click on: Project -> Share -> Shares -> Create Share

  2. Create a share with the following settings:

    • share name: a name of your choosing
    • share protocol: CephFS
    • size: the size of your manila share
    • share type: cephnfsnativetype

Edit the Share Rule

  1. Once your share is available, you can select Edit Share and Manage Rules and Add Rule:

    • access type: cephx
    • access level: read-write
    • access to: an arbitrary unique name

    In the example above the accessTo name is manilashare. The name assigned must be globally unique, if you use a name that is already in use you will see an error state.

  2. If you now go back to the share page (Project/Share/Shares) and click on the share you created, you should see your share's metadata.

    Important things to note here are:

    • Path: ips:ports followed by volume path (/volume/_no-group/...)
    • Access Key

Using Manila Share on a VM

This is the same whether you're using Horizon or the CLI. Please refer to Configuring a VM to use Manila Shares

Troubleshooting

Coming Soon!

Command Line Interface (CLI)

CLI Overview

There are many options and tools for using the OpenStack API from the command line. Follow the instructions in the table below to set up a security policy and network, launch and manage a VM and then remove the entire structure.

Note: There is also an API tutorial that the Jetstream team uses here: Jetstream API Tutorial - this tutorial goes into greater detail on some topics and may be of value to those learning the Openstack CLI.

Please note that the tutorial above presently reflects using the API on Jetstream1. It will be updated soon for Jetstream2.

Important Notes

  • It is possible to create entities with the same name; e.g. you could create two networks with the same name; however, they will have different Universally Unique Indentifiers (UUIDs). When this occurs, you may get a cryptic error message that entity may not exist or that there are multiple entities with that name. In this case, you must address the entity by its UUID.
  • It is important to understand that everything is owned by the project. Other users in your project can see and manipulate entities that you have created. Be careful in your naming and pay attention to the things you are manipulating.

Getting Started

You should be running the latest version of the clients. We recommend using python-openstack >= 5.0 as it uses Python3 and removes the dependencies on the now deprecated Python2. See Installing Openstack clients for more information.

The next thing you'll need to do before being able to do any CLI commands is have an appropriate openrc file.

Please note that openrc files on Jetstream2 require application credentials. Please refer to Openrc setup for information on generating an application credential and openrc file.

Source the openrc:

source openrc-file-name

You can also make the output look nicer in your terminal with the -fit-width option:

openstack image show Featured-Ubuntu20 --fit-width

You can make that permanent by adding the following to your environment.

export CLIFF_FIT_WIDTH=1

You'll then need to create a security group and network before launching your first VMs. More information may be found here:

Mac-specific Steps

This will help you get the Openstack clients working on Mac OS X 10.11.x and higher. It may work on recent older versions of Mac OS X but it has not been tested.

At this time, Python 3 is still not shipping on OS X. The latest Openstack clients require it.

Follow the instructions below at your own risk.

Mac-specific Steps
Task Command
Install Homebrew (this might take a few minutes): /usr/bin/ruby -e "$(curl -fsSL
https://raw.githubusercontent.com/Homebrew/install/master/install)"
Using brew we're going to install Python 3: brew install python
Now Python 3 is installed we can install the OpenStack command line tools: sudo pip3 install python-openstackclient

Windows-specific Steps

We recommend that Windows users install Windows Subsystem for Linux and install Ubuntu within it. Microsoft has a learning module for this. The page Enable Windows Subsystem for Linux and install a distribution can walk you through that.

Once installed, you can verify python3 is installed by doing:

which python3

If you get an error, you may need to install Python3 by doing:

sudo apt install python3 python3-pip

Then you should be able to proceed to the Linux/common steps below.

Common Linux Steps

Note: Python3 is required. This should already be installed by your operating system. Openstack CLI clients MUST be installed with Python3's pip/pip3!

Install the OpenStack clients:

pip install python-openstackclient

Additional clients that may also be useful depending on your custom needs are:

python-swiftclient python-heatclient python-magnumclient python-manilaclient

For current users, clients that you likely no longer need to install are:

python-keystoneclient python-novaclient python-neutronclient python-cinderclient python-glanceclient

Set up your OpenStack credentials (see Setting up openrc.sh for details):

source openrc.sh

Test an Open Stack command:

openstack flavor list

Following future OpenStack updates, all installed pip modules can be updated with this command:

pip3 list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip3 install -U

Optional Steps

Though not strictly necessary, we recommend using virtualenv to increase the stability of the openstack cli tools.

cd to your preferred directory; create a directory for the project:

mkdir <project_name>

Change to the project directory

cd <project_name>

Install the venv packages

sudo python3 -m pip install --user virtualenv

Start the VirtualEnvironment software

python3 -m venv env <project_name>

Activate the VirtualEnvironment for the project

source <project_name>/bin/activate

Setting up openrc.sh

Setting up application credentials and openrc.sh for the Jetstream2 CLI

New openrc format for Jetstream2 CLI!

One of the key changes to using Jetstream2's command line interface (CLI) is that it uses XSEDE credentials for authentication. To do that, you have to create an application credential via the Horizon interface. This will require a different sort of openrc than Jetstream1. This page will walk you through that process.

Please make sure to source the new Jetstream2 openrc in a fresh terminal session. If you invoke it in a session that's had another openrc sourced, you'll get an error like this: Error authenticating with application credential: Application credentials cannot request a scope.

Openrc Files are Allocation-specific

Each allocation you wish to use from the command line will need its own application credential and openrc file.

You CANNOT use the openrc generator like in Jetstream1

The openrc generator on the Horizon right side (username) menu will NOT work properly with Jetstream2! Please use the process below to get your application credential based openrc file.

Using the Horizon Dashboard to Generate openrc.sh

  1. Navigate to https://js2.jetstream-cloud.org

    Make sure it says "XSEDE Globus Auth" in the Authenticate Using box.

  2. The first time you log in you'll be directed to a Globus page to permit authorization.

    If you have linked institutional, Google, Orcid, or other credentials, you'll be able to use those to authenticate.

    We know XSEDE credentials work correctly so we will show that in our example.

  3. The next page should be the login screen for your credentials. We're showing the XSEDE login screen as our example.

  4. If you're using two-factor auth with your credentials as XSEDE does, you'll likely get a Duo or Authenticator screen here.

  5. You should be at the Horizon Dashboard home now.

  6. As application credentials are unique to each allocation, if you are on multiple XSEDE allocations, you'll want to verify you're using the correct one and change to the correct one if you are not.

    You do that by clicking at the top left next to the Jetstream2 logo where it has "XSEDE * TG-XXXXXXXXX * IU". That will show allocations under "Projects".

  7. From here, you'll select Identity and then Application Credentials from the sidebar menu on the left

    Once on that page, you'll click "Create Application Credential" towards the top right (noted by the red arrow)

  8. This will bring up the application credential creation screen.

    The sidebar has descriptions if you need help.

    We recommend using a descriptive name and to put details in the description so you can easily see what it is for.

    The Secret is the password or passphrase. We recommend using a strong password here or multi-word passphrase. As the page notes, you will not be able to retrieve it later if you forget it or delete the openrc file you generate.

    Set the expiration date and time. If you do not set a date, it will default to TODAY as noted on the sidebar.

    We do not recommend setting the roles, access rules, or selecting unrestricted unless you are an advanced user and understand the full implications of altering these.

  9. When you hit "Create Application Credential" it will then generate the credential and bring up a confirmation box. Please make sure to save the credential ID and secret if you need them for things other than the openrc.

    To get the openrc for CLI access, please click the "Download openrc file" button referenced by the red arrow in the screenshot. That will download a plain text file for use with the Openstack CLI client

    We recommend giving your new openrc file a descriptive name (e.g. openrc-TG-TRA111111.sh, using the XSEDE project name or some other meaningful description.)

Using Openstack command line

Managing SSH Keys from the CLI

Secure shell (SSH) keys are crucial to securely accessing your virtual machines. Jetstream2 only supports ssh key access to CLI and Horizon virtual machines.

You can only add a key pair to an instance at the time of its creation, not afterwards, so it is important not to overlook this step. You'll want to have a key pair created before hand and uploaded to the Jetstream2 cloud.

While several formats are supported, we recommend using RSA or ED25519 format keys.

Source your openrc if you haven't already! Refer to creating an openrc if you still need to create an openrc file for Jetstream2. We're assuming it is openrc.sh for this example.

source openrc.sh

Uploading an Existing Key

If you have an existing SSH key, you can upload it to the Jetstream2 cloud (note: key filenames may vary - this example assumes id_rsa.pub):

cd ~/.ssh
openstack keypair create --public-key id_rsa.pub my-api-key

Make sure to ONLY upload the public key! You can change the my-api-key to something more descriptive.

Create a new key and upload it

If you don't have an SSH key, you'll want to create a new key and upload it.

If you don't have the .ssh dir, you can create it:

mkdir .ssh ; chmod 700 .ssh

Then you can create your key

ssh-keygen -b 2048 -t rsa -C "Identifying comment for this ssh key" -f ~/.ssh/id_rsa

or

ssh-keygen -o -a 100 -t ed25519 -C "Identifying comment for this ssh key" -f ~/.ssh/id_ed25519

The first will create a 2048 bit RSA cryptography key with the comment you specify and the filename id_rsa (which is the default). The second will create an Ed25519 elliptical cryptography key. The comment is optional but we do recommend it to keep track of keys. You only need to do one of these, though you can create both.

You may also leave off the -f file option and ssh-keygen will prompt you for the filename.

openstack keypair create --public-key id_rsa.pub my-api-key

or

openstack keypair create --public-key id_ed25519.pub my-api-key

Make sure to ONLY upload the public key! You should change the my-api-key to something more descriptive.

Setting up a Security Group

Manage Security Groups in the CLI

Security groups can be thought of like firewalls. They ultimately control inbound and outbound traffic to your virtual machines. Under the CLI and Horizon, access defaults to all outbound allowed and NO inbound allowed.

To allow access to your VM for things like SSH, you will need to create a security group and add rules to it.

You can reuse a security group many times, so a best practice is to create groups by related services. For instance, you might create a basic group for ssh and icmp (which is what we will show as an example) and then a separate security group for http and https access if you're running a web service on your instance.

Create a security group that will enable inbound ping & SSH:

This will walk you through creating a basic security group on the command line and adding a couple of simple access rules.

openstack security group create --description "ssh and icmp enabled" my-username-ssh-and-icmp-access

This creates the security group named my-username-ssh-and-icmp-access with the description noted above. It becomes the container for holding security group rules.

openstack security group rule create --protocol tcp --dst-port 22:22 --remote-ip 0.0.0.0/0 my-username-ssh-and-icmp-access

This creates an SSH access rule, allowing inbound TCP protocol to port 22 from any IP number.

openstack security group rule create --protocol icmp my-username-ssh-and-icmp-access

This creates an ICMP (most notably ping) access rule, allowing inbound ICMP protocol from any IP number.

You can then add additional rules or additional security groups. This will allow the most basic of access to your VMs.

We do recommend limiting access as much as possible for best security practices.

Please refer to the Openstack Security Groups documentation for more information.

Also, as with all openstack CLI commands, you can type openstack help item to get more information on the options and syntax.

Create a network in the CLI

In Jetstream2, if you do not need multiple networks in your allocation, you can skip all of the steps below. However, if there's more than one network in your allocation already, you will either need to use one of those or create your own network using these steps.

This step creates a virtual network for your instances. You'll create a network, a subnet, and a router and then set the routing so your instances can talk outside of the Jetstream2 networks to the world. You can have multiple networks in an allocation or also multiple subnets on a single network. This effectively isolates instances from each other.

If you do not give your instances a public IP address (also called a floating ip by Openstack), your instances will not be reachable from the internet. While this is not a substitute for proper security groups, it is another level of protection for VMs that may used internally only by your VMs (e.g. private database servers).

  1. Create a private network

    openstack network create my-network-name

  2. Verify that the private network was created

    openstack network list

  3. Create a subnet within the private network space

    openstack subnet create --network my-network-name --subnet-range 10.0.0.0/24 my-subnet-name

  4. Verify that subnet was created

    openstack subnet list

  5. Create a router

    openstack router create my-router-name

  6. Connect the newly created subnet to the router

    openstack router add subnet my-router-name my-subnet-name

  7. Connect the router to the gateway named "public"

    openstack router set --external-gateway public my-router-name

  8. Verify that the router has been connected to the gateway

    openstack router show my-router-name

Creating a Virtual Machine

See what flavors (sizes) are available

openstack flavor list

See what Images are available. The Jetstream team makes Featured-* images available from which to build.

openstack image list

Create and boot an instance

Please note:

  • Make sure your SSH keyname matches
  • your-instance-name is the name of the instance; make it something meaningful for you.
  • Choose an appropriate flavor from the list in the flavors list
  • Choose an appropriate image from the list in the images list

Run the following command to create and boot an instance:

openstack server create my-server-name --flavor FLAVOR --image IMAGE-NAME --key-name my-keypair-name --security-group my-security-group-name --wait

Optional: If you have multiple networks, you'll need to also include this line. See Create a network in the CLI for more information.

--nic net-id=my-network-name

Create a public IP address for an instance

openstack floating ip create public

Add that public IP address with that instance

openstack server add floating ip my-server-name your.ip.number.here

Logging into Your Virtual Machine

Once your instance is up and has a floating IP, you are ready to ssh in and use it. If your ssh key is one of the default names (e.g. id_rsa or id_ed25519) and is in your ~/.ssh dir, you won't need to specify the location of the key. Otherwise, you'll need to use the ssh option -i path/key. For example:

ssh -i ~/.ssh/my-custom-key-name user@ip.number

Each distribution has a different default user. We will show examples for each without the -i path to your ssh key, assuming you have ~/.ssh/id_rsa or ~/.ssh/id_ed25519 as your default key.

For Ubuntu 18 or 20:

ssh ubuntu@your.ip.number.here

For CentOS 7:

ssh centos@your.ip.number.here

For Rocky 8:

ssh rocky@your.ip.number.here

You should be able to access and use your VM now! Please see Instance Management Actions in the CLI for all instance management actions.

Instance Management

Instance Management Actions in the CLI

Once you have an instance, you may need to stop, start, or do other actions with it. You may wish to review XSEDE Service Units and Jetstream2 for more information about the "costs" of the various states of Jetstream2 virtual machines.

Rebooting

If your instance is running and you need to reboot/restart it, you can do so from the instance itself by doing:

sudo shutdown -r now

You can also do it from the CLI with the command:

openstack server reboot my-server-name-or-UUID
openstack server reboot my-server-name-or-UUID --hard

Stopping and starting

Stopping an instance is equivalent to powering down your laptop. We recommend this as a means to conserve SUs when you are done working but plan to use your VM in the next day or two.

You can stop the instance from it's own command line by doing

sudo shutdown -h now

You can accomplish the same thing from the CLI with:

openstack server stop my-server-name-or-UUID

and start it again with:

openstack server start my-server-name-or-UUID

Note that state is not retained and that resources are still reserved on the compute host so that when you decide restart the instance, resources are available to activate the instance.

Suspending and resuming

Another option for conserving some SUs is suspending your instance. Suspending is similar to closing the lid on your laptop. We generally do not recommend using suspend, but if you opt to use it, we would recommend only using it when you want to conserve some SUs but plan to come back to work with your VM very soon.

openstack server suspend my-server-name-or-UUID

and then resume it again with:

openstack server resume my-server-name-or-UUID

Note that resources are still reserved on the compute host for when you decide restart the instance.

Shelving and unshelving

The last management option is shelving. This shuts down the instance down and moves to storage. Memory state is not maintained. Contents of your root disk are maintained. This is the most economical state for an unused VM as there are no SU charges for shelved VMs since it removes the VM from the hypervisor entirely. We recommend using shelving when you are done with a VM for multiple days or weeks.

openstack server shelve my-server-name-or-UUID

and unshelve it with:

openstack server unshelve my-server-name-or-UUID

Deleting Infrastructure in the CLI

When you're done with a VM, you'll want to remove the floating IP and return it to the pool if you aren't going to reuse it. Then you can delete an instance you no longer need.

We're providing instructions for deleting routers, subnets, and networks, as well, though you can keep those and reuse them. Network infrastructure does not incur any SU charges to leave though it will count against Openstack resource quotas.

There are a very limited number of IPs available. Please return any you are not using!

Clean Up Instance

Remove the public IP address from the instance

openstack server remove floating ip your-unneeded-instance your.ip.number.here

Return the public IP address to the pool of IP numbers

openstack floating ip delete your.ip.number.here

Delete an instance

openstack server delete your-unneeded-instance

Note: You often want to create infrastructure such as networks, subnets, routers, security groups/rules, etc. only once and not delete them.

These entities are reusable by all members of your project except for any keypairs you've created.

Deleting Network Infrastructure

Disconnect the router from the gateway

openstack router unset --external-gateway my-router-name

Delete the subnet from the router

openstack router remove subnet my-router-name my-subnet-name

Delete the router

openstack router delete my-router-name

Delete the subnet

openstack subnet delete my-subnet-name

Delete the network

openstack network delete my-network-name

Deleting Security Infrastructure

Delete a security group rule

openstack security group delete secgroup-rule-name

Delete a security group

openstack security group delete secgroup-name

Deleting Other Items

Deleting a keypair

openstack keypair delete my-keypair-name

Using Storage Under the CLI

All allocations receive 1 TB of storage alloation by default. You can use this storage quota in the form of volumes or manila shares. MANILA LINK

All of the commands below assume that you have an Openstack Client installed and a working openrc file.

To view any volumes you might have:

openstack volume list

To create a 10 GB volume, you can do:

openstack volume create --size 10 my-10GVolume

Then you can attach it to an instance for use:

openstack server add volume VM-Name-Or-UUID Volume-Name-Or-UUID

While you can usually assume it will be the next mounted disk (root should be /dev/sdaX in all cases on Jetstream2), you can look on your instance to see where the volume attached by doing:

dmesg | grep sd

The output of that should usually look something like this:

    [root@my-vm ~]# dmesg | grep sd
    [    1.715421] sd 2:0:0:0: [sda] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB)
    [    1.718439] sd 2:0:0:0: [sda] Write Protect is off
    [    1.720066] sd 2:0:0:0: [sda] Mode Sense: 63 00 00 08
    [    1.720455] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    1.725878]  sda: sda1
    [    1.727563] sd 2:0:0:0: [sda] Attached SCSI disk
    [    2.238056] XFS (sda1): Mounting V5 Filesystem
    [    2.410020] XFS (sda1): Ending clean mount
    [    7.997131] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    [    8.539042] sd 2:0:0:0: Attached scsi generic sg0 type 0
    [    8.687877] fbcon: cirrusdrmfb (fb0) is primary device
    [    8.719492] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device
    [  246.622485] sd 2:0:0:1: Attached scsi generic sg1 type 0
    [  246.633569] sd 2:0:0:1: [sdb] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
    [  246.667567] sd 2:0:0:1: [sdb] Write Protect is off
    [  246.667923] sd 2:0:0:1: [sdb] Mode Sense: 63 00 00 08
    [  246.678696] sd 2:0:0:1: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [  246.793574] sd 2:0:0:1: [sdb] Attached SCSI disk

From your instance, you can now create a mount point, view the device, and create the filesystem (using ext4 as the filesystem type for this example).

mkdir /vol_b
fdisk -l /dev/sdb
mkfs.ext4 /dev/sdb

If you get the following warning, it's safe to hit 'y' to proceed.

/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n)

mount -o noatime /dev/sdb /vol_b

Assuming you didn't get any errors, /dev/sdb should now be mounted on /vol_b

Note: Linux has a special mount option for file systems called noatime. If this option is set for a file system in /etc/fstab, then reading accesses will no longer cause the atime information (last access time - don't mix this up with the last modified time - if a file is changed, the modification date will still be set) that is associated with a file to be updated (in reverse this means that if noatime is not set, each read access will also result in a write operation). Therefore, using noatime can lead to significant performance gains.
    [root@my-vm ~]# touch /vol_b/foo
    [root@my-vm ~]# ls -la /vol_b/
    total 24
    drwxr-xr-x   3 root root  4096 Jul 13 13:37 .
    dr-xr-xr-x. 18 root root  4096 Jul 13 11:50 ..
    -rw-r--r--   1 root root     0 Jul 13 13:37 foo
    drwx------   2 root root 16384 Jul 13 13:36 lost+found
    
    [root@my-vm ~]# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sda1       8.0G  2.4G  5.7G  30% /
    devtmpfs        902M     0  902M   0% /dev
    tmpfs           920M     0  920M   0% /dev/shm
    tmpfs           920M   17M  904M   2% /run
    tmpfs           920M     0  920M   0% /sys/fs/cgroup
    tmpfs           184M     0  184M   0% /run/user/0
    /dev/sdb        9.8G   37M  9.2G   1% /vol_b

To make the volume mount persist, you can add an entry to /etc/fstab similar to this:

/dev/sdb /vol_b ext4 defaults,noatime 0 0

You would need to change as needed for a different device id, mount point, and file system type. We do recommend using the noatime option as shown in the example.

Once you are done with your volume or want to use it with another VM, if you are not shutting down the VM, you'll need to unmount it.

umount /vol_b

To detach it from the VM, you'll do:

openstack server remove volume VM-Name-Or-UUID Volume-Name-Or-UUID

Doing an openstack volume list now should show the volume as available:

    +--------------------------------------+------------------+------------+------+------------------------------+
    |  ID                                  |   Display Name   |  Status    | Size | Attached to                  |
    +--------------------------------------+------------------+------------+------+------------------------------+
    | af59d4fa-ced2-4049-a062-7b2a15807b7f | my-10GVolume     | available  |  10  |                              |
    +--------------------------------------+------------------+------------+------+------------------------------+

If you want to completely destroy a volume, you can do:

openstack volume delete Volume-Name-Or-UUID

Manila Shares in the CLI

To use Manila via Openstack CLI

On a terminal that has the Openstack Clients installed and the appropriate login credentials you can do the following:

1. Create a Share

openstack share create --name $manila-share-name cephfs $vol-size --os-share-api-version 2.63

Example:

openstack share create --name manila-share-cli cephfs 10 --os-share-api-version 2.63

Metadata for the share created above:

+---------------------------------------+--------------------------------------+
| Field                                 | Value                                |
+---------------------------------------+--------------------------------------+
| access_rules_status                   | active                               |
| availability_zone                     | None                                 |
| create_share_from_snapshot_support    | False                                |
| created_at                            | 2022-03-01T03:40:48.468421           |
| description                           | None                                 |
| has_replicas                          | False                                |
| id                                    | 23c511b2-66e7-4986-b6a6-231b490210d4 |
| is_public                             | False                                |
| metadata                              | {}                                   |
| mount_snapshot_support                | False                                |
| name                                  | manila-share-cli                     |
| progress                              | None                                 |
| project_id                            | 55d7efb46dd945a2b86f7ce8aa657e1a     |
| replication_type                      | None                                 |
| revert_to_snapshot_support            | False                                |
| share_group_id                        | None                                 |
| share_network_id                      | None                                 |
| share_proto                           | CEPHFS                               |
| share_type                            | de7b9e68-2357-4837-880f-858d7358b05c |
| share_type_name                       | cephfsnativetype                     |
| size                                  | 10                                   |
| snapshot_id                           | None                                 |
| snapshot_support                      | False                                |
| source_share_group_snapshot_member_id | None                                 |
| status                                | creating                             |
| task_state                            | None                                 |
| user_id                               | a9e55b395bcb494aaf5938f5f8382e71     |
| volume_type                           | cephfsnativetype                     |
+---------------------------------------+--------------------------------------+

id is the share_id. You can use this to look up information about your share. See step 4.

2. Create an Access Rule

openstack share access create $manila-share-name cephx $anyName --os-share-api-version 2.63

Example:

openstack share access create manila-share-cli cephx manilaAccess --os-share-api-version 2.63

Metadata for the access rule:

+--------------+--------------------------------------+
| Field        | Value                                |
+--------------+--------------------------------------+
| id           | 95067b4f-f77c-4b76-be12-ac5c3a8e8897 |
| share_id     | 23c511b2-66e7-4986-b6a6-231b490210d4 |
| access_level | rw                                   |
| access_to    | manilaAccess                         |
| access_type  | cephx                                |
| state        | queued_to_apply                      |
| access_key   | None                                 |
| created_at   | 2022-03-01T13:41:09.984126           |
| updated_at   | None                                 |
| properties   |                                      |
+--------------+--------------------------------------+

Make a note of the id value. This is the access rule id. In the above example it is 95067b4f-f77c-4b76-be12-ac5c3a8e8897. You can look up the access rule id in openstack to get your access_key.

3. Get Access Key

openstack share access show $access-rule-id --os-share-api-version 2.63

Example:

openstack share access show 95067b4f-f77c-4b76-be12-ac5c3a8e8897 --os-share-api-version 2.63

Metadata for the access rule:

+--------------+------------------------------------------+
| Field        | Value                                    |
+--------------+------------------------------------------+
| id           | 95067b4f-f77c-4b76-be12-ac5c3a8e8897     |
| share_id     | 23c511b2-66e7-4986-b6a6-231b490210d4     |
| access_level | rw                                       |
| access_to    | manilaAccess                             |
| access_type  | cephx                                    |
| state        | active                                   |
| access_key   | AQB2Ih5iyQKpChAAIKunDZ6Ztr1VfNn+AFxlGA== |
| created_at   | 2022-03-01T13:41:09.984126               |
| updated_at   | 2022-03-01T13:41:10.227543               |
| properties   |                                          |
+--------------+------------------------------------------+

The access rule is active and you can use the access_key generated above.

4. View Share Information

openstack share show $share_id --os-share-api-version 2.63

Example:

openstack share show 23c511b2-66e7-4986-b6a6-231b490210d4 --os-share-api-version 2.63

Metadata for your share will now look a little different:

+---------------------------------------+----------------------------------------------------------------------+
| Field                                 | Value                                                                |                                                                 |
+---------------------------------------+----------------------------------------------------------------------+
| access_rules_status                   | active                                                               |
| availability_zone                     | nova                                                                 |
| create_share_from_snapshot_support    | False                                                                |
| created_at                            | 2022-03-01T03:40:48.468421                                           |
| description                           | None                                                                 |
| export_locations                      |                                                                      |
|                                       | id = 1138760f-2ba4-4d9f-ad8b-312d39c4e4b8                            |
|                                       | path = 149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,  |
|                                       | 149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/1ca2d54e-  |
|                                       | 16a5-43b8-90de-75a91c1b96e9/fba3f935-5047-4eef-8b4d-3c27f356c2c7     |
|                                       | preferred = False                                                    |
| has_replicas                          | False                                                                |
| id                                    | 23c511b2-66e7-4986-b6a6-231b490210d4                                 |
| is_public                             | False                                                                |
| mount_snapshot_support                | False                                                                |
| name                                  | manila-share-cli                                                     |
| progress                              | 100%                                                                 |
| project_id                            | 55d7efb46dd945a2b86f7ce8aa657e1a                                     |
| properties                            |                                                                      |
| replication_type                      | None                                                                 |
| revert_to_snapshot_support            | False                                                                |
| share_group_id                        | None                                                                 |
| share_network_id                      | None                                                                 |
| share_proto                           | CEPHFS                                                               |
| share_type                            | de7b9e68-2357-4837-880f-858d7358b05c                                 |
| share_type_name                       | cephfsnativetype                                                     |
| size                                  | 10                                                                   |
| snapshot_id                           | None                                                                 |
| snapshot_support                      | False                                                                |
| source_share_group_snapshot_member_id | None                                                                 |
| status                                | available                                                            |
| task_state                            | None                                                                 |
| user_id                               | a9e55b395bcb494aaf5938f5f8382e71                                     |
| volume_type                           | cephfsnativetype                                                     |
+---------------------------------------+----------------------------------------------------------------------+

You will need the path of your export_locations. In the above example it is:

149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/1ca2d54e-16a5-43b8-90de-75a91c1b96e9/fba3f935-5047-4eef-8b4d-3c27f356c2c7

Important things to note down:

  • Share id (Step 1)
  • Access rule id (Step 2)
  • Acccess key (Step 3)
  • Export location path (Step 4)

This is the same whether you're using Horizon or the CLI. Please refer to Configuring a VM to use Manila Shares

CLI Troubleshooting

I get an error requesting scope when authenticating and doing openstack commands

If you source your openrc in a session that's had another openrc sourced from another cloud (inluding Jetstream1), you'll get an error like this:

Error authenticating with application credential: Application credentials cannot request a scope. (HTTP 401) (Request-ID:

Please make sure to source the new Jetstream2 openrc in a fresh terminal session.

If you're feeling adventurous, you can do something like this to remove all previous OpenStack environment variables:

while read -r varname; do unset "$varname"; done < <(env | grep ^OS_ | cut -d '=' -f1)

Please do that at your own risk.

I get an error saying my request requires authentication after sourcing my application credential openrc.

If your application credential secret in the openrc contains some punctuation/special characters, you might see an error like this:

The request you have made requires authentication. (HTTP 401) (Request-ID:

For example, if you had an ampersand in your credential password, it may have gotten escaped to &amp; instead of just the ampersand character. The same could happen for less than or greater than signs and potentially other special characters. Double check the openrc to verify that that has not happened.

Security

Firewalls

Jetstream2 staff encourages a defense-in-depth approach to security. This potentially involves several methods of restricting access and securing instances.

Firewalls are not enabled by default on Jetstream2 instances. Depending on the user interface you launched your instance from, you may have different security groups established for your instance. (See What is the default security profile for Jetstream2 VMs? for more information on that.)

We encourage keeping your instances patched, rebooting as needed for any kernel or glibc patches, limiting access to all services as much as possible, utilizing security groups if your interface allows it, and running your own host-based firewall if you're comfortable administering it.

If you are comfortable administering a firewall, we would encourage you to read the following tutorials for their respect Linux variants. Ubuntu's UFW (Uncomplicated FireWall) is very simple to use, though making sure you leave SSH access open is crucial (and often missed by first time UFW users) so you do not lock yourself out of your virtual machine.

Ubuntu 20 and 18

How to Set Up a Firewall with UFW on Ubuntu 20.04 is a good initial tutorial for setting up UFW.

Rocky 8 / Alma 8 / CentOS 7

The Redhat variants are a little less user friendly.

How to Open or close ports in AlmaLinux 8 or Rocky Firewall is a good reference for getting started with firewalld.

How to Set Up a Firewall with FirewallD on CentOS 7 will get you started with FirewallD on CentOS 7.

Storage

Manila: Filesystems-as-a-Service on Jetstream2

Manila is the file share service project for OpenStack. Manila provides the management of file shares, for example NFS and CIFS, as a core service to OpenStack. Manila works with a variety of proprietary backend storage arrays and appliances, with open source distributed filesystems, as well as with a base Linux NFS or Samba server. With Manila, you can have multiple instances on Jetstream2 share a filesystem.

Prereqs: Make sure you have these packages installed on your instance: ceph-commons and ceph-fuse

Follow the links below to use:

Using Manila Share on a VM

Configuring a VM to Use Manila Shares

1. Configuring your Instance

  1. Create the file /etc/ceph.$accessTo.secret and add the accessKey

Example:

/etc/ceph.manilashare.secret

AQAHfhZiwTf/NhAAT5ChE4tDXt3Nq1NyiURbMQ==
  1. Edit /etc/fstab to include the following line:
$path /mnt/ceph ceph name=$accessTo,secretfile=/etc/ceph.$accessTo.secret,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2

$path = ips:ports followed by volume path (/volume/_no-group/...)

Example:

149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/fe4f8ad4-2877-4e23-b5d3-46eb8476750b/ab404bac-9584-45f4-8a34-92dfc61fbb98 /mnt/ceph ceph name=manilashare,secretfile=/etc/ceph/ceph.manilashare.secret,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0   2

3. Mount the Share

Mount the manila share created with the following command: mount -a

If you then run a df -h|grep vol you should see something like this:

149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/fe4f8ad4-2877-4e23-b5d3-46eb8476750b/ab404bac-9584-45f4-8a34-92dfc61fbb98    1.8T  134G  1.7T   2% /mnt/ceph

Using the Object Store

The object store is presently experimental. It WILL be a production service in coming weeks/months, but is under active development presently. The object store is only available via Horizon and the CLI presently.

Documentation will continue to evolve.

The Jetstream object store utilizes Openstack Swift and is S3 compatible. You can utilize it via Horizon or the command line interface (CLI). From the CLI, you can use the python-swiftclient or the aws s3api or compatible tools.

Horizon instructions will be coming soon. Though it does not appear that you can generate the EC2 credentials in Horizon.

Prerequisites

If the you want to use s3 compatibility, you'll need to generate EC2 credentials. This assumes you already have the python-openstackclient installed and have generated an application credential for the CLI.

If you do not have an application credential openrc and CLI clients installed, please see these pages:

Optionally, you'll want an AWS s3api client like the AWS command line interface reference client.

Using the object store with s3api compatibility

Once you have sourced your application credential based openrc and installed the python-openstack and python-swiftclient client you will need to generate your ec2-style credentials.

The CLI command is:

openstack ec2 credentials create

You can save the creds in a config to use from the CLI/programatically. It's generally kept in the text file ~/.aws/credentials and looks like this:

[default]
region=RegionOne
aws_access_key_id=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
aws_secret_access_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

For your ~/.aws/credentials, make sure the value from the credential create under "Access" matches "aws_access_key_id" and the Secret matches "aws_secret_access_key".

You can retrieve your ec2 credentials later, as well, by doing

openstack ec2 credentials list
openstack ec2 credentials show *access_key_value*

The endpoint you'll need for S3 operations is https://js2.jetstream-cloud.org:8001/

For s3 style operations, you'll want to use the s3api functions. Those are documented here: https://docs.aws.amazon.com/cli/latest/reference/s3api/

Using the aws command line interface you can test operations.

aws s3api --endpoint-url "https://js2.jetstream-cloud.org:8001/" create-bucket --bucket my-unique-bucket-name

and to add a file to the bucket:

aws s3api --endpoint-url "https://js2.jetstream-cloud.org:8001/" put_object --bucket my-unique-bucket-name --key my-file.zip --body my-file.zip

and to see bucket contents:

aws s3api --endpoint-url "https://js2.jetstream-cloud.org:8001/" list-objects --bucket my-unique-bucket-name

Trying the object store from the CLI using Swift

To use the OpenStack CLI natively with the object store, you'll need the Swift client if you have not already installed it. You can install it by doing:

pip install python-swiftclient

Once you have the Swift client installed, you can test it by doing:

swift post my-unique-bucket-name

which will create a storage container called "my-unique-bucket-name". You can then list your buckets by doing:

swift list

If you want to delete the test bucket, you can do:

swift delete my-unique-bucket-name

As with all Openstack clients, you can see the full list of commands with

swift help

You can also add, remove, and otherwise work with swift containers (buckets in the S3 vernacular) in Horizon on the Project -> Object Store -> Containers tab.

Transferring Files

Coming Soon!!!

Maintenance and Administration

Step-by-step Guide

These steps will let the user that you create ssh to a running instance using a password you set. The user can reset the password once they login and/or add their ssh keys.

Step-by-step guide

All steps to be run as root or using sudo. should be replaced with an actual username

  1. Sudo or otherwise become root: sudo su -
  2. Create the user: adduser <username>
  3. Assign a temporary password: passwd <username>

    As an optional step, you can add the user to any other groups as needed. We'll use the docker group as an example.

  4. Add the user to group "docker" : usermod -a -G docker <username>

Research Software

Jupyter on Jetstream

Coming Soon!

Galaxy

Coming Soon!

Software Collection

Details on the Jetstream2 Software Collection will be forthcoming. At this time, we anticipate a shared disk that will be automatically mounted by instances. Researchers will then be able to use modules to load any combination of software they need, similar to how HPC systems do it.

This list will be ever evolving, but at this time, pre-launch, we anticipate the following software on the shared software store:

  • Matlab
  • R / R Studio / Shiny
  • Intel Compiler
  • NVIDIA (formerly PGI) Compiler
  • AOCC Compiler
  • Anaconda
  • Databases (e.g. Mysql, Postgresql, Mongo)
  • Java
  • Jupyter/JupyterLab/JupyterHub

Items like CUDA and the NVIDIA HPC SDK are not installed on the VMs or in the software collection. We recommend using the NVIDIA containers for GPU-related software from NVIDIA as the ecosystem can often be problematic for individual local installations. The NVIDIA Container Toolkit is installed on all featured images.

This is still an ongoing effort so this list will almost certainly change in coming weeks and months.

Using the Jetstream2 Software Collection

Jetstream2 utilizes Lmod to load and unload software you may need in your research or education pursuits. The software is served from a shared store utilizing Manila shares that are mounted automatically if you're using a Jetstream2 Featured Image.

All of the instructions on listing, loading, and unloading modules/software packages assume you are using a Jetstream2 featured image created after April 9, 2022.

[Listing available software from the collection]

You can use module avail to show available software packages. This may show multiple versions of the same package in the future, which will default to the latest version but allow you to load previous versions.

[exouser@rocky-lmod-host ~]$ module avail

---------------- /software/r8/modulefiles ------------------
  R/4.1.2 (L)    intel-OneAPI/2022.1.2       matlab/R2021a

---------------- /usr/share/lmod/lmod/modulefiles/Core -----
  lmod    settarg

  Where:
  L:  Module is loaded

[Loading software from the collection]

You can load software from the collection using module load.

[exouser@rocky-lmod-host ~]$ module load R

You can also explicitly load a version if there is more than one:

[exouser@rocky-lmod-host ~]$ module load R/4.1.2

[See what modules are loaded]

You can use module list to show what's loaded:

[exouser@rocky-lmod-host ~]$ module list

Currently Loaded Modules:
  1) R/4.1.2   2) intel-OneAPI/2022.1.2

[Saving your loaded software from the collection]

You can save the current loaded modules so it will automatically load every time you log in to your virtual machine by using module save. This will save to a file called default in an ~/.lmod.d directory that Lmod will create the first time you save your loaded module list.

You'll need to do this on any new virtual machine.

[Unloading software from the collection]

You can also unload any loaded modules by using module unload.

[exouser@rocky-lmod-host ~]$ module list

Currently Loaded Modules:
  1) R/4.1.2   2) intel-OneAPI/2022.1.2

[exouser@rocky-lmod-host ~]$ module unload R

[exouser@rocky-lmod-host ~]$ module list

Currently Loaded Modules:
  1) intel-OneAPI/2022.1.2

[exouser@rocky-lmod-host ~]$
This is an advanced topic for researchers and educators that are more comfortable with the system administration aspects of Linux. We generally advise using the Featured Images that have the mounts and Lmod already configured for you.

Do the following as the root user:

Create the file /etc/ceph.js2softwarero.secret with the contents

AQAYuStiw81/ABAAsvzI5h53WOC5K1vzmGB66g==

Make the mount point you want to use (we highly suggest it be /software)

mkdir /software

Add the following entry to the end of your /etc/fstab (make sure the mountpoint you created in the last step matches)

# JS2 Software Collection mount 149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/b7112570-f7cb-4bd2-8c0e-39b08609b9fd/01aa9d72-69bf-4250-9245-2eaddcdb251d /software ceph name=js2softwarero,secretfile=/etc/ceph.js2softwarero.secret,x-systemd.device-timeout=31,x-systemd.mount-timeout=30,noatime,_netdev,ro 0

Then mount the filesystem with

mount -a

If you then run a df -h|grep vol you should see something like this:

149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/fe4f8ad4-2877-4e23-b5d3-46eb8476750b/ab404bac-9584-45f4-8a34-92dfc61fbb98   1.8T  134G  1.7T   2% /mnt/ceph

You'll also need to install Lmod on your VM. The major distributions include this either in default software repositories or in the Powertools repo for CentOS/Alma/Rocky8.

You'll need to point your modulepath config to:

/software/xx/modulefiles

replacing the xx with u20, u18, r8, or c7 depending on the Linux variant you're using.

Software Licenses

Software licenses are generally the responsibility of the user to obtain, coordinate, implement, and maintain.

Some licenses may require users to coordinate with Jetstream staff to allow for proper configuration. Such coordination is handled on a case-by-base basis.

Licensed software maintained by staff but available to users are listed below.

Compilers and Parallel Libraries

  • The Intel Compiler is covered on the primary cloud by a site license and has been configured to point to the proper license server.
  • All components of the Intel Parallel Studio are covered by the IU site license, including the MKL libraries.

Specialty Software

  • MATLAB is covered on the primary cloud by a site license and has been configured to point to the proper license server.

If you need licenses for any other software that is not provided through the Jetstream2 shared software store then you will have to provide your own license. Licenses on regional clouds may vary and will need to be discussed with the regional cloud provider.

Please be aware that if you take an image or software from Jetstream and run it somewhere else then the license will not work. Jetstream license servers restrict access to only valid Jetstream IP addresses.

It is also the responsibility of the PI and those placed on each allocation to ensure that they are complying with the terms of any software license they are using on Jetstream2.

Containers

Containers are isolated environments in which to run your applications. They created by objects called images. An image is defined as a read only template with instructions for creating a container. These instructions define everything a container needs; Software, dependencies, system libraries, environment variables, configuration files, etc.

Some Basic Docker Commands

Give the installed version of Docker.

docker -version

Download the image from dockerhub.

docker pull <image_name>

Run the image pulled from dockerhub to create a container. If you don't have a local copy of the image, the run command will pull and then run the image to create a container.

docker run  <image_name>

Process status of containers. If no container is running, you get a blank line.

docker ps

Process status of all containers.

docker ps -a

Run a command in the docker container. The -it flag provides an interactive tty (shell) within the container.

docker exec -it <container_id> bash

Shut down a computer.

docker stop <container_id>

Build an image from the specified docker file.

docker build <path_to_docker_file>

Push an image to the docker hub repository.

docker push <docker_hub_username/image_name>

Kubernetes

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

Here is some more information on Kubernetes.

Some K8s basics

  1. A Pod is the smallest deployable unit of Kubernetes. A Pod can contain one or more containers on a single IP.
  2. A Node is a physical or virtual machine and its components include kubelet, kube-proxy and Container run-time.
    • Kubelet is the Node agent that manages containers created by Kubernetes.
    • Kube-proxy is used by Service to run a network proxy on each Node.
    • Container runtime is the software responsible for running containers. Ex: Docker, containerd, and CRI-O.
  3. Service allows the use of a stable network address for an application, including a persistent ip address and a local DNS entry within the cluster. Service also acts as a load-balancer across all pods allowing the addition removal of pods from a deployment.
  4. ReplicaSet is a convenient way to build multiple pods at once.
  5. Deployment provides declarative updates for Pods and ReplicaSets (rolling updates).
  6. Control Pane is a node that controls/manages worker nodes. The components of the Control Pane are the API Server, the cluster store, the controller manager, and the scheduler.
  • API server All communication between the various Kubernetes components must go through the API server.
  • Cluster Store or etcd Persistently stores the cluster's configuration, state, and metadata.
  • The Controller Manager runs controller processes that monitor and respond to controller events. Some examples of control loops include; node controller, job controller, end point controller, and replication controller.
  • The Scheduler assigns pods to nodes.

Building a Kubernetes Cluster

A k8s cluster is a collection of nodes running containers. Each node is responsible for running containers that are on that specific node.

In this example, we use the kubeadm tool to set up a cluster. Our cluster has 3 VMs: 1 control plane and 2 worker nodes.

Installation

Create three virtual machines and run the installation commands (1-16) on all three.

  1. Create configuration file for containerd

     
  2. Load modules

    sudo modprobe overlay
    sudo modprobe br_netfilter
  3. Set networking configurations for K8s

     
  4. Apply new settings

    sudo sysctl --system
  5. Install containerd

    sudo apt-get update && sudo apt-get install -y containerd
  6. Create default configuration file for containerd

    sudo mkdir -p /etc/containerd
  7. Generate default containerd configuration and save to the newly created default file

    sudo containerd config default | sudo tee /etc/containerd/config.toml
  8. Restart containerd to ensure new configuration file usage

    sudo systemctl restart containerd
  9. Verify that containerd is running

    sudo systemctl status containerd
  10. Disable swap

    sudo swapoff -a
  11. Install dependency packages

    sudo apt-get update && sudo apt-get install -y apt-transport-https curl
  12. Download and add GPG key

    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
  13. Add Kubernetes to repository list

     
  14. Update package listings

    sudo apt-get update
  15. Install Kubernetes packages

    sudo apt-get install -y kubelet=1.22.0-00 kubeadm=1.22.0-00 kubectl=1.22.0-00
  16. Turn off automatic updates

    sudo apt-mark hold kubelet kubeadm kubectl

Initialize the Cluster

  1. Before initializing the cluster, make sure the Cgroup driver is systemd on all three VMs. You can check that by running the following command: docker info | grep Cgroup

    To change the Cgroup drive to systemd you can do the following:

     
  2. You will need to restart Docker and reset kubeadm for the change in the previous step to take effect.

    sudo systemctl daemon-reload
    sudo systemctl restart docker
    sudo kubeadm reset
  3. Initialize the Kubernetes cluster on the control plane node using kubeadm.

    sudo kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.22.0
  4. Set kubectl access on the control plane.

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  5. On the Control Plane Node, install Calico Networking.

    kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
  6. Test access to cluster.

    kubectl get nodes

Join Worker Nodes to the Cluster

  1. On the Control Plane Node, create the token and copy the kubeadm join command. (Note: The join command can also be found in the output from kubeadm init command).

    kubeadm token create --print-join-command
  2. On the Worker Nodes, paste the kubeadm join command to join the cluster. Use sudo to run it as root.

    sudo kubeadm join ...
  3. On the Control Plane Node, view cluster status. (Note: You may have to wait a few moments to allow all nodes to become ready).

    kubectl get nodes

Clean Up

  1. On the Control Plabe Node, delete a resource using kubectl.

    kubectl delete node

Note

Before using the cluster confirm that all pods in the kube-system are running.

kubectl get pods -n kube-system -o wide

This command will give you a list of pods in the kube-system namespace, the status of the pod, the node it is running on. If the status of a pod is not Running, you can check the logs for error messages.

kubectl logs -n kube-system {name_of_your_pod}

Managing Applications with Kubernetes

To manage applications with Kubernetes we use the apply command. This command requires a file or directory of files. When run, the apply command makes the state of the Kubernetes cluster match the state defined in the file(s).

Using the Kubernetes CLI, (Kubectl), we can create objects such as Pods, Deployments. etc. by providing a yaml file for that object.

Pods

A Pod represents a single instance of an app running in the cluster.

Here's an example of a yaml file that defines a pod named, simple-pod.yaml The kind field describes the type of object you want to create. The pod spec must contain at least one container. Image specifies which image will be run in the pod. Finally, we list the port to expose from the container.

yaml manifest:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-deployment
  label:
  app: nginx  
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

To create the pod defined in the yaml file above, run the following command:

kubectl apply -f simple-pod.yaml

command line:

kubectl run nginx-deployment --image=nginx --port=80

ReplicaSet

ReplicaSet adds or deletes pods as needed. Creating replicas of a pod scales an application horizontally. Replicas are usually created as part of a deployment. Here's a sample yaml file for creating 2 replicas, to create a ReplicaSet without a deployment.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-replicaset
spec: 
  replicas: 2
  selector:
    matchLabels:
     app: nginx
spec: 
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    -containerPort: 80

command line:

kubectl scale --replicas=2 deployment nginx-deployment

Deployment

Deployment is an object that can provide updates to both pods and ReplicaSets. Deployment object allows you to do rolling updates of a pod, ReplicaSet object does not. A rolling update scales up the new version to the appropriate number of replicas and scales down the old version to zero.

yaml manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec: 
  replicas: 3
  selector:
    matchLabels:
     app: nginx
spec: 
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    -containerPort: 80

command line:

kubectl create deployment nginx-deployment --image=nginx

Service

Service enables network access from either within the cluster or between external processes.

kind: Service
metadata:
  name: nginx-ingress
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app: nginx

command line:

   kubectl create service nodeport nginx-deployment --tcp=80:80

Autoscaling

A Horizontal Pod Autoscaler (HPA) allows you to scale up or down depending on traffic. This can be configured by specifying the CPU or memory states. The master node periodically checks to see if the desired state is met and scales up or down as needed.

One way to do this is to enable autoscaling in a yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  maxReplicas: 10
  minReplicas: 5
  scaleTargetRef: 
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment
   targetCPUUtilizationPercentage: 10
...
...

command line:

kubectl autoscale deploy nginx-deployment --min=5 --max=10 --cpu-percent=50

Persistent Volumes in Kubernetes

Container filesystems are ephemeral. One way to persist data beyond the lifetime of a pod is by using volumes. PersistentVolume is storage external to the Container but within the same Pod spec.

Creating persistent storage requires the following definitions:

  • StorageClass
  • PersistentVolume
  • PersistentVolumeClaim
  • Pod that uses the PersistentVolumeClaim for storage

Here are some sample definition files.

StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: localdisk
provisioner: kubernetes.io/no-provisioner

PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: alpine-pv
  namespace: default
spec:
  storageClassName: localdisk
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /var/output

PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: alpine-pv-claim
spec:
  storageClassName: localdisk
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi

Pod

apiVersion: v1
kind: Pod
metadata:
  name: alpine
  namespace: default
spec:
  volumes:
    - name: alpine-volume
      persistentVolumeClaim:
         claimName: alpine-pv-claim
  containers:
    - image: alpine:3.2
      command: ['sh', '-c', 'while true; do echo This is a test! > /output/trial_run.txt; sleep 20; done']
      imagePullPolicy: IfNotPresent
      name: alpine
      volumeMounts:
         - name: alpine-volume
           mountPath: /output

The steps to create and use volumes are below:

  1. On the Control Plane, create the StorageClass, PersistentVolume, PersistentVolumeClaim, and Pod with the kubectl create command.
  2. Check if your pod is running and which worker node it is running on:

    The -o wide below list the node the pod is running on.

    kubectl get pods -o wide
  3. Log into the worker node and verify that /var/output/trial_run.txt exists and is populated.
  4. If you delete and recreate the pod, the volume will automatically bind to the new pod with the trial_run.txt file.

Minikube

Minikube allows you to run a Kubernetes cluster locally.

Installation

  1. Install Minikube on your Virtual Machine:

    Note: If you have VirtualBox installed, you can specify vm-driver=VirtualBox

    curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
    sudo install minikube-linux-amd64 /usr/local/bin/minikube
    sudo minikube config set vm-driver none
  2. Install kubectl:

    curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
    chmod +x kubectl
    sudo mv ./kubectl /usr/local/bin/kubectl
  3. Start Minikube:

    minikube start
  4. You can check your kubectl installation running the following command:

    kubectl get pods --all-namespaces

    This should list all the kube-system pods currently running on the machine.

  5. Create a Deployment

    kubectl create deployment --image nginx my-nginx
    kubectl expose deployment my-nginx --port=80 --type=NodePort
    kubectl get svc
  6. Start Dashboard

    To view your minikube dashboard, run the following command.

    minikube dashboard --url &

    This might throw an error depending on what's installed on your machine. To access your dashboard you need to run Kubeproxy.

  7. Kubeproxy

    kubectl proxy --address=0.0.0.0 --accept-hosts='.*'
  8. View Minikube Dashboard

    To view your minikube dashboard from your web browser, run the following command (replace host_ip with the ip address of your VM):

    http://{host_ip}:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
  9. Delete Cluster

    minikube delete

Terraform on Jetstream2

Terraform is a open-source code as infrastructure tool that works with Jetstream2 and allows you to build and destroy infrastructure quickly and easily.

Installing Terraform

The documentation to install Terraform is located below. We have linked to the latest documentation from the Terraform team.

https://www.terraform.io/downloads

Basic Operations

The basic setup is as follows:

  1. Git clone the terraform repo
  2. Download the openrc file into the repo
  3. Run source openrc.sh
  4. Initalize Terraform with terraform init
  5. See what changes Terraform wants to make to your infrastructure with terraform plan
  6. Apply the changes with terraform apply

Getting started with git

If you are unfamiliar with git, please see the git-scm documentation.

Git clone

Git clone allows you create a copy of a git repo that creates the directory and checks out the active branch.

git clone git@github.iu.edu:wellsaar/terraform-jetstream.git

This command creates a clone of the git repo that contains the code examples that we use below. The master branch of the git repo above should be a good starting point in developing your own terraform code.

Running Terraform

Make sure that you have setup and downloaded your openrc file. Setting up the openrc.sh for the Jetstream2 CLI.

You will need to source the openrc file once you have downloaded it in order to setup the variables that Terraform will need to connect to Jetstream2.

Terraform Init

terraform init

This is the first command that should be run after writing a new Terraform configuration or cloning an existing one. This command is used to initialize a working directory containing Terraform configuration files.

Please see Terraform Init Documentation for further information.

Example Terraform Init

terraform-jetstream git: (jetstream2) x terraform init
Initializing the backend..
Initializing provider plugins.

- Reusing previous version of terraform-provider-openstack/openstack from the dependency lock file
- Using previously-installed terraform-provider-openstack/openstackv1.42.0
Terraform has been successfully initialized!
* You may now begin working with Terraform. Try running
"terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
* If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

In this example, we have already initialized Terraform in this directory, and Terraform is checking the dependency lock file for any updates. You will need to run Terraform init if you make any changes to providers.

Terraform Plan

terraform plan

This command creates an execution plan which allows you to preview the changes that Terraform will change. The screenshot below is only a small section of output of a plan.

Please see Terraform Plan Documentation for further information.

Example Terraform Plan

# openstack_networking_subnet_v2.terraform_subnet1 will be created
+ resource "openstack_networking_subnet_v2" "terraform_subnet1" {
    + all_tags          = (known after apply)
    + cidr              = "192.168.0.0/24"
    + enable_dhcp       = true
    + gateway_ip        = (known after apply)
    + id                = (known after apply)
    + ip_version        = 4
    + ipv6_address_mode = (known after apply)
    + ipv6_ra_mode      = (known after apply)
    + name              = "terraform_subnet1"
    + network_id        = (known after apply)
    + no_gateway        = false
    + region            = (known after apply)
    + tenant_id         = (known after apply)

    + allocation_pool {
        + end   = (known after apply)
        + start = (known after apply)
      }

    + allocation_pools {
        + end   = (known after apply)
        + start = (known after apply)
      }
  }

Plan: 8 to add, 0 to change, 0 to destroy.

Changes to Outputs:
+ floating_ip_ubuntu20 = [
    + (known after apply),
  ]

────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the `-out` option to save this plan, so Terraform can't guarantee to take exactly these actions if you run `terraform apply` now.

In this example, due to space issues only the networking section is shown. The resource section in the example lists how the resource will be configured, including any tags and the name of the resource.

You can see on the Plan: line that 8 items will be added, 0 will be changed and 0 will be destroyed.

Terraform Apply

terraform apply

This command makes changes based on what's defined in the Terraform files and what's existing in your infrastructure already.

Please see Terraform Apply Documentation for further information.

Example Terraform Apply

# openstack_networking_subnet_v2.terraform_subnet1 will be created
+ resource "openstack_networking_subnet_v2" "terraform_subnet1" {
    + all_tags          = (known after apply)
    + cidr              = "192.168.0.0/24"
    + enable_dhcp       = true
    + gateway_ip        = (known after apply)
    + id                = (known after apply)
    + ip_version        = 4
    + ipv6_address_mode = (known after apply)
    + ipv6_ra_mode      = (known after apply)
    + name              = "terraform_subnet1"
    + network_id        = (known after apply)
    + no_gateway        = false
    + region            = (known after apply)
    + tenant_id         = (known after apply)

    + allocation_pool {
        + end   = (known after apply)
        + start = (known after apply)
      }

    + allocation_pools {
        + end   = (known after apply)
        + start = (known after apply)
      }
  }

Plan: 8 to add, 0 to change, 0 to destroy.

Changes to Outputs:
+ floating_ip_ubuntu20 = [
    + (known after apply),
  ]

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

Writing and Validating YAML Files

Terraform code is written in YAML (YAML Ain't Markup Language). Indentation matters with YAML so listed below are some free tools that will help you write in YAML.

Virtual Clusters on Jetstream2

A Virtual Cluster is a computer cluster that is made up of a single head node which runs the Slurm system, as well as a number of compute nodes. The head node is responsible for controlling the compute nodes. A large filesystem can be shared between all of the nodes. The head node is accessible to users via a public IP address and the compute nodes are created on-demand as jobs are submitted to the queue.

Slurm is a resource management system that helps organize and manage computer resources. It allocates resources to different tasks or jobs, making sure that everything runs as smoothly as possible.

Options to Create Your Own Virtual Cluster on Jetstream2