Containers at XSEDE
Last update: October 13, 2021
Containers are a standard unit of software that package up code and all its dependencies so that an application can run quickly and reliably on different computing platforms. There are a number of types of containers but the 2 widely used versions are Docker and Singularity. The differences between them are subtle but distinctive.
Singularity is a runtime container, like Docker, but doesn't require the specialized privileges that Docker requires. As such, many sites find Singularity a useful tool not only for administrators, but users as well.
Each XSEDE site offers different container related resources. Cloud resources like Jetstream focus specifically on providing users with container resources. Other sites and resources may have different focus areas but still provide access to containers.
As mentioned above, Docker and Singularity are the two most common types of containers. Some XSEDE sites discourage usage of Docker as it requires users to have escalated root privileges. Another popular container tool, Kubernetes is available at most XSEDE sites, but is more widely used on Jetstream which provides users assistance using the tool.
Several container tutorials were developed and presented at various conferences. The XSEDE Container Tutorial is available on GitHub.
Each site's area of expertise and offering for Containers is listed below:
Basic Docker container support is provided through pre-installed Docker and Docker Compose packages, available in many of the staff-maintained and featured CentOS 7, CentOS 8, Ubuntu 16, Ubuntu 18, and Ubuntu 20 images. Users of Jetstream's Atmosphere management interface have additional easy-to-use built-in command line options to quickly install Docker and Singularity on any image. Since Jetstream is a cloud provider, most container technologies are available – users get root access so they can use Docker, Singularity, CharlieCloud, Shifter, or whatever else they may need.
Other Helpful Links:
- Jetstream Atmosphere EZ tools
- Container Orchestration on Jetstream
- Container Orchestration Using the OpenStack Horizon GUI Interface
- Deploy scalable Jupyterhub with Kubernetes on Jetstream
- Deploy Kubernetes and JupyterHub on Jetstream with Magnum
- Kubeadm Bootstrapper
- Jetstream Public Wiki
The XSEDE link to Jetstream is Jetstream User Guide.
PSC Supports Singularity containers only. PSC offers a variety of pre-built containers for users, or users are welcome to build and deploy their own. This can be done through a reservation, the ‘interact' mode, or by requesting a VM and running containers on that VM.
SDSC continues to support Singularity-based containerization on Expanse. Users can search for available software on XSEDE resources with the XSEDE software search tool.
TACC offers training for Singularity containers - including Docker containers converted to Singularity on their systems (including Stampede2), curated container images for developers to use as a base container image to build their application to run at scale on various TACC systems, and documentation on how and where to use and interact with these base images.
TACC-supported images are specifically not portable, contrary to typical containerization philosophy, due to the added hardware-specific drivers needed for TACC systems.
A Quick Start guide can help users become familiar with Singularity and containers. One offered by Singularity is available here. Singularity Quick Start Guide
See the following web site for more information about public container registries: https://software.xsede.org/tiny-registries
Users needing assistance finding and using container software, services, training, or needing lightweight consulting may contact the XSEDE help desk at email@example.com. The Novel and Innovative Projects (NIP) team can provide this type of lightweight assistance. XSEDE allocated projects needing more in depth consulting on data preparation, data flows, data analytics, data locality, data visualization and other data techniques can request Extended Collaborative Support Services (ECSS) assistance. The Novel and Innovative Projects (NIP) area of ECSS provides lightweight consulting to assist in this process. This can be requested via firstname.lastname@example.org.