Abstract

Containers have transformed application deployment efficiency, moving from cloud environments into high‑performance computing (HPC) systems. By bundling applications with their dependencies inside isolated environments, containers improve compatibility and portability. In HPC, security constraints often restrict user‑level customisation, leading to containers carrying a heavy set of optimised libraries that trade portability for hardware performance. Cloud containers, by contrast, are lighter and more portable. While cloud systems incorporate sophisticated orchestration mechanisms, HPC systems have traditionally lacked such capabilities, though recent research proposes solutions for container orchestration on HPC. This paper surveys and taxonomises containerisation efforts and orchestration strategies on HPC platforms, highlights differences from cloud computing, and discusses open challenges and potential research directions.

Introduction

Containerisation encapsulates applications together with their dependencies, ensuring environment compatibility and easing deployement across clusters. Unlike virtual machines, containers share the host kernel, making them lightweight and fast. HPC systems typically run large‑scale simulations requiring low latency and high throughput. Users cannot install new software freely due to security and stability concerns, and the traditional module environment is cumbersome for dynamic AI workloads. Containers enable custom environments and support research reproducibility. However, HPC lacks efficient container scheduling, load balancing, auto‑scaling, and integrated environment provisioning. This work surveys state‑of‑the‑art containerisation and orchestration for HPC, compares them with cloud approaches, and discusses challenges and future opportunities.

The paper is organised as follows: Section II covers container concepts and techniques; Section III examines leading container engines for HPC; Section IV reviews orchestration strategies in both HPC and cloud; Section V discusses challenges and outlook; Section VI concludes.

Containerisation Concepts and Techniques

Containerisation is operating‑system‑level virtualisation that provides isolated execution environments. A container is a runnable instance of an image that bundles the program, libraries, data, and configuration files. Containers offer reproducibility: once defined, the environment remains unchanged regardless of where it runs. However, sharing the host kernel limits cross‑platform execution (e.g., Windows containers cannot run on Unix). HPC applications are often heavily optimised for specific hardware, forcing a trade‑off between performance and portability. Benchmarks show containers can achieve near‑native performance.

Namespaces and Resource Isolation

Linux namespaces isolate kernel resources: mount, PID, UTS, network, IPC, and user. Docker, one of the most widely used container engines, leverages namespaces and cgroups for isolation and resource limits. Docker uses three network types: host, bridge (default), and none. A software bridge enables communication among containers on the same host; cross‑host communication requires port mapping. On cloud systems, users typically have root privileges, making Docker adoption straightforward, but Unix sockets can become a security risk if exposed via TCP.

HPC Container Engines and Runtimes

Several container engines designed specifically for HPC systems address the limitations of Docker (e.g., root‑less operation, native MPI/GPU support, and integration with workload managers).

Shifter

Developed at NERSC, Shifter flattens Docker images into ext4 or squashfs images, which are replicated to parallel filesystems. It runs with user‑level privileges and supports MPI. GPU drivers are replaced at runtime with ABI‑compatible versions to maintain portability.

Charliecloud

Charliecloud converts Docker images into tar files and unpacks them on HPC nodes without requiring root. It supports MPI and injects host files (e.g., GPU libraries) to resolve compatibility issues.

Singularity

Designed explicitly for HPC, Singularity runs with user privileges and no daemon. Root is only required for building images. It natively supports GPUs, MPI, and InfiniBand. Singularity uses a single SIF image file for portability. MPI applications can be run using a hybrid model (MPI binaries and libraries inside the container) or a bind model (mounting host MPI libraries). The hybrid model is recommended becuase mounting volumes often requires privileged operations. Singularity can directly convert most Docker images and has become the de facto standard container engine in HPC.

SARUS

SARUS relies on runc for container instantiation. It consists of an image manager and a runtime component invoked via CLI. The runtime creates a root filesystem directory and a JSON configuration file, then calls runc to launch the container process.

UDocker

UDocker is a Python wrapper that runs simple Docker containers entirely in user space without requiring root. It provides a Docker‑like CLI with a subset of commands (pull, import, create, run, etc.) and does not require Docker or its daemon on the host.

Other Engines

Podman supports pods (groups of containers sharing namespaces) and uses runc or crun. Pods can integrate with Kubernetes environments but require advanced kernel features. Socker allows running Docker containers under SLURM. Enroot from NVIDIA is an enhanced unprivileged chroot that retains filesystem isolation but reduces other isolation; it uses user and mount namespaces.

Performance Evaluation

Studies show that HPC‑targeted containers deliver near‑native performance. Singularity achieves close‑to‑bare‑metal CPU, memory, and network bandwidth. Benchmarks using Linpack, NAMD, VASP, WRF, and AMBER indicate minimal overhead. Shifter on HPGMG‑FE shows no significant performance penalty. Comparisons of Charliecloud, Podman, and Singularity on CPU, memory, network, and I/O confirm that overhead is low. SARUS demonstrates strong scaling on hybrid GPU/CPU nodes.

Container Orchestration

Orchestration refers to automated configuration, coordination, and management of containers. HPC systems traditionally rely on workload managers, while cloud systems use dedicated orchestrators.

HPC Workload Managers

A workload manager comprises a resource manager and a job scheduler. Common HPC schedulers include PBS (OpenPBS, PBS Pro, TORQUE), Spectrum LSF, Grid Engine, OAR, and SLURM. They share features: central scheduling, queuing systems, and static resource management.

PBS (Portable Batch System) – TORQUE is a variant. Jobs are submitted via qsub to a head node, where the PBS server and scheduler reside. The scheduler (e.g., Maui) uses FIFO with backfill or gang scheduling. An example PBS script:

#!/bin/bash
#PBS -l nodes=1:ppn=4
#PBS -l walltime=1500:00:00
#PBS -j oe
#PBS -q batch
source /opt/mpi/openmpi-intel-2.1.2.sh
NP=$(wc -l < "$PBS_NODEFILE")
echo "Job ID: $PBS_JOBID"
cd "$PBS_O_WORKDIR"
mpirun -np $NP vasp_std

SLURM consists of slurmctld (central manager), slurmd (compute node agent), slurmdbd (accounting), and slurmrestd (REST API). It supports high availability and plugin‑based scheduling.

Spectrum LSF supports Docker, Singularity, and Shifter containers via application profiles. Users submit jobs to profiles; LSF manages container runtime automatically.

Cloud Orchestrators

Cloud containers benefit from mature orchestrators offering resource limits, scheduling, load balancing, health checks, fault tolerance, and auto‑scaling. Key cloud tools include Kubernetes, Docker Swarm, Apache Mesos + Marathon, YARN, Ansible, and OpenStack. Kubernetes provides modularity, custom scheduling (filtering and scoring nodes), and extensive ecosystem. Docker Swarm is simpler and lighter, suitable for small workloads. Mesos acts as a distributed resource negotiator, while Marathon provides container orchestration on Mesos. Ansible handles configuration management and multi‑node orchestration. OpenStack, when combined with container orchestrators (Kubernetes, Swarm, Mesos), enables rapid deployment of complex, highly available infrastructure.

Bridging HPC and Cloud Orchestration

Recent efforts aim to merge container orchestration from cloud into HPC. Examples include running Kubernetes on HPC clusters, using SLURM with container runtimes, and extending workload managers to support orchestrator features. The goal is to combine the efficiency of cloud orchestration with the high‑performance capabilities of HPC systems.

Research Challenges and Opportunities

Containerisation on HPC faces three major challenges:

• Compatibility: Containers must match the host architecture and utilise underlying hardware (CPU, interconnect, accelerators).
• Security: HPC systems have stricter security models, limiting container privileges and requiring careful isolation.
• Performance: Overheads from containerisation must be minimised to preserve near‑native execution.

Future research opportunities include developing lightweight runtimes that balance isolation and performance, integrating cloud‑style orchestration (e.g., Kubernetes) with HPC workload managers, and designing portable container images that can be optimised for diverse hardware without losing performance.

Conclusion

This survey has reviewed containerisation techniques and orchestration strategies for HPC systems, comparing them with cloud environments. Containers bring customisation and reproducibility to HPC, but must address compatibility, security, and performance challenges. While HPC container engines like Singularity, Shifter, and Charliecloud have made significant progress, orchestration remains less mature than in the cloud. Integrating cloud orchestration frameworks with HPC workload managers is a promising direction. Continued research and engineering will help HPC leverage containers more effectively, enabling scalable, portable, and high‑performance scientific computing.