Docker/Cloud

Docker/Cloud#

Use this page after completing the minimal container or cloud setup in Installation. It contains the operational details for developing in Docker, retaining artifacts, extending images, submitting jobs to a cluster, and managing remote workstations.

Isaac Lab provides a Dockerfile and Compose configuration that layer Isaac Lab onto the NVIDIA Isaac Sim container. The resulting image is reproducible across Docker hosts and can be converted to an Apptainer/Singularity image for shared HPC systems. For public-cloud workstations, Isaac Automator provisions Isaac Sim and Isaac Lab on AWS, GCP, Azure, and Alibaba Cloud.

Docker guide#

Docker Guide

Caution

Due to the dependency on Isaac Sim docker image, by running this container you are implicitly agreeing to the NVIDIA Software License Agreement. If you do not agree to the EULA, do not run this container.

Setup Instructions

Note

The following steps are taken from the Isaac Sim documentation on container installation. They have been added here for the sake of completeness.

Docker and Docker Compose

We have tested the container using Docker Engine version 26.0.0 and Docker Compose version 2.25.0 We recommend using these versions or newer.

To install Docker, please follow the instructions for your operating system on the Docker website.
To install Docker Compose, please follow the instructions for your operating system on the docker compose page.
Follow the post-installation steps for Docker on the post-installation steps page. These steps allow you to run Docker without using sudo.
To build and run GPU-accelerated containers, you also need install the NVIDIA Container Toolkit. Please follow the instructions on the Container Toolkit website for installation steps.

Note

Due to limitations with snap, please make sure the Isaac Lab directory is placed under the /home directory tree when using docker.

Directory Organization

The root of the Isaac Lab repository contains the docker directory that has various files and scripts needed to run Isaac Lab inside a Docker container. A subset of these are summarized below:

Dockerfile.base: Defines the base Isaac Lab image by overlaying its dependencies onto the Isaac Sim Docker image. Dockerfiles which end with something else, (i.e. Dockerfile.ros2) build an image extension.
docker-compose.yaml: Creates mounts to allow direct editing of Isaac Lab code from the host machine that runs the container. It also creates several named volumes such as isaac-cache-kit to store frequently reused resources compiled by Isaac Sim, such as shaders, and to retain logs, data, and documents.
.env.base: Stores environment variables required for the base build process and the container itself. .env files which end with something else (i.e. .env.ros2) define these for image extension.
docker-compose.cloudxr-runtime.patch.yaml: A patch file that is applied to enable CloudXR Runtime support for streaming to compatible XR devices. It defines services and volumes for CloudXR Runtime and the base.
.env.cloudxr-runtime: Environment variables for the CloudXR Runtime support.
container.py: A utility script that interfaces with tools in utils to configure and build the image, and run and interact with the container.

Running the Container

Note

The docker container copies all the files from the repository into the container at the location /workspace/isaaclab at build time. This means that any changes made to the files in the container would not normally be reflected in the repository after the image has been built, i.e. after ./container.py start is run.

For a faster development cycle, we mount the following directories in the Isaac Lab repository into the container so that you can edit their files from the host machine:

IsaacLab/source: This is the directory that contains the Isaac Lab source code.
IsaacLab/docs: This is the directory that contains the source code for Isaac Lab documentation. This is overlaid except for the _build subdirectory where build artifacts are stored.

The script container.py parallels basic docker compose commands. Each can accept an image extension argument, or else they will default to the base image extension. These commands are:

build: This builds the image for the given profile. It does not bring up the container.
start: This builds the image and brings up the container in detached mode (i.e. in the background).
enter: This begins a new bash process in an existing Isaac Lab container, and which can be exited without bringing down the container.
config: This outputs the compose.yaml which would be result from the inputs given to container.py start. This command is useful for debugging a compose configuration.
copy: This copies the logs, data_storage and docs/_build artifacts, from the isaac-lab-logs, isaac-lab-data and isaac-lab-docs volumes respectively, to the docker/artifacts directory. These artifacts persist between docker container instances and are shared between image extensions.
stop: This brings down the container and removes it.

The following shows how to launch the container in a detached state and enter it:

# Launch the container in detached mode
# We don't pass an image extension arg, so it defaults to 'base'
./docker/container.py start

# If we want to add .env or .yaml files to customize our compose config,
# we can simply specify them in the same manner as the compose cli
# ./docker/container.py start --file my-compose.yaml --env-file .env.my-vars

# Enter the container
# We pass 'base' explicitly, but if we hadn't it would default to 'base'
./docker/container.py enter base

The Isaac Lab base, ROS 2, and cuRobo images run as a non-root user with uid/gid 1000 to keep bind-mounted workspaces writable on GitHub runners. If you run one of these images directly with docker run and your host uid differs, pass Docker’s --user "$(id -u):1000" option so new files on bind mounts are owned by your host user while retaining runtime-home access.

If you are upgrading an existing Compose setup from older root-based images, recreate the named volumes before starting the new images. Older cache, log, and data volumes may contain root-owned files that the uid/gid 1000 runtime user cannot update. Copy any artifacts you want to keep, then remove the old Compose volumes from the docker directory:

docker compose --file docker-compose.yaml --profile base --env-file .env.base down --volumes

To copy files from the base container to the host machine, you can use the following command:

# Copy the file /workspace/isaaclab/logs to the current directory
docker cp isaac-lab-base:/workspace/isaaclab/logs .

The script container.py provides a wrapper around this command to copy the logs , data_storage and docs/_build directories to the docker/artifacts directory. This is useful for copying the logs, data and documentation:

# stop the container
./docker/container.py stop

CloudXR Runtime Support

To enable CloudXR Runtime for streaming to compatible XR devices, you need to apply the patch file docker-compose.cloudxr-runtime.patch.yaml to run CloudXR Runtime container. The patch file defines services and volumes for CloudXR Runtime and base. The environment variables required for CloudXR Runtime are specified in the .env.cloudxr-runtime file. To start or stop the CloudXR runtime container with base, use the following command:

# Start CloudXR Runtime container with base.
./docker/container.py start --files docker-compose.cloudxr-runtime.patch.yaml --env-file .env.cloudxr-runtime

# Stop CloudXR Runtime container and base.
./docker/container.py stop --files docker-compose.cloudxr-runtime.patch.yaml --env-file .env.cloudxr-runtime

X11 forwarding

The container supports X11 forwarding, which allows the user to run GUI applications from the container and display them on the host machine.

The first time a container is started with ./docker/container.py start, the script prompts the user whether to activate X11 forwarding. This will create a file at docker/.container.cfg to store the user’s choice for future runs.

If you want to change the choice, you can set the parameter X11_FORWARDING_ENABLED to ‘0’ or ‘1’ in the docker/.container.cfg file to disable or enable X11 forwarding, respectively. After that, you need to re-build the container by running ./docker/container.py start. The rebuilding process ensures that the changes are applied to the container. Otherwise, the changes will not take effect.

After the container is started, you can enter the container and run GUI applications from it with X11 forwarding enabled. The display will be forwarded to the host machine.

Python Interpreter

The container uses the Python interpreter provided by Isaac Sim. This interpreter is located at /isaac-sim/python.sh. We set aliases inside the container to make it easier to run the Python interpreter. You can use the following commands to run the Python interpreter:

# Run the Python interpreter -> points to /isaac-sim/python.sh
python

Understanding the mounted volumes

The docker-compose.yaml file creates several named volumes that are mounted to the container. These are summarized below:

Volume Name	Description	Container Path
isaac-cache-kit	Stores cached Kit resources	/isaac-sim/kit/cache
isaac-cache-ov	Stores cached OV resources	/root/.cache/ov
isaac-cache-pip	Stores cached pip resources	/root/.cache/pip
isaac-cache-gl	Stores cached GLCache resources	/root/.cache/nvidia/GLCache
isaac-cache-compute	Stores cached compute resources	/root/.nv/ComputeCache
isaac-logs	Stores logs generated by Omniverse	/root/.nvidia-omniverse/logs
isaac-carb-logs	Stores logs generated by carb	/isaac-sim/kit/logs/Kit/Isaac-Sim
isaac-data	Stores data generated by Omniverse	/root/.local/share/ov/data
isaac-docs	Stores documents generated by Omniverse	/root/Documents
isaac-lab-docs	Stores documentation of Isaac Lab when built inside the container	/workspace/isaaclab/docs/_build
isaac-lab-logs	Stores logs generated by Isaac Lab workflows when run inside the container	/workspace/isaaclab/logs
isaac-lab-data	Stores whatever data users may want to preserve between container runs	/workspace/isaaclab/data_storage

To view the contents of these volumes, you can use the following command:

# list all volumes
docker volume ls
# inspect a specific volume, e.g. isaac-cache-kit
docker volume inspect isaac-cache-kit

Isaac Lab Image Extensions

The produced image depends on the arguments passed to container.py start and container.py stop. These commands accept an image extension parameter as an additional argument. If no argument is passed, then this parameter defaults to base. Currently, the only valid values are (base, ros2). Only one image extension can be passed at a time. The produced image and container will be named isaac-lab-${profile}, where ${profile} is the image extension name.

suffix is an optional string argument to container.py that specifies a docker image and container name suffix, which can be useful for development purposes. By default ${suffix} is the empty string. If ${suffix} is a nonempty string, then the produced docker image and container will be named isaac-lab-${profile}-${suffix}, where a hyphen is inserted between ${profile} and ${suffix} in the name. suffix should not be used with cluster deployments.

# start base by default, named isaac-lab-base
./docker/container.py start
# stop base explicitly, named isaac-lab-base
./docker/container.py stop base
# start ros2 container named isaac-lab-ros2
./docker/container.py start ros2
# stop ros2 container named isaac-lab-ros2
./docker/container.py stop ros2

# start base container named isaac-lab-base-custom
./docker/container.py start base --suffix custom
# stop base container named isaac-lab-base-custom
./docker/container.py stop base --suffix custom
# start ros2 container named isaac-lab-ros2-custom
./docker/container.py start ros2 --suffix custom
# stop ros2 container named isaac-lab-ros2-custom
./docker/container.py stop ros2 --suffix custom

The passed image extension argument will build the image defined in Dockerfile.${image_extension}, with the corresponding profile in the docker-compose.yaml and the envars from .env.${image_extension} in addition to the .env.base, if any.

ROS2 Image Extension

In Dockerfile.ros2, the container installs ROS2 Humble via an apt package, and it is sourced in the .bashrc. The exact version is specified by the variable ROS_APT_PACKAGE in the .env.ros2 file, defaulting to ros-base. Other relevant ROS2 variables are also specified in the .env.ros2 file, including variables defining the various middleware options.

The container defaults to FastRTPS, but CylconeDDS is also supported. Each of these middlewares can be tuned using their corresponding .xml files under docker/.ros.

Running Pre-Built Isaac Lab Container

In Isaac Lab 2.0 release, we introduced a minimal pre-built container that contains a very minimal set of Isaac Sim and Omniverse dependencies, along with Isaac Lab 2.0 pre-built into the container. This container allows users to pull the container directly from NGC without requiring a local build of the docker image. The Isaac Lab source code will be available in this container under /workspace/IsaacLab.

This container is designed for running headless only and does not allow for X11 forwarding or running with the GUI. Please only use this container for headless training. For other use cases, we recommend following the above steps to build your own Isaac Lab docker image.

Note

Currently, we only provide docker images with every major release of Isaac Lab. For example, we provide the docker image for release 2.0.0 and 2.1.0, but not 2.0.2. In the future, we will provide docker images for every minor release of Isaac Lab.

To pull the minimal Isaac Lab container, run:

docker pull nvcr.io/nvidia/isaac-lab:3.0.0-beta2

Attention

If the pre-built image you use runs as a non-root user (uid/gid 1000) – as Isaac Lab 3.0.0-beta2 and later do – the bind-mounted host directories below must be writable by that user. Docker creates any missing bind-mount source directory as root, which the non-root runtime user cannot write to, leading to startup errors such as PermissionError: [Errno 13] Permission denied: '/root/.local/share/ov/data/exts'. Pre-create the host directories and make them writable by uid/gid 1000 before running the container:

mkdir -p ~/docker/isaac-sim/{cache/kit,cache/ov,cache/pip,cache/glcache,cache/computecache,logs,data,documents}
sudo chown -R 1000:1000 ~/docker/isaac-sim

To run the Isaac Lab container with an interactive bash session, run:

docker run --name isaac-lab --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
   -e "PRIVACY_CONSENT=Y" \
   -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
   -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
   -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
   -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
   -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
   -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
   -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
   -v ~/docker/isaac-sim/documents:/root/Documents:rw \
   nvcr.io/nvidia/isaac-lab:3.0.0-beta2

To enable rendering through X11 forwarding, run:

xhost +
docker run --name isaac-lab --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
   -e "PRIVACY_CONSENT=Y" \
   -e DISPLAY \
   -v $HOME/.Xauthority:/root/.Xauthority \
   -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
   -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
   -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
   -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
   -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
   -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
   -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
   -v ~/docker/isaac-sim/documents:/root/Documents:rw \
   nvcr.io/nvidia/isaac-lab:3.0.0-beta2

To run an example within the container, run:

uv (Recommended)

uv run python scripts/tutorials/00_sim/log_time.py

isaaclab.sh / isaaclab.bat

./isaaclab.sh -p scripts/tutorials/00_sim/log_time.py

Worked Docker example#

Running an example with Docker

From the root of the Isaac Lab repository, the docker directory contains all the Docker relevant files. These include the three files (Dockerfile, docker-compose.yaml, .env) which are used by Docker, and an additional script that we use to interface with them, container.py.

In this tutorial, we will learn how to use the Isaac Lab Docker container for development. For a detailed description of the Docker setup, including installation and obtaining access to an Isaac Sim image, please reference the Docker Guide. For a description of Docker in general, please refer to their official documentation.

Building the Container

To build the Isaac Lab container from the root of the Isaac Lab repository, we will run the following:

python docker/container.py start

The terminal will first pull the base IsaacSim image, build the Isaac Lab image’s additional layers on top of it, and run the Isaac Lab container. This should take several minutes for the first build but will be shorter in subsequent runs as Docker’s caching prevents repeated work. If we run the command docker container ls on the terminal, the output will list the containers that are running on the system. If everything has been set up correctly, a container with the NAME isaac-lab-base should appear, similar to below:

CONTAINER ID   IMAGE               COMMAND   CREATED           STATUS         PORTS     NAMES
483d1d5e2def   isaac-lab-base      "bash"    30 seconds ago   Up 30 seconds             isaac-lab-base

Once the container is up and running, we can enter it from our terminal.

python docker/container.py enter

On entering the Isaac Lab container, we are in the terminal as a non-root user. This environment contains a copy of the Isaac Lab repository, but also has access to the directories and libraries of Isaac Sim. We can run experiments from this environment using a few convenient aliases that have been put into the runtime user’s .bashrc. For instance, we have made the isaaclab.sh script usable from anywhere by typing its alias isaaclab.

Additionally in the container, we have bind mounted the IsaacLab/source directory from the host machine. This means that if we modify files under this directory from an editor on the host machine, the changes are reflected immediately within the container without requiring us to rebuild the Docker image.

We will now run a sample script from within the container to demonstrate how to extract artifacts from the Isaac Lab Docker container.

The Code

The tutorial corresponds to the log_time.py script in the IsaacLab/scripts/tutorials/00_sim directory.

Code for log_time.py

# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

"""
This script demonstrates how to generate log outputs while the simulation plays.
It accompanies the tutorial on docker usage.

.. code-block:: bash

    # Usage
    uv run python scripts/tutorials/00_sim/log_time.py

"""

"""Launch Isaac Sim Simulator first."""


import argparse
import os

from isaaclab.app import AppLauncher

# create argparser
parser = argparse.ArgumentParser(description="Tutorial on creating logs from within the docker container.")
# append AppLauncher cli args
AppLauncher.add_app_launcher_args(parser)
# parse the arguments
args_cli = parser.parse_args()
# launch omniverse app
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app

"""Rest everything follows."""

from isaaclab.sim import SimulationCfg, SimulationContext


def main():
    """Main function."""
    # Specify that the logs must be in logs/docker_tutorial
    log_dir_path = os.path.join("logs")
    if not os.path.isdir(log_dir_path):
        os.mkdir(log_dir_path)
    # In the container, the absolute path will be
    # /workspace/isaaclab/logs/docker_tutorial, because
    # all Python execution is done through the uv-managed workspace
    # and the calling process' path will be /workspace/isaaclab
    log_dir_path = os.path.abspath(os.path.join(log_dir_path, "docker_tutorial"))
    if not os.path.isdir(log_dir_path):
        os.mkdir(log_dir_path)
    print(f"[INFO] Logging experiment to directory: {log_dir_path}")

    # Initialize the simulation context
    sim_cfg = SimulationCfg(dt=0.01)
    sim = SimulationContext(sim_cfg)
    # Set main camera
    sim.set_camera_view([2.5, 2.5, 2.5], [0.0, 0.0, 0.0])

    # Play the simulator
    sim.reset()
    # Now we are ready!
    print("[INFO]: Setup complete...")

    # Prepare to count sim_time
    sim_dt = sim.get_physics_dt()
    sim_time = 0.0

    # Open logging file
    with open(os.path.join(log_dir_path, "log.txt"), "w") as log_file:
        # Simulate physics
        while simulation_app.is_running():
            log_file.write(f"{sim_time}" + "\n")
            # perform step
            sim.step()
            sim_time += sim_dt


if __name__ == "__main__":
    # run the main function
    main()
    # close sim app
    simulation_app.close()

The Code Explained

The Isaac Lab Docker container has several volumes to facilitate persistent storage between the host computer and the container. One such volume is the /workspace/isaaclab/logs directory. The log_time.py script designates this directory as the location to which a log.txt should be written:

    # Specify that the logs must be in logs/docker_tutorial
    log_dir_path = os.path.join("logs")
    if not os.path.isdir(log_dir_path):
        os.mkdir(log_dir_path)
    # In the container, the absolute path will be
    # /workspace/isaaclab/logs/docker_tutorial, because
    # all Python execution is done through the uv-managed workspace
    # and the calling process' path will be /workspace/isaaclab
    log_dir_path = os.path.abspath(os.path.join(log_dir_path, "docker_tutorial"))
    if not os.path.isdir(log_dir_path):
        os.mkdir(log_dir_path)
    print(f"[INFO] Logging experiment to directory: {log_dir_path}")

As the comments note, os.path.abspath() will prepend /workspace/isaaclab because in the Docker container all python execution is done through /workspace/isaaclab/isaaclab.sh. The output will be a file, log.txt, with the sim_time written on a newline at every simulation step:

    # Prepare to count sim_time
    sim_dt = sim.get_physics_dt()
    sim_time = 0.0

    # Open logging file
    with open(os.path.join(log_dir_path, "log.txt"), "w") as log_file:
        # Simulate physics
        while simulation_app.is_running():
            log_file.write(f"{sim_time}" + "\n")
            # perform step
            sim.step()
            sim_time += sim_dt

Executing the Script

This command does not select a visualizer, so it runs without a GUI while producing the log:

isaaclab -p scripts/tutorials/00_sim/log_time.py

If a config or command would otherwise select a visualizer, force-disable all visualizers with --visualizer none or --viz none.

Now log.txt will have been produced at /workspace/isaaclab/logs/docker_tutorial. If we exit the container by typing exit, we will return to IsaacLab/docker in our host terminal environment. We can then enter the following command to retrieve our logs from the Docker container and put them on our host machine:

./container.py copy

We will see a terminal readout reporting the artifacts we have retrieved from the container. If we navigate to /isaaclab/docker/artifacts/logs/docker_tutorial, we will see a copy of the log.txt file which was produced by the script above.

Each of the directories under artifacts corresponds to Docker volumes mapped to directories within the container and the container.py copy command copies them from those volumes to these directories.

We could return to the Isaac Lab Docker terminal environment by running container.py enter again, but we have retrieved our logs and wish to go inspect them. We can stop the Isaac Lab Docker container with the following command:

./container.py stop

This will bring down the Docker Isaac Lab container. The image will persist and remain available for further use, as will the contents of any volumes. If we wish to free up the disk space taken by the image, (~20.1GB), and do not mind repeating the build process when we next run ./container.py start, we may enter the following command to delete the isaac-lab-base image:

docker image rm isaac-lab-base

A subsequent run of docker image ls will show that the image tagged isaac-lab-base is now gone. We can repeat the process for the underlying NVIDIA container if we wish to free up more space. If a more powerful method of freeing resources from Docker is desired, please consult the documentation for the docker prune commands.

Clusters#

Cluster Guide

Clusters are a great way to speed up training and evaluation of learning algorithms. While the Isaac Lab Docker image can be used to run jobs on a cluster, many clusters only support singularity images. This is because singularity is designed for ease-of-use on shared multi-user systems and high performance computing (HPC) environments. It does not require root privileges to run containers and can be used to run user-defined containers.

Singularity is compatible with all Docker images. In this section, we describe how to convert the Isaac Lab Docker image into a singularity image and use it to submit jobs to a cluster.

Attention

Cluster setup varies across different institutions. The following instructions have been tested on the ETH Zurich Euler cluster (which uses the SLURM workload manager), and the IIT Genoa Franklin cluster (which uses PBS workload manager).

The instructions may need to be adapted for other clusters. If you have successfully adapted the instructions for another cluster, please consider contributing to the documentation.

Setup Instructions

In order to export the Docker Image to a singularity image, apptainer is required. A detailed overview of the installation procedure for apptainer can be found in its documentation. For convenience, we summarize the steps here for a local installation:

sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer

For simplicity, we recommend that an SSH connection is set up between the local development machine and the cluster. Such a connection will simplify the file transfer and prevent the user cluster password from being requested multiple times.

Attention

The workflow has been tested with:

apptainer version 1.2.5-1.el7 and docker version 24.0.7
apptainer version 1.3.4 and docker version 27.3.1

In the case of issues, please try to switch to those versions.

Configuring the cluster parameters

First, you need to configure the cluster-specific parameters in docker/cluster/.env.cluster file. The following describes the parameters that need to be configured:

Parameter	Description
CLUSTER_JOB_SCHEDULER	The job scheduler/workload manager used by your cluster. Currently, we support ‘SLURM’ and ‘PBS’ workload managers.
CLUSTER_ISAAC_SIM_CACHE_DIR	The directory on the cluster where the Isaac Sim cache is stored. This directory has to end on `docker-isaac-sim`. It will be copied to the compute node and mounted into the singularity container. This should increase the speed of starting the simulation.
CLUSTER_ISAACLAB_DIR	The directory on the cluster where the Isaac Lab logs are stored. This directory has to end on `isaaclab`. It will be copied to the compute node and mounted into the singularity container. When a job is submitted, the latest local changes will be copied to the cluster to a new directory in the format `${CLUSTER_ISAACLAB_DIR}_${datetime}` with the date and time of the job submission. This allows to run multiple jobs with different code versions at the same time.
CLUSTER_LOGIN	The login to the cluster. Typically, this is the user and cluster names, e.g., `your_user@euler.ethz.ch`.
CLUSTER_SIF_PATH	The path on the cluster where the singularity image will be stored. The image will be copied to the compute node but not uploaded again to the cluster when a job is submitted.
REMOVE_CODE_COPY_AFTER_JOB	Whether the copied code should be removed after the job is finished or not. The logs from the job will not be deleted as these are saved under the permanent `CLUSTER_ISAACLAB_DIR`. This feature is useful to save disk space on the cluster. If set to `true`, the code copy will be removed.
CLUSTER_PYTHON_EXECUTABLE	The path within Isaac Lab to the Python executable that should be executed in the submitted job.

When a job is submitted, it will also use variables defined in docker/.env.base, though these should be correct by default.

Exporting to singularity image

Next, we need to export the Docker image to a singularity image and upload it to the cluster. This step is only required once when the first job is submitted or when the Docker image is updated. For instance, due to an upgrade of the Isaac Sim version, or additional requirements for your project.

To export to a singularity image, execute the following command:

./docker/cluster/cluster_interface.sh push [profile]

This command will create a singularity image under docker/exports directory and upload it to the defined location on the cluster. It requires that you have previously built the image with the container.py interface. Be aware that creating the singularity image can take a while. [profile] is an optional argument that specifies the container profile to be used. If no profile is specified, the default profile base will be used.

Note

By default, the singularity image is created without root access by providing the --fakeroot flag to the apptainer build command. In case the image creation fails, you can try to create it with root access by removing the flag in docker/cluster/cluster_interface.sh.

Defining the job parameters

The job parameters need to be defined based on the job scheduler used by your cluster. You only need to update the appropriate script for the scheduler available to you.

For SLURM, update the parameters in docker/cluster/submit_job_slurm.sh.
For PBS, update the parameters in docker/cluster/submit_job_pbs.sh.

For SLURM

The job parameters are defined inside the docker/cluster/submit_job_slurm.sh. A typical SLURM operation requires specifying the number of CPUs and GPUs, the memory, and the time limit. For more information, please check the SLURM documentation.

The default configuration is as follows:

#SBATCH --cpus-per-task=8
#SBATCH --gpus=rtx_3090:1
#SBATCH --time=23:00:00
#SBATCH --mem-per-cpu=4048
#SBATCH --mail-type=END
#SBATCH --mail-user=name@mail
#SBATCH --job-name="training-$(date +"%Y-%m-%dT%H:%M")"

An essential requirement for the cluster is that the compute node has access to the internet at all times. This is required to load assets from the Nucleus server. For some cluster architectures, extra modules must be loaded to allow internet access.

For instance, on ETH Zurich Euler cluster, the eth_proxy module needs to be loaded. This can be done by adding the following line to the submit_job_slurm.sh script:

# in the case you need to load specific modules on the cluster, add them here
# e.g., `module load eth_proxy`

For PBS

The job parameters are defined inside the docker/cluster/submit_job_pbs.sh. A typical PBS operation requires specifying the number of CPUs and GPUs, and the time limit. For more information, please check the PBS Official Site.

The default configuration is as follows:

#PBS -l select=1:ncpus=8:mpiprocs=1:ngpus=1
#PBS -l walltime=01:00:00
#PBS -j oe
#PBS -q gpu
#PBS -N isaaclab
#PBS -m bea -M "user@mail"

Submitting a job

To submit a job on the cluster, the following command can be used:

./docker/cluster/cluster_interface.sh job [profile] "argument1" "argument2" ...

This command will copy the latest changes in your code to the cluster and submit a job. Please ensure that your Python executable’s output is stored under isaaclab/logs as this directory is synced between the compute node and CLUSTER_ISAACLAB_DIR.

[profile] is an optional argument that specifies which singularity image corresponding to the container profile will be used. If no profile is specified, the default profile base will be used. The profile has be defined directlty after the job command. All other arguments are passed to the Python executable. If no profile is defined, all arguments are passed to the Python executable.

The training arguments are passed to the Python executable. As an example, the standard ANYmal rough terrain locomotion training can be executed with the following command:

./docker/cluster/cluster_interface.sh job --rl_library rsl_rl --task IsaacContrib-Velocity-Rough-AnymalC --video

The above will, in addition, also render videos of the training progress and store them under isaaclab/logs directory.

Cloud workstations#

Cloud Deployment

Isaac Lab can be run in various cloud infrastructures with the use of Isaac Automator (v4).

Isaac Automator allows quick deployment of Isaac Sim, Isaac Lab, and Isaac Lab Arena onto public clouds (AWS, GCP, Azure, and Alibaba Cloud are currently supported). The result is a fully configured remote desktop cloud workstation (Isaac Workstation), which can be used for development and testing of Isaac Lab within minutes and on a budget. Isaac Automator supports a variety of GPU instances and stop/start functionality to save on cloud costs, and provides tools to aid the workflow (uploading and downloading data, autorun scripts, deployment management, etc.).

System Requirements

Isaac Automator requires having docker pre-installed on the system.

To install Docker, please follow the instructions for your operating system on the Docker Engine website.
Follow the post-installation steps for Docker on the Docker post-installation steps page. These steps allow you to run Docker without using sudo.

Installing Isaac Automator

For the most up-to-date and complete installation instructions, please refer to the Isaac Automator README.

To use Isaac Automator, first clone the repo:

HTTPS

git clone https://github.com/isaac-sim/IsaacAutomator.git

SSH

git clone git@github.com:isaac-sim/IsaacAutomator.git

Building the Container

Build the Isaac Automator container:

Linux / macOS

./build

Windows

docker build --platform linux/x86_64 -t isaac_automator .

This will build the Isaac Automator container and tag it as isaac_automator.

Deploying an Isaac Workstation

Linux / macOS

Enter the Automator container and run the deployment command:

./run
# inside container:
./deploy-aws

Alternatively, run it in one step:

./run ./deploy-aws

Windows

docker run --platform linux/x86_64 -it --rm -v .:/app isaac_automator bash
:: inside container:
./deploy-aws

Replace deploy-aws with deploy-gcp, deploy-azure, or deploy-alicloud for other cloud providers.

Note

The --isaaclab and --isaacsim flags accept any valid Git reference to specify the version to deploy. Use --isaaclab no or --isaacsim no to skip installation of the respective component.

./deploy-aws --isaaclab v3.0.0 --isaacsim main

On the first run (or when credentials expire), you will be prompted to enter your cloud credentials. Credentials are stored in state/ and persist across container restarts. Run ./deploy-<cloud> --help to see all available options.

Key deployment options:

--instance-type – Cloud VM instance type.
--isaacsim / --isaaclab / --isaaclab-arena – Git ref for the version to install, or no to skip.
--existing – What to do if a deployment already exists: ask (default), repair, modify, replace, or run_ansible.
--from-image – Deploy from a pre-built VM image for faster provisioning (AWS only at this time).

Connecting to the Isaac Workstation

Deployed Isaac Workstations can be accessed via:

SSH: ./ssh <deployment-name>
noVNC (browser-based): ./novnc <deployment-name>
NoMachine (remote desktop client)

Connection instructions are displayed at the end of the deployment command output and saved in state/<deployment-name>/info.txt.

Running Isaac Lab on the Cloud

Isaac Lab is installed from source on the deployed workstation at ~/IsaacLab. To run Isaac Lab commands, open a terminal on the workstation:

~/IsaacLab/isaaclab.sh train --rl_library rsl_rl \
  --task=Isaac-Cartpole-Direct

Pausing and Resuming

You can stop and restart instances to save on cloud costs:

# inside the Automator container:
./stop <deployment-name>
./start <deployment-name>

Use ./start <deployment-name> --quick to skip full Ansible provisioning and only run the autorun script.

Uploading and Downloading Data

# upload local uploads/ folder to the instance
./upload <deployment-name>

# download results from the instance to local results/ folder
./download <deployment-name>

Destroying a Deployment

To save costs, destroy deployments when no longer needed:

# inside the Automator container:
./destroy <deployment-name>

Note

Deployment metadata is stored in the state/ directory. Do not delete this directory, as it is required for managing deployments.

Docker/Cloud

Contents

Docker/Cloud#

Docker guide#

Worked Docker example#

Clusters#

Cloud workstations#