Actor Simulation and Synthetic Data Generation#

Detecting and tracking animated actors or agents like human characters and robots in diverse environments offers significant value across industries like retail, manufacturing, and logistics. It helps optimize layouts, improve safety, and enhance efficiency. However, collecting real-world data to train detection models is often costly and unscalable.

Synthetic data generation offers a flexible, scalable solution. The Omni.Metropolis.Pipeline (OMP), Isaacsim.Replicator.Agent (IRA), Isaacsim.Anim.Robot.Core (IAR) extensions together provide a way to set up human characters and robots in 3D environments and generate synthetic data. This framework also provides control over actor behaviors, environments, sensors, via configuration file. It aims to provide a GPU-accelerated solution for training computer vision models and testing software-in-the-loop systems.

This framework simplifies simulation customization with features like:

Codeless Interaction: Configurations are expressed in yaml file. No code is needed to get synthetic data.
Simplified Setup: Included in Isaac Sim, it offers both GUI and scripting interfaces for interactive and headless workflows.
High-Fidelity Data: Leverages Omniverse’s SimReady assets, physics, and rendering to produce realistic imagery and accurate annotations essential for AI training.
Seamless Integration: As part of Kit extensions, it works natively with omni.anim.behavior, omni.anim.navigation, and omni.replicator.core.

Before enabling this extension, read What Is Isaac Sim? to learn about Isaac Sim and follow Installation to install Isaac Sim.

characters and robots move around a warehouse 3D environment.

Enable Extensions#

Follow the Omniverse Extension Manager guide to enable the Omni.Metropolis.Pipeline, Isaacsim.Anim.Robot.Core and Isaacsim.Replicator.Agent.Core & UI.
- The extensions fetch sample assets from Isaac Sim Assets during start. Refer to Isaac Sim Assets if you encounter issues for loading assets.
- If loading the UI appears to be hanging, try starting Isaac Sim with the flag --/persistent/isaac/asset_root/timeout=1.0.
The UI panel is accessible by Tools > Action and Event Data Generation > Actor SDG and it opens on the right side of the screen.

Note

To have the extension auto-loaded on startup, check the autoload checkbox in the extension manager.
Because of extension dependencies, a restart of the Isaac Sim app might be required.

Tip

If you encounter unexpected errors, try launching Isaac Sim with the --reset-user flag to clear previous user settings.

./isaac-sim.sh --reset-user

Getting Started in UI#

It is recommended to use UI for first-time users. Please refer to Running from script section for running with python script in IsaacSim headless mode.

Follow the Enable Extensions and open the UI panel.
The default minimal config is loaded by default. You can also load a separate config file using the folder browser icon.
- All the sample config files are in [Isaac Sim App Path]/extscache/isaacsim.replicator.agent.core-[current-version]/data/sample_configs/.
- The minimal config file does not have actors and cameras. For a more comprehensive example, please use warehouse.yaml in the above folder. Note that this example can take up more loading time.

[Optional] Modify the configuration file to your needs.
- Use Save or Save As icon to save the changes in UI to config file.
- Use Reload icon to reset changes in UI and load the original config file again.
Click the Set Up Simulation button from the top of the UI and it will start loading simulation assets (scene, cameras, actors) according to the UI.
- The scene requires a NavMesh to spawn assets and control them correctly. The scenes in the example config has NavMesh set up in advance. If you are using a external scene, please refer to Navigation Mesh for NavMesh set up.
- You can also go to Window > Navigation > NavMesh and turn off Auto-Bake in the NavMesh settings. Turning it off can increase the performance.
Note

Clicking Set Up Simulation always fully reloads the scene from the current configuration. This includes reopening the base environment USD and re-creating all actors (characters and robots), sensors, and prop layers from scratch. Any manual edits made to the stage after a previous setup will be lost. If you want to iterate on the configuration, make your changes in the UI or config file first, then click Set Up Simulation to apply them.
Click the Start Data Generation button from the top of the UI and the simulation and data generation will start. It will run for the duration (in seconds) specified in the Simulation Duration in Actor SDG Setup panel.
When data generation finishes, the output data can be found from the Output Directory according to the output directory in Replicator panel.
- By default, it is in the User folder for Windows and the home folder for Linux.

Running from Script#

For large-scale data generation, it can be more efficient to launch it from script. IRA provides an automatic script (actor_sdg.py) to run offline data generation.

To run from script, open a terminal from where Isaac Sim is installed and run the following commands.

For Linux:
./python.sh tools/actor_sdg/actor_sdg.py -c [config file path]
For Windows:
.\python.bat tools\actor_sdg\actor_sdg.py -c [config file path]

Note

[config file path] is the path to the IRA configuration file.
You must use the python.sh or python.bat bundled with Isaac Sim to run the script.
An example config file is also provided in the /tools/actor_sdg folder. For a sample Linux run, execute: ./python.sh tools/actor_sdg/actor_sdg.py -c tools/actor_sdg/sample_config.yaml

API Usage#

This extension also exposes a Python API which you can use to set up simulations and generate data from your own script. Ensure that isaacsim.replicator.agent.core is enabled, and use the API as in the following example.

Note: The snippet below uses the minimal config bundled with the extension (data/sample_configs/minimal.yaml).

import os

from isaacsim import SimulationApp

# Start the application
simulation_app = SimulationApp({"headless": True})

# Get the utility to enable extensions
from isaacsim.core.utils.extensions import enable_extension

# Enable the IRA extension
enable_extension("isaacsim.replicator.agent.core")
simulation_app.update()


def _get_config_path() -> str:
    """Return path to the minimal IRA config bundled with the extension."""
    import omni.kit.app

    core_ext_path = (
        omni.kit.app.get_app().get_extension_manager().get_extension_path_by_module("isaacsim.replicator.agent.core")
    )
    default_config_file_path = os.path.join(core_ext_path, "data", "sample_configs", "minimal.yaml")
    if not os.path.isfile(default_config_file_path):
        raise FileNotFoundError(f"IRA config not found: {default_config_file_path}")
    return default_config_file_path


async def run_ira_data_generation(setup_simulation: bool = False, run_data_generation: bool = False):
    from isaacsim.replicator.agent.core import api as IRA

    # IRA: load config. Specify the config file path.
    config_path = _get_config_path()
    result = IRA.load_config_file(config_path)
    if not result:
        raise RuntimeError(f"Failed to load IRA config: {config_path}")

    # IRA: get config, you can modify the config here
    config = IRA.get_config_file()
    IRA.set_config(config)

    # IRA: setup simulation (only when setup_simulation is True)
    if setup_simulation:
        await IRA.setup_simulation()

        # Allow a few frames for scene to settle
        import omni.kit.app

        app = omni.kit.app.get_app()
        for _ in range(10):
            await app.next_update_async()

        # IRA: generate data (only when run_data_generation is True)
        if run_data_generation:
            await IRA.start_data_generation_async(will_wait_until_complete=True)


from omni.kit.async_engine import run_coroutine

task = run_coroutine(run_ira_data_generation(setup_simulation=False, run_data_generation=False))
while not task.done():
    simulation_app.update()

Configuration File#

The configuration file is the central place to define your simulation. It controls everything from the environment and characters to the sensors and data output. The file uses the YAML format.

The configuration file is organized into these top-level sections:

environment: Defines the simulation environment and assets.
character: Configures human characters.
robot: Configures robots.
sensor: Configures RTX sensors.
replicator: Configures data generation and output.

For detailed configuration instructions, parameter lists, and examples, refer to the following document:

Configuration File Guide

For editing the configuration files through UI or code, see the Configuration Editor API:

Configuration Editor API

Actor Behaviors#

Actor behaviors are achieved by OMP, IRA and IAR together.

Actors perform a “routine-trigger” behavior loop at play. This pattern is configurable by the behaviors and triggers assigned to the actor.

The Routine Trigger Loop#

When no actor triggers are activated, actors perform routine loop by repeatedly pick behaviors under routines to perform by their probability weights, using the actor global seed.

When any trigger is activated, actor will pause routine and start performing the behaviors under each active trigger. Running triggers will be paused and pushed to queue if a trigger with higher priority happens (triggers with lower priority will be skipped). The trigger will be marked complete when its behaviors are all finished. Then the first trigger in queue will resume running. Once all active triggers complete, actors will fallback to routine.

Configure Behaviors#

After actors are loaded into scene by config file, the configurations are embedded in the USD API schemas and USD Prims. Each actor is represented by MetroAgentAPI schema and its derived type. For human character, it is the IRACharacterAPI attached on the SkelRoot prim. For animated robot, it is the AnimRobotAPI attached on the root prim of the robot payload. Each behavior and trigger becomes individual USD Prims that actor USD API can have reference to, each actor trigger prim can also have reference to a list of behaviors.

The actor USD API schema defines basic information of the actor: name, group, seed, a routine reference slot and a trigger reference slot. At play, the name, group and seed will be combined and hashed into a single seed as actor global seed. This seed will be used for all the “randomness” of the actor, including random routine picking for the actor itself and the picking within each behavior such as picking a speed from speed range. This also means the same actor global seed will display same result if other settings and the environment don’t change.

Each type of actor behavior is represented by a USD Prim type. It defines the configuration of the behavior: weight, repeat and behavior specific parameters. For human characters, the behavior prim types follows CharacterXXXBehavior naming pattern. For animated robots, they are RobotXXXBehavior.

Each actor trigger is also a USD Prim. It defines the trigger priority and has a reference of behavior list to be executed sequentially when this trigger activates. Human characters and anim robots share the same trigger types that’s defined in OMP with naming MetroXXXTrigger.

In addition, actors leverage omni.behavior.behavior (Human characters) and isaacsim.anim.robot.core (Animated robots) as their animation implementation. For more information about them, please refer to the following documents:

Animated Robot Controller