[isaacsim.benchmark.services] Benchmark Services#

Version: 4.2.1

Overview#

isaacsim.benchmark.services provides a comprehensive framework for performance benchmarking in Isaac Sim applications. The extension offers structured approaches to measure and analyze CPU, GPU, memory, and frame time performance through both synchronous and asynchronous benchmark implementations.

Key Components#

`BaseIsaacBenchmark`#

The synchronous benchmark class designed for standalone scripts that need deterministic performance measurements. It provides phase-based benchmarking where different stages of execution can be measured independently.

benchmark = BaseIsaacBenchmark(benchmark_name="MyBenchmark", workflow_metadata={"metadata": []})
benchmark.set_phase("loading")
# load stage, configure sim, etc.
benchmark.store_measurements()
benchmark.set_phase("benchmark")
# run benchmark
benchmark.store_measurements()
benchmark.stop()

`BaseIsaacBenchmarkAsync`#

An asynchronous benchmark class that inherits from **omni.kit.test.AsyncTestCase**, enabling integration with Isaac Sim’s test framework. This approach is ideal for test cases that require async operations and proper test lifecycle management.

class MyBenchmark(BaseIsaacBenchmarkAsync):
    async def setUp(self):
        await super().setUp()

    async def test_my_benchmark(self):
        self.set_phase("loading")
        await self.fully_load_stage("path/to/stage.usd")
        await self.store_measurements()

        self.set_phase("benchmark")
        # ... run benchmark ...
        await self.store_measurements()

    async def tearDown(self):
        await super().tearDown()

Functionality#

Phase-Based Measurement#

Both benchmark classes organize measurements into distinct phases, allowing developers to isolate performance characteristics of different execution stages. Each phase can selectively enable frame time and runtime recording based on measurement requirements.

Data Collection#

The framework includes multiple data recorders that capture various performance metrics:

app_frametime: Application frame timing measurements
cpu_continuous: Continuous CPU utilization monitoring
gpu_frametime: GPU frame rendering measurements
hardware: Hardware specifications and capabilities
memory: Memory usage tracking
physics_frametime: Physics simulation timing
render_frametime: Rendering pipeline measurements
runtime: Overall execution time tracking

Stage Loading Integration#

Both classes provide specialized methods for USD stage loading that ensure complete asset and material loading before proceeding with measurements. This eliminates timing inconsistencies caused by background loading operations.

Custom Measurements#

The framework supports storing custom measurement data alongside the standard metrics, enabling domain-specific performance analysis through the store_custom_measurement method.

Configuration#

The extension provides settings for customizing benchmark behavior:

metrics.nvdataflow_default_test_suite_name: Sets the default test suite identifier for organized metric collection
metrics.metrics_output_folder: Specifies the directory for metric output files
metrics.randomize_filename_prefix: Controls whether output filenames include random prefixes to distinguish multiple benchmark runs

Dependencies#

The extension integrates with Isaac Sim’s core systems through several dependencies. It uses **omni.kit.test** to provide the async test case foundation for BaseIsaacBenchmarkAsync, and leverages **omni.physics** and **omni.physics.physx** for physics-related performance measurements.

Enable Extension#

The extension can be enabled (if not already) in one of the following ways:

Command-line interface

Define the next entry as an application argument from a terminal.

APP_SCRIPT.(sh|bat) --enable isaacsim.benchmark.services

Experience/extension configuration

Define the next entry under [dependencies] in an experience (.kit) file or an extension configuration (extension.toml) file.

[dependencies]
"isaacsim.benchmark.services" = {}

Extension Manager UI

Open the Window > Extensions menu in a running application instance and search for isaacsim.benchmark.services. Then, toggle the enable control button if it is not already active.

Extension: {{ extension_version }}

Documentation Generated: Jun 05, 2026

Settings#

Settings Provided by the Extension#

exts.”isaacsim.benchmark.services”.metrics.nvdataflow_default_test_suite_name#

Default Value: “Isaac Sim Benchmarks”
Description: Sets the default test suite name for Isaac Sim benchmarks.

exts.”isaacsim.benchmark.services”.metrics.metrics_output_folder#

Default Value: “” (empty string, currently commented out)
Description: Specifies the output folder path where metrics should be saved.

exts.”isaacsim.benchmark.services”.metrics.randomize_filename_prefix#

Default Value: false
Description: Controls whether to add a randomly generated string as a prefix to the output filename to distinguish runs.

exts.”isaacsim.benchmark.services”.rtf_stability.window_wall_ms#

Default Value: 100.0
Description: Wall-clock window size in milliseconds for each windowed RTF sample collected by the rtf_stability recorder.

exts.”isaacsim.benchmark.services”.rtf_stability.export_window_samples#

Default Value: false
Description: If true, the rtf_stability recorder also emits a list measurement of every windowed RTF sample (larger output files). It always emits mean, stdev, sample count, and mean-anchored stability metrics: fixed ±0.01/±0.10 absolute bands vs the phase mean windowed RTF, max absolute deviation from that mean, and the longest streak of consecutive windows outside the ±0.01 band. Band widths are fixed in code.

Python API#

`BaseIsaacBenchmarkAsync`	Benchmark class for async test cases.
`BaseIsaacBenchmark`	Benchmark class for standalone (synchronous) scripts.

class BaseIsaacBenchmarkAsync(*args: Any, **kwargs: Any)#

Bases: _BaseIsaacBenchmarkCore, AsyncTestCase

Benchmark class for async test cases.

Example:

class MyBenchmark(BaseIsaacBenchmarkAsync):
    async def setUp(self):
        await super().setUp()

    async def test_my_benchmark(self):
        await self.set_phase("loading")
        await self.fully_load_stage("path/to/stage.usd")
        await self.store_measurements()

        await self.set_phase("benchmark", warmup_frames=30)
        # ... run benchmark ...
        await self.store_measurements()

    async def tearDown(self):
        await super().tearDown()

async fully_load_stage(usd_path: str) → None#

Open a stage and wait for it to fully load.

Parameters:: usd_path – Path to USD stage.

Example:

await benchmark.fully_load_stage("/path/to/scene.usd")

async run_warmup(n_frames: int) → None#

Run warmup frames asynchronously without recording any metrics.

Parameters:: n_frames – Number of app update frames to run.

Example:

await benchmark.run_warmup(30)

async setUp( backend_type: str = 'JSONFileMetrics', report_generation: bool = False, workflow_metadata: dict | None = None, recorders: list[str] | None = None, ) → None#

Must be awaited by derived benchmarks to properly set up the benchmark.

Parameters:

backend_type – Type of backend used to collect and print metrics.
report_generation – Whether to generate a formatted report.
workflow_metadata – Metadata describing the benchmark workflow.
recorders – List of recorder names to use, or None for defaults.

async set_phase( phase: str, start_recording_frametime: bool = True, start_recording_runtime: bool = True, warmup_frames: int = 0, ) → None#

Set the active benchmarking phase and start recorders (async).

Parameters:

phase – Name of the phase, used in output.
start_recording_frametime – False to skip frametime recorders.
start_recording_runtime – False to skip runtime recorder.
warmup_frames – Number of app update frames to run before starting recorders.

Example:

await benchmark.set_phase("benchmark", warmup_frames=30)

async store_custom_measurement( phase_name: str, custom_measurement: Measurement, ) → None#

Store a custom measurement for the current benchmark.

Parameters:

phase_name – The phase name to which the measurement belongs.
custom_measurement – The measurement object to store.

Example:

await benchmark.store_custom_measurement("warmup", custom_measurement)

async store_measurements() → None#

Store measurements and metadata collected during the previous phase.

Example:

await benchmark.store_measurements()

async tearDown() → None#: Tear down the benchmark and finalize metrics.

class BaseIsaacBenchmark( benchmark_name: str = 'BaseIsaacBenchmark', backend_type: str = 'OmniPerfKPIFile', report_generation: bool = True, workflow_metadata: dict | None = None, recorders: list[str] | None = None, )#

Bases: _BaseIsaacBenchmarkCore

Benchmark class for standalone (synchronous) scripts.

Parameters:

benchmark_name – Name of benchmark to use in outputs.
backend_type – Type of backend used to collect and print metrics.
report_generation – Whether to generate a formatted report.
workflow_metadata – Metadata describing benchmark.
recorders – List of recorder names to use, or None for defaults.

Example:

benchmark = BaseIsaacBenchmark(benchmark_name="MyBenchmark", workflow_metadata={"metadata": []})
benchmark.set_phase("loading")
# load stage, configure sim, etc.
benchmark.store_measurements()
benchmark.set_phase("benchmark")
# run benchmark
benchmark.store_measurements()
benchmark.stop()

fully_load_stage(usd_path: str) → None#

Load a USD stage and block until it is fully loaded.

Parameters:: usd_path – Path to USD stage.

Example:

benchmark.fully_load_stage("/path/to/scene.usd")

run_warmup(n_frames: int) → None#

Run warmup frames without recording any metrics.

Call before set_phase to let the renderer, physics, and JIT pipelines stabilise so that the subsequent phase captures only steady-state behaviour.

Parameters:: n_frames – Number of app update frames to run.

Example:

benchmark.run_warmup(30)
benchmark.set_phase("benchmark")

set_phase( phase: str, start_recording_frametime: bool = True, start_recording_runtime: bool = True, warmup_frames: int = 0, ) → None#

Set the active benchmarking phase and start recorders.

Parameters:

phase – Name of the phase, used in output.
start_recording_frametime – False to skip frametime recorders.
start_recording_runtime – False to skip runtime recorder.
warmup_frames – Number of app update frames to run before starting recorders. Use this instead of statistical outlier trimming to exclude startup transients from measurements.

Raises:

RuntimeError – If the benchmark context or recorders are not initialized.

Example:

benchmark.set_phase("loading", start_recording_frametime=False)
benchmark.set_phase("benchmark", warmup_frames=30)

stop() → None#

Stop benchmarking and write accumulated metrics to file.

Example:

benchmark.stop()

store_custom_measurement( phase_name: str, custom_measurement: Measurement, ) → None#

Store a custom measurement for the current benchmark.

Parameters:

phase_name – The phase name to which the measurement belongs.
custom_measurement – The measurement object to store.

Example:

benchmark.store_custom_measurement("warmup", custom_measurement)

store_measurements() → None#

Store measurements and metadata collected during the previous phase.

Example:

benchmark.store_measurements()

Python API

[isaacsim.benchmark.services] Benchmark Services#

Overview#

Key Components#

BaseIsaacBenchmark#

BaseIsaacBenchmarkAsync#

Functionality#

Phase-Based Measurement#

Data Collection#

Stage Loading Integration#

Custom Measurements#

Configuration#

Dependencies#

Enable Extension#

Settings#

Settings Provided by the Extension#

exts.”isaacsim.benchmark.services”.metrics.nvdataflow_default_test_suite_name#

exts.”isaacsim.benchmark.services”.metrics.metrics_output_folder#

exts.”isaacsim.benchmark.services”.metrics.randomize_filename_prefix#

exts.”isaacsim.benchmark.services”.rtf_stability.window_wall_ms#

exts.”isaacsim.benchmark.services”.rtf_stability.export_window_samples#

Python API#

`BaseIsaacBenchmark`#

`BaseIsaacBenchmarkAsync`#