[isaacsim.benchmark.services] Benchmark Services

Version: 3.0.5

This extension provides benchmarking utilities

Enable Extension

The extension can be enabled (if not already) in one of the following ways:

Command-line interface

Define the next entry as an application argument from a terminal.

APP_SCRIPT.(sh|bat) --enable isaacsim.benchmark.services

Experience/extension configuration

Define the next entry under [dependencies] in an experience (.kit) file or an extension configuration (extension.toml) file.

[dependencies]
"isaacsim.benchmark.services" = {}

Extension Manager UI

Open the Window > Extensions menu in a running application instance and search for isaacsim.benchmark.services. Then, toggle the enable control button if it is not already active.

API

Python API

BaseIsaacBenchmarkAsync
BaseIsaacBenchmark	Front-end for benchmarking standalone scripts and other non-async snippets.

class BaseIsaacBenchmarkAsync(*args: Any, **kwargs: Any)

Bases: AsyncTestCase

async fully_load_stage(usd_path)

Await this function to open a stage and then wait for it to fully load

Parameters:

usd_path (str) – Path to USD stage.

async setUp(backend_type: str = 'JSONFileMetrics')

Must be awaited by derived benchmarks to properly set up the benchmark

set_phase(

phase: str,

start_recording_frametime: bool = True,

start_recording_runtime: bool = True,

) → None

Sets benchmarking phase. Turns on frametime and runtime collection.

Parameters:

• phase (str) – Name of phase, used in output.

• start_recording_frametime (bool) – False to not start recording frametime at start of phase. Default True.

• start_recording_runtime (bool) – False to not start recording runtime at start of phase. Default True.

async store_custom_measurement(

phase_name: str,

custom_measurement: <module 'isaacsim.benchmark.services.metrics.measurements' from '/builds/omniverse/isaac/omni_isaac_sim/_build/linux-x86_64/release/exts/isaacsim.benchmark.services/isaacsim/benchmark/services/metrics/measurements.py'>,

) → None

Stores the custom measurement specificed in the benchmark.

Parameters:

• phase (str) – The phase name to which the measurement belongs.

• measurement (Measurement) – The measurement object to store.

async store_measurements(

stop_recording_time: bool = True,

) → None

Stores measurements, metadata, and artifacts collected by all recorders during the previous phase. Optionally, ends frametime and runtime collection.

Parameters:

stop_recording_time (bool) – False to not stop recording runtime and frametime at end of phase. Default True.

async tearDown()

Must be awaited by derived benchmarks to properly tear down the benchmark

class BaseIsaacBenchmark(

benchmark_name: str = 'BaseIsaacBenchmark',

backend_type: str = 'OmniPerfKPIFile',

report_generation: bool = True,

workflow_metadata: dict = {},

gpu_frametime: bool = False,

)

Bases: object

Front-end for benchmarking standalone scripts and other non-async snippets.

By default this class will collect hardware performance data (see recorders), broken up by “phase”. Typical structure looks like:

benchmark = BaseIsaacBenchmark(benchmark_name=..., workflow_metadata=...)
benchmark.set_phase("loading")
# load stage, configure sim, etc.
benchmark.store_measurements()
benchmark.set_phase("benchmark")
# Actual code being benchmarked (running the sim for N frames, cloning an object, etc.)
benchmark.store_measurements()
benchmark.stop() # Shuts down benchmark, writes metrics to file

You can set any number of phases.

fully_load_stage(usd_path: str) → None

Loads provided USD stage, blocking execution until it is fully loaded.

Parameters:

usd_path (str) – Path to USD stage.

set_phase(

phase: str,

start_recording_frametime: bool = True,

start_recording_runtime: bool = True,

) → None

Sets benchmarking phase. Turns on frametime and runtime collection.

Parameters:

• phase (str) – Name of phase, used in output.

• start_recording_frametime (bool) – False to not start recording frametime at start of phase. Default True.

• start_recording_runtime (bool) – False to not start recording runtime at start of phase. Default True.

stop()

Stop benchmarking and write accumulated metrics to file.

store_custom_measurement(

phase_name: str,

custom_measurement: <module 'isaacsim.benchmark.services.metrics.measurements' from '/builds/omniverse/isaac/omni_isaac_sim/_build/linux-x86_64/release/exts/isaacsim.benchmark.services/isaacsim/benchmark/services/metrics/measurements.py'>,

) → None

Stores the custom measurement specificed in the benchmark.

Parameters:

• phase (str) – The phase name to which the measurement belongs.

• measurement (Measurement) – The measurement object to store.

store_measurements(

stop_recording_time: bool = True,

) → None

Stores measurements, metadata, and artifacts collected by all recorders during the previous phase. Optionally, ends frametime and runtime collection.

Parameters:

stop_recording_time (bool) – False to not stop recording runtime and frametime at end of phase. Default True.

Settings

Extension Settings

The table list the extension-specific settings.

Setting name	Description	Type	Default value
metrics.nvdataflow_default_test_suite_name	Default test suite name.	str	'Isaac Sim Benchmarks'
metrics.randomize_filename_prefix	Whether to add a randomly generated string as a prefix to the output filename to distinguish runs.	bool	False

The extension-specific settings can be either specified (set) or retrieved (get) in one of the following ways:

Set setting

Command-line interface

Define the key and value of the setting as an application argument from a terminal.

APP_SCRIPT.(sh|bat) --/exts/isaacsim.benchmark.services/SETTING_NAME=SETTING_VALUE

Experience/extension configuration

Define the key and value of the setting under [settings] in an experience (.kit) file or an extension configuration (extension.toml) file.

[settings]
exts."isaacsim.benchmark.services".SETTING_NAME = SETTING_VALUE

By programming

Define the key and value of the setting using the carb framework (in Python).

import carb

settings = carb.settings.get_settings()
settings.set("/exts/isaacsim.benchmark.services/SETTING_NAME", SETTING_VALUE)

Get setting

By programming

Define the key to query the value of the setting using the carb framework (in Python).

import carb

settings = carb.settings.get_settings()
value = settings.get("/exts/isaacsim.benchmark.services/SETTING_NAME")