[isaacsim.replicator.writers] Isaac Sim Replicator Writers#

Version: 1.5.3

Overview#

This extension provides specialized writers for synthetic data generation workflows in Isaac Sim. The extension extends the core Replicator framework with Isaac Sim-specific writers that handle various machine learning training data formats including pose estimation, DOPE (Detection of Pose Estimation), YCB Video dataset, and PyTorch tensors.

Key Components#

Dataset Format Writers#

DOPEWriter generates training data compatible with the DOPE (Detection of Pose Estimation) methodology. This writer processes pose annotations and creates structured datasets for training neural networks to detect and estimate 3D object poses from RGB images.

YCBVideoWriter formats synthetic data according to the YCB Video Dataset specification, a standard benchmark for 6D object pose estimation. The writer handles RGB images, semantic segmentation, depth data, and pose annotations while maintaining compatibility with the original dataset structure.

PoseWriter creates pose estimation datasets in multiple formats with configurable output options. This writer supports visibility thresholds, frame filtering, and debug visualizations to generate clean training datasets for pose estimation tasks.

Visualization and Analysis#

DataVisualizationWriter overlays annotation data onto rendered images for visual verification of synthetic data generation. The writer supports 2D tight bounding boxes, 2D loose bounding boxes, and 3D bounding box visualizations on RGB or normal backgrounds, enabling quick quality assessment of generated training data.

PyTorch Integration#

PytorchWriter integrates directly with PyTorch workflows by converting rendered data (e.g. RGB) from multiple cameras into batched PyTorch tensors. This writer works in conjunction with PytorchListener to provide real-time tensor data streaming, allowing machine learning pipelines to consume synthetic data without intermediate file storage.

PytorchListener acts as an observer that receives batched tensor data from PytorchWriter. The listener provides methods to directly retrieve data sent to the writer as PyTorch tensors without needing to access data stored by omni.replicator’s BackendDispatch, enabling seamless integration with training loops.

Integration#

The extension integrates with omni.replicator.core to extend the base Writer class functionality. Each writer registers specific annotators required for their data format and handles the processing pipeline from raw render products to formatted output files or tensor streams.

Writers support both local filesystem and S3 storage backends, with configurable output formats and frame processing options. The extension automatically registers all writers with the Replicator system at startup, making them available through the WriterRegistry interface.

Pytorch Online Writer and Listener#

The PytorchWriter and PytorchListener are APIs for using omni.replicator’s writer API to retrieve various data such as RGB from the specified cameras (supports multiple cameras) and provides them to the user in both default format (e.g.: PNG for RGB data) and batched pytorch tensors. The PytorchListener provides an API to directly retrieve data sent to the PytorchWriter without the need to access the stored by omni.replicator’s BackendDispatch.

Enable Extension#

The extension can be enabled (if not already) in one of the following ways:

Command-line interface

Define the next entry as an application argument from a terminal.

APP_SCRIPT.(sh|bat) --enable isaacsim.replicator.writers

Experience/extension configuration

Define the next entry under [dependencies] in an experience (.kit) file or an extension configuration (extension.toml) file.

[dependencies]
"isaacsim.replicator.writers" = {}

Extension Manager UI

Open the Window > Extensions menu in a running application instance and search for isaacsim.replicator.writers. Then, toggle the enable control button if it is not already active.

Python API#

`DataVisualizationWriter`	Data Visualization Writer.
`DOPEWriter`	Basic writer capable of writing built-in annotator groundtruth.
`PoseWriter`	Pose Writer.
`PytorchListener`	An Observer/Listener that keeps track of updated data sent by the writer.
`PytorchWriter`	A custom writer that uses omni.replicator API to retrieve RGB data via render products.
`YCBVideoWriter`	Writer capable of writing annotator groundtruth in the YCB Video Dataset format.

class DataVisualizationWriter(*args: Any, **kwargs: Any)#

Bases: Writer

Data Visualization Writer.

This writer can be used to visualize various annotator data.

Supported annotators: - bounding_box_2d_tight - bounding_box_2d_loose - bounding_box_3d

Supported backgrounds: - rgb - normals

Parameters:

output_dir – Output directory for the data visualization files forwarded to the backend writer.
bounding_box_2d_tight – If True, 2D tight bounding boxes will be drawn on the selected background (transparent by default).
bounding_box_2d_tight_params – Parameters for the 2D tight bounding box annotator.
bounding_box_2d_loose – If True, 2D loose bounding boxes will be drawn on the selected background (transparent by default).
bounding_box_2d_loose_params – Parameters for the 2D loose bounding box annotator.
bounding_box_3d – If True, 3D bounding boxes will be drawn on the selected background (transparent by default).
bounding_box_3d_params – Parameters for the 3D bounding box annotator.
frame_padding – Number of digits used for the frame number in the file name.

detach()#

Reset the writer state and detaches from the backend.

Resets the frame counter to zero and calls the parent class detach method to properly clean up backend connections.

Returns:: The result of the parent class detach method.

write(data: dict)#

Process annotation data and generates visualization images with overlays.

Iterates through render products and applies visualization overlays (2D/3D bounding boxes) onto background images, then saves the results to the output directory.

Parameters:: data – Annotation data containing render products and their associated annotator data.

BB_2D_LOOSE = 'bounding_box_2d_loose_fast'#: Annotator name for 2D loose bounding box fast annotation.

BB_2D_TIGHT = 'bounding_box_2d_tight_fast'#: Annotator name for 2D tight bounding box fast annotation.

BB_3D = 'bounding_box_3d_fast'#: Annotator name for 3D bounding box fast annotation.

SUPPORTED_BACKGROUNDS = ['rgb', 'normals']#: List of supported background types for visualization overlays.

class DOPEWriter(*args: Any, **kwargs: Any)#

Bases: Writer

Basic writer capable of writing built-in annotator groundtruth.

Deprecated since version This: class has been deprecated and will be removed in the next major release.

Parameters:

output_dir – Output directory string that indicates the directory to save the results. If use_s3 == True, this will be the bucket name.
class_name_to_index_map – Dictionary mapping class names to their corresponding indices.
semantic_types – List of semantic types to consider when filtering annotator data.
image_output_format – String that indicates the format of saved RGB images.
use_s3 – Boolean value that indicates whether output will be written to s3 bucket.
bucket_name – Name of the S3 bucket when using S3 output.
endpoint_url – Custom endpoint URL for S3 service.
s3_region – AWS region for S3 bucket.

Example

>>> import omni.replicator.core as rep
>>> camera = rep.create.camera()
>>> render_product = rep.create.render_product(camera, (512, 512))
>>> writer = rep.WriterRegistry.get("DOPEWriter")
>>> import carb
>>> tmp_dir = carb.tokens.get_tokens_interface().resolve("${temp}/rgb")
>>> writer.initialize(output_dir=tmp_dir, class_name_to_index_map=class_name_to_index_map)
>>> writer.attach([render_product])
>>> rep.orchestrator.run()

is_last_frame_valid() → bool#

Check if the last frame was valid (training data was present).

Returns:: True if the last frame was valid, False otherwise.
Return type:: bool

register_pose_annotator()#

setup_writer(writer_config: dict)#

Initialize writer and attach render product.

Parameters:

config_data – A dictionary containing the general configurations for the script.
writer_config – A dictionary containing writer-specific configurations.

write(data: dict)#

Write function called from the OgnWriter node on every frame to process annotator output.

Parameters:: data – A dictionary containing the annotator data for the current frame.

class PoseWriter(*args: Any, **kwargs: Any)#

Bases: Writer

Pose Writer.

Parameters:

output_dir – Output directory string that indicates the directory to save the results when no backend is supplied.
use_subfolders – If True, the writer will create subfolders for each render product, otherwise all data is saved in the same folder.
visibility_threshold – Objects with visibility below this threshold will be skipped.
skip_empty_frames – If True, the writer will skip frames that do not have visible objects.
write_debug_images – If True, the writer will include rgb images overlaid with the projected 3d bounding boxes.
frame_padding – Pad the frame number with leading zeroes.
format – Specifies which format the data will be outputted as.
use_s3 – If True, uses S3 backend for data storage.
s3_bucket – S3 bucket name when using S3 backend.
s3_endpoint_url – S3 endpoint URL when using S3 backend.
s3_region – S3 region when using S3 backend.
backend – Backend instance used to handle I/O scheduling. Prefer using this argument, for example with rep.backends.get("DiskBackend").
image_output_format – Image file format for RGB and debug outputs (jpeg, jpg, png or exr).

detach()#: Clear the writer state by resetting the frame counter to zero.

get_current_frame_id()#

Current frame ID counter.

Returns:: The current frame ID counter value.

write(data: dict)#

Write pose data for all render products in the frame.

Parameters:: data – Dictionary containing render product data with annotator information.

BB3D_ANNOT_NAME = 'bounding_box_3d_fast'#: Name of the 3D bounding box annotator used for retrieving object pose and geometry data.

CAM_PARAMS_ANNOT_NAME = 'camera_params'#: Name of the camera parameters annotator used for retrieving camera intrinsic and extrinsic data.

CUBOID_EDGE_COLORS = {'back': 'blue', 'connecting': 'green', 'front': 'red'}#: Colors used for drawing different types of cuboid edges in debug visualization images.

CUBOID_KEYPOINTS_ORDER_DEFAULT = ['Center', 'LDB', 'LDF', 'LUB', 'LUF', 'RDB', 'RDF', 'RUB', 'RUF']#: Default ordering of cuboid keypoints for 3D bounding box representation.

CUBOID_KEYPOINT_COLORS = ['white', 'red', 'green', 'blue', 'yellow', 'cyan', 'magenta', 'orange', 'purple']#: Colors used for drawing cuboid keypoints in debug visualization images.

CUBOID_KEYPOINT_ORDER_DOPE = ['LUF', 'RUF', 'RDF', 'LDF', 'LUB', 'RUB', 'RDB', 'LDB', 'Center']#: DOPE format ordering of cuboid keypoints for 3D bounding box representation.

RGB_ANNOT_NAME = 'rgb'#: Name of the RGB annotator used for retrieving color image data.

SUPPORTED_FORMATS = {'centerpose', 'dope'}#: Set of supported output formats for pose data.

class PytorchListener#

Bases: object

An Observer/Listener that keeps track of updated data sent by the writer. Is passed in the.

itialization of a PytorchWriter at which point it is pinged by the writer after any data is passed to the writer.

Deprecated since version 1.5.0: This class is deprecated and will be removed in a future version. No replacement is provided.

get_rgb_data() → torch.Tensor | None#

Return RGB data as a batched tensor from the current data stored.

Returns:: Images in batched pytorch tensor form.

write_data(data: dict)#

Update the existing data in the listener with the new data provided.

Parameters:: data – New data retrieved from writer.

class PytorchWriter(*args: Any, **kwargs: Any)#

Bases: Writer

A custom writer that uses omni.replicator API to retrieve RGB data via render products.

and formats them as tensor batches. The writer takes a PytorchListener which is able to retrieve pytorch tensors for the user directly after each writer call.

Deprecated since version 1.5.0: This class is deprecated and will be removed in a future version. No replacement is provided.

Parameters:

listener – A PytorchListener that is sent pytorch batch tensors at each write() call.
output_dir – Directory in which rgb data will be saved in PNG format by the backend dispatch. If not specified, the writer will not write rgb data as png and only ping the listener with batched tensors.
tiled_sensor – Whether to use tiled sensor mode.
device – Device in which the pytorch tensor data will reside. Can be “cpu”, “cuda”, or any other format that pytorch supports for devices.

write(data: dict)#

Send data captured by the attached render products to the PytorchListener and will write data to.

the output directory if specified during initialization.

Parameters:: data – Data to be pinged to the listener and written to the output directory if specified.

class YCBVideoWriter(*args: Any, **kwargs: Any)#

Bases: Writer

Writer capable of writing annotator groundtruth in the YCB Video Dataset format.

Deprecated since version This: class has been deprecated and will be removed in the next major release.

Parameters:

output_dir – Output directory string that indicates the directory to save the results.
num_frames – Total number of frames to be generated.
semantic_types – List of semantic types to consider when filtering annotator data.
rgb – Boolean value that indicates whether the rgb annotator will be activated and the data will be written or not.
bounding_box_2d_tight – Boolean value that indicates whether the bounding_box_2d_tight annotator will be activated and the data will be written or not.
semantic_segmentation – Boolean value that indicates whether the semantic_segmentation annotator will be activated and the data will be written or not.
distance_to_image_plane – Boolean value that indicates whether the distance_to_image_plane annotator will be activated and the data will be written or not.
image_output_format – String that indicates the format of saved RGB images.
pose – Boolean value that indicates whether the pose annotator will be activated and the data will be written or not.
class_name_to_index_map – Mapping between semantic label and index used in the YCB Video Dataset. These indices are used in the ‘cls_indexes’ field of the generated meta.mat file, in addition to being used to color the semantic segmentation (where pixels are colored according to the grayscale class index).
factor_depth – Depth scaling factor used in the YCB Video Dataset.
intrinsic_matrix – Camera intrinsic matrix. shape is (3, 3).

is_last_frame_valid() → bool#

Check if the last frame was valid (training data was present).

Returns:: True if the last frame was valid, False otherwise.

register_pose_annotator()#

Register the annotators for the specific writer.

Parameters:: config_data – A dictionary containing the configuration data for the current writer.

save_mesh_vertices( coord_prim: pxr.Usd.Prim, model_name: str, output_folder: str, )#

Create points.xyz file representing vertices of the mesh_prim, defined in the frame of the coord_prim. The.

points.xyz file will be saved in the output_folder/data/models/model_name/ directory.

Parameters:

mesh_prim (UsdGeom.Mesh) – mesh prim to get the vertice points.
coord_prim (Usd.Prim) – prim’s coordinate used to define the vertices with respect to.
model_name (str) – name of the part to get the vertices of. Note: This corresponds to the name used for the part in the YCB Video Dataset, and is unrelated to the name of the part in the scene.
output_folder (str) – path of the base output directory.

setup_writer(writer_config: dict)#

Initialize writer and attach render product.

Parameters:

config_data – A dictionary containing the general configurations for the script.
writer_config – A dictionary containing writer-specific configurations.

write(data: dict)#

Write function called from the OgnWriter node on every frame to process annotator output.

Parameters:: data – A dictionary containing the annotator data for the current frame.

Omnigraph Nodes#

The extension exposes the following Omnigraph nodes:

Dope
Pose

Python API