RTX Sensor Annotators

The isaacsim.sensors.rtx extension uses Omniverse Replicator to provide Annotators for RTX Lidar and Radar data collection. Annotators can be attached to render products, attached to OmniSensor prims (eg. OmniLidar or OmniRadar); for example, when run in Script Editor, the following snippet creates an OmniLidar prim at /lidar, a render product for the sensor, and attaches an IsaacExtractRTXSensorPointCloudNoAccumulator Annotator to the render product.

import omni
import omni.replicator.core as rep
from pxr import Gf

# Create an OmniLidar prim at prim path /lidar
_, sensor = omni.kit.commands.execute(
    "IsaacSensorCreateRtxLidar",
    translation=Gf.Vec3d(0.0, 0.0, 0.0),
    orientation=Gf.Quatd(1.0, 0.0, 0.0, 0.0),
    path="/lidar",
)

# Create a render product for the sensor.
render_product = rep.create.render_product(sensor.GetPath(), resolution=(1024, 1024))

# Create an annotator
annotator = rep.AnnotatorRegistry.get_annotator("IsaacExtractRTXSensorPointCloudNoAccumulator")

# Attach the render product after the annotator is initialized.
annotator.attach([render_product.path])

Annotators

Each isaacsim.sensors.rtx Annotator is associated with a specific isaacsim.sensors.rtx OmniGraph node; refer to that documentation for more information on the annotator’s outputs. This section details assumptions and limitations for each annotator.

Annotators support returning data on device where possible. Setting --/app/sensors/nv/lidar/outputBufferOnGPU=true or --/app/sensors/nv/radar/outputBufferOnGPU=true will cause the annotator to return data in GPU memory.

Note

In Isaac Sim 5.0, several existing isaacsim.sensors.rtx Annotators were removed in favor of simpler Annotators that can handle output from the new OmniLidar or OmniRadar prims, in addition to the deprecated Camera-prim-based workflows. See Deprecated Annotators for details.

Warning

If the Lidar rotation rate is slower than the frame rate, data from Annotators for accumulated Lidar scans will contain returns from multiple frames. If the Lidar prim moves between frames, or objects move in the scene, the buffer may contain returns from before the Lidar or object(s) moved, causing points to appear as though they are “dragging” behind objects when viewed with the DebugDrawPointCloud or DebugDrawPointCloudBuffer writers.

IsaacComputeRTXLidarFlatScan

IsaacComputeRTXLidarFlatScan extracts depth and azimuth data from an accumulated 2D RTX Lidar scan.

Note

The IsaacComputeRTXLidarFlatScan Annotator does not support RTX Radar.

Warning

If the annotator is attached to a 3D Lidar (defined as having emitters at nonzero elevation angles), the annotator will not return any data.

Warning

Even if --/app/sensors/nv/lidar/outputBufferOnGPU=true is set, IsaacComputeRTXLidarFlatScanSimulationTime output data will be on host memory.

IsaacExtractRTXSensorPointCloud and IsaacExtractRTXSensorPointCloudNoAccumulator

IsaacExtractRTXSensorPointCloud extracts the GenericModelOutput buffer’s point cloud data into a Cartesian vector data buffer for a full scan of an RTX Lidar or RTX Radar.

IsaacExtractRTXSensorPointCloudNoAccumulator is similar, but only includes one frame’s worth of data. For RTX Radar, the two Annotators are effectively identical, with IsaacExtractRTXSensorPointCloudNoAccumulator running faster.

Note

IsaacExtractRTXSensorPointCloud and IsaacExtractRTXSensorPointCloudNoAccumulator also output the “raw” GenericModelOutput buffer, which can be read manipulated via Python bindings provided in the isaacsim.sensors.rtx extension. See Reading Data from the GenericModelOutput Buffer for details.

Reading Data from the GenericModelOutput Buffer

Note

Isaac Sim 4.5 included the OgnIsaacReadRTXLidarData node, which provided an example of reading data from the GenericModelOutput buffer in Python. This node has been removed as of Isaac Sim 5.0 and replaced by the utility module and functions described below.

The isaacsim.sensors.rtx.generic_model_output Python module provides APIs for inspecting the GenericModelOutput buffer, returned by several annotators as an output named gmoBufferPointer or sensorOutputBuffer.

For an example of reading data from the GenericModelOutput buffer from Isaac Sim, checkout the standalone example located at standalone_examples/api/isaacsim.sensors.rtx/inspect_lidar_metadata.py.

Deprecated Annotators

Several Annotators have been removed and/or replaced by the Annotators described above, as of Isaac Sim 5.0.

New Annotator outputs are not guaranteed to be the same as the outputs of the deprecated Annotators, gmoBufferPointer the table below describes affected Annotators and how to replace them.

Deprecated Isaac Sim 4.5 Annotator	Replacement	Details
IsaacComputeRTXLidarFlatScanSimulationTime	IsaacComputeRTXLidarFlatScan	The new annotator outputs the same data as the old annotator. To get an associated timestamp, use the IsaacReadSimulationTime annotator.
IsaacComputeRTXLidarFlatScanSystemTime	IsaacComputeRTXLidarFlatScan	The new annotator outputs the same data as the old annotator. To get an associated timestamp, use the IsaacReadSystemTime annotator.
RtxSensorCpuIsaacComputeRTXLidarPointCloud	IsaacExtractRTXSensorPointCloudNoAccumulator	The new annotator outputs the same data as the old annotator, excluding azimuth, elevation, and range. These values can be computed from the Cartesian data buffer. The new annotator also automatically supports CPU or GPU output based on the --/app/sensors/nv/lidar/outputBufferOnGPU and --/app/sensors/nv/radar/outputBufferOnGPU settings, rather than Annotator type.
RtxSensorGpuIsaacComputeRTXLidarPointCloud	IsaacExtractRTXSensorPointCloudNoAccumulator	See above.
RtxSensorCpuIsaacComputeRTXRadarPointCloud	IsaacExtractRTXSensorPointCloudNoAccumulator	See above.
RtxSensorGpuIsaacComputeRTXRadarPointCloud	IsaacExtractRTXSensorPointCloudNoAccumulator	See above.
RtxSensorCpuIsaacCreateRTXLidarScanBuffer	IsaacExtractRTXSensorPointCloud	See above.
IsaacReadRTXLidarData	isaacsim.sensors.rtx.read_gmo_data utility.	See Reading Data from the GenericModelOutput Buffer for details.