Is this page helpful?

Add Noise to Camera#

Learning Objectives#

In this example, we will:

  • Have a brief introduction regarding adding an augmentation to sensor images

  • Publish image data with noise added

Getting Started#

Prerequisites

Running the Example#

  1. In one terminal source your ROS 2 workspace.

  2. In another terminal with your ROS 2 environment sourced, run the sample script:

    ./python.sh standalone_examples/api/isaacsim.ros2.bridge/camera_noise.py

    After the scene finishes loading, verify that you observe the viewport scanning a warehouse scene counterclockwise.

  3. In a new terminal with your ROS environment sourced, open an empty RViz window by running rviz2 on the command line.

  4. Add an Image window by clicking on “Add” on the bottom left. In the pop-up window, under the “By display type” tab, select “Image” and click “OK”.

  5. A new image window will appear somewhere on your RViz screen, along with a menu item labeled “Image” in the Display window. Dock the image window somewhere convenient.

  6. Expand the Image in the Display menu and change the “Image Topic” to /rgb_augmented. Verify that a slightly noisy version of the image in Isaac Sim is now showing in the RViz image window.

    ../_images/isim_5.0_ros_tut_gui_ros2_camera_noise.gif

Code explained#

First, set the camera on the render product used for capturing data. You can set the camera through the viewport API, but here we use set_camera_prim_path on the render product directly.

# Grab the render product and set the camera prim
render_product_path = get_active_viewport().get_render_product_path()
set_camera_prim_path(render_product_path, CAMERA_STAGE_PATH)

There are several methods for defining an augmentation within a sensor pipeline:

The numpy and warp kernel options are demonstrated below with a basic noise function. For brevity there are no out-of-bounds checks for color values.

The GPU warp kernel takes RGBA input and produces RGB output, applying per-pixel Gaussian noise:

import warp as wp


@wp.kernel
def image_gaussian_noise_warp(
    data_in: wp.array3d(dtype=wp.uint8), data_out: wp.array3d(dtype=wp.uint8), seed: int, sigma: float = 0.5
):
    i, j = wp.tid()
    dim_i = data_out.shape[0]
    dim_j = data_out.shape[1]
    pixel_id = i * dim_i + j
    state_r = wp.rand_init(seed, pixel_id + (dim_i * dim_j * 0))
    state_g = wp.rand_init(seed, pixel_id + (dim_i * dim_j * 1))
    state_b = wp.rand_init(seed, pixel_id + (dim_i * dim_j * 2))

    data_out[i, j, 0] = wp.uint8(float(data_in[i, j, 0]) + (255.0 * sigma * wp.randn(state_r)))
    data_out[i, j, 1] = wp.uint8(float(data_in[i, j, 1]) + (255.0 * sigma * wp.randn(state_g)))
    data_out[i, j, 2] = wp.uint8(float(data_in[i, j, 2]) + (255.0 * sigma * wp.randn(state_b)))

The equivalent CPU kernel uses numpy:

import numpy as np


def image_gaussian_noise_np(data_in: np.ndarray, seed: int, sigma: float = 25.0):
    np.random.seed(seed)
    return data_in + sigma * np.random.randn(*data_in.shape)

Either function can be passed to rep.Augmentation.from_function() to define an augmentation. The following registers a new rgb_gaussian_noise annotator that composes the noise function on top of the rgb annotator:

import omni.replicator.core as rep

rep.annotators.register(
    name="rgb_gaussian_noise",
    annotator=rep.annotators.augment_compose(
        source_annotator=rep.annotators.get("rgb", device="cuda"),
        augmentations=[
            rep.annotators.Augmentation.from_function(
                image_gaussian_noise_warp, sigma=0.1, seed=1234, data_out_shape=(-1, -1, 3)
            ),
        ],
    ),
)

Note

seed is an optional predefined Replicator Augmentation argument that works with both Python and warp functions. If set to None or < 0, Replicator uses its global seed together with the node identifier to produce a repeatable unique seed. For warp kernels, the seed initializes a random number generator that produces a new integer seed value for each kernel call.

Next, create a custom writer that uses the augmented rgb_gaussian_noise annotator and register it:

rep.writers.register_node_writer(
    name="CustomROS2PublishImage",
    node_type_id="isaacsim.ros2.bridge.ROS2PublishImage",
    annotators=[
        "rgb_gaussian_noise",
        omni.syntheticdata.SyntheticData.NodeConnectionTemplate(
            "IsaacReadSimulationTime", attributes_mapping={"outputs:simulationTime": "inputs:timeStamp"}
        ),
    ],
    category="custom",
)

(
    rep.WriterRegistry._default_writers.append("CustomROS2PublishImage")
    if "CustomROS2PublishImage" not in rep.WriterRegistry._default_writers
    else None
)

Finally, instantiate the writer, configure the ROS topic, and attach the render product. This begins capturing and publishing the augmented image data to ROS:

writer = rep.writers.get("CustomROS2PublishImage")
writer.initialize(topicName="rgb_augmented", frameId="sim_camera")
writer.attach([render_product_path])

Summary#

This tutorial covered the basics of adding an augmentation to the ROS sensor pipeline and adding noise to the RGB sensor output.

Next Steps#

Continue on to the next tutorial in our ROS Tutorials series, Publishing Camera’s Data.