Add Noise to Camera#

Learning Objectives#

In this example, we will

  • Have a brief introduction regarding adding an augmentation to sensor images

  • Publish image data with noise added

Getting Started#

Prerequisites

Running The Example#

  1. In one terminal source your ROS 2 workspace.

  2. In another terminal with your ROS 2 environment sourced, run the sample script:

    ./python.sh standalone_examples/api/isaacsim.ros2.bridge/camera_noise.py

    Once the scene finishes loading you should see the viewport scanning a warehouse scene counterclockwise.

  3. In a new terminal with your ROS environment sourced, open an empty rviz window by running rviz on the command line.

  4. Add a Image window by clicking on “Add” on the bottom left. In the pop-up window, under the “By display type” tab, select “Image” and click “OK”.

  5. A new image window will appear somewhere on your rviz screen, along with a menu item labeled “Image” in the Display window. Dock the image window somewhere convenient.

  6. Expand the Image in the Display menu and change the “Image Topic” to /rgb_augmented. A slightly noisy version of the image in Isaac Sim should now be showing in the Rviz image window.

    ../_images/isim_5.0_ros_tut_gui_ros2_camera_noise.gif

Code Explained#

The first step is to set the camera on the render product we want to use for capturing data. There are APIs to set the camera on the viewport but there are also lower level APIs that use the render product prim directly. Both achieve the same, in this case because we are already working with the render product path, we use set_camera_prim_path for illustrative purposes

# grab our render product and directly set the camera prim
render_product_path = get_active_viewport().get_render_product_path()
set_camera_prim_path(render_product_path, CAMERA_STAGE_PATH)

There are several methods for defining an augmentation within a sensor pipeline.

The numpy and omni.warp kernel options are demonstrated below to define a very simple noise function. For brevity there are no out of bounds checks for the color values

# GPU Noise Kernel for illustrative purposes
@wp.kernel
def image_gaussian_noise_warp(
    data_in: wp.array(dtype=wp.uint8, ndim=3),
    data_out: wp.array(dtype=wp.uint8, ndim=3),
    seed: int,
    sigma: float = 0.5,
):
    i, j = wp.tid()
    state = wp.rand_init(seed, wp.tid())
    data_out[i, j, 0] = wp.uint8(wp.int32(data_in[i, j, 0]) + wp.int32(255.0 * sigma * wp.randn(state)))
    data_out[i, j, 1] = wp.uint8(wp.int32(data_in[i, j, 1]) + wp.int32(255.0 * sigma * wp.randn(state)))
    data_out[i, j, 2] = wp.uint8(wp.int32(data_in[i, j, 2]) + wp.int32(255.0 * sigma * wp.randn(state)))

# CPU noise kernel
def image_gaussian_noise_np(data_in: np.ndarray, seed: int, sigma: float = 25.0):
    np.random.seed(seed)
    return data_in + sigma * np.random.randn(*data_in.shape)

Either of the two functions can be used with rep.Augmentation.from_function() to define an augmentation. In this case we want to augment the output of the existing IsaacConvertRGBAToRGB annotator that we use to convert the standard RGBA output to RGB so that it can be published to ROS

# get render car name for rgb data
rv_rgb = omni.syntheticdata.SyntheticData.convert_sensor_type_to_rendervar(sd.SensorType.Rgb.name)
rgba_to_rgb_annotator = f"{rv_rgb}IsaacConvertRGBAToRGB"

# register new augmented annotator that adds noise to rgb
# the image_gaussian_noise_warp variable can be replaced with image_gaussian_noise_np to use the cpu version
rep.annotators.register(
    name="rgb_gaussian_noise",
    annotator=rep.annotators.augment_compose(
        source_annotator=rgba_to_rgb_annotator,
        augmentations=[rep.Augmentation.from_function(image_gaussian_noise_warp, seed=1234, sigma=25)],
    ),
)

Note

seed is an optional predefined Replicator Augmentation argument that can be used with both Python and warp functions. If set to None or < 0, it will use Replicator’s global seed together with the node identifier to produce a repeatable unique seed. When used with warp kernels, the seed is used to initialize a random number generator that produces a new integer seed value for each warp kernel call.

Finally we use the existing ROS2PublishImage writer and replace the IsaacConvertRGBAToRGB annotator with our new augmented rgb_gaussian_noise annotator that combines the two operations. Once we replace the annotator, we initialize our replicator writer, and attach it to the render product to begin capturing and publishing data to ROS.

# replace the existing IsaacConvertRGBAToRGB annotator with the new noise augmented annotator
writer = rep.writers.get(f"{rv_rgb}" + "ROS2PublishImage")
writer.annotators[0] = "rgb_gaussian_noise"
writer.initialize(topicName="rgb_augmented", frameId="sim_camera")
writer.attach([render_product_path])

Summary#

This tutorial covered the basics of adding an augmentation into an existing ROS sensor pipeline and adding noise to the RGB sensor output.

Next Steps#

Continue on to the next tutorial in our ROS Tutorials series, Publishing Camera’s Data.