Grasping Synthetic Data Generation#

This tutorial introduces the isaacsim.replicator.grasping extension and its associated UI, isaacsim.replicator.grasping.ui. These tools provide a comprehensive workflow for generating synthetic grasping datasets in Isaac Sim.

Learning Objectives#

After completing this tutorial, you will be able to:

  • Understand the core components and data flow of the Grasping SDG extension.

  • Navigate and utilize the Grasping SDG UI to configure and run grasp generation workflows.

  • Define gripper properties, joint states, and multi-step grasp phases.

  • Configure object properties and grasp pose sampling parameters.

  • Execute and interpret the results of physics-based grasp evaluations.

  • Manage grasping configurations using YAML files for saving, loading, and sharing setups.

The extensions are automatically loaded in Isaac Sim, and the UI window can be opened from the main menu via Tools > Replicator > Grasping.

Grasping workflow overview

Getting Started#

Before proceeding, it is recommended to familiarize yourself with:

  • Simulation Fundamentals: For understanding physics simulation concepts and gripper rigging (e.g., drive joints).

  • Grasp Editor: This tutorial covers related concepts and provides a foundation for grasp definition.

Note

The grasp sampler requires the libspatialindex library. If you see related warnings please install it (e.g., on Ubuntu: sudo apt-get install libspatialindex-dev).

This tutorial utilizes an example stage that includes a pre-configured gripper and objects suitable for grasping exercises. You can find this stage at:

https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0
/Isaac/Samples/Replicator/Stage/sdg_grasping_xarm.usd

The stage asset can be found in the Content Browser under Isaac Sim > Samples > Replicator > Stage > sdg_grasping_xarm.usd, or can be loaded using by inserting the whole URL in the path field.

The example stage features a gripper with drive joints and three objects equipped with rigid body physics and colliders. Gravity is disabled for these objects to simplify initial interactions. The Grasping UI window typically docks in the Property panel upon opening.

Example stage with a gripper and objects for grasping

Overview#

The extension is designed to automate the process of finding and evaluating potential grasp poses for a given gripper-object pair. At its core, the workflow revolves around several key components and stages:

  1. Configuration: Defining the specific gripper, the target object, and the parameters that govern how grasps are found and tested.

  2. Grasp Pose Sampling: Algorithms (e.g., antipodal samplers) generate a set of candidate grasp poses around the target object. These poses represent potential ways the gripper might hold the object.

  3. Grasp Execution Phases: For each candidate grasp, a sequence of actions, termed “Grasp Phases” (e.g., move to pre-grasp, close fingers, lift), is simulated. This allows for defining complex, multi-step grasping behaviors analogous to real-world robot actions.

  4. Physics-Based Evaluation: Each phase of the grasp is simulated in the physics engine. The success or failure of the grasp attempt, along with other metrics (like contact forces, object displacement), can be recorded. In its current state the extension saves the gripper state as result from which the grasps can be evaluated.

  5. Data Logging and Management: Successful grasps and their associated parameters are logged. The entire setup can be saved to and loaded from configuration files (YAML format), ensuring reproducibility and facilitating batch processing.

The GraspingManager class is the central Python API orchestrating these steps, while the UI provides an intuitive way to configure and run this pipeline.

UI Window Overview#

The Grasping UI window provides the interface for setting up and running the grasping simulations workflows. It is organized into several sections, each addressing a specific part of the process. The general workflow involves configuring these sections, typically starting with the gripper and object, then defining the evaluation workflow and simulation parameters, and finally managing the overall configuration.

Grasping UI window main interface

Gripper Section#

This section is dedicated to defining the properties and behavior of the gripper, which is fundamental for any grasp attempt.

Grasping UI: Gripper joints configuration

  • Path: Specify the USD path to the root prim of your gripper (e.g., /World/Robot/gripper_base).

  • Joints: Once a gripper is selected, its articulated joints are listed. Here you can:

    • Include/Exclude: Select which joints are actively controlled during the grasp phases. These joints have to be drive joints.

    • Set pre-grasp positions: Define the initial state for each joint, typically an open configuration, before the grasp sequence begins.

    • Toggle visibility between all joints or of type drive (non-mimic) joints.

  • Grasp Phases: This powerful feature allows you to define a sequence of discrete actions that constitute a complete grasp attempt. This is analogous to defining a state machine or a sequence of motion primitives for the gripper.

    Grasping UI: Gripper grasp phases configuration

    For each phase (e.g., “Open”, “Close”), you specify:

    • Target joint positions for the active gripper joints.

    • Simulation step delta time (dt) for the physics steps within this phase.

    • Number of simulation steps to execute for this phase.

    Phases can be reordered, deleted, or simulated individually for debugging. If pre-grasp joint positions adequately prepare the gripper (e.g., fully open), an explicit “Open” phase might be unnecessary.

Object Section#

This section focuses on specifying the target object and configuring how potential grasp poses are generated for it.

Grasping UI: Object and grasp pose sampler configuration

  • Path: The USD path to the target object prim (e.g., /World/MyObject).

  • Grasp Pose Sampler: This configures the algorithm used to find potential grasp poses. This tutorial primarily uses an antipodal grasp sampler (implemented in sampler_utils.py). An antipodal grasp is typically stable for parallel-jaw grippers, involving two contact points on opposite sides of the object. Key parameters include:

    • Number of orientations per grasp axis: How many rotational variations around the primary grasp axis to sample.

    • Gripper standoff distance: The distance from the gripper’s Tool Center Point (TCP) or fingertips to the object surface during the approach phase, crucial for avoiding premature collision.

    • Maximum gripper aperture: The widest opening of the gripper jaws, filtering out grasps that are too wide for the object.

    • Alignment axes for the grasp: Defines local gripper axes to align with object features or the grasp line.

    • Gripper approach direction: The vector along which the gripper moves towards the object.

    • Lateral perturbation (sigma): Adds randomness to the grasp point location along the grasp axis, allowing for exploration around nominal contact points.

    • Random seed: For ensuring reproducible sampling results.

  • Grasp Poses: Manages the set of candidate grasp poses generated by the sampler.

    • Specify the desired number of candidate poses.

    • Clear previously generated poses.

    • Visualize the poses in the viewport (either in world or object-local frames) and cycle through them to inspect their placement.

    The following image shows example grasp poses generated by the antipodal sampler on various objects:

    Resulting grasp poses from the antipodal grasp sampler

  • Trimesh: Provides options for debug visualization of the object’s triangle mesh, which is used internally by the sampler for geometric calculations and collision checks.

Note

The Measure Tool can be useful for determining values like gripper aperture or standoff distance.

Using the Measure Tool for gripper and standoff distances

Workflow Section#

The Workflow section is where you orchestrate the actual grasp evaluation process using the configurations defined in the Gripper and Object sections.

Grasping UI: Workflow configuration (This image might be missing, using a placeholder name)

The system first saves the gripper’s initial pose. Then, for each generated grasp pose selected for evaluation, it sequentially executes the defined grasp phases within the physics simulation. After all phases for a given pose are completed, the outcome (e.g., success based on object stability, contact with target) and other relevant metrics are recorded.

  • Number of Grasps Samples: Specify how many of the generated grasp poses should be evaluated. Use -1 to evaluate all available poses.

  • Output Path: Define the directory and base file name for saving the evaluation results. The results are typically saved in a structured format like YAML, detailing each evaluated grasp and its outcome.

  • Overwrite Results: If enabled, existing result files at the output path will be overwritten. Otherwise, new files will be created (e.g., with incremental numbering) to avoid data loss.

  • Start Workflow: Initiates the grasp evaluation process. The UI will often provide feedback on the progress.

Simulation Section#

This section allows you to fine-tune global parameters that affect how the physics simulation is run during the grasp evaluation.

Grasping UI: Simulation settings

  • Render each simulation step: Control whether the viewport updates after each individual physics step within a grasp phase. Disabling this can speed up the evaluation process significantly for large datasets, with rendering potentially only occurring after each full grasp attempt or phase.

  • Simulate using timeline: Choose between advancing the simulation by stepping the main Isaac Sim timeline or by directly stepping the physics scene. Direct physics steps can offer more precise control for rapid evaluations, while timeline-based simulation might be closer to how a full robot application would run.

  • Isolated physics scene: Optionally specify a path to a Physics Scene prim. If provided, the grasping simulation can be run within this dedicated physics scene, preventing interference from other dynamic objects or physics settings in the main stage. This is useful for ensuring consistent and repeatable grasp evaluations.

Config Section#

The Config section provides the crucial functionality for saving your entire grasping setup to a YAML file and loading it back later. This is essential for reproducibility, sharing configurations, and running batch experiments.

Grasping UI: Configuration management

  • File Path: Specify the path to the YAML configuration file for saving or loading.

  • Config Includes: Selectively choose which components of the setup are included in the save/load operation. This allows for modular configurations. Options typically include:

    • Gripper Path

    • Joint Pregrasp States

    • Grasp Phases

    • Object Path

    • Sampler Parameters

    • Generated Grasp Poses (if you wish to save a specific set of poses)

  • Overwrite Existing File: When saving, this option determines if an existing file at the specified path should be overwritten.

  • Load/Save Buttons: Execute the respective file operations.

This structured UI and configuration system offers detailed control and flexibility for generating diverse grasping datasets.

Configuration File Example#

Below is a snippet illustrating the structure of a YAML configuration file. It can store settings for the gripper, object, sampler, and defined grasp phases. The specific content will depend on which components were selected for inclusion via the ‘Config Includes’ UI options.

grasp_phases:
- joint_drive_targets:
    /World/Grippers/xarm_gripper/joints/drive_joint: 48.0
  name: Close
  simulation_step_dt: 0.016666666666666666
  simulation_steps: 32
gripper_path: /World/Grippers/xarm_gripper
joint_pregrasp_states:
  /World/Grippers/xarm_gripper/joints/drive_joint: 0.0
  /World/Grippers/xarm_gripper/joints/left_finger_joint: 0.0
  # ... (other joints related to the xarm_gripper)
object_path: /World/Objects/_05_tomato_soup_can
sampler_config:
  grasp_align_axis: [0, 1, 0]
  gripper_approach_direction: [0, 0, 1]
  gripper_maximum_aperture: 0.08
  gripper_standoff_fingertips: 0.17
  lateral_sigma: 0.0
  num_candidates: 5
  num_orientations: 1
  orientation_sample_axis: [0, 1, 0]
  random_seed: -1
  sampler_type: antipodal
  verbose: false

Code Example#

The following scripts demonstrates a complete workflow for generating a grasping dataset using the GraspingManager API. This script programmatically performs the steps configurable through the UI: opening a stage, setting up the GraspingManager (potentially by loading a configuration file), generating grasp poses, evaluating these poses via physics simulation, and saving the results. This approach is highly suitable for batch processing or integration into larger robotics workflows. The script can be run directly from the Script Editor or as a Standalone Application.

To run the standalone example from the terminal (on Windows, use python.bat instead of python.sh):

./python.sh standalone_examples/api/isaacsim.replicator.grasping/grasping_workflow_sdg.py
Grasping Synthetic Data Generation Workflow
  1import asyncio
  2import os
  3
  4import omni.kit.app
  5import omni.usd
  6from isaacsim.core.utils.extensions import get_extension_path_from_name
  7from isaacsim.storage.native import get_assets_root_path_async
  8from isaacsim.replicator.grasping.grasping_manager import GraspingManager
  9
 10
 11async def run_example_async(
 12    stage_path,
 13    config_path=None,
 14    sampler_config=None,
 15    physics_scene_path=None,
 16    output_dir=None,
 17    gripper_path=None,
 18    object_prim_path=None,
 19):
 20    assets_root_path = await get_assets_root_path_async()
 21    print(f"Assets root path: {assets_root_path}")
 22    stage_url = assets_root_path + stage_path
 23    print(f"Opening stage: {stage_url}")
 24    await omni.usd.get_context().open_stage_async(stage_url)
 25    stage = omni.usd.get_context().get_stage()
 26
 27    grasping_manager = GraspingManager()
 28
 29    if config_path is not None:
 30        load_status = grasping_manager.load_config(config_path)
 31        print(f"Config load status: {load_status}")
 32
 33    # Make sure the object to grasp is set (either from the config file or from the argument)
 34    if not grasping_manager.get_object_prim_path() and object_prim_path:
 35        grasping_manager.object_path = object_prim_path
 36
 37    if not grasping_manager.get_object_prim_path():
 38        print("Warning: Object to grasp is not set (missing in config and argument). Aborting.")
 39        return
 40
 41    # Make sure the gripper is set (either from the config file or from the argument)
 42    if not grasping_manager.gripper_path and gripper_path:
 43        grasping_manager.gripper_path = gripper_path
 44
 45    if not grasping_manager.gripper_path:
 46        print("Warning: Gripper path is not set (missing in config and argument). Aborting.")
 47        return
 48
 49    # If there are already grasp poses in the configuration, don't generate new ones
 50    if grasping_manager.grasp_locations:
 51        print(
 52            f"Found {len(grasping_manager.grasp_locations)} grasp poses in the configuration file. No new poses will be generated."
 53        )
 54    else:
 55        print("No grasp poses found in configuration, generating new ones...")
 56
 57        # Determine Sampler Configuration
 58        if not (grasping_manager.sampler_config and grasping_manager.sampler_config.get("sampler_type")):
 59            if sampler_config:
 60                grasping_manager.sampler_config = sampler_config.copy()
 61            else:
 62                print(
 63                    "Warning: Sampler configuration is missing or invalid (not in config file and not provided as argument). Aborting pose generation."
 64                )
 65                return
 66
 67        # Generate the grasp poses
 68        success_generation = grasping_manager.generate_grasp_poses()
 69        if not success_generation or not grasping_manager.grasp_locations:
 70            print("Failed to generate grasp poses or no poses were generated.")
 71            return
 72        print(f"Generated {len(grasping_manager.grasp_locations)} new grasp poses.")
 73
 74    # Store the initial gripper pose to be able to restore it after the evaluation
 75    grasping_manager.store_initial_gripper_pose()
 76
 77    print("Evaluating grasp poses...")
 78    poses_to_evaluate = grasping_manager.get_grasp_poses(in_world_frame=True)
 79    if not poses_to_evaluate:
 80        print("No poses available to evaluate..")
 81        return
 82
 83    # Determine Output Path
 84    if not output_dir:
 85        print("Warning: Output path is not defined data will not be saved.")
 86
 87    # Set the output path and overwrite flag
 88    grasping_manager.set_results_output_dir(output_dir)
 89    grasping_manager.set_overwrite_results_output(True)
 90
 91    # Determine Physics Scene Path
 92    physics_scene_path_for_eval = None
 93    if physics_scene_path and stage.GetPrimAtPath(physics_scene_path):
 94        physics_scene_path_for_eval = physics_scene_path
 95    print(f"Physics scene path for evaluation: {physics_scene_path_for_eval}")
 96
 97    await grasping_manager.evaluate_grasp_poses(
 98        grasp_poses=poses_to_evaluate,
 99        render=True,
100        physics_scene_path=physics_scene_path_for_eval,
101        simulate_using_timeline=False,
102    )
103
104    print("Grasping workflow example finished.")
105    grasping_manager.clear()
106
107
108stage_path = "/Isaac/Samples/Replicator/Stage/sdg_grasping_xarm.usd"
109ext_path = get_extension_path_from_name("isaacsim.replicator.grasping")
110config_path = os.path.join(ext_path, "data/gripper_configs/xarm_antipodal_soup_can.yaml")
111output_dir = os.path.join(os.getcwd(), "xarm_antipodal")
112
113asyncio.ensure_future(run_example_async(stage_path=stage_path, config_path=config_path, output_dir=output_dir))
Grasping Synthetic Data Generation Workflow
  1from isaacsim import SimulationApp
  2
  3simulation_app = SimulationApp(launch_config={"headless": False})
  4
  5import asyncio
  6import os
  7
  8import omni.kit.app
  9import omni.usd
 10from isaacsim.core.utils.extensions import get_extension_path_from_name
 11from isaacsim.storage.native import get_assets_root_path
 12
 13# Make sure the grasping extension is loaded and enabled
 14ext_manager = omni.kit.app.get_app().get_extension_manager()
 15if not ext_manager.is_extension_enabled("isaacsim.replicator.grasping"):
 16    ext_manager.set_extension_enabled_immediate("isaacsim.replicator.grasping", True)
 17from isaacsim.replicator.grasping.grasping_manager import GraspingManager
 18
 19
 20def run_example(
 21    stage_path,
 22    config_path=None,
 23    sampler_config=None,
 24    physics_scene_path=None,
 25    output_dir=None,
 26    gripper_path=None,
 27    object_prim_path=None,
 28):
 29    assets_root_path = get_assets_root_path()
 30    print(f"Assets root path: {assets_root_path}")
 31    stage_url = assets_root_path + stage_path
 32    print(f"Opening stage: {stage_url}")
 33    omni.usd.get_context().open_stage(stage_url)
 34    stage = omni.usd.get_context().get_stage()
 35
 36    grasping_manager = GraspingManager()
 37
 38    if config_path is not None:
 39        load_status = grasping_manager.load_config(config_path)
 40        print(f"Config load status: {load_status}")
 41
 42    # Make sure the object to grasp is set (either from the config file or from the argument)
 43    if not grasping_manager.get_object_prim_path() and object_prim_path:
 44        grasping_manager.object_path = object_prim_path
 45
 46    if not grasping_manager.get_object_prim_path():
 47        print("Warning: Object to grasp is not set (missing in config and argument). Aborting.")
 48        return
 49
 50    # Make sure the gripper is set (either from the config file or from the argument)
 51    if not grasping_manager.gripper_path and gripper_path:
 52        grasping_manager.gripper_path = gripper_path
 53
 54    if not grasping_manager.gripper_path:
 55        print("Warning: Gripper path is not set (missing in config and argument). Aborting.")
 56        return
 57
 58    # If there are already grasp poses in the configuration, don't generate new ones
 59    if grasping_manager.grasp_locations:
 60        print(
 61            f"Found {len(grasping_manager.grasp_locations)} grasp poses in the configuration file. No new poses will be generated."
 62        )
 63    else:
 64        print("No grasp poses found in configuration, generating new ones...")
 65
 66        # Determine Sampler Configuration
 67        if not (grasping_manager.sampler_config and grasping_manager.sampler_config.get("sampler_type")):
 68            if sampler_config:
 69                grasping_manager.sampler_config = sampler_config.copy()
 70            else:
 71                print(
 72                    "Warning: Sampler configuration is missing or invalid (not in config file and not provided as argument). Aborting pose generation."
 73                )
 74                return
 75
 76        # Generate the grasp poses
 77        success_generation = grasping_manager.generate_grasp_poses()
 78        if not success_generation or not grasping_manager.grasp_locations:
 79            print("Failed to generate grasp poses or no poses were generated.")
 80            return
 81        print(f"Generated {len(grasping_manager.grasp_locations)} new grasp poses.")
 82
 83    # Store the initial gripper pose to be able to restore it after the evaluation
 84    grasping_manager.store_initial_gripper_pose()
 85
 86    print("Evaluating grasp poses...")
 87    poses_to_evaluate = grasping_manager.get_grasp_poses(in_world_frame=True)
 88    if not poses_to_evaluate:
 89        print("No poses available to evaluate..")
 90        return
 91
 92    # Determine Output Path
 93    if not output_dir:
 94        print("Warning: Output path is not defined data will not be saved.")
 95
 96    # Set the output path and overwrite flag
 97    grasping_manager.set_results_output_dir(output_dir)
 98    grasping_manager.set_overwrite_results_output(True)
 99
100    # Determine Physics Scene Path
101    physics_scene_path_for_eval = None
102    if physics_scene_path and stage.GetPrimAtPath(physics_scene_path):
103        physics_scene_path_for_eval = physics_scene_path
104    print(f"Physics scene path for evaluation: {physics_scene_path_for_eval}")
105
106    grasping_workflow_task = asyncio.ensure_future(
107        grasping_manager.evaluate_grasp_poses(
108            grasp_poses=poses_to_evaluate,
109            render=True,
110            physics_scene_path=physics_scene_path_for_eval,
111            simulate_using_timeline=False,
112        )
113    )
114
115    while not grasping_workflow_task.done():
116        simulation_app.update()
117
118    print("Grasping workflow example finished.")
119    grasping_manager.clear()
120
121
122stage_path = "/Isaac/Samples/Replicator/Stage/sdg_grasping_xarm.usd"
123ext_path = get_extension_path_from_name("isaacsim.replicator.grasping")
124config_path = os.path.join(ext_path, "data/gripper_configs/xarm_antipodal_soup_can.yaml")
125output_dir = os.path.join(os.getcwd(), "xarm_antipodal")
126
127run_example(stage_path=stage_path, config_path=config_path, output_dir=output_dir)
128
129
130simulation_app.close()