Data Generation with MobilityGen#

MobilityGen is a toolset built on NVIDIA Isaac Sim that enables you to easily generate and collect data for mobile robots.

../_images/isim_5.0_replicator_tut_gui_mobility_gen_robots.png ../_images/isim_5.0_replicator_tut_gui_mobility_gen_screencast.gif

MobilityGen supports:

  • Many robot types

    • Differential drive - Jetbot, Carter

    • Quadruped - Spot

    • Humanoid - H1

  • Many data collection methods

    • Manual - Keyboard Teleoperation, Gamepad Teleoperation

    • Automated - Random Accelerations, Random Path Following

Generate Data with MobilityGen#

Step 1 - Build an Occupancy Map#

You must create an occupancy map of your environment.

This tutorial uses an example warehouse scene.

  1. Load the warehouse stage:

    1. Select File > Open.

    2. Enter the following URL under File name:

      http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/Environments/Simple_Warehouse/warehouse_multiple_shelves.usd

    3. Click Open File.

  2. Create the occupancy map”

    1. Select Tools > Robotics > Occupancy Map to open the Occupancy Map extension.

    2. In the Occupancy Map window set Origin to:

      • X: 2.0

      • Y: 0.0

      • Z: 0.0

    3. In the Occupancy Map window set Upper Bound to:

      • X: 10.0

      • Y: 20.0

      • Z: 2.0 (Assumes the robot can move under 2 meter overpasses)

    4. In the Occupancy Map window set Lower Bound to:

      • X: -14.0

      • Y: -18.0

      • Z: 0.1 (Assume the robot can move over 5cm bumps)

    5. Click Calculate to generate the occupancy map.

    6. Click Visualize Image to view the occupancy map.

    7. In the Visualization window under Rotate Image select 180.

    8. In the Visualization window under Coordinate Type select ROS Occupancy Map Parameters File YAML.

    9. Click Regenerate Image.

    10. Copy the YAML text generated to your clipboard.

    11. In a text editor of choice, create a new file named ~/MobilityGenData/maps/warehouse_multiple_shelves/map.yaml.

      Please note: On windows replace ~ with a directory of choice.

    12. Paste the YAML text copied from the Visualization window into the created file.

    13. Edit the line image: warehouse_multiple_shelves.png to read image: map.png.

    14. Save the file.

    15. Back in the Visualization window click Save Image.

    16. In the tree explorer open the folder ~/MobilityGenData/maps/warehouse_multiple_shelves.

    17. Under the file name enter map.png.

    18. Click save.

That’s it! Verify that you now have a folder named ~/MobilityGenData/maps/warehouse_multiple_shelves/ with a file named map.yaml and map.png inside.

Step 2 - Record a Trajectory#

After creating a map of the environment, you can generate data with MobilityGen:

  1. Enable the MobilityGen UI extension.

    1. Navigate to Window > Extensions and search for MobilityGen UI.

    2. Click the toggle switch for the MobilityGen UI extension.

  2. Build the scenario:

    1. In the MobilityGen window under Stage paste the following USD:

      http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/Environments/Simple_Warehouse/warehouse_multiple_shelves.usd

    2. In the MobilityGen window under Occupancy Map enter the path to the map.yaml file created previously.

      ~/MobilityGenData/maps/warehouse_multiple_shelves/map.yaml

    3. Under the Robot dropdown select H1Robot.

    4. Under the Scenario dropdown select KeyboardTeleoperationScenario.

    5. Click Build.

      After a few seconds, verify that you see the scene and occupancy map appear.

  3. Test drive the robot using the following keys:

    • W - Move forward

    • A - Turn left

    • S - Move backwards

    • D - Turn right

  4. Start recording:

    1. Click Start recording to start recording a log.

    2. Move the robot around.

    3. Click Stop recording to stop recording.

The data is now recorded to ~/MobilityGenData/recordings by default.

Step 3 - Replay and Render#

After recording a trajectory, which includes data, like robot poses, you can now replay the scenario.

To do this, use the replay_directory.py Python script that ships with Isaac Sim.

To run the script call the following from inside the Isaac Sim directory:

./python.sh standalone_examples/replicator/mobility_gen/replay_directory.py --render_interval 40 --enable isaacsim.replicator.mobility_gen.examples

After the script finishes, verify that you have a folder ~/MobilityGenData/replays, which contains the rendered sensor data.

You can open this folder to explore the data.

For examples on ingesting the data for visualization and model training, review the open source MobilityGen GitHub Repository.

../_images/isim_5.0_replicator_tut_gui_mobility_gen_gradio_gui.png

Tips#

Tip 1 - How to generate procedural data#

Generating procedural mobility data with MobilityGen is done very similar to the basic teleoperation workflow above.

To generate procedural data

  1. Follow Step 1 - Build an occupancy map above to create an occupancy map of the environment

  2. Follow Step 2 - Record a trajectory above, but select RandomPathFollowingScenario instead of KeyboardTeleoperationScenario - You no longer need to manually teleoperate the robot. When the scenario is built, it will run and reset automatically. - You do need to hit “start recording” to enable recording to disk. However, when the scenario resets, a new recording will be created automatically. - You should see recordings collected in the ~/MobilityGenData folder the same as above.

  3. Follow Step 3 - Replay and render above to render the sensor data from the recorded trajectories.

That’s it! The process for other procedural scenarios (like RandomAccelerationScenario) is similar.

Tip 2 - How to add a custom robot#

You can implement a new robot for use with MobilityGen. This involves editing the robots.py file in the MobilityGen Examples extension.

The general workflow is as follows:

  1. Open the robots.py file in an editor of choice. This is located at <isaac sim path>/exts/isaacsim.replicator.mobility_gen.examples/isaacsim/replicator/mobility_gen/examples/robots.py

  2. Create a new class that subclasses the MobilityGenRobot class. Alternatively, if your robot fits one of the existing implementations (like WheeledMobilityGenRobot), you may subclass that.

    • We recommend starting by reviewing an existing robot implementation in robots.py as the easiest way to get started. A good way to start is simply by tweaking an existing robot.

  3. If you’re starting from scratch, implement the required abstract methods of MobilityGenRobot class

    • Implement the build() method. This method is responsible for adding the robot to the USD stage.

    • Implement the write_action() method. This method takes as input a linear and angular velocity command and performs any control logic.

    • Overwrite common class parameters (like physics_dt).

  4. Register the robot class by using the ROBOT.register() decorator. This makes the custom robot discoverable by MobilityGen.

After implementing this in the file above, save the file.

When you restart Isaac Sim, the new robot should be registered and appear in the MobilityGen UI, and ready for data collection.

Please note, since the registration of a new robot requires editing the Isaac Sim build file, you will want to make a copy of your robot.py externally so you do not lose it.

Next Steps#

That’s it for this tutorial!

In this tutorial, you:

  1. Built an occupancy map for use with MobilityGen.

  2. Recorded a MobilityGen trajectory using the H1 robot with keyboard Teleoperation.

  3. Rendered sensor data based on the recorded trajectory.

As next steps, try recording data:

  • for a different robot (for example: Spot)

  • using a different scenario (for example: Random Path Following)