isaaclab.devices

isaaclab.devices#

Sub-package providing interfaces to different teleoperation devices.

Currently, the following categories of devices are supported:

  • Keyboard: Standard keyboard with WASD and arrow keys.

  • Spacemouse: 3D mouse with 6 degrees of freedom.

  • Gamepad: Gamepad with 2D two joysticks and buttons. Example: Xbox controller.

  • OpenXR: Uses hand tracking of index/thumb tip avg to drive the target pose. Gripping is done with pinching.

All device interfaces inherit from the DeviceBase class, which provides a common interface for all devices. The device interface reads the input data when the DeviceBase.advance() method is called. It also provides the function DeviceBase.add_callback() to add user-defined callback functions to be called when a particular input is pressed from the peripheral device.

Classes

DeviceBase

An interface class for teleoperation devices.

RetargeterBase

Base interface for input data retargeting.

Se2Gamepad

A gamepad controller for sending SE(2) commands as velocity commands.

Se3Gamepad

A gamepad controller for sending SE(3) commands as delta poses and binary command (open/close).

Se2Keyboard

A keyboard controller for sending SE(2) commands as velocity commands.

Se3Keyboard

A keyboard controller for sending SE(3) commands as delta poses and binary command (open/close).

Se2SpaceMouse

A space-mouse controller for sending SE(2) commands as delta poses.

Se3SpaceMouse

A space-mouse controller for sending SE(3) commands as delta poses.

OpenXRDevice

An OpenXR-powered device for teleoperation and interaction.

isaaclab.devices.openxr.retargeters.GripperRetargeter

Retargeter specifically for gripper control based on hand tracking data.

isaaclab.devices.openxr.retargeters.Se3AbsRetargeter

Retargets OpenXR hand tracking data to end-effector commands using absolute positioning.

isaaclab.devices.openxr.retargeters.Se3RelRetargeter

Retargets OpenXR hand tracking data to end-effector commands using relative positioning.

isaaclab.devices.openxr.retargeters.GR1T2Retargeter

Retargets OpenXR hand tracking data to GR1T2 hand end-effector commands.

Modules

isaaclab.devices.openxr.retargeters

Retargeters for mapping input device data to robot commands.

Device Base#

class isaaclab.devices.DeviceBase[源代码]#

An interface class for teleoperation devices.

Derived classes have two implementation options:

  1. Override _get_raw_data() and use the base advance() implementation: This approach is suitable for devices that want to leverage the built-in retargeting logic but only need to customize the raw data acquisition.

  2. Override advance() completely: This approach gives full control over the command generation process, and _get_raw_data() can be ignored entirely.

Methods:

__init__([retargeters])

Initialize the teleoperation interface.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind keyboard.

advance()

Process current device state and return control commands.
__init__(retargeters: list[isaaclab.devices.retargeter_base.RetargeterBase] | None = None)[源代码]#

Initialize the teleoperation interface.

参数:

retargeters – List of components that transform device data into robot commands. If None or empty list, the device will output its native data format.

abstract reset()[源代码]#

Reset the internals.

abstract add_callback(key: Any, func: Callable)[源代码]#

Add additional functions to bind keyboard.

参数:

  • key – The button to check against.

  • func – The function to call when key is pressed. The callback function should not take any arguments.

advance() → torch.Tensor[源代码]#

Process current device state and return control commands.

This method retrieves raw data from the device and optionally applies retargeting to convert it to robot commands.

Derived classes can either: 1. Override _get_raw_data() and use this base implementation, or 2. Override this method completely for custom command processing

返回:

When no retargeters are configured, returns raw device data in its native format. When retargeters are configured, returns a torch.Tensor containing the concatenated outputs from all retargeters.

Retargeter Base#

class isaaclab.devices.RetargeterBase[源代码]#

Base interface for input data retargeting.

This abstract class defines the interface for components that transform raw device data into robot control commands. Implementations can handle various types of transformations including: - Hand joint data to end-effector poses - Input device commands to robot movements - Sensor data to control signals

Methods:

__init__(cfg)

Initialize the retargeter.

retarget(data)

Retarget input data to desired output format.
__init__(cfg: RetargeterCfg)[源代码]#

Initialize the retargeter.

参数:

cfg – Configuration for the retargeter

abstract retarget(data: Any) → Any[源代码]#

Retarget input data to desired output format.

参数:

data – Raw input data to be transformed

返回:

Retargeted data in implementation-specific format

Game Pad#

class isaaclab.devices.Se2Gamepad[源代码]#

基类：DeviceBase

A gamepad controller for sending SE(2) commands as velocity commands.

This class is designed to provide a gamepad controller for mobile base (such as quadrupeds). It uses the Omniverse gamepad interface to listen to gamepad events and map them to robot’s task-space commands.

The command comprises of the base linear and angular velocity: \((v_x, v_y, \omega_z)\).

Key bindings:

Command

Key (+ve axis)

Key (-ve axis)

Move along x-axis

left stick up

left stick down

Move along y-axis

left stick right

left stick left

Rotate along z-axis

right stick right

right stick left

参见

The official documentation for the gamepad interface: Carb Gamepad Interface.

Methods:

__init__(cfg)

Initialize the gamepad layer.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind gamepad.

advance()

Provides the result from gamepad event state.
__init__(cfg: Se2GamepadCfg)[源代码]#

Initialize the gamepad layer.

参数:

  • v_x_sensitivity – Magnitude of linear velocity along x-direction scaling. Defaults to 1.0.

  • v_y_sensitivity – Magnitude of linear velocity along y-direction scaling. Defaults to 1.0.

  • omega_z_sensitivity – Magnitude of angular velocity along z-direction scaling. Defaults to 1.0.

  • dead_zone – Magnitude of dead zone for gamepad. An event value from the gamepad less than this value will be ignored. Defaults to 0.01.

reset()[源代码]#

Reset the internals.

add_callback(key: carb.input.GamepadInput, func: Callable)[源代码]#

Add additional functions to bind gamepad.

A list of available gamepad keys are present in the carb documentation.

参数:

  • key – The gamepad button to check against.

  • func – The function to call when key is pressed. The callback function should not take any arguments.

advance() → torch.Tensor[源代码]#

Provides the result from gamepad event state.

返回:

A tensor containing the linear (x,y) and angular velocity (z).

class isaaclab.devices.Se3Gamepad[源代码]#

基类：DeviceBase

A gamepad controller for sending SE(3) commands as delta poses and binary command (open/close).

This class is designed to provide a gamepad controller for a robotic arm with a gripper. It uses the gamepad interface to listen to gamepad events and map them to the robot’s task-space commands.

The command comprises of two parts:

  • delta pose: a 6D vector of (x, y, z, roll, pitch, yaw) in meters and radians.

  • gripper: a binary command to open or close the gripper.

Stick and Button bindings:

Description

Stick/Button (+ve axis)

Stick/Button (-ve axis)

Toggle gripper(open/close)

X Button

X Button

Move along x-axis

Left Stick Up

Left Stick Down

Move along y-axis

Left Stick Left

Left Stick Right

Move along z-axis

Right Stick Up

Right Stick Down

Rotate along x-axis

D-Pad Left

D-Pad Right

Rotate along y-axis

D-Pad Down

D-Pad Up

Rotate along z-axis

Right Stick Left

Right Stick Right

参见

The official documentation for the gamepad interface: Carb Gamepad Interface.

Methods:

__init__(cfg)

Initialize the gamepad layer.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind gamepad.

advance()

Provides the result from gamepad event state.
__init__(cfg: Se3GamepadCfg)[源代码]#

Initialize the gamepad layer.

参数:

cfg – Configuration object for gamepad settings.

reset()[源代码]#

Reset the internals.

add_callback(key: carb.input.GamepadInput, func: Callable)[源代码]#

Add additional functions to bind gamepad.

A list of available gamepad keys are present in the carb documentation.

参数:

  • key – The gamepad button to check against.

  • func – The function to call when key is pressed. The callback function should not take any arguments.

advance() → torch.Tensor[源代码]#

Provides the result from gamepad event state.

返回:

A 7-element tensor containing:

  • delta pose: First 6 elements as [x, y, z, rx, ry, rz] in meters and radians.

  • gripper command: Last element as a binary value (+1.0 for open, -1.0 for close).

返回类型:

torch.Tensor

Keyboard#

class isaaclab.devices.Se2Keyboard[源代码]#

基类：DeviceBase

A keyboard controller for sending SE(2) commands as velocity commands.

This class is designed to provide a keyboard controller for mobile base (such as quadrupeds). It uses the Omniverse keyboard interface to listen to keyboard events and map them to robot’s task-space commands.

The command comprises of the base linear and angular velocity: \((v_x, v_y, \omega_z)\).

Key bindings:

Command

Key (+ve axis)

Key (-ve axis)

Move along x-axis

Numpad 8 / Arrow Up

Numpad 2 / Arrow Down

Move along y-axis

Numpad 4 / Arrow Right

Numpad 6 / Arrow Left

Rotate along z-axis

Numpad 7 / Z

Numpad 9 / X

参见

The official documentation for the keyboard interface: Carb Keyboard Interface.

Methods:

__init__(cfg)

Initialize the keyboard layer.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind keyboard.

advance()

Provides the result from keyboard event state.
__init__(cfg: Se2KeyboardCfg)[源代码]#

Initialize the keyboard layer.

参数:

  • v_x_sensitivity – Magnitude of linear velocity along x-direction scaling. Defaults to 0.8.

  • v_y_sensitivity – Magnitude of linear velocity along y-direction scaling. Defaults to 0.4.

  • omega_z_sensitivity – Magnitude of angular velocity along z-direction scaling. Defaults to 1.0.

reset()[源代码]#

Reset the internals.

add_callback(key: str, func: Callable)[源代码]#

Add additional functions to bind keyboard.

A list of available keys are present in the carb documentation.

参数:

  • key – The keyboard button to check against.

  • func – The function to call when key is pressed. The callback function should not take any arguments.

advance() → torch.Tensor[源代码]#

Provides the result from keyboard event state.

返回:

Tensor containing the linear (x,y) and angular velocity (z).

class isaaclab.devices.Se3Keyboard[源代码]#

基类：DeviceBase

A keyboard controller for sending SE(3) commands as delta poses and binary command (open/close).

This class is designed to provide a keyboard controller for a robotic arm with a gripper. It uses the Omniverse keyboard interface to listen to keyboard events and map them to robot’s task-space commands.

The command comprises of two parts:

  • delta pose: a 6D vector of (x, y, z, roll, pitch, yaw) in meters and radians.

  • gripper: a binary command to open or close the gripper.

Key bindings:

Description

Key (+ve axis)

Key (-ve axis)

Toggle gripper (open/close)

K

Move along x-axis

W

S

Move along y-axis

A

D

Move along z-axis

Q

E

Rotate along x-axis

Z

X

Rotate along y-axis

T

G

Rotate along z-axis

C

V

参见

The official documentation for the keyboard interface: Carb Keyboard Interface.

Methods:

__init__(cfg)

Initialize the keyboard layer.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind keyboard.

advance()

Provides the result from keyboard event state.
__init__(cfg: Se3KeyboardCfg)[源代码]#

Initialize the keyboard layer.

参数:

cfg – Configuration object for keyboard settings.

reset()[源代码]#

Reset the internals.

add_callback(key: str, func: Callable)[源代码]#

Add additional functions to bind keyboard.

A list of available keys are present in the carb documentation.

参数:

  • key – The keyboard button to check against.

  • func – The function to call when key is pressed. The callback function should not take any arguments.

advance() → torch.Tensor[源代码]#

Provides the result from keyboard event state.

返回:

A 7-element tensor containing:

  • delta pose: First 6 elements as [x, y, z, rx, ry, rz] in meters and radians.

  • gripper command: Last element as a binary value (+1.0 for open, -1.0 for close).

返回类型:

torch.Tensor

Space Mouse#

class isaaclab.devices.Se2SpaceMouse[源代码]#

基类：DeviceBase

A space-mouse controller for sending SE(2) commands as delta poses.

This class implements a space-mouse controller to provide commands to mobile base. It uses the HID-API which interfaces with USD and Bluetooth HID-class devices across multiple platforms.

The command comprises of the base linear and angular velocity: \((v_x, v_y, \omega_z)\).

备注

The interface finds and uses the first supported device connected to the computer.

Currently tested for following devices:

Methods:

__init__(cfg)

Initialize the spacemouse layer.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind spacemouse.

advance()

Provides the result from spacemouse event state.
__init__(cfg: Se2SpaceMouseCfg)[源代码]#

Initialize the spacemouse layer.

参数:

cfg – Configuration for the spacemouse device.

reset()[源代码]#

Reset the internals.

add_callback(key: str, func: Callable)[源代码]#

Add additional functions to bind spacemouse.

参数:

  • key – The keyboard button to check against.

  • func – The function to call when key is pressed. The callback function should not take any arguments.

advance() → torch.Tensor[源代码]#

Provides the result from spacemouse event state.

返回:

A 3D tensor containing the linear (x,y) and angular velocity (z).

class isaaclab.devices.Se3SpaceMouse[源代码]#

基类：DeviceBase

A space-mouse controller for sending SE(3) commands as delta poses.

This class implements a space-mouse controller to provide commands to a robotic arm with a gripper. It uses the HID-API which interfaces with USD and Bluetooth HID-class devices across multiple platforms [1].

The command comprises of two parts:

  • delta pose: a 6D vector of (x, y, z, roll, pitch, yaw) in meters and radians.

  • gripper: a binary command to open or close the gripper.

备注

The interface finds and uses the first supported device connected to the computer.

Currently tested for following devices:

Methods:

__init__(cfg)

Initialize the space-mouse layer.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind spacemouse.

advance()

Provides the result from spacemouse event state.
__init__(cfg: Se3SpaceMouseCfg)[源代码]#

Initialize the space-mouse layer.

参数:

cfg – Configuration object for space-mouse settings.

reset()[源代码]#

Reset the internals.

add_callback(key: str, func: Callable)[源代码]#

Add additional functions to bind spacemouse.

参数:

  • key – The keyboard button to check against.

  • func – The function to call when key is pressed. The callback function should not take any arguments.

advance() → torch.Tensor[源代码]#

Provides the result from spacemouse event state.

返回:

A 7-element tensor containing:

  • delta pose: First 6 elements as [x, y, z, rx, ry, rz] in meters and radians.

  • gripper command: Last element as a binary value (+1.0 for open, -1.0 for close).

返回类型:

torch.Tensor

OpenXR#

class isaaclab.devices.OpenXRDevice[源代码]#

基类：DeviceBase

An OpenXR-powered device for teleoperation and interaction.

This device tracks hand joints using OpenXR and makes them available as: 1. A dictionary of tracking data (when used without retargeters) 2. Retargeted commands for robot control (when retargeters are provided)

Raw data format (_get_raw_data output): * A dictionary with keys matching TrackingTarget enum values (HAND_LEFT, HAND_RIGHT, HEAD) * Each hand tracking entry contains a dictionary of joint poses * Each joint pose is a 7D vector (x, y, z, qw, qx, qy, qz) in meters and quaternion units * Joint names are defined in HAND_JOINT_NAMES from isaaclab.devices.openxr.common * Supported joints include palm, wrist, and joints for thumb, index, middle, ring and little fingers

Teleop commands: The device responds to several teleop commands that can be subscribed to via add_callback(): * “START”: Resume hand tracking data flow * “STOP”: Pause hand tracking data flow * “RESET”: Reset the tracking and signal simulation reset

The device tracks the left hand, right hand, head position, or any combination of these based on the TrackingTarget enum values. When retargeters are provided, the raw tracking data is transformed into robot control commands suitable for teleoperation.

Classes:

TrackingTarget

Enum class specifying what to track with OpenXR.

Methods:

__init__(cfg[, retargeters])

Initialize the OpenXR device.

reset()

Reset the internals.

add_callback(key, func)

Add additional functions to bind to client messages.

advance()

Process current device state and return control commands.
class TrackingTarget[源代码]#

基类：Enum

Enum class specifying what to track with OpenXR.

HAND_LEFT#

Track the left hand (index 0 in _get_raw_data output)

HAND_RIGHT#

Track the right hand (index 1 in _get_raw_data output)

HEAD#

Track the head/headset position (index 2 in _get_raw_data output)

__init__(cfg: OpenXRDeviceCfg, retargeters: list[isaaclab.devices.retargeter_base.RetargeterBase] | None = None)[源代码]#

Initialize the OpenXR device.

参数:

  • cfg – Configuration object for OpenXR settings.

  • retargeters – List of retargeter instances to use for transforming raw tracking data.

reset()[源代码]#

Reset the internals.

add_callback(key: str, func: Callable)[源代码]#

Add additional functions to bind to client messages.

参数:

  • key – The message type to bind to. Valid values are “START”, “STOP”, and “RESET”.

  • func – The function to call when the message is received. The callback function should not take any arguments.

advance() → torch.Tensor#

Process current device state and return control commands.

This method retrieves raw data from the device and optionally applies retargeting to convert it to robot commands.

Derived classes can either: 1. Override _get_raw_data() and use this base implementation, or 2. Override this method completely for custom command processing

返回:

When no retargeters are configured, returns raw device data in its native format. When retargeters are configured, returns a torch.Tensor containing the concatenated outputs from all retargeters.

Retargeters#

class isaaclab.devices.openxr.retargeters.GripperRetargeter[源代码]#

基类：RetargeterBase

Retargeter specifically for gripper control based on hand tracking data.

This retargeter analyzes the distance between thumb and index finger tips to determine whether the gripper should be open or closed. It includes hysteresis to prevent rapid toggling between states when the finger distance is near the threshold.

Features: - Tracks thumb and index finger distance - Implements hysteresis for stable gripper control - Outputs boolean command (True = close gripper, False = open gripper)

Methods:

__init__(cfg)

Initialize the retargeter.

retarget(data)

Convert hand joint poses to gripper command.
__init__(cfg: GripperRetargeterCfg)[源代码]#

Initialize the retargeter.

参数:

cfg – Configuration for the retargeter

retarget(data: dict) → torch.Tensor[源代码]#

Convert hand joint poses to gripper command.

参数:

data – Dictionary mapping tracking targets to joint data dictionaries. The joint names are defined in isaaclab.devices.openxr.common.HAND_JOINT_NAMES

返回:

Tensor containing a single bool value where True = close gripper, False = open gripper

返回类型:

torch.Tensor

class isaaclab.devices.openxr.retargeters.Se3AbsRetargeter[源代码]#

基类：RetargeterBase

Retargets OpenXR hand tracking data to end-effector commands using absolute positioning.

This retargeter maps hand joint poses directly to robot end-effector positions and orientations, rather than using relative movements. It can either: - Use the wrist position and orientation - Use the midpoint between thumb and index finger (pinch position)

Features: - Optional constraint to zero out X/Y rotations (keeping only Z-axis rotation) - Optional visualization of the target end-effector pose

Methods:

__init__(cfg)

Initialize the retargeter.

retarget(data)

Convert hand joint poses to robot end-effector command.
__init__(cfg: Se3AbsRetargeterCfg)[源代码]#

Initialize the retargeter.

参数:

  • bound_hand – The hand to track (OpenXRDevice.TrackingTarget.HAND_LEFT or OpenXRDevice.TrackingTarget.HAND_RIGHT)

  • zero_out_xy_rotation – If True, zero out rotation around x and y axes

  • use_wrist_rotation – If True, use wrist rotation instead of finger average

  • use_wrist_position – If True, use wrist position instead of pinch position

  • enable_visualization – If True, visualize the target pose in the scene

  • device – The device to place the returned tensor on (‘cpu’ or ‘cuda’)

retarget(data: dict) → torch.Tensor[源代码]#

Convert hand joint poses to robot end-effector command.

参数:

data – Dictionary mapping tracking targets to joint data dictionaries. The joint names are defined in isaaclab.devices.openxr.common.HAND_JOINT_NAMES

返回:

7D tensor containing position (xyz) and orientation (quaternion)

for the robot end-effector

返回类型:

torch.Tensor

class isaaclab.devices.openxr.retargeters.Se3RelRetargeter[源代码]#

基类：RetargeterBase

Retargets OpenXR hand tracking data to end-effector commands using relative positioning.

This retargeter calculates delta poses between consecutive hand joint poses to generate incremental robot movements. It can either: - Use the wrist position and orientation - Use the midpoint between thumb and index finger (pinch position)

Features: - Optional constraint to zero out X/Y rotations (keeping only Z-axis rotation) - Motion smoothing with adjustable parameters - Optional visualization of the target end-effector pose

Methods:

__init__(cfg)

Initialize the relative motion retargeter.

retarget(data)

Convert hand joint poses to robot end-effector command.
__init__(cfg: Se3RelRetargeterCfg)[源代码]#

Initialize the relative motion retargeter.

参数:

  • bound_hand – The hand to track (OpenXRDevice.TrackingTarget.HAND_LEFT or OpenXRDevice.TrackingTarget.HAND_RIGHT)

  • zero_out_xy_rotation – If True, ignore rotations around x and y axes, allowing only z-axis rotation

  • use_wrist_rotation – If True, use wrist rotation for control instead of averaging finger orientations

  • use_wrist_position – If True, use wrist position instead of pinch position (midpoint between fingers)

  • delta_pos_scale_factor – Amplification factor for position changes (higher = larger robot movements)

  • delta_rot_scale_factor – Amplification factor for rotation changes (higher = larger robot rotations)

  • alpha_pos – Position smoothing parameter (0-1); higher values track more closely to input, lower values smooth more

  • alpha_rot – Rotation smoothing parameter (0-1); higher values track more closely to input, lower values smooth more

  • enable_visualization – If True, show a visual marker representing the target end-effector pose

  • device – The device to place the returned tensor on (‘cpu’ or ‘cuda’)

retarget(data: dict) → torch.Tensor[源代码]#

Convert hand joint poses to robot end-effector command.

参数:

data – Dictionary mapping tracking targets to joint data dictionaries. The joint names are defined in isaaclab.devices.openxr.common.HAND_JOINT_NAMES

返回:

6D tensor containing position (xyz) and rotation vector (rx,ry,rz)

for the robot end-effector

返回类型:

torch.Tensor

class isaaclab.devices.openxr.retargeters.GR1T2Retargeter[源代码]#

基类：RetargeterBase

Retargets OpenXR hand tracking data to GR1T2 hand end-effector commands.

This retargeter maps hand tracking data from OpenXR to joint commands for the GR1T2 robot’s hands. It handles both left and right hands, converting poses of the hands in OpenXR format joint angles for the GR1T2 robot’s hands.

Methods:

__init__(cfg)

Initialize the GR1T2 hand retargeter.

retarget(data)

Convert hand joint poses to robot end-effector commands.
__init__(cfg: GR1T2RetargeterCfg)[源代码]#

Initialize the GR1T2 hand retargeter.

参数:

  • enable_visualization – If True, visualize tracked hand joints

  • num_open_xr_hand_joints – Number of joints tracked by OpenXR

  • device – PyTorch device for computations

  • hand_joint_names – List of robot hand joint names

retarget(data: dict) → torch.Tensor[源代码]#

Convert hand joint poses to robot end-effector commands.

参数:

data – Dictionary mapping tracking targets to joint data dictionaries.

返回:

Left wrist pose Right wrist pose in USD frame Retargeted hand joint angles

返回类型:

tuple containing