Ray Caster#
The Ray Caster sensor (and the ray caster camera) are similar to RTX based rendering in that they both involve casting rays. The difference here is that the rays cast by the Ray Caster sensor return strictly collision information along the cast, and the direction of each individual ray can be specified. They do not bounce, nor are they affected by things like materials or opacity. For each ray specified by the sensor, a line is traced along the path of the ray and the location of first collision with the specified mesh is returned. This is the method used by some of our quadruped examples to measure the local height field.
To keep the sensor performant when there are many cloned environments, the line tracing is done directly in Warp. This is the reason why specific meshes need to be identified to cast against: that mesh data is loaded onto the device by warp when the sensor is initialized. As a consequence, the current iteration of this sensor only works for literally static meshes (meshes that are not changed from the defaults specified in their USD file). This constraint will be removed in future releases.
Using a ray caster sensor requires a pattern and a parent xform to be attached to. The pattern defines how the rays are cast, while the prim properties defines the orientation and position of the sensor (additional offsets can be specified for more exact placement). Isaac Lab supports a number of ray casting pattern configurations, including a generic LIDAR and grid pattern.
# Pre-defined configs
##
from isaaclab_assets.robots.anymal import ANYMAL_C_CFG # isort: skip
@configclass
class RaycasterSensorSceneCfg(InteractiveSceneCfg):
"""Design the scene with sensors on the robot."""
# ground plane
ground = AssetBaseCfg(
prim_path="/World/Ground",
spawn=sim_utils.UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Environments/Terrains/rough_plane.usd",
scale=(1, 1, 1),
),
)
# lights
dome_light = AssetBaseCfg(
prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
)
# robot
robot = ANYMAL_C_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
ray_caster = RayCasterCfg(
prim_path="{ENV_REGEX_NS}/Robot/base/lidar_cage",
update_period=1 / 60,
offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)),
mesh_prim_paths=["/World/Ground"],
attach_yaw_only=True,
Notice that the units on the pattern config is in degrees! Also, we enable visualization here to explicitly show the pattern in the rendering, but this is not required and should be disabled for performance tuning.
Querying the sensor for data can be done at simulation run time like any other sensor.
def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene):
.
.
.
# Simulate physics
while simulation_app.is_running():
.
.
.
# print information from the sensors
print("-------------------------------")
print(scene["ray_caster"])
print("Ray cast hit results: ", scene["ray_caster"].data.ray_hits_w)
-------------------------------
Ray-caster @ '/World/envs/env_.*/Robot/base/lidar_cage':
view type : <class 'isaacsim.core.prims.xform_prim.XFormPrim'>
update period (s) : 0.016666666666666666
number of meshes : 1
number of sensors : 1
number of rays/sensor: 18000
total number of rays : 18000
Ray cast hit results: tensor([[[-0.3698, 0.0357, 0.0000],
[-0.3698, 0.0357, 0.0000],
[-0.3698, 0.0357, 0.0000],
...,
[ inf, inf, inf],
[ inf, inf, inf],
[ inf, inf, inf]]], device='cuda:0')
-------------------------------
Here we can see the data returned by the sensor itself. Notice first that there are 3 closed brackets at the beginning and the end: this is because the data returned is batched by the number of sensors. The ray cast pattern itself has also been flattened, and so the dimensions of the array are [N, B, 3] where N is the number of sensors, B is the number of cast rays in the pattern, and 3 is the dimension of the casting space. Finally, notice that the first several values in this casting pattern are the same: this is because the lidar pattern is spherical and we have specified our FOV to be hemispherical, which includes the poles. In this configuration, the “flattening pattern” becomes apparent: the first 180 entries will be the same because it’s the bottom pole of this hemisphere, and there will be 180 of them because our horizontal FOV is 180 degrees with a resolution of 1 degree.
You can use this script to experiment with pattern configurations and build an intuition about how the data is stored by altering the triggered variable on line 81.
Code for raycaster_sensor.py
1# Copyright (c) 2022-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
2# All rights reserved.
3#
4# SPDX-License-Identifier: BSD-3-Clause
5
6# Copyright (c) 2022-2025, The Isaac Lab Project Developers.
7# All rights reserved.
8#
9# SPDX-License-Identifier: BSD-3-Clause
10
11import argparse
12import numpy as np
13
14from isaaclab.app import AppLauncher
15
16# add argparse arguments
17parser = argparse.ArgumentParser(description="Example on using the raycaster sensor.")
18parser.add_argument("--num_envs", type=int, default=1, help="Number of environments to spawn.")
19# append AppLauncher cli args
20AppLauncher.add_app_launcher_args(parser)
21# parse the arguments
22args_cli = parser.parse_args()
23
24# launch omniverse app
25app_launcher = AppLauncher(args_cli)
26simulation_app = app_launcher.app
27
28"""Rest everything follows."""
29
30import torch
31
32import isaaclab.sim as sim_utils
33from isaaclab.assets import AssetBaseCfg
34from isaaclab.scene import InteractiveScene, InteractiveSceneCfg
35from isaaclab.sensors.ray_caster import RayCasterCfg, patterns
36from isaaclab.utils import configclass
37from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
38
39##
40# Pre-defined configs
41##
42from isaaclab_assets.robots.anymal import ANYMAL_C_CFG # isort: skip
43
44
45@configclass
46class RaycasterSensorSceneCfg(InteractiveSceneCfg):
47 """Design the scene with sensors on the robot."""
48
49 # ground plane
50 ground = AssetBaseCfg(
51 prim_path="/World/Ground",
52 spawn=sim_utils.UsdFileCfg(
53 usd_path=f"{ISAAC_NUCLEUS_DIR}/Environments/Terrains/rough_plane.usd",
54 scale=(1, 1, 1),
55 ),
56 )
57
58 # lights
59 dome_light = AssetBaseCfg(
60 prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
61 )
62
63 # robot
64 robot = ANYMAL_C_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
65
66 ray_caster = RayCasterCfg(
67 prim_path="{ENV_REGEX_NS}/Robot/base/lidar_cage",
68 update_period=1 / 60,
69 offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)),
70 mesh_prim_paths=["/World/Ground"],
71 attach_yaw_only=True,
72 pattern_cfg=patterns.LidarPatternCfg(
73 channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0
74 ),
75 debug_vis=not args_cli.headless,
76 )
77
78
79def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene):
80 """Run the simulator."""
81 # Define simulation stepping
82 sim_dt = sim.get_physics_dt()
83 sim_time = 0.0
84 count = 0
85
86 triggered = True
87 countdown = 42
88
89 # Simulate physics
90 while simulation_app.is_running():
91
92 if count % 500 == 0:
93 # reset counter
94 count = 0
95 # reset the scene entities
96 # root state
97 # we offset the root state by the origin since the states are written in simulation world frame
98 # if this is not done, then the robots will be spawned at the (0, 0, 0) of the simulation world
99 root_state = scene["robot"].data.default_root_state.clone()
100 root_state[:, :3] += scene.env_origins
101 scene["robot"].write_root_pose_to_sim(root_state[:, :7])
102 scene["robot"].write_root_velocity_to_sim(root_state[:, 7:])
103 # set joint positions with some noise
104 joint_pos, joint_vel = (
105 scene["robot"].data.default_joint_pos.clone(),
106 scene["robot"].data.default_joint_vel.clone(),
107 )
108 joint_pos += torch.rand_like(joint_pos) * 0.1
109 scene["robot"].write_joint_state_to_sim(joint_pos, joint_vel)
110 # clear internal buffers
111 scene.reset()
112 print("[INFO]: Resetting robot state...")
113 # Apply default actions to the robot
114 # -- generate actions/commands
115 targets = scene["robot"].data.default_joint_pos
116 # -- apply action to the robot
117 scene["robot"].set_joint_position_target(targets)
118 # -- write data to sim
119 scene.write_data_to_sim()
120 # perform step
121 sim.step()
122 # update sim-time
123 sim_time += sim_dt
124 count += 1
125 # update buffers
126 scene.update(sim_dt)
127
128 # print information from the sensors
129 print("-------------------------------")
130 print(scene["ray_caster"])
131 print("Ray cast hit results: ", scene["ray_caster"].data.ray_hits_w)
132
133 if not triggered:
134 if countdown > 0:
135 countdown -= 1
136 continue
137 data = scene["ray_caster"].data.ray_hits_w.cpu().numpy()
138 np.save("cast_data.npy", data)
139 triggered = True
140 else:
141 continue
142
143
144def main():
145 """Main function."""
146
147 # Initialize the simulation context
148 sim_cfg = sim_utils.SimulationCfg(dt=0.005, device=args_cli.device)
149 sim = sim_utils.SimulationContext(sim_cfg)
150 # Set main camera
151 sim.set_camera_view(eye=[3.5, 3.5, 3.5], target=[0.0, 0.0, 0.0])
152 # design scene
153 scene_cfg = RaycasterSensorSceneCfg(num_envs=args_cli.num_envs, env_spacing=2.0)
154 scene = InteractiveScene(scene_cfg)
155 # Play the simulator
156 sim.reset()
157 # Now we are ready!
158 print("[INFO]: Setup complete...")
159 # Run the simulator
160 run_simulator(sim, scene)
161
162
163if __name__ == "__main__":
164 # run the main function
165 main()
166 # close sim app
167 simulation_app.close()