在可视化器中显示相机流视图

在可视化器中显示相机流视图#

本指南介绍如何在可视化器的一个实时面板中显示多个环境的相机真值图像。该视图的工作方式和支持的可视化器见 相机流视图 。

快速开始#

运行 IsaacLab/scripts/tutorials/07_visualizers 中的 run_tiled_camera_visualizer.py 脚本：

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
run_tiled_camera_visualizer.py 的代码
  1# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
  2# All rights reserved.
  3#
  4# SPDX-License-Identifier: BSD-3-Clause
  5
  6"""
  7This script demonstrates the visualizer tiled camera panel.
  8
  9.. code-block:: bash
 10
 11    # Kit visualizer tiled camera panel
 12    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 13 --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit
 14
 15    # Newton visualizer tiled camera panel
 16    uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
 17        --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl
 18
 19"""
 20
 21from __future__ import annotations
 22
 23import argparse
 24import contextlib
 25import sys
 26
 27import gymnasium as gym
 28import torch
 29
 30import isaaclab_tasks  # noqa: F401
 31
 32with contextlib.suppress(ImportError):
 33    import isaaclab_tasks_experimental  # noqa: F401
 34from isaaclab.app import add_launcher_args, launch_simulation
 35
 36from isaaclab_tasks.utils import resolve_task_config, setup_preset_cli
 37
 38KIT_DEFAULT_TASK = "Isaac-Velocity-Rough-AnymalD"
 39NEWTON_DEFAULT_TASK = "IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor"
 40SUPPORTED_TILED_VISUALIZERS = {"kit", "newton", "newton_gl", "newton_rtx"}
 41UNSUPPORTED_TILED_VISUALIZERS = {"rerun", "viser"}
 42
 43
 44def _resolve_env_regex_path(prim_path: str) -> str:
 45    """Resolve scene config env namespace macros to the cloned-env regex."""
 46    return prim_path.format(ENV_REGEX_NS="/World/envs/env_.*")
 47
 48
 49def _requested_visualizers(args_cli: argparse.Namespace) -> list[str]:
 50    """Return requested visualizers, defaulting to Kit for this tutorial."""
 51    visualizers = args_cli.visualizer or ["kit"]
 52    visualizers = [str(visualizer).lower() for visualizer in visualizers]
 53
 54    if "none" in visualizers:
 55        raise ValueError("This demo requires a tiled-camera visualizer. Use '--viz kit' or '--viz newton_gl'.")
 56    unsupported = sorted(set(visualizers) & UNSUPPORTED_TILED_VISUALIZERS)
 57    if unsupported:
 58        raise ValueError(
 59            "The visualizer tiled camera panel is only implemented for Kit and Newton. "
 60            f"Unsupported selection: {unsupported}."
 61        )
 62    unknown = sorted(set(visualizers) - SUPPORTED_TILED_VISUALIZERS)
 63    if unknown:
 64        raise ValueError(f"Unknown visualizer selection for this demo: {unknown}.")
 65    return visualizers
 66
 67
 68def _make_kit_visualizer_cfg(env_cfg):
 69    """Create the Kit streaming-camera visualizer for the selected task."""
 70    from isaaclab_visualizers.kit import KitVisualizerCfg
 71
 72    visualizer_cfg = KitVisualizerCfg()
 73    visualizer_cfg.streaming_view = True
 74    visualizer_cfg.streaming_envs = 36
 75
 76    ego_cam_cfg = getattr(env_cfg.scene, "ego_cam", None)
 77    if ego_cam_cfg is not None:
 78        visualizer_cfg.streaming_sensor_prim_path = _resolve_env_regex_path(ego_cam_cfg.prim_path)
 79        return visualizer_cfg
 80
 81    visualizer_cfg.streaming_sensor_prim_path = None
 82    visualizer_cfg.streaming_cam_eye = (3.0, 3.0, 3.0)
 83    visualizer_cfg.streaming_cam_target_prim_path = "/World/envs/*/Robot/base"
 84    # Here is an alternative eye position for a top down view
 85    # visualizer_cfg.streaming_cam_eye = (0.0, 0.0, 5.0)
 86    return visualizer_cfg
 87
 88
 89def _make_newton_visualizer_cfg(env_cfg):
 90    """Create the Newton streaming-camera visualizer for the selected task."""
 91    from isaaclab_visualizers.newton import NewtonGLVisualizerCfg
 92
 93    visualizer_cfg = NewtonGLVisualizerCfg()
 94    visualizer_cfg.streaming_view = True
 95    visualizer_cfg.streaming_envs = 12
 96
 97    ego_cam_cfg = getattr(env_cfg.scene, "ego_cam", None)
 98    if ego_cam_cfg is not None:
 99        visualizer_cfg.streaming_sensor_prim_path = _resolve_env_regex_path(ego_cam_cfg.prim_path)
100        return visualizer_cfg
101
102    # Here are other robot mounted camera options for this environment
103    # visualizer_cfg.streaming_sensor_prim_path = "/World/envs/env_.*/Robot/left_arm_camera_sim_view_frame/left_camera"
104    # visualizer_cfg.streaming_sensor_prim_path = (
105    #     "/World/envs/env_.*/Robot/right_arm_camera_sim_view_frame/right_camera"
106    # )
107    visualizer_cfg.streaming_sensor_prim_path = None
108    visualizer_cfg.streaming_cam_eye = (3.0, 3.0, 3.0)
109    visualizer_cfg.streaming_cam_target_prim_path = "/World/envs/*/Robot/base"
110    return visualizer_cfg
111
112
113def _configure_visualizers(env_cfg, args_cli: argparse.Namespace) -> None:
114    """Attach tiled camera visualizer configs to the environment simulation config."""
115    visualizers = _requested_visualizers(args_cli)
116    args_cli.visualizer = visualizers
117    env_cfg.sim.visualizer_cfgs = [
118        _make_kit_visualizer_cfg(env_cfg) if visualizer == "kit" else _make_newton_visualizer_cfg(env_cfg)
119        for visualizer in visualizers
120    ]
121
122
123def _resolve_task(args_cli: argparse.Namespace) -> str:
124    """Resolve the task for the selected visualizer."""
125    if args_cli.task is not None:
126        return args_cli.task
127    if "newton" in _requested_visualizers(args_cli):
128        return NEWTON_DEFAULT_TASK
129    return KIT_DEFAULT_TASK
130
131
132# add argparse arguments
133parser = argparse.ArgumentParser(description="Showcase the Kit/Newton visualizer tiled camera panel.")
134parser.add_argument("--num_envs", type=int, default=None, help="Number of environments to simulate.")
135parser.add_argument("--task", type=str, default=None, help="Name of the task.")
136# append simulation launcher cli args
137add_launcher_args(parser)
138args_cli, hydra_args = setup_preset_cli(parser)
139args_cli.task = _resolve_task(args_cli)
140sys.argv = [sys.argv[0]] + hydra_args
141
142
143def main():
144    """Run a random-action environment with a tiled camera visualizer."""
145    # parse configuration via Hydra (supports preset selection, e.g. presets=newton_mjwarp)
146    env_cfg, _ = resolve_task_config(args_cli.task, "")
147    _configure_visualizers(env_cfg, args_cli)
148
149    with launch_simulation(env_cfg, args_cli):
150        # override with CLI arguments
151        env_cfg.scene.num_envs = args_cli.num_envs if args_cli.num_envs is not None else env_cfg.scene.num_envs
152        env_cfg.sim.device = args_cli.device if args_cli.device is not None else env_cfg.sim.device
153
154        # create environment
155        env = gym.make(args_cli.task, cfg=env_cfg)
156
157        # print info (this is vectorized environment)
158        print(f"[INFO]: Gym observation space: {env.observation_space}")
159        print(f"[INFO]: Gym action space: {env.action_space}")
160        env.reset()
161
162        # keep stepping until all visualizer windows have been closed
163        sim = env.unwrapped.sim
164        if not sim.visualizers:
165            print("[WARN]: No visualizers found. Exiting.")
166            env.close()
167            return
168
169        while True:
170            if sim.visualizers and not any(v.is_running() and not v.is_closed for v in sim.visualizers):
171                break
172            with torch.inference_mode():
173                actions = 2 * torch.rand(env.action_space.shape, device=env.unwrapped.device) - 1
174                env.step(actions)
175
176        env.close()
177
178
179if __name__ == "__main__":
180    main()

脚本的两种运行方式见下文的 示例；定制视频流的 VisualizerCfg 字段见 配置 和 故障排查。

示例#

运行 run_tiled_camera_visualizer.py 可演示相机流视图的两种用法：

  • 自动创建相机，朝向并跟随移动的 AnymalD 机器人，在 Kit 可视化器中显示

  • 使用机器人已有的腕部相机传输画面，在 Newton 可视化器中显示

示例 1：跟随 AnymalD 机器人#

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

uv run python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task Isaac-Velocity-Rough-AnymalD --num_envs 256 --viz kit

脚本中的 KitVisualizerCfg 会创建相机，朝向并跟随每个机器人的基座 prim；相机偏移量由 streaming_cam_eye 指定（此处为 (3.0, 3.0, 3.0) ；可以尝试 (0, 0, 5) 获取俯视画面）。相机视图从 256 个环境中随机抽取 36 个显示。

Kit 可视化器：交互式视口

Kit 可视化器：串流相机视图

示例 2：传输机器人搭载相机的画面#

uv run --extra teleop python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py \
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl

uv run --extra teleop python scripts/tutorials/07_visualizers/run_tiled_camera_visualizer.py ^
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor --num_envs 25 --viz newton_gl

Galbot 立方体堆叠环境自带腕部相机，可以从第一视角观察夹爪、桌面和立方体。脚本中的 NewtonGLVisualizerCfg 使用 /World/envs/env_.*/Robot/head_camera_sim_view_frame/head_camera 处的已有传感器传输画面；修改 streaming_sensor_prim_path 可显示其他相机。默认显示 25 个环境中 12 个相机的画面。

Newton 可视化器：交互式视口

Newton 可视化器：串流相机视图

配置#

要自定义相机流行为，请修改 run_tiled_camera_visualizer.py 中高亮的 VisualizerCfg 字段：

  • 对于自动创建的相机， streaming_cam_target_prim_path 选择跟随的 prim， streaming_cam_eye 设置相机相对该 prim 的偏移量。目标路径默认为 None ，此时可视化器会在初始化时采用发现的第一个场景相机；场景已有 TiledCamera 传感器时，无需显式指定路径。

  • 对于已有的场景相机， streaming_sensor_prim_path 必须与所选任务中的 Isaac Lab Camera 传感器 prim 路径匹配。

  • streaming_envs 控制显示的环境格数。传入 int 可随机抽取指定数量的环境，传入 list[int] 则可固定显示指定索引的环境。

  • streaming_gt_types 选择显示的真值类型，例如 ["rgb", "depth", "segmentation", "normals"] 。

  • streaming_depth_min / streaming_depth_max 设置深度色图的范围，单位为米。

故障排除#

  • 如果生成视图时提示缺少 prim，请确认 streaming_cam_target_prim_path 在每个所选环境中都能解析到目标；常用路径模板为 /World/envs/*/... 和 /World/envs/env_.*/... 。多数情况下，可以保留 None ，让可视化器自动采用已有的场景相机。

  • 如果已有相机视图提示该 prim 不属于任何 Isaac Lab 相机，请检查 streaming_sensor_prim_path 是否与任务中的 Camera 传感器匹配。

  • 如果深度面板只显示单一颜色，请调整 streaming_depth_min 和 streaming_depth_max ，使其覆盖场景中预期的深度范围。

  • 如果视图开销过大，可减小 streaming_envs 、 --num_envs 或相机分辨率。

警告

Newton MJWarp 在 replicate_physics=True 时使用自动创建的相机

设置 replicate_physics=True 后，物理初始化完成时只有 env_0 保留 USD prim。其余环境（ env_1 至 env_{N-1} ）的相机会被移除，导致初始化失败:

RuntimeError: Number of camera prims in the view (1) does not match
the number of environments (N).

解决方法 ：将 streaming_sensor_prim_path 指向在物理初始化前就已声明于场景配置中的相机（例如视觉任务中的 TiledCamera ）。

另请参阅#

目录