NVIDIA IsaacTeleop now includes a graph-based retargeting engine that maps hand and controller inputs directly to robot actions. The tutorial demonstrates this by building every input signal in NumPy rather than using a headset. This setup runs entirely on a CPU within a Colab environment and prints intermediate calculations. The process involves defining the data types, generating synthetic hand and controller data, creating a retargeter with adjustable parameters, and driving the built-in gripper and SE(3) retargeters. A final graph emits an action vector per step, applies a world-frame transform, and manages the state machine lifecycle.
Installation and environment check
The project installs the stable IsaacTeleop wheel from PyPI using the retargeters-lite extra. This dependency adds only SciPy. The code reports the package version, Python interpreter, and NumPy build. The package structure separates device I/O modules that wrap OpenXR and CloudXR from the pure-Python retargeting engine used here. Listing the schema types reveals the vocabulary of the data layer without opening a headset session.
The code executes the following steps to verify the environment:
- Installs the specific IsaacTeleop version with the lite extras.
- Prints the top-level modules available in the package.
- Lists the schema message types, showing the count and examples such as HandInput and ControllerInput.
Explicitly, no headset, OpenXR runtime, or simulator is used in this section.
The type contract
The engine relies on a TensorGroupType, an ordered list of typed slots, and a TensorGroup, the runtime container holding one value per slot. HandInput carries four NumPy arrays for the 26 OpenXR hand joints, while ControllerInput carries fourteen slots for poses, buttons, and axes, addressed through generated IntEnum indices.
Writing a float64 array where float32 is declared fails at the write stage. Reading a slot that has not been written raises an error instead of returning stale data. OptionalType marks inputs a tracker may not deliver; the matching OptionalTensorGroup starts as absent and becomes present on its first write. This mechanism tells downstream nodes when a hand leaves the tracking volume.
Synthetic tracking data
The tutorial builds the tracking data a headset would normally supply. The make_hand function lays out 26 joints in OpenXR order around a wrist position. It runs four finger chains and a thumb chain away from the palm and places the thumb tip a chosen pinch distance from the index tip. This ensures the one number the later steps depend on is under control.
The make_controller function fills every ControllerInput slot. This includes grip and aim poses with validity flags, four buttons, the thumbstick axes, and the analog squeeze and trigger values. Both functions return ordinary TensorGroups, which is all a retargeter sees, whether the numbers came from OpenXR or from NumPy.
What it means
Developers can now validate their retargeting logic without expensive hardware. By generating synthetic data, they verify that the graph processes inputs correctly before connecting real sensors. This approach isolates the logic from hardware drivers and allows testing on standard CPUs.
