Nvidia’s GR00t 2 watches video footage of humans and simulates robotic movements
A new Nvidia AI model, GR00t 2, watches humans perform tasks on video and then imagines itself as the robot executing those same movements.
Training robots usually requires engineers to spend hours remotely controlling the machines. This teleoperation is expensive, slow, and requires specific data for every robot model. Nvidia introduced this new AI foundation model to shorten that process.
Unlike previous approaches, including GR00t 1, the software does not translate camera images and language commands directly into motor movements. Nvidia describes the system as a world-action model. It simultaneously calculates the next movement and a short video of the immediate future. The system predicts how the camera image should change when a robotic arm picks up an object.
This video prediction allows the model to transfer movement sequences between different robot form factors. Nvidia trained the system on a cluster with more than a thousand GB300 chips. The training included about 400,000 hours of video footage showing people performing various tasks while wearing head-mounted cameras.
To teach a robot a new task, a person demonstrates the steps using data gloves, specialized hand grippers, or a finger exoskeleton. When these recordings are combined during training with general motion data from the target robot, the AI generates the robotic arm instead of the human arm in its video prediction and derives the matching control commands from that.
In lab tests, a robot with five-fingered hands was able to light a match, slice a banana, or toss packages onto a conveyor belt without having been teleoperated for those specific tasks.
Success rates varied between 10 and 80 percent depending on the task. From a single video of a human hand movement, the system also learned to hammer nails by way of a computer simulation. For 50 selected tasks where the model already performed well after just a few examples, less than three minutes of conventional training was enough to raise the success rate from 40 to 95 percent.
A higher-level language model helps with errors but slows the robot down
For longer assembly work, Nvidia pairs the system with a higher-level AI language model like OpenAI’s GPT-6 Astra. While GR00t 2 handles the gripping movements, the language model breaks the task into individual steps, color-codes grip points in the camera image, and launches helper routines when a part falls off.
When the researchers placed loose parts for toy models on a table in no particular order, the combined system assembled them in at least 97 percent of trials. Without the corrective language model and additional fine-tuning, the toy car assembly succeeded in only 23 out of 100 attempts. The team also trained precision tasks like plugging in network cables or chargers through repeated trial and error on the real device.
For bipedal robots, Nvidia developed the slimmed-down variant GR00t 2 Turbo, which processes fewer image details. It runs directly on the Jetson Thor onboard computer of a humanoid robot. In testing, the machine was able to scoop popcorn, clear tables, and wipe up liquids with a towel under its foot. That last task succeeded in just under half of all attempts.
The research team identifies significant hurdles before the system is ready for reliable real-world use. The model does not yet process direct sensor data for touch and sometimes misses subtle errors like half-turned screws. The higher-level language model also slows things down considerably. During trash sorting, the robot stood idle for roughly a third of the total time because it had to wait more than seven seconds on average for the planning model’s next computation.
Other labs are also looking for ways to teach robots new tasks with less teleoperation. We covered some of them in our AI Radar #5 on Neolabs.
