NVIDIA has launched Cosmos-H-Dreams, a tool that lets surgical robots generate future video frames in real time based on current actions and camera feeds.
Previous versions of the technology, known as Cosmos-H-Surgical-Simulator, could predict outcomes but ran too slowly for live interaction. The new model distills those capabilities into a faster version, running on a single NVIDIA RTX PRO 6000 GPU at interactive speeds.
Turning a predictive model into an interactive simulator
The original system was an action-conditioned world foundation model built on NVIDIA Cosmos-Predict2.5-2B. It was trained on the Open-H-Embodiment dataset. Given a surgical context frame and a future robot trajectory, it produced video showing the likely visual consequences of those actions. This was useful for offline policy evaluation and synthetic data generation. A recorded or policy-generated trajectory could be sent to the model, the corresponding rollout could be generated, and the result could be inspected or scored without repeatedly executing the motion on a physical robot.
Cosmos-H-Dreams moves the model into the real-time regime. Starting from the multi-embodiment surgical priors learned by Cosmos-H-Surgical-Simulator, the team specialised the model for da Vinci Research Kit (dVRK) tabletop suturing. They distilled it into a causal student that generates the scene autoregressively. The released model receives an initial RGB frame and a live stream of robot kinematics, then produces the next chunk of frames before continuing with the following action block.
Collaboration with CMR Surgical and Cambridge Consultants allowed integration with the Versius surgeon controller, enabling real-time operation on the Versius platform.
Reducing generation costs while keeping dynamics
The main challenge was preserving useful surgical dynamics while reducing the cost of generation. Cosmos-H-Dreams uses a teacher-to-student training pipeline designed for long, autoregressive rollouts.
2.1. A Surgical Teacher
The bidirectional teacher began from the Cosmos-H-Surgical-Simulator Open-H checkpoint, which uses a unified 44-dimensional action representation. For the released dVRK tabletop model, the dual-arm dVRK action content, consisting of relative end-effector translation, rotation, and gripper state, was mapped into this common representation.
The teacher was then fine-tuned on the JHU dVRK tabletop mixture, including successful demonstrations as well as failure and out-of-distribution episodes such as needle drops, missed throws, and unsuccessful knot ties. These failures are important: a simulator intended to evaluate policies must reproduce the consequences of poor actions, not only ideal demonstrations.
To improve stability during long rollouts, the training process progressively increased the teacher’s temporal horizon. They started training on a 12-frame horizon and progressively increased this number until 72 frames. At each horizon bump, they initialised the warmed-up model with pretrained weights.
2.2. Causal Warmup
The teacher’s denoising trajectories were first precomputed and cached. A causal student was initialised from the teacher and trained to imitate these cached trajectories. This warmup stage taught the student to operate with causal attention and a streaming key/value cache before it began learning from its own generated history.
2.3. Self-Forcing Distillation
Autoregressive models face a familiar problem: during training they may see clean, ground-truth context, while during deployment they must condition on their own imperfect outputs. Small errors can therefore compound over time.
Cosmos-H-Dreams addressed this mismatch with self-forcing distillation. During training, the student rolled forward using its own generated context. Distribution-matching supervision from the frozen teacher then guided those self-generated rollouts toward realistic surgical video. This prepared the student for the same conditions it would encounter during interactive inference.
The resulting model supported few-step diffusion, with as few as two denoising steps per latent frame, rather than the many-step process used by the full teacher. It combined the teacher’s surgical priors with the causal structure needed for streaming.
FlashDreams: The Real-Time Inference Engine
Model distillation was only part of the real-time story. Cosmos-H-Dreams was served through FlashDreams, an accelerated inference library for autoregressive world and video models.
FlashDreams turned the distilled student into a low-latency streaming system through several complementary optimizations, such as streaming KV cache, CUDA Graph capturing, or model compilation.
Together, these techniques brought the distilled surgical world model fine-tune from the roughly ten-frames-per-second regime of standard Cosmos-H-Surgical-Simulator inference to interactive operation (~160 frames per second) on a single NVIDIA RTX PRO 6000.
Cosmos-H-Dreams also provided the human-machine interfaces that turned generation into interaction. A browser client could send keyboard commands and receive generated frames over WebRTC. A Meta Quest client could map tracked controller motion into robot actions and display the synthesised scene through WebXR. The same model could also be connected to a learned surgical policy, with generated observations and predicted actions exchanged inside a closed loop.
Adapting to your own data
While Cosmos-H-Dreams included a pre-trained checkpoint for tabletop suturing, the system was designed to be extensible to specific embodiments. To train a real-time student model for their own dataset, they provided a complete recipe for teacher fine-tuning and self-forcing distillation in a step-by-step guide.
Moving toward closed-loop surgical physical AI
Cosmos-H-Dreams opened a new frontier for surgical simulation: environments learned from real robot data that were responsive enough to be inhabited.
The immediate next step was to evaluate more than visual quality. A useful surgical simulator must respond correctly to actions, preserve instrument and scene structure over long rollouts, and support conclusions that transfer to the physical robot. This motivated a new family of closed-loop benchmarks: tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, counterfactual action diversity, long-horizon drift, and agreement between simulated and real policy outcomes.
Real-time world models could also become active partners in surgical policy development. They could generate rare failures on demand, provide scalable environments for reinforcement or imitation learning, and enable rapid evaluation of new policies without tying every experiment to scarce robotic hardware.
Further ahead, real-time simulation enabled a series of downstream applications such as latency-aware telesurgery, where the world model helped maintain a more stable display; or interactive surgical rehearsal, procedure planning, and intraoperative decision support. Cosmos-H-Dreams was a research and development platform, not a diagnostic system, a replacement for intraoperative imaging, or a controller for a physical surgical robot. Yet it provided a foundation for exploring these possibilities safely.
As model fidelity, temporal stability, and hardware efficiency continued to improve, real-time generative simulation could help connect surgeon education, synthetic data generation, policy training, and policy evaluation within one shared Physical AI environment.
Resources
- Cosmos-H-Dreams code and examples: GitHub repository
- Cosmos-H-Dreams model: Hugging Face checkpoint
- Cosmos-H-Surgical-Simulator: Hugging Face model and GitHub repository
- Adapt to your own data: Step-by-step recipe for teacher fine-tuning and self-forcing distillation
- Open-H-Embodiment: Hugging Face dataset
- FlashDreams: GitHub repository
- NVIDIA Cosmos-Predict2.5: GitHub repository
- Cosmos-Surg-dVRK: World Foundation Model-based Automated Online Evaluation of Surgical Robot Policy Learning (arXiv paper)
Cosmos-H-Dreams brings action-conditioned surgical world modelling into the real-time loop, creating a new environment for people and policies to practice, explore, generate data, and evaluate what happens next.
