NVIDIA has released Cosmos3-H-Dreams, the first real-time generative surgical simulator trained on clinical cholecystectomy recordings from the CMR Surgical Versius platform.
The system runs interactively on a single NVIDIA RTX PRO 6000 GPU. It learns visual dynamics directly from synchronized video and robot kinematics, avoiding the need to explicitly model scene geometry or physical interactions.
From tabletop tasks to clinical scenes
Previous versions, such as Cosmos-H-Dreams, operated in controlled tabletop environments like dVRK suturing stations. Inside the human body, small instrument movements affect the entire scene. Tissue stretches under traction, reflections shift, structures become occluded, and smoke alters visibility.
A generative simulator must capture these responses while preserving anatomy and instruments as it generates each new block of video. Small errors can accumulate over time, gradually changing instrument geometry or anatomical structure. Temporal consistency is therefore essential.
Visual realism is a starting point for validation, not proof of biomechanical accuracy. The generated scene must follow the controls: translation, rotation, and gripper commands should produce coherent anatomical consequences. A plausible-looking rollout that ignores those commands cannot serve as an interactive environment.
Maintaining this fidelity in a live control loop requires low latency generation. Cosmos3-H-Dreams addresses this through a model specialized for clinical cholecystectomy and distilled for real-time generation, bringing action-conditioned interaction into scenes with complex anatomy, tissue motion, and occlusion.
What is new in Cosmos3-H-Dreams
A Cosmos 3 foundation and richer surgical controls
Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. It specialises a bidirectional teacher on curated cholecystectomy video and kinematics.
Sample selection targets two visible, engaged instruments: a left Cadiere grasper and a right monopolar hook. Camera commands are limited to emphasise instrument-driven interaction.
The model uses 28 action dimensions: pose and gripper information for both arms, plus energy controls, thumbstick axes and thumbstick buttons. Training uses 832 x 480 video with actions aligned at 10 Hz. This sampling rate is distinct from runtime generation throughput.
From teacher to four-step and two-step students
Cosmos3-H-Dreams combines a teacher-to-student training pipeline designed for long autoregressive rollouts with accelerated streaming inference through FlashDreams.
The process begins by adapting the bidirectional teacher on 24 future frames. The horizon then extends to 72 frames, covering 2.4 and 7.2 seconds at 10 Hz. This gives the teacher longer sequences over which to learn surgical dynamics.
Next, distillation turns the bidirectional teacher into a causal student that can generate video in short blocks as new actions arrive. Causal teacher forcing prepares the student for sequential generation using recorded context. Self-Forcing then trains it on its own generated context, guided by the teacher and a learned critic. This helps address the accumulation of errors during interactive rollouts.
The first released student uses four denoising steps per generated block. The team then trains that student further with a two-step schedule to produce the second release checkpoint. The two-step model is separately trained, not simply the four-step model run with fewer sampling steps.
Fewer denoising steps results in faster inference time at the expense of lower visual quality. Both checkpoints let developers explore the balance between visual quality and inference cost.
Streaming through FlashDreams
FlashDreams supplies the accelerated streaming runtime. Long rollouts also require managing the model’s attention history. The Cosmos3-H-Dreams integration uses an adapted FadeMem module that consolidates the growing key/value cache during long rollouts.
This complements Self-Forcing: distillation addresses how the model learns from its generated history, while cache consolidation addresses how that history is stored during streaming generation.
Exploring unseen trajectories
During a clinical procedure, complications may arise or the surgeon may perform novel movements. Therefore, a simulator must generalise beyond the training data and produce coherent frames with unseen movements.
The demonstrations show Cosmos3-H-Dreams generating different cholecystectomy scenes conditioned on instrument actions recorded with a VR headset.
The two videos begin with the same initial frame but follow different action sequences. This illustrates how instrument control leads to distinct continuations of the same scene with physically accurate tissue deformations.
These rollouts allow inspection of whether instruments follow the controls, whether tissue responds coherently to interaction, and whether anatomical and instrument appearance remain consistent over time.
Adapting to your own data and controller
Cosmos3-H-Dreams includes four-step and two-step checkpoints for cholecystectomy. The system can be adapted to other datasets and embodiments with synchronized video and compatible action representations.
To train a real-time student model on your own dataset, the release provides a complete recipe for teacher fine-tuning and Self-Forcing distillation. It describes data preparation, public teacher-training tools, and the distillation stages. Controller integrations must use the same action conventions and normalisation as training.
What is next
The advance in visual realism opens several research directions:
Model the full procedure. Expand coverage and temporal consistency so all phases of cholecystectomy can be explored systematically, with clear task boundaries and success criteria.
Connect generative and numerical simulation. Combine learned visual realism with explicit geometry, contact and tissue mechanics. Numerical models and measured data could help constrain and validate deformation, making simulation behaviour inspectable as well as convincing.
Evaluate with surgeons. Assess anatomical accuracy, instrument behaviour and tissue response, turning expert feedback into reproducible benchmarks.
Support more instruments. Move beyond two controlled tools to richer coordination, occlusion and interaction.
Develop autonomous surgical policies. Couple world action models to Cosmos3-H-Dreams in a closed loop, so policies can learn defined cholecystectomy subtasks.
These are research directions, not established clinical capabilities. Cosmos3-H-Dreams is a research simulator, not a validated system for patient care. It offers a foundation for studying surgical interaction in an increasingly realistic environment.
Get started
Explore the models, data, and runtime behind Cosmos3-H-Dreams:
- Cosmos3-H-Dreams code and examples: GitHub repository
- Cosmos3-H-Dreams model: Hugging Face checkpoint
- Cosmos3-H-Embodied-Nano: Hugging Face model and GitHub repository
- Adapt to your own data: Step-by-step recipe for teacher fine-tuning and self-forcing distillation
- Open-H-Embodiment: Hugging Face dataset
- FlashDreams: GitHub repository
- NVIDIA Cosmos 3: GitHub repository
Bring your controllers, your trajectories, and your research questions. The team looks forward to seeing what the community builds inside these new surgical worlds.
