MirroS has released Code-as-World, a system that converts video footage into executable MuJoCo physics simulations. The approach treats pixels merely as evidence of a physical scene rather than the scene’s underlying structure. A standard video model can predict plausible frames without ever representing mass, contact, or gravity. Instead, the system generates a scene.json file that an agent verifies against the source video and that anyone can edit to re-simulate. An agentic loop recovers these programs from real footage in up to five rounds. The verified worlds then serve as training data carrying exact physical labels, which raw video does not possess. Trained on that supervision, the Code-as-World-VL-9B model scores 55.4 MRA on QuantiPhy-validation, placing it ahead of Gemini-3.1 Flash at 54.8 and roughly 15 points above the strongest open-weight baseline.
Is it deployable?
Yes, at the research and internal-prototype tier. MirroS shipped the GitHub repo and two checkpoints — Code-as-World-VL-4B and Code-as-World-VL-9B — under Apache 2.0. These models were fine-tuned from Qwen3.5-4B and Qwen3.5-9B. Both run as BF16 safetensors served by vLLM behind an OpenAI-compatible /v1 endpoint, processing 16 sampled frames per video with a max-model-len of 4608.
The idea: pixels are evidence, not ontology
The MirroS technical report argues that video models, 3D reconstruction, and captions each recover part of a scene but none recovers its mechanism. Code-as-World represents a scene as an executable world representation (EWR), a triple p = (C, E, A):
- Composition: objects, geometry, metric dimensions, mass, friction, gravity. Floors and walls are static physical entities so they can support and collide.
- Evolution: initial states, forces, contacts, collisions, termination conditions, duration. Executing it expands composition into a full state trajectory.
- Appearance: camera, lighting, materials, background, frame rate, render config. Changing it never changes the physics.
In the released implementation, that triple compiles into a scene.json executed in MuJoCo, with two interchangeable engines: an animation engine (kinematic poses) and a physics engine (forces and contacts).
Agentic discovery instead of one-shot prediction
Recovering an EWR from a video is an inverse problem, so the team frames it as abductive search. An agent runs propose → instantiate → execute → render → verify for up to K = 5 rounds. For video input, SAM 3 supplies instance masks and image-plane tracks, VGGT-Omega estimates depth and camera geometry, and SAM 3D generates per-object meshes. Candidate rollouts are projected back into the input view and compared at selected key frames on RGB, depth, masks, and trajectories. Frame-level discrepancies aggregate into structured feedback Δ that guides the next revision; when the budget runs out without acceptance, the hypothesis is rejected.
At a matched five-evaluation budget, the loop beats Best-of-5 independent sampling on Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D — and the result repeats under the second execution engine. Candidate videos come from WISA-80K after motion-focused filtering; sim-to-real re-rendering uses Wan2.2-VACE plus an internal video model.
Verified worlds as training supervision
Phase 1 is supervised fine-tuning on 73,335 image-space QA pairs built from RefCOCO/+/g, RefCLEF and GOT-10K, covering extent, position, displacement, velocity and acceleration in raw pixels. Phase 2 applies GRPO to world-space VQA drawn from 1,585 text-driven and 988 video-driven executable worlds, rewarded on scale-normalized numerical accuracy plus unit and format terms. Training used eight NVIDIA H100 GPUs.
On QuantiPhy-validation (159 items, MRA macro-averaged over 2S/2D/3S/3D): 4B = 50.6, 9B = 55.4, 27B reasoning = 58.6, against Gemini-3.1 Flash at 54.8, ChatGPT-5.1 at 48.4, and the strongest open-weight baseline Qwen3-VL-32B-Instruct at 40.2. The ablation is the more useful number: image-space-only scores 44.2 (4B) and 50.9 (9B); adding both world-space sources lifts them to 50.6 and 55.4. Pixel-level grounding improves too — the 9B goes 63.7 → 68.3 on RefCOCO and 20.1 → 26.6 on GOT-10K after world-space RL.
What it means
For people making things, this shifts the workflow from guessing physics to editing code. Instead of training a model to predict how an object will fall based on visual patterns alone, a developer gets a script they can inspect and modify. If a simulation looks wrong, the agent rewrites the code until the rendered video matches reality. This allows for precise control over physical properties like friction and mass, which are usually hidden parameters in standard video generation. The verified outputs provide a dataset where every label is physically verifiable, removing the guesswork from training future models.
