BAAI’s Orca world model matches specialised robotics systems without ever seeing a single action label
BAAI has released Orca, a “world foundation model” that matches the performance of specialised systems across five robotics tasks. Crucially, its base model was trained without a single action label. The approach aims to address the chronic data shortage facing robotics.
The Beijing Academy of Artificial Intelligence (BAAI) argues against the dominant trend of predicting the next token, video frame, or robot action. Orca instead models the next state of the world in an abstract internal representation. Intelligence should not be defined by specialised prediction models like language models or robot controllers. What is needed are models that build a general grasp of how the world changes and use that as a base for many different tasks.
Two training methods that work together
Orca combines two learning modes. “Unconscious learning” uses raw videos without any captions. The model sees an image and predicts what the next one will look like, not at the pixel level, but in an abstract space. It picks up motion patterns, occlusions, and typical scene dynamics along the way.
“Conscious learning” adds verbal instructions. Videos are split into segments, each labeled with a description of the state change, so the model learns how a state shifts when a specific action happens. It also trains on classic video question-answering tasks so it can still respond to natural language queries.
A frozen core with swappable output heads
The pre-trained language-image model Qwen3.5 serves as the base. After training, this core stays frozen. For each output type, the researchers attach a separate, smaller module that converts the internal state into the desired form. Text runs through Qwen3.5’s existing language head. For images, Stable Diffusion 3.5 stays unchanged too, with only small upstream adapters trained to pass Orca’s internal state to the image generator. Robot actions come from a control module trained from scratch called “Action Expert.”
The split is intentional. The team says that it is not trying to chase top scores on single benchmarks. Instead, they want to show that a well-trained internal world state can serve as a shared base for very different tasks.
For training, the researchers put together 125,000 hours of video footage, 160 million event descriptions, and 11.5 million question-answer pairs. The videos span four views, covering first-person shots of everyday interactions, third-person shots of object handling, robot recordings without action data, and naturally occurring scenes. Only one-tenth of the video data went into the current version.
Scaling behavior and benchmark results
Orca was trained at two sizes, with 0.8 and 4 billion parameters. Training loss drops steadily with more data and larger models. Early tests also show that the better the internal world state gets during pre-training, the better results are across all three output modes.
On text benchmarks, Orca-4B posts the best overall average of 51.8 percent across MVBench, TemporalBench, 3DSRBench, and SWITCH among the compared small VLMs and larger world models. It beats several VLM baselines like Qwen3.5-4B, Gemma 4-4B, and DeepSeek-VL2-3B on average, though it does not win every single test since Qwen3.5-4B leads on MVBench, for instance. Orca-4B also tops the much larger world models Emu3 (8B) and Emu3.5 (34B) on average.
For image prediction, the researchers built their own benchmark called PRICE-V0.1. This uses real-world robot and first-person scenes and asks the model to generate an image showing the result of a command like “close the microwave.” Orca-4B hits 59.8 percent on average, beating specialised image generators like FLUX.2 small (56.1 percent), FLUX.1-context (40.9 percent), and OmniGen2 (39.6 percent). Orca keeps the robot’s shape, its contact points with objects, and its link to the instruction intact better than pure image models, which often add irrelevant objects or hallucinated hands.
Robot control without action data during pre-training
In five manipulation tasks using a two-armed humanoid robot on wheels, including shelving books, stacking bowls, and scooping sugar, Orca matches π0.5, a system built specifically on robot data. Orca’s base model never saw during pre-training which movement goes with which image. For actual robot control, a separate module was trained afterward on 200 real-world recordings per task, pairing camera images with the movements performed.
V-JEPA 2.1 and Qwen3.5, paired with the same control module as baselines, fall well behind. Orca also shows an edge in error recovery, retrying after a failed grasp while π0.5 stays stuck in repeated failures according to the examples in the paper. The authors see this as a potential way to ease robotics’ chronic shortage of labeled action data.
What’s still missing
For training, the team uses its in-house FlagScale library with several memory and communication tweaks, reaching 2.91 training samples per second per GPU on H100 cards. That is about 4.4 times faster than StarVLA, a codebase widely used in robotics.
Orca still only learns from images and text, with sound, force, and touch missing entirely. Visual prediction runs in the space of a pre-trained image encoder rather than learning its own world space from scratch. At 0.8 and 4 billion parameters, the models are too small for full world modeling. The event descriptions cover only short windows of a few minutes. BAAI says a native world model trained from scratch on many signal types remains the end goal.
How world models should even be defined is still debated in AI research. A team led by Peking University proposed a unified definition with OpenWorldLib that rules out pure text-to-video models like Sora. A Tsinghua University benchmark showed that even Sora 2 and Veo 3.1 struggle with physically plausible scene progression. In robotics, World Action Models combine state transition prediction with action generation, making it possible to learn from ordinary everyday videos that contain no action data at all.
