Allen Institute for AI has released ToyBlocks, a simulation framework designed to evaluate robot learning in controlled conditions.
Current robot demonstrations often cover narrow tasks without explaining why a system fails. Scientists usually study complex systems in small, controlled environments like petri dishes. Robotics requires similar settings where researchers can alter training data, repeat tests, and trace failure causes.
The team introduces FoundationWorlds, a framework for creating these environments. Each world pairs a controlled simulation with tools for generating tasks and training demonstrations, plus metrics to measure behavior over time. The goal is to study how models improve with more data and how to combine high-level reasoning with precise physical control.
ToyBlocks is the first FoundationWorld. It simulates a robot arm and blocks on a tabletop. The setting is simple, yet the robot must reason about block interactions while pushing, flipping, and stacking them.
The system automatically generates demonstrations without requiring time-consuming teleoperation, where a person remotely controls the robot. Researchers can easily vary these training experiences. A team could generate more pushing or flipping demonstrations, then evaluate performance on tasks kept out of training.
Starting with ToyBlocks, the team aims to extend the approach to increasingly complex environments.
How ToyBlocks works
Building on earlier work including MolmoAct 2, MolmoBot, and MolmoSpaces, ToyBlocks gives researchers a way to study robotics questions in a deliberately constrained domain.
As a FoundationWorld, ToyBlocks meets four requirements: realistic physics and appearance, automatic generation of tasks and demonstrations at scale, challenges combining reasoning with dexterity, and broad coverage of possible tasks within a domain.
Open-ended domains require modeling a wide range of objects and interactions, making it infeasible to fulfill all requirements. ToyBlocks’ narrow scope makes it possible to meet them all.
Within this smaller setting, researchers investigate how a robot learns to perform reasonable tasks while ensuring the simulated arm and blocks behave similarly to physical counterparts.
A physical robot gripper sits above wooden blocks in the ToyBlocks setup.
The system goes beyond picking and stacking. Pictures and letters on the block faces allow for searching, sorting, and arranging tasks. A robot may need to turn a block to see a hidden face before deciding where it belongs. Domain randomization varies block size, table height, and other physical properties from scene to scene, testing how well models generalise as geometry changes.
ToyBlocks automatically generates skills and task demonstrations for training models or adapting them through fine-tuning. High-level reasoning models decide what to do; low-level action models control the robot’s movements. The framework also helps design experiments and validate hypotheses.
The system uses reinforcement learning and motion planning to generate skills for short-horizon tasks. Skill training occurs in Isaac Lab, Nvidia’s open-source framework for robot learning in simulation. Through trial and error, these policies learn non-prehensile skills like pushing a block or flipping it onto another face without picking it up.
Skill generation can draw on privileged information the simulator supplies directly, such as a block’s exact position and orientation, instead of estimating it from camera images. This enables the automatic generation of skills that would require teleoperation in the real world.
To generate demonstration data, a task-and-motion planner automatically selects and sequences the skills needed to solve a task and plans the robot’s movements between them. Running each plan in simulation produces a dataset of recorded demonstrations along with labels describing which task or skill the robot performs. ToyBlocks scales data generation by executing hundreds of plans in parallel on a single graphics processing unit, reaching up to 24 times real-time performance.
Acting from sensor observations adds a perception problem. A model must estimate a block’s pose, which the simulator could supply directly. To reflect real-world conditions, ToyBlocks trains models that map raw sensor observations to actions.
Execution of Molmo 2 and ManiFlow trained in ToyBlocks shows Molmo 2 selecting skills such as rotation and flipping, indicating the relevant block via a pointing interface, while ManiFlow controls the simulated arm to execute the commanded skills.
The controllable data generator lets users create training mixtures that emphasise particular skills or fill gaps in existing datasets. Researchers can study how much data a model needs and which experiences help it generalise to unseen tasks. For example, they could test whether adding rotation and pick-and-place demonstrations improves flipping, or vary the balance of examples inside and outside a defined workspace region—evaluating their choices on tasks kept out of training.
Importantly, evaluation of trained models generates metrics beyond success rates. It provides information such as execution speeds, forces exerted on objects, and task progress measured via the same specifications used by the planning module.
A webpage interface lets users connect Claude or another frontier model and use natural language to instruct the robot or try tasks. This offers a way to explore how a model plans and acts without needing robotics expertise.
Research building on ToyBlocks
The evaluations below use ToyBlocks to study how training data and methods affect performance, spatial reasoning, generalisation to new letters and block appearances, multi-step planning, transfer to physical robots, and frontier models that act through tools. Users can easily design their own tasks and measures to investigate other questions—for example, whether a model performs better when it selects objects by pointing or by issuing text commands such as “pick the yellow block.”
Low-level action models: VLA spatial generalization
Low-level action models, such as vision-language-action models, map language instructions and visual observations to robot actions.
The premise of these models is that pretraining on large-scale datasets allows them to rapidly generalise to different scenarios or a specific workspace. However, evidence on how well these action models generalise remains conflicting. Some studies report that these models are brittle to changes in inputs, such as different object layouts and lighting, whereas others show they generalise well. The team argues these studies often rely on real-world datasets with limited flexibility in spatial variations. ToyBlocks enables testing how well action models generalise in a simple, easily reproducible setting with meaningful task-space diversity.
Generalisation tests vary the skill sequences needed to complete an instruction, the letters or colors on the blocks, and the number of blocks in a scene. A stacking study can withhold four-block stacks from training, then test whether a model can combine familiar movements to build them. Workspace studies distinguish performance at new positions within a training region from performance beyond its boundaries, where the model has to extrapolate.
The team considers four tasks: Stack, Unstack, Flip, and Rotate. For these tasks, they study interpolation, which involves new configurations within the training range, and extrapolation, which involves configurations beyond that range. For example, evaluation involves training on scenarios where objects spawn only in the left region and testing on the right region, or training on stack heights of 2 and 5 blocks and testing on heights 3 and 4.
Spatial generalisation studies for VLA models show that despite large-scale pretraining, popular action models generalise poorly in initial results. The team improves generalisation and analyses the contribution of each technique. The chart’s LoRA label stands for low-rank adaptation, a method for fine-tuning a model with a small set of trainable weight updates. The team believes action model generalisation remains far from solved; ToyBlocks provides a valuable testbed for systematically exploring generalisation techniques.
High-level reasoning: vision-language models
Long-horizon tasks require many steps and allow measurement of how well models can plan a sequence of actions to solve them. The framework enables creation of such tasks and oracles, reference components that provide correct plans or actions, at both high and low levels. This allows isolation of whether high-level planning or low-level execution failed when a model fails at a task. An example long-horizon task might be to spell “Ai2” with a block stack. Because the letters may not be immediately visible, the agent must first choose actions that reveal information, and then plan and execute a sequence of actions to build the stack.
The team evaluates hierarchical models with explicit high-level planning and low-level execution models. In this instance, they choose Molmo 2, a vision-language model, as the high-level planner, and ManiFlow as the low-level policy. Molmo 2 chooses the skill to execute and the block to execute it on, while the low-level model executes the instruction. They train low-level models such as ManiFlow to pause after completing the execution and query the high-level model for the next instruction. They evaluate the model’s planning abilities on three scenarios of increasing difficulty. The easiest scenario initialises with each character upright and facing the camera. The medium scenario has characters that may not be upright, and the hardest scenario contains characters that are not visible at the start of the episode.
Success rates on three-letter block-spelling tasks using Molmo 2 as the high-level reasoning model and either ManiFlow or a motion planner for low-level manipulation show an example task is “Stack Ai2”. Molmo 2 and ManiFlow are trained on ToyBlocks and evaluated across three difficulty settings and two generalisation conditions.
High-level reasoning: agentic systems
Experiments with agentic systems test how frontier models plan and act through tool calls. The team studies three settings using Claude Opus 5.5. In the search-and-spell setting, they expose ToyBlocks skills through the Model Context Protocol.
What it means
ToyBlocks provides a standardised way to test robot learning without the cost and variability of physical hardware. Researchers can now isolate specific failure modes, such as spatial reasoning gaps or planning errors, by controlling the simulation environment precisely.
