Ai2 Releases ToyBlocks to Test How Robots Learn and Fail

Ai2 released ToyBlocks on November 6, 2026, as the first environment in its FoundationWorlds framework, pairing an open robot simulation with generated demonstrations, two trained models and evaluation tools. The framework is designed to make robot-learning experiments repeatable and failures easier to trace.

The setting puts a simulated arm and labeled blocks on a tabletop. Pushing, flipping and stacking are joined by search, sorting and arrangement tasks: a block’s needed letter or picture can be hidden, so the robot must turn it before deciding where it belongs. Varying block size, table height and other properties between scenes tests whether policies cope with changing geometry.

The constrained domain lets Ai2 pursue four properties at once: credible physics and visuals, scalable generation of tasks and demonstrations, problems requiring reasoning as well as dexterity, and coverage of the tasks the domain can support.

Automated Demonstrations and Open Models

Reinforcement learning and motion planning create short-task skills in Nvidia’s Isaac Lab. Skill generation can use privileged simulator state, including exact block positions and orientations, instead of camera estimates, automating behaviors that would otherwise require teleoperation on a physical robot. A task-and-motion planner then selects and sequences skills to produce labeled demonstrations. Ai2 says hundreds of plans can execute concurrently on one GPU at up to 24 times real-time performance.

Researchers can weight the resulting training mixtures toward particular skills and evaluate policies on tasks withheld from training. That makes the data generator an experimental control rather than simply a source of larger datasets.

Goals are written as PDDL planning specifications in the ToyBlocks repository, allowing the evaluation system to score completion and intermediate progress without bespoke metrics for each task. Demonstrations are stored in LeRobot format. A hierarchical policy can swap either level for an oracle supplying a correct plan or action, separating bad decisions from faulty motor execution. Failed states can also return to data generation for fresh demonstrations and recoveries.

Ai2 reports that the initial release contains 85,336 long-horizon demonstrations, each comprising roughly 6–12 skill demonstrations depending on the set, covering nearly 1,600 hours of robot execution and 144 million frames. The public dataset uses Parquet-formatted LeRobot data under an ODC-BY license, with easy, medium and hard subsets. The main code is Apache-2.0 licensed; specified submodules and downloaded assets carry their own licenses.

Ai2’s model cards divide the released policy into two levels: Molmo2-4B-tb, a roughly 4.5-billion-parameter model, reads a front-camera image and text goal to produce a short plan, next action and target point; ManiFlow-tb, with 322.5 million parameters, uses front and gripper images, robot state, task information and a target heatmap to generate action chunks.

Generalization and Agent Experiments

The spatial experiments cover Stack, Unstack, Flip and Rotate, distinguishing interpolation within the training range from extrapolation beyond it. Examples include training with blocks in one workspace region and testing in another, or withholding particular stack heights. Ai2 reports poor initial generalization for ManiFlow, π0.5 and MolmoAct 2 despite large-scale pretraining, and concludes that action-model generalization remains far from solved.

The agentic tests expose a different weakness. Ai2 says its Claude Opus 5.5-based system solved all 30 classical block-stacking tasks from text descriptions and all 30 when choosing actions in a closed loop from repeated camera images. In search-and-spell scenes containing up to eight blocks, 50% lacked a required letter and had to be identified as impossible. The agent solved the task or correctly identified infeasibility in 63% of cases; look-alike-letter confusion accounted for 13% and incorrect letter orientation for 15%.

A third agent experiment allowed the system to test unexpected skill behavior and write new skill descriptions from the results. Ai2 reports that these learned descriptions raised success from 24% with default descriptions to 63%, exceeding the 36% achieved with descriptions written by a human expert.

Physical Robot Tests and Their Limits

For physical evaluation, models trained entirely on ToyBlocks simulation data ran without real-world training examples on a Franka FR3 arm with a Robotiq gripper. The 30 scene configurations contained 4–10 blocks seen during training. An episode counted as successful only when the relevant blocks reached their target poses without collisions; an operator intervention for unsafe motion counted as failure.

To narrow the simulation-to-reality gap, Ai2 tuned simulated joint friction and armature against real trajectories. Data generation randomized object size, friction and table height along with lighting, backgrounds, materials and camera parameters.

Ai2 reports ManiFlow atomic-skill success rates of 80.0% in simulation and 81.1% on the physical arm. The hierarchical Molmo 2–ManiFlow system succeeded in 55.5% of simulated episodes (111 of 200) and 50.0% of physical episodes (15 of 30).

Ai2 qualifies those results on two fronts. The physical hierarchy evaluation used the easiest simulated configuration, with block characters visible and upright, and further trials are needed to establish statistical confidence across tasks. Its ManiFlow card also records success rates from 54% to 59% across four runs of the same 128 simulated episodes, because simulation and batched inference are not bitwise deterministic.

Ai2 plans a second release with a larger dataset, more skills and broader benchmarking, plus fully integrated reinforcement-learning training that is absent from the current package. Longer-term directions include two-arm manipulation, dexterous hands with force and touch sensing, and interaction with unstructured block piles.

ToyBlocks’ immediate contribution is therefore experimental control: researchers can alter training mixtures, separate planning from execution and route failures back into data generation. The initial results also define its limits, with weak simulated generalization and a narrow physical evaluation tempering what these early transfer rates establish about robot learning beyond the block world.