WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions
AgiBot and NVIDIA researchers built a preprint benchmark that feeds robot world models the exact actions a real robot performed and checks whether the predicted video shows what really happened, including failed grasps and spills. Every one of seven models tested drifted on long runs, and the best rollouts of the robot arm often still got the physical outcome wrong, a warning for companies that plan to evaluate or train robots inside such models.
Given the same starting camera view and the same low-level action sequence, does an action-conditioned world model predict what actually happens when a real robot executes those actions, including mistakes and failures, across robots and over minutes rather than seconds?
Video-generation models are being turned into action-conditioned world models (ACWMs) that predict how a scene changes as a robot moves, with the promise of generating training data and testing policies without real robot time. A rollout can look convincing and still be wrong: the paper's opening example shows a model rendering a successful fruit pick when the real gripper missed. Existing benchmarks score visual quality or follow text instructions, or test executability in simulation, and none compare action-conditioned predictions directly with paired real executions over long episodes that include failures.
General video benchmarks judge visual quality, controllability and physical plausibility. Embodied benchmarks such as EWMBench, PAI-Bench, WoW-World-Eval and RBench condition on an image plus a task instruction rather than a specific action trajectory. WorldArena and WorldArena 2.0 test world models as data engines, policy evaluators and planners but mostly inside simulators, and RoboWM-Bench replays actions extracted from generated videos in real-to-sim environments.
WorldOracle replays recorded real action trajectories open loop through each model and compares the output with the real video on four scores: physical state correctness (a VLM checks annotated outcome criteria such as 'the green cup is grasped' at the end of each manipulation attempt), interaction spatial alignment (DINOv3 feature matches on SAM3-segmented arms, hands and objects), key interaction area fidelity (PSNR, LPIPS and V-JEPA similarity inside those masks only) and multi-view action coherence across head and two wrist cameras. It covers four tasks with different physics (picking fruit, hanging a towel, pouring water, pulling a drawer) and three embodiments: AgiBot G2, the YAM bimanual station and human hands. The new AgiBot G2 training set has 1,779 one-minute episodes with about 20% failed segments; the evaluation sets have 100 G2, 75 YAM and 100 human-hand episodes of about two minutes with about 40% failures, plus 60 G2 episodes driven by a pi0.5 policy, whose hesitations and corrections form a different action distribution. Scores are reported for the first 30 seconds and for full episodes.
Metric rankings agreed with human judgements in 88.1% to 93.2% of pairwise comparisons, with Spearman correlations of 0.72 to 0.82. Of the seven models (IRASim, Ctrl-World, DreamDojo, Cosmos 3 Nano, GE-Sim 2.0, JointControlVideo and OSCAR), GE-Sim 2.0 led on AgiBot G2 with a single-view score of 73.86 for the first 30 seconds and 63.33 over full episodes. DreamDojo fell from 50.56 to 27.08 and Cosmos 3 from 53.99 to 26.60 over the same horizons. GE-Sim 2.0 also led interaction spatial alignment on both robots, 81.40 short and 73.49 long on G2 and 74.97 and 59.78 on YAM, and scored 78.70 on the policy-rollout split. Yet over full G2 episodes its physical state correctness was 60.26 and key-area fidelity 56.24 while multi-view coherence was 96.96, meaning it tracked the arm well but often predicted the wrong grasp, contact or outcome. Across the four G2 tasks its average single-view score was 73.9 against 54.0 for the next best model.
This is a preprint and has not been peer reviewed. The authors say the three embodiments are not a controlled comparison: G2 models were adapted on the benchmark's own training data, YAM on the public ABC-130K set with a larger gap, and human-hand models were tested zero-shot. They also present the links between model design and scores as hypotheses, not causes. In our reading, the top model GE-Sim 2.0 comes from the AgiBot side and its paper shares authors with this benchmark, and G2 is AgiBot's own robot with a training set built for the test, so its lead should be confirmed by independent groups. Physical correctness relies on a VLM judge, and four tabletop tasks are a narrow slice of factory or home work. Datasets, code and leaderboard are promised but not yet released.
Companies selling world models as robot simulators, including NVIDIA with Cosmos and AgiBot with its GE models, now face a test that checks predictions against reality rather than visual polish. For robot makers that plan to evaluate policies or generate training data inside a world model, the finding that all models degrade over long horizons and can hallucinate success argues for keeping real-robot validation in the loop for now. A public leaderboard, once released, could become a procurement reference within a year; reliable long-horizon simulation looks further away.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.