LADDER: A Long-Horizon Robot Manipulation Benchmark

Abstract

General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-\infty-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only \sim25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only \sim62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: ladder.insait.ai

1 Introduction

General-purpose robot manipulation policies learn diverse skills from large datasets of teleoperated demonstrations (Brohan et al., 2023; Walke et al., 2023; Khazatsky et al., 2024; Black et al., 2025a; NVIDIA, 2026). However, these demonstrations predominantly cover short-horizon, single-subgoal tasks. Because collecting demonstrations for every multi-subgoal combination is infeasible (Mees et al., 2022; Liu et al., 2023), generalist robot models rely on zero-shot skill composition: using low-level policies trained on single-subgoal tasks to solve long-horizon tasks without task-specific fine-tuning.

When presented directly with full instructions involving multiple subgoals or conditional constraints, current policies degrade rapidly. Existing benchmarks offer limited ability to isolate why this failure occurs: suites such as CALVIN (Mees et al., 2022) and LIBERO-Long (Liu et al., 2023) evaluate short horizons and often require in-domain training rather than testing zero-shot composition. Recent multi-task evaluations such as RoboLab (Yang et al., 2026) scale task complexity, but conflate task horizon with scene clutter and instruction phrasing. Systematically isolating long-horizon composition requires evaluating low-level policies zero-shot across extended horizons while holding the environment and the object sizes fixed.

In this paper, we introduce LADDER, a benchmark for long-horizon robot manipulation composed of three evaluation suites: SimplerEnv-Long, RobotArena-\infty-Long, and RoboLab-Long. Because evaluating thousands of multi-subgoal trials on physical hardware is infeasible at scale, we use physics-based simulation environments that closely match real setups and have demonstrated strong sim-to-real correlation (Li et al., 2024; Jangir et al., 2026). Rather than inflating task horizon through unstructured scene changes, our benchmark structures long-horizon manipulation across eight spatial and logical task categories (Figure 1). Across all suites, task horizon scales systematically from 1 to 10 subgoals within the same environment, using explicit programmatic predicates to measure intermediate subgoal progress alongside full task completion.

The benchmark evaluates 145 tasks across two robot embodiments (WidowX and Franka) over 2,024 simulated episodes per system. On single-subgoal control tasks, low-level policies achieve a high success rate: 85.4% in simulation and 66.7% on the real WidowX. However, when given multi-subgoal tasks (2 to 10 subgoals), the low-level policy collapses, falling to 5.2% to 15.8% in simulation (0.5% at 8 to 10 subgoals) and, in real-world experiments, to 1.2% on WidowX and 5.6% on Franka.

To bridge the long-horizon gap, recent frameworks deploy high-level orchestrators across distinct execution backends: decomposing instructions into text subgoals for learned policies (Chen et al., 2026), synthesizing executable code over motion primitives (Fu et al., 2026), or generating symbolic plans for task-and-motion planners (Shen et al., 2026). Although not specifically designed for long-horizon tasks, GPT-Policy (Cheng et al., 2026) uses GPT-6 Astra to invoke robot tools and achieves the strongest performance on LADDER. We therefore systematically study the orchestrator design space, characterizing four modular mechanisms: subgoal planning, execution triggering, visual verification, and failure recovery. Orchestration enables low-level policies to execute tasks up to 10 subgoals without fine-tuning, improving multi-subgoal success to 20.6% to 22.7% in simulation and 31.0% to 58.3% in real-world experiments. Tool selection systems, which call motion primitives instead of a learned policy, go further. In all cases, LADDER remains challenging despite some methods spending impractical budgets. However, decomposition alone is insufficient: low-level policies frequently fail on individual subgoals. Offline, vision-language models decompose instructions accurately (up to 93.1%), but struggle with scene state tracking (at most 70.8%).

Our contributions are as follows:

• A long-horizon manipulation benchmark across three suites, spanning 1 to 10 subgoals and eight structured task categories with programmatic progress predicates (145 tasks, 2,024 simulated episodes per system).
• A systematic study of hierarchical orchestration over frozen low-level policies, characterizing the modular roles of planning, triggering, visual verification, and failure recovery, evaluated alongside representative alternative paradigms that achieve higher success at a higher cost.
• Real-world experiments across two robot platforms (WidowX and Franka) demonstrating that orchestration improves multi-subgoal execution.

2 Related Work

Manipulation benchmarks. Existing robot manipulation benchmarks provide limited means to study how task horizon affects the performance of a system (see Table 1 for an overview). CALVIN (Mees et al., 2022) and LIBERO-Long (Liu et al., 2023) include multi-step tasks, but their standard evaluation protocol for learned policies involves fine-tuning on in-domain demonstrations. Their simulated environments also have limited alignment with real-world robot appearance and dynamics, so success in these benchmarks does not reliably transfer to the real-world. SimplerEnv (Li et al., 2024) and RobotArena-\infty (Jangir et al., 2026) reduce this gap by matching simulated scenes and robot dynamics to real data, while RoboLab (Yang et al., 2026) uses photorealistic simulation. However, the first two focus primarily on single-step tasks, while RoboLab and its long-horizon extension RoboVoLo (Chen et al., 2026) vary task horizon along other axes of complexity, such as scene background and out of distribution objects. In contrast, building on SimplerEnv, RobotArena-\infty, and RoboLab, we propose a benchmark that isolates task horizon scaling by systematically increasing the number of subgoals of a task while holding the environment and underlying skill fixed. Hierarchical orchestrators. Recent advances in the visual and semantic reasoning capabilities of foundation models have motivated hierarchical orchestration as an approach to long-horizon manipulation. Orchestration systems use frontier multi-modal models to break an instruction into subgoals, and delegate their execution to a robot-control backend. Prior work has used code over motion primitives (Liang et al., 2023; Fu et al., 2026), task and motion planning (TAMP) (Shen et al., 2026), and learned skills or policies (Ahn et al., 2022; Huang et al., 2022; Belkhale et al., 2024; Shi et al., 2025; Li et al., 2025; Chen et al., 2026) as backends. However, it remains unclear how different design choices of these systems affect performance as task horizon systematically increases.

3 Long-Horizon Manipulation Benchmark

We build LADDER on three established platforms: SimplerEnv (Li et al., 2024), RobotArena-\infty (Jangir et al., 2026), and RoboLab (Yang et al., 2026). Each suite—SimplerEnv-Long, RobotArena-\infty-Long, and RoboLab-Long—keeps the original assets while adding four key features (Table 1): (1) controlled horizon scaling, from 1 to 10 subgoals with 138 tasks that extend a shorter task of the same scene or are extended by a longer one, and 36% of tasks comprising 6 or more subgoals; (2) diverse task categories, covering eight spatial and logical task structures (Figure 2); (3) explicit per-subgoal predicates, checking every subgoal programmatically; and (4) multiple embodiments, evaluating policies across multiple robot platforms.

Problem setting. We define a long-horizon task as one that requires completing several subgoals in sequence from a single full instruction. Each subgoal is a single pick-and-place action (for example, putting a carrot on a plate), the target location is its destination, and the number of subgoals is the task’s horizon. An episode is one execution rollout starting from a randomized scene layout.

The systems we evaluate control the robot with a low-level policy, a task-and-motion planner, or motion primitives. A low-level policy maps visual observations and a language instruction to robot control actions. Generalist low-level policies are trained on demonstration datasets (e.g., BridgeData and DROID) of short-horizon skills, like placing one object in a container (e.g., a plate, towel, bowl, or bin). We evaluate them without fine-tuning on data of the target tasks and environments.

Task horizon. Task horizon is the number of subgoals an instruction requires the robot to complete. To isolate the effect of horizon, tasks form ladders: within one scene and task category, a longer task keeps all subgoals of a shorter one and adds more (for example, “put the carrot on the plate” and “put the carrot and the spoon on the plate”). Every task on such a ladder is incremental. In total, 138 of our 145 tasks are incremental (Table 1). Except in exclusion tasks, instructions explicitly name each target object rather than using vague collective phrases (like “put everything in the bin”), ensuring that we test instruction following rather than random searching. Tasks scale from 1 up to 10 subgoals, which reaches the maximum object density that fits in the workspace without objects blocking basic grasps. Task categories. Task categories define the spatial and logical structure of each instruction (Figure 2). Across the three suites, we evaluate eight task categories (every task: Appendix D): single destination, moving multiple objects to one container (e.g., “put the spoon and the carrot on the plate”); split destinations, sorting objects into different containers (e.g., “put the lemon on the plate and the orange in the bin”); transfers, moving objects out of one container into another (e.g., “move the carrot and the spoon from the plate to the blue towel”); relay routing, moving an object through an intermediate stop (e.g., “first put the carrot on the plate, then on the towel”); revisitations, returning to a container used earlier (e.g., “put the spoon on the cloth, then in the pot, then back on the cloth”); exclusions, moving specific objects while leaving distractors alone (e.g., “put everything except the eggplant on the plate”); ordered sequences, following a strict order (e.g., “first put the banana in the bin, then the orange”); and duplicate instances, picking specific items among identical visual copies (e.g., “put both corn cobs on the cloth”). Evaluating all eight categories allows us to measure how different geometric and logical constraints affect policy success (§4).

Scene construction. Each scene is physically stable and within the robot’s reachable workspace. A longer task adds more objects and a larger container for them; object sizes stay the same, so grasping difficulty does not grow with the horizon. Every layout is validated by scripted policies that complete all subgoals, so failures are caused by sequential execution rather than reachability issues.

Evaluation metrics. We evaluate every subgoal with programmatic state checks. We report two main metrics: Task Success Rate (%), the percentage of episodes where all subgoals and constraints are satisfied at the end of the episode, and Subgoal Progress (%), the percentage of required subgoals completed, measuring partial progress on unfinished tasks. For every subgoal that is not completed, we also report a Failure Cause, such as a subgoal that was completed and later undone (§5.2). The step budget grows with the number of subgoals: 100 steps (20 s at 5 Hz) per subgoal on the WidowX suites, and a per-horizon budget of 900 to 4950 steps (60 s to 330 s at 15 Hz) on RoboLab-Long. The budget counts simulated time only: the simulation pauses while a system computes.

4 Characterizing orchestrators

To analyze how different orchestration methods support long-horizon manipulation, we characterize their operation around four design decisions (Figure 3, Table 2): (1) subgoal planning, (2) trigger mechanisms, (3) visual verification, and (4) failure recovery. This characterization informs our experimental comparisons and analysis, helping us identify how decisions in each of these aspects affect task execution and performance. We divide the evaluated systems into two groups. Learned-policy systems send text subgoals to a low-level policy. Tool selection systems do not use a low-level policy. Instead, they call motion primitives or a task-and-motion planner. In addition to existing systems, we introduce PSP-Agent (Plan-See-Pick Agent), a tool selection system in which a VLM plans text subgoals and executes each subgoal by calling manipulation primitives that verify their own outcome (Appendix B).

1. Subgoal planning. A plan decomposes a full instruction into sequential single-subgoal instructions (for example, decomposing “put the carrot and the spoon on the plate” into subgoal 1: “put the carrot on the plate” and subgoal 2: “put the spoon on the plate”). In our experiments, the planner is always a highly capable frontier Vision-Language Model (VLM). The only exception is the privileged verifier, which takes the ground-truth subgoals from the simulator. Our learned-policy variants and PSP-Agent generate the full list of text subgoals once, at the start of the episode (Text subgoals). VoLo (Chen et al., 2026), which we run with the low-level policy as its only execution tool, also generates a list of text subgoals from the first frame and replans the remaining ones during the episode. The other external systems use different plan representations: CaP-X (Fu et al., 2026) generates a Python program, TiPToP (Shen et al., 2026) generates symbolic goal predicates for its planner, and GPT-Policy (Cheng et al., 2026) selects only the next tool call. Low-level policy is the no-orchestrator baseline and passes the full instruction to the low-level policy without generating a subgoal plan. 2. Trigger mechanisms. Because the low-level policy runs continuously without knowing when to stop, the orchestrator needs a trigger condition to decide when to switch or advance to the next subgoal. We evaluate three subgoal-switching strategies: (1) Periodic trigger: advances after a fixed number of steps; (2) Proprioceptive trigger: advances to the next subgoal when internal robot state indicates an attempt has finished, specifically when the gripper opens after a sustained close (Appendix B); (3) Privileged trigger: a simulation-only diagnostic that checks ground-truth contacts at every step, advancing to the next subgoal the instant the current goal is reached; it is used by the Privileged verifier. Tool selection systems execute blocking commands and make the next decision when a command completes, either after each primitive call (Primitive completion) or after the full program (Program completion). TiPToP executes its plan open-loop, without a trigger. 3. Visual verification. When a trigger fires, a verifier determines whether the subgoal succeeded. Across our controlled variants and the privileged verifier, we use three verification settings: no verifier (None), which advances unconditionally; visual verification (Visual (VLM)), which queries a vision-language model using the latest camera observation and subgoal description; and simulator-state verification (Simulator state), used by the privileged verifier to check the task predicate directly. PSP-Agent verifies each primitive geometrically from the depth image (Geometric state), for example by checking that a grasped object has left its initial position. CaP-X and GPT-Policy query the VLM with the new camera image. 4. Failure recovery. After verification, the recovery mechanism determines whether to advance, retry, or replan. In our retry-based variants, a completed subgoal is removed from the queue and execution advances; an incomplete subgoal is retried in place. When a plan contains duplicate subgoals for interchangeable objects, an incomplete subgoal moves to the end of the queue so one difficult object does not consume the episode budget. The privileged verifier retries immediately. Without a verifier, the orchestrator always advances to the next subgoal (Advance). To replan (Replan), the VLM generates a new plan from the current camera image: VoLo replans the remaining subgoals, CaP-X generates a new program, and GPT-Policy selects a new tool call. PSP-Agent combines both mechanisms: it retries failed subgoals and replans after all subgoals have been attempted (Appendix B).

5 Experiments

We first study how task success scales with task horizon (§5.1). Then we investigate the dominant failures in different systems (§5.2) and whether simulation predicts real-world performance (§5.3).

Experimental setup and low-level policies. On SimplerEnv-Long and RobotArena-\infty-Long we evaluate a \pi_{0}-like (Black et al., 2025b) low-level policy that uses Qwen3-VL (Bai et al., 2025) as backbone and is trained on BridgeData v2 (Walke et al., 2023). On RoboLab-Long we use the public \pi_{0.5} (Black et al., 2025a) fine-tuned on DROID (Khazatsky et al., 2024) (Appendix A). For periodic-trigger variants, we use 27 steps (5.4 s at 5 Hz) on the WidowX suites and 250 steps (16.7 s at 15 Hz) on RoboLab-Long. In both cases, the interval is set to the median time the privileged verifier takes to complete a subgoal. Unless noted otherwise, visual verifiers use Gemini 3.7 Flash. GPT-Policy uses GPT-6 Astra. In simulation, the tool selection systems use the ground-truth depth image and camera calibration from the simulator. In real-world experiments, depth is estimated with MoGe-2 (Wang et al., 2025) from calibrated cameras. For the simulation results we report 95% Wilson confidence intervals over episodes, and test differences between systems paired over tasks (Appendix F).

5.1 Task Horizon Scaling

On single-subgoal tasks, the low-level policy reaches a similar success rate as the Proprioceptive verifier (85.4% vs. 80.2%). As task horizon grows, the low-level policy rapidly collapses to 26.9% at 2 subgoals and 0.0% at 5 subgoals. In contrast, orchestrator-based methods maintain higher success rates with increasing task horizon with Proprioceptive verifier achieving 58.4% at 2 subgoals, 8.3% at 5 subgoals and retaining partial progress up to 10 subgoals (Figure 4, Table 3). The Privileged verifier serves as a privileged reference for the specific low-level policy being evaluated. The best learned-policy orchestration methods, VoLo and the Proprioceptive verifier, decrease the gap of the low-level policy to this reference from 18.6 to 4.2–4.5 percentage points. Tool selection systems, which use ground-truth depth in simulation, are not limited by the low-level policy. GPT-Policy achieves the highest success rate on every suite (62.0% overall), and PSP-Agent is second (40.1%) while using the same VLM as the learned-policy orchestrators. Both exceed the privileged reference of the learned-policy systems, and on 5-subgoal tasks PSP-Agent still succeeds in 27.3% of episodes, against 11.1% for the privileged verifier. CaP-X performs on par with the best learned-policy methods (25.0%), while TiPToP (9.4%) does not improve over the low-level policy.

Trends also differ across task categories. On Exclusion and Relay tasks, the low-level policy nearly always fails while the Proprioceptive verifier approaches the privileged verifier. On Single destination, Split destination, Transfer and Ordered the gain of VoLo is also steady, but smaller. On Duplicates the low-level policy already approaches the privileged verifier and orchestration provides only tiny improvement. Revisit, whose tasks mostly have 6 or more subgoals, resists both: it needs intermediate transfers where execution errors compound, and reaches only 2.3% on average over the learned-policy systems (Table 3).

To assess the efficiency we analyze the elapsed time and the compute used by each method. We calculate elapsed time as the percentage of the simulated time budget needed to complete a task and compute as the number of VLM calls per episode (Figure 5). We do not compare wall-clock time, because it depends on the hardware of each run and on queueing at shared VLM endpoints (Appendix G). Most progress happens early: the Proprioceptive verifier completes 42.1% of all subgoals in the first half of the budget and 49.5% by the end, indicating that a longer budget would help little. These results suggest that the end-to-end low-level policy limits the returns to test-time scaling: VoLo makes more VLM calls than the Proprioceptive verifier but achieves similar success, while PSP-Agent and GPT-Policy pair higher call counts with higher success using motion primitives.

5.2 Failure Analysis

Next, we investigate the dominant failure modes for each method and try to understand whether learned-policy methods are bottlenecked by the orchestrator or the low-level policy.

To evaluate orchestrator reasoning independently from execution noise, we benchmark vision-language models offline on 145 scenes and 562 annotated frames across four capabilities (Table 4): plan decomposition, subgoal verification, scene state tracking, and subgoal switching. Gemini 3.7 Flash and GPT-5.6 Sol lead plan decomposition at 93.1%. Gemini 3.7 Flash also leads state tracking (70.8%), the hardest capability for most models, and serves as our default orchestrator VLM with the highest mean accuracy over the four capabilities (80.8%).

On average 32.6–43.4% of subgoals fail because an orchestrator without verification moves on before the subgoal is completed and never returns to it (Figure 6; Table 18 in the appendix). This failure extends to PSP-Agent, which incorrectly marks 16.2% of its subgoals as completed. With the introduction of verification mechanisms, this failure mode decreases to 3.4% and orchestrator-related failures comprise only 7.1% (13.8% for VoLo). This shifts the primary limitation of learned-policy systems to the low-level policy’s capabilities, with failure to manipulate subgoal target objects emerging as the main bottleneck (Low-level policy and VoLo). This is followed by a failure to reach later subgoals in the plan (Proprioceptive verifier and Privileged verifier), because the method keeps retrying earlier ones. We note that for these learned-policy methods, successful episodes on average complete within 50% of the time budget (Figure 5; Appendix G), which indicates that the allocated time budget is sufficient. Instead qualitative inspection reveals that the low-level policy ends up consuming most of the budget moving aimlessly and trying to reach the target object. Long tasks therefore fail because progress stalls: half of the failed episodes complete no subgoal after the first 28% of the budget. Reaching the target object is also the primary failure mode in tool selection systems such as CaP-X and GPT-Policy, while TiPToP fails to find a plan for 50.9% of its subgoals.

5.3 Real-World Experiments

To test whether LADDER is indicative of real-world performance, we repeat the comparison on physical WidowX and Franka robots. Due to the especially high costs and time requirements, we evaluate only a subset of the methods on task horizons ranging from 1 to 5. We observe a similar trend to simulation: the low-level policy’s success rate collapses much faster than orchestrator-based methods. Furthermore, we observe preserved rankings of the methods. This is an encouraging result for using LADDER as a reliable rank-measuring tool for long-horizon tasks.

6 Discussion and Conclusion

Our findings highlight three takeaways: (1) Task horizon is a primary failure mode. Low-level policies perform well on single-subgoal tasks, but collapse as horizon grows. High-level orchestration roughly triples overall multi-subgoal success. (2) Physical execution is the main bottleneck. VLMs decompose instructions accurately, but tasks often fail during execution even when given ground-truth subgoals. (3) The two best systems execute with tool-based primitives. GPT-Policy and PSP-Agent reach 62.0% and 40.1% success overall, against at most 29.6% for the learned-policy systems.

Limitations. High-level VLM inference is slow, limiting reactive adjustments during rapid movements. Our evaluation focuses on tabletop pick-and-place rather than pushing, sliding, or articulated objects. Visual state tracking remains challenging under heavy clutter, and physical robot experiments cover fewer scenes than simulation.

Reproducibility Statement

Section 3 describes how the benchmark is built, and Appendix D lists every task with its instruction, episodes and step budget. Section 4 and Appendix B specify every evaluated system and its settings, Appendix C gives the prompts of our systems, and Appendix A describes the low-level policies. We will release the long-horizon suites, the orchestrator code, the launch configurations and the offline VLM benchmark of Section 5.2. Rollout videos are available on the project website.

AI Use Statement

We used LLM-based coding and writing assistants. They helped implement the benchmark suites, the evaluated systems and the evaluation and analysis code; they helped analyze the results, review the methodology and check the paper’s claims against the recorded results; and they helped draft, edit and format the text. The authors reviewed and tested all code, verified every number, claim and reference, and take full responsibility for the paper.

Ethics Statement

This work involves no human subjects, personal data or crowdsourcing. We see no ethical concerns beyond the general safety considerations of deploying learned manipulation policies on physical robots.

References

1. Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL), 2022.
2. Bai et al. (2025) Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.
3. Belkhale et al. (2024) Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. RT-H: Action hierarchies using language. In Robotics: Science and Systems (RSS), 2024.
4. Black et al. (2025a) Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. \pi_{0.5}: a vision-language-action model with open-world generalization. In Conference on Robot Learning (CoRL), volume 305 of Proceedings of Machine Learning Research, pp. 17–40. PMLR, 2025a.
5. Black et al. (2025b) Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. \pi_{0}: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025b.
6. Brohan et al. (2023) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023.
7. Chen et al. (2026) Siyi Chen, Hugo Hadfield, Alex Zook, Mikaela Angelina Uy, Chan Hee Song, Erwin Coumans, Xuning Yang, Faisal Ladhak, Qing Qu, Stan Birchfield, Jonathan Tremblay, and Valts Blukis. VoLo: A physical orchestrator for open-vocabulary long-horizon manipulation. arXiv preprint arXiv:2606.07723, 2026.
8. Cheng et al. (2026) Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, and Tong Wu. In-context robot learning with VLM agents. arXiv preprint arXiv:2609.19138, 2026.
9. Fu et al. (2026) Max Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, and Linxi Jim Fan. CaP-X: A framework for benchmarking and improving coding agents for robot manipulation. In International Conference on Machine Learning (ICML), 2026.
10. Huang et al. (2022) Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning (CoRL), 2022.
11. Jangir et al. (2026) Yash Jangir, Yidi Zhang, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, and Katerina Fragkiadaki. RobotArena \infty: Scalable robot benchmarking via real-to-sim translation. In International Conference on Learning Representations (ICLR), 2026.
12. Khazatsky et al. (2024) Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O’Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems (RSS), 2024.
13. Li et al. (2024) Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. In Conference on Robot Learning (CoRL), 2024.
14. Li et al. (2025) Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. HAMSTER: Hierarchical action models for open-world robot manipulation. In International Conference on Learning Representations (ICLR), 2025.
15. Liang et al. (2023) Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
16. Liu et al. (2023) Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.
17. Mees et al. (2022) Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 2022.
18. NVIDIA (2026) NVIDIA. Cosmos 3: Omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800, 2026.
19. Shen et al. (2026) William Shen, Nishanth Kumar, Sahit Chintalapudi, Ryan Lindeborg, Jie Wang, Christopher Watson, Edward Hu, Jing Cao, Dinesh Jayaraman, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. TiPToP: A modular open-vocabulary robot manipulation system that plans. arXiv preprint arXiv:2603.09971, 2026.
20. Shi et al. (2025) Lucy Xiaoyang Shi, Brian Ichter, Michael Robert Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical vision-language-action models. In International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pp. 54919–54933. PMLR, 2025.
21. Walke et al. (2023) Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. BridgeData V2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
22. Wang et al. (2025) Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. MoGe-2: Accurate monocular geometry with metric scale and sharp details. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
23. Yang et al. (2026) Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. RoboLab: A high-fidelity simulation benchmark for analysis of task generalist policies. In Robotics: Science and Systems (RSS), 2026.

Appendix overview. Appendix A describes the low-level policies and compares \pi_{0.5} with Cosmos-3; Appendix B gives the configuration of every orchestrator and external system; Appendix C prints the prompts; Appendix D lists every task, its episodes and step budget; Appendix E gives the real-robot results per task category; Appendix F gives the results per suite and system, their uncertainty over tasks, and the construction of the offline VLM benchmark; Appendix G covers robot time, VLM calls, tokens and cost; and Appendix H analyzes the failures of every system per suite.

Appendix A Low-Level Policy Architectures and Training Details

We provide architecture, training, and deployment specifications for the base manipulation policies evaluated across our benchmarks:

• Bridge baseline (WidowX embodiment): Trained on the 43,851 trajectories of BridgeData v2 (Walke et al., 2023) that carry a language instruction. The model couples a Qwen3-VL-2B vision-language backbone (2.1B parameters) with a conditional flow-matching action expert (0.7B parameters, 2.8B in total). The action expert predicts end-effector deltas (translation and rotation) and a gripper command over an action chunk of 5 steps, of which the first 4 are executed before the next query. Training uses AdamW with a base learning rate of 5e-5, linear warmup for 250 steps, a batch size of 1024, and exponential moving average (EMA) checkpointing with decay 0.99 from step 20,000, trained for 30,000 steps. Control frequency is 5 Hz on SimplerEnv-Long and RobotArena-\infty-Long.
• DROID baseline (Franka embodiment): Evaluates the public \pi_{0.5} low-level policy (Black et al., 2025a) fine-tuned on the DROID dataset (Khazatsky et al., 2024) (\sim76,000 demonstration trajectories with a 7-DoF Franka Panda robot), in its joint-position version. The model pairs a PaliGemma 3B vision-language backbone (\sim2.9B parameters) with a flow-matching action expert (3.35B parameters in total), predicting joint positions and the gripper state over action chunks of 15 steps, of which 8 are executed. Control frequency is 15 Hz on RoboLab-Long.
• Cosmos-3 baseline (Franka embodiment): Evaluates the public Cosmos3-Nano-Policy-DROID checkpoint (NVIDIA, 2026), the Cosmos-3 world model released as a policy for the DROID setup, on RoboLab-Long. Like \pi_{0.5}, it predicts joint positions and the gripper state from the camera views, the joint state, and the text instruction.

Table 5 compares the two DROID-trained policies on the same RoboLab-Long tasks. Cosmos-3 completes fewer subgoals than \pi_{0.5} at almost every horizon, so the orchestrators in the main text use \pi_{0.5}. Orchestration helps both policies: VoLo raises task success from 11.1% to 25.6% with Cosmos-3 and from 19.6% to 29.0% with \pi_{0.5}, and the privileged verifier reaches 22.2% and 34.4%.

Appendix B Orchestrator Configurations and External Systems

Periodic trigger.

It advances every 27 steps on the WidowX suites and every 250 steps on RoboLab-Long, and stays on the last subgoal.

Proprioceptive trigger.

The trigger tracks the normalized gripper aperture ([0,1] travel). An attempt ends when the gripper opens past 0.5 after a close of at least 3 steps on the WidowX suites, and of at least 21 steps with a lift of 3 cm on RoboLab-Long. A cooldown of 8 steps (WidowX suites) or 57 steps (RoboLab-Long) prevents re-triggering while the jaws settle. On RoboLab-Long an attempt also ends after 300 steps without a release, which produces 68% of its advances, and the plan restarts from its first subgoal after the last. The WidowX suites have no such limit.

Proprioceptive verifier.

It detects attempts in the same way, without the cooldown. A verification query is sent 5 steps (WidowX suites) or 35 steps (RoboLab-Long) after a release, and after 45 or 300 steps without one. The verifier asks for an object count rather than a binary verdict, to disambiguate repeated identical objects: a subgoal is complete once the count reaches the number of times it occurs in the plan so far (one on revisit tasks). An incomplete subgoal is retried in place; on RoboLab-Long, an incomplete subgoal with interchangeable duplicates moves to the end of the queue instead.

Privileged verifier.

It reads the simulator’s success predicate for the current subgoal and advances once the predicate has held for a settle window, 3 steps on SimplerEnv-Long, 10 on RobotArena-\infty-Long and 120 on RoboLab-Long (to filter contact chatter on container rims), with the gripper open. On RoboLab-Long, a subgoal not completed within 600 steps is set aside and returned to later, except on ordered tasks; on the WidowX suites it is retried until the budget ends.

PSP-Agent.

PSP-Agent gives the VLM seven tools: pick_and_place, pick, place, push, measure, grasp_at and run_code (Appendix C). The VLM plans the task from the current image as a list of goals, each naming one object and one destination. The first call for a goal is pick_and_place on that object and destination; after a failure, the VLM chooses the next call from the goal, the tool catalog and the reports of the calls so far. Each tool returns whether it succeeded and, on failure, a short description of the cause (for example, that the gripper closed without an object). A goal is given up for the current pass after two failed calls that move the arm. After a pass over all goals, the VLM plans again from the new image and the reported outcomes; an empty plan is checked once more, and a second empty plan ends the episode. To localize an object, a tool fits an upright box to the depth points inside the bounding box the VLM returns for its name. A grasp is checked with depth on the WidowX suites (the object has left its start position) and with the jaw width on RoboLab-Long, and a release is checked with depth over the destination on SimplerEnv-Long and RoboLab-Long.

External systems.

We evaluate four external baselines and alternative execution backends:

• VoLo (Chen et al., 2026): Evaluated in its default subgoal mode, with the low-level policy as its only execution tool (the full system can also call action primitives). The VLM splits the task into an ordered list of subgoals from the first frame, checks progress every 20 steps on SimplerEnv-Long, 15 on RobotArena-\infty-Long and 150 on RoboLab-Long, and replans the remaining task when a subgoal fails, at most three times.
• CaP-X (Fu et al., 2026): We evaluate the multi-turn variant of CaP-X with high-level primitives. A VLM generates a Python program over six primitives: get_object_pose, sample_grasp_pose, goto_pose, open_gripper, close_gripper, and home_pose. get_object_pose segments the named object with SAM3 in the scene camera, lifts the mask into 3D with the calibrated depth, and fits a box, rather than reading privileged simulator state; grasps are computed from the box. Unlike the release, the grasp is taken from straight above the box center rather than sampled with Contact-GraspNet, the box extends down to the support surface, and on RoboLab-Long the arm ramps to each target within a clamped workspace. After each program, a second VLM query compares the frames before and after execution and decides whether to regenerate the program or finish.
• TiPToP (Shen et al., 2026): Performs open-loop task-and-motion planning from an initial scene perception. A VLM extracts 2D object bounding boxes and goal predicates, SAM-2 segments masks, metric depth reconstructs point clouds, M2T2 proposes grasp candidates, RANSAC estimates support planes, and cuTAMP with cuRobo solves for collision-free trajectory plans. Its planner finds a plan for most single-subgoal tasks but fails more often as subgoals are added: no plan is found for 5.2% of its subgoals at one subgoal and 74.0% at five.
• GPT-Policy (Cheng et al., 2026): A VLM agent that chooses one tool call at a time. We use its agent loop and tools (move_to, move_eef_chunk, set_gripper, locate_point, done, and give_up), with our description of the robot in its prompt. We answer locate_point, for up to 8 pixels per call, with the calibrated depth camera. We disable check_path, because the simulators do not provide the robot’s inverse kinematics.

VLM settings.

In simulation, every hosted VLM query uses Gemini 3.7 Flash (gemini-3.7-flash), except those of GPT-Policy, which uses GPT-6 Astra (gpt-6-astra); all were queried between 10 and 24 September 2026. The Proprioceptive verifier, PSP-Agent, VoLo and TiPToP decode at temperature 0.0, CaP-X at 1.0, and GPT-Policy at the provider’s default. Generation is capped at 2048 tokens for verification, for PSP-Agent’s planning and tool choice (1024 for grounding) and for GPT-Policy, at 8192 for VoLo and TiPToP, and at 20,480 for CaP-X. Reasoning effort is the provider’s default, except low for TiPToP. A verification query generates about 220 tokens, reasoning included.

Appendix C Vision-Language Prompts and Tool Catalog

We print the prompts of the two systems we built, as they ran: the verification query of the Proprioceptive verifier, and the prompt templates and tool catalog of PSP-Agent. Words in angle brackets are filled in at run time. The external systems use their own prompts: VoLo and TiPToP those of their releases, CaP-X those of its multi-turn variant, and GPT-Policy its agent prompt with our description of the robot.

Completion verification.

Sent by the Proprioceptive verifier with the current subgoal and frame on all three suites.

You are checking the CURRENT state of a robot tabletop scene from its camera views. The subtask below moves ONE object to a goal location, and the scene may hold SEVERAL interchangeable objects of that same kind. Count how many DISTINCT objects of the kind the subtask names are in the subtask’s END STATE RIGHT NOW – at the goal location the subtask asks for, regardless of how they got there. An object counts as placed when any part of it rests on the goal location, INCLUDING when partially hidden by the robot arm or other objects. An object the gripper is still holding does NOT count, however close to the goal location it is held: the release is not yet a placement and it is the release that most often fails. An object away from the goal location, or lifted well above it, is not counted. Count each physical object at most once, however many views show it. First write one short line listing what you counted, then output STRICT JSON: {“count”: <integer>}.

Verification, container suffix.

Appended to every verification query.

When a goal location is a container whose interior you cannot see into, do not look for the object inside it. Instead account for every object of that kind in the scene: if an object of that kind is no longer resting anywhere else in the scene, and the gripper is not holding it, it is in the container and counts as placed. Objects still resting elsewhere do not count.

Verification, multi-view suffix.

Appended when more than one camera is sent, which is the case on RoboLab-Long (main and wrist views).

Several camera views of the SAME scene at the SAME instant are provided, each labelled. They do not show different moments. A view may not contain the goal location at all – judge from whichever view shows it, and treat evidence from a close-up view as authoritative over a distant one where the goal location is small or occluded.

PSP-Agent, planning.

Sent with the current frame at the start of the episode and after every pass over the plan. Goals already worked on are listed with the tools’ reports in the form below.

You are planning for a <robot> working on a tabletop, seen from its camera. Task: <task>

<attempted>Break the task into the smallest goals that each move ONE object to ONE destination. Name every object by its colour and kind as it LOOKS in the image (“the yellow lemon”, “the blue bin”), never by a role or a bare noun, because those names are what will be pointed at in the image later and several objects may be of the same kind. Only include goals the task actually asks for. An object the task does not mention must be left where it is, even when it sits among the ones that move. When the task states an order (“first … then …”), keep the goals in the order the task gives them, and list a goal TWICE if the task asks for the same object to be moved twice. Otherwise order the goals so that anything placed on an open surface, where a later reach could knock it off, comes after anything dropped into a container. Leave out any goal the image shows is already met, and return an empty list of goals when the whole task is done. Give each goal as a sentence, plus the object it moves and where that object goes, named the same way. Output STRICT JSON and nothing else: {“goals”: [{“goal”: “…”, “object”: “…”, “destination”: “…”}]}

PSP-Agent, attempted goals.

Filled into the planning prompt from the second pass on.

Already worked on, with what the executor reported. The image is the evidence of what actually happened, so ask for one of these again ONLY if the picture shows it did not take – repeating a goal that took sends the arm back over it and knocks it over: <lines>

PSP-Agent, re-check.

Appended to the planning prompt, which is sent again, when a plan comes back empty; a second empty plan ends the episode.

Look again before stopping. The episode is NOT over, and stopping early throws the rest of it away: on a two-goal SimplerEnv unit 57% of the steps were spent standing still after the agent decided it was finished, while a third of the goals were still unmet. Name anything the task asks for that is not done, however small. Where the destination HIDES what goes into it – a bin, a box, a pot you cannot see the bottom of – the executor’s report is the evidence and the picture is not. A goal reported done there IS done, and asking for it again sends the hand back in to take the object out. Where the destination is OPEN – a plate, a cloth, a patch of table – the picture is the evidence and the report is not. If the object is not lying on it, that goal is NOT done, whatever was reported: a released object rolls off, and a later reach knocks it off. Return an empty list only when every part of the task is either visible in the picture or reported done into something you cannot see inside.

PSP-Agent, tool choice.

Sent with the current frame for every tool call after the first one of a goal; the catalog is listed below.

You are controlling a <robot>. The image is the scene right now. Goal: <goal> The gripper is <hand> right now.

Available tools: <catalog>

<history>Choose ONE tool call that makes progress on this goal. pick_and_place is the cheap path and handles most goals in one call. When it reports something the hand could not do, do not repeat it: the tools below it are yours – measure the thing, then take hold of it where YOU judge the jaws will close, or push it there along the surface when it is too wide to hold at all, and place it yourself. run_code drives the same API when a goal needs something none of the tools expresses. If a previous attempt failed, do not repeat it unchanged: name the OBJECT differently, since that name is what gets pointed at in the image and it may not match what is visible. The destination is what the goal asks for – describe it differently if it was not found, but never put the object somewhere else. Output STRICT JSON and nothing else: {“tool”: “<name>”, “args”: {…}}

PSP-Agent, grounding.

Sent with the current frame whenever a tool locates an object or a destination by name.

Point out <name> in this image. Draw it TIGHTLY around <name> alone. Anything lying beside or touching it must be OUTSIDE the box, even where that means cutting the box in close. Then add one box for EVERY other object lying near it, each with its own label, so <name> can be told apart from them. A name can say WHICH one it means by where it lies or what it is beside. Box the one it points at, and box the others of the same kind as well, so they can be told apart. Never reply with an empty list because the wording is not an exact name. Reply with ONLY a JSON list of boxes, each [ymin, xmin, ymax, xmax] normalised to 0-1000: [{“box_2d”: [y0, x0, y1, x1], “label”: “…”}] Return an empty list [] only if nothing of that kind is in the picture at all.

PSP-Agent, tool catalog.

The catalog of the tool-choice prompt: each tool’s signature and description.

pick_and_place(object_name: str, destination_name: str): Move one object to one destination. This is the whole of most goals; prefer it over calling pick and place separately.

pick(object_name: str): Grasp an object and lift it clear of the surface. Name it the way it looks, with its colour (“the yellow lemon”), so the right one is grounded when several are similar. The jaws close on the widest part of it they can span, since that is the part a slightly wide approach still catches; use `grasp_at` to take hold somewhere else.

place(destination_name: str): Put the object already in the gripper onto or into a destination, named the same way. This needs a pick to have succeeded first; to move something still on the table, use pick_and_place. Releases above the destination’s own top surface and on a clear part of it, so it works for a plate, for a cloth, and for the inside of a bin.

push(object_name: str, destination_name: str): Slide something along the surface to a destination with the closed hand, without lifting it. This is how to move what the jaws cannot take hold of: a pot or a bowl too wide to span, something too heavy, something flat with nothing to grip. It cannot put one thing INTO another or stack it on top, and it drags whatever stands in the way, so lift when you can and push when you cannot.

measure(object_name: str, surface: bool = False): What the cameras and depth say about one thing, so you can decide where to take hold of it yourself. Reports where it is, how big it is, how wide and how thick it is along its length section by section, and what the hand can actually close on. The jaws hold a prism, not a scoop, so the widest part of a thing is not always the part to take it by. Pass surface=True for something to put things ON, like a plate or a cloth, which is too flat to ground otherwise.

grasp_at(object_name: str, x_m: float, y_m: float, z_m: float, yaw_deg: Optional[float] = None): Take hold of an object at a point YOU choose, then lift it. Use it when `pick` reports it cannot take something – a pot too wide to span has a handle, a bowl has a rim, a spoon has a stem – and `measure` to find where those are. `yaw_deg` turns the hand about the vertical; leave it out to keep the object’s own angle. Everything else is as `pick`: the jaws open on the way, close on the point, and the object is checked and lifted.

run_code(code: str): Run Python against this same API when a goal needs something the other tools do not cover. `pick`, `place`, `pick_and_place`, `measure`, `grasp_at`, `go_home`, `locate(name, surface=False)` (returns an object with `center_m`, `extent_m`, `top_m`, `yaw_rad` and `cross_section_m()`, the width and height along its length; pass surface=True for something to put things ON, like a plate or a cloth, which is too flat to ground otherwise), `move_to(position_m, yaw_rad)`, `open_gripper`, `close_gripper` and `np` are already defined. Every skill returns a report with `ok`, True when it finished, and `text`. Print what you learn.

Appendix D Benchmark Suites and Complete Task Catalogs

We provide complete task specifications for all 145 evaluation tasks across the three benchmark suites: 54 tasks in SimplerEnv-Long (Table 7), 47 tasks in RobotArena-\infty-Long (Table 8), and 44 tasks in RoboLab-Long (Table 9); one RobotArena-\infty-Long task, scene45_all_3, repeats another (Table 8). Table 6 summarizes the eight task categories with representative instructions.

Episodes and step budgets.

Each episode of a task starts from its own initial layout: SimplerEnv-Long draws 24 seeded layouts per task, RobotArena-\infty-Long shifts the objects of one authored layout by 8 fixed offsets of up to 3 cm, and RoboLab-Long jitters every object’s position and yaw at reset. The step budget is 100 steps per subgoal on the WidowX suites. On RoboLab-Long it is 900, 1350, 2700, 3600, 4050, 4500 and 4950 steps for 1, 2, 3 to 6, 7, 8, 9 and 10 subgoals. In relay and revisit tasks a later leg of an object counts only once its earlier legs are done, and an ordered task fails when its objects are placed out of order. The primitives and plans of the tool selection systems execute as ordinary control steps, so their motions count against the same budget. Each task counts one run of each system, with all its episodes. A run that stopped before recording its results, or in which a check of the RoboLab-Long videos for unrendered scenes flagged an episode, was repeated; where a task was nevertheless measured twice, only one of the runs is counted. This happened for 20 tasks of a system, 11 of them VoLo’s on RoboLab-Long. Summed over each system’s repeated tasks, the counted runs never succeed more often than the others, and only for VoLo does the choice move a suite’s success by more than half a point: 29.0% on RoboLab-Long with its later runs counted, and 31.0% with the earlier ones. In total, the paper reports 21,296 simulated episodes: 2,024 for each of the ten systems of Table 3 and 1,056 for the three Cosmos-3 configurations of Table 5.

Appendix E Real-Robot Results per Task Category

Table 10 breaks the real-robot results of Section 5.3 down by task category.

Appendix F Results per Suite and System

The main text combines some results over the three suites or draws only some of the systems. This appendix gives every one of them per suite and for all ten systems, over the same episodes. Table 11 and Table 13 are the numbers behind Figure 4, Table 12 splits the task categories of Table 3 by suite, and Table 14 splits the offline VLM benchmark of Table 4 by the suite each scene comes from, without intervals, since each suite holds about a third of the samples. Failure causes per suite are in Appendix H and efficiency per suite in Appendix G.

Uncertainty over tasks.

The tasks are the benchmark, so the intervals of the tables cover the randomness of the episodes on these tasks. For a fixed set of tasks the Wilson interval is conservative, because the variance within tasks is smaller than the variance of all episodes taken as one sample. A claim that one system is better than another should also hold across tasks. Since all systems run the same tasks, we test each such difference paired: tasks are resampled within each suite (10,000 bootstrap draws) and both systems are scored on the same resampled tasks, so the difficulty of a task counts for both. The overall differences the main text draws, in points with their 95% intervals, are GPT-Policy over PSP-Agent 21.8 [18.1, 25.8], PSP-Agent over the privileged verifier 10.5 [7.4, 13.7], the privileged verifier over the Proprioceptive verifier 4.5 [2.6, 6.6] and over VoLo 4.2 [2.3, 6.3], the Proprioceptive verifier over the Proprioceptive trigger 3.4 [1.4, 5.3] and over the low-level policy 14.0 [9.8, 18.7]. VoLo and the Proprioceptive verifier (0.3 [-1.8, 2.6]), CaP-X and the Proprioceptive verifier (0.0 [-3.8, 3.6]), and the low-level policy and TiPToP (1.6 [-1.3, 4.7]) are not separated.

Offline VLM benchmark.

Plan decomposition is scored on 145 scenes, the reset frame and instruction of every task; a plan is an exact match when its object and destination pairs are exactly those of the task’s ground-truth subgoals, with their multiplicity, in any order. Requiring their order on the 47 ordered, relay and revisit tasks changes no score in Table 4. Verification, state tracking and next-subgoal selection use 562 frames recorded in rollouts of 143 of these tasks, mostly of the privileged verifier, CaP-X and TiPToP, labelled by the simulator’s per-subgoal predicates. The TP and TN rates are the shares of finished and unfinished subgoals judged correctly; state tracking asks about every subgoal of a frame and counts the frame as correct only when all answers are; a next subgoal is valid when it names a subgoal that is still unfinished, and on those 47 tasks the first unfinished one. Table 14 splits these scores by suite.