Bridging Frontier Reasoning and Robot Execution: From Autonomous Demonstration Generation to Dense Language Supervision

Abstract

Recent advances in frontier models enable robot manipulation from only a few demonstrations, but high inference latency limits their use for real-time robot control. To bridge this gap, we study two complementary approaches that connect frontier reasoning with low-latency local execution. First, we use a frontier model to autonomously generate demonstrations that supplement human demonstrations for training a fast local policy. We augment its in-context examples with corrective demonstration segments that show how to recover from physical errors, improving generation reliability. Generation time and cost decrease as successful examples accumulate in context, suggesting a path toward more efficient data collection. At deployment, a harness combines frontier-generated instructions with a fast local policy, enabling efficient execution while preserving the frontier model’s ability to guide and correct actions.

With low-latency execution delegated to the local policy, the bottleneck shifts to its capacity to reliably follow the frontier model’s diverse instructions. Our second bridge introduces dense language supervision across three nested granularities—primitive, atomic, and composite—with multi-aspect descriptions at each level. Across long-horizon tasks in RoboCasa 365 and BEHAVIOR-1K, where instructions change as execution progresses, the combined supervision achieves the highest performance under both oracle and frontier-model instructors, demonstrating more reliable instruction following through the policy’s language interface. Finally, we evaluate both bridges together on a crossword task that combines semantic planning and manipulation within a fixed time budget. These results support autonomous demonstration generation and dense language supervision as complementary components for connecting frontier reasoning to low-latency local execution.

1 Introduction

Robot learning has explored several approaches to translating frontier-level intelligence into physical robot actions. These approaches differ in how frontier models guide execution: they can issue instructions to a trained local policy (Shi et al., 2025), generate executable programs over control primitives (Liang et al., 2022), provide spatial guidance such as trajectories (Li et al., 2025), or produce robot commands directly (Menon et al., 2026). Recent advances in frontier models enable robot manipulation from only a few demonstrations through in-context learning, without task-specific fine-tuning. However, frontier-model inference and API latency limit the responsiveness of direct robot control, whereas practical manipulation requires high-frequency sensory feedback and timely action adjustments for reliable physical interaction (Berman et al., 2026). This motivates our central question: how can we retain the benefits of frontier-model reasoning while delegating motor execution to a fast local policy?

The ability to manipulate objects from a few examples makes frontier models a potential source of additional robot demonstrations, complementing teleoperation-based data collection. Our first bridge explores this capability by generating manipulation trajectories and incorporating them into local-policy training. Successful demonstrations alone do not reliably support autonomous generation, particularly during precise contact operations. We augment the in-context examples with corrective demonstration segments: short demonstrations of adjustments at moments when execution encounters physical errors. As successful trajectories accumulate, we retain useful segments for reuse in subsequent episodes and examine how generation time and API cost evolve. At deployment, a harness combines frontier-generated plans with the policy’s low-latency execution, allowing the frontier model to provide high-level direction while the policy generates motor actions within each execution segment.

Delegating motor execution reduces the need for frontier inference to generate actions, but makes the policy’s instruction-following capabilities a central constraint. We propose dense language supervision as the next bridge, enabling the policy to interpret and execute diverse, fine-grained instructions generated by the frontier model. We annotate the same demonstrations at three nested granularities—primitive, atomic, and composite—and pair these annotations with multi-aspect descriptions to broaden instruction diversity.

We study two complementary bridges between frontier reasoning and robot execution: generating demonstrations for local-policy training and expanding the policy’s language interface through dense supervision. Across four manipulation tasks, mixed human and generated demonstrations achieve the same aggregate policy success as an equally sized human-only corpus, supporting generated trajectories as a supplementary training source. Hierarchical and multi-aspect supervision together achieve the strongest results across both simulation benchmarks and instructor conditions. Finally, the crossword task demonstrates their integration, with dense supervision improving performance under dynamic spatial instructions. Together, these results support autonomous demonstration generation and dense language supervision as complementary bridges between frontier reasoning and low-latency local execution.

2 Related Work

Foundation Models for Robot Control. Foundation models guide robot execution through several interfaces, summarized in Figure 8 in Appendix B. Direct-control evaluations of frontier models demonstrate manipulation capabilities alongside limitations in execution speed and precision (Menon et al., 2026; Berman et al., 2026). Hierarchical and dual-system architectures couple vision-language processing with low-level control (NVIDIA et al., 2025; Shi et al., 2025; Chen et al., 2026a; Cheng et al., 2025), while other approaches provide spatial guidance (Li et al., 2025; Shridhar et al., 2025; Lee et al., 2026) or synthesized goal images (Black et al., 2024). Hi Robot fine-tunes a high-level VLM using labeled demonstrations and synthetic language interactions, and a low-level VLA using robot demonstrations (Shi et al., 2025). Harness frameworks such as Thea support feedback-driven task execution (Wang et al., 2026). Our study retains a frontier model without task-specific fine-tuning and examines how to train a local policy that executes its instructions.

Autonomous Data Generation and Policy Learning. MimicGen adapts human demonstrations to new scene configurations and embodiments (Mandlekar et al., 2023). Manipulate-Anything uses VLM-generated actions and visual verification to collect physical demonstrations for policy learning (Duan et al., 2024). Show-Harness collects human and frontier-generated demonstrations through predefined semantic actions mapped to robot commands by an embodiment-specific interpreter, and uses the resulting trajectories for policy training (Chen et al., 2026b). Program-synthesis approaches generate executable programs over perception and control primitives (Liang et al., 2022; Fu et al., 2026), with ASPIRE accumulating reusable skills (Lu et al., 2026) and ENPIRE extending agent-driven improvement to neural policies, training infrastructure, and algorithms (Xiao et al., 2026). Our first bridge studies autonomous demonstration generation through direct, low-level robot control by frontier models. Using successful demonstrations and corrective demonstration segments as in-context references, a frontier model executes tasks with visual feedback and produces new training trajectories. We examine generation reliability, track how generation time and cost evolve as successful examples accumulate, and evaluate the resulting demonstrations as a supplementary source for local-policy training.

Language Supervision and Instruction Following. RT-H learns language-motion prediction and action generation within a single VLM, enabling corrections through language (Belkhale et al., 2024). Hu et al. (2026) show that a controller’s responsiveness to varied instructions affects hierarchical execution. DeMiAn provides multi-aspect segment annotations and a learned instructor, including evaluation on composite tasks with changing atomic instructions (Kim et al., 2026). Building on multi-aspect supervision, our second bridge jointly annotates shared trajectories at three nested granularities—primitive, atomic, and composite—and adds alternative descriptions and rephrasings. We examine how temporal granularity and linguistic diversity affect instruction following under oracle and frontier-model instructors, with additional robustness evaluations across planners and instruction phrasings.

3 Bridge 1: Distilling Frontier Reasoning into Local Control

Our first bridge uses frontier-generated demonstrations to supplement human demonstrations for training a fast local policy. We first examine autonomous generation reliability, then generation efficiency, and finally deployment through a planner–policy harness.

3.1 Autonomous Rollouts via In-Context Learning

We first evaluate the capacity of frontier models to autonomously control a robot given only a minimal set of human demonstrations. We benchmark two models, Fable 5 and GPT-6 Astra, across four tasks spanning a spectrum of physical complexity: basic pick-and-place (claw machine), spatial alignment (stacking cups), deformable manipulation (folding a cloth), and high-precision contact (closing a lid). Each trial is provided five human demonstrations with zero subsequent human intervention, and the environment is fully reset between attempts. Trials exceeding a one-hour timeout are recorded as failures. This limit applies to the capability evaluation; the subsequent data-collection runs use a separate protocol.

Figure 1 shows that autonomous generation reliability varies across tasks. GPT-6 Astra succeeds in 8/10 Stack Cups trials, while Fable 5 succeeds in 6/10 Claw Machine trials. Precise contact remains challenging: on Close a Lid, Fable 5 succeeds in 0/10 trials and Astra in 2/10. The low success rates on Fold a Cloth and Close a Lid indicate that successful demonstrations alone do not reliably support generation across all four tasks.

Improving Precision through Corrective Demonstration Segments. Minor spatial misalignments during precise manipulation can lead to missed grasps or failed insertions. We augment the in-context examples with corrective demonstration segments: short demonstrations focused on moments when execution requires a corrective adjustment. These examples show how to use visual feedback to correct physical errors, providing guidance beyond successful task executions. Figure 2 illustrates corrective alignment during cube pickup; Appendix C reports per-task segment counts and additional examples (Figure 9).

Corrective demonstration segments improve observed success across the evaluated tasks (Figure 3). On Close a Lid, Astra improves from 2/10 to 5/10 successful trials, while Fable 5 improves from 0/10 to 2/10. Astra also succeeds in all ten Stack Cups trials. These results suggest that examples of visual corrections help recover attempts that would otherwise fail, although precise contact remains challenging.

3.2 Data Synthesis via Autonomous Generation

We use GPT-6 Astra to collect successful manipulation trajectories for local-policy training. During synthetic trajectory collection, frontier-controlled episodes can take tens of minutes, with some exceeding 100 minutes. These data-collection runs are distinct from the one-hour capability trials in Section 3.1. We therefore examine whether accumulating successful examples in context reduces generation time and API cost.

Generation Efficiency with Accumulated Examples. We track time and API cost across ten successful episodes per task. Unlike the independent trials with five demonstrations in Section 3.1, these generation sessions retain context across episodes. Useful segments from successful trajectories are added to a separate pool while the original five human demonstrations are retained. The model reuses and adapts these segments when the current situation matches, updating its strategy after each episode. Section C.2 provides the additional instruction used for this experiment. Figure 4 reports time and API cost per successful episode; time and API cost from failed attempts are excluded. These measurements describe successful-rollout efficiency rather than the total cost of collecting the dataset.

Generation time and API cost drop sharply after the first one or two successful episodes (Figure 4). On Close a Lid, both decrease by more than 50% between the first and second episodes; on Stack Cups, API cost falls from approximately $18 to $7 between the second and third episodes. Subsequent episodes generally maintain lower time and cost despite episode-to-episode variation. These observations suggest that retaining context across episodes can improve generation efficiency.

3.3 Harnessing Frontier Models for Local Policies

We incorporate autonomously generated trajectories into local-policy training alongside human demonstrations. For each task, the training corpus contains 40 successful episodes: 20 human-teleoperated demonstrations and 20 trajectories generated by GPT-6 Astra. Appendix C.6 compares this mixed corpus with 40 human demonstrations per task; both achieve 41/80 policy-alone successes across the four tasks (Figure 10). The autonomous trajectories include corrective movements, exposing the local policy to error-recovery behavior during training.

At deployment, the frontier model receives frames from three camera views, the task description, and the goal condition, and provides high-level instructions to the local policy. We fine-tune a pretrained OpenPI \pi_{0.5} policy (Black et al., 2025) on the mixed demonstration corpus described above to obtain the local controller, which predicts ten-action chunks of absolute joint targets. Appendix C documents the training corpus, policy architecture, and deployment interface, including timing limitations.

To evaluate the integrated harness during deployment, we compare four control configurations: GPT-6 Astra alone, Astra with corrective demonstration segments (CS), the local policy alone (Policy-alone), and the harness with Astra. We evaluate 20 trials per configuration and task, reporting success over all trials and mean elapsed time over successful trials only. Time includes frontier reasoning, planning, tool calls, API latency, and corrective retries, but excludes hardware-fix intervals (Section C.5).

Figure 5 compares execution time and success rate across the four control configurations. Across the four tasks, the harness achieves 65% success (52/80 trials), compared with 51.25% (41/80) for the local policy alone. Relative to GPT-6 Astra alone, mean elapsed time over successful trials is approximately 40–60% lower. For example, on Fold a Cloth, time decreases from 31.4 to 16.3 minutes while maintaining the same 55% success rate. These results support separating frontier-model guidance from local motor execution to balance task success and execution time, although precise contact remains challenging. The effectiveness of this separation depends on the local policy’s ability to follow the frontier model’s instructions, motivating the language-supervision study in Section 4.

4 Bridge 2: Expanding the Policy Language Channel

Within the harness, the frontier model provides high-level instructions while the local policy executes motor actions. The effectiveness of this division depends on whether the policy responds to the intended instruction granularity and wording. We address these requirements through supervision along two axes: 1) Temporal resolution: Task-level or subtask-level descriptions provide little supervision for individual movements. 2) Linguistic diversity: A single templated description per segment does not capture the varied instructions produced by a frontier planner. We introduce dense language supervision by annotating the same demonstration episodes at three nested granularities—primitive, atomic, and composite—and pairing each level with multiple instruction formats and rephrasings.

4.1 A Corpus at Three Hierarchical Granularities

We annotate the same demonstration episodes at three nested granularities: (1) a composite annotation names the full multi-stage task; (2) an atomic annotation names one subtask within it; and (3) a primitive annotation names a single-skill segment within a subtask, such as a reach, a grasp, or a carry. The three layers describe nested temporal spans within the same episodes. Table 1(a) illustrates the hierarchy for MakeFruitBowl, retaining the complete task and subtask sequence and expanding the apple-transfer subtask into primitive instructions. The other subtasks have their own primitive annotations, omitted from the example for space. The annotations describe the same underlying trajectories at different temporal resolutions. This provides a vocabulary for individual movements as well as complete subtasks.

RoboCasa 365 provides both atomic and composite annotations; we additionally construct primitive-level annotations. We derive primitive segments from proprioceptive signals, using gripper opening/closing and end-effector motion to identify physical events, with short boundary buffers and fallback windows. This yields eight primitive kinds over 27,113 segments, drawn from 4,100 episodes across 91 tasks on RoboCasa 365. For BEHAVIOR-1K, we use the primitive annotations and corresponding segments provided by the benchmark.

4.2 Dense Multi-aspect Language Supervision

We adapt the multi-aspect supervision approach from DeMiAn Kim et al. (2026) to provide four instruction formats for each segment: arm/gripper kinematics, spatial destinations, a terse command, and natural phrasing. These formats emphasize how the robot moves, where objects move, minimal wording, and conversational wording, respectively. Table 1(b) illustrates all four for the same apple-transfer subtask. These formats provide complementary descriptions of the same action segment, varying in both wording and detail. Appendix D reports corpus construction and physical consistency checks, and Section D.2 describes VLM screening and human review of object-identity annotations.

4.3 Experimental Setup

We evaluate our language-supervision strategies on two simulated benchmarks, RoboCasa 365 (Nasiriany et al., 2026; RoboCasa, 2026) and BEHAVIOR-1K (Li et al., 2023). We choose simulation-based benchmarks that provide long-horizon tasks requiring robust high-level planning and enable controlled comparisons of language inputs. This allows us to vary instruction granularity and phrasing while keeping initial conditions and evaluation criteria consistent. 1) RoboCasa 365: We evaluate ten composite kitchen tasks, scored via strict success rate over 240 paired episodes per seed (24 episodes per task). 2) BEHAVIOR-1K: We evaluate eight repeated rollouts of one instance for each of seven tasks (56 episodes per condition and seed). Due to the extreme difficulty of this benchmark, where complete task successes approach zero, we report mean subgoal progress (mean q), the benchmark’s predicate-based partial-success score, rather than binary success rates. Here, subgoals refer to benchmark goal predicates, not planner-generated instruction steps. To ensure fair comparison, all policies are initialized from the same pretrained checkpoint and trained for a matched budget of 120,000 steps, with training and evaluation seeds 42 and 142 for the main language-supervision comparison. Table 9 in Appendix E defines the prompt content for each condition. Benchmark tasks and training configuration details are provided in Appendix E.

Instructor conditions. We evaluate each language supervision recipe under two instructor conditions. Under the oracle instructor, the subgoal plan is authored and instruction switching follows simulator-state milestones, providing a privileged reference for policy performance. Under Fable 5, the frontier model serves as both planner and instructor: it determines the subgoal sequence and provides the corresponding language instructions for the local policy. The full sequence is generated once at the start of the episode from the composite goal text and reset-frame camera views, without access to simulator state. For each evaluation episode, the sequence is generated once and reused across all policy variants, keeping instruction content fixed across supervision conditions. We separate progress monitoring from high-level planning (Yi et al., 2026). A local open-weights VLM, Qwen3.6-27B, monitors execution from camera observations and decides when the current subgoal is complete. This makes routine progress checks more efficient by avoiding frequent calls to the slower frontier model. All policy variants share this progress monitor. When completion is confirmed, the harness passes the next instruction from the original plan to the local policy. Appendix F details this protocol and its connection to Probe 2.

4.4 Results

Table 2 shows that subtask instructions improve performance over fixed task-level instructions. Adding task-goal context improves both benchmarks. Hierarchical supervision further improves RoboCasa success over Task + Subtask, but not BEHAVIOR-1K progress; multi-aspect supervision alone improves both benchmarks. The combined recipe performs best under both instructors, reaching oracle scores of 0.517 and 0.174 on RoboCasa and BEHAVIOR-1K, respectively. Under Fable 5, it improves the corresponding scores from 0.381 to 0.485 and from 0.084 to 0.143 over subtask-level supervision.

4.4.1 Probe 1: Multi-Granularity Instruction Following

We first probe whether hierarchical supervision improves the local policy’s ability to follow fine-grained movement commands. We evaluate single-seed subtask-level and hierarchical-only checkpoints from the main evaluation on six primitive movement commands across 12 RoboCasa scenes, without additional training. We measure instruction compliance by checking whether the final displacement along the commanded axis meets a predefined threshold. Appendix G provides the protocol and results (Table 12). Hierarchical supervision improves compliance across all six commands, including “move the arm down” (0.111 to 0.778) and “move the arm back” (0.028 to 0.444). These results suggest that hierarchical supervision broadens the policy’s language channel to better support primitive movement commands alongside subtask-level instructions. We note that this probe measures directional movement, not successful manipulation or online correction during complete tasks.

4.4.2 Probe 2: Robustness to Instruction Diversity

A robust local policy must understand the semantic intent of an instruction regardless of the specific syntax or vocabulary used by the instructor. We test two forms of instruction variation: changing the source planner among Fable 5, Opus 5, and Gemini-3.1-Pro, and rephrasing the resulting instructions with Fable 5. In the rephrased condition, we prompt Fable 5 to preserve the original task goal while changing how the instructions are expressed. For example, “Open up the cabinet” becomes “Grasp the cabinet door handles and swing the doors open.” This condition tests goal-preserving variation in wording and descriptive detail; Fable 5 serves as the rephrasing model for the instructions from all three source planners. We first evaluate the quality of the complete high-level plans generated once at the start of each episode by these three frontier models, using the composite goal text and reset-frame camera views. We compare their predicted plans across 30 scenes against the ground-truth oracle plans. Within each planner and phrasing condition, the generated plan for an evaluation episode is reused across all policy variants; observation-based completion monitoring and instruction switching follow the shared protocol in Appendix F.

Table 11 in Appendix F reports decomposition step-count match and semantic similarity, showing that all three models generate plans highly similar to the oracle reference. These text-based metrics provide a check on plan quality, but do not establish that the plans are functionally equivalent. Execution differences across planners can therefore reflect both wording and differences in plan content or ordering. Table 3 reports single-seed end-to-end execution success across the three planners and their rephrased instruction conditions.

For the subtask-level baseline, changing the planner from Fable 5 to Gemini-3.1-Pro reduces success from 0.392 to 0.325, while using the rephrased versions of Fable 5’s instructions reduces success from 0.392 to 0.350. Multi-aspect supervision reduces sensitivity to rephrasing across all three planners, with absolute changes in success of at most 0.009, compared with up to 0.042 for the subtask-level baseline. The combined recipe achieves the highest success in all six conditions (0.454–0.492); even its lowest success exceeds the baseline’s highest (0.392). Together, these results suggest that diverse language supervision makes the policy less sensitive to how instructions are expressed, while combining it with hierarchical supervision sustains higher task success across the tested instructors and instruction phrasings.

5 Bringing Both Bridges Together

We evaluate both bridges together on the crossword task (Figure 6(a)), which requires forming valid words and placing alphabet cubes on a shared board. Each placement changes the board and remaining letter pool. In our qualitative observations, both frontier models proposed valid words, while physical execution was the main bottleneck.

Setup. The workspace contains an initially empty 6\times 6 grid and a randomized pool of 15 alphabet cubes. We use GPT-6 Astra to autonomously generate a demonstration corpus for the crossword task and annotate the trajectories with dense, multi-aspect language supervision. To assess the contribution of dense language supervision, we compare two harness configurations: one with a local policy trained on subtask-level instructions and the other with hierarchical and multi-aspect supervision. Both use the same demonstrations, policy architecture, training budget, and frontier controller. During harness deployment, the frontier model reads camera views at times of its choosing and issues language instructions with execution durations; the local policy generates motor actions. Subsequent observations support corrective instructions after failed pickups and revision of planned placements. Unlike the fixed-plan evaluation in Section 4, instructions can be revised during execution. Appendix H provides the policy training configuration and a separate 20-trial pick-and-place evaluation (Table 14). We evaluate performance using two reward components: +2 points for each cube successfully placed at its instructed grid coordinate and +2 points for each valid word formed.

Results. We report the mean total score over five games, each using a different alphabet set and a one-hour execution budget, for four control configurations: Fable 5 alone, GPT-6 Astra alone, the GPT-6 Astra harness, and the GPT-6 Astra harness with dense language supervision. Appendix H provides per-game results and an additional comparison prohibiting cell IDs in instructions (Table 13). We omit policy-alone execution because the task requires high-level word planning. As shown in Figure 6(b), both harness configurations achieve higher scores than either frontier model alone within the fixed time budget. The dense-supervision variant further improves the mean score. When board cell IDs are prohibited, the frontier model must issue instructions such as “put the cube to the left of the B cube” instead of “put the cube on 1A.” Under this restriction, dense supervision increases the mean score from 2.8 to 5.6 across five games (Appendix H), although both policies perform worse than with cell IDs. Together with the multi-granularity and instruction-diversity probes in Section 4.4, these results suggest that broadening the policy’s language channel supports execution across fine-grained movement commands, rephrased subgoals, and spatial descriptions of target locations.

Execution timing. The harness delegates motor execution to a low-latency local policy, but the overall system still pauses for frontier reasoning between execution segments. Figure 7 shows the first 20 minutes of the direct-control and frontier-harness runs. Over the full one-hour runs, cumulative action execution increases from 2 min 40 s under direct control to 9 min 8.55 s with the harness. This corresponds to approximately 3.4 times as much time spent executing robot actions within the same budget, illustrating the benefit of delegating motor execution to the local policy. Appendix H provides timing details.

6 Conclusion

We study two complementary bridges between frontier reasoning and robot execution: autonomous demonstration generation to supplement local-policy training, and dense language supervision to improve instruction following. We introduce corrective demonstration segments to improve generation reliability. Generation time and cost decrease across the observed successful episodes as examples accumulate. Combining hierarchical and multi-aspect language supervision improves instruction following on long-horizon simulation tasks. Evaluation on the crossword task brings both bridges together, with the harness achieving higher scores than direct frontier control within a fixed time budget. Delegating motor execution to a local policy mitigates the impact of frontier-inference latency, allowing more of the available time to be spent on physical execution. Extending this approach to broader physical tasks and reducing dependence on human demonstrations remain directions for future work. Together, these results suggest that effective delegation benefits from both capable local execution and a language interface that supports the frontier model’s guidance.

References

1. Belkhale et al. (2024) S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh RT-H: action hierarchies using language. In Proceedings of Robotics: Science and Systems, External Links: Document, Link
2. Berman et al. (2026) S. Berman, M. Ilie, J. Deng, and D. Freeman Claude plays robotics. Note: https://www.anthropic.com/research/claude-plays-roboticsAnthropic Frontier Red Team research post, July 9, 2026
3. Black et al. (2025) K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky \pi_{0.5}: A vision-language-action model with open-world generalization. In Proceedings of The 9th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 305, pp. 17–40. External Links: Link
4. Black et al. (2024) K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine Zero-shot robotic manipulation with pretrained image-editing diffusion models. In International Conference on Learning Representations (ICLR), Note: arXiv:2310.10639
5. Chen et al. (2026a) S. Chen, H. Hadfield, A. Zook, M. A. Uy, C. H. Song, E. Coumans, X. Yang, F. Ladhak, Q. Qu, S. Birchfield, J. Tremblay, and V. Blukis VoLo: a physical orchestrator for open-vocabulary long-horizon manipulation. arXiv preprint arXiv:2606.07723.
6. Chen et al. (2026b) Y. Chen, Z. Bai, Z. Cao, W. Zeng, K. Q. Lin, Y. Lin, G. Liang, K. Y. Ma, Q. Huang, and M. Z. Shou Show-Harness: just a VLM agent can play robots. arXiv preprint arXiv:2609.10522. External Links: Link
7. Cheng et al. (2025) A. Cheng, Y. Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang NaVILA: legged robot vision-language-action model for navigation. In Proceedings of Robotics: Science and Systems, External Links: Document, Link
8. Duan et al. (2024) J. Duan, W. Yuan, W. Pumacay, Y. R. Wang, K. Ehsani, D. Fox, and R. Krishna Manipulate-Anything: automating real-world robots using vision-language models. arXiv preprint arXiv:2406.18915. External Links: Link
9. Fu et al. (2026) L. Fu, J. Yu, K. El-Refai, E. Kou, H. Xue, H. Huang, W. Xiao, G. Wang, D. Niu, F. Li, G. Shi, J. Wu, S. Sastry, Y. Zhu, K. Goldberg, and L. Fan CaP-X: a framework for benchmarking and improving coding agents for robot manipulation. In International Conference on Machine Learning (ICML), Note: arXiv:2603.22435
10. Hu et al. (2026) J. Hu, M. Shridhar, C. Lu, D. Shah, H. L. Chiang, J. Tan, and A. Xie What matters in orchestrating robot policies: a systematic study of hierarchical VLA agents. arXiv preprint arXiv:2606.10267.
11. Kim et al. (2026) B. Kim, R. Wang, D. Acuna, J. Jung, A. Trevithick, B. Cui, Y. Choi, and P. Ammanabrolu How to instruct your robot: dense language annotations power robot policy learning. arXiv preprint arXiv:2605.17077. External Links: Document, Link
12. Lee et al. (2026) S. Lee, Y. Jung, J. Lee, J. Shin, A. H. Shahidzadeh, Y. Lee, H. J. Kim, J. Huang, and F. Huang \mu_{0}: A scalable 3D interaction-trace world model. arXiv preprint arXiv:2606.13769.
13. Li et al. (2023) C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, K. Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and L. Fei-Fei BEHAVIOR-1K: a benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In Proceedings of The 6th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 205, pp. 80–93. External Links: Link
14. Li et al. (2025) Y. Li, Y. Deng, J. Zhang, J. Jang, M. Memmel, R. Yu, C. R. Garrett, F. Ramos, D. Fox, A. Li, A. Gupta, and A. Goyal HAMSTER: hierarchical action models for open-world robot manipulation. In International Conference on Learning Representations (ICLR), External Links: Link
15. Liang et al. (2022) J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng Code as policies: language model programs for embodied control. arXiv preprint arXiv:2209.07753.
16. Lu et al. (2026) R. Lu, Y. Wu, E. Kou, L. Fu, W. Xiao, A. Mandlekar, Y. Xu, G. Shi, K. Goldberg, A. Chen, M. Chowdhury, Y. Zhu, L. Fan, and G. Wang ASPIRE: agentic skills discovery for robotics. arXiv preprint arXiv:2607.00272.
17. Mandlekar et al. (2023) A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox MimicGen: a data generation system for scalable robot learning using human demonstrations. In Proceedings of The 7th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 229, pp. 1820–1864. External Links: Link
18. Menon et al. (2026) A. Menon, S. Machcha, S. Zou, T. K. Chan, and J. Chooi GPT-6 Astra on robotic manipulation. Note: https://openai.robocurve.org/gpt-6-astra/Robocurve, September 4, 2026. Accessed 2026-09-18
19. Nasiriany et al. (2026) S. Nasiriany, S. Nasiriany, A. Maddukuri, and Y. Zhu RoboCasa365: a large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations (ICLR),
20. NVIDIA et al. (2025) NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu GR00T N1: an open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734.
21. RoboCasa (2026) RoboCasa RoboCasa365. Note: https://robocasa.ai/Benchmark v1.0 released 2026-02-18; per-frame subtask annotations released 2026-07-07
22. Shi et al. (2025) L. X. Shi, B. Ichter, M. R. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, A. Li-Bell, D. Driess, L. Groom, S. Levine, and C. Finn Hi robot: open-ended instruction following with hierarchical vision-language-action models. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, pp. 54919–54933. External Links: Link
23. Shridhar et al. (2025) M. Shridhar, Y. L. Lo, and S. James Generative image as action models. In Proceedings of The 8th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 270, pp. 2429–2455. External Links: Link
24. Wang et al. (2026) Q. Wang, T. Wang, C. Li, S. Ban, Y. Chen, Y. Ge, J. Qin, C. Li, and W. Zhu Towards the harness of embodied agents. arXiv preprint arXiv:2608.11246.
25. Xiao et al. (2026) W. Xiao, J. Xie, T. Zhang, H. Lin, L. Fu, H. Xue, J. Lu, Y. Yang, C. Dai, Z. Wang, J. Wu, G. Wang, S. S. Sastry, K. Goldberg, L. Fan, Y. Zhu, and G. Shi ENPIRE: agentic robot policy self-improvement in the real world. arXiv preprint arXiv:2606.19980.
26. Yi et al. (2026) P. Yi, Y. Ma, W. Xu, Y. Hao, S. Gan, W. Li, and S. Zhong Critic in the loop: a tri-system VLA framework for robust long-horizon manipulation. arXiv preprint arXiv:2603.05185.

Appendix A Limitations and Future Work

Experimental scope. The main language-supervision comparison uses two training/evaluation seed pairs, providing limited replication without separating training and evaluation variability. Evaluation tasks are represented in training: RoboCasa 365 tests held-out layouts, styles, and sampled assets, whereas BEHAVIOR-1K uses one held-out initial configuration per task within existing scenes. Physical evaluations cover four manipulation tasks and five crossword games; broader task coverage and additional independent runs are needed to assess generality.

Autonomous demonstration collection. Policies trained on 20 human and 20 generated demonstrations per task attain the same aggregate success as those trained on 40 human demonstrations (Appendix C.6). This result does not establish statistical equivalence or isolate the incremental benefit over 20 human demonstrations alone, which we do not evaluate. We are collecting additional autonomous demonstrations to evaluate policies trained entirely on generated trajectories.

Timing and deployment. Frontier reasoning introduces pauses between execution segments due to API latency, so low-latency local control does not make the complete system real-time. Generation-efficiency measurements exclude failed attempts and describe observational trends across successful episodes, rather than total collection cost or a causal effect of accumulating examples. Physical deployment times are conditional on success and exclude hardware-fix intervals. More responsive coordination between frontier reasoning and local execution is an important direction for future work.

Appendix B Interfaces for Frontier-Guided Robot Control

Figure 8 distinguishes five routes to robot control: a policy without a frontier model (a), frontier-generated conditioning for a trained policy (b), executable programs over a fixed API (c), geometric guides (d), and direct joint commands (e). The distinction is what the frontier model emits and how that output reaches the actuators. Geometric guides may be followed by a policy or converted to joint commands through inverse kinematics, so type (d) is defined by the guide rather than its receiver.

Appendix C Autonomous Demonstration Generation

C.1 Autonomous Rollout Prompt

The following prompt is used for autonomous rollouts in Section 3.1. Angle-bracketed fields denote the task instruction, task-specific success condition, and data paths.

Complete “<TASK_INSTRUCTION>” with an SO-101 robot, using five successful demonstrations from <DEMONSTRATION_DATASET> as in-context references.

1. Learn and act. Review all five demonstrations and observe the current scene. Infer a suitable strategy, execute with visual feedback, and adapt to the robot’s actual response. Preserve calibrated hardware limits and stop when safe execution cannot be established.
2. Verify success. Confirm that <SUCCESS_CONDITION> is satisfied.
3. Record trajectories. Save every attempt in <OUTPUT_DIRECTORY>, including measured states, executed actions, camera observations, timestamps, source references and outcome. Build the synthetic corpus from verified successes; label failures separately.
4. Report results. Report the outcome, segment reuse, episode-only estimated API cost, episode elapsed time, and robot command-execution time separately. Exclude preparation and postprocessing; state measurement limitations. Create a review video for successful episodes, omitting camera-free intervals.

C.2 Accumulated-Experience Prompt Extension

For the generation-efficiency experiment in Section 3.2, the following instruction is added to the rollout prompt:

Improve through experience. Add useful successful segments to <SEGMENT_POOL>, separate from the original demonstrations. Reuse and adapt them when the current situation matches. Update the strategy after each episode to improve reliability, time and cost.

The original five demonstrations and the growing segment pool are maintained separately. The model can reuse relevant successful segments in later episodes. The reported generation-efficiency measurements exclude preparation, postprocessing, and failed attempts; they describe successful episodes rather than the total cost of corpus construction. Episode elapsed time and robot command-execution time are recorded separately.

C.3 Corrective Demonstration Segments

Corrective demonstration segments are short examples of adjustments made when execution encounters physical errors. Table 4 reports the number of segments used for each task. These counts refer to corrective segments, rather than complete successful trajectories or evaluation trials.

Figure 9 illustrates alignment toward target cubes from different initial gripper offsets in four crossword pickup demonstrations.

C.4 Physical Policy and Deployment Interface

We fine-tune a pretrained OpenPI \pi_{0.5} vision-language-action policy on the mixed corpus of human and autonomously generated demonstrations to obtain the physical controller. The policy uses a PaliGemma backbone (gemma_2b) and a flow-matching action expert (gemma_300m). It controls an SO-101 arm using overhead and wrist RGB images, six measured joint/gripper state values, and a natural-language instruction. The deployment interface uses 224\times 224 RGB images and duplicates the overhead frame into the policy’s two external-camera input slots. This policy input format is distinct from the frontier model’s camera access.

Policy inference runs remotely over WebSocket. Each prediction provides ten absolute joint-target actions; the robot uses six action channels, corresponding to five arm joints and the gripper. The intended action rate is 30 Hz, so ten actions represent approximately 333 ms of motion. The documented scheduling strategy issues actions every 33.3 ms and replaces the unexecuted remainder of a chunk when a new prediction arrives.

The 30 Hz rate is a target rather than a consistently achieved measurement. Early diagnostics reported 3.38 end-to-end cycles per second, while later bench measurements reported control rates of 18.5–19.9 Hz. These are different timing measurements from diagnostic runs and do not establish a uniform achieved rate for the task evaluations. Camera acquisition, preprocessing, communication, and inference all contribute to the realized timing.

The instruction is supplied in the prompt field of every inference request. Updating that field can condition the next action chunk without retraining or reloading the policy. The interface alone does not establish a tested mid-execution interruption protocol or switching latency; the documented offline comparison of policy predictions under different instructions does not demonstrate successful physical task switching.

C.5 Elapsed-Time Accounting and Limitations

For the physical deployment comparison, elapsed time is averaged over successful trials only; failed and timed-out trials contribute to success rates but not to the time average. Elapsed time includes frontier reasoning and planning, camera observation and verification, tool calls, API and communication latency, physical action execution, and task-level corrective retries. Hardware-fix intervals are excluded. Consequently, these means measure time conditional on success rather than expected time per attempt or total time to obtain a success.

Even with the same prompt, the frontier agent can choose different sequences of observation, verification, and preparatory actions across trials. Service and communication issues introduce additional variability. The prompt does not enforce an identical operational sequence, and excluding hardware-fix intervals limits what the reported times establish about total deployment effort. We do not attribute this variability to unobserved internal model mechanisms. The horizontal coordinates in Figure 5 report elapsed task time averaged over successful trials. The crossword activity timeline in Figure 11 instead illustrates one run per configuration and is not an average across games.

C.6 Human-Only and Mixed Demonstration Corpora

We compare policy success after training on 40 human demonstrations per task with training on a mixed corpus of 20 human and 20 autonomously generated demonstrations per task (Figure 10). Each condition is evaluated over 20 trials per task. The mixed and human-only corpora yield success rates of 90% versus 90% on Operate a Claw Machine, 60% versus 55% on Stack Cups, 40% versus 50% on Fold a Cloth, and 15% versus 10% on Close a Lid. Both conditions achieve 41 successes across 80 trials (51.25%).

At the same total demonstration count, the mixed corpus uses half as many human demonstrations and attains the same aggregate success in this evaluation, with task-specific differences. This result supports the feasibility of using generated trajectories in place of part of the human training corpus; it does not establish statistical equivalence or consistent improvement across tasks. We do not evaluate a 20-human-only condition, so this comparison does not isolate the incremental benefit of adding generated trajectories to those 20 human demonstrations.

Appendix D Corpus Construction and Annotation Audit

D.1 Corpus Construction and Physical Validation

Annotation sources and segmentation. Both RoboCasa 365 and BEHAVIOR-1K provide atomic and composite annotations. BEHAVIOR-1K also provides primitive annotations and their corresponding segments, which we use directly. The primitive segmentation procedure described below applies to RoboCasa 365. For RoboCasa, we additionally construct a primitive corpus containing 27,113 segments from 4,100 episodes across 91 tasks. This is the corpus used for the reported policy training. Segmentation uses recorded action and state streams rather than video or language: gripper opening and closing identify grasp and release, smoothed vertical and horizontal end-effector motion distinguishes lift, lower, and transport, and direction reversals identify shaking. The median segment length is 18 frames, with an interquartile range of 13–35 frames. Table 5 gives the composition. Segmentation parameters. We smooth the end-effector trajectory with a five-frame moving average. Motion below 1.5 mm per frame is treated as stationary; otherwise a vertical-to-total motion ratio of at least 0.5 assigns an upward or downward label, with remaining motion assigned to transport. Carry segments (Lift, Lower, and Transport) require at least eight frames, net displacement of at least 2 cm, and a dominant-axis displacement ratio of at least 0.70. A finger gap below 7 cm identifies candidate held spans. Carry boundaries are trimmed to exclude grasp/release intervals and to limit within-segment finger-gap variation to 3 mm.

Grasp and release windows are identified from gripper opening and closing, using a 1 mm per-frame change threshold and a 4 mm net-change check. When that check fails, the implementation falls back to a three-frame window on either side of the event. Windows include two lead-in frames, with trailing buffers of four frames for grasp and eight for release. Reach and retract are bounded by end-effector motion and capped at 80 and 60 frames, respectively; retract begins after release. Shaking merges at least two carry segments when the dominant-axis motion reverses at least three times, the path length is at least 10 cm, net displacement is at most 15 cm, and the net-to-path ratio is at most 0.45.

Object identity. Object labels are derived from recorded simulator state rather than from nouns in task descriptions. Table 6 summarizes how these object labels are obtained. For phases with little object displacement, identity can be inherited from transport within the same grasp-to-release span. A consistency check against a directional inheritance rule found one disagreement in 10,314 cases across 222 multi-pick episodes. Unresolved cases retain the decomposition’s object name and are explicitly marked; these are concentrated in fixture manipulation, where the relevant part lacks a free joint. Asset mesh categories provide 112 distinct object names.

Language generation. We use Opus 5 for annotation generation. The language supervision uses the four formats described in Section 4: arm/gripper kinematics, spatial destinations, terse commands, and natural phrasing, with additional rephrasings. Captions are conditioned on the segment’s primitive kind and object identity and retained only after validation.

Physical consistency. Table 7 reports checks against the motion signals associated with each primitive. Among 230 sampled transport segments, 96.5% contain a single moving object. In eight inspected cases with multiple moving objects, the annotation agrees with the object rigidly held by the gripper. The visual audit is described in Section D.2.

Limitations. Object identity inherited across phases is inferred rather than directly observed within each segment, and 2.9% of identities remain unresolved by simulator-based checks. Shaking is rare in the corpus and its translation-reversal detector can miss tilting motions. Visual inspection covers only a sample. These measurements characterize annotation consistency; they do not isolate the causal effect of annotation quality on task success.

D.2 Annotation Audit

VLM screening. We use Opus 5 to inspect manipulated-object identity from RoboCasa’s default agent-left, agent-right, and wrist-camera views. The model receives segment imagery without the object label, task name, or annotation slots, and its predicted identity is compared with the simulator-derived annotation. The sample contains 207 segments across 61 tasks, selected round-robin across tasks within each primitive kind: 26 segments for each of seven kinds and all 25 available shake segments. This screening identifies cases for inspection rather than estimating annotation accuracy: the VLM can itself misidentify objects. Each segment is represented by 12 frames.

Human review. The authors manually reviewed 50 randomly sampled segments and agreed with the annotated object identity in 48 cases (96%). This result describes agreement within the reviewed sample, rather than a comprehensive assessment of annotation accuracy across the corpus. Physical consistency checks for primitive labels are reported in Section D.1.

Appendix E Training and Evaluation Details

Training/evaluation split. Evaluation tasks are included in the training task sets for both benchmarks. For RoboCasa 365, training uses pretrain demonstrations, whereas evaluation uses the target environment split with held-out kitchen layouts, visual styles, and sampled object assets. For BEHAVIOR-1K, the seven evaluation tasks belong to the ten-task training subset; evaluation uses a held-out public-test initial configuration, instance 301, for each task. This tests new object and robot initial configurations within the existing scene setup. The seven-task results use repeated rollouts of this single configuration per task and do not establish generalization across unseen houses or layouts.

RoboCasa evaluation. The evaluation set contains ten composite tasks selected to test multi-step execution and instruction switching across increasing task depth (Table 8). It includes cabinet and drawer articulation, repeated object transfers, and rinsing, with three tasks containing two subgoals, three containing three subgoals, and four containing four subgoals. Training demonstrations and milestone checkers are available for all ten tasks, and each task has a complete instruction sequence including required closing actions. Subgoal counts refer to distinct instruction phases in the execution plan, rather than the number of evaluation predicates. For the main language-supervision comparison, each condition evaluates 24 episodes per task and seed, giving 240 episodes per seed and 480 across seeds 42 and 142, with a 6,000-step limit. Performance is measured by strict task success, including required terminal conditions such as closing a cabinet; intermediate milestone completion alone is insufficient.

Instructor conditions. Oracle evaluation uses authored subgoal instructions and simulator-state milestones for switching. Frontier-instructor evaluation separates a fixed initial plan from observation-based completion monitoring, as described in Appendix F. The planner identity and the policy checkpoint must be matched to the recorded run when interpreting a table cell.

BEHAVIOR-1K evaluation. For the main language-supervision comparison, we evaluate each condition for eight episodes on each of seven tasks per seed, totaling 56 episodes per seed and 112 across seeds 42 and 142. The seven tasks are Chop an onion, Hiding Easter eggs, Preparing lunch box, Setting mousetraps, Chopping wood, Sorting vegetables, and Putting away Halloween decorations. All rollouts use task instance 301, with each rollout executed in a separate process. The execution limit is 12,000 simulator steps per episode. We compute the mean terminal partial-credit score q within each task, then report the unweighted mean of these seven task means. These are repeated rollouts of one instance per task, not eight distinct evaluation instances; they therefore do not establish generalization across a range of held-out task instances. The score measures benchmark goal-predicate satisfaction, not progress through planner-generated steps.

Training configuration. The policies in the simulation benchmark comparisons are trained for 120,000 steps. Crossword policy training uses a separate configuration described in Appendix H. The main language-supervision comparison uses two independently trained policies per condition, with training seeds 42 and 142; each is evaluated using the corresponding evaluation seed. The RoboCasa OpenPI configurations use a \pi_{0.5} policy with an action horizon of ten, a maximum text length of 384 tokens, and batch size 256. Optimization uses AdamW with (\beta_{1},\beta_{2})=(0.9,0.95), \epsilon=10^{-8}, weight decay 10^{-10}, and gradient-norm clipping at 1.0. The learning rate warms up over 1,000 steps to 5\times 10^{-5} and remains at that value; exponential moving average decay is 0.999. Because training and evaluation seeds change together, differences between the two runs reflect both sources of variation rather than isolating training-seed variability.

The BEHAVIOR-1K policy training configuration uses \pi_{0.5} with an action horizon of 32, a maximum text length of 256 tokens, and batch size 64. Its learning-rate schedule warms up over 1,000 steps to 2.5\times 10^{-5} and decays to 2.5\times 10^{-6}.

Language supervision recipes. Table 9 specifies the prompt construction for each condition. Here, goal is the episode-level task instruction and subgoal is the current subtask instruction. In the hierarchical conditions, hierarchical instruction denotes the instruction corresponding to the sampled segment’s hierarchy level, including primitive instructions such as “lift the arm”. The prompt pairs this instruction with the original episode-level goal using the fixed phrase “Current step:”. Multi-aspect supervision samples from the original segment instruction and descriptions in the four formats introduced in Section 4: arm/gripper kinematics, spatial destinations, terse commands, and natural phrasing.

The hierarchical-only and joint conditions use the same goal context and fixed phrase; the joint condition additionally varies the segment instruction through multi-aspect rephrasings.

E.1 Language Supervision Results by Seed

Table 10 reports the two seed results underlying the averages in Table 2. Each displayed mean is calculated before rounding. For RoboCasa, the mean is (s_{42}+s_{142})/480, where s_{42} and s_{142} are success counts over 240 trials per seed.

Appendix F Frontier Planning and Observation-Based Instruction Switching

This appendix describes the frontier-instructor protocol used in Section 4 and the high-level plan evaluation in Probe 2. The protocol separates initial plan generation, execution monitoring, and instruction switching.

F.1 Plan Generation and Reuse Across Policy Variants

At the start of each episode (t=0), a frontier vision-language model receives the composite task description and the reset-frame camera views. It generates the complete ordered subgoal sequence once. For a given evaluation episode and planner/phrasing condition, this sequence is reused across all policy variants. Instruction content is therefore held fixed across policy variants while the language-supervision condition varies, following the same discipline as the oracle condition.

The frontier model supplies the plan before execution. Subsequent observations determine when the harness advances through that plan; they do not trigger regeneration of the subgoal sequence.

F.2 Local Progress Monitoring

A separate local open-weights vision-language model, Qwen3.6-27B, checks progress every 40 control steps using the active subgoal instruction and three-camera RGB observations from the subgoal’s start and the current state. Its decisions use visual observations rather than simulator-state milestones. Monitor queries are synchronous.

The oracle condition instead uses an authored subgoal sequence and simulator-state milestones to determine completion. The frontier-instructor condition uses neither simulator state for initial planning nor simulator state for progress monitoring.

F.3 Instruction Switching for the Local Policy

The harness passes one instruction at a time from the pre-written subgoal list to the local policy. The policy executes the current instruction between monitor queries. Two consecutive positive judgments trigger advancement to the next instruction after the configured switching buffer. Uncompleted subgoals retain the current instruction without automatic skipping or replanning. Episodes terminate on benchmark success, environment termination, or exhaustion of the 6,000-step budget for RoboCasa 365 or the 12,000-step budget for BEHAVIOR-1K.

Thus, the instruction supplied to the local policy changes in response to observed completion, while the content and order of the subgoal list remain fixed. Plans are shared across policy variants, but switching times depend on each policy’s observed execution progress.

F.4 Connection to Probe 2: Robustness to Instruction Diversity

The high-level plan evaluation in Section 4, Probe 2, evaluates the initial plan-generation stage described above. Opus 5, Gemini-3.1-Pro, and Fable 5 each generate a full subgoal sequence from the composite goal text and reset-frame camera views. Their plans are compared with oracle reference plans across 30 scenes using decomposition step-count match and semantic similarity, with Fable 5 serving as the semantic-similarity judge. Fable 5 compares individual generated steps with oracle-reference steps and directly assigns a semantic-similarity score in [0,1] to each comparison. Higher scores indicate closer semantic agreement.

The subsequent execution evaluation varies the source planner and whether its instructions are rephrased. Table 3 reports RoboCasa 365 results from one training/evaluation seed (42). Its original-instruction Fable 5 results match the seed-42 entries in Table 10, whereas Table 2 averages seeds 42 and 142. Fable 5 generates the rephrased instructions for all three source planners. It is prompted to preserve the original task goal while changing the wording. For example, “Open up the cabinet” may be rewritten as “Grasp the cabinet door handles and swing the doors open.” Such rewrites can make action details explicit, so this condition tests variation in wording and descriptive detail rather than lexical substitution alone. Within each planner and phrasing condition, the generated sequence is reused across policy variants. All policy variants use the same local progress-monitoring and instruction-switching protocol. The plan-quality evaluation assesses the initial frontier-generated text; the execution evaluation measures how the policies perform with that text under observation-based instruction switching.

Appendix G Primitive Instruction Compliance

Table 12 reports the full primitive-compliance probe, including the instruction wording and movement threshold for each action.

We compare one checkpoint per condition from a single training seed of the subtask-level and hierarchical-only policies used in the main evaluation, without additional training for this probe.

Each command is evaluated three times in each of the 12 scenes, with a 24-second execution window per repetition. We average the three repetitions within each scene and then average equally across the 12 scenes. Each repetition is scored as compliant when final displacement along the commanded axis exceeds its threshold. Thus, each reported rate aggregates 36 binary trials per command and policy.

Appendix H Crossword Evaluation Results

Table 13 reports individual game scores and completed-word counts underlying the means in Section 5. Each game uses a different alphabet set and a one-hour execution budget, with scoring as defined in the main text. The harness baseline uses subtask-level supervision, while the dense-supervision harness uses the hierarchical + multi-aspect recipe. Within each instruction condition, both use the same demonstrations, policy architecture, training budget, and GPT-6 Astra frontier controller; the comparison varies language supervision only. Word validity is judged by the authors based on familiar English usage. This is human adjudication rather than an automated dictionary-based metric. Repositioning an already credited cube or repeating an already credited word earns no additional points.

Policy training configuration. Both crossword policies are initialized from pretrained \pi_{0.5} (Black et al., 2025) and trained with an action horizon of 10, batch size 32, and a budget of 8,000 steps. The peak learning rate is 2.5\times 10^{-5}. These settings are shared by the subtask-level and hierarchical + multi-aspect supervision conditions.

Instructions without cell IDs. In an additional comparison, we explicitly instruct the frontier model not to use board cell IDs such as “1A”, “3B”, or “6F” in its directives to the local policy. It must instead describe destinations spatially, for example, “put the cube on the left of the B cube” rather than “put the cube on 1A”. Table 13 reports five games under this restriction. Mean scores fall from 14.4 to 2.8 without dense supervision and from 15.6 to 5.6 with dense supervision. Under the restriction, dense supervision yields higher scores in four games and ties in one. These results support improved performance with spatially described targets, while both policies remain substantially less effective when cell IDs are unavailable in instructions. Qualitatively, placement was more difficult because spatial descriptions could leave the intended target ambiguous; placement and recovery success were not separately quantified.

Pick-and-place evaluation. Table 14 reports accuracy over 20 trials per configuration, excluding retries. This measures first-attempt execution separately from the one-hour game scores above. The local policy can be evaluated on individual pick-and-place trials even though it cannot independently perform the full crossword task.

H.1 Observation and Execution Timing

The frontier model autonomously decides when to read the available camera views and how long to execute each action segment. For example, it can select 2–3 s segments for transport and 0.12 s segments for fine adjustments. These durations describe physical execution rather than frontier-model inference latency.

Figure 11 compares one-hour activity timelines for GPT-6 Astra under direct control and frontier harness execution. In the illustrated direct-control run, GPT-6 Astra repeatedly reasons about successive raw actions, with 2 min 40 s spent on physical execution. In the illustrated harness run, action generation is delegated to the local policy, and physical execution occupies 9 min 8.55 s. This example is consistent with the harness allocating more time to robot motion while retaining observation-based guidance and correction. These durations correspond to 4.4% and 15.2% of the one-hour budget, respectively, or 3.43 times as much physical execution with the harness. This ratio measures time spent executing actions, not movement speed; the comparison uses one illustrated run per configuration and does not estimate an average timing advantage across games.