The Human--VLA Communication Gap: What Humans Leave Implicit and What Policies Cannot Recover

Abstract

A robot may know how to perform a task yet fail when a human asks for it naturally. We study this human–VLA communication gap by separating an information gap, where task semantics, grounding, or event structure remain unresolved, from a residual execution gap, where even a fully specified task depends on how it is organized and invoked. To do this, we characterize how people naturally express demonstrated objectives and evaluate controlled variants of the same physical tasks, with detailed analysis on \pi_{0.5} and consistent trends on \pi_{0}-FAST and Cosmos3-Edge. Our study reveals that humans increasingly compress instructions as tasks grow, relying more on contextual grounding and implicit execution structure. Current policies are misaligned with this behavior: they tolerate grounding dependence relatively well, but struggle when task semantics or event structure must be reconstructed. Even after the information gap is resolved, execution remains sensitive to command granularity and invocation state. These findings identify concrete targets for human-centered policy learning and interfaces that better bridge natural human objectives to robot behavior. Project website: https://commgap.github.io/.

I Introduction

Imagine asking a household robot, “Can you clear the table so I can work here?” The robot has already learned how to pick up cups, move plates, and place objects into containers, yet it freezes. Replace the request with “Pick up the white mug and place it in the bin,” and the same policy succeeds. What failed was not the physical skill but the interface to it: the human expressed an objective naturally, while the policy expected a narrow specification of what to do, on which object, and toward what target state.

This gap matters more as vision–language–action (VLA) models become general-purpose. Recent VLAs acquire diverse manipulation behaviors from large robot datasets and are evaluated by the breadth of tasks, objects, and environments they handle, so existing work has focused on the robot side of generality: how many skills are acquired, how they compose, and how they transfer. Yet broad physical competence alone does not make a policy accessible to ordinary users, who should not need to know the command templates used during training. Humans communicate physical objectives by saying an action, describing the outcome, or making an indirect request, so a useful generalist robot must bridge not only a physical generalization gap but a communication gap between how people express intent and how the policy expects it to be specified.

We study this human–VLA communication gap through two mismatches. A natural specification leaves some of the executable task structure unstated, which the policy must recover: an information gap. And even a complete specification may fail if its resolved events are organized or invoked outside the policy’s learned execution interface: an execution gap. Quantifying them gives a human-centered measure of generalist capability, how reliably a non-expert expressing the task naturally can reach it, and a diagnostic account that a success rate alone cannot give: which executable structure each form of human specification leaves unresolved, and what each costs the policy.

Studying this gap is difficult for three reasons. First, natural human task specification is open-ended: the same objective can be communicated through direct commands, desired outcomes, or partial references, and no fixed list of paraphrases captures it. Rather than enumerating that space, we begin from how people actually instruct another human. We collect natural instructions from participants shown task demonstrations and, informed by these and by prior work on situated instruction, develop a taxonomy that factors any specification by what task content it states, how it refers to objects, and how it composes multiple manipulations. The taxonomy does two things. It lets us generate human-like instructions for any demonstrated task, varying what is made explicit while holding the physical objective fixed. And it lets us measure what people naturally state, compress, or leave implicit as tasks grow more complex.

Second, resolving a human specification requires a policy-facing reference for what a fully explicit command looks like. Although VLAs inherit large vision–language backbones, their robot-training supervision is dominated by concise, action-centric instructions with regular structure in actions, object references, and destination descriptions [1]. We therefore mine recurring forms from robot-training datasets and use them to render each grounded event as an explicit action–object–destination command. These policy-facing realizations provide a common reference against which human specifications can be compared, and allow compound objectives to be expressed as sequences of atomic events when we study execution after the information gap has been removed.

Third, a success rate only says that a policy failed, but does not locate the gap. We introduce an information-gap analysis that characterizes, for each specification, which executable variables are explicitly provided, which require further grounding, and whether compound manipulations are decomposed into individual events. We compare each specification against a fully grounded command that explicitly identifies every action, target object, and target placement, and execute the same physical tasks under each form, measuring task success and partial progress. This allows us to relate the type and amount of unresolved structure to the VLA’s ability to recover the intended behavior. Once this information is fully resolved, we then probe the residual execution gap by holding the grounded event sequence fixed while varying whether events are issued together or sequentially, and whether each event begins from the preceding or home robot state. This separates task recovery from execution once the task is fully specified.

Together, our study separates what a VLA can physically do from what a human can reliably ask it to do, and finds them mismatched in a specific way. As tasks grow, people increasingly rely on contextual grounding and compress multiple manipulations into higher-level expressions, leaving more executable structure implicit, whereas the evaluated policies tolerate grounding dependence comparatively well but pay much more for genuinely missing task semantics and implicit execution structure, especially on complex tasks. We observe the same qualitative trends on \pi_{0.5} [2] and \pi_{0}-FAST [3], with similar behavior in Cosmos3-Edge [4], a world-action model. Nor does resolving the information gap guarantee reliable execution: even fully specified atomic events are more reliable when delivered one at a time and invoked from a familiar state.

II Related Work

II-A Human Objective Specification and Situated Instruction

Natural task-oriented communication extends beyond direct imperative commands: speakers express intended acts indirectly [5, 6], communicate purpose or constraints [7, 8], and rely on shared perceptual context to identify referents [9], and human–robot dialogue corpora similarly combine action semantics, spatial reference, and interaction context [10, 11, 12]. We treat these as alternative ways of specifying a physical objective: our taxonomy characterizes what executable information is stated, how referents are grounded, and how manipulations are composed, so that we can measure both how people compress objectives and how policies respond to the resulting implicit structure.

II-B The Learned Language Interface of VLA Policies

Robot-training language is substantially narrower than unrestricted human communication. Datasets such as RT-1 [13], BridgeData V2 [14], TacoPlay [15], Language Table [16], and LIBERO [17] exhibit highly repetitive, dataset-specific instruction distributions [1], unlike richer embodied-language datasets such as ALFRED [18] and SCOUT [12]. Recent studies show that VLA behavior can be sensitive to meaning-preserving linguistic variation [19, 20, 21, 22], but largely study variation in already explicit instructions. We focus on a preceding problem, natural specifications that leave task semantics or execution structure implicit, and use the regularities of the policy’s fine-tuning language only to construct consistent policy-facing realizations once this information is resolved.

II-C Human-Centered Evaluation of Generalist Robot Policies

Recent benchmarks increasingly move beyond aggregate task success to isolate visual, relational, procedural, and linguistic competencies. RoboLab probes controlled manipulation capabilities and instruction specificity [23], while LIBERO-Para and related work study linguistic and semantic robustness under controlled task variations [19, 21, 20]. Our evaluation asks a complementary question: how reliably can a learned capability be reached from the ways a non-expert naturally specifies the task? For a fixed objective, we measure the information gap between a specification and its executable structure, then the residual execution gap once that structure is resolved.

III Method

We study the communication gap around a common grounded representation of the physical task: we recover its sequence of manipulation events, characterize how humans naturally specify these events, and construct fully resolved policy-facing realizations of the same task. This shared representation lets us measure what a human specification leaves unresolved, and then test whether the fully resolved task remains sensitive to command organization and invocation state.

III-A Problem Formulation

We consider a physical task \tau demonstrated by a video V_{\tau}=(I_{1},\ldots,I_{T}), where I_{t} is the RGB observation at time t. From the demonstration, we recover an ordered sequence of grounded manipulation events E_{\tau}=(e_{1},\ldots,e_{K_{\tau}}), e_{k}=(A_{k},O_{k},D_{k}), where A_{k} denotes the physical action, O_{k} the manipulated object, and D_{k} its destination or resulting target state. For example, placing a banana and an apple into a bowl yields two events. E_{\tau} is the common semantic reference throughout: human specifications may selectively state, abstract, or compose information from these events, whereas robot-oriented commands express the same events using forms observed in robot-training data, so all variants preserve the same physical objective and differ only in how it is communicated. For each task, let \mathcal{H}_{\tau}=\{u_{\tau}^{1},\ldots,u_{\tau}^{N}\} denote its human-like specifications, and R_{\tau} the fully grounded robot-oriented realization of the same event sequence.

Given a fixed VLA policy \pi, the information gap is characterized by what structure in E_{\tau} is not directly available from a human specification and must be recovered for execution; once resolved, the residual execution gap asks whether the same executable task remains reliable as its events are organized differently and invoked from different robot states. We recover E_{\tau} from demonstration videos using object-centric tracking and persistent changes in state and relations, with a VLM providing factual descriptions of the extracted transitions for language generation (Appendix -B).

III-B Characterizing Human Objective Specification

Human Instruction Elicitation. Natural objective specification is open-ended, so rather than defining this space a priori, we first study how people instruct another human from demonstrated behavior. Participants observe task demonstrations of varying complexity and write the instruction they would give another human to achieve the demonstrated objective. We prescribe no command format, so participants decide what to state explicitly, how to identify relevant entities, and how to express compound objectives. These instructions provide both evidence for constructing our taxonomy and a natural distribution against which its forms can be analyzed.

Deriving the Specification Taxonomy. We analyze the collected instructions together with distinctions from prior work on task-oriented and situated language [5, 9, 7, 8]. Three recurring factors capture the major ways specifications differ (Table I): task content, whether the speaker states an action, a desired physical outcome, or an intent or preference from which the physical objective must be inferred; reference form, whether relevant entities are named directly, identified through scene-dependent descriptions, or omitted altogether; and composition, whether multiple manipulations are enumerated individually, grouped under a shared predicate, or compressed into a collective description. The taxonomy serves two roles: descriptively, to code the collected human instructions and measure how the information people provide changes with task complexity; and experimentally, to generate controlled specification variants for VLA evaluation.

Taxonomy-Conditioned Specification Generation. Given the grounded event sequence E_{\tau}, we generate human-like specifications by selecting a configuration of the three factors: a content operator (action, resulting physical state, or higher-level intent), a reference operator (direct naming, or description through attributes, spatial relations, or semantic groupings), and a composition operator (separate, grouped under a shared predicate, or summarized collectively). A language model then produces lexical and syntactic variants without changing the selected configuration, providing controlled specifications with known executable structure while keeping the physical objective fixed.

III-C Characterizing the VLA Language Interface

Robot-Training Language Prior. To characterize the other side of the interface, we model how fully resolved manipulation events are expressed in robot-training data: unlike the human space, which varies what task structure is made explicit, the policy-facing space keeps the grounded task fixed and varies only the linguistic forms through which the VLA was exposed to actions, objects, and destinations during training. For each evaluated policy, we mine the robot-language dataset corresponding to its fine-tuning distribution. Since robot-training language is substantially narrower and more repetitive than unrestricted human language [1], we extract from each instruction its command template with the action, manipulated-object reference, and destination reference, replacing grounded entities with typed placeholders to normalize commands with the same structure. These recurring forms are a language prior for constructing policy-facing commands compatible with the policy’s fine-tuning distribution (Appendix -D).

Policy-Facing Realization. Given a grounded event e_{k}=(A_{k},O_{k},D_{k}), we instantiate a compatible mined template with its action, object, and destination to obtain a fully explicit policy-facing command r_{k}; for a compound task, this yields an atomic command sequence R_{\tau}=(r_{1},\ldots,r_{K_{\tau}}). These realizations are a fully resolved instruction compatible with the policy’s training interface: the reference against which human specifications are compared and, for compound tasks, the atomic commands used to study execution. We keep them fixed so that performance differences reflect unresolved task information or execution conditions rather than incidental linguistic variation.

III-D Measuring the Communication Gap

We measure the information gap by comparing each specification with the grounded event sequence E_{\tau}, capturing what task information is available and how explicitly its events are decomposed. We then measure the residual execution gap by varying command organization and invocation state after the task is fully resolved.

Semantic Coverage and Grounding Dependence. For each event e_{k}=(A_{k},O_{k},D_{k}), let \mathcal{S}_{k}=\{A_{k},O_{k},D_{k}\}. For every required component s\in\mathcal{S}_{k}, we assign \sigma_{k,s}(u)\in\{0,1,2\}, where \sigma_{k,s}(u)=0 if the component is omitted from specification u, \sigma_{k,s}(u)=1 if it is specified but its grounded value must be resolved from the scene/task context, and \sigma_{k,s}(u)=2 if its grounded value is directly specified.

We define the semantic coverage ratio

\eta_{\tau}(u):=\frac{\sum_{k=1}^{K_{\tau}}\sum_{s\in\mathcal{S}_{k}}\mathbbm{1}[\sigma_{k,s}(u)\geq 1]}{3K_{\tau}},

which measures the fraction of required task components represented in the specification, either directly or through a grounding-dependent expression. We separately define the groundedness ratio

\gamma_{\tau}(u):=\frac{\sum_{k=1}^{K_{\tau}}\sum_{s\in\mathcal{S}_{k}}\mathbbm{1}[\sigma_{k,s}(u)=2]}{3K_{\tau}},

which measures the fraction whose grounded values are directly given in language. Thus, 1-\eta_{\tau}(u) captures omitted task information, while \eta_{\tau}(u)-\gamma_{\tau}(u) captures information that is present but still requires grounding.

Task Decomposition. Semantic coverage does not capture whether a compound objective is explicitly organized into its constituent manipulations. Let J_{\tau}(u) denote the number of distinct manipulation units individuated in specification u. We define the decomposition ratio \delta_{\tau}(u):=\frac{J_{\tau}(u)}{K_{\tau}}. A value of \delta_{\tau}(u)=1 indicates that every atomic event is separately expressed, while smaller values indicate that multiple events are compressed into a shared expression.

Reasoning Required to Resolve the Specification. The same amount of unresolved structure may require qualitatively different forms of recovery. We additionally categorize specifications by the reasoning needed to map them onto E_{\tau}, including visual grounding, action inference, task decomposition, and intent inference. We use this as a complementary analysis of what computation is required to close the information gap; details are provided in Appendix -H.

Information-Gap Evaluation. For each specification u, we execute the corresponding physical task with a fixed VLA policy and measure task success and partial progress. We compare it against a matched fully resolved realization r_{\tau}^{\mathrm{full}} constructed from the same event sequence E_{\tau}. This reference uses Action–Direct–Enumerated semantics: every required action, object, and destination is directly specified, and every manipulation event is explicitly individuated. All events are presented together in a single prompt, so this comparison resolves the task information. For evaluation metric M, we define the behavioral information gap as

\Delta_{M}^{\mathrm{info}}(u)=M(r_{\tau}^{\mathrm{full}})-M(u).

A larger \Delta_{M}^{\mathrm{info}}(u) indicates a larger performance cost from executable structure left unresolved in u. We analyze this cost primarily as a function of semantic coverage \eta_{\tau}(u), groundedness \gamma_{\tau}(u), decomposition \delta_{\tau}(u), and task complexity. A complementary analysis by required reasoning operation is provided in Appendix -H. Together, these analyses characterize both how much executable structure is unresolved and which kind the policy struggles to recover.

Residual Execution Gap. We next ask whether resolving the information gap is sufficient for reliable execution. All conditions use the same fully specified Action–Direct–Enumerated event sequence and the same policy-facing atomic commands R_{\tau}=(r_{1},\ldots,r_{K_{\tau}}). We vary only how these commands are executed. First, we compare presenting all atomic events together in one prompt with issuing the same events sequentially, one command at a time. This isolates sensitivity to execution organization while holding the task semantics fixed. For metric M, we measure

\Delta_{M}^{\mathrm{org}}=M(R_{\tau}^{\text{ sequential}})-M(R_{\tau}^{\text{ one prompt}}).

Second, under sequential execution, we vary the robot state from which each subsequent command is invoked. In the chained condition, each command begins from the terminal robot configuration left by the preceding event. In the reset condition, the arm is returned to its home configuration before the next command while preserving the task state produced by previous events. We measure

\Delta_{M}^{\mathrm{state}}=M(R_{\tau}^{\text{ sequential + home reset}})-M(R_{\tau}^{\text{ sequential}}).

The home reset is not proposed as a solution, but serves as a generic intervention for testing sensitivity to invocation state. Together, these comparisons separate information recovery from execution failure. Full execution and evaluation details are provided in Appendix -E.

IV Experiments

We investigate three questions about the human–VLA communication gap. Q1: How do natural human specifications change with task complexity, and what task information do people compress or leave implicit? Q2: Which forms of unresolved structure can the VLA recover, and how does their cost change as tasks become more complex? Q3: Once the task information is fully resolved, what residual execution gap remains in how the resolved events are presented and invoked?

IV-A Experimental Setup

Environment and Policies. We evaluate \pi_{0.5} [2], \pi_{0}-FAST [3], and Cosmos3-Edge [4], using checkpoints fine-tuned on DROID, on 120 RoboLab tasks with 10 rollouts for each task–instruction condition. All three policies exhibit similar qualitative trends across our information- and execution-gap analyses. For clarity, the main text focuses on \pi_{0.5}, which achieves the strongest overall task performance and therefore provides the clearest dynamic range for measuring the communication gap. Results for \pi_{0}-FAST and Cosmos3-Edge are reported in Appendix -F.

Human Instruction Study. To study how people naturally communicate physical objectives, we select 15 tasks spanning single-event manipulations and compound objectives with different structural regularities. Fifty participants view each demonstration without its canonical instruction and write the instruction they would naturally give to another human, yielding 750 specifications. We use these annotations to analyze how specification length, semantic coverage, grounding dependence, and decomposition change with task complexity.

Task Complexity and Compression Affordance. We measure physical task complexity by the number of grounded atomic events K_{\tau}. Because tasks with the same event count may differ substantially in how naturally they can be compressed, we additionally distinguish compound tasks that require event-wise enumeration, admit a compact grounded rule, or admit a higher-level task effect. These labels are assigned from the grounded objective independently of the human-written specifications. Full definitions and the RoboLab-120 distribution are provided in Appendix -A

Controlled VLA Evaluation. For all 120 tasks, we generate human-like specifications under controlled taxonomy configurations and evaluate them against the fully grounded r_{\tau}^{\text{full}}. This produces the factorial evaluation in Table II, where task content, reference form, and composition systematically vary the executable structure exposed to the policy. To study the residual execution gap, we then hold the grounded event sequence fixed and vary how its atomic events are organized and when they are issued to the policy. Full execution protocols are provided in Appendix -E.

IV-B Humans Compress as Task Complexity Increases

We first ask whether natural human communication actually creates the information gap studied in Sec. III-D. If people expressed a compound objective by simply concatenating atomic commands, instruction length and explicit event structure should grow with the number of required manipulations. Instead, we find people increasingly compress the objective, leaving more executable structure implicit.

Instructions Remain Compact as Task Complexity Grows. Fig. 2 compares natural specifications as the number of required atomic events K_{\tau} increases. Instruction length grows sublinearly in K_{\tau}, from 9.1 words for single-event tasks to 15.8 words for tasks requiring three or more events. At the same time, the executable structure made explicit declines substantially. Semantic coverage \eta_{\tau} decreases from 0.95 to 0.74, while groundedness \gamma_{\tau} falls from 0.89 to 0.27 and decomposition \delta_{\tau} from 1.00 to 0.21. Thus, people do not communicate increasingly complex objectives by proportionally increasing instruction length. They preserve much of the overall task content while increasingly relying on the listener to resolve referents and, especially, to expand a compact expression into atomic manipulations. Natural communication therefore produces a growing information gap as the physical objective becomes more compositional.

Compression Exploits Available Task Structure. This compression is not simply a consequence of having more events. Tasks with the same K_{\tau} differ in whether their events admit a compact shared description, and people exploit this structure when it is available. Weakly structured compound objectives remain comparatively action-oriented and enumerative, whereas rule-compressible objectives shift toward outcome-level and collective descriptions. When the task admits a compact semantic effect, participants more often describe the intent itself and leave the concrete physical realization implicit. The full distribution across task-content and composition forms is reported in Appendix -G. Together, these results show that human compression is structure-dependent: people preserve event-wise detail when the objective requires it, but use increasingly abstract specifications when a compact rule or intent can characterize the same physical goal.

IV-C VLAs: Selectively Sensitive to Unresolved Task Structure

Can the policy recover different unresolved structure equally well? As in the left panel of Fig. 5, the physical task is held fixed while the specification leaves different parts implicit; Table II evaluates this over the full factorial space of taxonomy-controlled specifications for \pi_{0.5} (\pi_{0}-FAST and Cosmos3-Edge show the same trends, Appendix -F).

What matters is not simply how much information is unresolved, but what must be recovered. The factorial design isolates different forms of unresolved structure while holding the remaining specification maximally explicit. Along the task-content axis, replacing an action specification with an outcome specification produces only a 4.5-point matched success gap, whereas an intent-level specification increases the gap to 15.2 points. An explicit outcome therefore often constrains the omitted action sufficiently for the policy to recover it, while an intent leaves the physical goal itself to be inferred before any executable behavior can be determined. The same asymmetry appears in reference: with action content and enumerated composition fixed, grounding-dependent references incur only a 2.8-point gap, whereas missing reference information incurs an 11.1-point gap. Thus, the policies are substantially more robust to resolution of information already constrained by language and observation than to reconstruction of task structure that is not present in the specification.

Missing event structure is as costly as missing task semantics. With action content and direct references fixed, grouping manipulations creates a 7.6-point success gap and collective specification a 10.4-point gap. Across all three policies, each undecomposed event costs 4.4–5.2 success points, comparable to a missing action, object, or destination (3.6–4.8), and far more than a component that only requires grounding (1.0–1.6). Thus, even when task semantics are available, VLAs remain sensitive to whether the objective is explicitly decomposed into executable events.

Structural reconstruction becomes harder as tasks grow. As task complexity increases, semantic coverage \eta_{\tau} and decomposition \delta_{\tau} become more strongly associated with the success gap, while groundedness \gamma_{\tau} remains weaker (Fig. 3, right). This creates a growing mismatch with human communication: as task complexity grows, human decomposition drops (Fig. 2), precisely as policies become more sensitive to missing event structure. The communication gap widens from both sides: people expose less structure as tasks grow, while VLAs become less able to recover it.

IV-D Execution Gap After Information Resolution

We next characterize what executable information the policy needs. We next ask whether providing that information completely is sufficient for reliable execution. As illustrated in the right panel of Fig. 5, we fix every compound task to the same fully resolved Action–Direct–Enumerated event sequence and vary only how those events are presented and invoked. This isolates a residual execution gap after task semantics, grounding, and event structure have already been resolved.

Sequentializing resolved events substantially improves execution. Is complete executable information sufficient for reliable execution? As in the right panel of Fig. 5, we fix every compound task to the same fully resolved Action–Direct–Enumerated event sequence and vary only how its events are presented and invoked. Across all three policies, issuing the grounded events sequentially outperforms presenting them in one upfront prompt in both success and partial progress (Fig. 4). No task information is added; the difference is execution organization: a compound prompt requires several manipulations from one invocation, whereas sequentialization reduces each invocation to one atomic event. Resolving the task semantics is thus not enough; the objective must also be exposed at a granularity the policy can reliably execute.

Invocation state creates a second residual gap. In sequential execution, later commands are invoked from robot configurations produced by preceding actions, often far from the states seen during training; resetting the arm to its home configuration before each event makes the invocation observation more in-distribution and improves success for all three policies (Fig. 4). Even with the correct atomic command, execution can thus fail because the policy is invoked from an unfamiliar state: each behavior must be accessed from an observation distribution in which it remains reliable.

IV-E Three Policies, One Gap, Different Weak Points

Across all three policies, we observe that grounding dependence is cheap, missing semantics and undecomposed events are expensive, and sequential invocation from a familiar state improves execution. However, within that shared pattern, the models differ in ways that track their architecture.

The information gap is shaped by the language backbone. Cosmos3-Edge pays the least for grounding-dependent references, consistent with a general-purpose vision–language backbone that resolves attribute and spatial reference. \pi_{0}-FAST pays the most (-1.6), and its penalties compress overall because it operates near floor on compound tasks. The decomposition penalty, however, is the largest term for every model, including \pi_{0.5}, whose explicit subtask prediction narrows but does not close it. No evaluated architecture recovers implicit event structure reliably.

The execution gap splits by mechanism. The sequentialization gain is shared, and largest for \pi_{0}-FAST (8.2% to 19.5%): it reflects the short, single-event commands common to all DROID-trained policies, not any particular action head. The invocation-state gain is not shared. Cosmos3-Edge gains 5.8 points from a home reset against 11.8 for \pi_{0.5}. A world-action model denoises its action chunk jointly with an imagined rollout from the current frame, so an unfamiliar arm configuration is partly absorbed by the video prior rather than falling outside the action distribution.

Implication. The communication gap is not one property of a single policy. Its grounding component is inherited from the vision–language backbone, its organization component from the robot-language distribution, and its invocation-state component from the action-generation paradigm. Only the structural-reconstruction component, the one humans exercise most as tasks grow, is unaddressed by any of the three, which marks it as the primary target for policy learning.

V Conclusion

We identify a human–VLA communication gap between how people naturally specify tasks and how current policies can reliably act on them, localizing where communication breaks down and what structure must be recovered as concrete targets for policy learning, evaluation, and interface design. Ultimately, generalist robots should adapt to how people communicate, rather than requiring people to adapt to the robot’s learned interface.

References

1. [1] S. L. Wanna, A. Luhtaru, J. Salfity, R. Barron, J. Moore, C. Matuszek, and M. Pryor (2026) Limited linguistic diversity in embodied AI datasets. In Proc. 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11058–11086.
2. [2] Physical Intelligence (2025) \pi_{0.5}: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054.
3. [3] K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine (2025) FAST: efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747.
4. [4] A. Agarwal, A. Ali, J. Allen, M. Antolini, A. Aubame, et al. (2026) Cosmos 3: omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800.
5. [5] J. R. Searle (1975) Indirect speech acts. In Syntax and Semantics, Volume 3: Speech Acts, P. Cole and J. L. Morgan (Eds.), pp. 59–82.
6. [6] H. H. Clark and D. H. Schunk (1980) Polite responses to polite requests. Cognition 8 (2), pp. 111–143.
7. [7] B. Di Eugenio (1992) Understanding natural language instructions: the case of purpose clauses. In Proc. 30th Annual Meeting of the Association for Computational Linguistics, pp. 120–127.
8. [8] K. Vander Linden and B. Di Eugenio (1996) A corpus study of negative imperatives in natural language instructions. In Proc. 16th International Conference on Computational Linguistics (COLING),
9. [9] H. H. Clark and D. Wilkes-Gibbs (1986) Referring as a collaborative process. Cognition 22 (1), pp. 1–39.
10. [10] E. Bastianelli, G. Castellucci, D. Croce, L. Iocchi, R. Basili, and D. Nardi (2014) HuRIC: a human robot interaction corpus. In Proc. 9th International Conference on Language Resources and Evaluation (LREC), pp. 4519–4526.
11. [11] S. Zarrieß, J. Hough, C. Kennington, R. Manuvinakurike, D. DeVault, R. Fernández, and D. Schlangen (2016) PentoRef: a corpus of spoken references in task-oriented dialogues. In Proc. 10th International Conference on Language Resources and Evaluation (LREC), pp. 125–131.
12. [12] S. M. Lukin, C. Bonial, M. Marge, T. A. Hudson, C. J. Hayes, K. Pollard, A. Baker, A. N. Foots, R. Artstein, F. Gervits, M. Abrams, C. Henry, L. Donatelli, A. Leuski, S. G. Hill, D. Traum, and C. Voss (2024) SCOUT: a situated and multi-modal human-robot dialogue corpus. In Proc. Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pp. 14445–14458.
13. [13] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich (2023) RT-1: robotics transformer for real-world control at scale. In Proc. Robotics: Science and Systems (RSS),
14. [14] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, A. Lee, K. Fang, C. Finn, and S. Levine (2023) BridgeData V2: a dataset for robot learning at scale. In Proc. 7th Conference on Robot Learning (CoRL), pp. 1723–1736.
15. [15] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard (2023) Latent plans for task-agnostic offline reinforcement learning. In Proc. 6th Conference on Robot Learning (CoRL), pp. 1838–1849.
16. [16] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence (2023) Interactive language: talking to robots in real time. IEEE Robotics and Automation Letters. Note: early access, doi: 10.1109/LRA.2023.3295255
17. [17] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems (NeurIPS),
18. [18] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox (2020) ALFRED: a benchmark for interpreting grounded instructions for everyday tasks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10740–10749.
19. [19] C. Kim, M. Kim, M. Kang, H. Kim, and D. Jung (2026) LIBERO-Para: a diagnostic benchmark and metrics for paraphrase robustness in VLA models. arXiv preprint arXiv:2603.28301.
20. [20] Y. Hou and L. Zhao (2026) LangGap: diagnosing and closing the language gap in vision-language-action models. arXiv preprint arXiv:2603.00592.
21. [21] I. Fang, J. Zhang, S. Tong, and C. Feng (2025) From intention to execution: probing the generalization boundaries of vision-language-action models. arXiv preprint arXiv:2506.09930.
22. [22] Z. Yin and Z. Zhang (2026) Grounded semantic re-binding for robust instruction generalization in vision-language-action models. arXiv preprint arXiv:2608.02497.
23. [23] X. Yang, R. Dagli, A. Zook, H. Hadfield, A. Goyal, S. Birchfield, F. Ramos, and J. Tremblay (2026) RoboLab: a high-fidelity simulation benchmark for analysis of task generalist policies. In Proc. Robotics: Science and Systems (RSS),
24. [24] T. Ren, Q. Jiang, S. Liu, Z. Zeng, W. Liu, H. Gao, H. Huang, Z. Ma, X. Jiang, Y. Chen, Y. Xiong, H. Zhang, F. Li, P. Tang, K. Yu, and L. Zhang (2024) Grounding DINO 1.5: advance the “edge” of open-set object detection. arXiv preprint arXiv:2405.10300.
25. [25] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer (2024) SAM 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714.
26. [26] L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao (2024) Depth anything V2. In Advances in Neural Information Processing Systems (NeurIPS),
27. [27] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923.

-A Task-Structure Complexity and Compression Affordance

Our analysis distinguishes the number of required physical events from the structure available for expressing those events compactly. Two tasks with the same number of manipulations may afford very different specifications: unrelated manipulations may require event-wise enumeration, whereas several placements may be summarized by a rule such as “put all the black objects in the bin.”

Atomic Event Count. For each task \tau, we use the grounded event sequence

E_{\tau}=(e_{1},\ldots,e_{K_{\tau}}),

where each e_{k}=(A_{k},O_{k},D_{k}) denotes one object-centric physical transition. K_{\tau} therefore measures physical manipulation complexity at the same granularity used throughout our information-gap analysis. It is derived from the extracted demonstration rather than from instruction length or linguistic structure. This differs from RoboLab’s internal num_subtasks statistic, which is defined by the benchmark’s success-condition representation [23]. We use the latter only as auxiliary metadata and compute K_{\tau} from E_{\tau}.

Compression Affordance. Event count alone does not determine how compactly an objective can be communicated. For tasks with K_{\tau}>1, we classify the strongest task-level structure that permits the objective to be expressed without enumerating every event. Classification is performed from E_{\tau} and the corresponding grounded scene, independently of the human-written specifications. Enumerative tasks require event-specific object, destination, or ordering information to remain explicit. Even when events share a generic action or destination, no single non-enumerative rule fully determines the required event sequence. Rule-compressible tasks admit a grounded rule P_{\tau} over observable entities, attributes, relations, or target states such that E_{\tau}=\mathcal{E}(P_{\tau},G_{\tau}), where \mathcal{E} instantiates the atomic events satisfying P_{\tau} in scene G_{\tau}. Examples include ‘all black objects in the bin,” ‘all mugs upright,” or sorting objects by color. Intent-compressible tasks admit a higher-level semantic effect or purpose F_{\tau} that compactly characterizes the objective without directly specifying the grounded object–destination assignments. Recovering the executable task therefore requires intent-to-goal inference: E_{\tau}=\mathcal{I}(F_{\tau},G_{\tau}), where \mathcal{I} maps the task-level effect and current scene to the corresponding grounded events. Examples include clearing a work surface, packing objects for storage, or discarding unwanted items. Table III summarizes the operational criteria.

RoboLab-120 Distribution. We apply this coding scheme to the 120 RoboLab tasks [23]. Table IV reports the resulting distribution. Nearly half of RoboLab tasks require multiple object-centric manipulations, but most compound objectives also exhibit reusable structure. RoboLab therefore contains many cases in which physical complexity increases while the objective remains linguistically compressible, allowing us to study whether people exploit this structure when deciding what to state explicitly and what to leave for the listener to recover.

-B Grounded Event Extraction

Our analysis assumes a common grounded event representation E_{\tau} for each demonstrated task. Here we describe the perception and verbalization pipeline used to recover this representation from demonstration videos. This component provides a shared semantic source for human-specification generation and policy-facing realization; it is not itself the focus of our study.

Object-Centric Temporal Representation. Given a demonstration V_{\tau}=(I_{1},\ldots,I_{T}), we convert the video into a sequence of object-centric scene graphs. We use Grounding DINO 1.5 Edge for open-vocabulary object detection [24] and SAM 2 for mask propagation and persistent object tracking [25]. Grounding DINO is applied to initialize object tracks and to recover objects when tracks are lost, while SAM 2 propagates object masks across intermediate frames. Each frame is represented as a graph G_{t}=(\mathcal{V}_{t},\mathcal{E}_{t}). Each node stores a persistent object identity together with its category, segmentation mask, bounding box, visual attributes, approximate position, and state. Edges encode spatial and semantic relations between entities, including left/right, front/behind, near, on, and inside. Image-plane relations are computed from object masks, while relative depth is estimated using Depth Anything V2 [26]. The resulting graph sequence maintains consistent entities and relations across the demonstration.

Event Extraction. We identify task-relevant events from persistent changes in object states and relations. A candidate change is retained only when the new state or relation remains stable across subsequent frames, reducing transient errors from tracking or segmentation. For the k-th transition, we define \Delta G_{k}=G_{t_{k}^{+}}\ominus G_{t_{k}^{-}}, where t_{k}^{-} and t_{k}^{+} denote representative observations immediately before and after the transition. Consecutive changes involving the same manipulated entity are grouped into a single atomic event. This yields an ordered event sequence E_{\tau}=(e_{1},\ldots,e_{K_{\tau}}), where each event is represented as e_{k}=(A_{k},O_{k},D_{k}). Here, A_{k} denotes the manipulation, O_{k} the manipulated object, and D_{k} its destination or resulting target state. For example, moving a white mug from the table into a blue bowl produces an event of the form (\mathrm{place},\mathrm{white\ mug},\mathrm{inside}(\mathrm{blue\ bowl})).

Factual Event Verbalization. The structured event representation is used as the semantic source of truth, but directly exposing graph serialization to a language model makes downstream generation unnecessarily dependent on implementation details. We therefore additionally convert each event into a concise factual description. For each transition, Qwen2.5-VL [27] receives the graph change \Delta G_{k} together with observations immediately before and after the event. It is prompted to identify which entity changed, what changed, and the resulting state or relation. The response is constrained to entities and relations represented in \Delta G_{k}. For example, a transition from \mathrm{ON}(\mathrm{mug},\mathrm{table}) to \mathrm{IN}(\mathrm{mug},\mathrm{bowl}) may be verbalized as “The white mug is moved from the table into the blue bowl.” These descriptions serve only as a natural-language interface to the downstream specification generators. The grounded event record E_{\tau}, rather than the generated description, determines the physical semantics of the task.

-C Human Study and Generation Validation

We conduct a human instruction study for two purposes. First, we use naturally elicited instructions to characterize how people communicate physical objectives and to validate that the proposed taxonomy captures the forms they actually use. Second, we evaluate whether taxonomy-conditioned generation preserves the intended task semantics while producing linguistically natural realizations.

Task Selection. We select 15 tasks from RoboLab spanning single-object manipulation, multi-object objectives, ordered manipulation sequences, set-level rules, sorting, reorientation, stacking, and higher-level cleanup objectives. The selected tasks are:

The set is chosen to span different numbers of atomic manipulation events and different opportunities for expressing a compound objective compactly.

Demonstration Generation. For each task, we generate a successful demonstration directly from the RoboLab simulator rather than using a learned policy. The task definition specifies the relevant target objects and their desired terminal poses or states. We use the simulator’s ground-truth object mesh models to represent the scene geometry and a motion planner to generate feasible robot trajectories that move each target object from its initial pose to the task-defined target pose. For compound tasks, the procedure is applied to each required manipulation in the order specified by the task objective. This produces a successful demonstration whose final scene satisfies the benchmark task predicate while avoiding failures or ambiguities introduced by a learned control policy. Participants observe only the resulting RGB demonstration video; RoboLab task names, benchmark instructions, event annotations, and success predicates are not shown.

Participants and Ethics. We recruit 50 college students between 20 and 27 years old (31 male and 19 female) for the instruction-elicitation study. Participants report no prior experience with robotics research or robot operation. The study involves minimal-risk instruction elicitation and does not collect sensitive or personally identifying information. Under our institutional policy, formal IRB review was not required. All participants provided informed consent before participation, and all responses were analyzed anonymously.

Instruction-Elicitation Procedure. Each participant observes demonstrations for all 15 selected tasks, yielding 50\times 15=750 human specifications. Task order is independently randomized for each participant to reduce systematic ordering effects. For each task, the interface displays the demonstration video together with a free-form text box. Participants may replay the video as many times as needed before submitting their response. No task name, canonical instruction, taxonomy label, or suggested vocabulary is shown. Participants are told that the robot in the video is acting to satisfy the objective of a human user. After observing the execution, they are asked to infer what the user wanted to accomplish and to express how they would naturally communicate the same objective to another person who could help complete the task. The prompt is:

Prompt shown to participants. You will watch a short video of a robot carrying out a task to meet the intent of a human user. Based on the robot’s actions, what do you think the human user wanted to accomplish? Imagine that you wanted to accomplish the same thing and were asking another person to help you. How would you naturally communicate what you want them to do? Write what you would naturally say to that person. There is no required format or level of detail. You may replay the video as many times as you need.

We deliberately frame the elicitation as human-to-human communication rather than asking participants to formulate instructions for a robot. Explicitly addressing a robot may cause speakers to accommodate to their assumptions about robot capabilities, for example by using more literal, procedural, or unnaturally explicit commands. Our goal instead is to capture how people naturally communicate an intended physical objective, including what they state explicitly and what they expect another agent to infer from context.

Coding Human Specifications. We annotate each of the 750 instructions both at the taxonomy level and at the executable-component level used in the information-gap analysis. For every task \tau, the grounded event sequence

E_{\tau}=(e_{1},\ldots,e_{K_{\tau}}),\qquad e_{k}=(A_{k},O_{k},D_{k})

serves as the semantic reference. For each required component s\in\{A_{k},O_{k},D_{k}\}, the instruction u receives a label \sigma_{k,s}(u)\in\{0,1,2\}, where \sigma_{k,s}(u)=0 indicates that the required information is omitted, \sigma_{k,s}(u)=1 indicates that it is expressed but its grounded value must be recovered from the observed scene or task context, and \sigma_{k,s}(u)=2 indicates that the grounded value is directly specified in language.

We use two independent model annotators, GPT-5.6 and Claude-Fable-5, to produce initial annotations for every human specification. Each model independently assigns the task-content, reference, and composition labels, together with the component-level \sigma_{k,s} annotations. Two human inspectors subsequently review the model annotations against the original instruction, demonstration, and grounded event representation. A label is entered into the final dataset only after the two human inspectors agree on the annotation. When the initial model annotations disagree, or when either inspector identifies an error, the case is reviewed explicitly and resolved by consensus. Thus, all statistics reported for the human study use the human-verified consensus annotations rather than either model’s prediction directly.

From the final \sigma annotations, we compute semantic coverage \eta_{\tau}(u) and groundedness \gamma_{\tau}(u) using Eqs. 1 and 2. We separately record the number J_{\tau}(u) of distinct manipulation units explicitly individuated by the instruction and compute the decomposition ratio \delta_{\tau}(u)=\frac{J_{\tau}(u)}{K_{\tau}}. For example, an instruction that describes three required object placements in three separate clauses has \delta_{\tau}=1, whereas a single collective expression covering the same three placements has J_{\tau}=1 and therefore \delta_{\tau}=1/3.

Taxonomy Coverage. The same consensus procedure is used to assign each specification its task-content, reference-form, and composition labels. A specification is considered covered by the taxonomy when all task-relevant aspects of its objective specification can be represented by these factors without introducing an additional Other category. Table V reports annotation agreement before human adjudication and final taxonomy coverage. The taxonomy covers 96.7\% of complete specifications. The remaining cases are primarily discourse-dependent expressions, such as “do the same with the rest,” whose interpretation depends on a previously established discourse pattern rather than on a recurring objective-specification form absent from the taxonomy.

Matched Human-Likeness Evaluation. We evaluate whether taxonomy-conditioned generation produces linguistically natural specifications, rather than merely satisfying the requested taxonomy constraints. For each human-written specification, we use its corresponding task and verified taxonomy labels, task content, reference form, and composition, to generate one matched specification. Each resulting pair therefore refers to the same physical task and instantiates the same taxonomy configuration, while differing in whether the surface realization was written by a human or generated by our procedure. Twenty additional participants, disjoint from the instruction-elicitation study, take part in a blinded source-identification evaluation. On each trial, participants view the task demonstration together with two candidate specifications: one human-written and one generated. The order of the two specifications is randomized independently on every trial, and the trials themselves are presented in randomized order. Participants are not told which source produced either specification.

Prompt shown to participants. Which of these two descriptions do you think was written by a person?

We report the fraction of trials on which participants correctly identify the human-written specification. Chance performance is 50\%. Because the two candidates are matched on both the underlying task and taxonomy configuration, differences in source-identification accuracy primarily reflect differences in linguistic realization, rather than differences in the amount or type of task information communicated.

Semantic Fidelity. Human-likeness alone does not guarantee that a generated specification preserves the intended physical objective. We therefore conduct a separate semantic-fidelity evaluation. A human inspector reviews each generated specification against the corresponding demonstration and grounded event sequence E_{\tau}, and determines whether the specification remains semantically consistent with the demonstrated task under its assigned taxonomy configuration. The inspector checks whether the generated instruction preserves the intended task semantics while respecting the requested task-content, reference, and composition forms. Surface linguistic differences are not considered errors as long as the resulting specification expresses the same intended objective at the prescribed level of explicitness. Table VI reports source-identification accuracy and semantic correctness by task structure. Overall source-identification accuracy is 52.8\%, close to the 50\% chance level, while semantic correctness is 96.0\%. Generation remains highly faithful for atomic and enumerative objectives and becomes somewhat more difficult for increasingly abstract task structures, particularly intent-compressible objectives. Together, the taxonomy-coverage, human-likeness, and semantic-fidelity results support using taxonomy-conditioned generation to construct controlled human-like specifications for the VLA evaluation.

-D Controlled Specification and Policy-Facing Realization

This appendix describes how we construct the robot-training language corpus and extract the policy-facing language prior used in Sec. III-C. The goal is not to recover the exact language seen by a policy during every optimization step, but to characterize the recurring ways in which grounded manipulation events are expressed in the robot data on which the evaluated policies were fine-tuned. We use this prior to construct fully resolved commands whose structure and surface realization are representative of the policies’ robot-training interface.

Corpus Construction. All three evaluated policies use checkpoints fine-tuned on DROID: \pi_{0.5} with pi05_droid_jointpos, \pi_{0}-FAST with pi0_fast_droid_jointpos, and Cosmos3-Edge with nvidia/Cosmos3-Edge-Policy-DROID. Although the implementations access DROID through different training pipelines and data formats, their robot-language supervision is derived from the same underlying dataset. We therefore construct a single DROID language corpus as a common proxy for the robot-language distribution shared by the three evaluated checkpoints. We collect the natural-language annotations associated with successful DROID demonstrations and treat each non-empty annotation as a corpus instance. When multiple annotations are available for the same demonstration, they are retained as separate instances. We do not deduplicate paraphrases, since repeated linguistic forms provide evidence about their prevalence in the training distribution. We apply only minimal text normalization, such as removing formatting artifacts and normalizing whitespace, before extracting command structure. The corpus is constructed independently of our evaluation language. In particular, RoboLab benchmark instructions, human-study specifications, and taxonomy-conditioned specifications generated for our experiments are excluded. Thus, the policy-facing realizations are derived only from robot-training language and do not use language from the evaluation tasks themselves.

Manipulation-Clause Extraction. We analyze the corpus at the level of manipulation clauses corresponding to the event representation used throughout the paper. Each clause is mapped, when possible, onto the semantic structure (A,O,D), where A denotes the manipulation, O the manipulated object, and D the destination or resulting target state. For instructions that explicitly contain multiple manipulation clauses, we treat the clauses separately when their event boundaries are unambiguous. This produces a collection of language realizations aligned with the same event granularity as e_{k}=(A_{k},O_{k},D_{k}). This processing is used only to extract linguistic regularities. It does not determine the event sequence E_{\tau} of a RoboLab task, which is obtained independently from the grounded event extraction pipeline described in Appendix -B.

Template Normalization. To separate recurring command structure from task-specific entities, we replace spans referring to manipulated objects and destinations with typed placeholders. For example, “Put the red mug inside the blue bowl” is normalized to “Put [OBJECT] inside [DESTINATION].” Instructions with the same normalized structure are grouped under a shared template t. Because the corpus is shared across the three policies, we estimate a single empirical template distribution

p(t)=\frac{N(t)}{\sum_{t^{\prime}}N(t^{\prime})},

where N(t) is the number of corpus instances realizing template t. We preserve these empirical frequencies rather than treating all observed templates as equally representative of the training language. Templates retain the linguistic frame of the command, including its action and relational wording. Consequently, surface forms such as “put [OBJECT] in [DESTINATION]” and “place [OBJECT] inside [DESTINATION]” may correspond to different templates even when they express the same grounded event. This allows the corpus to capture both structural and lexical regularities of robot-training instructions.

Lexical Realization. In addition to full-template frequency, we record the lexical forms used to express equivalent semantic components. For actions, this includes alternatives such as “pick up,” “grasp,” or “lift.” For destinations and target states, it includes alternative relational expressions such as “in,” “inside,” “onto,” or “on top of,” when they correspond to the same grounded relation. For a semantic component, we estimate the empirical frequency of each lexical realization \ell among compatible corpus instances. These statistics allow a grounded event to be expressed using language that is actually represented in the robot-training corpus rather than through unconstrained paraphrasing.

Reference Forms. We separately characterize how manipulated objects and destinations are referred to in the corpus. Reference expressions are grouped according to the information used to identify the grounded entity, including category names, attribute-qualified category names, attribute-based expressions, and relational descriptions. For example, depending on the available scene information, a grounded entity may be described as “the mug,” “the yellow mug,” “the yellow object,” or “the object beside the plate.” For each reference form, we retain both its structure and its empirical frequency in the DROID corpus. This distinction is important because two commands can share the same action semantics and syntactic structure while exposing different amounts of information about the referent.

Policy-Facing Realization Generation. Given a grounded event e_{k}=(A_{k},O_{k},D_{k}), we first identify the subset of corpus templates semantically compatible with the event’s action and target relation. A compatible template is then selected according to its empirical corpus frequency and instantiated with references to O_{k} and D_{k}. Lexical choices and reference forms are likewise restricted to realizations observed for compatible semantic components in the DROID corpus. When a reference form requires an attribute or spatial relation, the required information is obtained from the grounded scene representation associated with the task.

The resulting command r_{k}=\mathcal{R}(e_{k}) therefore expresses the complete grounded event while remaining close to the language interface represented in robot-training data. For a compound task E_{\tau}=(e_{1},\ldots,e_{K_{\tau}}), the procedure is applied independently to each event, yielding the atomic policy-facing sequence R_{\tau}=(r_{1},\ldots,r_{K_{\tau}}). Each r_{k} explicitly identifies the required manipulation, manipulated object, and destination or target state. The sequence R_{\tau} is therefore a fully resolved realization of the physical objective, rather than an alternative interpretation of it.

For the controlled communication-gap experiments, the selected policy-facing realizations are fixed across matched conditions. Thus, when human-like specifications are compared with the fully resolved reference, differences in behavior cannot be attributed to changes in the underlying event semantics or to independently varied command wording. The same atomic realizations are also used in the residual execution-gap experiments, where we vary only how the resolved events are organized and invoked.

-E Execution Protocol and Statistical Analysis

This section provides the execution details for the information-gap and residual execution-gap evaluations. Within each task, all compared conditions use the same physical objective, initial scene configuration, success criteria, and task-execution budget.

Simulation and Policy Observations. Experiments are run in RoboLab v0.3.1 using Isaac Sim 5.0.0 and Isaac Lab 2.2.0. Physics is simulated at 120 Hz, with a control decimation of 8, giving a policy control frequency of 15 Hz. The policy observes two RGB views: an over-the-shoulder left camera and a wrist-mounted camera. Both views are resized to 224\times 224 before being provided to the policy, together with the robot proprioceptive state.

Behavioral Metrics. We report task success and normalized partial progress. Success is determined by the RoboLab success predicate for the corresponding task. Progress measures the fraction of the task objective completed when the full success predicate is not satisfied. Both metrics are used for the information-gap and residual execution-gap analyses, with M\in\{\mathrm{Success},\mathrm{Progress}\}.

Execution Budget. Execution budgets are measured in policy-control steps at 15 Hz. For a task with benchmark episode horizon T_{\tau} seconds, the total task budget is B_{\tau}=15T_{\tau} control steps. The evaluated compound tasks use horizons of 60, 90, 120, 180, or 300 seconds, corresponding to total budgets of 900, 1350, 1800, 2700, or 4500 task steps, respectively. All presentation conditions receive the same total task budget. Under sequential execution, a task with K_{\tau} atomic events is assigned absolute event deadlines b_{k}=\left\lceil\frac{kB_{\tau}}{K_{\tau}}\right\rceil,\qquad k=1,\ldots,K_{\tau}. Thus, the nominal allocation is B_{\tau}/K_{\tau} steps per event, but unused time from an early-completed event remains available to later events. In the evaluated tasks, nominal per-event allocations are typically 450 steps (30 s), with longer allocations for tasks whose benchmark horizon is correspondingly larger. If an event has not completed by its deadline, it is marked as timed out and execution advances to the next event.

Information-Gap Execution. For each human-like specification u and the fully resolved reference r_{\tau}^{\mathrm{full}}, the complete specification is supplied once at the beginning of the rollout and remains fixed throughout execution. The policy acts until the benchmark success condition is satisfied or the total task budget is exhausted. The reference r_{\tau}^{\mathrm{full}} uses Action–Direct–Enumerated semantics: every required action, object, destination, and atomic event in E_{\tau} is explicitly specified. This provides the matched reference used to measure the behavioral cost of unresolved task information.

Residual Execution-Gap Conditions. To study execution after the information gap has been removed, we hold the fully grounded atomic command sequence R_{\tau}=(r_{1},\ldots,r_{K_{\tau}}) fixed and vary only how it is presented and invoked. In the one-prompt condition, all atomic commands are provided together at the beginning of the rollout. In the sequential condition, the policy receives only r_{1} initially and advances to r_{k+1} when the completion predicate for event e_{k} is satisfied or its deadline is reached. When the command changes, the policy’s cached action chunk is cleared so that the next action is generated from the new command.

Per-Event Completion Predicates. Sequential execution requires a completion criterion for each atomic event. We derive these predicates from RoboLab’s task definition and subtask structure rather than introducing separate manually specified success criteria. Each policy-facing command r_{k} is associated with the corresponding RoboLab stage or subtask group whose completion represents event e_{k}. During execution, the relevant RoboLab subtask state is queried read-only after each environment step. An event is considered complete when its associated stage or object group is marked complete by the benchmark. For repeated, visually interchangeable objects, completion is defined by the required count of completed object groups rather than by a particular object identity. Event completion is latched for the purpose of advancing through the command sequence.

Home-Reset Intervention. We additionally evaluate sequential execution with a home reset between events. After event e_{k} completes or reaches its deadline, and before r_{k+1} is issued, the gripper is opened for 6 control steps and the arm is moved back to the environment’s default home configuration by a 45-step linear interpolation in joint space. The robot then holds the home configuration for up to 10 additional steps, terminating the settling phase early when the maximum joint-position error falls below 0.02 rad. There is no reset before the first command or after the final command, and the simulator scene itself is not reset; object states produced by preceding events remain in the environment.

Reset motion does not consume the task-step budget used for policy execution. It does, however, consume simulated time. We therefore increase the environment time limit by up to 61(K_{\tau}-1) environment steps in the home-reset condition, corresponding to at most 6+45+10=61 reset steps at each event boundary. Consequently, the one-prompt, sequential, and sequential-plus-home-reset conditions all receive the same policy-controlled task budget.

Matched Evaluation. Each task–instruction condition is evaluated over 10 rollouts. Comparisons within a task use the same physical objective, initial scene distribution, evaluation criteria, and task budget. The information-gap experiments therefore vary only the information available in the specification, while the residual execution-gap experiments hold the resolved event sequence fixed and vary only its presentation and invocation state.

Aggregation and Confidence Intervals. The task is the unit of statistical analysis. For each task–condition pair, we first average outcomes over the 10 rollout trials and, where applicable, over surface realizations. Aggregate success and progress are then computed over these task-level means. We report 95% confidence intervals using a nonparametric bootstrap over tasks with 10,000 resamples. For matched comparisons, including the information-gap quantities \Delta^{\mathrm{info}}_{M}, we first compute the within-task difference between the compared conditions and bootstrap these paired task-level differences. Thus, repeated rollouts of the same task are not treated as independent samples.

Cost per Unresolved Unit. To estimate the behavioral cost of different forms of unresolved task structure in Fig. 3, we define, for each task–specification pair i, the number of missing semantic components m_{i}, grounding-dependent components g_{i}, and undecomposed events d_{i}:

m_{i}=\sum_{k,s}\mathbbm{1}[\sigma_{k,s}=0],\qquad g_{i}=\sum_{k,s}\mathbbm{1}[\sigma_{k,s}=1],\qquad d_{i}=K_{\tau_{i}}-J_{\tau_{i}}.

For each policy separately, we fit

\Delta^{\mathrm{info}}_{\mathrm{Success},i}=\beta_{\mathrm{miss}}m_{i}+\beta_{\mathrm{ground}}g_{i}+\beta_{\mathrm{decomp}}d_{i}+\epsilon_{i}.

The regression is fit without an intercept because the fully resolved reference has zero unresolved units and, by construction, zero matched information gap. The coefficients therefore give the estimated percentage-point success cost per missing semantic component, grounding-dependent component, and undecomposed event, respectively. Uncertainty is estimated by refitting the regression under the same task-level bootstrap procedure.

Association with Available Structure. For Fig. 3, we additionally measure how strongly the amount of available task structure is associated with the behavioral information gap as task complexity increases. Within each complexity group (K_{\tau}=1, K_{\tau}=2, and K_{\tau}\geq 3), we compute Spearman’s rank correlation between \Delta^{\mathrm{info}}_{\mathrm{Success}} and each of semantic coverage \eta_{\tau}, groundedness \gamma_{\tau}, and decomposition ratio \delta_{\tau}. Decomposition ratio is omitted for single-event tasks, for which \delta_{\tau}=1 by definition. More negative correlation indicates that greater availability of the corresponding executable structure is associated with a smaller success gap.

Reference for Grouped and Collective Conditions Grouped and collective specifications exist only for compound tasks, and we evaluate them on 54 and 50 of the 56 compound tasks, respectively. For these columns in Table II and Tables VII–IX, we compute \Delta^{\mathrm{info}}_{\mathrm{Success}} against the Action–Direct–Enumerated reference averaged over all 56 compound tasks, which is the one-prompt condition in Fig. 4 (23.5% for \pi_{0.5}, 8.2% for \pi_{0}-FAST, and 20.1% for Cosmos3-Edge).This departs from the per-subset matching used for the enumerated column.

-F Additional Models

We evaluate the communication gap on three policies with different architectures: two vision–language–action models, \pi_{0.5} and \pi_{0}-FAST, and the Cosmos3-Edge world–action model (WAM). All three models are evaluated using the same RoboLab task set, taxonomy-controlled specifications, physical objectives, rollout protocol, and evaluation metrics. This allows us to examine whether the observed communication gap is specific to a particular policy or persists across different model families. The main text focuses primarily on \pi_{0.5} because it achieves the strongest overall task performance and therefore provides the clearest dynamic range for analyzing the effect of unresolved task structure. For completeness, we report the full factorial results for all three models here. Table VII reports the results for \pi_{0.5}, Table VIII for \pi_{0}-FAST, and Table IX for Cosmos3-Edge. Across Tables VII–IX, the same qualitative pattern is visible. Grounding-dependent references produce comparatively small performance degradation, whereas omitting task semantics or compressing multiple events into grouped or collective specifications produces substantially larger losses. The effect is especially pronounced on compound tasks, where all three models are sensitive to whether the constituent manipulation events are explicitly exposed.

The magnitude of these effects differs across models. As shown in Table VII, \pi_{0.5} achieves the highest overall success and therefore exhibits the clearest separation between specification conditions. Table VIII shows that \pi_{0}-FAST operates at a lower overall success level, particularly on compound tasks, which compresses some absolute information-gap differences. Table IX shows that Cosmos3-Edge is comparatively robust to grounding-dependent references, while remaining sensitive to missing semantics and implicit event structure. Despite these differences in magnitude, the three models exhibit the same overall asymmetry: locally grounded information is comparatively recoverable, whereas reconstructing missing task semantics or latent event structure is substantially less reliable. The consistency of this pattern across Tables VII, VIII, and IX suggests that the communication gap is not specific to a single policy architecture.

-F1 Cross-Model Comparison

All three policies share the pattern reported in the main text: grounding dependence is cheap, missing semantics and undecomposed events are expensive, and sequential invocation from a familiar state improves execution. Within that shared pattern, however, the models differ in ways that track their architecture.

The information gap is shaped by the language backbone. Cosmos3-Edge pays the least for grounding-dependent references, consistent with a general-purpose vision–language backbone that resolves attribute and spatial reference. \pi_{0}-FAST pays the most (-1.6), and its penalties compress overall because it operates near floor on compound tasks. The decomposition penalty, however, is the largest term for every model, including \pi_{0.5}, whose explicit subtask prediction narrows but does not close it. No evaluated architecture recovers implicit event structure reliably.

The execution gap splits by mechanism. The sequentialization gain is shared, and largest for \pi_{0}-FAST (8.2% to 19.5%): it reflects the short, single-event commands common to all DROID-trained policies, not any particular action head. The invocation-state gain is not: Cosmos3-Edge gains 5.8 points from a home reset against 11.8 for \pi_{0.5}, since a world-action model denoises its action chunk jointly with an imagined rollout from the current frame, so an unfamiliar arm configuration is partly absorbed by the video prior rather than falling outside the action distribution.

-G Human Specification Strategies by Task Structure

The main text shows that natural specifications become less grounded and less explicitly decomposed as task complexity increases. Here we examine how people achieve that compression by analyzing the taxonomy forms they select for tasks with different structural affordances.

We group tasks according to the compression-affordance classes defined in Appendix -A: single atomic events, compound objectives that remain primarily enumerative, compound objectives that admit a compact grounded rule, and compound objectives that admit a compact task-level intent. For each elicited specification, we record its task-content form, reference form, and composition form. These labels are assigned from the specification itself, while the task-structure class is assigned independently from the grounded objective. Table XI reports the resulting distributions.

Task Content. Single-event tasks are predominantly expressed as direct actions, with 82\% of specifications using action-level descriptions. Compound objectives that do not admit a strong compact abstraction remain similarly action-oriented, with 68\% expressed at the action level. By contrast, when a grounded rule captures the shared structure of the task, participants more often describe the desired outcome: outcome-level specifications account for 58\% of rule-compressible tasks. Intent-compressible objectives shift further toward abstraction, with 55\% expressed through an intent or higher-level effect. This progression suggests that speakers do not choose task-content form independently of the objective structure. Concrete shared regularities favor outcome-level descriptions, whereas objectives naturally characterized by a purpose or effect invite intent-level specification.

Reference Form. Reference strategies shift in parallel. For atomic tasks, 92\% of specifications use direct references to the relevant objects or destinations. Direct reference remains dominant for compound enumerative tasks at 78\%, but falls to 46\% for rule-compressible tasks and 39\% for intent-compressible tasks. As a result, increasingly compressible objectives place more responsibility on the listener to resolve the intended entities from scene context, set membership, or the inferred task goal.

Composition. The strongest shift appears in how multiple manipulations are linguistically organized. Among compound enumerative tasks, 63\% of specifications explicitly enumerate the required manipulations. When a shared grounded rule is available, only 12\% remain enumerated, while 57\% use a collective description. Intent-compressible tasks show an even stronger preference for collective specification, at 69\%.

Thus, event count alone does not explain whether speakers enumerate or compress a compound objective. Participants preserve event-wise structure when the task requires it, but collapse multiple manipulations into a single expression when the grounded objective supports a shared rule or higher-level effect.

Implication for the Information Gap. The three taxonomy factors therefore shift together rather than independently. As task structure becomes more compressible, specifications move from action-level to outcome- or intent-level content, from direct to grounding-dependent reference, and from enumeration to collective composition. This produces a coordinated transfer of responsibility from speaker to listener: the speaker communicates a compact characterization of the desired objective, while the listener must recover the grounded entities, atomic manipulations, and, for intent-level descriptions, the physical goal itself.

This analysis complements the complexity result in Sec. IV-B. Increasing K_{\tau} creates more opportunity for compression, but whether speakers actually exploit that opportunity depends on the structure of the objective. Natural human specification is therefore shaped jointly by physical complexity and by the compact abstractions the task affords.

-H Reasoning Requirements and VLA Reliability

The main analysis characterizes the information gap by the executable structure available in a specification: semantic coverage, groundedness, and decomposition ratio. Here we provide a complementary view based on the reasoning operation required to recover the unresolved structure.

Reasoning Operations. We distinguish four recurring forms of recovery, summarized in Table XII. Visual grounding resolves a linguistically specified but scene-dependent object or destination. Action inference recovers an omitted manipulation from an explicitly specified entity and desired physical state. Task decomposition expands a compressed multi-object objective into its constituent grounded events. Intent inference recovers the intended physical goal from a higher-level purpose or preference before the corresponding executable structure can be determined. These operations may co-occur in unrestricted human specifications. To estimate their behavioral cost more directly, we use controlled taxonomy conditions that introduce one focal recovery requirement while keeping the remaining specification structure maximally explicit. Visual grounding is isolated by replacing direct references with grounding-dependent references under Action–Enumerated specifications; action inference by replacing Action with Outcome under Direct–Enumerated specifications; task decomposition by replacing Enumerated composition with Grouped or Collective composition under Action–Direct specifications; and intent inference by replacing Action with Intent under Direct–Enumerated specifications. Table XIII reports task success, partial progress, and the matched success gap relative to the fully resolved Action–Direct–Enumerated reference over the same eligible task subset.

Locally Constrained Recovery Is Comparatively Reliable. Visual grounding produces a 2.8 percentage-point success gap, while action inference produces a 4.5-point gap. In both cases, the missing value is strongly constrained by information that remains available: a scene-dependent reference can often be resolved from the observation, while an omitted action can often be inferred from an explicitly stated object and desired outcome.

Structural and Intent Recovery Are Harder. Compressing multiple manipulations into grouped or collective specifications produces matched success gaps of 7.6 and 10.4 percentage points, respectively. Here, the policy must reconstruct the individual manipulation events underlying a compact expression rather than resolve a single local argument. Intent-level specifications are more difficult still, producing a 15.2-point gap even when references are direct and event structure is explicitly enumerated. In this case, the policy must first infer the intended physical objective before mapping it onto executable behavior. Partial progress shows the same overall ordering.

Implication. The difficulty of an information gap depends not only on how much structure is unresolved, but also on what kind of recovery is required. Locally constrained ambiguities, such as contextual grounding or an omitted action, are comparatively recoverable. Reconstructing latent event structure or inferring the physical objective itself is substantially less reliable. This reasoning-based analysis therefore complements the structural metrics in the main text by distinguishing different computational demands that can arise from similar amounts of implicit task information.