从人类视频到机器人操作：用因果束选择“该迁移什么”的跨本体迁移

Abstract

Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manipulation videos, in contrast, provide abundant and diverse task experience that could potentially benefit robot learning. However, differences in morphology, action spaces, and control mechanisms make human motions unsuitable as direct supervision for robot actions, raising a scientific question: what knowledge in human videos should be transferred across embodiments to robot manipulation? To address this question, we analyze subtask transfer through the lens of causal beams, focusing on which contextual conditions and interaction dependencies should be retained to support local task effects. Based on this perspective, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST learns manipulation-related representations from human videos and grounds them in task-level semantic variables using robot demonstrations. Within a predefined manipulation model, causal-beam constraints guide contextual information selection by requiring retained assignments to sustain local mechanism outputs while preserving nontrivial responsiveness to retained inputs. The selected semantic information conditions a world-action model, allowing the robot to generate actions from its own observations without requiring human–robot motion correspondence. Experiments on robot manipulation benchmarks show improvements in task success over the corresponding world-action model baselines.

October 15, 2026

1 Introduction

Recent robot manipulation models are evolving toward World Action Models (WAMs) that jointly model future scene evolution and robot behavior, enabling predictions of interaction outcomes (Kim et al., 2026; Li et al., 2026). However, scaling real-world robot data requires dedicated hardware, human operation, and physical deployment, making broad coverage of tasks, objects, and scenes expensive. In contrast, human manipulation videos are abundant and capture diverse task processes and object interactions, making them an attractive source of additional experience for robot learning (Khazatsky et al., 2025; Ma et al., 2026; Wang et al., 2023). Yet humans and robots differ substantially in morphology, action spaces, and control mechanisms, so concrete human actions cannot be directly reused as robot supervision (Bahl et al., 2022; Xu et al., 2023; Kim et al., 2025a). This leads to the central question of this work: what knowledge in human manipulation videos should be transferred across embodiment differences to robot manipulation?

A complete human manipulation sequence contains two types of information: object-interaction relations that support task progress, and concrete motion realizations that are closely tied to the human embodiment. For long-horizon manipulation tasks, we first organize continuous execution into local task processes using temporally extended subtasks, thereby avoiding the requirement that the robot reproduce human motion step by step. However, defining a subtask only specifies the local scope within which transfer takes place; it does not further determine which information within that scope should be preserved together with the task experience. Retaining only the task intention or boundary states of a subtask may discard the conditions and interaction relations that determine how a local effect is realized, whereas retaining the complete human execution trajectory reintroduces substantial embodiment-specific motion details.

We study this selective information-preservation problem through the lens of causal beams (Pearl, 2009). Given a structural model and a specific context, a causal beam allows part of the inputs to be projected out in a context-dependent manner, while requiring the actual values of the retained variables to sustain the actual outcome of the current mechanism and the projected mechanism to retain nontrivial responsiveness to its retained inputs. This perspective motivates separating task-level abstraction from contextual information selection. Task-level variables provide an abstraction of execution, while causal-beam conditions constrain which semantic dependencies are retained within a specified model.

Based on this, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST first learns manipulation-related temporal representations through human-video pretraining and grounds them, under robot demonstration supervision, into a candidate SCM instantiated from shared manipulation templates. A contextual subtask-conditioning module then selects a contextual causal beam from the candidate SCM according to the current execution context, producing the local task effect, relevant conditions, and their effect-relevant dependencies. This structural information conditions the target robot WAM, while concrete actions are still generated from the robot’s current state and its own action space. CBST therefore combines human-video representation transfer with causal-beam-constrained selection of contextual subtask information for robot execution.

Our contributions are summarized as: (1) We study human-to-robot learning from the perspective of cross-embodiment transfer, focusing on which task-progress structures should remain transferable after embodiment-specific motion is discarded; (2) We propose Causal-Beam-Guided Subtask Transfer (CBST), which combines a shared candidate manipulation SCM with learned contextual causal beams to preserve effect-relevant conditions and dependencies for conditioning robot WAMs; (3) We systematically evaluate CBST on WAMs including LingBot-VA and Cosmos Policy, examining the contributions of human video pretraining, context-specific causal selection, and their effects on robot control.

2 Related Works

Human-to-robot transfer aims to exploit abundant human demonstrations while addressing the embodiment gap between human and robot execution. Motion-oriented approaches seek representations that remain useful across embodiments while retaining execution information. WHIRL aligns human and robot videos to infer task intent and initializes robot policy learning from third-person human demonstrations (Bahl et al., 2022), while EgoMimic uses egocentric human videos with 3D hand tracking, cross-domain alignment, and joint training on human and robot demonstrations (Kareer et al., 2024). Latent Action Pretraining (LAPA) instead learns discrete latent actions from videos without robot action labels and subsequently maps them to robot actions using limited robot data (Ye et al., 2025). Other approaches move toward higher-level transferable representations. MimicPlay learns latent plans from large-scale human play videos and uses them to guide a low-level robot controller (Wang et al., 2023); XSkill discovers cross-embodiment skill prototypes from unlabeled human and robot videos and maps them to robot actions with a conditional policy (Xu et al., 2023); Translating Robot Skills learns unsupervised correspondences between skill sequences across morphologically different agents (Shankar et al., 2022); and UniSkill learns embodiment-agnostic skill representations that allow skills extracted from human video prompts to guide robot policies trained on robot data (Kim et al., 2025a). More recently, Gen2Act generates human future videos for novel tasks and conditions a robot policy on the generated motion information (Bharadhwaj et al., 2024).

Temporal abstraction organizes long-horizon behavior into reusable units beyond primitive actions. The options framework formalizes temporally extended policies through initiation sets, internal policies, and termination conditions (Sutton et al., 1999), while later robotic methods model skill feasibility and composition, such as SayCan (Ahn et al., 2022) and relational skill-precondition learning. Another line of work represents task progress through intermediate subgoals: Video Language Planning generates hierarchical visual and language plans (Du et al., 2023), TaKSIE conditions visual subgoal generation on task progress (Kang and Kuo, 2024), and Latent Space Backward Planning recursively constructs intermediate latent goals from the final task goal (Liu et al., 2025). More recent world-model methods integrate such temporal abstraction directly into planning: Hierarchical Planning with Latent World Models learns dynamics at multiple temporal scales and uses high-level latent subgoals to guide low-level planning (Zhang et al., 2026), while SAGE similarly exploits temporally extended latent subgoals for action-sequence generation (Cheng et al., 2026).

World models learn predictive dynamics to support action selection and planning. Early latent-dynamics approaches such as PlaNet perform planning directly in learned latent spaces (Hafner et al., 2019), while more recent methods extend this idea to visual robot manipulation: DINO-WM predicts future visual features for zero-shot planning (Zhou et al., 2025), V-JEPA 2-AC learns action-conditioned latent dynamics for model-predictive control (Assran et al., 2025), and IRASim models fine-grained action-conditioned visual futures for simulation and planning (Zhu et al., 2025). Recent World-Action Models further unify future prediction with action generation. WorldVLA jointly models visual and action tokens in an autoregressive framework (Cen et al., 2025); Cosmos Policy generates robot actions together with future-state images and value estimates (Kim et al., 2026); LingBot-VA jointly learns future-frame prediction and policy execution in a shared visual–action latent space (Li et al., 2026); and \tau_{0}-WM further combines video prediction, action generation, and action evaluation within a unified model (Zhou et al., 2026).

3 Problem Definition and Analysis

3.1 Problem Formulation

Robot manipulation is a sequential decision-making process: given a task instruction g, the robot generates actions based on the available information and obtains new observations through interaction with the environment. Let s_{t} denote the robot observation at time t, comprising visual input and available proprioceptive measurements, and let a_{t} denote a robot action. The interaction history is h_{t}=(s_{0},a_{0},\ldots,a_{t-1},s_{t}). We consider a world-action model (WAM) that jointly models robot actions and future observations conditioned on the interaction history h_{t} and task instruction g.

Given h_{t} and g, the WAM generates an action chunk and predicts future observations through the joint conditional distribution p_{\theta}(A_{t},S_{t}^{+}\mid h_{t},g). Here, A_{t}=(a_{t},\ldots,a_{t+H-1}) denotes an action chunk of length H, and S_{t}^{+}=(s_{t+\delta})_{\delta\in\mathcal{T}_{H}} denotes the future-observation sequence at offsets \mathcal{T}_{H}\subseteq\{1,\ldots,H\}. After executing the action chunk, the robot updates the interaction history using newly acquired observations, forming a closed-loop control process.

3.2 Causal Beams

We use a recursive structural causal model M=(U,V,\mathcal{F}), where each non-root variable V_{i}\in V is determined by a structural mechanism V_{i}=f_{i}(\mathrm{PA}_{i},U). A context U=u determines the actual values V_{i}(u). For the causal-beam construction, the parent set of each mechanism is partitioned as

\mathrm{PA}_{i}=C_{i}\mathbin{\dot{\cup}}N_{i},

where C_{i} denotes the retained parents and N_{i} the remaining parents. The retained parents must sustain the actual output:

f_{i}(C_{i}(u),n_{i},u)=V_{i}(u),\qquad\forall n_{i}\in\operatorname{Val}(N_{i}).

In this work, we use the natural-beam specialization of the general causal-beam construction (Pearl, 2009). The remaining inputs are fixed at their actual values, giving the projected mechanism

f_{i}^{u}(c_{i})=f_{i}(c_{i},N_{i}(u),u).

The projection must remain nontrivially responsive to its retained inputs:

\exists c_{i}^{\prime}\in\operatorname{Val}(C_{i})\quad\text{s.t.}\quad f_{i}^{u}(c_{i}^{\prime})\neq V_{i}(u).

Thus, the actual assignments of the retained parents sustain the current mechanism’s output, while the projected mechanism is nonconstant in the retained inputs considered jointly. This local responsiveness condition does not establish that every retained parent is individually necessary or that each retained upstream variable has an intervention effect on the downstream effect Y in the composed model. A collection of such projected mechanisms forms a natural beam only when their retained and projected variables are mutually compatible across mechanisms. The full compatibility conditions are given in Appendix A.

3.3 Causal Analysis of Cross-Embodiment Transfer

Humans and robots may accomplish the same task through different motion realizations because of differences in morphology, action spaces, and control mechanisms. This motivates describing local task processes using semantic variables for task conditions, manipulation attempts, object-state changes, and effects, without requiring step-by-step correspondence between human and robot motion. Defining these variables, however, does not determine which conditions and dependencies should be retained in a particular execution context. We use causal-beam conditions to guide this contextual selection within a specified task-level structural model.

To formalize the task-relevant structure under consideration, we use a candidate SCM as Fig.2(a) for a local manipulation process. We assume that human and robot executions of the same local manipulation can be described using a shared set of task-level variable semantics and candidate dependency relations, while their embodiment-specific structural functions may differ. This shared schema defines the hypothesis space in which cross-embodiment structural correspondence is considered; it does not assert that every candidate dependency is active in every execution context. Sharing this schema does not guarantee that a beam valid in one embodiment remains valid in another; the beam conditions must be evaluated relative to the instantiated mechanisms and execution context. Let X denote task-relevant conditions at the beginning of the interval, I the manipulation attempt, D the resulting object-state change, and Y the realized local effect:

X=U_{X},I=U_{I},D=f_{D}^{e}(X,I,U_{D}),Y=f_{Y}^{e}(X,D,U_{Y}),

where e\in\{h,r\} denotes the embodiment. Thus, humans and robots share the task-level variables and the hypothesis space of possible dependencies, but not necessarily the concrete mechanisms that realize these dependencies. The mechanisms f_{D}^{e} and f_{Y}^{e} may therefore differ between humans and robots. The distinction between I and D separates an attempted manipulation from the object change that is actually realized. This schema specifies candidate task-relevant variable roles and dependencies rather than a complete physical model; task-specific variables are instantiated from the corresponding manipulation specification. Within this candidate structure, causal-beam selection determines which dependencies are retained in a particular execution context.

Consider moving a bowl into a drawer. Let B indicate whether a transport motion is executed, G whether the grasp is effective, O whether the drawer entrance is accessible, D whether the bowl is carried by the transport motion, and Y whether it reaches the target region. Here X=(G,O) and I=B. A simplified local model is

D=B\land G,Y=D\land O.

In a successful context with B=G=O=1, no proper subset of \{B,G\} sustains D=1, and no proper subset of \{D,O\} sustains Y=1. A projection of the original mechanism that omits one of these parents may still match the observed endpoint, but it is not a valid sustaining projection in this context. In contrast, when transport is attempted with an ineffective grasp, B=1,G=0,O=1, one valid natural projection retains G for D and D for Y, yielding D=G and Y=D. Thus, the same candidate SCM can admit different valid projections under different execution contexts. Appendix A provides the full verification of the causal-beam conditions. This example highlights two points relevant to transfer. First, matching an observed endpoint does not imply that the dependencies producing that endpoint have been preserved. Second, the structure retained for a local process is context dependent.

This motivates using causal-beam conditions to constrain which task-level dependencies are preserved in the transferred representation, with contextual selection performed under the robot-side reference mechanisms rather than by directly reusing a human-side beam.

Definition 3.1 (Causal Subtask Representation).

Given a candidate manipulation SCM M for a local task process and a context u, let M_{u} denote a causal beam of M. We define the causal subtask representation as

\sigma=(Y(u),\mathcal{C}_{u},\mathcal{R}_{u}),

where Y(u) denotes the realized local effect, \mathcal{C}_{u}=\{(C_{i},C_{i}(u))\}_{i} collects the retained parents and their actual values, and \mathcal{R}_{u} records the parent–child relations retained within the projected local mechanisms. The projected mechanisms f_{i}^{u} are used to determine whether a retained parent set satisfies the causal-beam conditions, but the representation transfers the retained task-level dependencies rather than requiring identical projected functions across embodiments. Excluding a parent from the transferred dependency set does not remove its role in the projection context: its fixed value can still determine how the projected mechanism responds to changes in the retained inputs.

The local task goal specifies the desired change, whereas \sigma describes dependencies supporting the observed local effect in a given context. An unrealized goal alone does not distinguish ongoing execution from failure.

4 Method

We propose Causal-Beam-Guided Subtask Transfer (CBST). Human-video pretraining initializes a visual–temporal module, which is then adapted with robot demonstrations to estimate task-level semantic variables from robot observations. Using these estimates, a learned selector chooses contextual parent assignments subject to causal-beam constraints within a predefined manipulation model. The selected assignments are rendered as language conditions for WAM, which combines them with the robot’s native observations to generate actions and future observation predictions.

4.1 Overall Architecture

Following the task-level formulation in Section 3.3, CBST maps robot observations to semantic estimates, selects contextual dependencies, and uses the resulting subtask information to condition action generation. Given the interaction history h_{t} and task instruction g, a visual–temporal encoder and semantic prediction heads produce

z_{t}=E_{\eta}(h_{t},g),\hat{\mathbf{v}}_{t}=P_{\omega}(z_{t}),

where z_{t} summarizes the observed manipulation process and \hat{\mathbf{v}}_{t} estimates the conditions, manipulation attempts, object-state changes, and effects defined by the local SCM. Discretizing these predictions gives the contextual assignment \bar{\mathbf{v}}_{t}. Predicted states and task-specified prerequisite rules also determine the currently admissible goals \hat{G}_{t}.

The estimated context is then used to instantiate the causal-beam conditions in Section 3.2. Under fixed reference mechanisms f_{i}^{\mathrm{ref}}, let \hat{\mathcal{A}}_{i,t} contain the parent subsets satisfying sustenance and responsiveness at \bar{\mathbf{v}}_{t}. A learned selector assigns probabilities p_{i,t}(C) from z_{t} and the semantic predictions, and jointly selects compatible subsets:

\displaystyle\{C_{i,t}^{*}\}_{i}\in\operatorname*{arg\,max}_{\{C_{i}\}_{i}} \\ \displaystyle\sum_{i}\log p_{i,t}(C_{i}) \\ \displaystyle\text{subject to} \\ \displaystyle C_{i}\in\hat{\mathcal{A}}_{i,t},\quad(\mathrm{PA}_{i}\setminus C_{i})\cap C_{j}=\varnothing\quad\text{for all }i\neq j.

For a feasible selection, the retained assignments and relations form \hat{\sigma}_{t}=(\bar{Y}_{t},\hat{\mathcal{C}}_{t},\hat{\mathcal{R}}_{t}), an estimated instance of the causal subtask representation in Definition 3.1. Its validity is relative to the reference mechanisms and estimated context. A deterministic renderer combines the task instruction, admissible goals, and selected subtask information into a contextual description. The text encoder and conditional generative network complete the data flow:

r_{t}=\mathcal{R}(g,\hat{G}_{t},\hat{\sigma}_{t}),c_{t}=T_{\theta}(r_{t}),(A_{t},\hat{S}_{t}^{+})\sim p_{\theta}(\,\cdot\mid\mathcal{O}_{t}^{W},c_{t}).

Here, \mathcal{O}_{t}^{W} denotes the robot-observation condition, A_{t} the generated action chunk, and \hat{S}_{t}^{+} the predicted future observations. The renderer includes realized effects and selected variable assignments; retained relations guide field selection but are not explicitly serialized. If beam selection is infeasible, its fields are omitted while task and state conditioning are retained. The WAM and its text encoder remain frozen. After action execution, new observations update h_{t}, and the estimation–selection–generation process repeats.

4.2 Learning Objectives and Training

Following the analysis in Section 3.3, training grounds the task-level semantic variables and learns contextual parent selection under causal-beam constraints. We first initialize the visual–temporal encoder through action-category classification on human videos:

\mathcal{L}_{h}=\mathbb{E}_{(o^{h},y^{h})}\left[\operatorname{CE}\!\left(H(E_{\eta}(o^{h})),y^{h}\right)\right],

where o^{h} is a video clip, y^{h} its category label, and H a classification head. The pretrained encoder then initializes robot adaptation. Robot demonstrations supervise task states and interval variables through \mathcal{L}_{\mathrm{sem}}=\mathcal{L}_{\mathrm{state}}+\lambda_{v}\mathcal{L}_{\mathrm{var}}, with each component normalized over valid labels. An auxiliary mechanism-consistency loss \mathcal{L}_{\mathrm{mech}} encourages predicted parent variables to reproduce annotated outputs under the fixed reference mechanisms. This loss uses valid, reference-consistent annotations, while other valid labels remain available for semantic supervision. The reference mechanisms remain frozen during training.

Structural supervision follows the sustenance and responsiveness conditions in Section 3.2. For an annotated context v, finite-domain enumeration under the fixed reference mechanisms produces the acceptable parent subsets \mathcal{A}_{i}(v). The selector learns to assign probability mass to this set, while a separate regularizer favors compact selections:

\mathcal{L}_{\mathrm{beam}}=-\sum_{i\in\mathcal{I}(v)}\log\sum_{C\in\mathcal{A}_{i}(v)}p_{i}(C),\mathcal{L}_{\mathrm{size}}=\sum_{i\in\mathcal{I}(v)}\mathbb{E}_{C\sim p_{i}(\cdot\mid C\in\mathcal{A}_{i}(v))}[\frac{|C|}{|\mathrm{PA}_{i}|}].

Here, \mathcal{I}(v) contains mechanisms with valid, reference-consistent annotations, complete variable domains, and nonempty acceptable sets. Other valid annotations remain available for semantic supervision. The beam loss permits multiple acceptable subsets; the size regularizer expresses a compactness preference rather than an additional beam-validity condition. Cross-mechanism compatibility is enforced by the joint selection procedure in the preceding subsection. The robot adaptation objective is

\mathcal{L}_{\mathrm{robot}}=\mathcal{L}_{\mathrm{sem}}+\lambda_{m}\mathcal{L}_{\mathrm{mech}}+\lambda_{b}\mathcal{L}_{\mathrm{beam}}+\lambda_{\mathrm{size}}\mathcal{L}_{\mathrm{size}}.

Training updates the trainable encoder components, semantic prediction heads, and selector. The reference mechanisms, WAM, and native text encoder remain frozen. At inference, beam validity is checked using the discrete estimated context, and compatible selections are used to construct the action-generation condition.

5 Experiments

5.1 Experimental Setup

We pretrain CBST’s ResNet18–GRU visual-temporal encoder on Something-Something V2 (SSv2) (Goyal et al., 2017) and adapt the planner using robot demonstrations, keeping the WAM and its text encoder frozen. We evaluate LingBot-VA on RoboTwin 2.0 and LIBERO, and Cosmos Policy (Predict2.5) on LIBERO and RoboCasa (Chen et al., 2025; Liu et al., 2023; Nasiriany et al., 2024). Comparisons include the underlying WAMs and representative published policies, using officially reported baseline scores (Li et al., 2026; Kim and Gu, 2026). We report task success rates (%) under official success criteria, averaged across evaluated tasks. Dataset composition, baseline descriptions, implementation and hardware configurations, and evaluation protocols are provided in Appendix B.

5.2 Results on Simulation Benchmarks

We evaluate CBST with two frozen WAM backbones across three simulation benchmarks (Tables 2–4). On RoboTwin 2.0, LingBot-VA + CBST achieves 94.00% and 92.54% success under Easy and Hard settings. On LIBERO-Long, CBST achieves 98.60% with LingBot-VA and 97.33% with Cosmos Policy (Predict2.5). On RoboCasa, Predict2.5 + CBST reaches 72.44% with 50 demonstrations per task, compared with 71.1% for the backbone. Figure 4 illustrates a CBST execution on LIBERO. Full baseline comparisons, detailed results, and additional visualizations appear in Appendices C.

5.3 Ablation Studies

We examine human-video pretraining, contextual structural selection, and mechanism-consistency supervision on RoboTwin Easy with LingBot-VA as the frozen backbone (Table 4). Pretraining variants remove human-video initialization or shuffle its temporal order; conditioning variants use state-only information, the full candidate structure, or size-matched random subsets; the objective variant removes mechanism-consistency supervision. Robot adaptation data and evaluation settings remain fixed, while conditioning variants share the same trained predictor. More results across both datasets, detailed variant definitions, and computational costs are provided in Appendix D.

5.4 Real-World Experiments

We evaluate CBST on an AgileX dual-arm platform across ten object-placement tasks, with 20 trials per task and method. Each task specifies an object and a fixed target location. In cross-workspace configurations, no single arm can reach both the object’s initial position and its destination, requiring coordinated transport through a shared handover region. We compare Cosmos Policy with the Predict2.5 backbone against the same backbone augmented with CBST, and measure full-task success after object release. Task definitions, deployment settings, and the complete result table are provided in Appendix E. Figure 5 shows a recorded CBST cup-placement execution, including the intermediate handover. Additional sequences appear in Appendix E.4.

5.5 Human-Video Pretraining

We examine the role of temporal ordering during human-video pretraining by comparing ordered and frame-shuffled SSv2 clips. Both variants use the same ResNet18–GRU architecture, action labels, sampled frame content, and optimization budget; only the temporal order of the pretraining inputs differs. Figure 6 shows the training classification loss and final validation top-1 accuracy after three epochs. Ordered pretraining achieves 25.88% accuracy, compared with 22.15% for the shuffled control, supporting the utility of temporal order for SSv2 action recognition.

We further evaluate the downstream contribution of this pretraining through the robot ablations in Table 4. Under the same robot adaptation protocol, ordered pretraining yields 94.00% success on RoboTwin Easy, compared with 92.94% for shuffled pretraining and 92.80% without human-video pretraining. These comparisons support the practical benefit of ordered human-video pretraining for robot control. The pretrained representation initializes the robot planner; robot-specific prediction heads and structural selection are learned during adaptation.

6 Conclusion

We presented Causal-Beam-Guided Subtask Transfer (CBST), which uses a causal-beam perspective to guide contextual information selection for human-to-robot transfer. Human-video pretraining initializes manipulation-related representations, while robot demonstrations ground these representations in task-level semantic variables. Within a predefined manipulation model, causal-beam conditions constrain which contextual assignments are retained and supplied to a frozen world-action model for closed-loop control. Experiments on robot manipulation benchmarks show task-success improvements over the corresponding backbone baselines.

References

1. Ahn et al. (2022) M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng Do as i can, not as i say: grounding language in robotic affordances. External Links: 2204.01691, Link
2. Assran et al. (2025) M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, Mojtaba, Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, and N. Ballas V-jepa 2: self-supervised video models enable understanding, prediction and planning. External Links: 2506.09985, Link
3. Bahl et al. (2022) S. Bahl, A. Gupta, and D. Pathak Human-to-robot imitation in the wild. External Links: 2207.09450, Link
4. Bharadhwaj et al. (2024) H. Bharadhwaj, D. Dwibedi, A. Gupta, S. Tulsiani, C. Doersch, T. Xiao, D. Shah, F. Xia, D. Sadigh, and S. Kirmani Gen2Act: human video generation in novel scenarios enables generalizable robot manipulation. External Links: 2409.16283, Link
5. Bi et al. (2025) H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, H. Zhao, H. Liu, Z. Su, L. Ma, H. Su, and J. Zhu Motus: a unified latent action world model. External Links: 2512.13030, Link
6. Black et al. (2026) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky \pi_{0}: A vision-language-action flow model for general robot control. External Links: 2410.24164, Link
7. Cen et al. (2025) J. Cen, C. Yu, H. Yuan, Y. Jiang, S. Huang, J. Guo, X. Li, Y. Song, H. Luo, F. Wang, D. Zhao, and H. Chen WorldVLA: towards autoregressive action world model. External Links: 2506.21539, Link
8. Chen et al. (2025) T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, W. Deng, Y. Guo, T. Nian, X. Xie, Q. Chen, K. Su, T. Xu, G. Liu, M. Hu, H. Gao, K. Wang, Z. Liang, Y. Qin, X. Yang, P. Luo, and Y. Mu RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. External Links: 2506.18088, Link
9. Cheng et al. (2026) L. Cheng, Q. Zhang, and Y. Wang SAGE: subgoal-conditioned action generation for latent world model planning. External Links: 2607.17973, Link
10. Cho et al. (2014) K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans (Eds.), Doha, Qatar, pp. 1724–1734. External Links: Link, Document
11. Du et al. (2023) Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, L. Kaelbling, A. Zeng, and J. Tompson Video language planning. External Links: 2310.10625, Link
12. Goyal et al. (2017) R. Goyal, S. E. Kahou, V. Michalski, J. Materzyńska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic The ”something something” video database for learning and evaluating visual common sense. External Links: 1706.04261, Link
13. Hafner et al. (2019) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson Learning latent dynamics for planning from pixels. External Links: 1811.04551, Link
14. He et al. (2015) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. External Links: 1512.03385, Link
15. Intelligence et al. (2025) P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky \pi_{0.5}: A vision-language-action model with open-world generalization. External Links: 2504.16054, Link
16. Kang and Kuo (2024) X. Kang and Y. Kuo Incorporating task progress knowledge for subgoal generation in robotic manipulation through image edits. External Links: 2410.11013, Link
17. Kareer et al. (2024) S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu EgoMimic: scaling imitation learning via egocentric video. External Links: 2410.24221, Link
18. Khazatsky et al. (2025) A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, V. Guizilini, D. A. Herrera, M. Heo, K. Hsu, J. Hu, M. Z. Irshad, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O’Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Martín-Martín, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn DROID: a large-scale in-the-wild robot manipulation dataset. External Links: 2403.12945, Link
19. Kim et al. (2025a) H. Kim, J. Kang, H. Kang, M. Cho, S. J. Kim, and Y. Lee UniSkill: imitating human videos via cross-embodiment skill representations. External Links: 2505.08787, Link
20. Kim et al. (2025b) M. J. Kim, C. Finn, and P. Liang Fine-tuning vision-language-action models: optimizing speed and success. External Links: 2502.19645, Link
21. Kim et al. (2026) M. J. Kim, Y. Gao, T. Lin, Y. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M. Liu, C. Finn, and J. Gu Cosmos policy: fine-tuning video models for visuomotor control and planning. In The Fourteenth International Conference on Learning Representations, External Links: Link
22. Kim and Gu (2026) M. J. Kim and J. Gu Cosmos policy: fine-tuning video models for visuomotor control and planning. NVIDIA, Stanford University. Note: https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/post_training/predict2/cosmos_policy/post_training.htmlNVIDIA Cosmos Cookbook
23. Li et al. (2026) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu Causal world modeling for robot control. External Links: 2601.21998, Link
24. Liang et al. (2025) J. Liang, P. Tokmakov, R. Liu, S. Sudhakar, P. Shah, R. Ambrus, and C. Vondrick Video generators are robot policies. External Links: 2508.00795, Link
25. Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone LIBERO: benchmarking knowledge transfer for lifelong robot learning. External Links: 2306.03310, Link
26. Liu et al. (2025) D. Liu, H. Niu, Z. Wang, J. Zheng, Y. Zheng, Z. Ou, J. Hu, J. Li, and X. Zhan Efficient robotic policy learning via latent space backward planning. External Links: 2505.06861, Link
27. Ma et al. (2026) J. Ma, E. Zhang, H. Yang, D. Li, C. Xu, G. Wang, and H. Wang Robot learning from human videos: a survey. External Links: 2604.27621, Link
28. Nasiriany et al. (2024) S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu RoboCasa: large-scale simulation of everyday tasks for generalist robots. External Links: 2406.02523, Link
29. NVIDIA et al. (2025) NVIDIA, :, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. ”. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu GR00T n1: an open foundation model for generalist humanoid robots. External Links: 2503.14734, Link
30. Pearl (2009) J. Pearl Causality. Causality: Models, Reasoning, and Inference, Cambridge University Press. External Links: ISBN 9780521895606, LCCN 99042108, Link
31. Shankar et al. (2022) T. Shankar, Y. Lin, A. Rajeswaran, V. Kumar, S. Anderson, and J. Oh Translating robot skills: learning unsupervised skill correspondences across robots. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, pp. 19626–19644. External Links: Link
32. Sutton et al. (1999) R. S. Sutton, D. Precup, and S. Singh Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112 (1), pp. 181–211. External Links: ISSN 0004-3702, Document, Link
33. Wang et al. (2023) C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar MimicPlay: long-horizon imitation learning by watching human play. External Links: 2302.12422, Link
34. Xu et al. (2023) M. Xu, Z. Xu, C. Chi, M. Veloso, and S. Song XSkill: cross embodiment skill discovery. External Links: 2307.09955, Link
35. Ye et al. (2025) S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y. Chao, B. Y. Lin, L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo Latent action pretraining from videos. In The Thirteenth International Conference on Learning Representations, External Links: Link
36. Zhang et al. (2026) W. Zhang, B. Terver, A. Zholus, S. Chitnis, H. Sutaria, M. Assran, R. Balestriero, A. Bar, A. Bardes, Y. LeCun, and N. Ballas Hierarchical planning with latent world models. External Links: 2604.03208, Link
37. Zheng et al. (2025) J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, Y. Zhang, J. Pang, J. Liu, T. Wang, and X. Zhan X-vla: soft-prompted transformer as scalable cross-embodiment vision-language-action model. External Links: 2510.10274, Link
38. Zhou et al. (2025) G. Zhou, H. Pan, Y. LeCun, and L. Pinto DINO-WM: world models on pre-trained visual features enable zero-shot planning. In Forty-second International Conference on Machine Learning, External Links: Link
39. Zhou et al. (2026) P. Zhou, S. Chen, D. Chen, J. Wang, R. Jin, B. Zhu, Y. Pan, S. Gu, K. Wang, S. Nan, X. Qiu, C. Qiu, P. Yang, Y. Cai, J. Gao, Y. Li, Y. Fu, X. Yue, Z. Chen, and J. Luo \tau_{0}-WM: a unified video-action world model for robotic manipulation. External Links: 2606.01027, Link
40. Zhu et al. (2025) F. Zhu, H. Wu, S. Guo, Y. Liu, C. Cheang, and T. Kong IRASim: a fine-grained world model for robot manipulation. External Links: 2406.14540, Link

AI Usage Statement

In this work, we used generative AI tools for grammar and language polishing. They were not involved in research ideation, methodology, analysis, or substantive writing. The authors take full responsibility for all contents of the paper.

Appendix A Details of Causal Beams

A.1 Formal Details of Causal Beam Construction

We consider a recursive structural causal model M=(U,V,\mathcal{F}), where U denotes the exogenous variables, V the endogenous variables, and \mathcal{F} the set of structural mechanisms. Each non-root endogenous variable V_{i} is determined by V_{i}=f_{i}(\mathrm{PA}_{i},U), where \mathrm{PA}_{i}\subseteq V denotes the endogenous parents of V_{i}. A context U=u determines the actual value V_{i}(u) of every endogenous variable. For a variable set Z, we write Z_{W\leftarrow w}(u) for its value in the original model under the intervention W\leftarrow w and context u.

Definition A.1 (Causal Beam [Pearl, 2009]).

Given a recursive structural causal model M=(U,V,\mathcal{F}) and a context u, a causal beam M_{u}=(u,V,\mathcal{F}^{u}) is constructed by projecting the structural mechanisms as follows.

For each projected non-root mechanism V_{i}=f_{i}(\mathrm{PA}_{i},U), partition its parent set as \mathrm{PA}_{i}=C_{i}\mathbin{\dot{\cup}}N_{i}, where C_{i} denotes the retained parents and N_{i} the remaining parents. The actual values of the retained parents must sustain the actual output:

f_{i}(C_{i}(u),n_{i},u)=V_{i}(u),\qquad\forall n_{i}\in\operatorname{Val}(N_{i}).

Next, select W_{i}\subseteq N_{i} and an assignment w_{i}. Let N_{i,W_{i}\leftarrow w_{i}}(u) denote the values attained by the remaining parents N_{i} in the original model under the intervention W_{i}\leftarrow w_{i} and context u. The projected mechanism is

f_{i}^{u}(c_{i})=f_{i}\!\left(c_{i},N_{i,W_{i}\leftarrow w_{i}}(u),u\right).

The projected mechanism must remain nontrivially responsive to the retained inputs: there exists c_{i}^{\prime}\in\operatorname{Val}(C_{i}) such that f_{i}^{u}(c_{i}^{\prime})\neq V_{i}(u). Thus, the projection cannot simply replace the original mechanism with a constant function that always returns the actual output.

Selections across different mechanisms must additionally be compatible. In particular, N_{i}\cap C_{j}=\varnothing for i\neq j, and whenever intervention assignments overlap across mechanisms, they must assign the same value to every shared intervened variable. The compatible projected mechanisms, together with the retained assignment rules for root variables, constitute the causal beam M_{u}=(u,V,\mathcal{F}^{u}).

The sustaining condition requires the actual values C_{i}(u) to preserve the actual mechanism output under arbitrary assignments to the remaining parents N_{i}. The responsiveness condition serves a different role: it requires the projected mechanism to preserve a nontrivial dependence on its retained inputs. These two requirements distinguish a causal projection from a representation that merely records the observed output.

A special case is obtained when no non-actual contingency is introduced.

Definition A.2 (Natural Beam [Pearl, 2009]).

A causal beam is called natural when W_{i}=\varnothing for every projected mechanism. In this case, the remaining parents are fixed at their actual values under context u, and the projected mechanism reduces to f_{i}^{u}(c_{i})=f_{i}(c_{i},N_{i}(u),u).

A variable that is fixed in a particular projection is not thereby declared irrelevant in general. The projection is defined relative to the specified model and context, and a different context may admit a different set of retained parents and hence a different projected mechanism.

A.2 An Illustrative Construction of a Causal Beam

We illustrate the construction using a binary structural causal model with three root variables B,G,O and two non-root variables D,Y, with mechanisms D=B\land G and Y=D\land O. The parent sets are therefore \mathrm{PA}_{D}=\{B,G\} and \mathrm{PA}_{Y}=\{D,O\}.

We first consider a context u_{1} with B=1,G=1,O=1, which implies D=1,Y=1. For the mechanism D=B\land G, retaining only B does not sustain the actual output because setting G=0 changes D from 1 to 0. Likewise, retaining only G fails because setting B=0 changes the output. A valid choice is therefore C_{D}=\{B,G\} and N_{D}=\varnothing. The corresponding projection is f_{D}^{u_{1}}(b,g)=b\land g. It preserves the actual output at b=g=1 and remains nontrivially responsive because, for example, f_{D}^{u_{1}}(0,1)=0\neq 1=D(u_{1}).

For Y=D\land O, the same reasoning gives C_{Y}=\{D,O\} and N_{Y}=\varnothing, with f_{Y}^{u_{1}}(d,o)=d\land o. Again, the actual output is preserved, while changing either retained input appropriately can change the projected output. Because both excluded parent sets are empty, the cross-mechanism compatibility condition is satisfied. The resulting natural beam therefore retains the original mechanisms for this context.

Now consider a second context u_{2} with B=1,G=0,O=1, which gives D=0,Y=0. For D=B\land G, choose C_{D}=\{G\} and N_{D}=\{B\}. The sustaining condition holds because f_{D}(b,0)=b\land 0=0 for all b\in\{0,1\}. Using a natural projection fixes B at its actual value B(u_{2})=1, yielding f_{D}^{u_{2}}(g)=f_{D}(1,g)=g. The projection remains nontrivially responsive since f_{D}^{u_{2}}(1)=1\neq 0=D(u_{2}).

For Y=D\land O, choose C_{Y}=\{D\} and N_{Y}=\{O\}. The sustaining condition is satisfied because f_{Y}(0,o)=0\land o=0 for all o\in\{0,1\}. Fixing O at its actual value O(u_{2})=1 gives f_{Y}^{u_{2}}(d)=f_{Y}(d,1)=d, which is also nontrivially responsive because f_{Y}^{u_{2}}(1)=1\neq 0=Y(u_{2}).

The cross-mechanism compatibility conditions are satisfied because N_{D}\cap C_{Y}=\{B\}\cap\{D\}=\varnothing and N_{Y}\cap C_{D}=\{O\}\cap\{G\}=\varnothing. No non-actual intervention assignments are introduced, so there are no additional assignment-consistency constraints to check. The resulting natural beam contains the projected mechanisms D=G and Y=D.

The two contexts therefore yield different valid projections of the same underlying structural causal model. In u_{1}, sustaining the actual successful outputs requires retaining both parents of each mechanism, whereas in u_{2}, the actual zero values allow smaller retained parent sets. The construction is context specific: fixing B and O in the second projection does not imply that these variables are generally irrelevant, only that their actual values can be treated as fixed within that particular projection.

Appendix B Experimental Details

B.1 Datasets and Benchmark Composition

We distinguish the data used to pretrain the planner from the robot demonstrations used for adaptation and the environments used for closed-loop evaluation. Table 5 summarizes their roles.

B.1.1 Something-Something V2

SSv2 contains short videos of everyday human–object interactions [Goyal et al., 2017]. Its 174 labels describe actions and their outcomes using object-independent templates. The official release contains 220,847 videos, divided into 168,913 training videos, 24,777 validation videos, and 27,157 test videos. Test labels are not publicly provided.

Action coverage.

The label space includes object displacement, placement, containment, deformation, and interactions with different observable outcomes. Table 7 summarizes representative distinctions relevant to manipulation. The groups are descriptive summaries, not an alternative official taxonomy. Training retains the original 174-class classification target.

Use in CBST.

Human pretraining optimizes action classification. The visual encoder, temporal module, and representation projection initialize the robot-side planner; robot prediction heads and structural selectors are introduced during adaptation. Human category labels are not treated as robot action labels or as direct supervision for causal parent sets.

For the temporal-order control, sampled frames are shuffled within each clip while preserving the clip label, sampled frame content, and training budget. The no-human-pretraining variant uses the same robot adaptation procedure without SSv2 pretraining. Its visual initialization is specified separately to distinguish absence of human-video pretraining from random initialization.

B.1.2 RoboTwin 2.0

RoboTwin 2.0 provides simulated bimanual manipulation tasks [Chen et al., 2025]. We use the 50-task evaluation set reported for LingBot-VA. The tasks cover object transport, handover, placement, stacking, tool interaction, and articulated-object manipulation. Easy and Hard results are reported separately.

Table 8 lists the complete task set, grouped by the horizon annotations in the LingBot-VA evaluation table [Li et al., 2026]. These annotations describe task groups; they are distinct from the action-chunk length and the simulator episode limit.

Evaluation settings.

Easy and Hard follow the configurations associated with the reported LingBot-VA results. Hard introduces additional variation in object poses and scene configurations. We retain the benchmark-defined success checks. Overall performance averages the task success rates over all 50 tasks; horizon-specific results use only the tasks in the corresponding group.

The robot embodiment, camera configuration, demonstration budget, and episode limits used for CBST are specified in Tables 14 and 15.

B.1.3 LIBERO

LIBERO evaluates knowledge transfer in language-conditioned robot manipulation [Liu et al., 2023]. Its released collections include Spatial, Object, Goal, and LIBERO-100. LIBERO-100 is partitioned into LIBERO-90 and the 10-task LIBERO-Long set, also registered as libero_10.

We evaluate CBST on all four LIBERO suites—Spatial, Object, Goal, and Long—with both WAM backbones. Each suite contains ten tasks. The main text focuses on LIBERO-Long, while Appendix C.2 reports results on all four suites. Table 9 distinguishes these evaluation suites from the broader data collection.

Spatial, Object, and Goal.

Table 10 summarizes the task variants in the three short-horizon suites. LIBERO provides scene and language specifications together with visual observations and robot proprioception. The released demonstrations include workspace and wrist views.

Long-horizon tasks.

Table 11 lists all ten tasks used for CBST evaluation. Task identifiers and success predicates follow the official benchmark; the descriptions below are shortened for readability.

Data and evaluation.

Robot demonstrations are partitioned by trajectory before observation windows are constructed. Planner inputs use observations available up to the current decision time, together with robot proprioception and task information. Simulator state may provide adaptation labels, but is not an input to the learned planner during evaluation.

We retain the official task success predicates. Success rates are averaged over the ten Long tasks. Scores for the complete four-suite benchmark, when quoted from prior work, retain the aggregation reported by that source.

B.1.4 RoboCasa

RoboCasa provides kitchen environments for learning everyday manipulation skills [Nasiriany et al., 2024]. We use the 24-task protocol adopted by Cosmos Policy [Kim et al., 2026]. This evaluation set contains seven task families, listed in Table 12; it excludes kitchen navigation.

Task coverage.

Pick-and-place tasks vary the source and destination fixture. Door and drawer tasks require changing articulated states. Sink, stove, and microwave tasks involve fixture controls, whereas coffee tasks cover mug positioning, retrieval, and button operation.

Reference protocol.

The Cosmos Policy report uses 50 human-teleoperated demonstrations per task. Its evaluation includes five scenes per task, with ten trials per scene and three evaluation seeds, giving 3,600 trials over the 24 tasks. Evaluation uses unseen object instances, and two of the five scenes introduce styles absent from training.

These quantities describe the published reference protocol. CBST’s adaptation data and rollout counts are specified separately below. Demonstration budgets remain explicit in the result tables because the published baselines use different amounts of data.

B.2 Baseline Details

We include two types of comparison methods: the WAM backbones to which CBST is attached, and external robot manipulation methods reported on the same benchmarks. Only LingBot-VA and Cosmos Policy (Predict2.5) serve as CBST backbones.

LingBot-VA.

LingBot-VA jointly generates visual futures and robot actions using an autoregressive video–action model [Li et al., 2026]. We use it as the underlying WAM for RoboTwin and LIBERO-Long.

Cosmos Policy.

Cosmos Policy adapts a pretrained video model to generate actions together with future-state and value predictions [Kim et al., 2026]. CBST uses the Predict2.5 configuration. The earlier Predict2 configuration is retained as an additional published reference, with its results identified separately [Kim and Gu, 2026].

X-VLA.

X-VLA uses a flow-based action model and soft prompts to accommodate heterogeneous embodiments and data sources [Zheng et al., 2025].

\pi_{0} and \pi_{0.5}.

\pi_{0} combines pretrained vision–language representations with flow-based action generation [Black et al., 2026]. \pi_{0.5} extends this line with heterogeneous training sources and semantic supervision for broader generalization [Intelligence et al., 2025].

Motus.

Motus combines understanding, video generation, and action modeling within a unified latent-action framework [Bi et al., 2025].

OpenVLA-OFT.

OpenVLA-OFT applies an optimized adaptation recipe to OpenVLA, including parallel action decoding, action chunking, and continuous action prediction [Kim et al., 2025b].

GR00T-N1.5.

GR00T-N1.5 combines vision–language encoding with a diffusion action module and updates the model and training recipe of GR00T-N1 [NVIDIA et al., 2025].

Video Policy.

Video Policy couples video generation with action generation, using visual prediction to support robot policy learning [Liang et al., 2025].

Sources of reported results.

RoboTwin comparisons follow the LingBot-VA report. LIBERO comparisons combine the published LingBot-VA and Cosmos Policy tables. RoboCasa comparisons follow Cosmos Policy, with Predict2.5 results obtained from the official Cosmos Cookbook.

We preserve the reported model version, demonstration budget, evaluation setting, and uncertainty notation. When two sources report different results for the same method, the corresponding source is identified explicitly; entries from different sources are not combined into a single synthetic result.

B.3 Implementation Details

Visual-temporal encoder.

The planner uses a ResNet18 visual encoder [He et al., 2015] and a GRU temporal module [Cho et al., 2014]. Available camera views share the visual encoder, and their features are averaged at each sampled timestep. The resulting feature is combined with robot proprioception and task features before temporal aggregation. Observation histories combine sparse earlier observations with denser recent observations.

Human pretraining and robot adaptation.

Human pretraining uses action-category classification. The pretrained encoder and representation projection initialize the robot planner, while robot-specific semantic heads and parent-set selectors are newly initialized.

During robot adaptation, the final visual stage, temporal module, representation projection, semantic heads, and selectors are optimized using the objectives defined in the method section. The earlier visual stages, reference mechanisms, WAM, and native text encoder remain frozen. Trajectory splits are constructed before sampling observation windows.

Semantic and structural supervision.

Robot demonstration replay provides labels for semantic states, interval variables, and realized effects. Unknown labels are masked rather than assigned negative values. Candidate dependencies are instantiated from the task specification and shared manipulation templates.

For structural supervision, parent subsets are evaluated within the declared finite variable domains using frozen reference mechanisms. Multiple acceptable subsets are retained when they satisfy the structural conditions. Examples lacking the required labels or variable domains are excluded from the corresponding structural loss.

Conditioning and execution.

A deterministic renderer preserves the original task instruction and appends predicted task information and selected dependency fields. The WAM’s native text encoder processes the resulting prompt. The WAM retains its original observation encoder, action representation, and generation procedure.

The planner updates the condition from newly available observations before the next action request. Evaluation does not provide the learned planner with future observations or simulator ground-truth semantic states. Any oracle condition is identified separately.

B.4 Evaluation Metrics and Protocols

Task success.

Let s_{k,e}\in\{0,1\} denote the official success indicator for episode e of task k, with N_{k} evaluated episodes. The task success rate is

\mathrm{SR}_{k}=\frac{100}{N_{k}}\sum_{e=1}^{N_{k}}s_{k,e}.

The aggregate score over K tasks is

\mathrm{SR}_{\mathrm{avg}}=\frac{1}{K}\sum_{k=1}^{K}\mathrm{SR}_{k}.

When every task has the same number of episodes, this equals the success rate pooled over all episodes.

RoboTwin Easy and Hard are aggregated separately. LIBERO-Long averages over ten tasks, and the RoboCasa protocol averages over 24 tasks. Published baseline values retain the aggregation and uncertainty conventions of their respective sources.

Evaluation configurations.

Table 15 records the protocol for each CBST evaluation. Action-chunk length, executed actions per request, and episode limits are specified separately because they determine different aspects of closed-loop execution.

Training and evaluation randomness.

Training seeds and evaluation seeds are recorded separately. Repeated rollout seeds assess variation in evaluation, whereas independently trained planners assess variation in learning. The meaning of any reported error bar is specified in the corresponding table or figure caption.

Appendix C Additional Simulation Results

This section expands the benchmark comparisons in Section 5.2. Published values retain their original precision and aggregation.

C.1 RoboTwin 2.0

Table 16 reports the horizon-grouped results [Li et al., 2026]. Easy and Hard results remain separate. Source-reported aggregate values are retained without recomputing them from the rounded per-task entries.

C.2 LIBERO

Table 17 combines the four-suite comparisons reported by LingBot-VA and Cosmos Policy, together with the official Predict2.5 results. Identical entries are consolidated, while source-dependent differences are preserved as separate rows. In particular, the reported \pi_{0} averages and UniVLA results differ between the two papers.CBST results are reported for Spatial, Object, Goal, and Long with both WAM backbones. For CBST, the Average column is the arithmetic mean of the four suite success rates; source-reported baseline averages are preserved.

C.3 RoboCasa

Table 18 includes all methods in the RoboCasa comparison reported by Cosmos Policy, together with its Predict2.5 update. Training demonstrations per task are retained because the methods use different data budgets. The CBST entry reports the robot data used by the underlying policy and the additional planner, with overlapping demonstrations counted only once.

C.4 Additional Qualitative Examples

Figures 7 provide three additional simulation task sequences. Each example contains temporally ordered frames from a recorded execution. Task names and recording sources are specified in the captions. Annotations describe observable object interactions and state changes; model predictions are included only when corresponding outputs are available.

Appendix D Additional Ablation Studies

This section specifies the ablation variants and extends the analysis in Section 5.3. We distinguish changes to pretraining, robot adaptation objectives, and inference-time conditioning. This distinction identifies which comparisons require independently trained planners and which can be evaluated using a shared checkpoint.

D.1 Variant Definitions

Common configuration.

Unless explicitly changed, variants use the same robot demonstration split, observation inputs, candidate variable domains, reference mechanisms, renderer, and frozen WAM checkpoint. Retrained variants use the same robot adaptation budget and checkpoint-selection rule. Inference-only variants share the full CBST checkpoint and differ solely in the information rendered for the WAM.

Without human-video pretraining.

This variant removes the human-video classification stage. The visual encoder retains the common initialization specified in Appendix B.3, and components normally initialized through human pretraining use their corresponding untrained initialization. Robot heads and selectors follow the same initialization procedure as full CBST. The comparison holds the robot adaptation budget fixed, rather than equating the total compute spent across human pretraining and adaptation.

Shuffled human-video pretraining.

We randomly permute the sampled frame order within each human clip while keeping its original action label. The sampled images, class distribution, architecture, optimization settings, and number of pretraining updates are otherwise unchanged. Robot adaptation subsequently uses correctly ordered robot observations. This control evaluates the contribution of temporal ordering during human pretraining.

State-only conditioning.

Using the full CBST checkpoint, we retain the original task instruction, admissible goals, and predicted semantic-state fields. Fields describing the selected parent dependencies and associated interval-interaction variables are omitted from the rendered condition. The underlying state predictions remain unchanged, so this comparison isolates the information presented to the WAM rather than differences in state-estimation training.

Full-structure conditioning.

For every mechanism, we replace the selected parent set with the complete candidate parent set, C_{i,t}=\mathrm{PA}_{i}. Parent values are obtained from the same predictions as in CBST. Variable naming, field ordering, update frequency, and text encoding remain unchanged. This variant evaluates whether selecting a context-specific subset is useful beyond supplying all available structural fields.

Random selection.

For mechanism i at time t, let k_{i,t} be the number of parents retained by CBST. We sample a subset uniformly from the candidate subsets of \mathrm{PA}_{i} having cardinality k_{i,t}. The sampled subset uses the same predicted variable values and renderer as CBST. Sampling does not use ground-truth task outcomes or reference-validity labels. Random-selection seeds are fixed before evaluation and reported separately from training and rollout seeds.

Without mechanism consistency.

We remove \mathcal{L}_{\mathrm{mech}} from robot adaptation by setting \lambda_{m}=0. Semantic-state supervision, interval-variable supervision, and beam supervision remain unchanged. The reference mechanisms are still used to construct acceptable parent sets; the ablation therefore removes the mechanism-consistency training signal without removing the structural supervision source.

D.2 Results across Backbones and Benchmarks

Table 20 extends the principal ablations to WAM backbones and associated benchmarks. RoboTwin Easy and Hard are reported separately. Published backbone scores are included as references and are not treated as paired measurements from the ablation runs.

D.3 Temporal Architecture

We compare GRU, Transformer, temporal convolutional network (TCN), and MLP temporal modules while retaining the same visual encoder, observation samples, robot supervision, and WAM interface. The MLP receives the sampled frame features in their fixed temporal order through concatenation; the other modules process the corresponding feature sequence. Architecture-specific widths and depths, parameter counts, and optimization settings are reported with the results.

The comparison uses the same data and training schedule. Parameter counts are reported explicitly rather than assuming that the architectures have identical capacity.

D.4 Training and Validation Curves

We report human-pretraining and robot-adaptation curves separately. Human-pretraining plots include classification loss and validation accuracy for correctly ordered and shuffled clips. Robot-adaptation plots separate semantic-state, interval-variable, mechanism-consistency, and structural-selection objectives, together with the corresponding held-out diagnostics. Curves use the original logged values. Training steps and epochs are identified explicitly, and resumed training segments retain their boundaries. Duplicate logging streams are not counted as additional optimization steps. Model selection follows the rule specified in Appendix B.3.

D.5 Sensitivity to Observation History

We vary observation history along two axes. First, we change the number of sampled frames while holding the lookback duration fixed, testing sampling density. Second, we change the lookback duration while holding the number of sampled frames fixed, testing temporal coverage. The current observation is retained in every setting. Other planner and WAM settings remain unchanged. Each sampling configuration is used consistently during adaptation and evaluation. The tested values and the default configuration are specified in Table 22.

D.6 Sensitivity to Structural Compactness

We vary the size-regularization coefficient \lambda_{\mathrm{size}} while retaining the same semantic supervision, reference mechanisms, and acceptable-set construction. A separate planner is trained for each coefficient using the same adaptation budget. Setting \lambda_{\mathrm{size}}=0 removes the explicit compactness preference while preserving the structural objectives. For each setting, we report control success, the mean retained-parent fraction, and the number of text tokens delivered to the WAM. Token counts are measured using each backbone’s native tokenizer. Structural compactness and prompt length are reported separately because equal parent counts need not produce equal token counts.

D.7 Computational Cost

We measure the cost of the additional planner and conditioning path separately from WAM generation. Measurements use a fixed accelerator, input specification, batch size, and action-generation configuration for each backbone. Warm-up iterations are excluded, and accelerator operations are synchronized at the declared timing boundaries.

Planner time includes visual-temporal encoding, semantic prediction, and parent-set selection. Conditioning time separates text rendering from native text encoding. End-to-end request latency is measured directly rather than inferred by summing components that may overlap. Simulator stepping and video recording are excluded from policy-request timing.

Appendix E Real-World Experimental Details

E.1 Platform and Task Formulation

The platform consists of two AgileX manipulators with parallel-jaw grippers operating over a shared tabletop. Each task is specified by a manipulated object and a fixed target region or receptacle. The controller must acquire the object, transport it, and establish the required final placement. When a single arm cannot reach both the source and destination, the arms coordinate through their overlapping reachable workspace. No particular handover trajectory is prescribed by the task. The supplied videos show a fixed external recording view.

E.2 Task Definitions

Table 23 describes the ten task entries. All names specify placement outcomes rather than a required handover.Target locations are fixed within a task configuration; any changes across configurations must be documented separately from variation in initial object pose. The detailed results in Table 24.

E.3 Baseline and Evaluation Protocol

The comparison pairs Cosmos Policy with a Predict2.5 backbone and its CBST-augmented counterpart. The intended controlled comparison holds the task-adapted low-level policy checkpoint, robot observations, control settings, and evaluation conditions fixed, while adding CBST’s contextual structural conditioning. The official Cosmos Policy ALOHA experiments provide a related tabletop-manipulation reference [Kim et al., 2026]; those published real-world results use Predict2, and the official Cookbook states that Predict2.5 was not trained on ALOHA [Kim and Gu, 2026]. Those results are therefore not substituted for evaluation on our AgileX tasks.

Each method is evaluated for 20 trials on each of ten tasks. This specifies the number of executions, not the number of distinct layouts or random seeds. Initial object poses, target geometry, and the mapping of initializations between methods must be documented to establish comparability. For the final protocol, the same initialization set should be used for both methods, with scene resets between executions and balanced method order.

Success criteria.

Full-task success requires the designated object to occupy its specified target region or receptacle, satisfy the task-specific placement requirement, and remain stable after gripper release within the evaluation time limit. Handover is neither an additional task objective nor an independently rewarded outcome. Grasping or transferring an object without completing its final placement does not constitute success. The handling of autonomous retries, intermediate drops, and human interventions must follow the same documented rule for both methods.

Metrics.

Let S_{m,k} denote the number of successful trials for method m on task k. With 20 trials per task, the task success rate and ten-task average are

\mathrm{SR}_{m,k}=100\frac{S_{m,k}}{20},\qquad\overline{\mathrm{SR}}_{m}=\frac{1}{10}\sum_{k=1}^{10}\mathrm{SR}_{m,k}=100\frac{\sum_{k=1}^{10}S_{m,k}}{200}.

Single-task rates therefore change in increments of 5 percentage points, while the ten-task average changes in increments of 0.5 percentage points. Improvements are reported as percentage-point differences. A mean computed over the three visualized tasks is explicitly labeled as a subset mean and is not used as the ten-task result. Stage-level annotations, if available, distinguish acquisition, transport, handover where applicable, and final placement; they are diagnostic and do not replace the full-task metric.

E.4 Qualitative Executions

The three supplied recordings are identified by the authors as CBST executions. Each visualization contains six original video frames selected around visible manipulation events. In the cup-placement sequence shown in the main text, the initiating arm lifts the cup and transfers it to the receiving arm for final tabletop placement. Figure 8 shows plate transport into a rack and remote-control placement into a container. In the plate sequence, grasp transfer enables transport between arm workspaces; in the remote sequence, one arm completes the placement while the other remains inactive. These examples illustrate execution strategies and do not measure their frequency across the ten-task suite.

The figure labels describe visible events rather than predicted states or causal dependencies. All selected frames preserve the source image content. The encoded timestamps and frame rate are used only to identify frames, not to infer control frequency or execution latency. Selected successful videos provide qualitative examples and are not used to estimate task success rates.