EWAM: Tiefenweise Spezialisierung in einem vereinheitlichten Embodied Model

Abstract

Vision-language-action (VLA) policies emphasize semantic understanding, whereas world-action models (WAMs) learn predictive representations of environment dynamics. Systems that expose a policy to both sources often still concentrate action computation on a single expert. We present EWAM, an action-centric unified embodied model whose asymmetric joint attention lets action tokens read semantic, current-visual, predicted-future, and action information at every layer while the perceptual experts retain their distinct roles. Without layer-wise supervision, EWAM develops an emergent depth-wise specialization: action queries attend mainly to vision-language features in shallow layers, to predicted future frames in intermediate layers, and to action tokens themselves in deep layers. This handoff replicates across tasks and is stable across denoising steps. Checkpoint tracking and causal interventions show that it is learned and that action generation depends on it. EWAM is pretrained in two separate regimes, one on cross-embodiment robot trajectories and one on human egocentric video. In simulation and real-robot experiments, it surpasses existing VLA, WAM, and hybrid baselines. Human egocentric data improve both cross-embodiment transfer and real-robot robustness, and subtask-phase supervision improves long-horizon completion. Together, these results suggest that unified embodied learning can induce an ordered internal progression from semantic understanding, through visual foresight, to action formation.

Project page: https://wanghao00pro.github.io/EWAM-project

1 Introduction

Kenneth Craik’s classic account of internal models proposed that an intelligent organism carries a “small-scale model” of external reality, allowing it to “react to future situations before they arise” [14]. This view casts purposeful action as an ordered computation: understanding the present situation, anticipating how it may evolve, and then determining what to do. Generalist robot policies face an analogous challenge. They must connect at least three forms of computation: understanding what an instruction and the current scene imply, anticipating how the physical world may evolve, and producing precise actions.

Two dominant paradigms emphasize different parts of this pipeline. Vision-language-action (VLA) models inherit rich semantic representations from vision-language models, but their static pretraining provides limited explicit supervision for physical dynamics [7, 17, 76, 27, 5, 4]. World-action models (WAMs), in contrast, learn from temporal visual data and can acquire useful motion priors [64, 29, 67], yet predicting plausible futures does not by itself guarantee semantic reasoning or fine-grained control [74].

Prior attempts to obtain both capabilities generally follow one of two paths. The first remains centered on a single paradigm, adding auxiliary outputs while leaving action generation predominantly dependent on either vision-language or video features [57, 69, 47]. The second attaches an independently operating world model or understanding module to a policy [41, 34]. Such composition adds capabilities, but does not necessarily induce structured cooperation inside the action policy. Even a multi-expert model [3] can leave action computation concentrated on one expert. This raises a more fundamental question than how to add another module: when semantic understanding, visual prediction, and action generation are truly unified, do they continue to compete throughout the network, or does the model develop an ordered progression reminiscent of purposeful human action—first understanding the task, then imagining what should happen next, and finally committing to an action?

Figure 1 makes this distinction concrete by visualizing where action queries retrieve information across representative paradigms. In \pi_{0.5} [4], action attention is consistently concentrated on VLM tokens, reflecting a VLA-style computation in which semantic representations are the primary source for control (Figure 1(a)). Fast-WAM [67] exhibits the complementary pattern: action queries attend predominantly to video tokens, consistent with a WAM whose control computation is centered on learned visual dynamics (Figure 1(b)). Motus [3] integrates multiple expert modules through an externally attached frozen VLM, yet its action attention remains dominated by the understanding expert (Figure 1(c)). This observation suggests that merely stacking experts may still yield a single-expert bias resembling the VLA paradigm, rather than inducing an internal division of labor.

To determine whether a unified policy can move beyond this single-expert bias, we develop EWAM, a unified embodied model containing a vision-language (VL) expert, a video expert, and an action expert. At every layer, action queries can attend to semantic tokens, predicted visual tokens, and action tokens. An asymmetric mask prevents video and vision-language queries from reading action tokens or each other, making the action stream the integration site while preserving the distinct predictive and semantic roles of the two perceptual experts. Video and action outputs are trained jointly with flow-matching objectives.

Despite receiving no explicit layer-wise supervision, EWAM develops a consistent internal progression (Figure 1(d)). In shallow layers, action tokens attend primarily to vision-language tokens, including both language and current-image content. In intermediate layers, attention to current-image tokens declines while attention to predicted future-frame tokens rises. In deep layers, action self-attention becomes dominant as the policy consolidates the information into a control sequence. We refer to this depth-dependent handoff as emergent depth-wise specialization: an ordered progression from semantic understanding, through visual foresight, to action formation.

We validate this claim along four dimensions: observational evidence, developmental dynamics, causal interventions, and policy performance. Our observational analysis confirms that the hierarchical handoff is stable across tasks, while tracking training checkpoints shows that this internal organization emerges during training. We then examine depth-specific functional dependencies on vision-language, observed-frame, and predicted-future features through layer-targeted training and fixed-checkpoint interventions. Same-task counterfactual future-feature injection across five tasks further tests whether actions depend on the specific content of predicted futures. Finally, we evaluate the resulting policy in simulation and real-world robot settings. We study two separate pretraining regimes, one on cross-embodiment robot trajectories and the other on human egocentric video. Initialized from cross-embodiment robot pretraining, EWAM achieves 72.1% on RoboTwin clean-to-random (C2R) evaluation and a 92.9% average in the in-domain setting. It also obtains 98.8% on LIBERO and demonstrates consistent performance across multiple real-world robot embodiments. Human pretraining raises success on a held-out robot embodiment from 40.1% to 66.9%, and egocentric co-training raises real-robot success from 10% to 60% under scene variations. Subtask-phase supervision during post-training further improves long-horizon completion in both simulated and real-world tasks.

Our contributions are:

• We develop EWAM, an action-centric unified embodied model that exposes semantic, predictive, and action representations to a common action stream through asymmetric joint attention while preserving the provenance of the perceptual experts.
• We identify and characterize emergent depth-wise specialization: a semantic-to-foresight-to-action handoff that replicates across diverse tasks and remains stable across denoising steps. Checkpoint tracking, layer-targeted interventions, and counterfactual future-feature injection show that it is learned and functionally consequential.
• We show that EWAM is competitive across complementary evaluation protocols, reaching 72.1% on RoboTwin C2R, a 92.9% in-domain average, and 98.8% on LIBERO, with consistent performance across three real-robot embodiments. Human egocentric data improve cross-embodiment transfer via pretraining and robustness under scene variations via co-training. Subtask-phase supervision further improves long-horizon completion.

2 Related Work

Vision-language-action models.

VLA policies adapt vision-language representations to robot control, providing a direct route for transferring semantic and web-scale knowledge into action prediction. RT-1 [7] demonstrated large-scale tokenized action prediction, while PaLM-E [17] connected embodied observations to a multimodal language model. RT-2 [76] casts actions as language tokens and co-trains robot trajectories with vision-language data. Octo [18] learns a cross-embodiment generalist policy from heterogeneous robot datasets, while OpenVLA [27] scales an open VLA built from pretrained vision and language components. The \pi family [5] couples a pretrained VLM to a flow-matching action expert, extending continuous generative policy learning beyond the earlier Diffusion Policy [13], and \pi_{0.5} [4] further combines multi-robot data, semantic subtask prediction, and web data for open-world operation. These systems establish the strength of semantic pretraining for instruction following and generalization. However, when predictive visual dynamics are introduced alongside VLM features, it remains unclear how the action stream coordinates these complementary sources across network depth.

World-action models.

WAMs instead build robot policies on predictive video representations and jointly model future observations and actions. DreamZero [64] adapts a pretrained video diffusion backbone into a large joint video-action policy, demonstrating zero-shot transfer across embodiments. LingBot-VA [29] learns causal frame prediction and policy execution in a shared latent space with closed-loop rollout. Fast-WAM [67] separates video co-training from test-time imagination, showing that predictive supervision can benefit action learning even when future frames are not generated at inference. A recent controlled robustness study [74] reports complementary strengths between VLA and WAM policies, with their relative performance shaped by architecture and training-data diversity. Our work does not seek to determine which paradigm is universally superior; instead, it examines how their semantic and predictive representations are internally coordinated when exposed to one action stream.

Hybrid VLA–WAM systems.

Recent systems begin to combine the semantic strengths of VLAs with the visual foresight of world models, but most retain a modular pipeline. At inference, \pi_{0.7} [41] uses a high-level semantic policy to produce subtask instructions and a separately trained world model based on BAGEL [16] to generate subgoal images; these external outputs then condition its VLA action policy. Cortex 2.0 [1] similarly augments an existing VLA with a distinct world-model planning module: the world model rolls out candidate futures, a reward module selects one, and the selected trajectory is passed to the downstream controller. Motus [3] brings video generation and action prediction into a Mixture-of-Transformers, but introduces semantic understanding through an externally attached frozen VLM. These approaches establish the value of combining semantics, foresight, and control, while preserving explicit module boundaries or one-way intermediate interfaces. In contrast, our model couples semantic, future-visual, and action representations within one layer-aligned architecture: the action stream can interact with all three sources throughout depth, allowing the understanding–foresight–action division of labor to emerge internally rather than being prescribed by a serial pipeline.

Learning manipulation from human egocentric video.

Human video provides manipulation supervision beyond robot demonstrations. Corpora such as Ego4D [21], EPIC-Kitchens [15], and Something-Something v2 [20] support several learning routes. R3M [39] and VC-1 [38] learn visual representations without action supervision, transferring knowledge through the perceptual encoder. GR-1 [53] and LAPA [65] derive surrogate signals through future prediction and latent action learning, respectively, without directly specifying metric hand motion. Hand-centric approaches provide explicit motion targets: VITRA [30] constructs hand-centric vision-language-action episodes, EgoDex [23] pairs video with tracked hand motion, and EgoMimic [26] co-trains human and robot demonstrations captured through matched hardware. EWAM places three heterogeneous human corpora and downstream robot data in a shared camera-relative end-effector action space. Human pretraining uses the same joint video–action objective and layer-aligned experts as robot pretraining, supplying shared perceptual and predictive representations for downstream robot learning.

3 Preliminaries and Problem Formulation

Let o_{t} denote the current image observation, \ell a language instruction, v_{t+1:t+H} future visual observations, and a_{t:t+K-1} an action chunk. The policy models future-video and action trajectories conditioned on (o_{t},\ell). Both output spaces are trained with flow matching [31]. For a clean target x and Gaussian noise \epsilon, we construct

x_{\tau}=(1-\tau)x+\tau\epsilon,\qquad u^{\star}(x_{\tau},\tau)=\epsilon-x.

We regress a velocity field from the noisy sample to u^{\star}. Video and action use independently sampled noise levels.

4 A Unified Embodied Model

4.1 Architecture Overview

As illustrated in Figure 2, given an image observation o_{t} and a language instruction \ell, our model jointly predicts a future visual trajectory v_{t+1:t+H} and an action chunk a_{t:t+K-1}. It contains three layer-aligned experts. A vision-language expert encodes the instruction and current image into semantic tokens h^{L}; a video expert processes the condition-frame latent together with noisy future-video latents h^{V}; and an action expert embeds the noisy action chunk into h^{A}. The video expert is initialized from Wan2.2-TI2V-5B [48]. The vision-language expert is initialized from Qwen3-VL-2B [2]. Unlike a pipeline in which a separately generated future is passed to a downstream VLA, the three streams are executed in one forward process and interact at every network depth through the action stream.

Training proceeds in two stages. During large-scale pretraining, we optimize future prediction and action generation over cross-embodiment trajectories. During task-specific post-training, we retain these objectives and introduce subtask-phase supervision. Long-horizon demonstrations are partitioned into semantic phases; the same partition supplies a textual label of the active phase and a scalar phase index. The post-trained model therefore produces

(\hat{v}_{t+1:t+H},\hat{a}_{t:t+K-1},\hat{p}_{t},\hat{y}_{t})=f_{\theta}(o_{t},\ell),

where \hat{y}_{t} is the currently active subtask and \hat{p}_{t} is the normalized index of that phase in the subtask sequence.

4.2 Action-Centric Asymmetric Attention

Unrestricted trimodal attention can blur the pretrained roles of semantic and predictive representations. We instead make action tokens the sole integration interface. For video (V), action (A), and vision-language (L) tokens, the attention mask permits

A\leftarrow\{V,A,L\},\qquad V\leftarrow V,\qquad L\leftarrow L.

Thus, action queries can retrieve semantic and predictive evidence, while the video and vision-language streams retain modality-specific computation and cannot read one another or the action stream. In implementation, the three query-side computations in Figure 2 are evaluated through a single concatenated masked-attention operation, with each stream retaining its own normalization and projection parameters.

The instruction and current image are first processed by Qwen3-VL [2], yielding 28 hidden states. To expose these semantic features across the 30 unified blocks, we align them to the unified depth using the fixed mapping detailed in Appendix B. Let q_{l}^{L}\in\mathbb{R}^{N_{L}\times 2{,}048} denote the representation aligned to block l, and let z_{l-1}^{L} denote the VL state accumulated through the preceding unified block. At unified block l, we form

x_{l}^{L}=\begin{cases}q_{l}^{L},&l=0,\\[2.0pt]
\frac{1}{2}\left(q_{l}^{L}+z_{l-1}^{L}\right),&l>0.\end{cases}

Thus, x_{l}^{L} combines the layer-matched Qwen3-VL feature with the VL updates accumulated through unified depth; its VL-only attention output is projected back to 2,048 dimensions and added residually to give z_{l}^{L}. The video stream requires no such alignment, since the 30 unified blocks follow the native depth of the Wan2.2 video expert. Both streams expose their keys and values to action queries, making the action stream the sole cross-modal reader; the action objective shapes semantic and predictive representations only through these exposed keys and values. This separation also makes action-query attention a well-defined measure of which source is retrieved.

4.3 Joint Video–Action Flow Matching

Let x_{v} and x_{a} denote clean future-video and action targets. We independently sample noise levels for the two modalities, construct their noisy interpolants, and regress the corresponding flow velocities. The observed condition-frame latent is kept fixed, while future-video and action latents are updated jointly. The base objective is

\mathcal{L}_{\mathrm{base}}=\lambda_{v}\lVert\hat{u}_{v}-u^{\star}_{v}\rVert_{2}^{2}+\lambda_{a}\lVert\hat{u}_{a}-u^{\star}_{a}\rVert_{2}^{2},

where \hat{u}_{v} and \hat{u}_{a} are the predicted video and action velocities, u^{\star}_{v} and u^{\star}_{a} are the corresponding target flow velocities, and \lambda_{v} and \lambda_{a} are loss weights. At inference, we jointly integrate the predicted video and action velocity fields. The two trajectories are therefore denoised concurrently within the same forward process, allowing evolving future-video features to participate directly in action computation rather than serving as a separately generated visual prompt.

4.4 Subtask-Aware Post-Training

Subtask-phase partition.

Long-horizon demonstrations are divided into N semantically meaningful phases using annotated transition checkpoints. If the current observation lies in phase k_{t}\in\{0,\ldots,N-1\}, we associate a short textual description y_{t}=(y_{t,1},\ldots,y_{t,M}) of the active subtask and define a normalized phase index

p_{t}=\frac{k_{t}}{\max(N-1,1)}.

The scalar p_{t} is the position of the current phase in this partition, not elapsed trajectory time. It is therefore derived from the same subtask annotation rather than an independently defined progress target.

Subtask prediction.

The vision-language stream receives only the high-level instruction, not the subtask text. After the joint forward pass, a lightweight autoregressive decoder predicts y_{t} from a concatenated cross-attention memory, obtained by linearly projecting the final action representation h_{t}^{A} and the aligned VL tokens h_{t}^{L} with W_{A} and W_{L}:

\mathrm{Mem}_{t}=\bigl[W_{A}\,h_{t}^{A};\;W_{L}\,h_{t}^{L}\bigr].

The decoder reuses the Qwen3-VL token embedding and language-model output head, and attends to \mathrm{Mem}_{t} rather than to an independently executed planner:

p_{\theta}(y_{t}\mid\mathrm{Mem}_{t})=\prod_{j=1}^{M}p_{\theta}(y_{t,j}\mid y_{t,<j},\,\mathrm{Mem}_{t}),

where p_{\theta} is the decoder distribution with parameters \theta and y_{t,<j} denotes the preceding tokens, with token-level objective

\mathcal{L}_{\mathrm{sub}}=-\sum_{j=1}^{M}m_{t,j}\log p_{\theta}(y_{t,j}\mid y_{t,<j},\,\mathrm{Mem}_{t}),

where m_{t,j} masks padding. We supervise the currently active subtask throughout each annotated phase.

Phase-index head.

We mean-pool the final action representation and regress p_{t} as

\bar{h}_{t}^{A}=\operatorname{MeanPool}(h_{t}^{A}),\qquad\hat{p}_{t}=W_{2}\,\operatorname{SiLU}(W_{1}\bar{h}_{t}^{A}),

where W_{1} and W_{2} are the weights of a two-layer MLP head and \hat{p}_{t} is the predicted phase index, with \mathcal{L}_{\mathrm{prog}}=\lVert\hat{p}_{t}-p_{t}\rVert_{2}^{2}.

Post-training objective.

The full objective is

\mathcal{L}_{\mathrm{post}}=\mathcal{L}_{\mathrm{base}}+\lambda_{p}\mathcal{L}_{\mathrm{prog}}+\lambda_{s}\mathcal{L}_{\mathrm{sub}}.

Both terms come from the same phase partition and augment the base video–action objective only during task-specific post-training; no external high-level policy is required.

5 Emergent Depth-Wise Specialization

5.1 Measuring Modality Allocation

We analyze how the information retrieved by action queries changes with model depth. Let \mathcal{Q}_{A} denote the set of action queries, \mathcal{K} the set of all keys visible to them, and \mathcal{K}_{m}\subseteq\mathcal{K} the keys from source m. With \alpha_{qk}^{(l)} the post-softmax attention weight from query q to key k in block l, the allocation to source m is

R_{l}^{m}=\frac{\sum_{q\in\mathcal{Q}_{A}}\sum_{k\in\mathcal{K}_{m}}\alpha_{qk}^{(l)}}{\sum_{q\in\mathcal{Q}_{A}}\sum_{k\in\mathcal{K}}\alpha_{qk}^{(l)}}.

We distinguish five sources: VL-text, VL-image, condition-frame, future-frame, and action tokens. We aggregate over attention heads and action queries, normalize within each layer, and average over samples and denoising steps, using the robot-pretrained in-domain checkpoint of Table 5. This statistic shows where action tokens retrieve information; Section 5.4 tests functional necessity.

5.2 Emergent Specialization Across Depth

Action queries exhibit three overlapping computational regimes rather than a hard-routed pipeline. Unless stated otherwise, every value reported in this section is a macro average over one representative sequence from each of the 50 RoboTwin 2.0 tasks under the randomized protocol, obtained by averaging the five measured denoising steps within a task before averaging across tasks with equal weight, using the robot-pretrained in-domain checkpoint. Appendix C.2 reports the same quantities with 95% task-bootstrap confidence intervals, and Appendix C reports their token-count-normalized counterparts.

Shallow understanding.

In layers 0–9, VL tokens receive 75.4% of action attention, compared with 14.1% for video and 10.6% for action tokens. Action computation therefore begins by retrieving the instruction and current-scene information needed to identify and ground the task.

Intermediate visual foresight.

Across the intermediate stage (layers 10–19), attention becomes more distributed across VL, video, and action tokens, with allocations of 36.6%, 24.3%, and 39.1%, respectively. The visual-source handoff is concentrated in layers 11–15, with a cross-task median raw crossover at layer 13 (IQR 12–13). Figure 3 reveals the structure hidden by the stage-level aggregates: attention to VL-image tokens drops sharply, while attention to predicted future-frame tokens increases (see Appendix C for token-count-normalized results); attention to the observed condition frame remains comparatively weak. Thus, the model retains linguistic guidance from the VL expert while shifting its visual dependence from the current observation toward predictive representations of future scene evolution.

Deep action formation.

In layers 20–29, action self-attention rises to 59.5%, while VL and video attention fall to 25.2% and 15.3%, respectively. What the deep stage retains from the two perceptual sources is selective rather than uniform: averaged over layers 16–29, VL-image tokens hold only 1.3% of action attention against 11.8% for predicted future frames (Table 12). Deep layers therefore primarily refine motor trajectories while preserving concise semantic and predictive context. Together, these regimes constitute a gradual understanding–foresight–action handoff rather than three isolated modules.

Cross-task replication and stability.

We repeat the analysis on one representative sequence from each of the 50 RoboTwin 2.0 tasks, first averaging the five denoising steps within each task and then macro-averaging across tasks. Figure 3(b) shows that the image-to-future visual handoff is not specific to Adjust Bottle: all 50 tasks exhibit the raw handoff, with a stable future–image crossover at median layer 13 (IQR 12–13). Across the complete five-source routing profile, VL attention is largest in the shallow stage for all tasks and action attention is largest in the deep stage for all tasks. After task-specific token-count normalization, the visual handoff remains present in 49 of 50 tasks and its median crossover shifts to layer 14. Attention profiles are also highly stable across denoising steps (mean pairwise correlation 0.998); complete raw curves, normalized curves, and full statistics are reported in Appendix C.2.

Figure 1 further shows that \pi_{0.5}, Fast-WAM, and Motus concentrate action attention on one dominant source, whereas our model changes its primary source with depth.

5.3 Learning the Visual Handoff during Training

We trace action-attention routing across twelve checkpoints spanning 50 to 40,000 optimizer steps of one training run without robot pretraining, to determine whether the depth-wise division of labor is present from the start or emerges through training. All checkpoints belong to the same thirty-two-GPU training run and share the asymmetric mask and initialization regime. At 50 steps, action queries attend to VL-image tokens at every depth, and VL-text receives only 2.9% of shallow attention; no layer-wise division is yet visible. A coarse organization appears within the first few hundred steps. By 150 steps, shallow text attention has risen to 22.6%, image attention concentrates in shallow layers, and raw future-frame attention mass first exceeds VL-image mass at layer 8. Continued training refines this organization until, at 40,000 steps, the three regimes of Section 5 are established: shallow layers read the instruction and current image, and raw future-frame attention mass exceeds VL-image mass at every layer from 13 onward (Figure 4). This refinement is not monotonic: under token-count normalization, shallow image enrichment rebounds to 1.79 at 5,000 steps before deep image enrichment falls to 0.04 at convergence.

The mask determines which sources action queries can access, but not the depth at which each source is used. The layer-specific routing therefore emerges during the training of our model. The trajectory is measured on one representative sequence from one run; Appendix C.1 and Table 13 give full values, including token-normalized measurements.

5.4 Causal Validation through Layer-Targeted Interventions

The attention profiles above are observational: they show where information is routed, but not whether the routed sources affect behavior. We therefore use two complementary tests in Table 1. The left panel reports structural ablations trained independently without pretraining; these test whether the proposed layer-wise access pattern is beneficial when a model can adapt during training. The right panel masks selected keys only at inference time in one fixed cross-embodiment robot-pretrained checkpoint, thereby measuring its learned dependence without allowing compensation through retraining. We evaluate the inference interventions on the full RoboTwin Randomized protocol of 50 tasks and 100 episodes per task, with matched episode seeds.

Training-time structural ablations.

Without pretraining, the asymmetric model reaches 83%, compared with 81% for symmetric full attention. Removing VL access in layers 0–9 or video access in layers 10–19 reduces success to 78%, whereas restricting layers 24–29 to action tokens preserves the 83% score. These results are consistent with early semantic grounding, intermediate visual integration, and late action refinement. Because each variant is trained separately and may adapt to its mask, however, they provide architectural support rather than direct evidence that the pretrained policy uses each source at the corresponding depth.

Fixed-checkpoint depth interventions.

Inference-time masking reveals a sharper stage-specific dependence. Relative to the 92.80% pretrained baseline, removing all video keys in shallow layers 0–9 changes success by only -0.82 points (91.98%), while shallow VL masking costs 16.27 points (76.53%). In contrast, the matched all-video intervention across depth yields 91.98% in layers 0–9, 0.25% in layers 10–19, and 65.82% in layers 20–29. Thus, the policy has little shallow dependence on video, maximal dependence in the intermediate stage, and a smaller but still substantial dependence in the deep stage.

Decomposing the intermediate video pathway identifies which content drives this effect. Masking condition-frame keys alone retains 81.40% success, but masking intermediate future features alone nearly collapses performance, reducing success from 92.80% to 3.55%; masking both sources reduces it further to 0.25%. The dominant intermediate dependence is therefore on predicted-future representations, not on visual tokens indiscriminately. Together with the shallow VL intervention, this depth- and source-specific pattern functionally corroborates the observational handoff.

Counterfactual future-feature injection.

Masking establishes necessity, but a collapse could arise merely because the network expects the future-token pathway to be present. To test whether actions depend on its content, we conduct same-task counterfactual interventions across five tasks. For each task, we cache the future-frame keys and values from a successful donor rollout (seed 100) at layers 10–19 during the first three policy calls. We inject them during the corresponding calls of five target rollouts (seeds 43–47), while preserving each target’s condition-frame, VL, state, and action representations. All 25 target rollouts succeed without injection.

As summarized in Table 2, injection reduces aggregate success from 25/25 to 4/25. All injected rollouts fail on Place Empty Cup and Stack Blocks Two. Only seed 46 succeeds on Click Bell, and only seed 45 succeeds on Grab Roller. On Adjust Bottle, success decreases from 5/5 to 2/5. These results show that sensitivity to counterfactual future representations extends across tasks rather than being specific to bottle adjustment.

Adjust Bottle additionally provides an interpretable view of how the injected representations alter behavior. A bottle whose opening faces left should be adjusted with the right arm, whereas a right-facing opening requires the left arm. Injection causes all five targets to initially follow the donor’s right-arm behavior despite changes in bottle identity, orientation, and position. For the two left-facing targets (seeds 43 and 46), the donor-directed first grasp misses the target position; once injection stops, both policies re-localize the bottle and succeed on a second grasp. For the three right-facing targets (seeds 44, 45, and 47), the transferred right-arm choice conflicts with the required left-arm strategy, and all three episodes fail.

The consistent degradation across five tasks provides cross-task, content-sensitive causal evidence that action generation depends on intermediate future representations. The Adjust Bottle trajectories further show that these representations can steer spatial targeting and arm selection, while target-conditioned control can re-emerge after injection ends.

6 Human Egocentric Learning: Representation and Data Construction

To extend the unified model to human demonstrations, we use two complementary data interfaces, evaluated in Section 7.8. Large-scale human pretraining uses camera-relative wrist-motion targets from existing corpora, whereas G1-D co-training uses task-specific recordings retargeted to the robot interface. The final subsection assesses the retargeting quality of that offline procedure and illustrates robot replay, separately from learned-policy evaluation.

6.1 Human Egocentric Pretraining

The human-pretraining experiments in Section 7.8 use the same architecture and joint video–action objective as robot pretraining. To provide consistent motion targets across the three egocentric corpora, we express wrist displacement in the anchor camera frame and mask unavailable action dimensions.

A camera-relative end-effector action space.

Our starting point is that the egocentric camera is the one reference frame that a human recording and a robot platform both provide. We therefore express manipulation as wrist motion relative to that camera. Let (R_{we},t_{we}) denote the wrist pose in world coordinates at an anchor frame, (R_{we^{\star}},t_{we^{\star}}) its pose at a later frame, and R_{wc} the orientation of the egocentric camera at the anchor. We first form the motion in the wrist’s own frame,

R_{ee^{\star}}=R_{we}^{\top}R_{we^{\star}},\qquad t_{ee^{\star}}=R_{we}^{\top}\bigl(t_{we^{\star}}-t_{we}\bigr),

where R_{ee^{\star}} and t_{ee^{\star}} are the rotation and translation of the wrist from the anchor to the later frame, expressed in the anchor wrist frame; we then map it into the camera frame through R_{ce}=R_{wc}^{\top}R_{we}:

\Delta R_{c}=R_{ce}R_{ee^{\star}}R_{ce}^{\top},\qquad\Delta t_{c}=R_{ce}\,t_{ee^{\star}},\qquad\Delta\omega_{c}=\log_{SO(3)}\bigl(\Delta R_{c}\bigr),

where \Delta R_{c} and \Delta t_{c} are the same wrist motion expressed in the anchor camera frame and \log_{SO(3)} maps \Delta R_{c} to its axis–angle vector \Delta\omega_{c}. Each hand contributes a translation and a rotation vector together with two reserved slots, giving a sixteen-dimensional action

a=\bigl[\,\Delta t_{c}^{L},\,\Delta\omega_{c}^{L},\,0,\,0,\;\Delta t_{c}^{R},\,\Delta\omega_{c}^{R},\,0,\,0\,\bigr],

where superscripts L and R denote the left and right hands, and the two reserved dimensions for each hand are excluded from the loss using the same validity mask as for cross-embodiment padding. All actions in a chunk are referenced to the same anchor pose and the same anchor camera, so the i-th action is the displacement from the anchor to frame i{+}1 rather than a step-to-step increment; the state is the current wrist pose expressed in that camera frame. This construction removes dependence on the world-frame origin and axis convention while preserving metric translation magnitudes; it does not normalize embodiment scale or remove kinematic differences. It gives human wrists and robot end-effectors a common task-space layout. On the robot side, the reserved slots carry gripper commands.

Reducing heterogeneous sources to a common frame.

The three corpora we pretrain on expose camera geometry in different forms, and each requires its own route to the anchor camera pose. VITRA-1M [30] publishes world-to-camera extrinsics, which we invert; EgoDex [23] stores a camera-to-world transform directly; and Xperience [44] provides a SLAM body pose T_{wb} together with a stereo-rectified calibration T_{cb}, from which the camera pose follows as T_{wc}=T_{wb}T_{cb}^{-1}. Once a source is reduced to world-frame wrist poses and an anchor camera pose, all three enter the identical computation in Eq. 14, so a given dimension carries the same physical meaning across the mixture. Per-hand tracking validity, derived from each source’s own confidence or visibility signal, masks that hand’s six dimensions whenever the estimate is unreliable, which keeps invalid reconstructions from contributing gradients. No action normalization is applied at this stage. The coordinate conversion preserves each source’s metric scale, so consistent units and calibration remain necessary.

6.2 Robot-Compatible Demonstrations from Egocentric Capture

Wearable systems such as BifrostUMI [50] combine human motion sensing with instrumented manipulation interfaces to collect demonstrations without the target robot. Video-based pipelines reconstruct hand motion and camera trajectories to obtain metric action supervision [37]. These motions cannot be used directly as robot actions: head motion changes the observed hand trajectory, and human wrists and robot tools have different interaction frames. We construct robot-compatible demonstrations by separating task-space motion and grasp timing from the joint configurations that realize them. Metric wrist tracking and visual hand geometry define the human-side reference; interaction-aware tool alignment and sequence optimization convert it into bimanual robot trajectories. Paired with egocentric images, these references support task-specific human–robot co-training on G1-D, complementing the wrist-motion pretraining in Section 6.1.

Metric hand-motion reconstruction.

Recovering manipulation motion requires separating wrist movement from camera movement while retaining finger articulation. We combine a Lumos Ego headset, which records fisheye RGB, time-of-flight depth, and camera poses from simultaneous localization and mapping (SLAM), with two wrist-mounted Vive Tracker 3.0 devices that measure metric six-degree-of-freedom motion (Figure 5).

HaWoR [70] estimates wrist poses and hand parameters from the egocentric video; MANO [43] decodes the hand parameters into articulated geometry. Camera poses map the visual estimates into the SLAM world frame, where the registered Tracker measurements provide metric motion constraints. This combines measured wrist motion with the finger geometry needed for tool alignment and grasp inference. Appendix H gives the visual tracking and temporal processing details.

Interaction-aware action alignment.

Robot action targets must specify tool motion and when to grasp or release. We infer grasp timing from the relation between the hand and the manipulated object. SAM 3 masks [10], propagated by Lucas–Kanade optical flow [35], provide image-space object geometry. Projected finger-to-mask distances, object occupancy within the pinch region, and hand-aperture changes supply grasp evidence. Distinct temporal conditions for grasp and release convert this evidence into stable binary gripper commands.

EgoInfinity [49] and Do as I Do [40] recover hand–object interactions and map reconstructed motion to robot embodiments. In our pipeline, human-to-gripper alignment [66] must also account for the difference between the anatomical wrist and the tool center point (TCP). We estimate a wrist-local interaction point from the median thumb–index midpoint in valid closed-hand frames. Fixing this offset separates the reference motion from instantaneous finger articulation: the point follows wrist translation and rotation while the fingers open or close.

A shared rigid transform places both hands in the robot workspace using a real-robot reference trajectory, retaining metric scale and the relative geometry of the two reference points. The median thumb–index direction and the orthogonalized wrist-to-interaction direction define the tool’s closing and approach axes, with signs fixed across the episode. These targets refer to the robot’s nominal pinch-center TCP. Together with the grasp schedule, they specify the task-space reference that the retargeter must track. Figure 6 summarizes this construction; Appendix H gives the coordinate definitions.

Kinematically constrained retargeting.

The same tool motion can be realized by different robot joint configurations, and independent framewise inverse kinematics can introduce abrupt joint changes. We therefore optimize the 14 arm joints over the full sequence, balancing tool tracking, reference-posture regularization, and temporal smoothness. The tool targets and grasp schedule remain fixed, so optimization changes how the robot realizes the reference without redefining that reference.

Grasp timing also determines orientation tolerance. Tighter tolerances during closure prioritize the closing and approach directions; looser tolerances when open allow posture adjustment. A real-robot reference regularizes redundant shoulder–elbow and wrist configurations, while temporal penalties control velocity, acceleration, and jerk on the original 30 Hz timeline. Joint and derivative bounds constrain the solve, and workspace and collision checks assess the resulting trajectory. Appendix H details the objective and optimization procedure.

Observation–action supervision.

Each episode pairs the captured human images with robot end-effector trajectories and inferred gripper commands on a common timeline. Forward kinematics (FK) recomputes the end-effector trajectories from the optimized joint reference using the robot data’s tool convention. This pairing supplies the correspondence between visual evolution and motion needed for EWAM’s joint video–action learning. In the co-training setting of Section 7.8.2, human and robot episodes use the same sixteen-dimensional end-effector action interface. All reconstruction and retargeting are performed offline. Figure 7 illustrates demonstration replay, while Sections 6.3 and 7.8.2 evaluate retargeting fidelity and learned-policy performance, respectively.

6.3 Retargeting Quality and Robot Replay

We assess the offline retargeting pipeline on five recorded human demonstration sequences, examining compliance with the specified kinematic checks and positional agreement with the human-derived tool targets. TCP position error is computed between the positions obtained by forward kinematics from the retargeted joint references and the corresponding tool targets. We report the 95th percentile (P95) of this distance separately for the left and right tools. These measurements characterize the offline references rather than measured tracking error during physical execution. As shown in Table 3, all five references pass the constraint and consistency checks defined in Appendix H. Nevertheless, the largest per-sequence P95 errors are 77.01 mm for the left tool and 19.69 mm for the right tool, indicating residual target mismatch despite compliance with the specified checks.

As a qualitative illustration, Figure 7 shows a recorded bimanual pouring demonstration adapted for replay in simulation and on the physical Unitree G1-D robot. The comparison aligns seven visible action phases, from grasping through pouring to placement and release, providing a view of the demonstrated action sequence across human and robot embodiments. The robot sequences are replays of demonstration-derived trajectories. The effect of human demonstrations on learned-policy performance is evaluated separately in Section 7.8.2.

7 Experiments

7.1 Experimental Setup

We evaluate EWAM across simulation, cross-embodiment transfer, and real-world deployment. In simulation, we use RoboTwin 2.0 [12] and LIBERO [32], reporting task success rate as the primary metric. Baselines are grouped into VLA, WAM, and hybrid VLA–WAM paradigms.

RoboTwin experiments follow three distinct evaluation protocols. First, the clean-to-random protocol adapts a single policy on clean demonstrations of all 50 tasks and evaluates it under both clean and randomized conditions. Second, the in-domain protocol adapts and evaluates each method on both clean and randomized demonstrations. Third, the cross-embodiment protocol trains on four robot platforms and evaluates on Aloha-Agilex-2 to isolate transfer from human egocentric pretraining.

The checkpoints used for the C2R and in-domain evaluations are initialized from cross-embodiment robot pretraining. The checkpoints used in the human-pretraining experiments in Section 7.8.1 are obtained from human egocentric pretraining. Appendix G provides details of both data sources. LIBERO evaluation covers its four standard task suites. Real-world evaluation covers a Franka single arm, a Dobot dual-arm platform, and a G1-D dual-arm humanoid. We further evaluate subtask-phase supervision on three instruction-conditioned block-manipulation tasks and a Unitree G1-D Dual-Basket Sorting sequence in Section 7.7 (Appendix E.4). Appendix F reports end-to-end inference latency and the optimizations behind it.

7.2 RoboTwin Clean-to-Random Evaluation

In our primary simulation comparison, a single policy is initialized from a cross-embodiment robot-pretrained checkpoint, adapted on clean demonstrations of all 50 tasks, and then evaluated under domain randomization. No randomized data from the target embodiment are used during downstream adaptation. Table 4 compares representative VLA, WAM, and hybrid VLA–WAM policies under both clean-to-clean (C2C) and clean-to-random (C2R) evaluation [46, 25, 9, 75, 61, 4, 67, 73, 22, 60, 19].

EWAM reaches 82.2% C2C and 72.1% C2R, exceeding GigaBrain-0.7 by 4.2 C2R points and \pi_{0.5} by 26.1. Overall, EWAM achieves the highest C2R and average scores, generalizing well from clean to randomized scenes. Because pretraining data differ across methods, this comparison alone does not isolate the contribution of the attention architecture.

7.3 RoboTwin In-Domain Results

We additionally report the in-domain setting, in which every method, including ours, is trained on both clean and randomized demonstrations and evaluated under each condition. We organize the baselines into VLA, WAM, and hybrid VLA+WAM paradigms. The VLA comparisons cover recent open and large-scale policies [61, 56, 71, 68, 28]. The WAM group comprises policies centered on predictive pixel-space or latent dynamics representations [63, 11, 29, 67]. The hybrid VLA+WAM group comprises systems that combine VLA and WAM components [3, 8, 62].

EWAM achieves the highest average, 92.9%, with 93.0% on clean scenes and 92.8% under randomization. Relative to the strongest baseline in each paradigm, it improves average success by 2.0 points over ACE-Ego-0 among VLAs, 0.7 points over LingBot-VA among WAMs, and 1.4 points over WLA-0 among hybrid methods. Its 0.2-point gap between clean and randomized scores indicates consistent performance across conditions. Together with the clean-to-random results in Section 7.2, these results show that EWAM preserves strong in-domain performance while maintaining randomized performance after clean-only downstream adaptation.

Without pretraining, the same architecture reaches 83.0% under the training-ablation protocol of Table 1; the robot-pretrained model reaches 92.8% under randomization in the in-domain evaluation. The architecture therefore remains effective from scratch while benefiting substantially from cross-embodiment pretraining.

7.4 LIBERO Simulation Results

We additionally evaluate EWAM on the LIBERO manipulation benchmark [32], covering spatial, object-centric, goal-conditioned, and long-horizon task suites. As shown in Table 6, EWAM achieves success rates of 98.6%, 99.8%, 98.6%, and 98.2% on LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, respectively, yielding an overall average of 98.8%. This is the highest overall average among the compared policies.

7.5 Real-World Evaluation

We evaluate seven real-world tasks across a Franka single arm, a Dobot dual-arm platform, and a Unitree G1-D dual-arm humanoid. As shown in Tables 7 and 8, EWAM attains the highest success rate on every task. Relative to \pi_{0.5} [4], it is higher on six of the seven tasks and ties on the Dobot towel task. Comparisons with the additional baselines show the same trend where results are available: EWAM outperforms Motus [3], LingBot-VA [29], Xiaomi Robotics-0 [9], and Fast-WAM [67] on Franka, and outperforms Motus and Fast-WAM [67] across all Dobot and G1-D tasks. These results demonstrate consistent real-world performance across single-arm, dual-arm, and humanoid embodiments and across rearrangement, deformable-object manipulation, and pouring tasks.

7.6 Real-Robot Generalization and Failure Localization

We examine generalization and failure stages on G1-D pour-beans. The task requires grasping both cups, transporting them, aligning the pouring cup over the receiving cup, pouring, and replacing both cups. We vary cup instances, object placement, or background and clutter one factor at a time relative to an in-domain baseline, and score each stage separately. Each condition receives 10 physical trials, for 40 trials in total.

Table 9 reports 8/10 successes in-domain and 20/30 across the held-out conditions (66.7%), a 13.3-point decrease from the concurrent baseline. Cup-instance and placement changes each yield 7/10, while background and clutter changes yield 6/10. Differences between these conditions amount to one trial, so their ordering should not be overinterpreted.

Failures concentrate near pouring. Across all 40 trials, grasping succeeds in 39, transport and replacement in 40, alignment in 29, and pouring in 28. Ten of the twelve incomplete-success trials are annotated with spills associated with inaccurate cup alignment. The single outright failure occurs when the robot closes on both cups without grasping either and continues into the pouring sequence. The stage metrics describe individual motion outcomes; they are not nested measures of full-task success.

Spills also occur in two in-domain trials, indicating a precision bottleneck that persists across the tested conditions. These observations motivate improved alignment and feedback during pouring, while leaving open whether the underlying limitation arises from perception, action prediction, or controller tracking.

7.7 Subtask-Phase Supervision

We next test the subtask-phase prediction heads of Section 4.4 as a current-goal signal for long-horizon control. The same architecture is post-trained with and without this supervision, then evaluated on three instruction-conditioned block tasks and on a Unitree G1-D dual-basket sorting sequence (Appendix E.4). Table 10 shows higher later-stage completion and higher full-episode success on all three simulation tasks: instructed ranking rises from 83.5% to 91.0%, instructed stacking from 44.5% to 58.5%, and instructed ranking & stacking from 1.5% to 16.0%. The same supervision raises later-stage completion on dual-basket sorting, from a mean of 1.0/4 objects placed without the heads to 2.2/4 with them. The gains are consistent with training the policy to recognize and complete the currently active subtask, rather than with adding an independently executed planner.

7.8 Learning from Human Egocentric Data

We study human data as an additional source of supervision for EWAM through three related comparisons. First, we test whether human-pretrained initialization improves transfer to a held-out robot embodiment. Second, we compare human- and robot-pretrained initialization under the same real-robot post-training mixture. Third, without pretraining and with 60 robot episodes fixed, we vary the number of added egocentric episodes and measure real-robot robustness under tablecloth and cup changes. The first two comparisons examine initialization; the third isolates the contribution of co-training data. These are separate training regimes, rather than successive stages of one recipe.

Human pretraining uses approximately 2,084 hours from VITRA-1M, EgoDex, and Xperience, with wrist motion expressed as camera-relative end-effector displacements. For real-robot co-training, we instead use task-specific human recordings retargeted to the G1-D action interface. Section 6.1 details the pretraining representation, and Section 6.2 describes demonstration construction.

7.8.1 Human Pretraining for Cross-Embodiment Transfer

We train on the 50 RoboTwin 2.0 tasks using ARX-X5, UR5, Franka, and Piper, and evaluate on the held-out Aloha-Agilex-2. The compared policies share the architecture, downstream robot data, and optimization, and differ only in whether they use human-pretrained initialization.

Human pretraining improves cross-embodiment success at every evaluated checkpoint (Figure 8). At 120K robot-training steps, success increases from 36.2% to 66.9%, a 30.7-point gain under a matched training budget. Comparing each policy’s best evaluated checkpoint gives 40.1% versus 66.9%, a 26.8-point gain.

7.8.2 Human Pretraining for Real-Robot Co-Training

We next evaluate the same initialization question by co-training Unitree G1-D on a long-horizon pouring task. Two policies are post-trained on the same mixture of 60 teleoperated robot episodes and 30 retargeted human episodes. They share the schedule, seed, and evaluation episodes, and differ only in human- versus robot-pretrained initialization. A robot-pretrained policy using 90 robot episodes provides a robot-only reference. All three are evaluated on 30 held-out robot trajectories for action error and on the physical robot for task success.

Under the matched 60-robot/30-human mixture, human pretraining increases real-robot success from 40% to 60% (Table 11). Training loss decreases from 0.068 to 0.034, and held-out action error decreases from 0.0231 to 0.0164 (Figure 9). The 90-robot reference reaches 70% success, but changes the data composition and therefore does not isolate the effect of adding human demonstrations. The next comparison holds the robot set fixed and varies only the amount of egocentric co-training data.

7.8.3 Effect of Egocentric Co-Training Data

Without pretraining and with 60 robot episodes fixed, adding 30 to 1,000 egocentric episodes raises real-robot success under tablecloth and cup changes from 10% to 60% without saturating (Figure 10; Appendix E.5). Adding just 30 egocentric episodes reaches 20%, matching robot-only training with 90 episodes. These results show that low-cost egocentric data can effectively improve robustness.

8 Discussion

From a fixed handoff to adaptive computation.

The emergent depth-wise handoff suggests that semantic grounding, visual foresight, and action refinement need not follow a fixed global schedule. Different tasks and execution phases may require different amounts of each form of computation. Ambiguous instructions may demand prolonged semantic grounding, uncertain interactions may require more predictive computation, and well-specified motions may permit an earlier transition to action refinement. Future models could therefore learn task- and state-conditioned routing that dynamically shifts the transition between experts.

From visual foresight to counterfactual planning.

The counterfactual future-feature intervention provides a further direction. Changing the content of intermediate future representations alters spatial targeting and arm selection, indicating that these representations carry control-relevant information rather than serving only as an auxiliary prediction target. The current model produces a future trajectory jointly with its action prediction, but does not explicitly compare alternative outcomes. A natural extension is to generate multiple action-conditioned futures and evaluate their consequences before committing to an action. This would move unified world–action modeling from a single predictive trajectory toward counterfactual planning, while retaining semantic grounding from the vision-language expert.

Limitations and future directions.

Our analysis characterizes information routing rather than the complete contents of the learned representations. Attention allocation identifies where action queries retrieve information, and the interventions establish functional dependence on selected sources, but neither provides a complete causal decomposition of the computation. Representation probing, activation patching, and cross-layer causal mediation could test whether semantic and predictive information itself changes together with the observed routing. Finally, although the deployed system achieves real-time operation through chunked execution, maintaining three large experts remains computationally expensive. The layer-wise division of labor observed in this work suggests a route to structured optimization. Rather than compressing all three experts uniformly, future work could optimize computation over the expert–depth grid: model capacity could be allocated according to the measured contribution of each expert at different depths, while structured pruning or parameter sharing could target regions with consistently low functional contribution. These improvements could reduce the model’s computational overhead while preserving its functional organization.

9 Conclusion

We present EWAM, a unified embodied model in which action queries read from vision-language and video experts through asymmetric attention. Its computation organizes into a depth-wise handoff from semantic grounding, through predicted visual futures, to action refinement. This organization emerges during training, and layer-targeted masking and counterfactual injection show that action generation depends on it. Across simulation benchmarks and real-robot embodiments, EWAM exceeds strong VLA, WAM, and hybrid baselines. Low-cost human egocentric data further improve cross-embodiment transfer and robustness, and subtask-phase supervision improves long-horizon completion. These results suggest that how unified policies coordinate semantic and predictive computation offers a principled path toward more capable embodied models.

Authors

Core Contributors Hao Wang1,2, Jiajun Wen3, Jingzhi Liu1, Shuoshuo Xue1, Zhiliang Chen2,4

Contributors Min Lin1, Yicheng Chang1, Xiaoyu Guo5, Yukang Zhuo1, Zheng Chong1, Yunshuang Nie3, Jian Zhang6, Weijia Liufu1, Qingman Wu1, Heming Xu7, Bingchang Song8, Dantong Wu9, Zhiyuan Wang9

Guidance Group Hang Xu9, Jianhua Han9, Bokui Chen3, Shen Zhao1

Project Leaders Rui Li9, Xiaodan Liang1,9

1Sun Yat-sen University 2Beijing Zhongguancun Academy 3Tsinghua University 4Fudan University 5South China University of Technology 6Mohamed bin Zayed University of Artificial Intelligence 7Northwest University, Xi’an 8Southern University of Science and Technology 9Yinwang Intelligent Technology Co. Ltd.

References

1. [1] A. Aida, W. Amer, K. Bankovic, D. Behl, F. Busch, A. Bhalla, M. Duong, F. Gienger, R. Godse, D. Grachev, R. Gulde, E. Hagensieker, J. Hu, S. Joshi, T. Knobloch, L. Kumar, D. LaRocque, K. Lokesh, O. Moured, K. Nguyen, C. Preyss, R. Sriganesan, V. Singh, C. Sponner, A. Tong, D. Tuscher, M. Tuscher, and P. Upputuri (2026) Cortex 2.0: grounding world models in real-world industrial deployment. arXiv preprint arXiv:2604.20246. External Links: Link
2. [2] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-VL technical report. arXiv preprint arXiv:2511.21631.
3. [3] H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, H. Zhao, H. Liu, Z. Su, L. Ma, H. Su, and J. Zhu (2026) Motus: a unified latent action world model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 35101–35113. External Links: Link
4. [4] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky (2025) \pi_{0.5}: A vision-language-action model with open-world generalization. In Proceedings of the 9th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 305, pp. 17–40. External Links: Link
5. [5] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2025) \pi_{0}: A vision-language-action flow model for general robot control. In Robotics: Science and Systems, External Links: 2410.24164
6. [6] K. Black, M. Y. Galliker, and S. Levine (2025) Real-time execution of action chunking flow policies. In Advances in Neural Information Processing Systems, pp. 33383–33407.
7. [7] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, et al. (2023) RT-1: robotics transformer for real-world control at scale. In Robotics: Science and Systems,
8. [8] J. Cai, Z. Cai, J. Cao, Y. Chen, Z. He, L. Jiang, H. Li, H. Li, Y. Li, Y. Liu, et al. (2026) InternVLA-A1: unifying understanding, generation and action for robotic manipulation. arXiv preprint arXiv:2601.02456.
9. [9] R. Cai, J. Guo, X. He, P. Jin, J. Li, B. Lin, F. Liu, W. Liu, F. Ma, K. Ma, F. Qiu, H. Qu, Y. Su, Q. Sun, D. Wang, D. Wang, Y. Wang, R. Wu, D. Xiang, Y. Yang, H. Ye, Y. Zhang, and Q. Zhou (2026) Xiaomi-robotics-0: an open-sourced vision-language-action model with real-time execution. arXiv preprint arXiv:2602.12684.
10. [10] N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris Coll-Vinent, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al. (2026) SAM 3: segment anything with concepts. In International Conference on Learning Representations, pp. 138846–138923.
11. [11] J. Chen, K. Wang, K. Chen, S. Chen, F. Gao, et al. (2026) LaWAM: latent world action models for efficient dynamics-aware robot policies. arXiv preprint arXiv:2606.15768.
12. [12] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Q. Liang, Z. Li, X. Lin, Y. Ge, Z. Gu, et al. (2026) RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In International Conference on Machine Learning (ICML), External Links: Link
13. [13] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion. In Robotics: Science and Systems,
14. [14] K. J. W. Craik (1943) The nature of explanation. Cambridge University Press.
15. [15] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, et al. (2018) Scaling egocentric vision: the EPIC-KITCHENS dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 753–771.
16. [16] C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, G. Shi, and H. Fan (2025) Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683.
17. [17] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, et al. (2023) PaLM-E: an embodied multimodal language model. In International Conference on Machine Learning,
18. [18] D. Ghosh, H. R. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, Y. L. Tan, L. Y. Chen, Q. Vuong, T. Xiao, P. R. Sanketi, D. Sadigh, C. Finn, and S. Levine (2024) Octo: an open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, External Links: Document
19. [19] GigaBrain Team et al. (2026) GigaBrain-0.7: scaling embodied foundation models to emergent capabilities with a three-system architecture. arXiv preprint arXiv:2608.15875.
20. [20] R. Goyal, S. E. Kahou, V. Michalski, J. Materzynska, S. Westphal, et al. (2017) The “something something” video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5843–5851.
21. [21] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, et al. (2022) Ego4D: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18995–19012.
22. [22] J. Guo, Q. Li, P. Li, Z. Chen, N. Sun, Y. Su, H. Wang, Y. Zhang, X. Li, and H. Liu (2026) Unified 4D world action modeling from video priors with asynchronous denoising. arXiv preprint arXiv:2604.26694.
23. [23] R. Hoque, P. Huang, D. Yoon, M. Sivapurapu, and J. Zhang (2026) EgoDex: learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations, pp. 4218–4237.
24. [24] C. Hou, K. Wu, J. Liu, Z. Che, D. Wu, F. Liao, G. Li, J. He, Q. Feng, Z. Jin, et al. (2025) RoboMIND 2.0: a multimodal, bimanual mobile manipulation dataset for generalizable embodied intelligence. arXiv preprint arXiv:2512.24653. External Links: Link
25. [25] T. Jiang, T. Yuan, Y. Liu, C. Lu, J. Cui, X. Liu, S. Cheng, J. Gao, H. Xu, and H. Zhao (2025) Galaxea open-world dataset and G0 dual-system VLA model. arXiv preprint arXiv:2509.00576. External Links: Link
26. [26] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu (2025) EgoMimic: scaling imitation learning via egocentric video. In IEEE International Conference on Robotics and Automation (ICRA), pp. 13226–13233.
27. [27] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn (2024) OpenVLA: an open-source vision-language-action model. In Conference on Robot Learning,
28. [28] H. Li, G. Zhao, Y. Liu, H. Hou, G. Ye, et al. (2026) ACE-Ego-0: unifying egocentric human and robotic data for vla pretraining. arXiv preprint arXiv:2606.17200.
29. [29] L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu (2026) Causal world modeling for robot control. arXiv preprint arXiv:2601.21998. External Links: Link
30. [30] Q. Li, Y. Deng, Y. Liang, L. Luo, L. Zhou, C. Yao, L. Zeng, Z. Feng, H. Liang, S. Xu, et al. (2025) Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571.
31. [31] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In International Conference on Learning Representations,
32. [32] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, pp. 44776–44791.
33. [33] S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu (2025) RDT-1B: a diffusion foundation model for bimanual manipulation. In International Conference on Learning Representations, pp. 29982–30009.
34. [34] Q. Long, Y. Wang, J. Song, J. Zhang, P. Li, W. Wang, Y. Wang, H. Li, S. Xie, G. Yao, H. Zhang, X. Wang, Z. Wang, X. Lan, H. Liu, and X. Li (2026) Scaling world model for hierarchical manipulation policies. arXiv preprint arXiv:2602.10983.
35. [35] B. D. Lucas and T. Kanade (1981) An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 674–679.
36. [36] Y. Ma, Z. Chen, E. Zhou, Z. Liu, H. Zhang, K. Liu, and H. Fan (2026) Table30 V2: evaluating generalized models by real robots at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4461–4467.
37. [37] Macrodata Labs (2026) Turning egocentric video into robot actions. Note: https://macrodata.co/blog/turning-egocentric-video-into-3d-hand-actionsTechnical blog, August 6, 2026. Accessed September 10, 2026
38. [38] A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, et al. (2023) Where are we in the search for an artificial visual cortex for embodied intelligence?. In Advances in Neural Information Processing Systems (NeurIPS), pp. 655–677.
39. [39] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta (2022) R3M: a universal visual representation for robot manipulation. In Conference on Robot Learning (CoRL),
40. [40] B. Paliwal, H. Etukuru, W. Liang, P. Abbeel, N. M. Shafiullah, and J. Malik (2026) Do as i do: dexterous manipulation data from everyday human videos. arXiv preprint arXiv:2606.19333.
41. [41] Physical Intelligence, B. Ai, A. Amin, R. Aniceto, A. Balakrishna, G. Balke, K. Black, G. Bokinsky, S. Cao, et al. (2026) \pi_{0.7}: A steerable generalist robotic foundation model with emergent capabilities. arXiv preprint arXiv:2604.15483. External Links: Link
42. [42] R. A. Potamias, J. Zhang, J. Deng, and S. Zafeiriou (2025) WiLoR: end-to-end 3d hand localization and reconstruction in-the-wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12242–12254.
43. [43] J. Romero, D. Tzionas, and M. J. Black (2017) Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics 36 (6), pp. 245:1–245:17.
44. [44] Ropedia (2026) Xperience-10M: a large-scale egocentric multimodal dataset with structured 3d/4d annotations. Hugging Face. Note: https://huggingface.co/datasets/ropedia-ai/xperience-10mDataset
45. [45] A. Savitzky and M. J. E. Golay (1964) Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry 36 (8), pp. 1627–1639. External Links: Document
46. [46] StarVLA Community (2026) StarVLA: a lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014.
47. [47] J. Sun, W. Zhang, Z. Qi, S. Ren, Z. Liu, H. Zhu, G. Sun, X. Jin, and Z. Chen (2026) VLA-JEPA: enhancing vision-language-action model with latent world model. In European Conference on Computer Vision (ECCV), pp. 478–497. External Links: Document
48. [48] Wan Team et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314.
49. [49] G. Wang, K. Ren, A. Morgan, Y. Chen, H. H. Qian, P. Chanrungmaneekul, and K. Hang (2026) EgoInfinity: a web-scale 4d hand-object interaction data engine for any-view robot retargeting and video-to-action robot learning. arXiv preprint arXiv:2606.17385.
50. [50] H. Wang, C. Yu, Y. Hu, J. Zhang, Y. Li, and S. Luo (2026) BifrostUMI: bridging robot-free demonstrations and humanoid whole-body manipulation. arXiv preprint arXiv:2605.03452.
51. [51] Q. Wang, M. Li, J. Guan, J. Ye, S. Xie, Y. Liu, J. Chen, Z. Liang, J. Zhang, X. Hu, et al. (2026) Qwen-VLA: unifying vision-language-action modeling across tasks, environments, and robot embodiments. arXiv preprint arXiv:2605.30280. External Links: Link
52. [52] Z. Wang, C. Liu, Y. Xiang, R. Zhang, Q. Hao, H. Lu, H. Chen, Z. Feng, K. Zheng, D. Ye, et al. (2026) The Great March 100: 100 detail-oriented tasks for evaluating embodied ai agents. arXiv preprint arXiv:2601.11421. External Links: Link
53. [53] H. Wu, Y. Jing, C. Cheang, G. Chen, J. Xu, et al. (2024) Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations (ICLR), pp. 10641–10662.
54. [54] K. Wu, C. Hou, J. Liu, Z. Che, X. Ju, Z. Yang, M. Li, Y. Zhao, Z. Xu, G. Yang, et al. (2025) RoboMIND: benchmark on multi-embodiment intelligence normative data for robot manipulation. In Robotics: Science and Systems (RSS), External Links: Link
55. [55] S. Wu, X. Liu, S. Xie, P. Wang, X. Li, B. Yang, Z. Li, K. Zhu, H. Wu, Y. Liu, et al. (2025) RoboCOIN: an open-sourced bimanual robotic data collection for integrated manipulation. arXiv preprint arXiv:2511.17441. External Links: Link
56. [56] W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, et al. (2026) A pragmatic vla foundation model. arXiv preprint arXiv:2601.18692.
57. [57] W. Wu, F. Wang, F. Lu, H. Sun, S. Liu, Y. Wang, Y. Yan, Y. Wang, S. Ma, X. Wang, Y. Liu, S. Yang, T. Zhou, K. Zhang, L. Zhou, C. Su, N. Xue, B. Tan, H. Zhang, Y. Zhang, F. Liao, X. Zhu, Y. Shen, and K. Zheng (2026) From foundation to application: improving VLA models in practice. arXiv preprint arXiv:2607.06403.
58. [58] Xiaomi Robotics Team, J. Guo, P. Jin, J. Li, P. Li, Y. Li, F. Liu, W. Peng, O. Qin, Y. Su, et al. (2026) Xiaomi-Robotics-1: scaling vision-language-action models with over 100k hours of real-world trajectories. arXiv preprint arXiv:2607.15330.
59. [59] A. Yakefu, B. Xie, C. Xu, E. Zhang, E. Zhou, F. Jia, H. Yang, H. Fan, H. Zhang, H. Peng, et al. (2025) RoboChallenge: large-scale real-robot evaluation of embodied policies. arXiv preprint arXiv:2510.17950. External Links: Link
60. [60] L. Yang, W. Song, X. Wang, P. Sheng, Z. Fang, Z. Zhou, J. He, H. Yan, J. Chen, N. Sun, Q. Sun, P. Wang, L. Liu, Y. Wang, Y. Gao, F. Dayoub, and H. Li (2026) 4D-WAM: infusing spatiotemporal awareness into world action models through trajectory fields. arXiv preprint arXiv:2608.08023.
61. [61] Y. Yang, S. Zeng, T. Lin, X. Chang, D. Qi, et al. (2026) ABot-M0: vla foundation model for robotic manipulation with action manifold learning. arXiv preprint arXiv:2602.11236.
62. [62] Y. Yang, Z. Liu, S. Kou, Y. Chen, Y. Hu, et al. (2026) World-language-action model for unified world modeling, language reasoning, and action synthesis. arXiv preprint arXiv:2606.05979.
63. [63] A. Ye, B. Wang, C. Ni, G. Huang, G. Zhao, H. Li, H. Li, J. Li, J. Lv, J. Liu, et al. (2026) GigaWorld-policy: an efficient action-centered world-action model. arXiv preprint arXiv:2603.17240.
64. [64] S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, et al. (2026) World action models are zero-shot policies. arXiv preprint arXiv:2602.15922. External Links: Link
65. [65] S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, et al. (2025) Latent action pretraining from videos. In International Conference on Learning Representations (ICLR), pp. 28213–28239.
66. [66] H. Yuan, Z. Liang, A. Chen, Y. Wang, H. Li, P. Lin, Y. Huang, Z. Lei, T. Zhang, J. Zhang, et al. (2026) Qwen-RobotManip technical report: alignment unlocks scale for robotic manipulation foundation models. arXiv preprint arXiv:2606.17846.
67. [67] T. Yuan, Z. Dong, Y. Liu, and H. Zhao (2026) Fast-WAM: do world action models need test-time future imagination?. arXiv preprint arXiv:2603.16666. External Links: Link
68. [68] H. Zhang, L. Xiang, H. Lin, Z. Huang, M. Wang, et al. (2026) Hy-Embodied-0.5-VLA: from vision-language-action models to a real-world robot learning stack. arXiv preprint arXiv:2606.14409.
69. [69] J. Zhang, Y. Hu, Y. Guo, X. Chen, Y. Liu, W. Chen, C. Lu, and J. Chen (2026) UniJEPA: enhancing robot policy via unified continuous and discrete representation learning. In International Conference on Machine Learning,
70. [70] J. Zhang, J. Deng, C. Ma, and R. A. Potamias (2025) HaWoR: world-space hand motion reconstruction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1805–1815.
71. [71] T. Zhang, Z. Yuan, D. Chi, P. Liu, D. Li, et al. (2026) JoyAI-RA 0.1: a foundation model for robotic autonomy. arXiv preprint arXiv:2604.20100.
72. [72] Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang (2022) ByteTrack: multi-object tracking by associating every detection box. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 1–21.
73. [73] Y. Zhang, W. Zhang, Z. Qi, H. Zhang, H. Lin, et al. (2026) ImageWAM: do world action models really need video generation, or just image editing?. arXiv preprint arXiv:2606.19531.
74. [74] Z. Zhang, Z. Li, B. Rahmati, R. H. Yang, Y. Ma, A. Rasouli, S. Pakdamansavoji, Y. Wu, L. Zhang, T. Cao, F. Wen, X. Wang, X. Quan, and Y. Zhang (2026) Do world action models generalize better than VLAs? a robustness study. arXiv preprint arXiv:2603.22078. External Links: Link
75. [75] J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, T. Wang, Y. Zhang, J. Liu, and X. Zhan (2026) X-VLA: soft-prompted transformer as scalable cross-embodiment vision-language-action model. In International Conference on Learning Representations, pp. 60580–60606.
76. [76] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. (2023) RT-2: vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165–2183.

Appendix A Cross-Paradigm Attention Measurement

Figure 1 compares action-query attention routing across four policies. The measurement details are as follows.

Checkpoints.

All baseline weights are publicly released. Figure 1(b) uses the released Fast-WAM [67] weights and Figure 1(c) the released Motus [3] weights. Figure 1(a) uses the \pi_{0.5} baseline checkpoint distributed with Motus, that is, a \pi_{0.5} policy trained on RoboTwin 2.0 by the Motus authors. Figure 1(d) uses the in-domain EWAM checkpoint of Table 5, which is also the checkpoint used for the inference-time interventions of Table 1.

Evaluation condition.

Every panel is measured on the same single RoboTwin 2.0 Adjust Bottle episode under the randomized protocol, at denoising step 0. Attention allocation is stable across tasks and denoising steps, so a single episode at one denoising step is representative (Appendix C.2). The horizontal axis is the policy inference step, that is, the successive action-chunk queries within the episode; the number of steps differs across panels because the four policies complete the episode in different numbers of chunks.

Key grouping.

For each model we assign every attended key to one of the sources that model possesses and apply Eq. 12 over those groups. The available sources differ by architecture: \pi_{0.5} has no video stream, and Fast-WAM has no separate vision-language stream.

Appendix B Model and Training Details

B.1 Architecture

EWAM contains three depth-aligned transformer experts for semantic understanding, future-visual prediction, and action generation. The vision-language expert receives the language instruction and current image. The video expert receives a latent representation of the observed condition frame and noisy future-video latents. The action expert receives a noisy action chunk. Corresponding blocks from the three experts are executed at the same network depth, allowing the action representation to aggregate information progressively rather than waiting for a separately generated visual subgoal.

The attention mask is asymmetric. Vision-language queries attend only within the vision-language stream, and video queries attend only within the video stream. Action queries may attend to vision-language, video, and action keys. This design prevents the auxiliary generative stream from overwriting semantic representations while making both sources directly available to action computation. It also gives a well-defined quantity for measuring how action queries allocate attention among modalities at each depth.

VL layer alignment and recurrent fusion.

The 28 Qwen3-VL [2] hidden states are extracted in a single Qwen3-VL forward pass before the 30-block unified computation. Writing the original Qwen3-VL outputs as (\bar{q}_{0}^{L},\ldots,\bar{q}_{27}^{L}), the aligned sequence is

(q_{0}^{L},\ldots,q_{29}^{L})=(\bar{q}_{0}^{L},\ldots,\bar{q}_{27}^{L},\bar{q}_{26}^{L},\bar{q}_{27}^{L}).

At each unified block, Eq. 4 combines the aligned Qwen3-VL state with the accumulated VL state. The normalized fused tokens are mapped by block-specific query, key, and value projections from 2,048 dimensions to 24 heads of width 128. Video, action, and VL tensors are then concatenated and processed by one masked-attention call. For the VL query rows, the mask retains only VL keys; the same VL keys and values remain visible to action queries.

The VL-side attention result is mapped back to 2,048 dimensions by a block-specific output projection and applied through a residual update,

z_{l}^{L}=x_{l}^{L}+W_{O,l}^{L}\,\operatorname{Attn}_{l}^{L}(x_{l}^{L}).

Here \operatorname{Attn}_{l}^{L} denotes the VL-query rows of the shared masked attention operation. The resulting z_{l}^{L} is carried to the next fusion step. No additional VL feed-forward layer follows this update because each q_{l}^{L} already includes the native Qwen3-VL layer’s feed-forward computation.

The implementation used in our analysis contains 30 aligned transformer blocks, indexed from 0 to 29. Based on the observed attention transitions, we refer to blocks 0–9 as the shallow stage, blocks 10–19 as the intermediate stage, and blocks 20–29 as the deep stage. These equal-width ranges are descriptive partitions rather than architecturally hard-coded modules.

B.2 Training Objectives

Training follows two stages. During cross-embodiment pretraining, EWAM jointly optimizes future-video and action flow-matching losses. Independent noise levels are sampled for video and action targets so that the model cannot solve one modality by relying on an identical denoising state in the other. Padded action dimensions introduced by cross-embodiment alignment are masked from the loss. The base objective is

\mathcal{L}_{\mathrm{base}}=\lambda_{v}\mathcal{L}_{\mathrm{video}}+\lambda_{a}\mathcal{L}_{\mathrm{action}}.

Task-specific post-training retains both base losses and adds subtask-phase supervision. Annotated phases of each demonstration supply a textual label of the active subtask and a normalized phase index in [0,1] derived from the same partition:

\mathcal{L}_{\mathrm{post}}=\mathcal{L}_{\mathrm{base}}+\lambda_{p}\mathcal{L}_{\mathrm{prog}}+\lambda_{s}\mathcal{L}_{\mathrm{sub}}.

The scalar is regressed from the action-side representation; the textual label identifies the currently active semantic phase. Both terms are applied only during task-specific post-training.

Appendix C Token-Count-Normalized Attention Analysis

The allocation R_{l}^{m} in Eq. 12 measures the total attention mass received by source m. Because the five sources contain different numbers of keys, we additionally measure attention enrichment relative to a uniform-token reference:

E_{l}^{m}=\frac{R_{l}^{m}}{|\mathcal{K}_{m}|/\sum_{j}|\mathcal{K}_{j}|},

where E_{l}^{m}>1 indicates that a source receives more attention than expected if attention were distributed uniformly over all available keys. We compute this statistic on one representative sequence from each of the 50 RoboTwin 2.0 tasks, using the robot-pretrained in-domain checkpoint of Table 5. Every sequence contains 120 condition-frame, 240 future-frame, 21 action, and 120 VL-image tokens, while VL-text length varies from 59 to 85 tokens. We therefore compute E_{l}^{m} using each task’s own token counts, average over its five measured denoising steps, and then macro-average with equal task weight.

Table 12 reports both total attention mass and token-normalized enrichment for the two visual sources involved in the observed handoff. VL-image tokens are strongly enriched in shallow layers but decline rapidly with depth, while future-frame attention rises. In the cross-task macro average, future-frame raw mass first exceeds VL-image mass at layer 13, and token-normalized future enrichment first exceeds image enrichment at layer 14. The handoff therefore persists after controlling for the two-to-one difference in future-frame and VL-image token counts.

C.1 Checkpoint Evolution of the Visual Handoff

Table 13 reports the checkpoint trajectory of the asymmetric model over twelve checkpoints, and Figure 11 plots it against training step in the same layout as the bidirectional-mask trajectory of Figure 15. The table resolves the concurrent redistribution of attention from image to text tokens and separates the two halves of the handoff.

Three phases are visible. At 50 steps, action queries attend to VL-image tokens at every depth: image enrichment is 2.07, 1.74, and 2.33 across the three stages, and text accounts for only 2.9% of shallow attention. Between 100 and 400 steps the shallow stage rapidly acquires its linguistic character. Shallow text mass rises from 14.3% to 54.8%, while deep image enrichment falls from 0.75 to 0.21. A future-over-image crossover is already detectable during this phase, but its onset shifts from layer 23 at 100 steps to layers 11–12 thereafter, and its persistence varies from 71% to 95%.

The subsequent refinement is not monotonic. At 600 and 1,000 steps, stable crossovers begin at layer 14 but persist over only 62% and 56% of the remaining layers. At 5,000 steps, shallow image enrichment rebounds to 1.79 and no stable crossover is detected. By 40,000 steps, shallow image enrichment remains high at 1.75, deep image enrichment falls to 0.04, and the layer-14 crossover persists through every subsequent layer. Thus, coarse semantic-to-action depth organization appears within the first few hundred steps, whereas the specific shallow-image-to-intermediate-future handoff fluctuates before stabilizing at convergence. All checkpoints come from the same thirty-two-GPU run, so this trajectory is not confounded by a change in global batch size.

C.2 Cross-Task Mechanism Replication

Using the same 50-task macro-averaging protocol and the same robot-pretrained in-domain checkpoint, we next quantify how often the depth-wise mechanism occurs. For stage-level allocations, uncertainty is estimated with 95% task-bootstrap confidence intervals (CIs), while mechanism prevalence uses 95% Wilson CIs. We define the stable crossover as the first of three consecutive layers for which future-frame attention exceeds VL-image attention, and summarize its location by the median and interquartile range (IQR). Figure 12 reports the complete raw five-source routing profile. Table 14 summarizes the stage-level attention values, raw and token-normalized mechanism prevalence, crossover locations, and denoising-step stability.

The upper block of Table 14 quantifies the three-stage reallocation. VL tokens receive 75.38% of shallow attention, compared with 14.05% for video and 10.58% for action. The middle stage is substantially more distributed (36.55% VL, 24.31% video, and 39.14% action), consistent with a gradual handoff rather than a hard module boundary. In the deep stage, action attention rises to 59.46%, while VL and video attention fall to 25.22% and 15.32%, respectively.

The lower block measures how consistently this organization appears across tasks. All 50 tasks exhibit shallow VL dominance, the intermediate raw image-to-future handoff, and deep action dominance. The stable raw crossover is tightly concentrated around layers 12–13. After controlling for each task’s token counts, 49 of 50 tasks still exhibit the handoff over layers 13–15, and all tasks eventually reach a stable normalized crossover, with a median at layer 14. The mean correlation of 0.998 across denoising-step profiles further shows that these depth trends are largely invariant to denoising time.

The only task without a late-middle normalized handoff is Stamp Seal, for which future-frame enrichment is slightly below VL-image enrichment when averaged over layers 13–15 (margin -0.020). Its normalized stable crossover occurs later, at layer 17, and it still exhibits the complete raw understanding–foresight–action progression.

Appendix D Attention Routing under a Fully Bidirectional Mask

The asymmetric mask in Eq. 3 restricts the two perceptual streams to their own tokens. To determine what this restriction contributes, we train a variant in which every stream may attend to every other stream, so that A\leftarrow\{V,A,L\}, V\leftarrow\{V,A,L\}, and L\leftarrow\{V,A,L\}, and apply the same depth-wise attention analysis. As for the asymmetric trajectory of Appendix C.1, the variant is trained without robot pretraining; all other components, objectives, and data are unchanged. We report two independently trained runs: a trajectory of checkpoints between 50 and 5,000 optimizer steps trained on four GPUs, and a converged 40,000-step run trained on thirty-two GPUs. Because the two runs use different global batch sizes, we treat the converged run as an independent replication rather than as a continuation of the trajectory. Both are analysed on a single representative sequence with ten denoising steps.

The coarse depth organization does not depend on the mask.

At convergence, the bidirectional variant reproduces the three-stage action-attention profile reported in Section 5, which is visible in the aggregate maps of Figure 13(a). VL tokens receive 85.6% of shallow action attention, the intermediate stage is distributed across VL, video, and action tokens (40.6%, 29.0%, and 30.5%), and action self-attention rises to 49.9% in the deep stage. The concentration of predicted-future usage in intermediate layers also recurs: future-frame enrichment rises from 0.14 in the shallow stage to 0.50 in the middle stage before falling to 0.31, peaking at layer 14, against a corresponding 0.21–0.41–0.24 profile for the asymmetric model at the same step. At the level of the three-source aggregate, and on the future-frame side of the visual transition, the depth-wise organization studied in this paper is therefore not produced by the asymmetric mask and is not an artifact of one particular choice of cross-stream connectivity. We regard this as a partial replication that strengthens the generality of the phenomenon. As in Section 5, however, the stage-level aggregate conceals the sub-modality structure, and it is there that the two configurations diverge.

The role of the semantic stream does depend on the mask.

The two configurations differ in what the vision-language stream itself computes. Under the asymmetric mask the vision-language stream attends only to its own tokens at every depth, by construction. Under the bidirectional mask it does not, as Figure 14 and Figure 13(b) show: its shallow layers retain self-attention, but from roughly layer 11 onward video tokens become the dominant key source, and in the deep stage the vision-language stream directs 82.0% of its attention to video tokens while retaining only 15.3% on its own. In other words, the semantic stream ceases to act primarily as a source of semantic keys and becomes a consumer of the generative stream.

The consequence on the action side is that the visual handoff itself does not survive, even though its future-frame half does. A handoff requires two things: an early reliance on the observed image, and its subsequent release in favour of predicted future frames. Table 15 shows that the bidirectional variant has neither. Under the asymmetric mask, action queries use VL-image tokens in a sharply depth-specific way: image enrichment is 1.75 in the shallow stage, well above the uniform-token reference, and falls to 0.04 in the deep stage, a shallow-to-deep ratio of roughly 44. The model reads the current image where it grounds the instruction and gives it up once actions are being refined. Under the bidirectional mask this differentiation is absent. Figure 13(c) shows the pattern directly: the image column is pale at every depth, whereas the text column is dark throughout the shallow stage. Image enrichment is flat across depth (0.81, 0.70, and 0.67 for the three stages, a ratio of 1.2), never reaches the uniform-token reference at the stage level, and attains its per-layer maximum at layer 18 rather than in the shallow stage. Its shallow stage is instead almost purely linguistic, with text enrichment of 6.29 against image enrichment of 0.81, so the shallow visual grounding that the asymmetric model performs has no counterpart here.

The bidirectional variant does exhibit layers at which future-frame enrichment numerically exceeds image enrichment, at layers 13–15. That inequality, however, holds between two signals that both remain below the uniform-token reference throughout the network, and it is produced entirely by the rise of future-frame attention rather than by any release of the observed image: image enrichment recovers to 0.67 in the deep stage rather than collapsing. We therefore do not regard it as the same phenomenon. Consistent with this, the inequality is not sustained. Measuring the fraction of layers at or beyond the stable crossover for which token-normalized future enrichment exceeds image enrichment, the bidirectional variant reaches only 24%, whereas the asymmetric model maintains the ordering at every layer from the crossover to layer 29.

Training dynamics.

Figure 15 traces these quantities across training. Three observations follow. First, the vision-language stream moves in opposite directions at the two ends of the network. Its shallow layers reclaim their own tokens early, with self-attention rising from 27.1% at 50 steps to 43.0% at 150 and 66.4% at 250. Its deep layers do not: deep self-attention recovers only to 39.6% near 350 steps and then declines steadily, reaching 27.0% at 5,000 steps in the four-GPU run and 15.3% in the independently trained converged run. Attention to video moves oppositely at the two ends over the same span: the shallow share falls from 69.3% to 13.7% between 50 and 5,000 steps and stands at 26.4% in the converged run, whereas the deep share rises from 64.9% to 68.8% over the same interval and reaches 82.0% at convergence. The deep-layer capture is therefore progressive rather than a property of initialization. Second, the action stream reorganizes over the same short interval: between 150 and 250 steps, shallow action attention to VL-text rises from 13.1% to 37.7% while shallow action attention to video falls from 60.9% to 24.0%. Because the shallow vision-language recovery occupies exactly this window, the two streams reorganize together rather than in a resolvable order. Third, crossover persistence declines monotonically over training, from 83% at 150 steps to 55% at 400 steps and 31% at 5,000 steps, with the converged run at 24%. Under a bidirectional mask the future-over-image ordering therefore becomes less rather than more sustained as training proceeds, and does not consolidate into a handoff.

Scope.

These measurements characterize where each stream retrieves information, not what its representations encode. Attention to video keys does not by itself establish that semantic content has been displaced from the vision-language representation, since residual connections can carry shallow-layer semantics through deeper blocks irrespective of where those blocks attend. Establishing whether the routing change is accompanied by a representational change requires decoding semantic targets directly from the deep vision-language activations of both configurations, which we leave to future work. We therefore restrict the claim to routing: the asymmetric mask is not what produces the coarse depth-wise organization, but it is what makes the observed image a source that shallow layers rely on and deeper layers release, and what keeps the vision-language stream a pure semantic source. It is therefore also what makes token identity an interpretable indicator of information source in the analysis of Section 5.

Appendix E Experimental Details

E.1 RoboTwin 2.0

RoboTwin 2.0 contains 50 bimanual manipulation tasks and applies structured randomization to clutter, illumination, background, tabletop height, and language [12]. For the standard multi-task experiment, we evaluate one policy across all tasks in both Clean and Randomized scenes. Each reported aggregate is the unweighted mean of per-task success rates, with 100 evaluation episodes per task and setting.

For clean-to-random evaluation, no target-embodiment randomized data are used during downstream adaptation; the adapted policy is evaluated directly in the fully randomized Hard setting. In contrast, the in-domain results in Table 5 use both clean and randomized data during downstream adaptation. Table 16 summarizes the RoboTwin evaluation settings detailed above. Table 17 reports the complete task-level Clean and Randomized results for the in-domain multi-task evaluation.

E.2 LIBERO

LIBERO evaluates knowledge transfer across spatial, object-centric, goal-conditioned, and long-horizon manipulation suites [32]. We train and evaluate a single EWAM policy under the same observation and action interface used by the comparison methods. The primary metric is average task success, reported separately for LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long in addition to the overall mean.

E.3 Real-World Evaluation Details

We evaluate the same model family on a Franka single-arm platform, a Dobot dual-arm platform, and a Unitree G1-D humanoid. The selected tasks cover single-arm object rearrangement, bimanual coordination, and contact-rich liquid transfer. Franka evaluation includes stacking bowls and placing an object into a box. Dobot evaluation includes pouring water, tidying a desk, and manipulating a towel. Unitree G1-D evaluation includes kettle pouring and pouring beans. Together, these tasks test grasp stability, object rearrangement, deformable-object manipulation, bimanual coordination, and controlled pouring. Figure 16 illustrates the real-world robot evaluation setup and corresponding tasks. Table 18 summarizes the embodiments and tasks included in this evaluation.

Every method is evaluated with 10 trials per task on each platform, and all real-world success rates reported in Tables 7, 8, and 11 are computed over those 10 trials. An episode is counted as successful only when the task-level terminal condition is satisfied, rather than when an intermediate grasp or motion succeeds. Initial object poses are varied across trials within the reachable workspace. We use the same task instruction for EWAM and the corresponding baseline and do not manually intervene after execution begins.

E.4 Long-Horizon Subtask Evaluation

This appendix details the two long-horizon evaluations of Section 7.7: three instruction-conditioned block tasks in simulation, and Dual-Basket Sorting on a Unitree G1-D.

E.4.1 Instruction-Conditioned Block Tasks

The three tasks used in Section 7.7 are adapted from the Blocks Ranking and Blocks Stacking layouts of RoboTwin and are distinct from the 50-task suite evaluated in Sections 7.2 and 7.3. In the original designs, three colored blocks (red, green, and blue) are initialized at random table positions and must be arranged in a fixed red–green–blue order, from left to right for ranking and from bottom to top for stacking. Our variants preserve the object set, workspace, and embodiment, and modify only the language interface. Each instruction specifies a sequence of local objectives that changes within the episode, so later stages remain conditioned on completing the currently active subtask.

Instructed Ranking.

The policy grasps the three blocks in one instructed order and places them along a left-to-right line in a second instructed order. The two orders need not coincide. For example, an instruction may require grasping blue–red–green and placing red–green–blue from left to right.

Instructed Stacking.

The policy grasps the three blocks in an instructed order and stacks them bottom to top in that same order. With three blocks this admits only 3!=6 instruction variants. Because the stacking order is not specified independently, an episode that completes the stack necessarily completes the full task, so Stack and Full coincide in Table 10. For example, an instruction may require grasping blue–red–green and stacking blue–red–green from bottom to top.

Instructed Ranking & Stacking.

This task composes the two skills within a single episode. The policy first completes an instructed ranking, grasping in one specified order and placing from left to right in a second, and then restacks the same blocks bottom to top in a third instructed order. For example, an instruction may require grasping blue–red–green, placing red–green–blue from left to right, and then stacking green–red–blue from bottom to top.

Evaluation protocol.

Each task is evaluated for 200 episodes. Both variants share the same unified backbone, pretraining, and post-training data, and differ only in whether the subtask-phase heads are trained. Grasp, Place, and Stack are the percentages of episodes that complete the named stage, with Place and Stack counted only when the currently instructed target is met; they are computed independently over the same 200 episodes and are therefore not nested, so a later-stage rate may exceed an earlier-stage rate. Full is complete-episode success, counted only when the terminal spatial configuration satisfies every specified order. Dashes in Table 10 mark stages a task does not define: Stack is absent from instructed ranking, and Place is absent from instructed stacking. Because the commanded orders differ from the fixed convention of the original tasks, success requires finishing each active subtask and then the remaining sequence, rather than reproducing a memorized layout.

E.4.2 Real-World Dual-Basket Sorting

We evaluate the same with- versus without-supervision pair on a Unitree G1-D Dual-Basket Sorting sequence. Four objects are placed on the table: a doll, a marker, a crumpled tissue, and a used water bottle. The policy must place useful objects (the doll and the marker) in the right basket and waste (the tissue and the bottle) in the left basket, thereby clearing the table. We score each trial by the number of objects placed in the designated basket (out of four); a trial is fully successful only when all four objects are cleared.

With subtask-phase supervision, ten trials complete 4,3,0,1,3,3,2,3,1,2 objects (mean 2.2/4); one trial clears the table. After repeated failures on the marker, the policy typically switches to another remaining object and later returns, rather than persisting on the same failed grasp. Without this supervision, ten trials complete 0,1,1,1,2,1,2,1,1,0 objects (mean 1.0/4), with no full clearance. After a failed grasp the policy typically retries the same object and does not reallocate to an easier remaining target.

Together with the block tasks, these results indicate that subtask-phase labels supply a current-goal signal. Training the policy to recognize and complete the active phase increases later-stage completion and full-sequence success on both the instructed block tasks and the physical multi-object sequence.

Figure 17 illustrates the behavioral contrast. With subtask-phase supervision (top), the policy maintains a clear local objective and clears the table: it successfully grasps each of the four objects and places them in the designated baskets. Without this supervision (bottom), rollouts frequently trigger grasp height errors and grasp pose-and-position errors. The policy then retries the same failed grasp rather than switching to another object, so progress stalls and the table is not cleared.

E.5 Generalization under Tabletop and Cup Changes

Evaluation setting.

This section details the real-robot pouring evaluation summarized in Effect of Egocentric Co-Training Data (Section 7.8.3), which changes table colours or tablecloths and replaces the pouring cup. All seven configurations omit EWAM robot and human pretraining and use end-effector (EEF) actions. We vary robot-only demonstration counts or add egocentric data to 60 robot demonstrations. Each configuration is evaluated over 10 physical trials, randomly switching among two to three tablecloth and cup variants. Rates pool these trials. This evaluation is separate from Table 11.

Observed results.

Robot-only training remains at 0–20% success (Figure 10). With robot demonstrations fixed at 60, adding 1,000 egocentric demonstrations raises success from 10% to 60%, a gain of 50 percentage points. Two findings follow from these results. First, a small amount of egocentric data can substitute for an equal amount of robot data: 60 robot plus 30 egocentric demonstrations reach 20%, matching 90 robot-only demonstrations, while egocentric data are far cheaper to collect. Second, performance continues to scale with egocentric data: as egocentric demonstrations increase from 30 to 1,000, success rises from 20% to 30%, 40%, and 60%, with no sign of saturation in this range. Egocentric data thus offer a cost-effective way to improve robustness.

Appendix F Inference Latency and Real-Time Deployment

Inference configuration and timing.

EWAM generates a chunk of 16 actions, each with 16 dimensions, in 258 ms on a single RTX 5090 using five denoising (flow-integration) steps, including chunk-overlap correction (Table 19). Timings are measured at batch size one in bfloat16 after warm-up, with GPU synchronization around each inference call. We report the minimum of three repetitions, which differ by at most 1 ms.

Reducing inference cost.

We omit VAE decoding of predicted future frames because robot control requires only the action output. The video expert and the latent future features used by action tokens remain active. Rewriting rotary position embeddings to avoid graph breaks enables full-graph compilation of each denoising step. At ten denoising steps, these changes reduce chunk latency from 1,668 to 461 ms; overlap correction adds 3 ms, giving 464 ms in total and a 3.6\times speedup over the original implementation. With five steps, the corresponding timings are 254 ms before correction and 258 ms after correction. The reduction in denoising steps changes the sampling procedure and is separate from the implementation optimizations. Neither float16 nor the tested FP8 implementations improved on compiled bfloat16.

Action execution and replanning.

With a temporal stride of four, a 16-action prediction has a nominal horizon of 16\times 4/30=2.13 s at a 30 Hz command rate. The measured 258 ms inference time is approximately 12% of this horizon, providing a latency margin relative to the nominal prediction horizon. Real-time chunking [6] generates a new chunk while the previous one executes and constrains generation using the remaining actions to improve continuity. The command execution rate and the observation-conditioned replanning rate are distinct: the latter depends on inference latency and the execution schedule.

Comparison with other policies.

Table 20 compares action-output inference on the same GPU, with batch size one, bfloat16, and the same action-chunk shape. Baseline models use randomly initialized weights, so the results characterize their execution cost under this benchmark configuration. At five denoising steps, EWAM takes 258 ms versus 901 ms for Motus, while remaining slower than \pi_{0.5}, Xiaomi Robotics-1, and Fast-WAM.

Appendix G Pretraining Data

This section describes the data used in EWAM’s two separate pretraining regimes, with further details provided in Sections G.1 and G.2. The human pretraining regime uses approximately 2,084 hours of egocentric video, in which wrist motion supplies the action target. Independently, the robot pretraining regime uses 300K cross‑embodiment trajectories, comprising approximately 1,800 hours of curated robotic manipulation data collected across multiple robot embodiments, and provides synchronized observations, language instructions, robot states, actions, and future visual observations. In each regime, heterogeneous state and action spaces are mapped into a shared padded representation, while validity masks ensure that unavailable dimensions do not contribute to the action loss.

G.1 Human Egocentric Data

The human stage of Section 6.1 draws on three egocentric corpora summarized in Table 21. Combined, they provide approximately 2,084 hours of first-person manipulation video paired with per-frame hand pose and camera geometry. The datasets offer complementary coverage: VITRA‑1M [30], compiled from unconstrained internet footage, provides the broadest diversity of objects, environments, and activities; EgoDex [23] contains scripted tabletop manipulation with high-quality on-device hand tracking; Xperience [44] supplies SLAM-registered recordings captured via a head-mounted stereo rig.

VITRA‑1M provides MANO‑based hand reconstructions together with corrected camera intrinsics and extrinsics for episodes derived from Ego4D, EPIC‑Kitchens, and Something‑Something v2 [21, 15, 20]. We use its ego4d_cooking_and_cleaning, ego4d_other, epic, and ssv2 subsets, retaining 1,155,865 episodes; the EgoExo4D subset is excluded. Frames flagged as untracked by the released per‑hand masks are treated as invalid. EgoDex contributes 338,000 demonstrations across 194 tabletop tasks recorded at 30 Hz, with per‑frame hand tracking confidences; we discard hand observations whose confidence falls below 0.3. Xperience is a partially released corpus recorded with a head‑mounted stereo rig, for which hand poses are stored in the per‑frame rectified stereo‑left camera frame and must first be lifted to world coordinates using the SLAM trajectory before the shared computation applies. Episodes lacking a SLAM pose, lacking stereo calibration, or shorter than one training window are dropped at indexing time.

Every retained episode is reduced to world‑frame wrist poses plus an anchor camera pose and then converted by Eq. 14, so the sixteen action dimensions carry identical physical meaning across the mixture. Sampling is weighted in proportion to the number of indexed windows in each corpus. Because the targets are camera‑relative displacements whose scale is already comparable across sources, no action normalization is applied during this stage.

G.2 Robotic Manipulation Data

During pretraining, EWAM leverages multiple open-source robotic manipulation datasets containing approximately 3,250 hours of raw recordings, including GM-100 [52], RoboCOIN [55], RoboMIND1.0 [54], RoboChallenge1.0 [59], RoboChallenge2.0 [36], RDT [33], and Simulation data [12, 24, 54]. To focus on tabletop manipulation, we apply task‑relevance filtering, data cleaning, and quality assessment. After removing invalid, low‑quality, and task‑irrelevant samples, we obtain approximately 1,800 hours of high‑quality pretraining data. Figure 18 shows the retained duration and corresponding proportion of each source dataset.

The pretraining data of EWAM cover a wide range of robot platforms and manipulation tasks. Substantial disparities exist among robots in terms of embodiment, degrees of freedom, and manipulation approaches. The data encompass representative manipulation tasks, including grasping, placing, moving, organizing, switch operation, and cleaning, together with a wide variety of interactive objects. As shown in Figure 19, the composition of robot platforms and the keyword distributions of manipulation tasks and interactive objects are presented.

To systematically characterize the composition of the pretraining data, we summarize the kinematic joint dimensions across various robot platforms in Table 22. The reported statistics correspond to joint-state dimensionality. As several platforms support both single-arm and bimanual setups, joint dimensions differ accordingly.

During data processing, we find substantial differences among the open‑source datasets in terms of acquisition devices, data formats, annotation protocols, and data quality. Common defects include corrupted videos, black frames, severe occlusion, prolonged inactivity, and missing frames. Certain episodes also exhibit state‑action misalignment, abrupt signal changes, and numerical outliers. These issues may disrupt the temporal continuity of action sequences and the logical consistency between states and actions, thereby impairing model pretraining.

According to the overall quality inspection results, most datasets exhibit varying degrees of state‑action misalignment, abrupt numerical changes, and anomalous values. Numerical outliers have the most detrimental effect on model optimization. After the model has fitted the normal data, anomalous samples substantially increase the difficulty of optimization, causing the loss to rise sharply and potentially preventing training from converging. Several datasets contain a substantial number of anomalies, including extreme outliers with magnitudes as high as tens of billions. Regarding state‑action alignment, some samples exhibit state signals that precede action signals or motion trends that are inconsistent between the two. Further analysis indicates that these problems typically arise from inconsistent sampling frequencies or temporal synchronization errors among different signals. Moreover, some episodes contain prolonged static segments, predominantly near their beginning and end, with occasional occurrences during task execution. Mid‑trajectory inactivity often indicates hesitation or pauses during data collection, potentially encouraging the model to learn uninformative or suboptimal action patterns.

To address these issues, we design a two‑stage data cleaning and quality inspection pipeline. The first stage detects file corruption, black frames, occlusions, static frames, and state‑action misalignment, and removes unqualified data at the trajectory level. The second stage detects abrupt numerical changes and anomalous values, and removes abnormal data at the frame level. This pipeline adopts a coarse‑to‑fine filtering strategy that preserves the temporal continuity and state‑action consistency of the cleaned data while maintaining a stable data retention rate.

Overall, systematic curation, format standardization, and two‑stage quality assurance yield approximately 1,800 hours of high‑quality robotic manipulation data for pretraining EWAM on tabletop manipulation. Spanning diverse robot platforms, objects, and task categories, the resulting corpus combines reliable state‑action synchronization with broad behavioral coverage. It therefore provides a solid foundation for learning generalizable visuomotor representations and supports transfer to downstream tasks.

Appendix H Egocentric Reconstruction and Retargeting Details

This appendix gives the coordinate definitions and implementation details for the demonstration construction in Section 6.2.

Hand representation and interaction reference.

Let s\in\{\mathrm{L},\mathrm{R}\} index the hands, W the reconstruction world frame, H_{s} the wrist frame, and G the robot base frame. The recovered wrist pose is \mathbf{T}^{W}_{H_{s}}(t)=[\mathbf{R}^{W}_{H_{s}}(t),\mathbf{p}^{W}_{H_{s}}(t)]; the articulated joints \mathbf{J}^{H_{s}}(t) are expressed locally. A local joint is placed in the world as \mathbf{j}^{W}=\mathbf{R}^{W}_{H_{s}}\mathbf{j}^{H_{s}}+\mathbf{p}^{W}_{H_{s}}. We estimate a fixed local interaction offset \bar{\mathbf{c}}_{s}^{H} as the median thumb–index midpoint in valid closed-hand frames, giving

\mathbf{c}_{s}^{W}(t)=\mathbf{R}^{W}_{H_{s}}(t)\bar{\mathbf{c}}_{s}^{H}+\mathbf{p}^{W}_{H_{s}}(t).

where t is the frame index and \mathbf{c}_{s}^{W}(t) is the interaction reference point of hand s in the world frame. The point follows wrist translation and rotation while remaining independent of instantaneous finger articulation. Its use assumes that one wrist-local interaction reference is appropriate for the episode.

Shared robot-frame mapping.

One episode-level rigid transform (\mathbf{C}_{G\leftarrow W},\mathbf{b}_{G\leftarrow W}) places both hands in the robot workspace. The tool targets are

\displaystyle\mathbf{p}^{\star}_{t,s} \\ \displaystyle=\mathbf{C}_{G\leftarrow W}\mathbf{c}_{s}^{W}(t)+\mathbf{b}_{G\leftarrow W}, \\ \displaystyle\mathbf{R}^{\star}_{t,s} \\ \displaystyle=\mathbf{C}_{G\leftarrow W}\mathbf{R}^{W}_{H_{s}}(t)\mathbf{B}_{s}.

where \mathbf{p}^{\star}_{t,s} and \mathbf{R}^{\star}_{t,s} are the target TCP position and orientation of hand s at frame t in the robot base frame. Here \mathbf{C}_{G\leftarrow W} aligns the axes and \mathbf{b}_{G\leftarrow W} sets the placement using a real-robot reference trajectory. The transform is shared across hands and has unit scale. The fixed basis \mathbf{B}_{s} maps the wrist orientation to the tool orientation using an episode-consistent closing axis and an orthogonalized approach axis. The targets refer to the nominal pinch-center TCP and are held fixed during joint optimization.

Sequence optimization.

Let Q=\{\mathbf{q}_{t}\}_{t=0}^{T-1}, with \mathbf{q}_{t}\in\mathbb{R}^{14} containing seven joints per arm. The retargeter seeks to minimize

\underset{Q\in\mathcal{F}_{\mathrm{solve}}}{\operatorname{minimize}}\;\sum_{t=0}^{T-1}\left[\lambda_{\mathrm{task}}E_{\mathrm{task}}(\mathbf{q}_{t})+\lambda_{\mathrm{post}}E_{\mathrm{post}}(\mathbf{q}_{t})\right]+\lambda_{\mathrm{temp}}E_{\mathrm{temp}}(Q).

where T is the number of frames and \lambda_{\mathrm{task}}, \lambda_{\mathrm{post}}, and \lambda_{\mathrm{temp}} are the weights of the corresponding terms. The task term E_{\mathrm{task}} groups TCP position error and closing- and approach-axis errors outside phase-dependent tolerances. The posture term E_{\mathrm{post}} regularizes redundant shoulder–elbow and wrist configurations around a real-robot reference posture. The temporal term E_{\mathrm{temp}} penalizes velocity, acceleration, and jerk through finite differences on the original 30 Hz timeline. \mathcal{F}_{\mathrm{solve}} denotes the constraints enforced during the solve, including joint, velocity, and acceleration bounds. Workspace and collision conditions are also checked on the returned trajectory. This is a grouped description of the objective; the returned reference need not be a global minimizer. Output admissibility is reported separately in Section 6.3.

Visual tracking and temporal processing.

The visual branch applies WiLoR hand detection [42] and ByteTrack association [72] to fisheye-rectified images. We retain detections for each anatomical hand, reject tracks dominated by image boundaries, and interpolate short detection gaps only when the endpoint detections have sufficient overlap. The resulting contiguous clips are supplied to HaWoR. Short, bounded gaps in wrist-local articulation are interpolated, and valid articulation segments are smoothed with a Savitzky–Golay filter [45]. World-frame wrist poses, wrist-local geometry, and reconstruction-source and validity indicators are retained.

Staged optimization and initialization.

We solve sparse quadratic subproblems within a trust region after local linearization, using multiple initial inverse-kinematics candidates and staged warm starts. The selected refinement first regularizes the full trajectory and then increases position-tracking priority. A final prefix refinement frees the initial joint pose and uses a stable later segment to initialize the first 1.2 s, keeping the remainder fixed. This avoids locking the sequence to an unfavorable first-frame solution. Later frames are available because the procedure operates offline. Throughout refinement, the task-space targets, gripper commands, and original timeline are held fixed.

Articulated-hand adaptation.

For articulated robot hands, an additional branch aligns palm frames and hand scales, then solves fingertip-position inverse kinematics with joint limits and temporal regularization. The G1-D two-finger interface instead uses the inferred binary gripper commands alongside the arm trajectory.

Output audit.

Before export, we check finite values, joint and derivative limits, workspace conditions, collision clearance at recorded and interpolated configurations, FK consistency, and preservation of target, gripper, and time signals. The exported trajectories are reference actions, distinct from controller commands and hardware measurements. Grasp states are inferred from image-space evidence and do not measure contact forces. Mechanical admissibility is evaluated under the tested robot model and sampled configurations, separately from physical task performance.