WING überträgt egozentrisches Videowissen über interaktionszentrierte spektrale Latentführung auf Roboter-Policies

\addtolist

[]The Hong Kong University of Science and Technology Hong Kong SAR, China\affiliationlist\affiliationformat\addtolist[†]Equal contribution\contributionlist\contributionformat\addtolist[🖂]Correspondence: \contributionlist\contributionformat\metadata[ Project Page]https://mikuz12.github.io/wing/

Abstract

Learning general-purpose robot policies requires large-scale real-world interaction data, yet collecting robot demonstrations through teleoperation remains expensive and difficult to scale. Egocentric videos provide a rich source of human interaction experience that shares task-relevant semantics with robotic manipulation, creating an opportunity to align human and robot actions in a shared latent action space for cross-embodiment knowledge transfer. However, existing latent action approaches typically infer actions through reconstruction between consecutive frames, which are not inherently interaction-centric and can be dominated by nuisance variations, such as ego-camera motion. Moreover, although human interactions and robot actions share interaction semantics, they often exhibit substantially different temporal dynamics, making direct transfer to robot policies challenging. To address these challenges, we propose WING (World Action Learning via INteraction-Centric Spectral Latent Guidance), a framework for transferring interaction knowledge from egocentric videos to robot policies. First, we introduce an interaction–motion decoupling mechanism that separates interaction-relevant signals from task-irrelevant motion and selectively distills the interaction-centric components into latent actions. Second, motivated by the observation that cross-embodiment task semantics are primarily encoded in slowly varying temporal structures, we identify shared components between egocentric latent actions and robot behaviors in the spectral domain and use them as guidance for action generation at inference time. With these designs, WING achieves average success rates of 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1, while also demonstrating strong performance across four real-world tasks under diverse generalization settings. These results demonstrate that WING can effectively distill embodied interaction knowledge from large-scale egocentric videos and transfer it to robot control, providing a scalable pathway for acquiring physical interaction knowledge from human experience.

Introduction

Building generalist robot models that can reliably operate across diverse real-world environments requires large-scale and diverse training data (Intelligence et al., 2025; Kim et al., 2024; Li et al., 2026b; Ye et al., 2026b; Shou et al., 2026; Hu et al., 2026). Directly collecting such data through robot teleoperation remains costly and difficult to scale. Human videos (Deng and Zhou, 2026; Goyal et al., 2017; Wang et al., 2023; Punamiya et al., 2026; Grauman et al., 2022), in contrast, contain abundant embodied interactions across diverse tasks and environments, offering a potentially scalable source of experience for robot learning. Egocentric videos (Hoque et al., 2026; Punamiya et al., 2026; Zheng et al., 2026b; Grauman et al., 2022; Wang et al., 2023; Damen et al., 2021) are particularly appealing in this context, as they capture rich human-object interactions from a first-person perspective that closely resembles the visual observations available to robots.

Transferring human experiences to robot control remains challenging because egocentric videos lack direct and generalizable action supervision. Explicit annotations such as hand poses are often tightly coupled to human morphology and kinematics, limiting their ability to provide transferable supervision for robot action learning. Latent action learning (Bu et al., 2025; Ye et al., 2025; Wei et al., 2026; Gao et al., 2026) offers a more flexible alternative by inferring action-like representations directly from visual transitions and encoding them in a latent space, without requiring explicit action labels. However, extracting transferable interaction knowledge from visual transitions is fundamentally different from merely explaining how images change over time. Existing latent action models are typically trained to reconstruct future observations, encouraging them to capture any visually predictive variation rather than selectively modeling interaction-relevant dynamics. This leads to a central question: How can latent action learning extract interaction-centric knowledge from large-scale egocentric videos for transferable robot learning?

We first examine the sources of nuisance factors in latent actions extracted from egocentric videos. We observe that a major source stems from task-irrelevant observer motion. Unlike robot manipulation settings where the head camera is typically mounted on the robot and remains stable, human-captured egocentric videos often exhibit substantial viewpoint changes caused by rapid head and body movements. These camera-induced variations can become entangled with interaction-related dynamics and even dominate the learned latent representations (Xu et al., 2026; Lee et al., 2026). However, transferable information primarily lies in the underlying hand-object interactions rather than task-irrelevant motion. This motivates us to design a latent action model that can disentangle interaction dynamics from observer-induced variations (Wei et al., 2026; Lin et al., 2026).

Moreover, we also need to account for differences in execution dynamics between human and robot interactions. Even when meaningful interaction structures are captured, human and robot behaviors can differ in how the same interaction is executed due to differences in embodiment, execution speed, and control granularity. Such discrepancies are particularly pronounced in fine-grained temporal patterns, which can vary substantially even when the underlying interaction semantics are shared. We therefore analyze the temporal structures of human egocentric interactions and robot actions in the frequency domain. Our analysis reveals substantially stronger correspondence in the low-frequency components of semantically matched human-robot interaction pairs. Motivated by this observation, we treat the low-frequency components of latent actions as transferable interaction information and use them as guidance for robot policy learning and action generation.

Motivated by these observations, we introduce WING (World Action Learning via INteraction-Centric Spectral Latent Guidance), a framework for extracting transferable interaction dynamics from large-scale egocentric videos and leveraging them for robot learning. WING first learns interaction-centric latent actions by suppressing sensitivity to observer motion while preserving interaction-relevant changes. Specifically, we decompose visual transitions into global observer-induced motion and local interaction-related motion, and selectively distill the latter into the latent action representation. WING then decomposes latent action trajectories in the frequency domain using the discrete cosine transform (DCT) (Ahmed et al., 1974) and retains low-frequency components that exhibit more consistent structure across human and robot executions. At inference time, a lightweight predictor estimates this spectral guidance from the current observation and task context, which is then used to condition robot action generation.

We conduct extensive evaluations of both the representations learned by WING-LAM and the downstream performance of the complete WING framework. Our analysis shows that WING-LAM is substantially less sensitive to observer motion while preserving interaction-related structure, and that low-frequency latent dynamics exhibit stronger correspondence between semantically matched human and robot executions. We further evaluate WING across multiple simulated benchmarks and real-world manipulation tasks. In simulation, WING achieves average success rates of 99.2% on LIBERO (Liu et al., 2023) and 93.80% on RoboTwin 2.0 (Chen et al., 2025), and reaches 57.7% on RoboCasa–GR1 (Nasiriany et al., 2024) after pre-training on egocentric video data, outperforming strong baselines including \pi_{0.5} (Intelligence et al., 2025) and LingBot-VA (Li et al., 2026b). In real-world experiments, we evaluate WING on four bimanual manipulation tasks under both standard and diverse generalization settings, where it maintains strong performance across variations. Ablation studies further confirm that both interaction-centric latent action learning and low-frequency spectral guidance contribute consistently to downstream policy performance.

Overall, our contributions are threefold:

• We show that observer-induced variations can become entangled with interaction-relevant dynamics in egocentric latent actions, and further reveal that low-frequency latent dynamics exhibit stronger correspondence across human and robot interactions.
• Motivated by these findings, we introduce WING, which combines interaction-centric latent action learning with spectral latent guidance to suppress observer-induced variations and exploit temporal structures that are more consistently shared across embodiments.
• We validate WING through extensive representation analyses and closed-loop experiments across simulated and real-world manipulation tasks. WING consistently outperforms representative robot policies and alternative egocentric pretraining strategies.

Related Work

Recent vision-language-action (VLA) and world-action models (WAMs) have demonstrated strong generalization by combining large-scale vision-language or video pretraining with robot demonstrations (Brohan et al., 2023; Kim et al., 2024; Black et al., 2024; Intelligence et al., 2025; NVIDIA et al., 2025; Ye et al., 2026b; Yuan et al., 2026b; Li et al., 2026b). However, their performance still relies heavily on robot data, which remains expensive to collect and difficult to scale across diverse tasks and environments. To alleviate this limitation, recent studies have explored egocentric human videos as a scalable source of interaction experience. Prior works have explored this direction through representation learning, behavior transfer, and latent-action modeling (Nair et al., 2022; Ma et al., 2022; Kareer et al., 2025; Zheng et al., 2026b; Ye et al., 2025; Bu et al., 2025; Wei et al., 2026; Xu et al., 2026). However, ego-centric transitions entangle interaction dynamics with observer-induced camera motion, making it difficult to identify transferable manipulation structure. Existing approaches either model full visual transitions or reduce action-irrelevant variation, while other works exploit camera motion as an active-perception cue (Wei et al., 2026; Xu et al., 2026; Lin et al., 2026), without explicitly separating observer-specific motion from interaction dynamics.

WING addresses this gap by learning interaction-centric latent action guidance for ego-to-robot transfer. It disentangles observer motion from interaction dynamics and extracts low-frequency temporal structure that is more consistently shared between human and robot executions. A more comprehensive discussion of related work is provided in Appendix B.

Methods

In this section, we present the technical details of WING, including interaction-centric latent action learning, the spectral latent guidance module, and the overall training pipeline. Together, these components extract transferable interaction structure from large-scale egocentric videos and use it as structured latent guidance for robot action learning.

Overview

A robot policy \pi_{\theta}(\mathbf{a}_{t:t+m}\mid\mathbf{l},\mathbf{o}_{t},\mathbf{s}_{t}) maps a language instruction \mathbf{l}, visual observation \mathbf{o}_{t}, and robot proprioceptive state \mathbf{s}_{t} to an executable action chunk \mathbf{a}_{t:t+m}. Our goal is to learn generalizable action knowledge from large-scale egocentric videos through latent action modeling and transfer it to downstream robot manipulation policies. As discussed above, this transfer faces two key challenges: (1) reconstruction-based latent action models entangle interaction-relevant dynamics with task-irrelevant motion, particularly ego-camera motion; and (2) discrepancies in temporal dynamics between human interactions and robot actions hinder direct cross-embodiment transfer.

To address these challenges, WING comprises two key components. First, WING-LAM is an interaction-centric latent action model that encodes egocentric visual transitions (\mathbf{o}^{\mathrm{ego}}_{t},\mathbf{o}^{\mathrm{ego}}_{t+\delta}) into latent actions \mathbf{z}_{t}=E_{\phi}^{\mathrm{WING\text{-}LAM}}(\mathbf{o}^{\mathrm{ego}}_{t},\mathbf{o}^{\mathrm{ego}}_{t+\delta}) while suppressing task-irrelevant motion, thereby encouraging the latent representation to capture interaction-relevant dynamics. Second, the Spectral Latent Guidance Module decomposes latent action trajectories \mathbf{z}_{t:t+H} in the spectral domain and retains the first K low-frequency components, \mathbf{g}^{\mathrm{low}}_{t}=\left[\operatorname{DCT}(\mathbf{z}_{t:t+H})\right]_{0:K-1}, as transferable latent guidance. By suppressing rapidly varying high-frequency components, this module isolates slowly varying temporal structures that are more consistently shared across human and robot interactions.

At inference time, a lightweight predictor P_{\psi} estimates the spectral guidance \widehat{\mathbf{g}}^{\mathrm{low}}_{t}=P_{\psi}(\mathbf{l},\mathbf{o}_{t},\mathbf{s}_{t}), which conditions the world-action model \pi_{\theta}(\mathbf{a}_{t:t+m}\mid\mathbf{l},\mathbf{o}_{t},\mathbf{s}_{t},\widehat{\mathbf{g}}^{\mathrm{low}}_{t}) for fine-grained robot action generation. In this way, WING transfers knowledge learned from egocentric human videos as a high-level action prior, while allowing the robot policy to generate embodiment-specific fine-grained controls, thereby bridging large-scale human interaction and downstream robot manipulation.

Learning Interaction-Centric Latent Actions

Egocentric videos contain both observer-induced motion and hand-object interactions. Consequently, a latent action learned to reconstruct the full transition can encode both sources of change (Zhang et al., 2025). However, for robot guidance, this calls for the latent to retain interaction dynamics while reducing its sensitivity to observer motion. WING-LAM therefore uses an explicit motion-routing design that isolates interaction dynamics from observer motion and distills the resulting interaction latent into an RGB-only student.

Camera–Interaction Motion Decomposition. Observer motion and hand–object interaction exhibit different spatial motion patterns in egocentric videos: observer motion induces coherent image-wide changes, while interaction motion is more localized. We exploit this distinction to construct motion-aware supervision. Specifically, we decompose each frame transition into a global observer-motion component and a local interaction-motion component. Given two video frames \mathbf{o}_{t} and \mathbf{o}_{t+\delta}, an offline point tracker estimates point correspondences: \mathbf{p}_{t,i},\mathbf{p}_{t+\delta,i}\in\mathbb{R}^{2}, where i identifies the same tracked point across frames. Training-time region masks separate background tracks from hand–object interaction tracks. Using the background tracks, we estimate an invertible global image warp W_{t\rightarrow t+\delta} as the observer-motion target, capturing the dominant coherent transformation between the two frames. After compensating this global motion, the remaining displacement of each point represents motion unexplained by the observer movement:

\mathbf{r}_{t,i}=\underbrace{W_{t\rightarrow t+\delta}^{-1}(\mathbf{p}_{t+\delta,i})}_{\text{camera-compensated location}}-\quad\mathbf{p}_{t,i},

where \mathbf{r}_{t,i}\in\mathbb{R}^{2} denotes the residual displacement expressed in the first frame’s coordinates, which serves as the interaction-motion target.

Motion-Routed Latent Learning. These decomposed motion signals provide privileged supervision for learning a factorized latent representation. We then train a two-branch teacher to encode visual features and tracked point coordinates, producing camera and interaction latents, \mathbf{z}^{T}_{t,\mathrm{cam}}\in\mathbb{R}^{D_{c}} and \mathbf{z}^{T}_{t,\mathrm{int}}\in\mathbb{R}^{D_{i}}, where the superscript T denotes the teacher and D_{c},D_{i} are the respective latent dimensions. The camera branch predicts the global warp, while the interaction branch predicts residual displacements, and their decoded motions jointly reconstruct the tracked locations in the second frame. Although motion routing separates the two sources, observer-specific cues may still leak into the interaction latent; we therefore apply camera interventions that preserve interaction dynamics and enforce latent consistency.

Interaction Distillation. Although the teacher learns interaction-centric representations using privileged motion cues, these cues are unavailable during deployment. We therefore distill the teacher’s interaction latent into an RGB-only student S_{\phi} that directly extracts latent actions from frame pairs. For each pair (\mathbf{o}_{t}^{\mathrm{ego}},\mathbf{o}_{t+\delta}^{\mathrm{ego}}), we construct a camera-perturbed view by applying a camera-like global warp to the second frame, yielding \tilde{\mathbf{o}}_{t+\delta}^{\mathrm{ego}}. The two pairs share the same interaction target \mathbf{z}^{T}_{t,\mathrm{int}} while differing only in observer-induced motion. The student then encodes the original and augmented pairs as: \mathbf{z}_{t}^{\mathrm{nat}}=S_{\phi}(\mathbf{o}_{t}^{\mathrm{ego}},\mathbf{o}_{t+\delta}^{\mathrm{ego}}) and \mathbf{z}_{t}^{\mathrm{aug}}=S_{\phi}(\mathbf{o}_{t}^{\mathrm{ego}},\tilde{\mathbf{o}}_{t+\delta}^{\mathrm{ego}}), where both outputs lie in \mathbb{R}^{D_{i}}, and \mathrm{nat} and \mathrm{aug} denote the original and augmented inputs, respectively. We optimize the student using:

\mathcal{L}_{\mathrm{distill}}=\frac{1}{2D_{i}}\sum_{v\in\{\mathrm{nat},\mathrm{aug}\}}\left\|\mathbf{z}_{t}^{v}-\mathbf{z}^{T}_{t,\mathrm{int}}\right\|_{2}^{2}+\frac{\lambda_{\mathrm{view}}}{D_{i}}\left\|\mathbf{z}_{t}^{\mathrm{nat}}-\mathbf{z}_{t}^{\mathrm{aug}}\right\|_{2}^{2},

where v indexes the two inputs and \lambda_{\mathrm{view}} weights their prediction consistency. The first term transfers the teacher’s interaction representation; the second encourages the student to retain that representation under camera-like image warps. Details of implementation are provided in Appendix C.1.

Overall, the above design separates observer-induced variation from interaction dynamics, yielding interaction-centric latent actions that capture transferable manipulation structure and serve as the foundation for the subsequent spectral guidance module.

Spectral Latent Action Guidance

Although WING-LAM extracts interaction-centric latent actions, semantically matched human and robot executions can differ in temporal dynamics. We find that low-frequency latent components exhibit stronger cross-embodiment correspondence than full trajectories, and therefore retain these components as spectral guidance for robot action generation.

Spectral Latent Representation. Given visual observations sampled at a fixed interval \Delta t, WING-LAM encodes H successive transitions into \mathbf{Z}_{t}=[\mathbf{z}_{t},\ldots,\mathbf{z}_{t+H-1}]^{\top}\in\mathbb{R}^{H\times D_{i}}, where \mathbf{z}_{t+h}=S_{\phi}(\mathbf{o}_{t+h},\mathbf{o}_{t+h+1}). An orthonormal DCT is then applied along the temporal dimension, and the lowest K components are retained as the guidance target \mathbf{g}_{t}^{\mathrm{low}}. This truncation affects only the latent guidance, not the robot actions.

Predicting Low-Frequency Guidance. Since \mathbf{g}_{t}^{\mathrm{low}} requires future observations, we train a guidance predictor P_{\psi} to estimate it from the current observation and instruction, together with proprioception when available. Writing these inputs as \mathbf{c}_{t}, we minimize the mean squared prediction error:

\widehat{\mathbf{g}}_{t}^{\mathrm{low}}=P_{\psi}(\mathbf{c}_{t}),\qquad\mathcal{L}_{\mathrm{pred}}=\frac{1}{KD_{i}}\left\|\widehat{\mathbf{g}}_{t}^{\mathrm{low}}-\mathbf{g}_{t}^{\mathrm{low}}\right\|_{F}^{2},

where \|\cdot\|_{F} denotes the Frobenius norm. During robot training, we jointly optimize this loss and the action-generation objective. Future frames are only used to construct training targets.

Conditioning Robot Actions. We convert the predicted guidance \widehat{\mathbf{g}}_{t}^{\mathrm{low}} into \mathbf{U}_{t}\in\mathbb{R}^{K\times D_{\mathrm{hid}}}, where D_{\mathrm{hid}} is the action model’s hidden dimension. Each frequency coefficient vector is projected to this dimension, normalized with LayerNorm, and given a learned embedding identifying its frequency mode. The action model attends to these guidance tokens through a gated residual update:

\mathbf{H}_{a}^{(l)}\leftarrow\mathbf{H}_{a}^{(l)}+\gamma_{l}\operatorname{Attn}\!\left(Q=\operatorname{LayerNorm}(\mathbf{H}_{a}^{(l)}),K=\mathbf{U}_{t},V=\mathbf{U}_{t}\right),

where \mathbf{H}_{a}^{(l)} denotes the action hidden sequence at layer l, and the learned scalar \gamma_{l} controls the magnitude of latent guidance.

Training Strategy

We train the world action model (WAM) in two stages: pretraining on egocentric videos, followed by robot policy learning from robot demonstrations.

Egocentric Pretraining. Using our curated egocentric data, we adopt Wan2.2 (Wan et al., 2025) and train the video expert for future-frame generation and the guidance predictor P_{\psi} for low-frequency spectral guidance prediction. The two pathways learn visual and interaction dynamics through the generation objective and latent guidance prediction, respectively. Full details of egocentric pretraining are provided in Appendix C.2.

Robot Policy Learning. After pretraining, we transfer the pretrained visual backbone and guidance predictor to the robot policy and jointly train them with the action expert on robot demonstrations. The guidance predictor is supervised on robot data, while the WING-LAM encoder remains frozen. The predicted guidance conditions downstream robot action generation during both training and inference. This stage adapts the interaction dynamics learned from egocentric videos to robot control, with robot demonstrations providing direct supervision for executable actions.

Experiments

In this section, we evaluate WING from both representation and policy perspectives. Specifically, we aim to answer the following four questions:

1. What structures in egocentric latent actions reveal transferable interaction knowledge?
2. Can WING effectively transfer latent action knowledge to improve robot performance?
3. How do interaction-centric latent actions and spectral latent action guidance contribute?
4. How effective is WING’s pretraining strategy for downstream robot policy learning?

Experimental Setup

We first evaluate whether WING-LAM reduces observer-related variation while retaining interaction-relevant semantics, and then investigate which temporal structures in latent trajectories exhibit stronger cross-embodiment correspondence between human and robot executions. We compare WING-LAM with representative LAM baselines using motion readout, camera perturbations, and action classification, and analyze semantic similarity between matched and mismatched human–robot trajectories across frequency bands. Evaluation details are provided in Appendix D.1 and Appendix D.2. We further evaluate WING on LIBERO (Liu et al., 2023), RoboTwin 2.0 (Chen et al., 2025), RoboCasa–GR1 (Nasiriany et al., 2024), and a real-world dual-arm platform. We compare against representative vision-language-action, latent-action, and world-action modeling methods, reporting success rate in simulation and both task completion and progress in the real world. Full evaluation protocols are provided in Appendix D.

(Q1): Characterizing Egocentric Latent Structure For Robot Learning.

We first examine whether WING-LAM suppresses observer-related variation while preserving interaction semantics. As shown in Figure 3(a), both its teacher and student retain strong interaction-motion information but substantially less camera-motion information than DreamDojo (Gao et al., 2026). Notably, although CD-LAM (Wei et al., 2026) is explicitly designed to reduce action-irrelevant variation, it still retains more camera-motion information under our evaluation protocol. This selectivity persists under direct perturbation: as shown in Figure 3(b), the student exhibits the lowest latent sensitivity under increasing camera motion. Importantly, this robustness does not come at the expense of action semantics, as WING-LAM achieves the highest average action-classification accuracy on LARYBench (Nie et al., 2026) (Figure 3(c)). We then ask which temporal components of these interaction-centric trajectories are most consistent across human and robot executions. As shown in Figure 3(d), low-frequency components yield a substantially larger separation between semantically matched and mismatched pairs than either the full trajectory or the high-frequency components, indicating that shared interaction structure is concentrated more strongly in low-frequency dynamics and motivating their use as guidance for action generation.

(Q2): Robot Policy Performance

Results on Simulation.

Across single-arm, bimanual, and humanoid manipulation benchmarks, WING achieves average success rates of 99.20\% on LIBERO (Table 2(a)), 93.80\% on RoboTwin 2.0 (Table 2(b)), and 57.7\% on RoboCasa–GR1 (Table 1), outperforming the strongest baselines on all three. Notably, its advantage extends to LIBERO’s long-horizon suite and RoboTwin’s randomized scenes, suggesting that the gains are not confined to shorter tasks or clean visual conditions. Together, these results demonstrate the effectiveness of WING for robot manipulation across diverse embodiments and task settings. Full results for RoboTwin 2.0 and RoboCasa are provided in Appendices E.4.2 and E.4.1, respectively.

Results on the real-world robot.

As illustrated in Figure 4, WING achieves the highest average success rate of 75.0% under the standard setting. Across both standard and generalization settings, WING consistently outperforms FastWAM and GR00T-N1.7 in average success rate and progress score. Under generalization settings, WING achieves comparable average success to the strong baseline \pi_{0.5}, while attaining a slightly higher average progress score. These results demonstrate the effectiveness of WING in real-world manipulation and its ability to maintain strong performance under diverse generalization settings.

(Q3): Ablation Studies on WING-LAM and Spectral Guidance.

Ablation on observer debiasing and spectral guidance.

Table 3 evaluates the contribution of both components under a controlled setting without ego-pretraining of the WAM. When WING-LAM is ablated, we replace it with the pretrained DreamDojo LAM while keeping the spectral guidance unchanged. When DCT module variant is ablated, we retain the same latent encoder but bypass DCT and use the full time-domain latent trajectory directly as guidance. Removing either component reduces average performance across both LIBERO and real-world evaluations. The gains from the DCT variant show that frequency-domain decomposition and low-frequency retention yield more effective guidance by emphasizing slowly varying interaction dynamics. The additional gains from WING-LAM indicate that latent actions with reduced observer-induced variation provide more useful guidance for robot control. This advantage of WING-LAM becomes even clearer in the ego-pretraining analysis below, where it leads to larger improvements.

Ablation on the number of DCT components.

We further investigate how much spectral information is required for effective latent guidance by varying the number of retained DCT coefficients K\in\{2,4,8\}, where K=8 corresponds to the full spectrum. As shown in Figure 5(a), K=4 consistently performs best across all suites. Retaining only two coefficients is overly restrictive and discards useful temporal dynamics, while the full spectrum introduces additional high-frequency components without improving downstream control. These results suggest that effective latent guidance requires a sufficient, but not excessive, spectral bandwidth, with K=4 providing the best balance between preserving control-relevant temporal structure and filtering variation that is less consistently shared across human and robot executions.

(Q4): Analysis of Ego Pre-training Strategies.

Having established WING’s downstream performance, we further examine whether its interaction-centric spectral guidance also provides an effective pre-training strategy for WAMs from egocentric videos. We compare pre-training methods under the same 200h budget which differ in how video-derived information supervises the WAM. In Figure 5(b), Vid-P. performs video-only pre-training, isolating the effect of exposure to egocentric videos themselves. HP-P. uses hand-pose labels as supervision. LA-P. follows LAPA (Ye et al., 2025) which uses latent actions as direct action-prediction targets. In contrast, Ours predicts spectral latent guidance to condition action generation. To isolate the effect of suppressing observer-induced variation, w/o OD. denotes variants in which WING-LAM is replaced with DreamDojo LAM. Implementation details are provided in Appendix C.2.

As shown in Figure 5(b), video-only pre-training consistently underperforms our method, with particularly clear gaps on the real-world standard and generalization settings, indicating that the gains of egocentric pre-training cannot be explained by video exposure alone. Replacing WING-LAM with a non-debiased latent-action model further degrades performance in most settings, supporting the importance of suppressing observer-induced variation. Direct latent-action prediction also performs worse, possibly reflecting the gap between visual transition representations and executable robot actions. Together, these results suggest that WING is most effective when ego-derived latent actions serve as auxiliary guidance, rather than as direct action targets, while the robot policy retains its original action-prediction objective. We further analyze the effectiveness and efficiency of pre-training in Appendix E.3.3 and E.3.4 by examining downstream validation loss after different amounts of egocentric pretraining and downstream performance under varying amounts of robot training data. These analyses show that WING not only learns effective representations during pre-training, but also retains stronger downstream performance as robot supervision becomes more limited.

Limitations and Future Work

Although WING provides an effective way to exploit egocentric videos for robot learning, several limitations remain. First, constructing the WING-LAM teacher relies on external motion cues and an image-space camera–interaction decomposition. The dominant observer motion is approximated by a single global affine warp, which can become less accurate under strong parallax, large depth variation, or pronounced 3D camera motion, potentially introducing noise into the interaction supervision. Second, latent-action pretraining and spectral guidance add training cost and pipeline complexity. Future work could explore more self-contained, depth-aware motion modeling and more efficient guidance mechanisms.

Conclusion

In this work, we study how to effectively transfer interaction knowledge from egocentric human videos to robot learning. We show that transferable latent actions require suppressing observer-induced variation and capturing temporal structures that remain consistent across human and robot executions. Based on these findings, we propose WING, a framework that learns and transfers interaction-centric latent knowledge from egocentric videos. WING suppresses observer-induced variations through interaction-centric latent action learning and leverages low-frequency latent dynamics that are more consistent across embodiments through spectral latent guidance. Extensive experiments across simulation and real-world manipulation benchmarks demonstrate that WING effectively transfers embodied interaction knowledge from egocentric videos to robot control, achieving strong performance and generalization across diverse tasks. Our results suggest that scaling robot learning with human experience requires not simply modeling all observed dynamics, but identifying interaction structures that remain consistent across observers and embodiments.

References

1. Ahmed et al. (1974) N. Ahmed, T. Natarajan, and K. R. Rao Discrete cosine transform. IEEE Transactions on Computers.
2. Bi et al. (2026) H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, et al. Motus: a unified latent action world model. In CVPR,
3. Black et al. (2024) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. \pi_{0}: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164.
4. Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818.
5. Bu et al. (2025) Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li Univla: learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111.
6. Chen et al. (2026a) J. Chen, K. Wang, K. Chen, S. Chen, F. Gao, W. Tang, Z. Li, W. Liu, Z. Yao, B. Li, Y. Xu, and C. Yu LaWAM: latent world action models for efficient dynamics-aware robot policies. arXiv preprint arXiv:2606.15768.
7. Chen et al. (2025) T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, et al. RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088.
8. Chen et al. (2026b) X. Chen, H. Wei, P. Zhang, C. Zhang, K. Wang, Y. Guo, R. Yang, Y. Wang, X. Xiao, L. Zhao, et al. Villa-x: enhancing latent action modeling in vision-language-action models. In ICLR,
9. Community (2026) S. Community StarVLA: a lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014.
10. Dai et al. (2026) W. Dai, K. Lan, J. Zhou, B. Zhao, X. Su, J. Tong, W. Guan, and S. Yang Conla: contrastive latent action learning from human videos for robotic manipulation. arXiv preprint arXiv:2602.00557.
11. Damen et al. (2021) D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. The epic-kitchens dataset: collection, challenges and baselines. TPAMI.
12. Deng and Zhou (2026) Y. Deng and D. Zhou HumanNet: scaling human-centric video learning to one million hours. arXiv preprint arXiv:2605.06747.
13. Fischler and Bolles (1981) M. A. Fischler and R. C. Bolles Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM.
14. Fu et al. (2024) Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn Humanplus: humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454.
15. Gao et al. (2026) S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W. Tseng, Y. Dong, K. Mo, C. Lin, et al. Dreamdojo: a generalist robot world model from large-scale human videos. arXiv preprint arXiv:2602.06949.
16. Goyal et al. (2017) R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV,
17. Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: around the world in 3,000 hours of egocentric video. In CVPR,
18. Harley et al. (2025) A. W. Harley, Y. You, X. Sun, Y. Zheng, N. Raghuraman, Y. Gu, S. Liang, W. Chu, A. Dave, P. Tokmakov, S. You, R. Ambrus, K. Fragkiadaki, and L. J. Guibas AllTracker: efficient dense point tracking at high resolution. In ICCV,
19. Hoque et al. (2026) R. Hoque, P. Huang, D. Yoon, J. Zhang, et al. Egodex: learning dexterous manipulation from large-scale egocentric video. In ICLR,
20. Hu et al. (2026) B. Hu, Z. Li, R. Shao, J. Chen, A. H. Liu, W. Zheng, and L. Nie From abstraction to instantiation: learning behavioral representation for vision-language-action model. arXiv preprint arXiv:2605.22671.
21. Intelligence et al. (2025) P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. \pi_{0.5}: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054.
22. Kareer et al. (2025) S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu Egomimic: scaling imitation learning via egocentric video. In ICRA,
23. Kim et al. (2026) M. J. Kim, Y. Gao, T. Lin, Y. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M. Liu, C. Finn, et al. Cosmos policy: fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163.
24. Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246.
25. Lee et al. (2026) J. M. Lee, T. Cho, L. Zhao, and J. Lee Why latent actions fail, and how to prevent it. arXiv preprint arXiv:2605.20223.
26. Li et al. (2026a) H. Li, G. Zhao, Y. Liu, H. Hou, G. Ye, T. Fang, C. Liu, S. Huang, J. Liu, X. Wang, et al. ACE-ego-0: unifying egocentric human and robotic data for vla pretraining. arXiv preprint arXiv:2606.17200.
27. Li et al. (2026b) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, et al. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998.
28. Li et al. (2024) Q. Li, Y. Liang, Z. Wang, L. Luo, X. Chen, M. Liao, F. Wei, Y. Deng, S. Xu, Y. Zhang, et al. Cogact: a foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650.
29. Lin et al. (2026) X. Lin, G. Zhong, T. Lu, Z. Ye, Y. Zhu, Z. Wu, and Y. Jiang ActiveMimic: egocentric video pretraining with active perception. arXiv preprint arXiv:2606.06194.
30. Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone LIBERO: benchmarking knowledge transfer for lifelong robot learning. In NeurIPS,
31. Liu et al. (2025) S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu Rdt-1b: a diffusion foundation model for bimanual manipulation. In ICLR,
32. Luo et al. (2026) H. Luo, W. Zhang, Y. Feng, S. Zheng, H. Xu, C. Xu, Z. Xi, Y. Fu, and Z. Lu Being-h0.7: a latent world-action model from egocentric videos. arXiv preprint arXiv:2605.00078.
33. Lyu et al. (2026) J. Lyu, K. Liu, X. Zhang, H. Liao, Y. Feng, W. Zhu, T. Shen, J. Chen, J. Zhang, Y. Dong, et al. Lda-1b: scaling latent dynamics action model via universal embodied data ingestion. arXiv preprint arXiv:2602.12215.
34. Ma et al. (2026) T. Ma, J. Zheng, Z. Wang, C. Jiang, A. Cui, J. Liang, and S. Yang DiT4DiT: jointly modeling video dynamics and actions for generalizable robot control. arXiv preprint arXiv:2603.10448.
35. Ma et al. (2023) Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman Liv: language-image representations and rewards for robotic control. In ICML,
36. Ma et al. (2022) Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang Vip: towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030.
37. Nair et al. (2022) S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta R3m: a universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601.
38. Nasiriany et al. (2024) S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu RoboCasa: large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523.
39. Nie et al. (2026) D. Nie, F. Chen, Q. Lv, J. Kuang, X. Li, X. Cao, and X. Cai Lary: a latent action representation yielding benchmark for generalizable vision-to-action alignment. arXiv preprint arXiv:2604.11689.
40. Niu et al. (2026) Y. Niu, H. Lv, X. Zhang, X. Wan, S. Gao, Y. Ai, H. Xu, Y. Hu, H. Zhang, Y. Xie, et al. EgoAERO: learning dexterous manipulation from a single egocentric video without object assets. arXiv preprint arXiv:2606.08057.
41. NVIDIA et al. (2025) NVIDIA, J. Bjorck, N. C. Fernando Castañeda, X. Da, R. Ding, L. ". Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu GR00T N1: an open foundation model for generalist humanoid robots. In ArXiv Preprint, External Links: 2503.14734
42. O’Neill et al. (2024) A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open x-embodiment: robotic learning datasets and rt-x models. In ICRA,
43. Punamiya et al. (2026) R. Punamiya, S. Kareer, Z. Liu, J. Citron, R. Qiu, X. Cai, A. Gavryushin, J. Chen, D. Liconti, L. Y. Zhu, et al. Egoverse: an egocentric human dataset for robot learning from around the world. arXiv preprint arXiv:2604.07607.
44. Shi et al. (2025) J. Shi, Z. Zhao, T. Wang, I. Pedroza, A. Luo, J. Wang, J. Ma, and D. Jayaraman Zeromimic: distilling robotic manipulation skills from web videos. In ICRA,
45. Shou et al. (2026) Q. Shou, F. Zhu, S. Chen, P. Yan, Z. Yan, Y. Miao, X. Pang, Z. Hong, R. Shi, H. Huang, et al. Halo: a unified vision-language-action model for embodied multimodal chain-of-thought reasoning. arXiv preprint arXiv:2602.21157.
46. Strang (1999) G. Strang The discrete cosine transform. SIAM Review.
47. Sun et al. (2026) J. Sun, W. Zhang, Z. Qi, S. Ren, Z. Liu, H. Zhu, G. Sun, X. Jin, and Z. Chen Vla-jepa: enhancing vision-language-action model with latent world model. arXiv preprint arXiv:2602.10098.
48. Team et al. (2024) O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213.
49. Wan et al. (2025) T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Yan, L. Huang, M. Feng, N. Zhang, P. Li, P. Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. Weng, T. Shen, W. Lin, W. Wang, W. Wang, W. Zhou, W. Wang, W. Shen, W. Yu, X. Shi, X. Huang, X. Xu, Y. Kou, Y. Lv, Y. Li, Y. Liu, Y. Wang, Y. Zhang, Y. Huang, Y. Li, Y. Wu, Y. Liu, Y. Pan, Y. Zheng, Y. Hong, Y. Shi, Y. Feng, Z. Jiang, Z. Han, Z. Wu, and Z. Liu Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314.
50. Wang et al. (2023) X. Wang, T. Kwon, M. Rad, B. Pan, I. Chakraborty, S. Andrist, D. Bohus, A. Feniello, B. Tekin, F. V. Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In ICCV,
51. Wang et al. (2026) Y. Wang, P. Lin, X. Chen, H. Yuan, Z. Liang, Y. Huang, A. Chen, Z. Lei, J. Zhang, T. Zhang, et al. Ego2Robot: scalable robot data synthesis from egocentric human data. arXiv preprint arXiv:2608.02580.
52. Wei et al. (2026) Y. Wei, K. Zhou, L. Mao, Z. Zhang, Z. Xu, Z. Xi, S. Liang, R. Han, Y. Yan, X. Wang, et al. Causally debiased latent action model for embodied action conditioned world models. arXiv preprint arXiv:2607.09185.
53. Wu et al. (2026) W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, F. Wang, Q. Zhu, H. Sun, Y. Wang, S. Ma, et al. A pragmatic vla foundation model. arXiv preprint arXiv:2601.18692.
54. Xu et al. (2026) R. Xu, Y. Zhang, J. Wang, Y. Wang, and J. Yu Motion-focused latent action enables cross-embodiment vla training from human egovideos. arXiv preprint arXiv:2606.18955.
55. Yan et al. (2026) G. Yan, J. Liu, Y. Fan, L. Cai, M. Liao, J. Zhang, and D. Fox Flex-\pi: a multi-stream world-action model with compute flexibility. arXiv preprint arXiv:2608.10860.
56. Yang et al. (2026) J. Yang, Y. Shi, H. Zhu, M. Liu, K. Ma, Y. Wang, G. Wu, T. He, and L. Wang Como: learning continuous latent motion from internet videos for scalable robot learning. In CVPR,
57. Yang et al. (2025) R. Yang, Q. Yu, Y. Wu, R. Yan, B. Li, A. Cheng, X. Zou, Y. Fang, X. Cheng, R. Qiu, et al. Egovla: learning vision-language-action models from egocentric human videos. arXiv preprint arXiv:2507.12440.
58. Ye et al. (2026a) A. Ye, B. Wang, C. Ni, G. Huang, G. Zhao, H. Li, H. Li, J. Li, J. Lv, J. Liu, et al. GigaWorld-policy: an efficient action-centered world–action model. arXiv preprint arXiv:2603.17240.
59. Ye et al. (2026b) S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922.
60. Ye et al. (2025) S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y. Chao, B. Y. Lin, et al. Latent action pretraining from videos. In ICLR,
61. Yuan et al. (2026a) H. Yuan, Z. Liang, A. Chen, Y. Wang, H. Li, P. Lin, Y. Huang, Z. Lei, T. Zhang, J. Zhang, et al. Qwen-robotmanip technical report: alignment unlocks scale for robotic manipulation foundation models. arXiv preprint arXiv:2606.17846.
62. Yuan et al. (2026b) T. Yuan, Z. Dong, Y. Liu, and H. Zhao Fast-WAM: do world action models need test-time future imagination?. arXiv preprint arXiv:2603.16666.
63. Zhang et al. (2025) C. Zhang, T. Pearce, P. Zhang, K. Wang, X. Chen, W. Shen, L. Zhao, and J. Bian What do latent action models actually learn?. In NeurIPS,
64. Zhang et al. (2022) L. Zhang, S. Zhou, S. Stent, and J. Shi Fine-grained egocentric hand-object segmentation: dataset, model, and applications. In ECCV,
65. Zheng et al. (2026a) J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, et al. X-vla: soft-prompted transformer as scalable cross-embodiment vision-language-action model. In ICLR,
66. Zheng et al. (2026b) R. Zheng, D. Niu, Y. Xie, J. Wang, M. Xu, Y. Jiang, F. Castañeda, F. Hu, Y. L. Tan, L. Fu, et al. Egoscale: scaling dexterous manipulation with diverse egocentric human data. arXiv preprint arXiv:2602.16710.

Appendix A Diagnostic Analysis

Observer Variation in a Reconstruction Bottleneck

We analyze when reconstruction-based latent action learning retains observer-induced variation. Following the observation decomposition in (Lee et al., 2026) and the linear LAM analysis in (Zhang et al., 2025), we separate interaction and observer contributions to a visual transition.

Setup.

Let \mathbf{m} denote the hand–object interaction state and \mathbf{c} the observer state. The observation map h generates visual representations \mathbf{o}=h(\mathbf{m},\mathbf{c}) and \mathbf{o}^{\prime}=h(\mathbf{m}^{\prime},\mathbf{c}^{\prime}), both in \mathbb{R}^{d_{o}}. Using the intermediate observation h(\mathbf{m}^{\prime},\mathbf{c}), we decompose the observation change as

\displaystyle\Delta\mathbf{o}_{\mathrm{int}} \\ \displaystyle=h(\mathbf{m}^{\prime},\mathbf{c})-h(\mathbf{m},\mathbf{c}), \\ \displaystyle\Delta\mathbf{o}_{\mathrm{obs}} \\ \displaystyle=h(\mathbf{m}^{\prime},\mathbf{c}^{\prime})-h(\mathbf{m}^{\prime},\mathbf{c}), \\ \displaystyle\Delta\mathbf{o} \\ \displaystyle=\mathbf{o}^{\prime}-\mathbf{o}=\Delta\mathbf{o}_{\mathrm{int}}+\Delta\mathbf{o}_{\mathrm{obs}}.

The first component changes the interaction state at a fixed observer state; the second changes the observer state at the final interaction state. This identity holds for nonlinear h and does not require the two components to be independent.

For the linear analysis below, we subtract the respective means from all observation and transition variables and reuse the same notation. The additive identity above is preserved. All expectations are taken over the distribution of observation pairs, with finite second moments assumed.

Consider a linear LAM with a d_{z}-dimensional bottleneck, where 1\leq d_{z}<d_{o}:

\mathbf{z}=\mathbf{C}\mathbf{o}+\mathbf{D}\mathbf{o}^{\prime},\qquad\widehat{\mathbf{o}}^{\prime}=\mathbf{A}\mathbf{o}+\mathbf{B}\mathbf{z}.

Here, \mathbf{A}\in\mathbb{R}^{d_{o}\times d_{o}}, \mathbf{B}\in\mathbb{R}^{d_{o}\times d_{z}}, and \mathbf{C},\mathbf{D}\in\mathbb{R}^{d_{z}\times d_{o}} are unconstrained trainable matrices. They minimize the expected squared reconstruction error,

\mathcal{L}=\mathbb{E}\!\left[\|\widehat{\mathbf{o}}^{\prime}-\mathbf{o}^{\prime}\|_{2}^{2}\right].

Following (Zhang et al., 2025), we assume that the current observation and the total observation change are uncorrelated:

\mathbb{E}[\mathbf{o}\,\Delta\mathbf{o}^{\top}]=\mathbf{0}.

This is an additional condition of the linear analysis, not a consequence of centering.

Proof.

Substituting \mathbf{o}^{\prime}=\mathbf{o}+\Delta\mathbf{o} into Eq. 6 gives

\widehat{\mathbf{o}}^{\prime}-\mathbf{o}^{\prime}=[\mathbf{A}+\mathbf{B}(\mathbf{C}+\mathbf{D})-\mathbf{I}]\mathbf{o}+(\mathbf{B}\mathbf{D}-\mathbf{I})\Delta\mathbf{o},

where \mathbf{I} is the d_{o}\times d_{o} identity matrix. The expected cross term vanishes by Eq. 8, so

\displaystyle\mathcal{L}={} \\ \displaystyle\mathbb{E}\!\left[\|[\mathbf{A}+\mathbf{B}(\mathbf{C}+\mathbf{D})-\mathbf{I}]\mathbf{o}\|_{2}^{2}\right] \\ \displaystyle+\mathbb{E}\!\left[\|(\mathbf{B}\mathbf{D}-\mathbf{I})\Delta\mathbf{o}\|_{2}^{2}\right].

For any \mathbf{B},\mathbf{C},\mathbf{D}, choosing \mathbf{A}=\mathbf{I}-\mathbf{B}(\mathbf{C}+\mathbf{D}) eliminates the first term.

The product \mathbf{B}\mathbf{D} can represent any matrix of rank at most d_{z}. The remaining objective therefore seeks a linear reconstruction of \Delta\mathbf{o} through at most d_{z} dimensions. By PCA optimality (Zhang et al., 2025), it is minimized by projecting onto the eigenvectors of \bm{\Sigma} with the d_{z} largest eigenvalues.

To obtain the minimum loss, write \mathbf{P}=\sum_{i=1}^{d_{z}}\mathbf{u}_{i}\mathbf{u}_{i}^{\top}. The reconstruction discards the remaining orthogonal components:

\displaystyle\Delta\mathbf{o}-\mathbf{P}\Delta\mathbf{o} \\ \displaystyle=\sum_{i=d_{z}+1}^{d_{o}}(\mathbf{u}_{i}^{\top}\Delta\mathbf{o})\mathbf{u}_{i}, \\ \displaystyle\mathcal{L}^{\star} \\ \displaystyle=\mathbb{E}\!\left[\|\Delta\mathbf{o}-\mathbf{P}\Delta\mathbf{o}\|_{2}^{2}\right] \\ \displaystyle=\sum_{i=d_{z}+1}^{d_{o}}\mathbb{E}[(\mathbf{u}_{i}^{\top}\Delta\mathbf{o})^{2}]=\sum_{i=d_{z}+1}^{d_{o}}\lambda_{i}.

The squared components add because the eigenvectors are orthonormal. The minimum loss is therefore the total variance along the discarded directions. This optimum is attained in the original LAM by \mathbf{A}=\mathbf{I}, \mathbf{B}=\mathbf{U}, \mathbf{C}=-\mathbf{U}^{\top}, and \mathbf{D}=\mathbf{U}^{\top}.

Under this reconstruction, the predicted transition is \mathbf{P}\Delta\mathbf{o}_{\mathrm{int}}+\mathbf{P}\Delta\mathbf{o}_{\mathrm{obs}}. Since the observer component is centered,

\displaystyle\mathbb{E}\!\left[\|\mathbf{P}\Delta\mathbf{o}_{\mathrm{obs}}\|_{2}^{2}\right] \\ \displaystyle=\operatorname{tr}(\mathbf{P}\bm{\Sigma}_{\mathrm{obs}}\mathbf{P}^{\top}) \\ \displaystyle=\operatorname{tr}(\mathbf{P}\bm{\Sigma}_{\mathrm{obs}}) \\ \displaystyle=\sum_{i=1}^{d_{z}}\mathbf{u}_{i}^{\top}\bm{\Sigma}_{\mathrm{obs}}\mathbf{u}_{i}.

The second equality uses \mathbf{P}^{\top}=\mathbf{P}, \mathbf{P}^{2}=\mathbf{P}, and the cyclic property of the trace. Each term in the final sum is the observer variance along a selected direction and is nonnegative. The retained observer component is therefore nonzero in mean square exactly when at least one selected direction has positive observer variance. ∎

Interpretation and scope.

The reconstruction objective selects directions according to total transition variance, without distinguishing interaction from observer motion. The result concerns a selected optimal reconstruction subspace; it does not establish statistical recoverability of observer state from the latent. When eigenvalues are tied at the cutoff, the optimal subspace may be nonunique.

Occurrence and Sensitivity of Observer Variation

The preceding analysis explains why reconstruction can retain observer variation. We now examine how observer motion affects a latent-action encoder. Following the occurrence–response decomposition in (Lee et al., 2026), we separate how often observer motion occurs from the latent response when it occurs.

Setup.

Let f:\mathbb{R}^{d_{o}}\times\mathbb{R}^{d_{o}}\to\mathbb{R}^{d_{z}} be a fixed latent-action encoder, which may be nonlinear. Using the observation map h defined above, fix an interaction transition (\mathbf{m},\mathbf{m}^{\prime}) and an initial observer state \mathbf{c}. For a final observer state \mathbf{c}^{\prime}, define

\displaystyle\mathbf{z}(\mathbf{c}^{\prime}) \\ \displaystyle=f\bigl(h(\mathbf{m},\mathbf{c}),h(\mathbf{m}^{\prime},\mathbf{c}^{\prime})\bigr), \\ \displaystyle\mathbf{z}(\mathbf{c}) \\ \displaystyle=f\bigl(h(\mathbf{m},\mathbf{c}),h(\mathbf{m}^{\prime},\mathbf{c})\bigr).

The second expression keeps the observer fixed across the interaction transition. Their difference,

\Delta\mathbf{z}=\mathbf{z}(\mathbf{c}^{\prime})-\mathbf{z}(\mathbf{c}),

therefore measures the latent response to observer motion while holding the interaction transition and current observation fixed.

All probabilities and expectations below are over \mathbf{c}^{\prime} conditional on the fixed (\mathbf{m},\mathbf{m}^{\prime},\mathbf{c}). Let p=\Pr(\mathbf{c}^{\prime}\neq\mathbf{c}) be the probability of observer motion, and assume \mathbb{E}[\|\Delta\mathbf{z}\|_{2}^{2}]<\infty. No centering or linearity assumption is required.

Proof.

When \mathbf{c}^{\prime}=\mathbf{c}, the two encoder inputs in Eq. 14 coincide, so \Delta\mathbf{z}=\mathbf{0}. For 0<p<1, the law of total expectation gives

\displaystyle\mathbb{E}[\|\Delta\mathbf{z}\|_{2}^{2}]={} \\ \displaystyle p\,\mathbb{E}\!\left[\|\Delta\mathbf{z}\|_{2}^{2}\mid\mathbf{c}^{\prime}\neq\mathbf{c}\right] \\ \displaystyle+(1-p)\,\mathbb{E}\!\left[\|\Delta\mathbf{z}\|_{2}^{2}\mid\mathbf{c}^{\prime}=\mathbf{c}\right].

The second term is zero, proving Eq. 16. For p=1, conditioning on observer motion leaves the expectation unchanged; for p=0, \Delta\mathbf{z}=\mathbf{0} almost surely. ∎

Interpretation.

For a fixed observer-motion distribution, the conditional expectation in Eq. 16 measures the encoder’s average sensitivity to observer motion. It includes the effects of motion magnitude and direction. Comparisons between encoders therefore require the same perturbation distribution and a common latent scale. If observer motion occurs almost surely, p=1 and the expected latent change is determined entirely by this response term.

WING-LAM aims to reduce the response of its interaction channel to observer motion while preserving interaction information.

Information Preservation and Frequency-Ordered Coordinates

We use the discrete cosine transform (DCT) to decompose latent-action trajectories into temporally ordered components. Here we briefly recall the standard properties needed for our analysis.

Consider H\geq 2 latent actions \mathbf{z}_{0},\ldots,\mathbf{z}_{H-1}\in\mathbb{R}^{D_{i}}, sampled at a fixed temporal interval, and stack them as

\mathbf{Z}=[\mathbf{z}_{0},\ldots,\mathbf{z}_{H-1}]^{\top}\in\mathbb{R}^{H\times D_{i}}.

Let \mathbf{C}\in\mathbb{R}^{H\times H} denote the orthonormal DCT-II matrix,

C_{kh}=\alpha_{k}\cos\left[\frac{\pi k}{H}\left(h+\frac{1}{2}\right)\right],\qquad\alpha_{k}=\begin{cases}1/\sqrt{H},&k=0,\\
\sqrt{2/H},&k>0,\end{cases}

where h=0,\ldots,H-1 indexes time and k=0,\ldots,H-1 indexes temporal frequency. The DCT coefficients are

\mathbf{Q}=\mathbf{C}\mathbf{Z},

with \mathbf{q}_{k}^{\top} denoting the k-th row of \mathbf{Q}.

Because \mathbf{C} is orthonormal, the full transform is invertible and preserves the trajectory energy:

\mathbf{Z}=\mathbf{C}^{\top}\mathbf{Q},\qquad\|\mathbf{Z}\|_{F}^{2}=\|\mathbf{Q}\|_{F}^{2}=\sum_{k=0}^{H-1}\|\mathbf{q}_{k}\|_{2}^{2}.

The mode index also provides a natural ordering of temporal variation. Let \mathbf{D}\in\mathbb{R}^{(H-1)\times H} be the first-difference operator,

(\mathbf{D}\mathbf{v})_{h}=v_{h+1}-v_{h}.

For the DCT-II basis, \mathbf{C} diagonalizes \mathbf{D}^{\top}\mathbf{D} (Strang, 1999), giving

\mathbf{C}\mathbf{D}^{\top}\mathbf{D}\mathbf{C}^{\top}=\operatorname{diag}(\lambda_{0},\ldots,\lambda_{H-1}),\qquad\lambda_{k}=4\sin^{2}\left(\frac{\pi k}{2H}\right).

Therefore,

\sum_{h=0}^{H-2}\|\mathbf{z}_{h+1}-\mathbf{z}_{h}\|_{2}^{2}=\sum_{k=0}^{H-1}\lambda_{k}\|\mathbf{q}_{k}\|_{2}^{2}.

Since 0=\lambda_{0}<\lambda_{1}<\cdots<\lambda_{H-1}, higher-index modes correspond to progressively faster temporal variation.

These properties only establish that the DCT provides an information-preserving, frequency-ordered representation of the full trajectory. We further evaluate this property empirically through the cross-embodiment analyses in Appendix E.2.1.

Appendix B Related Work

Vision-Language-Action and World-Action Models

Vision-language-action (VLA) models have emerged as a dominant paradigm for generalist robot manipulation, leveraging large-scale vision-language pretraining and heterogeneous robot demonstrations to map visual observations and language instructions directly to actions (Brohan et al., 2023; Kim et al., 2024; Team et al., 2024; Black et al., 2024; Intelligence et al., 2025; NVIDIA et al., 2025; Hu et al., 2026). Recent works further scale model capacity, data diversity, and cross-embodiment training, leading to increasingly general robot policies across tasks and platforms (O’Neill et al., 2024; Liu et al., 2025; Li et al., 2024). Despite their strong semantic and visual priors, these policies primarily optimize action prediction and do not explicitly model how the observed world evolves under interaction.

World-action models (WAMs) provide a complementary direction by jointly modeling robot actions and future visual dynamics. Early approaches explored video prediction as a planning interface or as auxiliary supervision for control, while recent WAMs increasingly build on large-scale pretrained video generation models and learn actions together with future visual representations (Ye et al., 2026b; Yuan et al., 2026b; Li et al., 2026b; Yan et al., 2026; Ye et al., 2026a). Such joint prediction allows action learning to benefit from the spatiotemporal priors encoded by video models and has shown strong data efficiency and generalization in robot manipulation. Recent studies have further explored more efficient future prediction, action-centered world modeling, and richer geometric or semantic prediction targets (Yuan et al., 2026b; Yan et al., 2026; Ye et al., 2026a).

Our work builds on this world-action modeling paradigm but focuses on a different source of supervision. Rather than relying only on robot trajectories to learn interaction dynamics, we study how large-scale egocentric human videos can provide interaction priors for robot action generation. WING extracts these priors into a compact latent guidance signal, allowing human interaction dynamics to complement the visual world-modeling capability of a pretrained WAM without requiring human actions to be directly mapped to robot controls.

Learning from Egocentric Human Videos for Robot Manipulation

Egocentric human videos offer a scalable source of manipulation experience, covering diverse objects, environments, and interaction behaviors that are costly to collect through robot teleoperation. Large-scale datasets such as Ego4D, EPIC-KITCHENS, EgoDex, and EgoVerse have made such experience increasingly accessible (Grauman et al., 2022; Damen et al., 2021; Hoque et al., 2026; Punamiya et al., 2026). Early efforts primarily used human videos to learn transferable visual representations or reward functions for downstream robot learning (Nair et al., 2022; Ma et al., 2022; Ma et al., 2023). More recent approaches extract explicit motion cues, such as hand, wrist, or body trajectories, and convert them into robot-compatible supervision through retargeting, inverse kinematics, or embodiment-aware action representations (Li et al., 2026a; Kareer et al., 2025; Yang et al., 2025; Shi et al., 2025; Fu et al., 2024). These methods provide stronger behavioral supervision than visual pretraining, but generally require recovering human motion in a form that can be explicitly aligned with robot actions.

Latent-action learning provides a way to exploit human videos without recovering executable actions. These methods infer action-related representations from visual transitions and use them for policy or VLA pretraining (Ye et al., 2025; Bu et al., 2025; Dai et al., 2026; Yang et al., 2026). Because visual change is not equivalent to interaction dynamics, however, the resulting representations may also encode appearance, static scene content, and motion unrelated to manipulation. Recent approaches reduce such variation through task-conditioned representations, contrastive objectives, temporal cues, or foreground–background disentanglement (Bu et al., 2025; Dai et al., 2026; Yang et al., 2026; Xu et al., 2026). CD-LAM (Wei et al., 2026) further studies action-irrelevant bias in latent actions and improves their controllability through embodiment-centric reconstruction, action-centric contrastive learning, and latent-space calibration, including an evaluation of sensitivity to synthetic camera shifts. Together, these methods establish the importance of making latent actions more closely reflect embodied behavior, but primarily focus on action relevance or world-model controllability rather than the transferable temporal structure of egocentric interactions.

Observer motion makes this transfer problem particularly challenging. Head- and body-induced camera motion can dominate the visual transition even when the underlying hand–object interaction changes little. Prior work has recovered or compensated for camera motion when reconstructing human trajectories (Niu et al., 2026), while other methods treat observer motion as a useful component of active perception (Lin et al., 2026). These approaches clarify the role of observer motion in egocentric behavior, but pursue different objectives from learning latent interaction dynamics for robot policy guidance. Meanwhile, existing latent-action debiasing methods do not explicitly use observer and interaction motion as separate supervision channels. It therefore remains unclear how to reduce observer-induced variation in egocentric latent actions while preserving the interaction structure that can transfer to robot control.

WING addresses this problem in two stages. WING-LAM first uses geometric motion routing to separate dominant observer-induced image motion from local interaction residuals, and distills the resulting interaction representation into an RGB-only latent-action encoder. Reducing observer sensitivity alone, however, does not resolve differences in how humans and robots execute the same interaction, particularly in execution speed and fine-scale motion. WING therefore analyzes latent-action trajectories across temporal frequency bands, identifies low-frequency dynamics with stronger correspondence between semantically matched human and robot interactions, and uses these dynamics as guidance for robot action generation. In this way, WING connects observer-aware latent-action learning with the temporal structure used for ego-to-robot transfer.

Appendix C Implementation and Training Details

WING-LAM Architecture and Training Details

This section provides the implementation details of WING-LAM introduced in Section 3.2. WING-LAM is trained in two stages. We first construct structured motion supervision from egocentric frame pairs and train a motion-routed teacher that separates dominant observer-induced image motion from local interaction residuals. We then freeze the teacher and distill its interaction representation into an RGB-only student. The resulting student is frozen and used offline to construct latent-action targets for subsequent WING training. The optimization hyperparameters are summarized in Table 4.

Offline Motion Supervision Construction

Point tracks and reliability.

Given a frame pair (\mathbf{o}_{t},\mathbf{o}_{t+\delta}), we use AllTracker (Harley et al., 2025) to extract point correspondences:

(\mathbf{p}_{t,i},\mathbf{p}_{t+\delta,i}),\qquad i=1,\ldots,N_{t},

where \mathbf{p}_{t,i}\in\mathbb{R}^{2} denotes the image-plane location of the i-th tracked point in frame \mathbf{o}_{t}, i indexes tracked correspondences across the frame pair, and N_{t} is the number of correspondences returned for that pair. AllTracker additionally provides visibility and tracking-confidence estimates. We retain reliable correspondences and associate each retained track with a scalar reliability weight \rho_{i} derived from these estimates. EgoHOS (Zhang et al., 2022) provides an initial hand-region seed.

Global observer-motion estimation.

We exclude tracks inside the initial hand region and fit an affine transformation to the remaining reliable background correspondences using deterministic RANSAC (Fischler and Bolles, 1981). The resulting warp W_{t\rightarrow t+\delta} provides an image-space approximation of the dominant coherent motion induced by the moving observer. We also record a fitting confidence \rho_{\rm cam} from the RANSAC fit and use it to weight the camera-motion supervision.

Interaction-region construction.

For every reliable track, we compute its camera-compensated residual following Eq. 1:

\mathbf{r}_{t,i}=W_{t\rightarrow t+\delta}^{-1}(\mathbf{p}_{t+\delta,i})-\mathbf{p}_{t,i}.

Starting from the hand-region seed, we expand the interaction region to spatially nearby tracks with sufficiently large compensated residual motion. The resulting binary indicator m_{i}\in\{0,1\} distinguishes interaction tracks from background tracks. We denote

\mathcal{I}_{t}=\{i:m_{i}=1\},\qquad\mathcal{B}_{t}=\{i:m_{i}=0\},

and collect the privileged teacher inputs as

\mathcal{G}_{t}=\{\mathbf{p}_{t,i},\mathbf{p}_{t+\delta,i},m_{i},\rho_{i}\}_{i=1}^{N_{t}}.

All quantities in \mathcal{G}_{t} are used only during WING-LAM training.

Motion-Routed Teacher Architecture and Objectives

Teacher architecture.

The teacher follows the two-frame spatiotemporal Transformer architecture of DreamDojo (Gao et al., 2026). The visual backbone encodes (\mathbf{o}_{t},\mathbf{o}_{t+\delta}) into spatiotemporal visual features. For each reliable correspondence from \mathbf{p}_{t,i} to \mathbf{p}_{t+\delta,i}, we sample the visual features at the corresponding locations in the two frames and concatenate them with the normalized point coordinates, displacement, and tracking confidence to obtain the track descriptor \mathbf{d}_{t,i}. Two gated set-pooling modules aggregate background and interaction descriptors separately:

\displaystyle\mathbf{z}_{t,\rm cam}^{T} \\ \displaystyle=P_{\rm cam}\left(\{\mathbf{d}_{t,i}\}_{i\in\mathcal{B}_{t}}\right), \\ \displaystyle\mathbf{z}_{t,\rm int}^{T} \\ \displaystyle=P_{\rm int}\left(\{\mathbf{d}_{t,i}\}_{i\in\mathcal{I}_{t}}\right),

where \mathbf{z}_{\rm cam}^{T}\in\mathbb{R}^{D_{c}} and \mathbf{z}_{\rm int}^{T}\in\mathbb{R}^{D_{i}}. T denotes the teacher. The two latent codes are decoded independently as:

\widehat{W}_{t\rightarrow t+\delta}=D_{\rm cam}(\mathbf{z}_{t,\rm cam}^{T}),\qquad\widehat{\mathbf{r}}_{t,i}=D_{\rm int}(\mathbf{p}_{t,i},\mathbf{z}_{t,\rm int}^{T}).

For every reliable track, the decoded global and interaction motions are composed as:

\widehat{\mathbf{p}}_{t+\delta,i}=\widehat{W}_{t\rightarrow t+\delta}\left(\mathbf{p}_{t,i}+m_{i}\widehat{\mathbf{r}}_{t,i}\right).

Thus, background tracks are explained only by the global warp, whereas interaction tracks additionally receive the decoded residual motion.

Motion supervision.

Let h_{\beta}(\cdot,\cdot) denote the coordinate-wise Smooth-L_{1} distance with \beta=0.01, and let \langle f_{i}\rangle_{w_{i}} denote the normalized weighted average of f_{i} with weights w_{i}. We supervise the global motion, canonical interaction residual, and their composition by:

\displaystyle\mathcal{L}_{\rm cam} \\ \displaystyle=\rho_{cam}\left\langle h_{\beta}\left(\widehat{\mathbf{W}}_{t\rightarrow t+\delta}(\mathbf{p}_{t,i}),\mathbf{W}_{t\rightarrow t+\delta}(\mathbf{p}_{t,i})\right)\right\rangle_{\rho_{i},\,i\in\mathcal{B}_{t}}, \\ \displaystyle\mathcal{L}_{\rm int} \\ \displaystyle=\left\langle h_{\beta}\left(\widehat{\mathbf{r}}_{t,i},\mathbf{r}_{t,i}\right)\right\rangle_{\rho_{i},\,i\in\mathcal{I}_{t}}, \\ \displaystyle\mathcal{L}_{\rm trk} \\ \displaystyle=\left\langle h_{\beta}\left(\widehat{\mathbf{p}}_{t+\delta,i},\mathbf{p}_{t+\delta,i}\right)\right\rangle_{\rho_{i}}.

Here, \mathcal{L}_{\rm cam} constrains the decoded observer-motion warp on background tracks, \mathcal{L}_{\rm int} supervises the camera-compensated interaction residuals, and \mathcal{L}_{\rm trk} ensures that the two components jointly explain the observed track endpoints.

Camera Intervention and Representation Regularization

Camera intervention.

Motion routing reduces observer-induced motion in the interaction branch, but does not guarantee that the resulting representation is insensitive to observer-specific cues. We therefore construct a camera-intervened counterpart of each transition by perturbing only the global camera motion while preserving the underlying interaction motion. Encoding the natural and camera-intervened transitions yields teacher interaction representations \mathbf{z}_{t,\mathrm{int}}^{T,\mathrm{nat}} and \mathbf{z}_{t,\mathrm{int}}^{T,\mathrm{cam}}\in\mathbb{R}^{D_{i}}, respectively. We encourage them to agree through:

\mathcal{L}_{\mathrm{cons}}=\frac{1}{D_{i}}\sum_{d=1}^{D_{i}}h_{\beta}\left([\mathbf{z}_{t,\mathrm{int}}^{T,\mathrm{nat}}]_{d},[\mathbf{z}_{t,\mathrm{int}}^{T,\mathrm{cam}}]_{d}\right),

where [\mathbf{z}]_{d} denotes the d-th coordinate of \mathbf{z}\in\mathbb{R}^{D_{i}}. This consistency objective encourages the interaction representation to retain interaction information while reducing its sensitivity to camera motion.

Representation regularization.

To prevent collapse and reduce redundancy, we apply variance and covariance regularization to the interaction representations from the natural and intervened views jointly. Let \sigma_{d} denote the batchwise standard deviation of latent dimension d. The variance regularizer is

\mathcal{L}_{\rm var}=\frac{1}{D_{i}}\sum_{d=1}^{D_{i}}\max(0,\gamma-\sigma_{d})^{2},\qquad\gamma=0.1.

Let \mathbf{C}\in\mathbb{R}^{D_{i}\times D_{i}} denote the covariance matrix of the standardized latent dimensions, where C_{de} is the covariance between latent dimensions d and e. We further use

\mathcal{L}_{\rm cov}=\frac{1}{D_{i}(D_{i}-1)}\sum_{d\neq e}C_{de}^{2}.

The complete teacher objective is

\displaystyle\mathcal{L}_{\rm teacher}={} \\ \displaystyle\lambda_{\rm trk}\mathcal{L}_{\rm trk}+\lambda_{\rm cam}\mathcal{L}_{\rm cam}+\lambda_{\rm int}\mathcal{L}_{\rm int} \\ \displaystyle+\lambda_{\rm cons}\mathcal{L}_{\rm cons}+\lambda_{\rm var}\mathcal{L}_{\rm var}+\lambda_{\rm cov}\mathcal{L}_{\rm cov}.

The motion-supervision terms establish the camera–interaction routing, while the consistency and representation regularizers further suppress observer-motion leakage and stabilize the interaction latent space.

Interaction Distillation

The motion-routed teacher depends on privileged point tracks, interaction masks, and global-motion estimates, making it unsuitable for large-scale latent extraction. We therefore freeze the teacher and distill only its interaction representation into an RGB-only student S_{\phi}.

The student is initialized from the teacher visual encoder but receives only the RGB frame pair. Its spatiotemporal visual tokens are mean-pooled and linearly projected into a D_{i}-dimensional interaction latent; no point tracks, masks, or motion parameters are provided. For each training transition, we use the frozen teacher’s interaction representation from the natural view as the common distillation target:

\mathbf{z}_{t,\mathrm{int}}^{T}:=\mathbf{z}_{t,\mathrm{int}}^{T,\mathrm{nat}}=E_{T,\mathrm{int}}(\mathbf{o}_{t},\mathbf{o}_{t+\delta},\mathcal{G}_{t}),

where E_{T,\mathrm{int}} denotes the interaction output of the teacher E_{T}. The student encodes the natural and camera-intervened frame pairs into \mathbf{z}_{t}^{\mathrm{nat}} and \mathbf{z}_{t}^{\mathrm{aug}}, respectively. Both representations are trained to match (\mathbf{z}_{t,\mathrm{int}}^{T}) and to agree with each other through Eq. 2. The teacher remains frozen, and gradients update only the student S_{\phi}. After distillation, all privileged motion-processing components and the teacher are discarded. The frozen RGB-only student defines the WING-LAM latent action

\mathbf{z}_{t}=S_{\phi}\left(\mathbf{o}_{t},\mathbf{o}_{t+\delta}\right),

which is subsequently used offline to construct the latent-action trajectories for WING.

Egocentric-Video Pretraining Details

Pre-training Data and Preprocessing.

We pre-train our model on three large-scale egocentric video datasets: Ego4D, EgoDex, and EgoVerse, totaling approximately 200 hours of ego-videos. Specifically, Ego4D contributes approximately 20 hours, EgoDex contributes about 85 hours, and EgoVerse contributes 95 hours. Language instructions are obtained from the original task descriptions of EgoDex and EgoVerse, while those for Ego4D are constructed from the verb–noun action labels in its official long-term anticipation annotations. All egocentric videos are standardized to 30 FPS. The sampled RGB frames are resized to 384\times 640 and encoded by the frozen Wan2.2 VAE.

Latent Action Preprocessing.

To construct latent-action supervision, we sample egocentric videos at a fixed physical interval of 0.2\,\mathrm{s}, corresponding to every 6 frames at 30 FPS. The sampled frames are center-cropped to a 4:3 aspect ratio, resized to 240\times 320, and consecutive sampled endpoints are encoded by the frozen WING-LAM into 32-dimensional latent transitions. Nine sampled endpoints yield H=8 transitions, spanning a 1.6\,\mathrm{s} temporal window.

Hand Pose Preprocessing

For hand-pose-based pretraining, we construct a 200-hour corpus by supplementing it with 10 hours of hand-annotated data from each of EgoDex and EgoVerse, replacing the Ego4D portion that lacks the required annotations. We then follow the Ego2Robot (Wang et al., 2026) pipeline in RobotManip (Yuan et al., 2026a) to extract hand end-effector (EEF) trajectories from these annotations. The resulting trajectories serve as action supervision during model pretraining.

Guidance Predictor Training

During egocentric-video pretraining, we train the guidance predictor to estimate interaction-centric spectral guidance from the current visual observation and language instruction. For each training sample, the frozen WING-LAM encoder extracts an 8\times 32 latent-action trajectory from the sampled video frames. We apply an orthonormal temporal DCT and retain the lowest four frequency components, producing a 4\times 32 guidance target.

The predictor is a lightweight, current-observation-conditioned query Transformer. It uses four learned queries that cross-attend to a context formed by concatenating linearly projected current-frame tokens and language-instruction tokens. A single Transformer block with a hidden dimension of 1,024, eight attention heads, and a fourfold-expanded feed-forward network produces four query features, which are projected to 32 dimensions to predict the 4\times 32 normalized guidance target. The prediction loss is weighted by 0.1. Future frames are used only to construct the training target and are never provided to the predictor.

Optimization Setup

We conduct egocentric-video pre-training for 40 epochs on 32 NVIDIA H100 GPUs. We use a per-GPU batch size of 16, resulting in an effective global batch size of 512. The learning rate is set to 1\times 10^{-4}. To reduce training overhead, both the Wan2.2 VAE latents and WING-LAM latent transitions are pre-computed and cached before training. The video backbone is initialized from Wan2.2-TI2V-5B (Wan et al., 2025).

Robot Policy Training Details

Training Framework

For our model, the guidance predictor and shared visual backbone are initialized from the egocentric-video pretrained checkpoint. During robot policy training, the guidance predictor is jointly optimized with the robot policy. Given the current visual observation, language instruction, and robot proprioceptive state, the predictor estimates the low-frequency spectral representation of the anticipated interaction dynamics. The predicted coefficients are projected into guidance tokens and injected into the action model through gated cross-attention, providing an additional conditioning signal for action prediction. The spectral prediction targets are constructed following the same procedure as in egocentric-video pretraining. The overall training objective is

\mathcal{L}_{\mathrm{total}}=\lambda_{\mathrm{video}}\mathcal{L}_{\mathrm{video}}+\lambda_{\mathrm{action}}\mathcal{L}_{\mathrm{action}}+\lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}},

where \lambda_{\mathrm{video}}=\lambda_{\mathrm{action}}=1, and \lambda_{\mathrm{pred}}=0.1.

Optimization Setup

Robot policy post-training is conducted on 8 NVIDIA H100 GPUs with a global batch size of 128. We use the AdamW optimizer with a peak learning rate of 2\times 10^{-4}. The learning rate is linearly warmed up and then decayed according to a cosine schedule.

Appendix D Evaluation Details

In this section, we present the details of our evaluation experiments.

Latent-Action Representation Evaluation Details

We evaluate frozen latent-action representations along three complementary dimensions: (i) what camera- and interaction-motion information can be decoded from the latent representation, (ii) how sensitive the representation is to controlled changes in camera motion, and (iii) how well it preserves high-level action semantics. This section describes the datasets, targets, and evaluation protocols used for these analyses.

Datasets and Frozen Representations

We evaluate WING-LAM together with DreamDojo (Gao et al., 2026), CD-LAM (Wei et al., 2026), LAPA (Ye et al., 2025), VILLA-X (Chen et al., 2026b), and UniVLA (Bu et al., 2025) using their frozen latent-action encoders. The motion decodability and camera-sensitivity analyses use EgoDex (Hoque et al., 2026), EgoVerse (Punamiya et al., 2026), and Ego4D (Grauman et al., 2022). EgoDex provides synchronized head-pose annotations, while EgoVerse and Ego4D provide additional diversity in natural observer and interaction motion. We split videos by source trajectory to avoid overlap between probe training and evaluation. The action-semantic evaluation follows the LARYBench protocol and is described separately in Appendix D.1.4.

Motion Decodability

We evaluate how much camera- and interaction-motion information can be decoded from each frozen latent representation. For camera motion, we use the relative head motion when ground-truth head pose is available, and otherwise use the estimated global image motion as a proxy. For interaction motion, we remove the estimated global camera motion from the tracked point motion and aggregate the remaining local motion into a fixed-dimensional target. We evaluate each frozen latent representation using three probe families: a ridge linear probe, a two-layer MLP with GELU activations, and a random Fourier feature (RFF) probe followed by regularized linear regression. The MLP is trained with AdamW, while the ridge and RFF probes use regularized linear regression. In all cases, the latent encoder remains frozen and only the probe is optimized.

For each target, we train a probe on the frozen latent representation and report the coefficient of determination (R^{2}) on the held-out test set:

R^{2}=1-\frac{\sum_{n=1}^{N}\left\|\mathbf{y}_{n}-\widehat{\mathbf{y}}_{n}\right\|_{2}^{2}}{\sum_{n=1}^{N}\left\|\mathbf{y}_{n}-\bar{\mathbf{y}}\right\|_{2}^{2}},\qquad\bar{\mathbf{y}}=\frac{1}{N}\sum_{n=1}^{N}\mathbf{y}_{n},

where \mathbf{y}_{n} and \widehat{\mathbf{y}}_{n} denote the ground-truth and predicted motion targets for sample n, respectively. Higher R^{2} indicates that the corresponding motion information is more readily decoded from the latent representation. An R^{2} of zero corresponds to the constant-mean baseline, while negative values indicate lower predictive accuracy than this baseline.

Sensitivity to Camera Perturbations

We complement motion decodability with a controlled intervention that directly measures how much a frozen latent representation changes under additional camera motion. For each frame pair (\mathbf{x}_{t},\mathbf{x}_{t+\delta}), we construct matched dynamic and static perturbations. Dynamic perturbations apply different global transformations to the two frames, introducing additional inter-frame camera motion, whereas static perturbations apply the same transformation to both frames. The transformation magnitude and image-processing pipeline are matched between the two conditions, so that their difference primarily reflects sensitivity to relative camera motion rather than interpolation or boundary artifacts. For the multi-model comparison in Fig. 3, we instantiate this protocol using two transformations A and B of equal magnitude and opposite direction. The pairs AB and BA form the dynamic conditions, while AA and BB serve as their matched static controls. Let \mathbf{z}^{PQ} denote the latent representation obtained after applying transformation P to the first frame and Q to the second, and let \mathbf{z}^{0} denote the representation of the unperturbed pair.

Because latent scales differ across models, each latent dimension is standardized using statistics computed from the corresponding training split. We measure the standardized deviation of each perturbed representation from the unperturbed pair as

d^{PQ}=\sqrt{\frac{1}{D_{i}}\sum_{d=1}^{D_{i}}\left(\frac{[\mathbf{z}^{PQ}]_{d}-[\mathbf{z}^{0}]_{d}}{\sigma_{d}}\right)^{2}},\qquad PQ\in\{AB,BA,AA,BB\},

where \sigma_{d} is the training-set standard deviation of latent dimension d and D_{i} denotes the latent action’s dimension. We report the dynamic-minus-static deviation,

\frac{1}{2}\left[d^{AB}+d^{BA}\right]-\frac{1}{2}\left[d^{AA}+d^{BB}\right].

A smaller value indicates that the representation changes less in response to additional inter-frame camera motion after accounting for matched image perturbations.

We evaluate five increasing perturbation levels, (0.25\,\mathrm{px},0.05^{\circ}), (0.5\,\mathrm{px},0.1^{\circ}), (1\,\mathrm{px},0.2^{\circ}), (2\,\mathrm{px},0.4^{\circ}), and (4\,\mathrm{px},0.8^{\circ}), where each pair specifies the translation and rotation magnitude.

Action-Semantic Preservation

We further evaluate whether reducing camera-motion sensitivity preserves interaction-relevant semantic information. Following the LARYBench (Nie et al., 2026) evaluation protocol, we evaluate frozen latent representations on action-category classification across Ego4D (Grauman et al., 2022), EgoDex (Hoque et al., 2026), EPIC-KITCHENS (Damen et al., 2021), and HoloAssist (Wang et al., 2023). For each method, the latent encoder remains frozen and the same classification protocol is applied to the representations.

Pair Construction for Spectral Similarity Analysis

To evaluate cross-embodiment correspondence, we construct matched and mismatched human–robot clip pairs through a hierarchical annotation protocol. We first retrieve candidate human and robot episodes from EgoDex and EgoDex-Eval according to their task annotations. These task labels are used only for coarse-grained episode-level filtering, since episodes assigned to the same task may still contain substantially different local motion patterns, interaction phases, or even opposite motion directions.

Within each candidate episode pool, human annotators further inspect the temporal content and select short clips according to their dominant action semantics. Specifically, a matched pair is formed by selecting a human clip and a robot clip that exhibit similar dominant interaction dynamics, rather than requiring frame-wise or trajectory-level alignment. Consequently, matched clips may differ substantially in object instance, scene appearance, viewpoint, operator, and precise end-effector trajectory, while still sharing the same underlying interaction pattern. This procedure therefore performs coarse task-level retrieval first, followed by fine-grained clip-level semantic matching within the retrieved episodes.

Mismatched pairs are constructed from the same task-conditioned candidate pools, but the selected human and robot clips exhibit inconsistent dominant motion semantics. This preserves a comparable task distribution while removing action-level correspondence. To reduce dependence among evaluation samples, each source episode is used at most once in the matched set and at most once in the mismatched set. The final evaluation set contains 100 matched and 100 mismatched pairs, covering 25 distinct tasks. The pair set and all match/mismatch labels were finalized before computing any latent trajectories or spectral similarities. To assess annotation reliability, two additional annotators independently reviewed the constructed pairs and showed high agreement with the original labels.

We use our frozen WING-LAM, the same latent-action encoder used to construct spectral guidance in WING, to encode both human and robot clips. WING-LAM is trained only on egocentric videos and is not adapted using robot data for this analysis. Therefore, this evaluation directly tests whether the interaction-centric latent space learned from egocentric data generalizes across embodiments, and whether semantically matched human and robot interactions exhibit consistent temporal structure in this shared representation.

For both egocentric and robot clips, we use the same physical sampling interval of 0.2\mathrm{s}, corresponding to an effective sampling rate of 5 Hz. Each H=8 latent-action trajectory is constructed from 9 sampled observation endpoints and spans 1.6\mathrm{s}. We then apply DCT-II along the temporal dimension and compute cosine similarity between the corresponding low-, high-, and full-frequency representations. We report the average similarity over the 100 matched and 100 mismatched pairs in Figure 3.

Simulation Evaluation Details

Benchmark Protocols and Metrics

LIBERO.

LIBERO contains four suites—Spatial, Object, Goal, and Long—with 10 tasks and 500 demonstrations per suite. We train one checkpoint across the four suites and evaluate each task for 50 rollouts, giving 2,000 rollouts in total. We use the benchmark episode budgets for each suite and report the mean task success rate for every suite together with their four-suite average.

RoboTwin 2.0.

RoboTwin evaluates 50 bimanual tasks in clean and domain-randomized scenes. Following the standard multi-task protocol, training uses 2,500 clean demonstrations and 25,000 randomized demonstrations. Each task is evaluated for 100 rollouts under each scene condition. We report the 50-task clean mean, the randomized mean, and their arithmetic average.

RoboCasa–GR1.

RoboCasa–GR1 contains 24 tabletop tasks executed by a Fourier GR1 humanoid with two 7-DoF arms, two 6-DoF hands, and a 3-DoF waist. We use the Unified 1K dataset with 1,000 teleoperated demonstrations per task and egocentric RGB observations. Each task is evaluated over 50 rollouts, and the main metric is the average of the 24 task success rates.

Evaluation.

All simulation tables report success rate in percent. Notably, the FastWAM result on RoboCasa–GR1 is our reproduction trained to convergence. Complete RoboTwin and RoboCasa task-level tables are reported below; method-specific optimization details are given in Appendix C.3.

Baselines

We compare our method with recent robot manipulation policies spanning three representative paradigms: vision-language-action models, latent-action based policies, and world-action modeling approaches. Together, these baselines cover methods that directly map multimodal observations to robot actions, methods that introduce an intermediate latent action representation, and methods that explicitly exploit predicted future dynamics or world representations for policy learning.

Vision-language-action models.

We first consider representative vision-language-action (VLA) policies that directly predict robot actions from visual observations and language instructions. These include \pi_{0} (Black et al., 2024) and \pi_{0.5} (Intelligence et al., 2025), X-VLA (Zheng et al., 2026a), LingBot-VLA (Wu et al., 2026), GR00T (N1.6 and N1.7) (NVIDIA et al., 2025), StarVLA-OFT (Community, 2026) and LDA-1B (Lyu et al., 2026). This group represents strong recent VLA systems that learn robot control directly in the action space without explicitly introducing a latent-action or future-dynamics interface.

Latent-action based methods.

We additionally compare against approaches that introduce latent actions as an intermediate representation for learning manipulation behaviors, including LAPA (Ye et al., 2025), UniVLA (Bu et al., 2025), LaWAM (Chen et al., 2026a), Being-H0.7 (Luo et al., 2026), and VLA-JEPA (Sun et al., 2026). These methods are particularly relevant to our setting because they investigate whether interaction or action information can be represented in a latent space rather than being directly tied to embodiment-specific robot commands. In contrast, our method uses interaction-centric latent dynamics with reduced sensitivity to observer motion and emphasizes their low-frequency temporal structure as guidance for robot action learning.

World-action modeling methods.

The third group contains methods that explicitly leverage predictive world representations, future observations, or future latent dynamics to facilitate action learning. We include Motus (Bi et al., 2026), LingBot-VA (Li et al., 2026b), FastWAM (Yuan et al., 2026b), GigaWorld-Policy (Ye et al., 2026a), LDA-1B (Lyu et al., 2026) and DiT4DiT (Ma et al., 2026). These baselines are the closest to our method in their use of anticipated future dynamics for control. Our method differs by constructing guidance from interaction-centric latent dynamics extracted from egocentric videos and using their low-frequency components, which exhibit stronger correspondence across human and robot executions, rather than conditioning the policy on the full latent trajectory.

Real-world baselines.

For the real-world experiments, we focus on representative strong methods from the above families and compare against GR00T-N1.7 (NVIDIA et al., 2025), \pi_{0.5} (Intelligence et al., 2025), and FastWAM (Yuan et al., 2026b). These baselines provide comparisons with a strong general-purpose VLA model, a recent large-scale robot policy, and a predictive world-action modeling approach, respectively, under the same real-world evaluation protocol.

Real-World Evaluation Protocols

Tasks and Progress Metrics

The real-world suite contains four tasks. Pack Objects requires placing a two-brick LEGO assembly, a tape roll, and an inverted plastic cup in a container. Stack Cups requires arranging three cups into a stable tower. Battery Insertion requires grasping a battery, aligning it with a mouse compartment, and seating it fully. Long-Horizon Battery Assembly additionally requires relocating the mouse and battery to the workspace before insertion and moving the assembled mouse to a target region. We collect 100 demonstrations each for Pack Objects and Stack Cups, and 150 demonstrations each for Battery Insertion and Long-Horizon Battery Assembly. For each task, we evaluate the policy over 20 trials. We report binary success and a stage-wise score. Success requires the final task condition to remain satisfied for at least three seconds. The score is the furthest valid milestone reached while all preceding milestones remain satisfied; Pack Objects is order-independent and scores the number of objects placed. Table 5 gives the complete rubric. The deployment code will be released.

Each rollout is autonomous: no human correction, object reset, or policy restart is permitted before termination. Hardware- or communication-only interruptions are discarded and repeated, whereas policy-induced safety terminations count as failures. The recorded protocol uses 20 trials per reported setting and the same predefined initial configurations, observation inputs, language instructions, action limits, termination rules, and low-level controller for all methods.

Generalization Conditions

The generalization study changes one factor at a time: object appearance, task-irrelevant background, object geometry, or initial spatial layout. Camera placement, robot base pose, and all other factors are held fixed. Changed objects, colors, backgrounds, and poses are excluded from task-specific robot training demonstrations. Geometry is not evaluated for the battery tasks because changing the insertion dimensions would change the task mechanics. The task-specific generalization settings are defined as follows:

• Pack Objects: Appearance: Replace the original cup and LEGO bricks with instances of different colors while preserving their shapes and sizes. Background: Introduce task-irrelevant tabletop objects or vary the illumination without obstructing the target objects. Geometry: Change the way the LEGO pieces are connected, while keeping its function in the task, and change the size of the cup. Spatial layout: Relocate the tape roll and cup within predefined reachable regions while keeping the container fixed.
• Stack Cups: Appearance: Replace the original cups with cups of different colors but identical geometry. Background: Add task-irrelevant clutter to the tabletop while keeping the manipulation area reachable. Geometry: Replace the original cups with a larger affordance-equivalent set that can still be arranged into the same tower structure. Spatial layout: Randomize the initial positions of all three cups within predefined regions while preserving task feasibility.
• Battery Insertion: Appearance: Replace the original battery with a color-different but dimension-matched instance. Background: Add task-irrelevant tabletop clutter without occluding the mouse or battery. Spatial layout: Randomize the initial positions and orientations of the mouse and battery within predefined reachable regions. Geometry: Not evaluated, because changing the battery dimensions or compartment geometry would alter the insertion mechanics and therefore change the task itself.
• Battery Assembly: Appearance: Replace the original battery with a color-different but dimension-matched instance. Background: Introduce task-irrelevant tabletop clutter while preserving the visibility of all task objects. Spatial layout: Vary the initial positions and orientations of the mouse and battery while retaining the required relocation, insertion, and final-placement stages. Geometry: Not evaluated, because changing the insertion geometry would alter the underlying task requirements.

All shifted conditions use the same milestone rubric as the base setting. Multi-factor shifts, if evaluated, must be reported separately rather than mixed into the single-factor averages.

Platform and Deployment

We conduct real-world experiments on a dual-arm Agilex Cobot Magic robot platform. The robot is equipped with three Orbbec DaBai DC1 cameras, including one front-facing camera and two wrist-mounted cameras, which provide visual observations for policy inference. The robot action space consists of 14 dimensions, including the 6-DoF pose commands for each arm and additional gripper control signals. All real-world experiments use the same inference pipeline without additional task-specific or model-specific modifications. The deployment framework and corresponding code will be released to facilitate reproducibility.

Appendix E Additional Results

Additional Latent Action Results

Camera–Interaction Readout Results

Factor-selective routing across datasets.

As shown in Table 6, across EgoDex, EgoVerse, and Ego4D, WING-LAM consistently exhibits complementary factor routing: \mathbf{z}_{\mathrm{int}} preserves interaction-related information while suppressing observer-motion information, whereas \mathbf{z}_{\mathrm{cam}} captures observer-motion variation with minimal interaction information. Notably, for EgoDex, observer-motion readout is evaluated against the recorded 6D head pose rather than the affine motion proxy used to supervise WING-LAM during training. The reduced head-motion decodability from \mathbf{z}_{\mathrm{int}} therefore indicates that suppression learned from the affine proxy transfers, at least partially, to genuine egocentric head motion. The student model further inherits this routing behavior from the trajectory-assisted teacher, demonstrating that the interaction-centric representation can be obtained without privileged inputs at inference time.

Probe robustness.

We further evaluate the readout results using probes with different capacities, including Ridge, MLP, and RFF. Although stronger probes improve absolute readout scores for some baselines, the qualitative routing behavior of WING-LAM remains unchanged, with complementary interaction and camera-motion selectivity preserved across probe settings.

Detailed LARYBench Results

Table 7 reports the detailed LARYBench (Nie et al., 2026) action-classification results across Ego4D, EPIC-KITCHENS, HoloAssist, and EgoDex. All latent-action encoders are frozen and evaluated under the same classification protocol. WING-LAM achieves the highest average accuracy across the four datasets, indicating that observer debiasing preserves action-related semantic information in the learned latent representation. These per-dataset results provide the details corresponding to the aggregate LARYBench result reported in the main text.

Qualitative Visualization of Interaction Motion

To further examine what motion patterns are captured by WING-LAM, we conduct a qualitative analysis by feeding its latent actions into the teacher decoder to reconstruct motion trajectories. Specifically, we extract trajectories from three consecutive clips and concatenate them sequentially to visualize longer-term motion patterns. The decoded trajectories are then visualized across diverse egocentric interactions in Figure 8. Across all examples, the decoded trajectories consistently concentrate on moving hands, forearms, manipulated objects, and their surrounding interaction regions. Their dominant directions broadly follow the local changes between the two frames across diverse activities, including object transfer, tool use, surface wiping, and food preparation. Comparatively few interaction responses appear in background regions, even when the egocentric viewpoint changes noticeably. These examples provide qualitative evidence that the WING-LAM latent preferentially preserves interaction-relevant motion rather than camera-induced global scene motion.

Additional Spectral Results

Cross-Embodiment Spectral Similarity Analysis

Experimental settings are detailed in Appendix D.2. As shown in Figure 3(d), low-frequency latents achieve the largest separation between semantically matched and mismatched human–robot interaction pairs, with a matched–mismatched gap of 0.51 (95% bootstrap CI: [0.48,0.53]). In contrast, full-spectrum and high-frequency representations obtain substantially smaller gaps of 0.26 ([0.22,0.27]) and 0.05 ([0.03,0.10]), respectively. Importantly, repeating the same analysis with the pretrained DreamDojo LAM yields the same qualitative ordering, with low-frequency components exhibiting the largest matched–mismatched separation, suggesting that this frequency-selective correspondence is not specific to WING-LAM. To verify that this advantage is not merely driven by the dominant energy of the zeroth component, we further exclude q_{0} and evaluate the remaining low-frequency dynamics. The q_{1}:q_{3} components retain a matched-pair similarity of 0.42, compared with only 0.08 for mismatched pairs. In contrast, extending the representation to q_{1}:q_{7} decreases the matched similarity to 0.31 while increasing the mismatched similarity to 0.14. This shows that the low-frequency correspondence persists even after removing the zeroth component, and that incorporating higher-frequency components weakens the separation between semantically matched and mismatched interactions.

Overall, these results suggest that low-frequency latent components preserve interaction semantics with stronger cross-embodiment correspondence, whereas higher-frequency components contain variation that is less consistently shared across human and robot executions.

Spectral Bandwidth Gap Between Egocentric and Robot Data

We additionally examine the spectral distributions of latent-action trajectories in egocentric and robot data. For this dataset-level bandwidth analysis, we use the pretrained DreamDojo LAM rather than WING-LAM. DreamDojo has been trained on both egocentric and robot data, making it better suited for characterizing differences in the temporal frequency distributions of the two domains without strongly confounding the analysis with domain-specific encoder generalization. In contrast, WING-LAM is trained specifically on egocentric videos; applying it directly to robot observations could make the estimated spectral distribution partly reflect its out-of-domain generalization behavior rather than the intrinsic temporal characteristics of robot trajectories. Therefore, this analysis is intended to provide a more domain-balanced characterization of the temporal bandwidth gap between egocentric and robot interaction data, rather than to evaluate the cross-embodiment correspondence of the WING-LAM representation itself.

Using the pretrained DreamDojo LAM, we encode ego and robot trajectories with the same physical temporal sampling interval of 0.2\,\mathrm{s}, forming H=8 latent-action sequences that span 1.6\,\mathrm{s}. We then apply DCT-II along the temporal dimension. We measure the fraction of spectral energy retained by the lowest K=4 frequency components. The spectral analysis reveals a noticeable difference in temporal bandwidth between egocentric and robot trajectories, with egocentric trajectories exhibiting broader frequency distributions than robot trajectories. Robot trajectories are substantially more concentrated in low-frequency components, whereas egocentric trajectories contain relatively more high-frequency variation. We observe the same qualitative trend when repeating the analysis with WING-LAM, indicating that this bandwidth difference is not specific to the DreamDojo representation. This observation provides additional motivation for examining cross-embodiment correspondence separately across frequency bands.

Action Semantics Across Frequency Bands

We examine how action-semantic information is organized across temporal frequency bands on the official Ego4D validation split of LARYBench (Nie et al., 2026). For each video, we apply temporal DCT to its 8-step WING-LAM latent trajectory and construct low-frequency (q_{0:3}), high-frequency (q_{4:7}), and full-spectrum (q_{0:7}) representations. Each representation is reduced to 32 dimensions using PCA and evaluated with the same clustering and visualization procedures. We measure the agreement between the resulting clusters and the official action labels using normalized mutual information (NMI). The low-frequency representation achieves substantially higher NMI (0.55\pm 0.02) than the full-spectrum (0.31\pm 0.01) and high-frequency (0.17\pm 0.05) representations, indicating that action semantics are more clearly organized in the low-frequency latent subspace.

Additional Policy Analyses

Latency Measurement

We measure policy inference latency on a single NVIDIA RTX 4090 GPU and report both the median and 95th-percentile latency. For evaluation, we follow the real-world deployment setting, using an action chunk size of 32 and 10 denoising steps. As shown in Table 8, our model incurs higher latency than lightweight policy baselines such as GR00T and \pi_{0.5}, due to the additional computation introduced by the dense world-modeling architecture. Cosmos-Policy (Kim et al., 2026) and Motus (Bi et al., 2026) require 845.6 ms and 1532.8 ms per policy inference, respectively. Thus, although inference latency remains a limitation of the current implementation, our model remains computationally competitive with generative policies of comparable complexity.

Ablation on Denoising Steps

The number of denoising steps N controls the amount of iterative refinement at inference time. We conduct this ablation on RoboTwin 2.0 and fix the action chunk size to 32 for all settings. As shown in Table 9, single-step inference achieves an average success rate of only 80.46\%, whereas increasing N to 2 substantially improves it to 93.20\%, with performance remaining relatively stable for larger N. Among the evaluated settings, N=12 achieves the highest average success rate, with 94.56\% and 93.04\% on the Clean and Randomized settings, respectively, and is therefore used for the RoboTwin 2.0 result reported in the main table. Notably, nearby settings perform similarly (e.g., 93.55\% success rate at N=10), indicating that the overall performance is not sensitive to the precise choice of N once sufficient iterative refinement is used.

Egocentric Pretraining Efficiency

We investigate how the extent of egocentric pretraining affects subsequent robot policy optimization. All pre-training checkpoints are trained on the same 200h egocentric video corpus, and we select checkpoints obtained after 15k, 30k, 45k, and 60k pretraining steps. Each checkpoint is then used to initialize the robot policy, which is trained on the Pack Objects task under the same optimization setting. We use a held-out split of the Pack Objects data as the validation set and report the total validation loss of the full model throughout robot training. As shown in Fig. 11, longer egocentric pretraining consistently leads to lower validation loss and faster convergence, indicating that Stage-I pretraining provides a stronger initialization and improves the optimization efficiency of downstream robot policy learning.

Downstream Data Efficiency

To further evaluate the effectiveness of egocentric pretraining, we study how performance changes as the amount of downstream robot training data is reduced. We compare models initialized with and without our pretraining under 100%, 50%, and 25% of the downstream training data on LIBERO and the real-world standard setting.

As shown in Figure 12, egocentric pretraining consistently improves downstream performance on both LIBERO and real-world manipulation. More importantly, the advantage becomes increasingly pronounced as the amount of downstream robot data decreases, with the largest improvement observed in the 25% real-world data setting. These results demonstrate that our pretraining provides transferable knowledge that improves robot-data efficiency, particularly in low-data regimes.

Full Simulation Results

RoboCasa–GR1 Per-Task Results

Table 10 reports all 24 tabletop tasks. The FastWAM aggregate is our own reproduction.

RoboTwin 2.0 Per-Task Results

Table 11 provides the complete 50-task clean/randomized breakdown.

Full Real-World Generalization Results

Here in this section, we report the full real-world generalization results. All aggregate metrics are computed from unrounded underlying values; rounding is applied only to the final reported number.