What Visual Representations Do World Action Models Need?

Abstract

World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. Recent studies show that external visual representations improve manipulation performance and robustness. However, these methods select representations based on presumed capability gaps and use different integration mechanisms, making it difficult to disentangle the effects of representation choice and integration. To better understand how external representations strengthen WAMs, we conduct a systematic study through controlled comparisons of multiple visual representations, five integration mechanisms, and feature combinations. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert’s effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R2-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.

1 Introduction

World Action Models (WAMs) adapt pretrained video generation models to robot control by jointly predicting future video and actions. This allows robot policies to draw on the visual structure and temporal dynamics learned through large-scale video pretraining, as demonstrated by DreamZero (Ye et al., 2026b), LingBot-VA (Li et al., 2026a), and Fast-WAM (Yuan et al., 2026). Along with these priors, WAMs inherit the latent space of their variational autoencoders (VAEs).

The inherited VAE is optimized for visual reconstruction (Wan Team et al., 2025), but how well its representations support robot manipulation remains unclear. Its reconstruction objective does not explicitly prioritize manipulation-relevant information, such as object semantics and spatial relationships. Recent studies augment WAMs with external visual features and report improvements in manipulation performance and robustness (Qiu et al., 2026; Yan et al., 2026b; Ma et al., 2026a). However, these methods select different representations based on prior assumptions about which visual capabilities WAMs lack. Systematic comparisons across representation types remain limited, leaving unclear which visual representations best support robot manipulation in WAMs.

Beyond representation choice, WAM performance also depends on the mechanism used to integrate external features into the model (Yang et al., 2026). A further question is whether combining more representations leads to continued performance gains. External features can be integrated through representation alignment (Yu et al., 2025; Qiu et al., 2026), action conditioning (Wang et al., 2026c), or joint prediction (Yan et al., 2026a). Existing studies often couple selected representations with specific integration designs (Qiu et al., 2026; Yan et al., 2026a; Zhang et al., 2026a), making the effects of representation choice and integration mechanism difficult to disentangle across methods. Besides, the diverse demands of manipulation—including semantics, geometry, object extent, motion, and interaction—suggest that different representations may be needed. Yet additional sources may also introduce redundant information or interference during optimization. Whether and when different feature combinations improve WAM performance therefore remains an empirical question.

To address these questions, we systematically study visual representations in WAMs, focusing on capabilities, integration mechanisms, and composition. Using a common WAM backbone built on a pretrained video generation model, we compare diverse visual representations, five integration mechanisms, and multiple source combinations under a shared training protocol. Our study addresses three research questions (RQs):

• Which visual representations improve WAM performance?: Which feature injection mechanism yields the greatest performance gains from external features?: Does combining more feature sources consistently improve WAM performance?

Our controlled comparisons yield three findings that guide the selection, integration, and composition of external visual representations for WAMs. (1) General-purpose visual representations should be the default choice for augmenting WAMs. They can match or outperform representations pretrained for specific visual capabilities—even under perturbations relevant to those capabilities—while providing broader robustness gains. (2) Channel-wise joint prediction yields the largest average gains across matched sources. Our analyses suggest that future-feature prediction and effective use of the added visual information by the action expert contribute to these gains and should guide future integration designs. (3) Additional sources should be selected for the task-relevant information they add beyond the existing representations. The same source can improve one combination but provide little benefit or even reduce performance in another. Although feature composition can provide further gains, these are limited compared with the initial improvement. Feature composition should therefore serve as a supplementary route to improvement.

Our contributions are threefold:

• A controlled study of visual representations in WAMs. We systematically compare representation sources, five integration mechanisms, and feature combinations within a common WAM backbone and shared training protocol, enabling controlled comparisons of their effects on manipulation performance and robustness.
• Empirical guidance for WAM representation design. We derive practical recommendations for selecting, integrating, and combining external visual representations. These findings motivate future research on stronger general-purpose visual foundation models and more efficient and effective mechanisms for integrating their representations into WAMs.
• R2-WAM and validation beyond the controlled study. We instantiate these findings as a Representation Recipe for WAMs, termed R2-WAM. We validate the effectiveness of the recipe across different WAM formulations on LIBERO-Plus, and further demonstrate improvements over the corresponding VAE-only baselines on RoboCasa GR1 Tabletop and RoboTwin 2.0, as well as in real-world robot experiments.

2 Related Work

World Action Models.

WAMs learn action prediction alongside future visual dynamics. DreamZero (Ye et al., 2026b) transfers a pretrained video diffusion backbone to joint video–action generation, and LingBot-VA (Li et al., 2026a) combines autoregressive diffusion with closed-loop visual feedback. Fast-WAM (Yuan et al., 2026) shows that video co-training can benefit control even when future video generation is omitted at inference. Other studies modify the visual prediction space or its supervision. LDA-1B (Lyu et al., 2026) learns dynamics in DINO feature space, while LaWAM (Chen et al., 2026a) conditions action generation on predicted latent visual subgoals. RepWAM (Wang et al., 2026b) and LingBot-VA 2.0 (Zhang et al., 2026b) learn semantically aligned visual–action tokenizers. ST-WAM (Wang et al., 2026c) diagnoses failures under visual shifts and combines DINO future prediction and history retrieval with VAE dynamics. Robust-WAM (Yan et al., 2026b) aligns queries in the action stream with future semantic features while retaining the VAE generation pathway. Flex-\pi (Yan et al., 2026a) jointly denoises RGB latents, 3D pointmaps, DINO features, and actions, supporting different combinations of input and prediction streams at deployment. These studies demonstrate the effectiveness of specific visual representations within their respective WAM designs, but systematic comparisons are still needed to identify which features complement the capabilities that WAMs inherit from pretrained video generators.

Visual Representations.

Pretraining objectives shape the information captured by visual representations. Visual self-supervision has produced encoders such as DINOv3 (Siméoni et al., 2026), VC-1 (Majumdar et al., 2023), LingBot-Vision (Fu et al., 2026), VideoMAE V2 (Wang et al., 2023), and V-JEPA 2.1 (Mur-Labadia et al., 2026). Language-associated approaches include image–text models such as CLIP (Radford et al., 2021) and SigLIP 2 (Tschannen et al., 2025), as well as multimodal language models such as Qwen3.5 (Qwen Team, 2026). Other pretraining objectives target segmentation with SAM 3 (Carion et al., 2025), geometry with VGGT-\Omega (Wang et al., 2026a), and interaction-related information with HRP (Srirama et al., 2024) and DynaFLIP (Lee et al., 2026). UniTok (Ma et al., 2025) and C-RADIOv4 (Ranzinger et al., 2026) further combine multiple learning objectives or teacher models within a single representation. These approaches provide different but overlapping visual capabilities, but how their pretraining emphases translate into manipulation gains in WAMs remains an empirical question.

3 Study Design

3.1 Base WAM Architecture

Given RGB observations, a language instruction, and the current robot state, the base WAM jointly predicts future-video latents z_{1:T}^{\mathrm{vae}} and an action chunk a. The model comprises a video expert initialized from Wan 2.2 (Wan Team et al., 2025) and a separate action expert. The experts interact through joint attention with an asymmetric mask: action tokens attend to both video and action tokens, whereas video tokens attend only to video tokens. Both experts attend to T5-encoded language features (Raffel et al., 2020) through cross-attention and are additionally conditioned on the robot state and flow time.

Throughout, t=0 denotes the current observation, t=1,\ldots,T indexes the future sequence, and \tau denotes flow time. Thus z_{0}^{\mathrm{vae}} denotes current-frame VAE latents, and z_{1:T,\tau}^{\mathrm{vae}} denotes future-video latents at flow time \tau.

We train the model with a weighted sum of video and action flow-matching losses (Lipman et al., 2023):

\mathcal{L}_{\mathrm{WAM}}=\mathcal{L}_{\mathrm{FM}}(z_{1:T}^{\mathrm{vae}})+\lambda_{\mathrm{act}}\mathcal{L}_{\mathrm{FM}}(a),

where \mathcal{L}_{\mathrm{FM}} denotes the modality-specific mean-squared error between predicted and target flow velocities, and \lambda_{\mathrm{act}} weights the action loss. At inference, future-video latents and actions are initialized from noise and jointly denoised.

3.2 Visual Representations

We compare thirteen pretrained visual encoders, using their pretraining objectives as an initial indication of the information their representations capture and the visual capabilities they may support. All encoders remain frozen during WAM training and extract representations from the RGB observations available to the base WAM. We consider six pretraining emphases:

Visual self-supervision: VC-1 (Majumdar et al., 2023) and VideoMAE V2 (Wang et al., 2023) learn visual structure through masked image or video reconstruction. V-JEPA 2.1 (Mur-Labadia et al., 2026) predicts visual features to learn spatial and temporal relationships; DINOv3 (Siméoni et al., 2026) uses self-distillation to learn semantic and dense spatial features; and LingBot-Vision (Fu et al., 2026) emphasizes fine spatial structure through masked boundary modeling.

Language-associated pretraining: SigLIP 2 (Tschannen et al., 2025) combines image–text alignment with captioning and self-supervised objectives to learn semantic and dense visual features. Qwen3.5 (Qwen Team, 2026) uses multimodal language modeling for diverse visual understanding tasks.

Segmentation supervision: SAM 3 (Carion et al., 2025) learns object regions and boundaries through concept-conditioned detection and segmentation.

Geometric supervision: VGGT-\Omega (Wang et al., 2026a) learns scene geometry through 3D reconstruction.

Interaction-oriented pretraining: HRP (Srirama et al., 2024) uses hand, object, and contact supervision to learn manipulation affordances. DynaFLIP (Lee et al., 2026) aligns images, language, and 3D motion to capture information relevant to interaction dynamics.

Multi-objective learning and multi-teacher distillation: UniTok (Ma et al., 2025) combines reconstruction and image–text learning to retain visual detail and semantics, while C-RADIOv4 (Ranzinger et al., 2026) distills DINOv3, SigLIP 2, and SAM 3.

Based on these pretraining objectives and capability emphases, we designate representations intended for broad transfer across visual or vision–language tasks as general-purpose, and those pretrained primarily for specific visual capabilities as specialized. Table 1 lists the sources and their group assignments.

3.3 Feature Integration Mechanisms

We compare five mechanisms that use external features for representation alignment, action conditioning, or joint prediction (Figure 2). The three joint-prediction mechanisms differ in how feature and VAE representations are fused and whether their Transformer parameters are shared.

We denote current and future external features by f_{0} and f_{1:T}, respectively, with a superscript k identifying the source when needed. The state of future features at flow time \tau is written as f_{1:T,\tau}.

Representation alignment (REPA): A trainable projector maps intermediate WAM hidden states into the external feature space. A cosine-distance loss aligns the projected states with spatially corresponding features from the frozen source encoder (Yu et al., 2025).

Gated cross-attention: An additional cross-attention module uses action hidden states as queries and projected features from the current observations as keys and values. A learnable gate scales the module’s output before it is added to the action hidden states (Alayrac et al., 2022). The existing cross-attention to T5 language features is retained.

Channel-wise concatenation: We spatially align external features with the VAE latent grid and concatenate them along the channel dimension. The resulting representation is processed by the video expert, whose input and output projections are expanded to jointly predict VAE and external-feature flow velocities. The video-token layout and attention mask remain unchanged (Tan et al., 2026).

Token-wise concatenation. We represent external features as a separate token sequence with its own input projection and future-feature prediction head. Feature and VAE tokens share the video expert’s Transformer layers (Yan et al., 2026a).

Feature expert: We introduce a dedicated Transformer expert to predict future external features. Unlike token-wise concatenation, this mechanism uses independent feature-processing parameters. Feature, video, and action tokens interact through masked joint attention, while each expert retains its own projections and feed-forward layers (Bi et al., 2026).

For REPA and the three joint-prediction mechanisms, the training objective is

\mathcal{L}=\mathcal{L}_{\mathrm{WAM}}+\lambda_{\mathrm{aux}}\mathcal{L}_{\mathrm{aux}},

where \mathcal{L}_{\mathrm{aux}} is the cosine-distance alignment loss for REPA and the future-feature flow-matching loss for the joint-prediction mechanisms. Gated cross-attention is trained using \mathcal{L}_{\mathrm{WAM}} alone.

We evaluate feature composition using channel-wise concatenation as the common integration mechanism. Features from each source are spatially aligned with the VAE latent grid and concatenated along the channel dimension.

4 Experiments

4.1 Experimental Setup

Training.

We use the LIBERO (Liu et al., 2023) and RoboTwin 2.0 (Chen et al., 2026b) training datasets from Fast-WAM (Yuan et al., 2026) and the RoboCasa GR1 Tabletop (Nasiriany et al., 2024) dataset used in DIAL (Chen et al., 2026c). Across the LIBERO comparisons (RQ1–RQ3), we hold the training data, base observation inputs, action horizon, optimizer, and learning-rate schedule fixed while varying representation sources, integration mechanisms, or source combinations. Training configurations for simulation and real-robot experiments are provided in Appendix C.

Evaluation.

We evaluate LIBERO (Liu et al., 2023) with 50 rollouts per task following Fast-WAM (Yuan et al., 2026); full results are reported in the Appendix D since performance is near saturation. The same models are evaluated on LIBERO-Plus (LP) (Fei et al., 2026) without further training, using three evaluation seeds and all 10,030 episodes per seed across seven perturbation categories. We report per-category and overall success rates. RoboCasa GR1 Tabletop (Nasiriany et al., 2024; Chen et al., 2026c) uses 50 trials per task across 24 tasks, while RoboTwin 2.0 (Chen et al., 2026b; Yuan et al., 2026) uses 100 trials per task across 50 tasks in each of the clean and randomized settings. We evaluate the VAE-only WAM and R2-WAM on four real-robot tasks with 20 trials per task per model; metric definitions are provided in Appendix C.3.

4.2 RQ1: Which visual representations improve WAM performance?

We compare the visual representations introduced in Section 3.2. All representations are integrated using channel-wise concatenation, with the VAE-only WAM serving as the baseline. Table 1 reports success rates as mean \pm standard deviation across three evaluation seeds for each of the seven LP perturbation categories and overall. We examine both overall performance and the range of perturbations under which each source improves robustness.

Viewpoint, background, and noise perturbations expose key robustness gaps in the VAE-only WAM.

The VAE-only WAM achieves high success rates under language perturbations (91.1\%) and lighting changes (94.3\%), yet performs substantially worse under viewpoint changes (51.2\%), background changes (62.0\%), and image noise (66.6\%). This uneven robustness motivates examining whether external visual representations can address these weaknesses and improve performance across perturbation categories.

General-purpose representations show stronger robustness across perturbations.

Across the sources evaluated in Table 1, general-purpose representations achieve a higher average success rate than specialized representations (82.2\% vs. 80.0\%). A general-purpose source also achieves the highest observed mean success rate in each of the seven perturbation categories, including camera, background, and image noise, where the VAE-only WAM is weakest.

Takeaway 1. General-purpose visual representations should be the default for augmenting WAMs.

Rather than choosing a specialized encoder to remedy an assumed missing capability, start with a general-purpose visual representation. In our study, general-purpose representations provide broader robustness gains and can match or exceed specialized alternatives even where their specialization appears relevant.

4.3 RQ2: Which feature integration mechanism yields the greatest performance gains from external features?

We compare the five integration mechanisms in Section 3.3 across thirteen visual representations (Figure 3(a); per-source results in Appendix D). We then examine future-feature prediction, predicted feature structure, and action sensitivity to investigate the gains from channel-wise concatenation (Figure 3(b–d)).

Channel-wise concatenation achieves the highest average success rate.

Across the thirteen visual representations in Table 1, channel-wise concatenation achieves an average LP success rate of 81.5\%, exceeding the next-best mechanism by 5.1 percentage points (Figure 3(a)).

Why channel-wise concatenation performs well.

Figure 3(b) separates current-feature conditioning from future-feature prediction. Future-only prediction reaches 81.9%, close to the full configuration (84.0%) and above current-only conditioning (76.5%), supporting future-feature prediction as a major contributor.

The shared-PCA visualizations in Figure 3(c) show that the video expert recovers coarse spatial structure in future features. Token-wise predictions also retain recognizable structure, whereas the feature-expert prediction is coarser in the displayed example.

We analyze how action generation depends on external features (Figure 3(d)). Channel-wise concatenation exhibits higher translation and rotation sensitivities in these states, indicating stronger dependence of action generation on the added visual representations.

Takeaway 2. Use channel-wise concatenation as the default.

For new designs, support future-feature prediction and encourage the action expert to make effective use of the added features. Check prediction quality and how added features affect generated actions, and verify that design changes improve task success rates.

4.4 RQ3: Does combining more feature sources consistently improve WAM performance?

Building on RQ1 and RQ2, we use DINOv3 (Siméoni et al., 2026) as the initial external representation and keep channel-wise concatenation fixed. The candidate additions span different pretraining emphases, with representatives from each category in Section 3.2: V-JEPA 2.1 (Mur-Labadia et al., 2026) for visual self-supervision, SigLIP 2 (Tschannen et al., 2025) for language-associated learning, SAM 3 (Carion et al., 2025) for segmentation, VGGT-\Omega (Wang et al., 2026a) for geometry, HRP (Srirama et al., 2024) for interaction-oriented pretraining, and UniTok (Ma et al., 2025) and C-RADIOv4 (Ranzinger et al., 2026) for multi-objective / multi-teacher learning. We first add each candidate separately to DINOv3, then evaluate selected larger combinations through single-source additions (Figure 4).

A general-purpose source provides the main gain.

DINOv3 (Siméoni et al., 2026) alone improves LP success by 9.3 percentage points over VAE-only, while adding other features yields at most a further 3.7-point gain among the evaluated combinations.

The benefit of an additional source depends on the existing representations.

The composition results suggest that overlap in pretraining objectives can limit complementary gains. Both UniTok (Ma et al., 2025) and SigLIP 2 (Tschannen et al., 2025) learn language-aligned visual representations. Adding SigLIP 2 improves DINOv3 (Siméoni et al., 2026) by 1.8 percentage points, but reduces success by 0.4 points once UniTok is included and adds only 0.2 points when V-JEPA 2.1 (Mur-Labadia et al., 2026) is also present. Spatial capabilities similarly overlap: DINOv3 already provides dense features supporting segmentation and depth estimation, while VGGT-\Omega (Wang et al., 2026a) uses a DINOv3-initialized visual encoder. Adding VGGT-\Omega or SAM 3 (Carion et al., 2025) to DINOv3 produces little change, although both improve some larger combinations. A direct teacher–student relationship occurs with C-RADIOv4 (Ranzinger et al., 2026), which distills DINOv3, SigLIP 2, and SAM 3: combining it with DINOv3 reduces LP success to 78.6\%. Meanwhile, adding V-JEPA 2.1 (Mur-Labadia et al., 2026) to DINOv3 and UniTok improves performance, consistent with its video-based predictive pretraining providing complementary cues.

Takeaway 3. Additional sources should be selected for complementarity.

Build on a strong general-purpose representation and use composition for selective refinement. When choosing additional sources, consider which task-relevant information they may contribute beyond the existing features, accounting for overlap in pretraining objectives and teacher models.

4.5 Representation Recipe and Validation

R2-WAM combines DINOv3 (Siméoni et al., 2026), UniTok (Ma et al., 2025), and V-JEPA 2.1 (Mur-Labadia et al., 2026) through channel-wise concatenation. Adding HRP (Srirama et al., 2024) yields the highest mean LP success (87.7\% versus 87.3\%), but only a 0.4-percentage-point gain. We retain three sources to limit auxiliary encoders and feature-prediction targets.

Validation across WAM formulations.

Table 2(a) compares VAE-only baselines with their recipe-augmented counterparts under two additional WAM formulations: direct action prediction (Yuan et al., 2026) and video-then-action prediction (IDM) (Li et al., 2026a). Implementation details are provided in Appendix B.1. These results support the benefit of the recipe across both evaluated formulations.

Cross-benchmark validation.

R2-WAM achieves a state-of-the-art success rate of 84.7\% on RoboCasa GR1 Tabletop (Nasiriany et al., 2024). On RoboTwin 2.0 (Chen et al., 2026b), it reaches 92.8\%. The gains over our VAE-only baselines are 5.9 and 2.2 percentage points, respectively, supporting the effectiveness of the same representation configuration across robot embodiments and environments.

Real-robot validation.

Table 2(b) reports performance on the four physical tasks shown in Figure 5. R2-WAM improves complete-task success by 10.0 percentage points on each of air-fryer food preparation, instructed pick-and-place, and cup pushing. On tool sweeping, the mean debris collection rate increases from 48.3\% to 66.7\%. These results extend the improvements from the selected representation recipe to physical manipulation, including sequential manipulation and tool use.

5 Conclusion

In this work, we systematically study how the selection, integration, and composition of external visual representations affect VAE-based WAMs. Our findings suggest a practical recipe: start with a strong general-purpose representation, integrate it through channel-wise concatenation, and retain additional sources when they improve task success rates in the intended combination. R2-WAM instantiates this recipe and demonstrates gains across the evaluated robot embodiments and environments.

Our findings provide a starting point for further research on visual representations and their use in WAMs. Future work should focus on training stronger general-purpose visual representations and developing integration mechanisms that improve future-feature prediction and enable action experts to use visual information more effectively.

References

1. Alayrac et al. (2022) J. Alayrac, J. Donahue, P. Luc, A. Miech, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems,
2. Bi et al. (2026) H. Bi, H. Tan, S. Xie, et al. Motus: a unified latent action world model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
3. Black et al. (2025) K. Black, N. Brown, J. Darpinian, et al. \pi_{0.5}: a vision-language-action model with open-world generalization. In Proceedings of The 9th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 305, pp. 17–40. External Links: Link
4. Cai et al. (2026) J. Cai, Z. Cai, J. Cao, et al. InternVLA-A1: unifying understanding, generation and action for robotic manipulation. arXiv preprint arXiv:2601.02456. External Links: Link
5. Carion et al. (2025) N. Carion, L. Gustafson, Y. Hu, et al. SAM 3: segment anything with concepts. arXiv preprint arXiv:2511.16719.
6. Chen et al. (2026a) J. Chen, K. Wang, K. Chen, et al. LaWAM: latent world action models for efficient dynamics-aware robot policies. arXiv preprint arXiv:2606.15768.
7. Chen et al. (2026b) T. Chen, Z. Chen, B. Chen, et al. RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In International Conference on Machine Learning,
8. Chen et al. (2026c) Y. Chen, Y. Ge, H. Zhou, M. Ding, Y. Ge, and X. Liu DIAL: decoupling intent and action via latent world modeling for end-to-end VLA. arXiv preprint arXiv:2603.29844. External Links: Link
9. Fei et al. (2026) S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu LIBERO-plus: a progressive robustness benchmark for visual-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
10. Fu et al. (2026) Z. Fu, B. Tan, C. Sun, S. Liu, K. Zheng, Y. Xu, X. Zhu, Y. Shen, and N. Xue Vision pretraining for dense spatial perception. arXiv preprint arXiv:2607.05247. External Links: Link
11. Hoffman et al. (2019) J. Hoffman, D. A. Roberts, and S. Yaida Robust learning with Jacobian regularization. arXiv preprint arXiv:1908.02729. External Links: Link
12. Lee et al. (2026) J. Lee, S. Lee, J. Shin, H. Jung, S. Kim, D. Cho, H. J. Kim, J. Huang, and F. Huang DynaFLIP: rethinking robotics perception via tri-modal-dynamics guided representation. arXiv preprint arXiv:2605.30350. External Links: Link
13. Li et al. (2026a) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu Causal world modeling for robot control. arXiv preprint arXiv:2601.21998.
14. Li et al. (2026b) X. Li, Y. Zheng, S. Gu, H. Ma, F. Yu, Y. Zheng, X. Nie, S. Yuan, Y. Zang, W. Li, S. Tian, M. Liu, Y. Zhang, and W. Ding Latent action as intention enables efficient future imagination for world action models. arXiv preprint arXiv:2608.24882. External Links: Link
15. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In International Conference on Learning Representations,
16. Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track,
17. Liu et al. (2026) J. Liu, Z. Xia, S. Tian, H. Li, Y. Zheng, N. Ma, X. Fu, X. Liu, J. Li, Y. Li, S. Zhou, Z. Xing, L. Wang, C. Li, H. Li, and D. Zhao WALA learning executable latent actions from action-labeled demonstrations and action-free videos. arXiv preprint arXiv:2607.11397. External Links: Link
18. Lyu et al. (2026) J. Lyu, K. Liu, X. Zhang, et al. LDA-1b: scaling latent dynamics action model via universal embodied data ingestion. In Proceedings of Robotics: Science and Systems,
19. Ma et al. (2025) C. Ma, Y. Jiang, J. Wu, J. Yang, X. Yu, Z. Yuan, B. Peng, and X. Qi UniTok: a unified tokenizer for visual generation and understanding. In Advances in Neural Information Processing Systems, Vol. 38. External Links: Document, Link
20. Ma et al. (2026a) F. Ma, D. Peng, W. Yue, J. Cao, B. Wang, Q. Zhang, and J. Ma GeoSem-wam: geometry- and semantic-aware world action models. arXiv preprint arXiv:2606.03188.
21. Ma et al. (2026b) T. Ma, J. Zheng, Z. Wang, C. Jiang, A. Cui, J. Liang, and S. Yang DiT4DiT: jointly modeling video dynamics and actions for generalizable robot control. arXiv preprint arXiv:2603.10448. External Links: Link
22. Majumdar et al. (2023) A. Majumdar, K. Yadav, S. Arnaud, J. Ma, C. Chen, S. Silwal, A. Jain, V. Berges, T. Wu, J. Vakil, P. Abbeel, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and F. Meier Where are we in the search for an artificial visual cortex for embodied intelligence?. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Document, Link
23. Mur-Labadia et al. (2026) L. Mur-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, N. Ballas, and A. Bardes V-JEPA 2.1: unlocking dense features in video self-supervised learning. In European Conference on Computer Vision,
24. Nasiriany et al. (2024) S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu RoboCasa: large-scale simulation of household tasks for generalist robots. In Proceedings of Robotics: Science and Systems,
25. NVIDIA GEAR Team (2025) NVIDIA GEAR Team GR00T N1.6: an improved open foundation model for generalist humanoid robots. Note: NVIDIA Research External Links: Link
26. Qiu et al. (2026) L. Qiu, Y. Li, Y. Chen, Y. Ge, Y. Ge, and X. Liu Making foresight actionable: repurposing representation alignment in world action models. arXiv preprint arXiv:2606.12217.
27. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link
28. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning,
29. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research.
30. Ranzinger et al. (2026) M. Ranzinger, G. Heinrich, C. McCarthy, J. Kautz, A. Tao, B. Catanzaro, and P. Molchanov C-RADIOv4 (tech report). arXiv preprint arXiv:2601.17237.
31. Siméoni et al. (2026) O. Siméoni, H. V. Vo, M. Seitzer, et al. DINOv3. Transactions on Machine Learning Research.
32. Simonyan et al. (2014) K. Simonyan, A. Vedaldi, and A. Zisserman Deep inside convolutional networks: visualising image classification models and saliency maps. In International Conference on Learning Representations Workshop,
33. Srirama et al. (2024) M. K. Srirama, S. Dasari, S. Bahl, and A. Gupta HRP: human affordances for robotic pre-training. In Proceedings of Robotics: Science and Systems, Delft, Netherlands. External Links: Document, Link
34. Tan et al. (2026) B. Tan, X. Zhang, N. Liao, Y. Zhang, S. Zhang, X. Yang, Q. Fan, and Y. Zhang DreamWorld: unified world modeling in video generation. arXiv preprint arXiv:2603.00466.
35. Tschannen et al. (2025) M. Tschannen, A. Gritsenko, X. Wang, et al. SigLIP 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786.
36. Wan Team et al. (2025) Wan Team, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, et al. Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314.
37. Wang et al. (2026a) J. Wang, M. Chen, S. Zhang, N. Karaev, J. Schönberger, P. Labatut, P. Bojanowski, D. Novotny, A. Vedaldi, and C. Rupprecht VGGT-\Omega. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
38. Wang et al. (2026b) J. Wang, Q. Zhang, S. Yang, Y. Luo, Y. Shen, Z. Wu, Y. Jiang, and Y. Xu RepWAM: world action modeling with representation visual-action tokenizers. arXiv preprint arXiv:2606.13674.
39. Wang et al. (2023) L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao VideoMAE V2: scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14549–14560. External Links: Link
40. Wang et al. (2026c) M. Wang, B. Hu, B. Qian, et al. ST-wam: semantic-temporal world action model for robust manipulation under visual distribution shifts. arXiv preprint arXiv:2607.28993.
41. Yan et al. (2026a) G. Yan, J. Liu, Y. Fan, L. Cai, M. Liao, J. Zhang, and D. Fox Flex-\pi: a multi-stream world-action model with compute flexibility. arXiv preprint arXiv:2608.10860.
42. Yan et al. (2026b) H. Yan, J. Li, J. He, et al. Robust-wam: bridging generative pretraining and semantic foresight in world-action models. arXiv preprint arXiv:2608.05903.
43. Yang et al. (2026) L. Yang, W. Song, X. Wang, P. Sheng, Z. Fang, Z. Zhou, J. He, H. Yan, J. Chen, N. Sun, Q. Sun, P. Wang, L. Liu, Y. Wang, Y. Gao, F. Dayoub, and H. Li 4D-WAM: infusing spatiotemporal awareness into world action models through trajectory fields. arXiv preprint arXiv:2608.08023. External Links: Link
44. Ye et al. (2026a) J. Ye, N. Gao, S. Yang, J. Zheng, Z. Wang, Y. Chen, P. Chen, Y. Chen, S. Liu, and J. Jia StarVLA-\alpha: reducing complexity in vision-language-action systems. arXiv preprint arXiv:2604.11757. External Links: Link
45. Ye et al. (2026b) S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, A. Malik, K. Lee, W. Liang, N. Ranawaka, J. Gu, Y. Xu, G. Wang, F. Hu, A. Narayan, J. Bjorck, J. Wang, G. Kim, D. Niu, R. Zheng, Y. Xie, J. Wu, Q. Wang, R. Julian, D. Xu, Y. Du, Y. Chebotar, S. Reed, J. Kautz, Y. Zhu, L. Fan, and J. Jang World action models are zero-shot policies. arXiv preprint arXiv:2602.15922.
46. Yu et al. (2025) S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie Representation alignment for generation: training diffusion transformers is easier than you think. In International Conference on Learning Representations,
47. Yu et al. (2023) W. Yu, C. Yang, C. McGreavy, E. Triantafyllidis, G. Bellegarda, M. Shafiee, A. J. Ijspeert, and Z. Li Identifying important sensory feedback for learning locomotion skills. Nature Machine Intelligence 5, pp. 919–932. External Links: Document, Link
48. Yuan et al. (2026) T. Yuan, Z. Dong, Y. Liu, and H. Zhao Fast-wam: do world action models need test-time future imagination?. arXiv preprint arXiv:2603.16666.
49. Zhang et al. (2026a) J. Zhang, J. Zhu, T. Su, C. Ma, Z. Huang, Y. Xu, and H. Wang Learning 4D geometric priors for inference-efficient world action models. arXiv preprint arXiv:2607.05468. External Links: Link
50. Zhang et al. (2026b) Q. Zhang, L. Li, L. Zhang, S. Yang, Y. Luo, S. Li, R. Wang, J. Wang, J. Shao, G. Xu, J. Zhou, Y. Shen, Y. Jin, F. Xu, S. Ma, J. Liao, G. Lu, Z. Shi, Y. Wen, Y. Zhao, W. Tang, X. Wang, C. Li, J. Zhu, K. L. Cheng, N. Xue, X. Zhu, Y. Shen, and Y. Xu Native video-action pretraining for generalizable robot control. arXiv preprint arXiv:2607.08639.
51. Zheng et al. (2025) R. Zheng, J. Wang, S. Reed, et al. FLARE: robot learning with implicit world modeling. arXiv preprint arXiv:2505.15659. External Links: Link

Appendix A Visual Representation Construction

Encoder configurations.

We primarily use large (L) variants of the visual encoders. For Qwen3.5, we use Qwen3.5-4B.

Feature extraction.

We use token features from the final layer of each encoder. For Qwen3.5, we extract the final-layer hidden states at visual-token positions from a single forward pass.

Appendix B Feature Integration Implementation Details

B.1 WAM Formulations

We implement three WAM formulations within our base architecture, differing in the visual information available to the action expert and the training and inference procedures (Figure 6).

Joint video–action denoising (default).

Future visual tokens and action tokens are jointly denoised. As described in Section 3.1, action tokens attend to current-frame visual tokens, noisy future visual tokens, and action tokens, whereas visual tokens do not attend to action tokens. At inference, future visual tokens and actions are initialized from noise and jointly denoised.

Direct action prediction.

Following the direct-prediction paradigm of Fast-WAM (Yuan et al., 2026), the attention mask restricts the action expert’s visual context to current-frame tokens, preventing access to future visual tokens. At inference, actions are denoised conditioned on the current observation without generating future visual tokens.

Video-then-action prediction (IDM).

Following the inverse-dynamics paradigm of LingBot-VA (Li et al., 2026a), the action expert predicts actions conditioned on current and future visual tokens. During training, it receives clean ground-truth future visual tokens. At inference, the video expert first predicts future visual tokens, which then serve as conditions for action denoising.

VAE-only baselines.

For each formulation, VAE-only denotes the corresponding model using VAE representations without additional external visual representations.

B.2 Integration Mechanisms

We use the notation introduced in Section 3.1 and Section 3.3: current and future VAE latents are z_{0}^{\mathrm{vae}} and z_{1:T}^{\mathrm{vae}}, and the corresponding external features are f_{0} and f_{1:T}. Observation indices t=0,1,\ldots,T are distinct from flow time \tau, which is appended as a separate subscript for noised future states. Each integration mechanism adapts source features to its required spatiotemporal layout.

Representation alignment (REPA). The student representation is the output of the eighth video-expert block. A trainable three-layer MLP P maps this state to the source feature dimension, with SiLU after the first two linear layers. The teacher trajectory is aligned to the video-token grid: spatial resizing is bilinear, the observed temporal slice is preserved, and the eight future slices are averaged into two groups of four. Padding masks are aligned with the features and remove invalid locations. After \ell_{2} normalization, the alignment objective is

\mathcal{L}_{\mathrm{REPA}}=1-\frac{1}{|\Omega|}\sum_{i\in\Omega}\left\langle\widehat{P(h_{i}^{8})},\widehat{f_{i}}\right\rangle,

where \Omega contains valid future tokens and excludes the observed slice. The frozen teacher and the trainable projector are used only during training and are absent from inference.

Gated cross-attention. An independent feature cross-attention branch is inserted into each of the 30 action-expert blocks. The normalized action state after mixed attention is used as the query, while normalized features from the current observation f_{0} provide keys and values through an adapter to the action-expert width. The original T5 language cross-attention is retained, and both residual branches are computed from the same post-mixed-attention state \bar{h}_{a}^{\ell}:

\widetilde{h}_{a}^{\ell}=\bar{h}_{a}^{\ell}+\operatorname{CA}_{\mathrm{text}}^{\ell}(\bar{h}_{a}^{\ell},c)+\tanh(g_{\ell})\,\operatorname{CA}_{\mathrm{feat}}^{\ell}(\bar{h}_{a}^{\ell},f_{0}).

The resulting state is subsequently processed by the block FFN. Every layer has its own scalar gate, initialized with g_{\ell}=0, so the added branch is initially neutral. This mechanism has no auxiliary feature loss. The frozen visual encoder and feature cross-attention branches remain active at inference.

Channel-wise concatenation. Following the channel-wise integration design of DreamWorld (Tan et al., 2026), we align external visual features with the VAE latent grid and concatenate them along the channel dimension. This preserves the video token grid and attention sequence length. The input and output projections are expanded to accommodate the added features while retaining the pretrained VAE weights. Observed features remain clean, whereas future VAE latents and external features are jointly noised and predicted. At inference, future features are initialized from noise and participate in joint denoising; only the VAE latents are decoded.

Token-wise concatenation. Dense external features are flattened into a separate token sequence, mapped from C_{f} to the video-expert hidden width by an independent input projection, and appended to the VAE tokens. Feature and VAE tokens share all video-expert Transformer blocks, while a separate prediction head maps feature hidden states back to the source dimension C_{f}. Features are flattened in temporal, row, and column order into observed tokens F_{0} and future tokens F_{1:T}. LayerNorm precedes both the input projection and the prediction head; their linear projections use Xavier-uniform weights and zero biases. The combined token ordering is [V_{0},V_{1:T},F_{0},F_{1:T},A].

Feature tokens use the video expert’s three-dimensional rotary positions, with feature-grid coordinates mapped to the video-token coordinate system. The attention mask allows observed video and feature tokens to interact only with observed visual tokens. Future video and feature tokens jointly attend to both observed and future video/feature streams. Action tokens attend to observed and noisy future video/feature tokens, as well as action tokens. Video and feature tokens do not attend to action tokens.

During training, F_{0} remains clean and F_{1:T} receives Gaussian noise sampled independently from the video noise. Feature and video flows use the same sampled timestep and the same scheduler, with feature velocity target \epsilon_{f}-f_{1:T}, where \epsilon_{f} is the sampled feature noise. The scheduler-weighted MSE is evaluated over valid future feature tokens only. At inference, F_{0} is extracted from the observed views and kept clean, while F_{1:T} is initialized from Gaussian noise and jointly denoised with video and actions. The observed prefix F_{0} is restored after every solver step, and the final predicted feature trajectory is discarded.

Feature expert. This mechanism introduces an independent 30-block Transformer expert with hidden width 1024, FFN width 4096, 24 attention heads, and head dimension 128. Its input and output projections map between the source dimension C_{f} and the expert hidden width.

At every layer, the video, feature, and action experts independently form their Q/K/V tensors. The tensors are concatenated for masked joint attention, then split by modality before each expert applies its own attention output projection, text cross-attention, normalization layers, and FFN. Joint attention does not imply parameter sharing: the feature expert owns independent Transformer parameters. Its attention mask follows the same routing as token-wise concatenation: action queries attend to observed and noisy future video/feature tokens and action tokens, while video and feature queries cannot attend to action tokens. Feature noise is independent of video noise, while the feature and video flows share the same timestep and scheduler. The observed feature prefix remains clean, and only valid future features enter the flow-matching loss. During inference, future features are initialized from noise and participate in joint denoising; the observed prefix is restored after every step, and the predicted feature trajectory is discarded.

Unless otherwise specified, auxiliary feature-prediction losses use a weight of 1.0. The feature expert uses a feature-prediction loss weight of 0.1, and REPA uses an alignment weight of 0.1. Sensitivity experiments across five visual encoders compare REPA depths of 8, 15, and 22 at this weight, and weights of 0.1 and 0.5 at depth 8 (Table 5). Increasing the alignment weight does not improve the mean LIBERO-Plus success rate, and the mean varies by less than 0.1 percentage points across the four configurations, although individual encoders exhibit different preferences.

We compare five feature-integration mechanisms using a common WAM backbone and training budget. The feature preprocessing and auxiliary-loss settings for each mechanism are specified above.

Appendix C Training and Evaluation Configurations

C.1 Simulation Training

Table 6 lists the common training settings, and Table 7 summarizes the benchmark-specific configurations. Within the main LIBERO comparisons (RQ1–RQ3), these settings are held fixed while the representation source, integration mechanism, or source set varies as specified in the main text.

C.2 Simulation Evaluation and Reporting

Benchmark coverage and aggregation.

LIBERO uses 50 rollouts per task across 40 tasks, giving 2,000 rollouts per evaluation; we report the average across its four equally sized suites. A complete LIBERO-Plus evaluation contains 10,030 episodes across seven perturbation categories. Overall LP success is the total number of successes divided by the total number of evaluated episodes. RoboCasa GR1 Tabletop uses 50 trials per task across 24 tasks. RoboTwin 2.0 uses 100 trials per task across 50 tasks in each of the clean and randomized settings; its reported average weights these settings equally.

Instruction processing.

The official LIBERO-Plus task instructions include appended descriptions of the perturbation types. We remove these descriptions for evaluation, while retaining the rewritten instructions in the language perturbation category.

Evaluation seeds.

Results use one trained checkpoint per configuration. The RQ1 summaries and RQ3 composition results report means and sample standard deviations across three evaluation seeds.

C.3 Real-Robot Training and Evaluation

Robot setup.

We conduct all four real-robot tasks on the same Galbot G1 platform, equipped with two arms and grippers. The model takes images from one head-mounted camera and two wrist-mounted cameras as input. Actions consist of joint positions and gripper states.

Training configuration.

We use 200 demonstrations each for air-fryer food preparation and instructed pick-and-place, and 100 each for tool sweeping and cup pushing. Real-robot models are trained for 10 epochs with a batch size of 512.

Evaluation protocol.

We evaluate the VAE-only WAM and R2-WAM on the four real-robot tasks in Table 2, with 20 trials per task for each model. Complete-task success is reported for air-fryer food preparation, instructed pick-and-place, and cup pushing; tool sweeping is evaluated using the mean debris collection rate, as defined below.

Complete-task success.

For air-fryer food preparation, instructed pick-and-place, and cup pushing, we report the percentage of trials in which the full task is completed. Air-fryer food preparation requires opening the air fryer, placing the corn cob inside, and closing it; instructed pick-and-place requires selecting the instructed object and placing it at the specified location; cup pushing requires pushing the paper cup next to the tissue box.

Debris collection rate.

Each sweeping trial starts with six pieces of debris. Let c_{i}\in\{0,\ldots,6\} denote the number swept into the dustpan in trial i. We report the mean collection rate, \frac{100}{20}\sum_{i=1}^{20}\frac{c_{i}}{6}, as a percentage. This metric measures partial task completion rather than binary success, with each rollout treated as one evaluation trial.

Appendix D Feature Integration Results and Diagnostics for RQ2

D.1 Full Source–Mechanism Results

Table reports LIBERO averages and LP perturbation-level and overall success rates for the thirteen main-study representations across all five integration mechanisms.

All values are success rates (%). LIBERO reports the four-suite average; LP Total is the full-benchmark micro-average. Each mechanism is evaluated with the thirteen main-study sources.

D.2 Current-Feature Conditioning and Future-Feature Prediction

We compare three DINOv3 configurations under channel-wise concatenation (Figure 3(b)):

The current-only configuration removes the future-feature channels and their prediction loss; no future features are generated at inference. The future-only configuration omits the current-frame DINOv3 features while retaining future-feature supervision and denoising. The full configuration includes both current-feature conditioning and future-feature prediction. All three configurations use the same attention-mask rule, training duration, remaining loss weights, and evaluation settings.

Feature prediction visualization.

Following LDA-1B (Lyu et al., 2026), we visualize DINOv3 features by projecting them onto three principal components and mapping these components to RGB colors. A shared PCA projection and color scale are used for the reference and predictions. The resulting feature maps are overlaid on the ground-truth future RGB image to illustrate their spatial structure.

D.3 Local Action Sensitivity Analysis

We detail the gradient diagnostic in Figure 3(d). The analysis is inspired by gradient-based sensitivity and attribution methods (Simonyan et al., 2014; Hoffman et al., 2019; Yu et al., 2023); our statistic uses local Jacobians.

Differentiation target and sampling path.

Let F denote the raw current-frame DINOv3 features from the agent-view and wrist cameras. Each view has a 14\times 14 grid of 1,024-dimensional features. For mechanism m, we write the final normalized action chunk as A_{m}(F;\xi)\in\mathbb{R}^{32\times 7}, where \xi collects the fixed non-feature inputs and realized sampling noise. We differentiate through the full ten-step denoising process, including each mechanism’s transformations of the raw features, while holding the original VAE latents and other inputs fixed. The derivative therefore measures total local dependence on the current external-feature input across the sampling computation. It does not isolate dependence on a generated future-feature state.

We analyze channel-wise concatenation, token-wise concatenation, the feature expert, and gated cross-attention. REPA has no external-feature input at inference, so this diagnostic is not applicable to it.

Action groups and gradient estimator.

We flatten translation, rotation, and gripper outputs into separate vectors a_{g} with dimensions d_{g}=96, 96, and 32, respectively. Within each camera, each non-overlapping 2\times 2 block of raw feature locations forms one cell p on a 7\times 7 diagnostic grid. Let F_{c,p}\in\mathbb{R}^{4096} contain all feature channels in this block, and let

J_{g,c,p}=\frac{\partial a_{g}}{\partial F_{c,p}}

be its action–feature Jacobian. The local score is the Frobenius norm of this block Jacobian divided by \sqrt{d_{g}}, i.e., the root-mean-square input-gradient norm over the action outputs in group g.

To avoid constructing the full Jacobian, we use random output projections (Hoffman et al., 2019). For each group, draw independent Rademacher vectors z_{r}\in\mathbb{R}^{d_{g}} with entries \pm 1/\sqrt{d_{g}} and compute vector–Jacobian products by automatic differentiation:

\widehat{s}_{g,c,p}=\left[\frac{1}{R}\sum_{r=1}^{R}\left\|\frac{\partial(z_{r}^{\mathsf{T}}a_{g})}{\partial F_{c,p}}\right\|_{2}^{2}\right]^{1/2}.

Since \mathbb{E}[z_{r}z_{r}^{\mathsf{T}}]=I/d_{g}, the quantity inside the square root estimates \|J_{g,c,p}\|_{F}^{2}/d_{g}. We then average the local scores over the 49 cells and two camera views:

\widehat{S}_{m,g}(o)=\frac{1}{2\cdot 49}\sum_{c=1}^{2}\sum_{p=1}^{49}\widehat{s}_{m,g,c,p}(o),

where o identifies the observation state. Averaging occurs after the square root; this statistic is a mean of local sensitivity estimates, not the Frobenius norm of the entire camera Jacobian.

Observation states and plotted points.

All four mechanisms use the same nine fixed observation states: approach, grasp, and placement in each of three LIBERO trajectories. The recorded frame indices are (28,36,93) for spatial_e000, (34,42,136) for object_e000, and (24,32,84) for goal_e114, in approach–grasp–placement order. The spatial_e000_f028_approach state uses R=16 probes per action group; the other eight states use R=8, consistently across mechanisms.

Each point in Figure 3(d) is one \widehat{S}_{m,g}(o) value. Colors distinguish translation, rotation, and gripper outputs, giving nine points per color and 27 points per mechanism. The horizontal axis is logarithmic; vertical offsets separate overlapping points and have no quantitative meaning. Points show variation across observation states, not independent training replications or confidence intervals.

Interpretation.

For fixed other inputs and sampling noise, a small feature change satisfies \Delta a_{g}\approx J_{g}\Delta F to first order. Larger scores therefore indicate stronger local action sensitivity in the measured feature coordinates.

Appendix E Feature Composition Protocol and Results for RQ3

E.1 Feature Construction and Training Objective

For source set \mathcal{K}, channel-wise concatenation uses aligned external features in a fixed source order. With t=0 indexing the current observation and t=1,\ldots,T the future sequence, we write

\displaystyle f_{t} \\ \displaystyle=\operatorname{Concat}_{\mathrm{ch}}\{f_{t}^{k}:k\in\mathcal{K}\},\qquad t=0,1,\ldots,T, \\ \displaystyle\widetilde{z}_{0} \\ \displaystyle=[z_{0}^{\mathrm{vae}}\|_{\mathrm{ch}}f_{0}], \\ \displaystyle\widetilde{z}_{1:T,\tau} \\ \displaystyle=[z_{1:T,\tau}^{\mathrm{vae}}\|_{\mathrm{ch}}f_{1:T,\tau}].

Here f_{1:T,\tau} concatenates the source-wise future-feature states f_{1:T,\tau}^{k} at flow time \tau in the same order. The current state \widetilde{z}_{0} remains clean. At inference, future VAE latents and external features are initialized from noise and jointly denoised. Each source contributes a future-feature loss:

\mathcal{L}_{\mathrm{aux}}=\sum_{k\in\mathcal{K}}\mathcal{L}_{\mathrm{aux}}^{k},\qquad\mathcal{L}_{\mathrm{aux}}^{k}=\mathcal{L}_{\mathrm{FM}}(f_{1:T}^{k}).

The feature losses are evaluated on valid future targets f_{1:T}^{k}; current features f_{0}^{k} remain clean and are excluded from the loss. Source-specific processing and concatenation order are held fixed between training and evaluation.

E.2 Progressive Composition and Conditional Gains

DINOv3 is the initial external source. The seven candidate additions are V-JEPA 2.1, SigLIP 2, SAM 3, VGGT-\Omega, HRP, UniTok, and C-RADIOv4. Figure 4 first evaluates each candidate added to DINOv3. The next group uses DINOv3 plus UniTok as its parent, and the four-source group uses DINOv3 plus UniTok plus V-JEPA 2.1. Each bar adds one source not already in its parent set.

E.3 Per-Seed Results and Evaluation Settings

Table 10 reports the evaluations underlying the composition means and standard deviations. Each combination uses one fixed trained checkpoint, evaluated with seeds 41, 42, and 43. Every seed covers all 10,030 LP episodes. The mean is computed from the three full-benchmark success rates, and the standard deviation uses the sample convention.

These composition evaluations use ten inference steps, replanning every ten steps, 30 initial waiting steps, no action ensembling, text guidance scale 1.0 and binarized gripper outputs. The LP evaluation protocol follows Appendix C.2.

Appendix F Task-Level RoboCasa Results

Appendix G Task-Level RoboTwin Results