LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

Abstract

Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pre-training paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and optical-flow alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-robot success from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions.

Keywords: Latent action models, human-video pre-training, robot policy scaling

1 Introduction

Scaling data drives progress in vision and language, but embodied intelligence faces a sharper bottleneck: interaction data requires physical robots or simulators, making it far more expensive than text or static images. Large-scale human videos, especially egocentric ones, offer a promising way to relieve this bottleneck [1]. They naturally record diverse everyday interactions at a scale beyond robot-collected trajectories, and transforming such videos into action-like supervision could complement robot-only datasets such as Open X-Embodiment [2].

The central challenge is that human videos lack robot action labels, and human motion cannot be directly mapped to robot control spaces. Existing methods, such as VITRA [3], often convert human videos into pseudo-actions through explicit motion estimation, where estimation errors can lead to noisy action annotations; simply adding more human video does not necessarily improve policies. Latent Action Models (LAMs) offer an alternative: they infer implicit actions between consecutive frames end-to-end, without predefined robot action labels [4, 5, 6, 7]. Because these implicit actions are optimized to explain visual transitions, they provide compact supervision for unlabeled videos.

Despite this potential, existing LAMs are limited by their architectural priors and alignment signals. Many recent methods use Vector Quantized (VQ) latent spaces [8, 5, 6]. While compact, a fixed discrete codebook can over-compress the continuous changes that define egocentric manipulation, weakening the supervision needed for fine-grained policy learning. In addition, pixel-wise reconstruction signals such as MSE are often dominated by large static regions, providing weak signals for small but important hand-object interactions. As a result, learned latent actions may miss transition-relevant structure needed for downstream embodied policies.

To address these limitations, we introduce LAPT-Scale, a scalable latent action pre-training paradigm for embodied policy learning. LAPT-Scale contains two components: LAPT-Scale-LAM, which constructs human-derived supervision with a continuous Latent Action Model trained on large-scale human videos, and LAPT-Scale-Policy, which uses this supervision to train a VLA policy. The policy training follows three stages: human-video latent-action pre-training, embodiment alignment with target-robot data, and task-specific post-training in simulation and the real world. Unlike discrete LAMs, LAPT-Scale-LAM parameterizes implicit actions in a continuous space to preserve fine-grained transition dynamics. To make this space perceptually grounded and temporally consistent, we train the LAM with alignment signals beyond pixel reconstruction: perceptual alignment for local contact fidelity [9], semantic alignment with self-supervised features for hand-object structure [10], and optical-flow alignment for observed dynamics [11].

We train LAPT-Scale-LAM on 23.3K hours of diverse egocentric videos and use it to annotate a 5.1K-hour cleaned human-video subset for policy pre-training. With 250 hours of self-collected robot data for embodiment alignment, LAPT-Scale reaches 90.0%/89.3% success on RoboTwin Easy/Hard, 64.8% on RoboCasa-GR1. Scaling LAM-annotated human video from 1K to 5K hours further improves real-robot success from 63.3% to 85.0%, suggesting that continuous latent action annotation is a promising route for benefiting from larger human-video corpora. On four harder real-robot tasks, it retains 70.0–75.0% success under unseen position, illumination, and background shifts.

Our contributions are summarized as follows:

• LAPT-Scale-LAM: a continuous latent action model with perceptual, self-supervised feature, and optical-flow alignment signals for learning a more expressive latent action space from human videos.
• LAPT-Scale-Policy: a three-stage framework for human-video latent-action pre-training, target-robot embodiment alignment, and downstream post-training.
• Real-robot scaling: increasing only Stage-1 human-video pre-training data improves downstream real-robot success from 63.3% to 85.0%.

2 Related Work

Vision-language-action policy learning.

Vision-language-action (VLA) models unify visual perception, language conditioning, and action generation for generalist manipulation. Recent systems scale VLA policies with web knowledge, robot data, flow-based action modeling, and cross-embodiment training [12, 13, 14, 15, 16, 17, 18, 19, 20], while other work studies backbone, adaptation, spatial-representation, and action-space choices [21, 22, 23, 24]. However, VLA training remains constrained by action-labeled robot trajectories. Egocentric VLA methods such as VITRA and EgoScale reduce this bottleneck by converting human videos into action-like targets before robot alignment [3, 25], but such targets depend on explicit motion estimation or retargeting signals. Simulation-mediated pipelines also convert reconstructed human demonstrations into physically checked robot-centered trajectories [26]. LAPT-Scale instead annotates human videos with LAM-derived implicit actions.

Human video data for embodied learning.

Human video datasets provide broad coverage of physical interaction, from object manipulation in Something-Something and EPIC-KITCHENS to daily-life egocentric video in Ego4D [27, 28, 1]. Recent resources expand scale and annotation density with paired ego-exo views, factory videos, 3D/4D annotations, dexterous demonstrations, multimodal interaction labels, and filtered Ego4D text/action annotations [29, 30, 31, 32, 33, 34]. These datasets are useful visual experience, but their human observations and motion annotations do not directly specify robot-executable actions. LAPT-Scale learns latent actions to bridge this missing-label gap before robot embodiment alignment.

Latent action models from videos.

Latent Action Models infer action-like variables from observation transitions, making unlabeled videos useful for behavior learning. Related work learns compact transition variables through inverse-dynamics, world-model, or latent-action objectives and integrates them into VLA or world-model pipelines [4, 5, 35, 36, 37, 6, 7, 38, 39, 40]. However, many LAMs rely on discrete or vector-quantized latents, which can over-compress continuous egocentric manipulation dynamics [8, 5, 6]. Reconstruction-only training can be dominated by static background regions. LAOF augments reconstruction with optical-flow constraints to improve robustness to visual distractors [41]. LAPT-Scale-LAM therefore uses continuous latents together with perceptual, self-supervised semantic, and optical-flow alignment signals to preserve fine-grained implicit actions for human-video policy pre-training [9, 10, 11].

3 Method

3.1 Problem Formulation & Data

Pipeline.

As shown in Fig. 1, LAPT-Scale contains two modules. LAPT-Scale-LAM learns continuous latent actions from an egocentric human-video corpus \mathcal{D}_{h}=\{x_{1:T}^{(i)}\} and annotates a cleaned subset as \mathcal{D}_{z}^{h}=\{(\bm{c}_{t}^{h},\bar{\bm{z}}_{t}^{h})\}. LAPT-Scale-Policy trains a VLA policy in three stages: human-video latent-action pre-training on \mathcal{D}_{z}^{h}, target-robot embodiment alignment on \mathcal{D}_{r}^{\mathrm{align}}, and downstream post-training on \mathcal{D}_{r}^{\mathrm{task}}.

Human-video data and filtering.

Our LAM is trained on 23.3K hours of diverse egocentric videos, while policy pre-training uses a 5.1K-hour cleaned subset with more reliable manipulation content. Fig. 2 summarizes the corpus diversity. Following EgoVid-5M [34], we remove corrupted or blurry clips and discard segments dominated by camera motion or weak hand-object interaction. This lets the LAM learn from broad visual diversity while policy scaling uses cleaner action-like supervision.

3.2 LAPT-Scale-LAM: Constructing Human Latent-Action Supervision

LAPT-Scale-LAM explains visual transitions in human videos. Given adjacent frames (x_{t},x_{t+1}) from a clip x_{1:T}, the encoder E_{\phi}:(x_{t},x_{t+1})\mapsto\bm{z}_{t} produces a transition-level latent action \bm{z}_{t}\in\mathbb{R}^{d} that captures hand-object dynamics from x_{t} to x_{t+1}. Applying the same transition encoder over the clip yields \bm{z}_{1:T-1}. We implement the LAM as a spatio-temporal VAE [42]: for each transition, a learnable action query \bm{q}_{0} is prepended to the two-frame patch embeddings and processed by a Transformer encoder [43]. The query output parameterizes a diagonal Gaussian posterior q_{\phi}(\bm{z}_{t}\mid x_{t},x_{t+1}), from which the continuous latent action is sampled. A low-dimensional bottleneck (e.g., d=32), together with a decoder that separately receives current visual context, encourages \bm{z}_{t} to capture frame-to-frame change rather than static appearance.

To reconstruct the future frame, we inject the latent action into current-frame patch embeddings, \bm{d}_{t,n}=W_{p}\bm{p}_{t,n}+W_{z}\bm{z}_{t}, and use a spatial Transformer decoder D_{\theta} to predict \hat{x}_{t+1}. The LAM is aligned with visual fidelity, semantic structure, and observed motion signals, as illustrated in Fig. 3. The latent action encoder infers continuous transition variables, and the decoder is trained with reconstruction, perceptual, semantic, and optical-flow alignment terms. These signals are complementary: pixel and perceptual alignment keep the predicted frame visually faithful, semantic features preserve object-centric structure under appearance changes, and optical-flow alignment encourages the predicted transition to match the observed motion field. We therefore supervise the decoder at multiple levels rather than relying on reconstruction alone. Concretely, the training alignment contains the following terms.

• Perceptual alignment. Pixel reconstruction and LPIPS [9] preserve local structure: \mathcal{L}_{\mathrm{perc}}=\lambda_{\mathrm{mse}}\|\hat{x}_{t+1}-x_{t+1}\|_{2}^{2}+\lambda_{\mathrm{lpips}}\mathrm{LPIPS}(\hat{x}_{t+1},x_{t+1}). (1)
• Semantic alignment. Frozen DINOv2 [10] features preserve hand-object structure: \mathcal{L}_{\mathrm{dino}}=\frac{1}{2}\|S(P(\hat{x}_{t+1}))-S(P(x_{t+1}))\|_{2}^{2}+\frac{1}{2}\|c(\hat{x}_{t+1})-c(x_{t+1})\|_{2}^{2}, (2) where P and c denote patch and CLS features, and S(\cdot) denotes spatial pooling over patch features.
• Motion alignment. A frozen RAFT [11] estimator matches reconstructed and observed optical-flow fields: \mathcal{L}_{\mathrm{flow}}=\left\|R(x_{t},\hat{x}_{t+1})-\mathrm{sg}[R(x_{t},x_{t+1})]\right\|_{1}, (3) where \mathrm{sg}[\cdot] stops gradients through the target flow.

The full LAM training objective combines these alignment terms:

\mathcal{L}_{\mathrm{LAM}}=\mathcal{L}_{\mathrm{perc}}+\lambda_{\mathrm{dino}}\mathcal{L}_{\mathrm{dino}}+\lambda_{\mathrm{flow}}\mathcal{L}_{\mathrm{flow}}+\beta\mathcal{L}_{\mathrm{KL}},

where \mathcal{L}_{\mathrm{KL}} keeps the continuous latent space close to the prior. The frozen encoder then generates the latent-action labels used in Stage 1 of VLA training.

3.3 LAPT-Scale-Policy: Three-Stage VLA Policy Training

LAPT-Scale-Policy contains a VLM backbone F_{\omega} that encodes the policy context \bm{c} and uses the JoyAI-RA policy architecture [44], with a Perceiver-style action expert A_{\eta} trained by Conditional Flow Matching [45]. We keep the same action expert across three stages: Stage 1 predicts chunks of LAM-produced latent actions \bm{y}=\bar{\bm{z}}_{t:t+H-1}^{h} on \mathcal{D}_{z}^{h}, where each \bar{\bm{z}}_{\tau}^{h} is the posterior mean for one human-video transition and h indicates the human-video source; Stage 2 switches the target to robot action chunks \bm{y}=\bm{a}_{t:t+H-1}^{r} on \mathcal{D}_{r}^{\mathrm{align}} for embodiment alignment; Stage 3 continues on \mathcal{D}_{r}^{\mathrm{task}} for downstream post-training. No separate human-video head, adapter, or reinitialization is introduced.

For each target \bm{y} and context \bm{c}, we sample noise \bm{\xi}\sim\mathcal{N}(\bm{0},\mathbf{I}) and flow time \tau\in[0,1] to construct \bm{y}^{\tau}=\tau\bm{y}+(1-\tau)\bm{\xi}. The flow-matching objective is

\mathcal{L}_{\mathrm{FM}}(\bm{c},\bm{y})=\left\|A_{\eta}(\bm{c},\bm{y}^{\tau},\tau)-(\bm{y}-\bm{\xi})\right\|_{2}^{2}.

Targets are padded to the action expert’s maximum dimension and masked in the loss when needed. At inference time, the policy predicts executable robot actions; LAM-encoded actions are used only as scalable Stage-1 supervision.

4 Experiments

We evaluate LAPT-Scale along three axes: downstream VLA performance in simulation and the real world, scaling with LAM-annotated human-video data, and standalone LAM quality. The downstream benchmarks test executable robot behavior after embodiment alignment, while the LAM diagnostics test whether the annotation model preserves transition-relevant visual and semantic information.

4.1 Experimental Setup

LAPT-Scale-LAM is trained on a 23.3K-hour egocentric human-video corpus. The main policy run uses a 5.1K-hour cleaned subset annotated by the frozen LAM, and the scaling study uses nested 1K-, 3K-, and 5K-hour subsets with all robot data held fixed.

For downstream evaluation, we test on RoboTwin 2.0 [46], RoboCasa-GR1 Tabletop [47, 16], and seven real-robot tasks on an AgiBot G1 robot after 250 hours of target-robot embodiment alignment. We report task success rate for policy evaluation and use human-video reconstruction metrics plus LARY linear-probe accuracy for LAM diagnostics. Data quantities and evaluation details are in Appendix D.

4.2 Main Results

We compare the full LAPT-Scale pipeline against strong VLA baselines after Stage-1 pre-training on the 5.1K-hour LAM-annotated subset, target-robot embodiment alignment, and downstream post-training. Baseline sources and evaluation details are summarized in the appendix.

On RoboTwin 2.0, LAPT-Scale reaches 90.0% in the Easy setting and 89.3% in the Hard setting, slightly outperforming the strongest listed baselines. This indicates that the benefit is not limited to fixed initial configurations, since RoboTwin Hard varies object poses and scene layouts.

On RoboCasa-GR1, LAPT-Scale reaches 64.8% average success, showing that the gain also transfers to household tabletop manipulation.

The real-robot results test whether human-video latent supervision remains useful after embodiment alignment. LAPT-Scale achieves 85.0% average success over the three tasks, improving over \pi_{0.5} and JoyAI-RA. At the task level, it improves over JoyAI-RA on Mouse Pad and Cup while matching it on Headphones. Overall, Tables 1–3 suggest that Stage-1 latent-action pre-training can benefit both simulation and physical robot deployment. We view these results as evidence that LAM-based human-video pre-training remains compatible with strong downstream policy training; the scaling behavior is analyzed separately below.

4.3 Scaling with Annotated Human Video

We next study the central scaling question: whether increasing only the amount of LAM-annotated human video improves downstream robot performance. The LAM is always trained on the full 23.3K-hour corpus; we vary only the amount of cleaned, filtered, and annotated human video used for Stage-1 VLA pre-training, while keeping robot data, policy architecture, and evaluation protocol fixed.

We compare three forms of human-video supervision. Hand-motion annotation uses explicit hand-derived pseudo-actions, the DreamDojo LAM [37] uses its official pretrained reconstruction-based LAM, and LAPT-Scale-LAM uses the full perceptual, semantic, and motion-aware alignment signals. This comparison tests whether scale helps from more video alone or from higher-quality action-like supervision. The pretrained DreamDojo LAM differs from the MSE-only diagnostic trained on our data with its reconstruction-plus-KL recipe.

Fig. 4 shows that LAPT-Scale-LAM benefits most consistently from additional human video, improving from 63.3% at 1K hours to 80.0% at 3K hours and 85.0% at 5K hours. Larger LAM-annotated datasets also reach lower Stage-1 pre-training losses.

The baselines show why annotation quality matters. Explicit hand-motion pipelines can accumulate source-dependent hand-estimation noise under camera motion, occlusion, hand-visibility changes, and viewpoint shifts. A hand tracker can produce plausible but wrong pseudo-actions in these cases, and downstream policy training does not correct the labels. Such noise may explain why hand motion improves from 1K to 3K hours but degrades to 55.0% at 5K hours.

DreamDojo LAM scales more reliably than hand motion and reaches 76.7% at 5K hours, but remains below LAPT-Scale-LAM. This suggests that pixel-level future-frame reconstruction alone may not fully capture the semantic and motion structure needed for moving hand-object interactions, motivating the controlled alignment ablation below. The full per-task breakdown and shared 0-hour control (55.0%) are in Appendix E.

4.4 Controlled Stage-1 and Alignment Ablations

We remove Stage 1 with initialization, robot data, Stage-2/3 schedules, and evaluation fixed, and extend Stage-2 training in a compute-matched control (Table 4; Appendix E.1). Both remain below the full model.

We retrain the annotation-to-policy pipeline after removing each LAM alignment term (Table 5).

Removing flow yields the largest RoboTwin Hard drop (11.5 points), despite high PSNR, illustrating the limits of pixel fidelity as a diagnostic.

4.5 Harder Real-Robot Tasks and Generalization

Four harder tasks cover paper-ball disposal, pen and self-occluded eraser grasping, and multi-stage book stowing. One joint policy uses 200 demonstrations per task, also used to post-train \pi_{0.5}. We test seen conditions and unseen position, illumination, and background shifts with 20 trials per task per domain (protocols and domain images in Appendix H).

LAPT-Scale and \pi_{0.5} match at 85.0% in seen conditions, but reach 70.0–75.0% and 40.0–50.0%, respectively, under the tested shifts (Table 6).

4.6 Human-Video LAM Evaluation

We evaluate the LAM before policy training with two diagnostics: future-frame reconstruction on human-video transitions and first-person action recognition on LARY [50] using frozen latent actions with a linear probe.

We compare three LAM variants under the same transition-prediction protocol. VQ-LAM uses a discrete codebook [8]; MSE-only LAM keeps the continuous latent space but removes the LPIPS, DINO, and optical-flow alignment terms; LAPT-Scale-LAM uses the full continuous latent space and action-aware alignment signals. This tests latent-space and alignment choices.

Table 7 shows that LAPT-Scale-LAM improves both transition reconstruction and latent-action semantics. The gap to VQ-LAM suggests that discrete codes are too coarse for fine-grained egocentric hand-object dynamics, while the gain over MSE-only LAM shows the value of perceptual, semantic, and motion alignment.

The qualitative example in Fig. 5 illustrates the same trend: VQ-LAM produces blurry reconstructions with noticeable hand-position errors, and MSE-only LAM remains visually blurred. LAPT-Scale-LAM reconstructs the transition most similarly to the ground truth, especially around the moving hand and object.

These diagnostics complement downstream evaluation by showing that Stage-1 annotations retain transition information before robot embodiment alignment.

4.7 Interpreting the Learned Latent Action Space

To probe what the learned latent action token represents, we perform counterfactual latent-action replacement on human ego-view videos, which match the training source of LAPT-Scale-LAM. For each donor-target pair, we keep the target visual context fixed but replace its latent action with one inferred from a donor video. We then decode three short-horizon predictions: the donor self-rollout, the target base rollout, and the target rollout under donor-latent replacement. This isolates the latent action from scene context: if the token carries transition information, the replacement should alter the target rollout without simply copying donor appearance.

Across the examples in Figure 6, replacement changes how the target sequence evolves without globally rewriting the target scene. The clearest effects appear as camera-relative hand-object motion changes: the donor latent action redirects the short-horizon movement tendency, while the scene layout remains largely target-specific. These results suggest that LAM tokens encode transferable information about how the scene changes, rather than only what the scene contains.

5 Conclusion and Limitations

We presented LAPT-Scale, a latent action pre-training paradigm that turns large-scale egocentric human videos into supervision for embodied policy learning. By learning continuous transition variables directly from video and using them for Stage-1 VLA pre-training, LAPT-Scale provides a scalable way to exploit human videos without explicit robot action labels. Across simulation, real-robot, scaling, and LAM diagnostic experiments, our results suggest that LAM-based human-video supervision improves downstream manipulation and becomes more useful as the amount of annotated human video increases, supporting latent actions as scalable human-video annotations. Four harder real-robot tasks also show robustness under the tested position and appearance shifts.

The main limiting assumption is that visual latent actions learned from human videos can become useful robot-policy supervision after embodiment alignment; they are not guaranteed to align with independently controllable robot degrees of freedom. As a result, target-robot embodiment alignment remains necessary. The approach may also fail when videos contain weak manipulation signal, large embodiment gaps, heavy occlusion, or visual changes dominated by ego-camera motion. Our experiments are limited in robot and task coverage: the real-robot study uses seven tasks and one target platform, with 20 trials per condition. This limits the precision of success estimates, and broader validation is needed before claiming deployment robustness. Future work should study stronger embodiment alignment, longer-horizon latent-action composition, broader robot coverage, and more systematic scaling laws across human-video data, model size, and robot demonstrations.

References

1. [1] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
2. [2] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation, pages 6892–6903. IEEE, 2024.
3. [3] Q. Li, Y. Deng, Y. Liang, L. Luo, L. Zhou, C. Yao, L. Zeng, Z. Feng, H. Liang, S. Xu, Y. Zhang, X. Chen, H. Chen, L. Sun, D. Chen, J. Yang, and B. Guo. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571, 2025.
4. [4] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning, 2024.
5. [5] S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y.-W. Chao, B. Y. Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024.
6. [6] Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. UniVLA: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025.
7. [7] H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, et al. Motus: A unified latent action world model. arXiv preprint arXiv:2512.13030, 2025.
8. [8] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
9. [9] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. URL https://arxiv.org/abs/1801.03924.
10. [10] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
11. [11] Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, 2020.
12. [12] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023.
13. [13] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406.09246.
14. [14] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. \pi_{0}: A vision-language-action flow model for general robot control, 2024. URL https://arxiv.org/abs/2410.24164.
15. [15] Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky. \pi_{0.5}: a vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054.
16. [16] NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. GR00T N1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.
17. [17] Gemini Robotics Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. Gemini Robotics: Bringing AI into the physical world. arXiv preprint arXiv:2503.20020, 2025.
18. [18] W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, F. Wang, Q. Zhu, H. Sun, Y. Wang, S. Ma, et al. A pragmatic VLA foundation model. arXiv preprint arXiv:2601.18692, 2026.
19. [19] Y. Yang, S. Zeng, T. Lin, X. Chang, D. Qi, J. Xiao, H. Liu, R. Chen, Y. Chen, D. Huo, et al. ABot-M0: VLA foundation model for robotic manipulation with action manifold learning. arXiv preprint arXiv:2602.11236, 2026.
20. [20] JoyAI-RA Team. JoyAI-RA 0.5: Scaling robot manipulation learning via dual action alignment, 2026. URL https://arxiv.org/abs/2608.05674.
21. [21] X. Li, P. Li, L. Qian, M. Liu, D. Wang, J. Liu, B. Kang, X. Ma, X. Wang, D. Guo, T. Kong, H. Zhang, and H. Liu. What matters in building vision-language-action models for generalist robots. arXiv preprint arXiv:2412.14058, 2024.
22. [22] Y. Wang, P. Ding, L. Li, C. Cui, Z. Ge, X. Tong, W. Song, H. Zhao, W. Zhao, P. Hou, et al. VLA-Adapter: An effective paradigm for tiny-scale vision-language-action model. arXiv preprint arXiv:2509.09372, 2025.
23. [23] D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, and X. Li. SpatialVLA: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025.
24. [24] J. Zheng, J. Li, D. Liu, Y. Zheng, Z. Wang, Z. Ou, Y. Liu, J. Liu, Y.-Q. Zhang, and X. Zhan. Universal actions for enhanced embodied foundation models. arXiv preprint arXiv:2501.10105, 2025.
25. [25] R. Zheng, D. Niu, Y. Xie, J. Wang, M. Xu, Y. Jiang, F. Castañeda, F. Hu, Y. L. Tan, L. Fu, T. Darrell, F. Huang, Y. Zhu, D. Xu, and L. Fan. EgoScale: Scaling dexterous manipulation with diverse egocentric human data, 2026. URL https://arxiv.org/abs/2602.16710.
26. [26] P. Liu, Y. Liu, S. Guo, F. Ma, Z. Yuan, A. Li, Z. Chen, W. Li, T. Zhang, M. Li, J. Zhang, J. Xiong, Z. Xiang, D. Chi, Y. Zhuang, L. Luo, W. Tan, D. Li, N. Jiang, Y. Li, Q. He, J. Liang, C. Cai, M. Luo, H. Zhang, P. Hao, S. Wang, N. Qiao, Y. Gao, L. Kang, J. Xiong, R. Li, J. Li, H. Shen, Y. Gong, N. Duan, and L. Lin. JoyAI-Sim: A simulation-enabled interconversion toolchain for the embodied data pyramid, 2026. URL https://arxiv.org/abs/2606.16776.
27. [27] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The something something video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
28. [28] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The EPIC-KITCHENS dataset. In European Conference on Computer Vision, pages 720–736, 2018.
29. [29] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, et al. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. International Journal of Computer Vision, 133(12):8356–8435, 2025. doi:10.1007/s11263-025-02557-6.
30. [30] Build AI. Egocentric-10K, 2025. URL https://huggingface.co/datasets/builddotai/Egocentric-10K.
31. [31] Ropedia. Xperience-10M: A large-scale egocentric multimodal dataset with structured 3D/4D annotations, 2026. URL https://huggingface.co/datasets/ropedia-ai/xperience-10m. Dataset.
32. [32] R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11709, 2025.
33. [33] R. Punamiya, S. Kareer, Z. Liu, J. Citron, R.-Z. Qiu, X. Cai, A. Gavryushin, J. Chen, D. Liconti, L. Y. Zhu, P. Aphiwetsa, B. Li, A. Cheluva, P. Kuppili, Y. Liu, D. Patel, A. Gao, H.-Y. Chung, R. Co, R. Zbizika, J. Liu, X. Xu, H. Xiong, G. Chen, S. Oliani, W. Xuan, C. Yang, X. Wang, J. Fort, R. Newcombe, J. Gao, J. Chong, G. Matsuda, A. Doriwala, M. Pollefeys, R. Katzschmann, X. Wang, S. Song, J. Hoffman, and D. Xu. EgoVerse: An egocentric human dataset for robot learning from around the world, 2026. URL https://arxiv.org/abs/2604.07607.
34. [34] X. Wang, K. Zhao, F. Liu, J. Wang, G. Zhao, X. Bao, Z. Zhu, Y. Zhang, and X. Wang. EgoVid-5M: A large-scale video-action dataset for egocentric video generation, 2024. URL https://arxiv.org/abs/2411.08380.
35. [35] X. Chen, J. Guo, T. He, C. Zhang, P. Zhang, D. C. Yang, L. Zhao, and J. Bian. IGOR: Image-goal representations are the atomic control units for foundation models in embodied AI. arXiv preprint arXiv:2411.00785, 2024.
36. [36] S. Gao, S. Zhou, Y. Du, J. Zhang, and C. Gan. AdaWorld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025.
37. [37] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W.-C. Tseng, Y. Dong, K. Mo, C.-H. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. R. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M.-Y. Liu, Y. Zhu, J. Jang, and L. Fan. DreamDojo: A generalist robot world model from large-scale human videos, 2026. URL https://arxiv.org/abs/2602.06949.
38. [38] X. Chen, H. Wei, P. Zhang, C. Zhang, K. Wang, Y. Guo, R. Yang, Y. Wang, X. Xiao, L. Zhao, et al. Villa-X: Enhancing latent action modeling in vision-language-action models. arXiv preprint arXiv:2507.23682, 2025.
39. [39] C. Zhang, J. Wang, Z. Gao, Y. Su, T. Dai, C. Zhou, J. Lu, and Y. Tang. CLAP: Contrastive latent action pretraining for learning vision-language-action models from human videos. arXiv preprint arXiv:2601.04061, 2026.
40. [40] P. Liu, Z. Xiang, M. Li, W. Li, J. Zhang, J. Sun, and J. Li. WLA3: World latent action modeling for semantics, dynamics, and kinematics, 2026. URL https://arxiv.org/abs/2609.15870.
41. [41] X. Bu, J. Lyu, F. Sun, R. Yang, Z. Ma, and W. Li. LAOF: Robust latent action learning with optical flow constraints. arXiv preprint arXiv:2511.16407, 2025.
42. [42] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
43. [43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
44. [44] T. Zhang, Z. Yuan, D. Chi, P. Liu, D. Li, K. Hu, L. Zhang, J. Nie, Z. Wei, Z. Chen, Y. Tang, J. Li, Z. Xiang, M. Li, T. Luo, H. Wan, A. Li, L. Zhai, Z. Zhan, X. Bai, J. Cai, P. Cao, K. Chen, S. Chen, Y. Dai, S. Di, Y. Gong, C. Gui, Y. Guo, P. Hao, Q. He, H. Huang, K. Huang, Z. Huang, S. Jin, Y. Jin, A. Li, D. Li, J. Li, R. Li, Y. Li, Y. Li, J. Liang, F. Liu, J. Long, M. Luo, X. Pan, H. Shen, X. Tian, D. Wang, S. Wang, J. Xiong, H. Xu, W. Xu, Z. Yu, H. Zhang, J. Zhang, L. Zhao, C. Zhou, N. Duan, Y. Zhuang, and L. Lin. JoyAI-RA 0.1: A foundation model for robotic autonomy, 2026. URL https://arxiv.org/abs/2604.20100.
45. [45] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
46. [46] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025.
47. [47] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems, 2024.
48. [48] StarVLA Community. StarVLA: A lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014, 2026.
49. [49] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. C. Burchfiel, and S. Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, 2023. doi:10.15607/RSS.2023.XIX.026.
50. [50] D. Nie, F. Chen, Q. Lv, J. Kuang, X. Li, X. Cao, and X. Cai. LARY: A latent action representation yielding benchmark for generalizable vision-to-action alignment. arXiv preprint arXiv:2604.11689, 2026.

Appendix A Human-Video Data Construction

A.1 Source Composition

LAPT-Scale-LAM is trained on a 23.3K-hour egocentric human-video corpus collected from six egocentric human-video sources: Egocentric-10K [30], EgoDex [32], Ego4D [1] after EgoVid-style filtering [34], Xperience [31], EgoVerse [33], and self-collected human videos. The corpus is built by merging complementary sources rather than relying on a single dataset. Broad daily-activity datasets provide scene and object diversity, manipulation-focused datasets increase the density of hand-object interaction, and self-collected videos improve coverage of target-relevant activities and camera viewpoints.

A.2 Filtering, Mixing, and Annotation

We first standardize all videos to a common decoding format, normalize frame rates and resolutions for training, remove clips with severe corruption or missing frames, and segment long videos into short temporal clips. We also remove near-duplicate clips after segmentation to avoid over-representing repeated scenes or repeated camera trajectories.

For policy pre-training, we apply an additional manipulation-focused filtering stage. A clip is retained when it contains visible hand-object interaction and temporal visual change that can plausibly correspond to an action. We discard segments dominated by strong blur, extreme ego-camera motion, missing hands, long static intervals, severe occlusion, or weak manipulation content. This produces a 5.1K-hour cleaned subset used for the main Stage-1 policy pre-training run.

The source mixture is constructed in two stages. First, all candidate videos are standardized and pooled into the 23.3K-hour corpus used to train LAPT-Scale-LAM. This stage favors coverage: even clips with moderate manipulation density can help the LAM learn generic egocentric visual dynamics, camera-relative motion, and object appearance variation. Second, we apply the stricter manipulation filter described above to build the 5.1K-hour Stage-1 policy pre-training subset. This stage favors supervision quality: clips are kept when they contain visible hands, object interaction, and meaningful temporal change.

The 1K-, 3K-, and 5K-hour scaling subsets are sampled from the cleaned pool and are nested when possible: the 3K-hour subset contains the 1K-hour subset plus additional clips, and the 5K-hour subset contains the 3K-hour subset plus additional clips whenever the source manifests allow it. We also cap or rebalance very large sources during sampling so that scaling primarily increases the amount of action-like supervision rather than simply increasing the dominance of one source. In practice, this preserves a mixture of household manipulation, dexterous-hand interaction, and general egocentric daily activity across all scales.

For fair comparison with the hand-motion annotation baseline, the cleaned Stage-1 pool only includes sources where the same filtering and hand-pose-based pseudo-action pipeline can be applied. Egocentric-10K remains useful for broadening LAM training coverage, but it does not provide the hand-pose annotations required by the hand-motion baseline. We therefore exclude it from the cleaned Stage-1 scaling subsets, which keeps the comparison focused on the form of human-video supervision rather than source-specific annotation availability.

We also audit the semantic diversity of the available human-video metadata. The fine-grained taxonomy identifies 12 scene categories, 18 task categories, 77 scene-task cells, 21 action labels, 54 object labels, and 272 action-object pairs. We use these statistics as a metadata diversity audit rather than as exact training-hour composition, because weakly labeled or unlabeled sources remain part of the large-scale video pool.

After filtering, the frozen LAPT-Scale-LAM encoder annotates each retained clip with transition-level latent actions. For a clip x_{1:T}, the encoder is applied to adjacent frames to produce posterior means \bar{\bm{z}}_{1:T-1}, which are used as deterministic Stage-1 policy targets. The latent annotations are stored together with the corresponding visual context, language or clip-level text when available, source metadata, and masks for valid temporal positions. Invalid transitions caused by decoding failure or clip boundaries are excluded from the Stage-1 loss. This annotation format lets the same policy training code consume human-video latent-action chunks in Stage 1 and executable robot action chunks in Stages 2–3.

Appendix B Training and Implementation Details

B.1 LAM and Policy Training

The controlled LAM diagnostic variants are trained on the same human-video corpus with the same clip sampling and optimization protocol. LAPT-Scale-LAM encodes each video transition into a continuous latent action and reconstructs the next frame from the current visual context and latent action. The full model uses the reconstruction, LPIPS, DINO feature, optical-flow, and KL terms described in the main paper. The MSE-only diagnostic uses DreamDojo’s reconstruction-plus-KL training recipe on our data, omitting the LPIPS, DINO, and optical-flow alignment terms. In contrast, the DreamDojo baseline used for Stage-1 scaling uses its official pretrained LAM. It is not the same checkpoint as the controlled MSE-only diagnostic. The VQ-LAM baseline replaces the continuous latent action with a discrete codebook while keeping the transition prediction protocol fixed.

The LAM uses a patch size of 16, model dimension 1024, 24 Transformer encoder blocks, 24 Transformer decoder blocks, and 16 attention heads. The latent action is parameterized as a 32-dimensional continuous Gaussian posterior. The training objective uses MSE reconstruction as the base term, KL weight 1\times 10^{-6}, LPIPS weight 0.01, DINO feature weight 0.003, and optical-flow weight 3.3\times 10^{-5}. DINO alignment uses a frozen DINOv2 ViT-B/14 backbone, and flow alignment uses RAFT-large from the torchvision optical-flow models.

The LAM training configuration also uses paired augmentation. For each paired sample, the configuration uses four repeats and resamples the latent action under augmentation. The color augmentation ranges are 0.3 for brightness, contrast, and saturation, 0.08 for hue, gamma in [0.7,1.4], Gaussian noise standard deviation 0.035, blur probability 0.2, and minimum crop scale 0.9.

LAPT-Scale-Policy uses the same VLM backbone and action expert across all experiments. Stage 1 trains the policy on LAM-annotated human-video clips, where the target is the continuous latent action produced by the frozen LAM encoder. Stage 2 continues training the same action expert on 250 hours of target-robot embodiment-alignment data, replacing the Stage-1 latent target with executable robot action chunks without introducing a separate action head or adapter. The action expert supports up to 32 target dimensions; lower-dimensional targets are padded and excluded from the loss with a mask. Stage 3 further adapts the same policy on task-specific robot data. Across scaling experiments, the embodiment-alignment data, downstream task data, policy architecture, and training schedule are fixed; only the amount and source of human-video pre-training supervision are changed.

B.2 Compute and Hyperparameters

We separate the computational cost into three parts: LAM training, offline human-video annotation, and policy training. LAM training is a one-time cost on the 23.3K-hour human-video corpus. After training, the frozen LAM encoder is used offline to annotate the cleaned 5.1K-hour subset and the nested scaling subsets. Policy training then consumes the cached latent-action annotations in Stage 1, so downstream scaling experiments do not require rerunning the LAM encoder except when changing the annotated human-video pool. Table 9 summarizes the implementation, training, and compute configuration used in our experiments.

Appendix C Baseline Implementation Details

Table 12 summarizes the main implementation differences among the human-video supervision baselines. For fair comparison, all variants use the same cleaned human-video subsets, the same Stage-1 policy architecture, the same 250-hour target-robot embodiment-alignment data, and the same downstream post-training protocol. The comparison therefore isolates the form and quality of the supervision extracted from human videos rather than changes in robot data or policy capacity.

For the explicit annotation baseline in Table 15, we convert egocentric human videos into action-like supervision through reconstructed hand motion. Specifically, this baseline treats the human hand as a dexterous end-effector and uses hand-pose trajectories as pseudo-action targets for human-video pre-training. It uses the same clip sampling, filtering, policy architecture, robot data, and downstream training schedule as LAPT-Scale-LAM. This comparison is intended to isolate the effect of replacing explicit hand-motion supervision with LAM-derived implicit actions.

The hand-motion baseline depends on explicit hand reconstruction, which can produce noisy pseudo-actions under occlusion, motion blur, unusual viewpoints, or contact-rich manipulation. The LAM baselines avoid explicit hand-pose estimation by learning transition variables directly from video. DreamDojo LAM and MSE-only LAM primarily test reconstruction-driven latent actions, VQ-LAM tests whether a discrete bottleneck is sufficient for egocentric manipulation, and LAPT-Scale-LAM tests whether continuous latents with perceptual, semantic, and motion-aware alignment produce better action-like supervision.

Appendix D Evaluation Protocol Details

For all downstream robot evaluations, we report task success rate. A rollout is counted as successful only when the task-specific goal condition is satisfied within the evaluation horizon; otherwise it is counted as a failure. For simulation benchmarks, we follow the benchmark-provided success detectors and episode termination rules. For real-robot tasks, success is judged by the final task state after execution and uses the same criteria for all compared methods.

D.1 Robot-Data Quantities and Evaluation Accounting

The 250-hour Stage-2 corpus is heterogeneous embodiment-alignment data, comprising 162,767 episodes and 26.7M frames rather than per-task data. For Stage 3, we use 50 demonstrations per original real-robot task, 200 per added hard task, 50 clean plus 500 randomized demonstrations per RoboTwin task, and 1,000 demonstrations per RoboCasa task. The original tasks use 20 evaluation trials per task. The four harder tasks use 20 trials per task per domain, or 320 trials per method across the seen domain and three shifted domains.

D.2 Simulation Benchmarks

RoboTwin 2.0 evaluation follows the benchmark Easy and Hard settings. For RoboTwin post-training, each task uses 50 clean demonstrations and 500 heavily randomized demonstrations. The Easy setting evaluates policies under the standard benchmark initialization distribution, while the Hard setting introduces stronger variation in object poses and scene layouts. We report the average success rate across the benchmark task suite for each setting; the main LAPT-Scale policy reaches 90.0% in Easy and 89.3% in Hard. All compared methods are evaluated with the same task split, observation interface, action interface, and success definition.

RoboCasa-GR1 evaluation uses the tabletop manipulation setting with 24 tasks and 50 rollouts per task. We follow the benchmark-provided simulator reset distribution and task success checks, and report the average success rate across all RoboCasa rollouts. The reported RoboCasa-GR1 number averages over the evaluated task instances rather than selecting the best task-specific checkpoint; the main LAPT-Scale policy reaches 64.8% average success.

D.3 Real-Robot Tasks

Real-robot evaluation is conducted on an AgiBot G1 robot after target-robot embodiment alignment. We evaluate three tasks: Mouse Pad, Headphones, and Cup. Each task is evaluated over 20 trials per method. Between trials, the scene is reset to the task-specific initial distribution, and all methods are evaluated under the same reset and success criteria. A Mouse Pad trial succeeds if the robot completes the requested manipulation of the mouse pad; a Headphones trial succeeds if the headphones are manipulated to the target state; and a Cup trial succeeds if the cup task reaches the specified final configuration.

Appendix E Additional Real-Robot Scaling Results

E.1 Stage-1 Removal Control

The robot-only control removes Stage 1 while retaining the same initialization, 250-hour embodiment-alignment corpus, Stage-2/3 data, and evaluation protocol. The compute-matched control extends Stage-2 training to match the training compute budget. Unlike the Diffusion Policy comparison, Stage-1 removal tests human-video supervision within the same policy architecture.

Appendix F Additional LAM Diagnostics

F.1 Reconstruction Protocols

In addition to the single-step reconstruction results reported in the main paper, we evaluate each LAM variant under two rollout protocols. The first is a teacher-forced next-frame reconstruction protocol: for every time step, the model receives the ground-truth current frame x_{t} and predicts \hat{x}_{t+1}. This measures whether the latent action extracted from a local transition preserves enough information to reconstruct the next visual state without accumulating temporal errors. The second is an autoregressive rollout protocol: the model receives the ground-truth first frame x_{1}, predicts \hat{x}_{2}, and then conditions on its own prediction to generate subsequent frames. This evaluates temporal stability and whether reconstruction errors compound over multiple latent-action steps.

For this quantitative comparison, we include LAPA [5] and VILLA-X [38] as additional latent-action model baselines. LAPA represents a discrete latent-action annotation approach, while VILLA-X represents a video latent-action baseline trained for egocentric transition prediction. Together with MSE-only LAM, these baselines test whether the gains come from the LAPT-Scale alignment design rather than only from using a latent transition model.

The metrics in Table 17 are computed over 3500 frames sampled from a randomly held-out 100-hour evaluation split of our mixed human-video dataset. Qualitative examples for both protocols are shown in Figs. 7 and 8.

These two protocols expose the differences progressively. In the teacher-forced setting, LAPA and VILLA-X already show weak local transition fidelity: hand-object boundaries are less stable and the predicted motion is often visually inconsistent with the ground-truth next frame. MSE-only LAM is substantially stronger under teacher forcing because every prediction is conditioned on a clean ground-truth frame, but it still tends to produce smoother hands and object contacts. LAPT-Scale-LAM gives the sharpest teacher-forced reconstructions, while DreamDojo-style reconstruction remains slightly blurrier around the moving hand and contacted objects.

Fig. 7 illustrates the easier teacher-forced case. Because every prediction receives a clean previous frame, the comparison mainly reflects how well each latent action preserves the local transition. The strongest differences appear around the hand and the object being manipulated, where reconstruction-only objectives tend to smooth contact boundaries.

The autoregressive setting makes this gap much more visible. Once each model must condition on its own previous prediction, small contact and hand-position errors compound over time. LAPA and VILLA-X drift quickly, while MSE-only LAM gradually loses fine hand structure; by the final generated frames, the hand is often no longer clearly recognizable. LAPT-Scale-LAM remains more stable under the same rollout protocol, preserving a visible hand region and coherent short-horizon object motion for longer. This supports the view that the additional perceptual, semantic, and motion alignment terms improve temporal latent-action structure rather than only single-step reconstruction quality.

Fig. 8 evaluates the same models under a stricter rollout protocol. Here, the model must reuse its own generated frames, so small hand-position errors and blurred contacts become part of the next input. This is why the gap between MSE-only LAM and LAPT-Scale-LAM is more visible in later frames than in the teacher-forced comparison.

F.2 Similar Motions Across Different Scenes

To test whether the learned latent action is tied to visual appearance or to transition-level motion, we compare clips that contain similar short-horizon actions in different scenes. For each query transition, we retrieve nearest neighbors in the LAPT-Scale-LAM latent-action space while excluding clips from the same source video. This produces groups where the background, object identity, lighting, and hand appearance can differ substantially, but the short-horizon motion pattern is similar.

For the cluster visualization, we sample 50K motion-filtered windows from 2.36M candidate windows across 11,981 episodes. Each window contains 16 frames with stride 16. We retain windows above the 70th percentile of the motion score (threshold 4.805), reduce latent-action features to 64 PCA dimensions (explained-variance sum 0.7756), and cluster the resulting features into 50 groups. The selected clusters in Fig. 9 illustrate repeated hand-object motion patterns across visually different clips.

This analysis complements the latent-action replacement experiment reported in the main paper. Nearest-neighbor retrieval asks whether similar motions naturally cluster together across scenes, while replacement asks whether a latent action inferred from one scene can change the rollout in another scene without copying the donor appearance. Together, these diagnostics support the interpretation that the LAM token encodes how the scene changes rather than only what the scene looks like.

The selected clusters in Fig. 9 are not meant to show near-duplicate frames. Instead, they show that clips with different backgrounds, objects, and hand appearances can still be close in latent-action space when their short-horizon motion is similar. This is the desired behavior for Stage-1 supervision: the policy should learn reusable transition patterns rather than memorize source-specific visual appearance.

Appendix G Stage-1 Latent Rollout Diagnostics

To further diagnose Stage-1 human-video pre-training, we evaluate whether the learned latent-action space is meaningful enough for decoder rollout and whether the VLA policy can predict action-relevant latent variables from context. We compare two rollout settings: using the ground-truth latent action encoded by LAPT-Scale-LAM and using the latent action sampled from the Stage-1 VLA policy. The ground-truth-latent rollout tests whether the compressed latent space preserves enough transition information for decoding, while the VLA rollout tests whether Stage-1 pre-training learns to predict latent actions that still produce coherent human motion.

Table 18 shows that ground-truth latent actions can reconstruct future visual transitions with strong fidelity, indicating that the compressed LAM latent space retains meaningful motion information. More importantly, the VLA-predicted latent rollout also achieves reasonable reconstruction quality, showing that Stage-1 pre-training learns action-related latent structure rather than only static appearance cues.

The two examples below visualize this diagnostic at different manipulation scales. The dice example emphasizes short, fast hand-object motion, while the sandwich example contains longer contact sequences and larger appearance changes. In both cases, the Stage-1 policy-predicted latents produce rollouts that preserve the dominant motion pattern even though they are not used as executable robot actions.

The sandwich sequence is visually harder because multiple ingredients and hand contacts change across the clip. The rollout remains useful as a diagnostic because it tests whether predicted latent actions encode the transition tendency rather than merely reconstructing a static scene.

Appendix H Additional Robustness Evaluation

H.1 Harder Real-Robot Tasks

To test whether the gains extend beyond the three original tabletop tasks, we add paper-ball disposal, slender-pen grasping, small eraser grasping under self-occlusion, and multi-stage book stowing. The first task requires disposal into a target receptacle; pen and eraser require precision grasping despite small contact regions and partial occlusion; book stowing requires a controlled edge push, grasp, vertical reorientation, and shelving. We jointly train one policy with 200 demonstrations per task. The same downstream demonstrations are used to post-train \pi_{0.5} from its official checkpoint.

H.2 Generalization to Unseen Conditions

We evaluate the four harder tasks under the training distribution and three unseen shifts: object position, illumination, and background/tablecloth appearance. Each entry averages 20 trials per task and domain. LAPT-Scale and \pi_{0.5} are tied in-distribution, whereas LAPT-Scale degrades less under all three shifts.