Skeleton-Grounded Latent Action Model for Robot World Models

Abstract

Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.

1 Introduction

Robot world models simulate how environments evolve in response to robot actions, supporting policy evaluation and model-based planning (Yang et al., 2024; Wu et al., 2024). Accurate simulation requires modeling robot motion and resulting scene changes. Latent action models encode visual transitions as proxy actions (Bruce et al., 2024; Ye et al., 2025), allowing world models to learn dynamics from large-scale videos without explicit action annotations (Gao et al., 2026). These latent actions are learned through future-observation reconstruction, with the current observation providing visual context (Gao et al., 2025; Liang et al., 2026). For action-conditioned simulation, inferred latent actions serve as supervision for mapping robot controls into the learned latent action space.

Learning this mapping is challenging since reconstruction objectives encourage latent actions to encode robot motion together with object responses and other scene changes (Nikulin et al., 2025; Zhang et al., 2025). For example, similar end-effector motions may produce different object responses under different contact states (Gu et al., 2026; Xu et al., 2026), introducing scene-dependent variation beyond robot motion into the adaptation target. These interaction outcomes are useful for world modeling, since accurate simulation requires predicting how robot motion affects the surrounding scene. This motivates a dedicated kinematic component for action adaptation, together with a residual component retaining complementary transition information for dynamics learning.

To this end, we propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. Robot skeletons describe robot configuration and motion while abstracting scene appearance. Related robot world models use skeleton sequences or kinematic renderings as direct conditioning signals for generation (Wang et al., 2025b; Wu and Gao, 2026; Xu et al., 2026). SkeLAM uses paired RGB and skeleton transitions to learn kinematic and residual components from RGB observations. Skeleton supervision grounds the kinematic latent in robot motion, while the residual latent supplies complementary information for RGB reconstruction. Together, the two components jointly support reconstruction, providing a dedicated target for action adaptation while retaining information for dynamics learning. During representation learning, skeletons supervise the RGB latents, while the deployed interface uses RGB observations and robot actions.

We further introduce cross-modal swap reconstruction, which replaces the RGB kinematic latent with its paired skeleton counterpart while retaining the RGB residual. Reconstructing the same transition under this substitution encourages compatible motion conditioning in the RGB decoder, extending cross-modal correspondence beyond alignment in latent space. The world model learns from both components through separate conditioning pathways, while branch dropout prepares it for the zero-residual condition used at deployment. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, generation uses the current observation and robot actions.

To assess motion accuracy and visual fidelity, we evaluate SkeLAM on DROID (Khazatsky et al., 2024). Compared with DreamDojo, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1%. Motion probing shows that the kinematic component approaches the performance of the concatenated representation with half the latent dimensions, while latent interventions and component ablations support its role in generation. Qualitative results further demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Open-loop policy evaluation shows agreement between predicted success rates and those measured in real robot executions.

Our contributions are summarized as follows:

• We propose SkeLAM, a skeleton-guided latent action model that separates the conditioning roles of robot motion and complementary visual dynamics, providing a dedicated kinematic target for robot action adaptation.
• We introduce cross-modal swap reconstruction to encourage compatible RGB and skeleton motion conditioning during generation. Combined with branch dropout, this design supports robot action conditioning through the kinematic pathway learned from visual transitions.
• Experiments on DROID demonstrate gains in motion accuracy and visual fidelity, supported by representation analyses and component ablations. We further show zero-shot human-to-robot motion transfer and agreement between predicted and measured policy success rates.

2 Related Work

Action-conditioned robot world models. Robot world models support interactive simulation (Yang et al., 2024; Wu et al., 2024; Wang et al., 2025a) and policy evaluation (Zhu et al., 2025; Guo et al., 2026; Liao et al., 2025). Visual prediction supports foresight-guided policy learning (Lv et al., 2025a; Cai et al., 2026a) and multi-view data augmentation (Wang et al., 2026). Related research addresses video procedure planning (Ye et al., 2026), temporal answer grounding (Zhang et al., 2026), and cloud–edge agent execution (Huang et al., 2026). DALI-R (Luo et al., 2026) reranks actions with 3D point-cloud latent predictions and task-completion scores. Control representations include action embeddings, skeleton sequences (Wang et al., 2025b; Wu and Gao, 2026), and kinematic renderings (Xu et al., 2026). Recent models further incorporate spatial grounding (Gu et al., 2026; Yang et al., 2026b). Separating camera and object motion also supports camera-controlled video generation (Cheng et al., 2026b).

Latent action learning from videos. Latent action models encode visual transitions for world modeling (Bruce et al., 2024; Gao et al., 2025; Gao et al., 2026; Garrido et al., 2026) and robot policy learning (Ye et al., 2025; Liang et al., 2026; Yang et al., 2026a). Reconstruction-based learning is sensitive to distracting dynamics and ambiguous action correspondence (Nikulin et al., 2025; Zhang et al., 2025). Olaf-World (Jiang et al., 2026) aligns integrated latent actions with temporal differences in frozen video features across contexts. LAOF (Bu et al., 2026b) uses agent optical flow as motion supervision to reduce sensitivity to distractors. Shared RGB–depth features provide geometric supervision (Govind et al., 2026), alongside advances in latent representations for policy learning (Liu et al., 2026a; Shen et al., 2026; Xie et al., 2026; Liu et al., 2026b). LARYBench (Nie et al., 2026) evaluates representations using semantic action classification and physical action regression, independently of downstream policies. A systematic study finds that proxy metrics support coarse model selection but poorly rank downstream policy performance (Bu et al., 2026a). SkeLAM grounds its kinematic component in robot skeletons while retaining complementary visual dynamics in a separate residual.

Structured action representations. Quantized skills support autoregressive action generation (Li et al., 2025), while flow-matching policies preserve latent-action structure and align latent and physical actions through interpolation-based sampling during decoder training (Lyu et al., 2026). Spatial and kinematic priors include voxel heatmaps (Yang et al., 2026c), robot graphs with differentiable kinematics (Lv et al., 2025b), and trajectories defined by endpoint positions and velocities (Lv et al., 2026). Joint image, LiDAR, and action modeling supports autonomous-driving trajectory generation (Cheng et al., 2026a). SkeleWAM (Sheng et al., 2026) jointly learns action and future-state prediction over sparse 3D robot–object skeletons. Skel-WAM (Cai et al., 2026b) cotrains visual and hand-skeleton dynamics on human and robot videos, with a robot-trained action expert. SkeLAM uses skeleton supervision to ground video-derived kinematic latents for action-conditioned video prediction.

3 Methodology

SkeLAM learns structured latent actions for video-based world modeling and subsequent adaptation to robot controls. As illustrated in Figure 1, skeleton supervision grounds a kinematic component in robot motion, while a residual component retains complementary visual dynamics. The world model learns from both components, with branch dropout preparing it for action-conditioned inference. An action adapter is then trained to predict kinematic latents from robot actions, followed by world-model fine-tuning using these predictions.

Preliminary. Let o_{t} and a_{t} denote the visual observation and robot action at time t. An action-conditioned world model represents the distribution of H future observations:

p\!\left(o_{t+1:t+H}\mid o_{t},a_{t:t+H-1}\right).

To learn dynamics from videos, latent action models infer proxy actions z_{t} from visual transitions (o_{t},o_{t+1}). These representations are learned by reconstructing o_{t+1} from o_{t} and z_{t}, and subsequently provide conditioning signals for world-model training.

3.1 Skeleton-Grounded Latent Actions

Latent decomposition. To learn a kinematic representation for robot action adaptation while retaining complementary information for RGB reconstruction, we encode each RGB transition into kinematic latents z_{\mathrm{kin},t}^{\mathrm{rgb}} and residual latents z_{\mathrm{res},t}^{\mathrm{rgb}}. Further, to supervise the kinematic latent, we use skeleton images depicting the corresponding robot joint configurations.

Given paired RGB observations (o_{t},o_{t+1}) and their corresponding skeleton images (s_{t},s_{t+1}), Transformer encoders parameterize Gaussian posteriors for the RGB kinematic and residual latents, and the skeleton motion latent:

\displaystyle\bigl(z_{\mathrm{kin},t}^{\mathrm{rgb}},z_{\mathrm{res},t}^{\mathrm{rgb}}\bigr) \\ \displaystyle\sim q_{\phi}^{\mathrm{rgb}}(z_{\mathrm{kin}},z_{\mathrm{res}}\mid o_{t},o_{t+1}), \\ \displaystyle z_{\mathrm{kin},t}^{\mathrm{ske}} \\ \displaystyle\sim q_{\eta}^{\mathrm{ske}}(z_{\mathrm{kin}}\mid s_{t},s_{t+1}).

where z_{\mathrm{kin},t}^{\mathrm{rgb}},z_{\mathrm{kin},t}^{\mathrm{ske}}\in\mathbb{R}^{d_{\mathrm{kin}}} and z_{\mathrm{res},t}^{\mathrm{rgb}}\in\mathbb{R}^{d_{\mathrm{res}}}. The d_{\mathrm{kin}} and d_{\mathrm{res}} denote the dimensions of the kinematic and residual latents, respectively. We initialize the RGB and skeleton branches through conditional reconstruction:

\displaystyle\hat{o}_{t+1} \\ \displaystyle=D_{\mathrm{rgb}}\!\left(o_{t},z_{\mathrm{kin},t}^{\mathrm{rgb}},z_{\mathrm{res},t}^{\mathrm{rgb}}\right), \\ \displaystyle\hat{s}_{t+1} \\ \displaystyle=D_{\mathrm{ske}}\!\left(s_{t},z_{\mathrm{kin},t}^{\mathrm{ske}}\right).

Here, skeleton reconstruction establishes the motion reference used for subsequent supervision of the RGB kinematic representation.

Skeleton grounding. To ground the RGB kinematic representation in robot motion, we freeze the trained skeleton encoder and decoder. We reconstruct the next skeleton from the RGB kinematic posterior mean through the frozen decoder and align it with the paired skeleton motion representation. Let u_{t}=\operatorname{norm}(\mu_{\mathrm{kin},t}^{\mathrm{rgb}}) and v_{t}=\operatorname{sg}[\operatorname{norm}(\mu_{\mathrm{kin},t}^{\mathrm{ske}})], where \mu is the posterior mean, \operatorname{norm} denotes \ell_{2} normalization, and \operatorname{sg} denotes stop-gradient. The grounding and alignment objectives are:

\displaystyle\mathcal{L}_{\mathrm{ground}} \\ \displaystyle=\mathbb{E}\!\left[\ell_{\mathrm{ske}}\!\left(D_{\mathrm{ske}}(s_{t},\mu^{\mathrm{rgb}}_{\mathrm{kin},t}),s_{t+1}\right)\right], \\ \displaystyle\mathcal{L}_{\mathrm{align}} \\ \displaystyle=\mathbb{E}\!\left[1-u_{t}^{\mathsf{T}}v_{t}+\frac{\|u_{t}-v_{t}\|_{2}^{2}}{d_{\mathrm{kin}}}\right].

Here, \ell_{\mathrm{ske}} is the skeleton reconstruction loss detailed in Appendix B.1.

Cross-modal swap reconstruction. To extend motion supervision from the skeleton branch to the RGB decoder, we introduce cross-modal swap reconstruction. We replace the RGB kinematic input with the posterior mean of the paired skeleton motion latent, while retaining the residual sample used in RGB reconstruction:

\displaystyle\hat{o}_{t+1}^{\mathrm{swap}} \\ \displaystyle=D_{\mathrm{rgb}}\!\left(o_{t},\operatorname{sg}\!\left[\mu_{\mathrm{kin},t}^{\mathrm{ske}}\right],z_{\mathrm{res},t}^{\mathrm{rgb}}\right), \\ \displaystyle\mathcal{L}_{\mathrm{swap}} \\ \displaystyle=\mathbb{E}\!\left[\operatorname{MAE}\!\left(\hat{o}_{t+1}^{\mathrm{swap}},o_{t+1}\right)\right].

The RGB and swapped reconstructions share the decoder, current observation, and target, encouraging compatible motion conditioning across modalities. Gradients propagate to \phi through the residual latent, while the skeleton branch remains frozen.

Latent-action objective. The RGB objective combines reconstruction MSE with KL regularization of the kinematic and residual posteriors toward standard Gaussian priors:

\displaystyle\mathcal{L}_{\mathrm{rgb}}={} \\ \displaystyle\mathbb{E}_{\mathcal{D},q_{\phi}^{\mathrm{rgb}}}\!\left[\operatorname{MSE}(\hat{o}_{t+1},o_{t+1})\right] \\ \displaystyle+\beta\mathbb{E}_{\mathcal{D}}\!\left[D_{\mathrm{KL}}\!\left(q_{\phi,\mathrm{kin}}^{\mathrm{rgb}}\,\|\,\mathcal{N}(0,I)\right)+D_{\mathrm{KL}}\!\left(q_{\phi,\mathrm{res}}^{\mathrm{rgb}}\,\|\,\mathcal{N}(0,I)\right)\right].

Here \mathcal{D} denotes the paired training data and \beta controls the KL regularization strength. The component posteriors q_{\phi,\mathrm{kin}}^{\mathrm{rgb}} and q_{\phi,\mathrm{res}}^{\mathrm{rgb}} are the marginals of q_{\phi}^{\mathrm{rgb}}(\cdot\mid o_{t},o_{t+1}).

With the skeleton branch frozen, we optimize the RGB encoder and decoder using:

\displaystyle\mathcal{L}_{\mathrm{latent}}={} \\ \displaystyle\mathcal{L}_{\mathrm{rgb}}+\lambda_{\mathrm{ground}}\mathcal{L}_{\mathrm{ground}}+\lambda_{\mathrm{align}}\mathcal{L}_{\mathrm{align}} \\ \displaystyle+\lambda_{\mathrm{swap}}\mathcal{L}_{\mathrm{swap}}+\lambda_{\mathrm{var}}\mathcal{L}_{\mathrm{var}}.

The regularizer \mathcal{L}_{\mathrm{var}} penalizes low per-dimension standard deviations and cross-covariance between the RGB latent components. Grounding and latent alignment are introduced after conditional reconstruction initialization, followed by swap reconstruction. Initialization losses, regularizer definitions, loss weights, and the optimization schedule are provided in Appendix B.1.

3.2 Action-Conditioned World Modeling

Structured world-model conditioning. To preserve a kinematic conditioning pathway for robot action adaptation, we introduce separate projections for the kinematic and residual components. Given a training clip o_{t:t+H}, the frozen RGB encoder extracts z_{\mathrm{kin},i}^{\mathrm{rgb}} and z_{\mathrm{res},i}^{\mathrm{rgb}} from each transition (o_{i},o_{i+1}), where i\in\{t,\ldots,t+H-1\}. We use separate MLP projections P_{b} with branch-presence masks m_{b}\in\{0,1\}, leaving latent standardization implicit in the notation (Appendix A):

\displaystyle\tilde{z}_{b,i} \\ \displaystyle=m_{b}z^{\mathrm{rgb}}_{b,i},\qquad b\in\{\mathrm{kin},\mathrm{res}\}, \\ \displaystyle h_{i} \\ \displaystyle=P_{\mathrm{kin}}\!\left(\tilde{z}_{\mathrm{kin},i};m_{\mathrm{kin}}\right)+P_{\mathrm{res}}\!\left(\tilde{z}_{\mathrm{res},i};m_{\mathrm{res}}\right),

where a dropped branch has a zero latent and a zero presence mask. The world model p_{\theta}(o_{t+1:t+H}\mid o_{t},h_{t:t+H-1}) learns from video-derived latents through full-backbone fine-tuning after projector warmup. Robot-action latents reuse the kinematic projection.

Branch dropout and world-model training. Residual latents use future observations unavailable at deployment. To retain complementary information while preparing for missing conditions, \mathcal{R} samples one pair of branch masks per clip:

(m_{\mathrm{kin}},m_{\mathrm{res}})=\begin{cases}(1,1),&\text{full (keep)},\\
(0,1)\text{ or }(1,0),&\text{drop\_kin or drop\_res (partial)},\\
(0,0),&\text{drop\_both (null)}.\end{cases}

These four whole-branch modes share masks across all H transitions; their step-dependent probabilities are given in Appendix B.2. Partial means dropping one entire branch, not individual coordinates; null drops both branches using fixed zeros in standardized latent space. The supervision target remains the complete future clip o_{t+1:t+H} in every mode. The drop_res mode trains prediction from o_{t} and kinematic latents, matching action-conditioned inference.

Let \rho collect the learnable conditioning parameters. The world-model generation objective is

\mathcal{L}_{\mathrm{wm}}(\theta,\rho)=\mathbb{E}_{\begin{subarray}{c}\mathcal{D}_{\mathrm{video}},q_{\phi}^{\mathrm{rgb}},\mathcal{R},\xi\end{subarray}}\!\left[\ell_{\mathrm{gen}}\!\left(\theta;o_{t+1:t+H},o_{t},h_{t:t+H-1},\xi\right)\right],

where \mathcal{D}_{\mathrm{video}} contains training clips, \ell_{\mathrm{gen}} is the generation loss, and \xi collects its stochastic variables. The RGB encoder remains frozen.

Robot action adaptation and inference. To condition the world model on robot controls, we train a lightweight MLP adapter A_{\psi}:\mathcal{A}\rightarrow\mathbb{R}^{d_{\mathrm{kin}}}, where \mathcal{A} denotes the robot action space. For each transition (o_{i},a_{i},o_{i+1}), the frozen RGB encoder provides the kinematic posterior mean \mu_{\mathrm{kin},i}^{\mathrm{rgb}} as the regression target:

\mathcal{L}_{\mathrm{adapt}}(\psi)=\mathbb{E}_{\mathcal{D}_{\mathrm{robot}}}\!\left[d_{\mathrm{adapt}}\!\left(A_{\psi}(a_{i}),\operatorname{sg}\!\left[\mu_{\mathrm{kin},i}^{\mathrm{rgb}}\right]\right)\right],

where \mathcal{D}_{\mathrm{robot}} denotes the robot training data and d_{\mathrm{adapt}} is the latent regression criterion. Restricting regression to the kinematic component focuses adaptation on the motion representation learned through skeleton supervision. Only \psi is optimized, with the RGB encoder and (\theta,\rho) held fixed.

We then fine-tune the full world-model backbone using predicted kinematic latents and a curriculum residual–mask pair (\tilde{z}_{\mathrm{res},i}^{(s)},m_{\mathrm{res}}^{(s)}) (Appendix B.2):

\hat{h}_{i}^{\mathrm{act}}=P_{\mathrm{kin}}\!\left(A_{\psi}(a_{i});1\right)+P_{\mathrm{res}}\!\left(\tilde{z}_{\mathrm{res},i}^{(s)};m_{\mathrm{res}}^{(s)}\right).

The action-conditioned generation objective is

\mathcal{L}_{\mathrm{wm}}=\mathbb{E}_{\mathcal{D}_{\mathrm{robot}},\xi}\!\left[\ell_{\mathrm{gen}}\!\left(\theta;o_{t+1:t+H},o_{t},\hat{h}_{t:t+H-1}^{\mathrm{act}},\xi\right)\right].

At inference, \tilde{z}_{\mathrm{res},i}=0 and m_{\mathrm{res}}=0, with no future teacher residual; the current observation and actions specify the rollout:

\hat{o}_{t+1:t+H}\sim p_{\theta}\!\left(\,\cdot\mid o_{t},\hat{h}_{t:t+H-1}^{\mathrm{act}}\right).

4 Experiment

We evaluate the effect of the kinematic–residual decomposition on action-conditioned world modeling, and examine motion transfer and rollout evaluation as downstream applications. Specifically, we investigate the following questions:

• RQ1 (World modeling): Does adapting robot actions to a dedicated kinematic component improve motion accuracy while preserving visual fidelity? (Section 4.2)
• RQ2 (Motion representation): Do skeleton-grounded kinematic latents support more accurate linear decoding of robot motion than unstructured latent actions? (Section 4.3)
• RQ3 (Motion transfer): Can a kinematic representation learned from robot interactions transfer human motion to generated robot rollouts in a zero-shot setting? (Section 4.4)
• RQ4 (Policy evaluation): How closely do open-loop world-model rollouts reproduce policy rankings and success rates measured in real robot executions? (Section 4.5)

4.1 Experimental Setup

Data. DROID (Khazatsky et al., 2024) pairs visual observations with robot actions, allowing us to assess whether predicted dynamics follow robot controls. We train the latent action model and world model on DROID, incorporating AgiBot (Bu et al., 2025) to broaden the range of robot interactions observed during training. Evaluation uses 400 DROID clips, each containing one conditioning frame and 12 future frames at 480\times 640 resolution. More details are provided in Appendix A.

Evaluation Metrics. To quantify trajectory, direction, and magnitude agreement, we report 2D end-effector average displacement error (ADE, px), direction macro-F1 (\mathrm{F1}_{\mathrm{dir}}), normalized magnitude MAE (L_{1}^{\mathrm{mag}}), and magnitude Pearson correlation (r_{\mathrm{mag}}). PSNR, SSIM, and LPIPS (Zhang et al., 2018) assess pixel, structural, and perceptual fidelity, respectively, with tLPIPS reported as a temporal diagnostic. More details are provided in Appendix C.1.

Implementation Details. We implement the latent action model using Transformer VAEs with d_{\mathrm{kin}}=d_{\mathrm{res}}=32 and use Cosmos-Predict2.5-2B as the world-model backbone. We train with AdamW, using stage-specific learning rates and global batch sizes. SkeLAM training uses NVIDIA A100 GPUs and requires approximately 2 days for one complete run. Architectural specifications, loss weights, training schedules, and inference settings are provided in Appendix B.

Baselines. We compare SkeLAM with latent-action world models DreamDojo (Gao et al., 2026) and CD-LAM (Wei et al., 2026). Alternative conditioning strategies include direct robot actions in IRASim (Zhu et al., 2025), Ctrl-World (Guo et al., 2026), Cosmos-Predict2.5 (Ali et al., 2025), Cosmos3 (Agarwal et al., 2026), and GE-Sim (Liao et al., 2025), and geometric guidance in EnerVerse-AC (Jiang et al., 2025) and OSCAR (Wu and Gao, 2026). TesserAct (Zhen et al., 2025), and GE-Base (Liao et al., 2025) serve as text-conditioned references for visual fidelity, with evaluation details provided in Appendix C.

4.2 Action-Conditioned World Modeling

Quantitative results. To assess how accurately SkeLAM simulates the consequences of robot actions, we evaluate motion accuracy and visual fidelity on DROID. Table 1 shows that SkeLAM ranks first on seven of eight metrics among the reported action-conditioned models, including all four visual fidelity metrics. Compared with DreamDojo and CD-LAM, SkeLAM reduces ADE by 19.5% and 23.7%, respectively, alongside LPIPS reductions of 11.1% and 17.4%. These improvements indicate closer agreement with both the target robot trajectories and visual observations. Ctrl-World and OSCAR achieve higher direction F1, while SkeLAM attains lower trajectory error and stronger visual fidelity. In particular, compared with Ctrl-World, SkeLAM reduces ADE from 42.96 to 41.15 px and LPIPS from 0.4029 to 0.1310. The combined improvements in trajectory accuracy and visual fidelity support the generation of robot motion together with the associated scene dynamics.

Qualitative results. To evaluate whether generated rollouts reproduce robot motion while preserving scene structure, we compare predictions with ground truth in Figure 3. The overlaid trajectories reveal deviations from the reference motion in DreamDojo and OSCAR, accompanied by visible arm deformation in later frames. Ctrl-World produces a shorter trajectory and local gripper distortions, while Cosmos-Predict2.5 alters the apparent table scale and bowl position. SkeLAM more closely follows the curved reference trajectory while retaining the tabletop layout, although local gripper errors remain. This example highlights the importance of assessing motion correspondence alongside scene consistency. Two additional comparisons appear in Appendix E.

Ablation studies. To assess how each component supports action-conditioned generation, we compare variants in Table 2. (i) Skeleton supervision grounds the RGB kinematic latent in robot motion; removing it increases ADE from 41.15 to 64.59 and reduces magnitude correlation from 0.5277 to 0.2009, supporting geometric supervision for learning motion representations suitable for action conditioning. (ii) Cross-modal swap couples skeleton supervision to RGB generation beyond latent alignment; removing it increases ADE to 66.74 and reduces magnitude correlation to 0.1897, while LPIPS rises from 0.1310 to 0.1436, supporting reconstruction-level RGB–skeleton alignment for downstream action conditioning. (iii) Residual latent retains complementary scene dynamics; zeroing it from world-model pretraining maintains similar visual fidelity (0.1305 vs. 0.1310 LPIPS) while ADE increases to 43.37 and magnitude correlation decreases to 0.4737, suggesting that complementary visual information benefits motion conditioning. (iv) Action adaptation supervises the action-to-kinematic mapping; omitting adapter regression increases ADE to 65.84, supporting explicit supervision of the mapping from robot actions to the learned kinematic representation. Further details and adaptation-budget analysis are provided in Appendix D.

4.3 Motion Representation Analysis

Motion decodability. To examine how robot motion is encoded, we train linear probes on frozen latent representations. As shown in Table 4, the kinematic component achieves an EEF MAE of 5.99\times 10^{-3} and a magnitude macro-F1 of 0.7467, improving on both latent-action baselines. Within SkeLAM, it also outperforms the residual component on all three metrics. Concatenating the residual increases EEF R^{2} from 0.3200 to 0.3277 and magnitude macro-F1 from 0.7467 to 0.7482. The kinematic component thus approaches the probing performance of the full representation using half the latent dimensions, supporting its specialization for displacement and magnitude encoding.

Cross-modal compatibility. To assess cross-modal motion conditioning, we intervene on the latent inputs to the RGB decoder. Figure 4 shows that substituting paired and mismatched skeleton kinematic means increases reconstruction MAE by 0.53\% and 11.67\%, respectively. SSIM exhibits the same ordering. Zeroing the kinematic component increases MAE by 5.78\%, compared with 0.61\% for the residual. The greater degradation under mismatched substitution than kinematic removal indicates sensitivity to motion correspondence. These results support compatible RGB and skeleton conditioning and a functional role for the kinematic component in reconstruction.

4.4 Zero-Shot Human-to-Robot Motion Transfer

To examine whether the learned motion representation transfers across embodiments, we use human demonstration videos from UniSkill (Kim et al., 2025) as motion prompts. The RGB encoder extracts latent actions from these videos to directly condition SkeLAM, with a robot observation specifying the initial scene. All model parameters remain fixed during transfer. We compare with the pretrained checkpoint of DreamDojo (Gao et al., 2026), which supports latent-action conditioning. As shown in Figure 5, SkeLAM generates distinct end-effector motions corresponding to the human demonstrations: a downward reach for the towel-pushing prompt and an upward movement for the tissue-pulling prompt. DreamDojo exhibits limited end-effector displacement in both examples. These results provide qualitative evidence that human-derived latent actions can guide robot motion despite differences in embodiment and scene appearance.

4.5 World Model for Robot Policy Evaluation

To assess whether generated rollouts preserve task outcomes and relative policy performance, we evaluate seven policies across 15 RoboArena sessions (105 episodes). Both models receive the same initial observations and recorded actions. Human annotators label task success in generated rollouts for comparison with real execution outcomes. Table 4 reports 33.33\% success recall and 63.02\% balanced accuracy for SkeLAM, compared with 0.00\% and 48.44\% for Cosmos-Predict2.5. At the policy level, success-rate MAE decreases from 7.62 to 4.76 percentage points, with positive Spearman (\rho=0.5347) and Pearson (r=0.6091) correlations for SkeLAM. Together, these metrics indicate improved agreement in both absolute policy success rates and relative policy ordering. Figure 6 compares predicted and real success rates for individual policies, with proximity to the diagonal indicating closer agreement. The results support agreement in both individual task outcomes and relative policy performance under open-loop action replay. Nevertheless, the modest success recall indicates that reproducing successful task outcomes remains challenging. Further evaluation details are provided in Appendix C.3.

5 Conclusion

We presented SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. The kinematic component provides a motion-grounded target for robot action adaptation, while the residual retains complementary visual dynamics. Cross-modal swap reconstruction and branch dropout support reuse of the learned kinematic pathway for action-conditioned generation. Experiments on DROID show improved motion accuracy and visual fidelity, with representation analyses supporting kinematic specialization. Qualitative results demonstrate zero-shot human-to-robot motion transfer, while policy evaluation yields success-rate estimates consistent with real robot executions. These results support structuring latent actions around robot motion while retaining complementary information for generation. Future work will extend the current recorded-action replay protocol to closed-loop policy evaluation. A key question is how prediction errors affect subsequent policy decisions and task outcomes over longer interaction horizons.

References

1. Agarwal et al. (2026) N. Agarwal, A. Ali, J. Allen, M. Antolini, A. Aubame, A. Azzolini, J. Bai, M. Bala, Y. Balaji, J. Bapst, et al. Cosmos 3: omnimodal world models for physical ai. arXiv preprint arXiv:2606.02800.
2. Ali et al. (2025) A. Ali, J. Bai, M. Bala, Y. Balaji, A. Blakeman, T. Cai, J. Cao, T. Cao, E. Cha, Y. Chao, et al. World simulation with video foundation models for physical ai. arXiv preprint arXiv:2511.00062.
3. Bruce et al. (2024) J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. M. E. Bechtle, F. Behbahani, S. C.Y. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. D. Freitas, S. Singh, and T. Rocktäschel Genie: generative interactive environments. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, pp. 4603–4623. External Links: Link
4. Bu et al. (2025) Q. Bu, J. Cai, L. Chen, X. Cui, Y. Ding, S. Feng, X. He, X. Huang, et al. AgiBot World Colosseo: a large-scale manipulation platform for scalable and intelligent embodied systems. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), External Links: Link
5. Bu et al. (2026a) X. Bu, Q. Hu, L. Zhou, L. Zhang, Y. Tang, Z. Liu, X. Tao, Z. Ma, Q. Huang, C. Tang, H. Wang, J. Zhang, J. Ma, H. Ye, W. Li, and X. Hao What Matters for Latent Actions in Robot Learning. arXiv preprint arXiv:2608.19613. External Links: Link
6. Bu et al. (2026b) X. Bu, J. Lyu, F. Sun, R. Yang, Z. Ma, and W. Li LAOF: Robust Latent Action Learning with Optical Flow Constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27334–27344. External Links: Link
7. Cai et al. (2026a) J. Cai, Z. Cai, J. Cao, Y. Chen, Z. He, L. Jiang, H. Li, H. Li, Y. Li, Y. Liu, Y. Lu, Q. Lv, H. Ma, J. Pang, Y. Qiao, Z. Qiu, Y. Shen, X. Shi, Y. Tian, B. Wang, H. Wang, J. Wang, T. Wang, X. Wei, C. Wu, Y. Xie, B. Xing, Y. Yang, Y. Yang, Q. Yu, F. Yuan, J. Zeng, J. Zhang, S. Zhang, S. Zhang, Z. Zhaxi, B. Zhou, Y. Zhou, Y. Zhou, H. Zhu, Y. Zhu, and Y. Zhu InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation. arXiv preprint arXiv:2601.02456. External Links: Link
8. Cai et al. (2026b) Z. Cai, Y. Li, Y. Wang, X. Zhan, Y. Yang, H. Ma, K. Li, T. Lu, J. Pang, L. Xu, and D. Lin Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer. arXiv preprint arXiv:2609.21514. External Links: Link
9. Cheng et al. (2026a) Z. Cheng, X. Zhang, D. Di, L. Fan, B. Ma, H. Li, and X. Yang MOJITO: modal joint learning for unified end-to-end autonomous driving. In European Conference on Computer Vision, pp. 620–639.
10. Cheng et al. (2026b) Z. Cheng, X. Zhang, D. Di, C. Wei, H. Li, and X. Yang MoCa: modeling object consistency for 3D camera control in video generation. In The Fourteenth International Conference on Learning Representations,
11. Gao et al. (2026) S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W. Tseng, Y. Dong, K. Mo, C. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. R. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M. Liu, Y. Zhu, J. Jang, and L. ”. Fan DreamDojo: a generalist robot world model from large-scale human videos. arXiv preprint arXiv:2602.06949. External Links: Link
12. Gao et al. (2025) S. Gao, S. Zhou, Y. Du, J. Zhang, and C. Gan AdaWorld: learning adaptable world models with latent actions. In International Conference on Machine Learning, External Links: Link
13. Garrido et al. (2026) Q. Garrido, T. Nagarajan, B. Terver, N. Ballas, Y. LeCun, and M. Rabbat Learning Latent Action World Models In The Wild. arXiv preprint arXiv:2601.05230. External Links: Link
14. Govind et al. (2026) M. K. Govind, D. Reilly, P. Wang, and S. Das UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models. arXiv preprint arXiv:2602.20231. External Links: Link
15. Gu et al. (2026) C. Gu, H. Yu, J. Zhang, H. Lin, W. Zhang, J. Wang, H. Jin, S. Xie, J. Jiang, and Z. Wang GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions. arXiv preprint arXiv:2608.06332. External Links: Link
16. Guo et al. (2026) Y. Guo, L. X. Shi, J. Chen, and C. Finn Ctrl-World: A Controllable Generative World Model for Robot Manipulation. In International Conference on Learning Representations, External Links: Link
17. Huang et al. (2026) Y. Huang, M. Zhang, H. Lei, C. Ji, J. Wang, and H. Zhu CE-Agent: a cloud-edge collaborative agent framework with dynamic offloading and synergistic distillation. Journal of Cloud Computing. External Links: Document, Link
18. Jiang et al. (2025) Y. Jiang, S. Chen, S. Huang, L. Chen, P. Zhou, Y. Liao, X. He, C. Liu, H. Li, M. Yao, and G. Ren EnerVerse-ac: envisioning embodied environments with action condition. arXiv preprint arXiv:2505.09723.
19. Jiang et al. (2026) Y. Jiang, Y. Gu, I. W. Tsang, and M. Z. Shou Olaf-World: Orienting Latent Actions for Video World Modeling. In International Conference on Machine Learning, External Links: Link
20. Khazatsky et al. (2024) A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, V. Guizilini, D. A. Herrera, M. Heo, K. Hsu, J. Hu, M. Z. Irshad, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O’Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Martín-Martín, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn DROID: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. External Links: Link
21. Kim et al. (2025) H. Kim, J. Kang, H. Kang, M. Cho, S. J. Kim, and Y. Lee UniSkill: imitating human videos via cross-embodiment skill representations. In 9th Annual Conference on Robot Learning,
22. Li et al. (2025) H. Li, Q. Lv, R. Shao, X. Deng, Y. Li, J. Hao, and L. Nie STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization. In Proceedings of the 42nd International Conference on Machine Learning, pp. 35562–35580. External Links: Link
23. Liang et al. (2026) A. Liang, P. Czempin, M. M. Hong, Y. Zhou, J. Wang, E. Biyik, and S. Tu CLAM: continuous latent action models for robot learning from unlabeled demonstrations. In IEEE/RSJ International Conference on Intelligent Robots and Systems, External Links: Link
24. Liao et al. (2025) Y. Liao, P. Zhou, S. Huang, D. Yang, S. Chen, Y. Jiang, Y. Hu, J. Cai, S. Liu, J. Luo, L. Chen, S. Yan, M. Yao, and G. Ren Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation. arXiv preprint arXiv:2508.05635. External Links: Link
25. Liu et al. (2026a) J. Liu, Z. Xia, S. Tian, H. Li, Y. Zheng, N. Ma, X. Fu, X. Liu, J. Li, Y. Li, S. Zhou, Z. Xing, L. Wang, C. Li, H. Li, and D. Zhao WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos. arXiv preprint arXiv:2607.11397. External Links: Link
26. Liu et al. (2026b) Y. Liu, P. Yuan, X. Zhu, Y. Guo, and J. Chen GALA: Geometry-Aware Latent Action Modeling for Vision-Language-Action Model Pretraining across Embodiments. arXiv preprint arXiv:2609.21948. External Links: Link
27. Luo et al. (2026) L. Luo, X. Bu, R. Liu, Q. Huang, C. Tang, X. Hao, H. Wang, and W. Li Data-Asymmetric Latent Imagination and Reranking for 3D Robotic Imitation Learning. arXiv preprint arXiv:2605.10166. External Links: Link
28. Lv et al. (2025a) Q. Lv, W. Kong, H. Li, J. Zeng, Z. Qiu, D. Qu, H. Song, Q. Chen, X. Deng, and J. Pang F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actions. arXiv preprint arXiv:2509.06951. External Links: Link
29. Lv et al. (2025b) Q. Lv, H. Li, X. Deng, R. Shao, Y. Li, J. Hao, L. Gao, M. Y. Wang, and L. Nie Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17394–17404. External Links: Link
30. Lv et al. (2026) Q. Lv, J. Xing, Z. Yang, M. Yao, Y. Shi, Y. Jueluo, M. Z. Shou, and X. Deng Hermite Curves as Trajectory Priors for Vision-Language-Action Models. arXiv preprint arXiv:2608.01265. External Links: Link
31. Lyu et al. (2026) J. Lyu, X. Bu, Q. Huang, C. Tang, X. Hao, H. Wang, and W. Li LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching. arXiv preprint arXiv:2606.10517. External Links: Link
32. Nie et al. (2026) D. Nie, F. Chen, Q. Lv, J. Kuang, X. Li, X. Cao, and X. Cai LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment. arXiv preprint arXiv:2604.11689. External Links: Link
33. Nikulin et al. (2025) A. Nikulin, I. Zisman, D. Tarasov, N. Lyubaykin, A. Polubarov, I. Kiselev, and V. Kurenkov Latent action learning requires supervision in the presence of distractors. In International Conference on Machine Learning, External Links: Link
34. Shen et al. (2026) Z. Shen, J. Liang, J. Lu, F. Jiang, Y. Wang, C. Wei, J. Liu, J. Yang, Q. Yu, J. You, C. Hao, G. He, C. Xie, and R. Wu LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models. arXiv preprint arXiv:2608.22403. External Links: Link
35. Sheng et al. (2026) J. Sheng, H. Wang, and M. Liu SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation. arXiv preprint arXiv:2610.02120. External Links: Link
36. Wang et al. (2026) B. Wang, H. Zhang, S. Zhang, J. Hao, M. Jia, Q. Lv, Y. Mao, Z. Lyu, J. Zeng, X. Xu, and J. Pang RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation. arXiv preprint arXiv:2601.05241. External Links: Link
37. Wang et al. (2025a) L. Wang, K. Zhao, C. Liu, and X. Chen Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression. arXiv preprint arXiv:2502.04296. External Links: Link
38. Wang et al. (2025b) Y. Wang, C. Wen, H. Guo, S. Peng, M. Qin, H. Bao, X. Zhou, and R. Hu Precise action-to-video generation through visual action prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, External Links: Link
39. Wei et al. (2026) Y. Wei, K. Zhou, L. Mao, Z. Zhang, Z. Xu, Z. Xi, S. Liang, R. Han, Y. Yan, X. Wang, F. Feng, and B. Huang Causally debiased latent action model for embodied action conditioned world models. arXiv preprint arXiv:2607.09185.
40. Wu et al. (2024) J. Wu, S. Yin, N. Feng, X. He, D. Li, J. Hao, and M. Long iVideoGPT: interactive VideoGPTs are scalable world models. In Advances in Neural Information Processing Systems, External Links: Link
41. Wu and Gao (2026) Z. Wu and J. Gao OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics. arXiv preprint arXiv:2606.04463. External Links: Link
42. Xie et al. (2026) Y. Xie, H. Tian, J. Liu, Y. Wang, Q. Sun, and W. Ding GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos. arXiv preprint arXiv:2609.17099. External Links: Link
43. Xu et al. (2026) M. Xu, T. Zhang, T. Liu, Z. Chen, X. Han, and Z. Liu Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation. arXiv preprint arXiv:2603.16669. External Links: Link
44. Yang et al. (2026a) J. Yang, Y. Shi, H. Zhu, M. Liu, K. Ma, Y. Wang, G. Wu, T. He, and L. Wang CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link
45. Yang et al. (2026b) J. Yang, J. Zhange, J. Li, H. Guo, W. Huang, J. Wang, P. Liu, and S. Xia LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models. arXiv preprint arXiv:2608.05706. External Links: Link
46. Yang et al. (2026c) P. Yang, H. Ci, Y. Chen, Q. Lv, H. Cai, and M. Z. Shou ActionMap: Robot Policy Learning via Voxel Action Heatmap. arXiv preprint arXiv:2606.06904. External Links: Link
47. Yang et al. (2024) S. Yang, Y. Du, S. Ghasemipour, J. Tompson, L. Kaelbling, D. Schuurmans, and P. Abbeel Learning interactive real-world simulators. In International Conference on Learning Representations, External Links: Link
48. Ye et al. (2025) S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y. Chao, B. Y. Lin, L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo Latent action pretraining from videos. In International Conference on Learning Representations, External Links: Link
49. Ye et al. (2026) Z. Ye, L. Zhang, S. Zhou, Y. Yu, Y. Shi, J. Xuan, S. Dong, G. Tong, M. Zhang, and B. Li OSEF: one-step evidence fusion for cross-video scene procedure planning. arXiv preprint arXiv:2607.29401. External Links: 2607.29401, Document, Link
50. Zhang et al. (2025) C. Zhang, T. Pearce, P. Zhang, K. Wang, X. Chen, W. Shen, L. Zhao, and J. Bian What do latent action models actually learn?. In Advances in Neural Information Processing Systems, External Links: Link
51. Zhang et al. (2026) M. Zhang, X. Zhang, H. Lei, Z. Xu, X. Qi, S. Liu, and J. Ruan Difficulty-aware instructional step reasoning for temporal answer grounding in single video. In NLPCC 2026 Shared Tasks, Note: Accepted, to appear External Links: Link
52. Zhang et al. (2018) R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang The unreasonable effectiveness of deep features as a perceptual metric. In CVPR,
53. Zhen et al. (2025) H. Zhen, Q. Sun, H. Zhang, J. Li, S. Zhou, Y. Du, and C. Gan Tesseract: learning 4d embodied world models. arXiv preprint arXiv:2504.20995.
54. Zhu et al. (2025) F. Zhu, H. Wu, S. Guo, Y. Liu, C. Cheang, and T. Kong IRASim: a fine-grained world model for robot manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, External Links: Link

Appendix A Data and Preprocessing

Data and splits.

DROID and AgiBot World (Bu et al., 2025) are used in latent-action learning, world-model training, and action adaptation. DROID contributes 60,000 training clips to Stage 1 and 45,358 clips to each of Stages 2A, 2B, 3A, and 3B. RGB and skeleton observations are matched by episode, camera, and frame index; overlapping windows from an episode remain in the same partition. Normalization statistics are fitted on training data only. The DROID evaluation selects one clip from each of 400 trajectories: 100 trajectories per motion quartile, with additional camera stratification. AgiBot evaluation uses 400 held-out clips and 800 arm trajectories.

Preprocessing.

RGB and pre-rendered skeleton observations come from the corresponding OSCAR (Wu and Gao, 2026) subsets, following its kinematics, projection, and rasterization protocol. Stage 1 letterboxes both modalities to 240\times 320 (height \times width): resize by \min(240/H_{o},320/W_{o}) using OpenCV INTER_AREA, round the resized dimensions, and center on a zero canvas, assigning odd padding pixels to the bottom or right. Intensities are divided by 255. The target skeleton mask selects pixels with an 8-bit RGB channel sum above 15, then applies one dilation with a 5\times 5 elliptical kernel. It is used only in the skeleton loss. Stage 2/3 uses a 4{:}3 center crop resized to 480\times 640 for video generation; the frozen latent encoder keeps the Stage 1 resolution and uses the same source frame indices.

Temporal alignment and action interface.

A DROID window samples 13 frames from 60-FPS video at indices f_{k}=f_{0}+4k, k=0,\ldots,12, giving 15 FPS and a 0.8-second span; consecutive window starts differ by 24 source frames. The first frame conditions 12 future frames. Each transition pairs (o_{k},o_{k+1}) with an executed state-delta proxy

a_{k}=[p_{k+1}-p_{k};\,\Delta\omega_{k};\,g_{k+1}]\in\mathbb{R}^{7},

where translation is in the robot base frame, \Delta\omega_{k} is the rotation vector of the relative \mathrm{SO}(3) rotation, and g_{k+1} is destination gripper openness. Deltas are between consecutive sampled states, not relative to the clip start or joint-space commands. Manifests retain source indices, timestamps, and transition-validity indicators.

Deterministic latent extraction.

The frozen Stage 1 RGB encoder exports posterior means as two 12\times 32 arrays, z_{\mathrm{kin}} and z_{\mathrm{res}}; the skeleton encoder is not used for world-model conditioning. Each component is standardized elementwise as \bar{z}=(z-m_{z})\oslash\max(s_{z},\epsilon) using training statistics. The same kinematic statistics define adapter targets and world-model inputs; normalization is implicit in the main-text notation. Offline caches retain episode, camera, frame, preprocessing, and checkpoint identifiers and receive no gradients.

Appendix B Model and Training Details

Architecture and initialization.

Both Transformer-VAE branches start from random weights and use 16\times 16 patches, width 1,024, 16 encoder and 16 decoder blocks, 16 attention heads, causal temporal attention, and zero Transformer dropout. The RGB posterior has 32 kinematic and 32 residual dimensions; the skeleton posterior has 32 kinematic dimensions. The world model starts from official Cosmos-Predict2.5 2B weights (Ali et al., 2025), independently of the DreamDojo baseline. The final Stage 1 teacher, video tokenizer, and text encoder remain frozen throughout Stage 2/3.

Figure 7 summarizes the training stages, selected checkpoints, and dependencies.

B.1 Latent-action training

Stages 1A/B/C initialize RGB and skeleton reconstruction, add grounding and alignment, then add swap, respectively (Figure 7; \mathrm{K}=1{,}000). The skeleton branch is frozen in Stage 1B/C. RGB reconstruction uses MSE and the two KL terms in Eq. (6), with \beta=2\times 10^{-4}; swapping adds no KL term. Grounding uses the RGB kinematic posterior mean through the frozen skeleton decoder; alignment uses paired, \ell_{2}-normalized means, with the skeleton target detached. Swap reconstruction uses the frozen skeleton mean and the same residual sample as ordinary RGB reconstruction.

For target skeleton image s, foreground mask m, image prediction \hat{s}, and mask prediction \hat{m}, the grounding criterion is

\displaystyle\ell_{\mathrm{ske}} \\ \displaystyle=\ell_{\mathrm{fg}}+0.5\ell_{\mathrm{mask}}+0.05\ell_{\mathrm{bg}}, \\ \displaystyle\ell_{\mathrm{fg}} \\ \displaystyle=\frac{\sum_{p,c}m_{p}|\hat{s}_{pc}-s_{pc}|}{C_{s}\sum_{p}m_{p}+\epsilon},\qquad\ell_{\mathrm{bg}}=\frac{\sum_{p,c}(1-m_{p})|\hat{s}_{pc}|}{C_{s}\sum_{p}(1-m_{p})+\epsilon}, \\ \displaystyle\ell_{\mathrm{mask}} \\ \displaystyle=\operatorname{BCE}(\hat{m},m)+1-\frac{2\sum_{p}\hat{m}_{p}m_{p}+\epsilon}{\sum_{p}\hat{m}_{p}+\sum_{p}m_{p}+\epsilon},

where C_{s} is the number of color channels. For batch representations Z, the regularizer is

\displaystyle V_{0.5}(Z) \\ \displaystyle=\frac{1}{d}\sum_{j=1}^{d}\max(0,0.5-\operatorname{Std}_{b}Z_{bj}), \\ \displaystyle\mathcal{L}_{\mathrm{var}} \\ \displaystyle=\frac{V_{0.5}(Z_{\mathrm{kin}})+V_{0.5}(Z_{\mathrm{res}})}{2}+\frac{0.1\|\operatorname{Cov}(Z_{\mathrm{kin}},Z_{\mathrm{res}})\|_{F}^{2}}{d_{\mathrm{kin}}d_{\mathrm{res}}}.

Cross-covariance uses centered RGB components and excludes the frozen skeleton latent; this discourages linear redundancy, without enforcing independence. Stage 1B minimizes \mathcal{L}_{\mathrm{rgb}}+\mathcal{L}_{\mathrm{ground}}+0.5\mathcal{L}_{\mathrm{align}}+0.1\mathcal{L}_{\mathrm{var}}. Stage 1C adds \lambda_{\mathrm{swap}}\mathcal{L}_{\mathrm{swap}}, with weight 0.5 for 500 updates and 1.0 thereafter; 20% of updates instead use only \mathcal{L}_{\mathrm{rgb}}, including its KL terms.

Stages 1B/C use AdamW with (\beta_{1},\beta_{2})=(0.9,0.999), \epsilon=10^{-8}, zero weight decay, gradient-norm clipping 0.3, and constant learning rates 5\times 10^{-5} and 2\times 10^{-5}, respectively, without warmup. The batch is 32 frame pairs on eight A100 80-GB GPUs (four pairs per GPU); Stage 1C uses bfloat16. At phase transitions only model weights are restored. The configured limits are 40K/10K updates, but the selected checkpoints are 20K/5K.

B.2 World-model training and action adaptation

Stage 2A performs 5K conditioning-projector updates, followed by 20K full-DiT and conditioning updates in Stage 2B. Stage 3A fits only the action adapter for 2K updates with the world model fixed; its learning-rate cycle is configured for 8K updates. Stage 3B starts from these Stage 2B/3A checkpoints and performs 30K action-conditioned generation updates.

Branch dropout.

Standardized latents are masked before projection; the carrier contains two 32-dimensional vectors and two binary presence masks (66 values). A dropped branch has zero latent and zero mask. One mode is sampled per clip and shared over all 12 transitions. For modes (1,1),(0,0),(0,1),(1,0), probabilities are (0.80,0.10,0.05,0.05) for global Stage 2 steps 0–999 and (0.50,0.10,0.05,0.35) thereafter. Stage 2B starts at global step 5,000.

Adapter and residual curriculum.

The DROID adapter is a per-transition MLP, 7\rightarrow 256\rightarrow 256\rightarrow 32, with GELU between linear layers. Training-standardized actions predict standardized RGB kinematic posterior means. With valid-transition indicator v_{ni}, its regression loss is

\mathcal{L}_{\mathrm{adapt}}=\frac{\sum_{n,i}v_{ni}\|A_{\psi}(a_{ni})-\operatorname{sg}[\mu^{\mathrm{rgb}}_{\mathrm{kin},ni}]\|_{2}^{2}}{32\sum_{n,i}v_{ni}+\epsilon}.

In Stage 3B, kinematic conditions always come from actions with mask one. The paired residual is retained with probability 0.5 for updates 0–1,999, decaying linearly to zero over updates 2,000–7,999, and zero thereafter. One choice is shared across a clip; discarded residuals and their masks are zero.

Both stages use AdamW, (0.9,0.999), \epsilon=10^{-8}, weight decay 10^{-3}, global gradient clipping 0.1, batch 32, bfloat16, eight-way FSDP, EMA, and 1K checkpoint intervals. Stage 2B uses seed 0 and LambdaLinear scheduling: 100-step warmup, f_{\mathrm{start}}=10^{-6}, f_{\max}=0.99, f_{\min}=0.4, cycle 400K. Normalization statistics remain fixed in Stage 3B.

DROID inference.

One observed frame and 12 action transitions generate 12 future frames at 480\times 640, always with zero residual and mask. We use FlowUniPCMultistepScheduler, order 2, solver bh2, 35 steps, shift 5, and a non-Karras schedule. Guidance is f_{\mathrm{cond}}+g(f_{\mathrm{cond}}-f_{\mathrm{uncond}}) with g=0 (conditional prediction); text is the empty string encoded online, with negative prompts disabled. Results use regular, not EMA, weights at Stage 3B step 30K. Each clip is generated once with seed 20260823+p for zero-based manifest position p; eight ranks partition clips without padding or duplication. All 400 generations completed.

Appendix C Evaluation Protocols

C.1 Video and motion metrics

Video fidelity.

Canonical clips have 13 RGB frames in [0,1] at 480\times 640. Video metrics include the initial conditioning frame; motion metrics use the 12 future frames. MSE and MAE average errors over all channels, pixels, and frames per clip, then over valid clips. PSNR is computed from each clip’s MSE e_{n} before averaging: N_{v}^{-1}\sum_{n}10\log_{10}(1/e_{n}) dB. SSIM uses skimage.metrics.structural_similarity, data_range=1.0, channel_axis=-1, and the default 7\times 7 uniform window. LPIPS uses TorchMetrics AlexNet with normalize=True. Both are averaged over frames and then clips. Temporal LPIPS is the mean over clips of

\mathrm{tLPIPS}_{n}=\frac{1}{12}\sum_{t=1}^{12}\left|D_{\mathrm{LPIPS}}(\hat{I}_{n,t},\hat{I}_{n,t-1})-D_{\mathrm{LPIPS}}(I_{n,t},I_{n,t-1})\right|.

It measures agreement in adjacent-frame perceptual change, not end-effector motion.

Validity and displacement.

A valid arm trajectory requires in-image reference positions and complete prediction tracking at every evaluated frame, with finite coordinates throughout. Invalid trajectories are excluded from scores but remain in the coverage denominator (400 for DROID, 800 for AgiBot). Thus methods can have different valid subsets; generated-video and valid-video counts are reported separately. For valid trajectories, let \Delta p_{nt}=p_{nt}-p_{n0} and \Delta\hat{p}_{nt}=\hat{p}_{nt}-\hat{p}_{n0}. Then \mathrm{ADE}=(12N_{m})^{-1}\sum_{n=1}^{N_{m}}\sum_{t=1}^{12}\|\Delta\hat{p}_{nt}-\Delta p_{nt}\|_{2} pixels.

Direction and magnitude.

Direction uses initial-to-final displacement: the sign of the horizontal component determines left/right when |\Delta x|\geq|\Delta y|; otherwise the vertical sign determines up/down (positive image y is down). Displacements below 0.005 image diagonals (4 pixels) are stationary. Four-class direction macro-F1 includes only trajectories for which both reference and prediction are nonstationary, a potentially smaller subset than ADE. Magnitude is m_{n}=\|\Delta p_{n,12}\|_{2}/800; magnitude MAE is N_{m}^{-1}\sum_{n}|\hat{m}_{n}-m_{n}|, and r_{\mathrm{mag}} and \rho_{\mathrm{mag}} are Pearson and Spearman correlations over valid trajectories. Undefined correlations are unavailable, not zero.

C.2 Baselines, probes, and interventions

Table 1 lists control interfaces; text-only TesserAct (RGB-LoRA) and GE-Base are visual references. Linear probes use frozen encoders and \widehat{\Delta p}=Wz+b: displacement MAE/R^{2} and magnitude macro-F1. Individual latents are 32D, their concatenation 64D; the skeleton teacher uses privileged input.

For interventions, deterministic means k,r,s denote RGB kinematic, RGB residual, and skeleton kinematic latents. Normal, correct/wrong swap, and zero-kinematic/residual inputs are [k;r], [s;r], [s^{\prime};r], [0;r], and [k;0]; s^{\prime} comes from another episode. The decoder, current frame, and target are fixed. Figure 4 reports 100(\bar{e}_{j}-\bar{e}_{0})/\bar{e}_{0} for MAE and 10^{3}(\overline{\mathrm{SSIM}}_{0}-\overline{\mathrm{SSIM}}_{j}). Raw decoder-latent zeroing differs from world-model dropout of standardized latents and presence masks. These tests support motion specialization without establishing complete disentanglement.

C.3 Human transfer and policy evaluation

Human-to-robot transfer.

UniSkill (Kim et al., 2025) videos provide RGB-encoder latent sequences directly to the world model, bypassing the action adapter; a robot observation specifies the initial scene. All weights remain fixed, and DreamDojo uses its latent-conditioned checkpoint. The two examples assess visible motion correspondence, not task completion or reliable transfer across arbitrary viewpoints and embodiments.

Open-loop policy evaluation.

Seven RoboArena policies are evaluated in 15 sessions (105 paired episodes), with nine real successes and 96 failures. Policy identifiers are pi0_fast, paligemma_fast, paligemma_fast_specialist, paligemma_vq, pi0, paligemma_binning, and paligemma_diffusion, each with suffix _droid. Models receive the same initial observations and recorded actions; human annotators assign binary task-success labels to generated rollouts using the real-execution criterion. Actions are replayed without policy feedback from generated observations. Episode-level metrics pool all pairs, treating success as positive: success recall, balanced accuracy (mean sensitivity and specificity), and MCC. For policy p, real and generated success rates r_{p},\hat{r}_{p} average the 15 session labels. Pearson and Spearman correlations use seven rate pairs, with average ranks for ties; success-rate MAE is \Delta_{\mathrm{SR}}=(100/7)\sum_{p}|r_{p}-\hat{r}_{p}| percentage points. The seven policies and nine real successes limit the ranking evidence.

Appendix D Supplementary Results and Ablations

Additional metrics and coverage.

Table 6 complements Table 1 with DROID metrics and trajectory coverage. OSCAR has lower tLPIPS than SkeLAM (0.0174 vs. 0.0222), while SkeLAM has lower ADE and LPIPS. Tables 7 and 8 report AgiBot results: SkeLAM improves SSIM over DreamDojo but has worse LPIPS and ADE, so DROID gains do not extend uniformly to AgiBot. Table 9 compares adaptation budgets.

Component ablations.

In Table 2, removing skeleton supervision omits grounding, alignment, and swap, retaining RGB reconstruction; removing swap retains grounding and alignment with \lambda_{\mathrm{swap}}=0. Removing the residual uses zero residual from Stage 2 onward. Removing action adaptation skips adapter regression and jointly trains a randomly initialized adapter and world model for action-conditioned generation. The latter two retain the full model’s action interface and inference settings, including zero residual. These change training; decoder interventions keep weights fixed.

Appendix E Additional Qualitative Comparisons

Figures 9 and 9 supplement Figure 3 with two DROID scenes. Both grids compare recorded observations and generated rollouts at four time steps, showing end-effector motion, arm geometry, and scene appearance.

AI use statement.

Generative AI assisted language polishing and readability; all AI-assisted content was reviewed by authors.