XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation
XGenAct is a preprint describing a world action model that turns RGB observations, robot actions, metric depth, surface normals, and functional-role segmentation into RGB video streams and trains one video diffusion transformer on all of them. On five held-out RLBench tasks it averages 52% closed-loop success versus 26% for the strongest evaluated baseline. This matters because it adds explicit geometric and semantic future prediction to robot action generation without modality-specific heads or auxiliary losses.
Can multiple geometric and semantic future-prediction tasks expressed as RGB video streams in a single shared diffusion objective improve robot action generation over RGB-only world action models?
Current world action models predict future RGB frames and robot actions but mostly capture temporal dynamics, not explicit metric depth, surface orientation, or task-relevant functional regions. Prior attempts add such signals through auxiliary heads, branches, or separate decoders, leaving spatial supervision fragmented and not unified with action generation.
Earlier work such as PAD, GeoSem-WAM, DreamWAM, WAM4D, X-WAM, and RynnWorld-4D added depth, flow, or geometry through separate encoders/decoders, auxiliary branches, residual modules, or multi-branch architectures; SA-WAM co-encodes RGB and depth but keeps actions in fixed dedicated latent slots. These designs rely on modality-specific components and do not make structured perception a direct part of the same action-generating video objective.
XGenAct uses a frozen Wan2.2-TI2V-5B video VAE and one fine-tuned diffusion transformer. RGB is identity-mapped; 7-DoF actions use the Action Images multiview video codec; metric depth uses a monotonic power transform mapped onto an RGB cube path; surface normals use (n+1)/2; and functional-role segmentation uses fixed task-role RGB codes. Training samples pick one perception stream and pair it with action streams from two camera views as [m1,a1,m2,a2], with joint, policy, or inverse conditioning plans; all templates share the same flow-matching loss on predicted positions only and no auxiliary learned heads. Models are fine-tuned from the Action Images checkpoint with batch 4, learning rate 5e-7, on self-generated RLBench data of 16 tasks and 3,960 episodes plus ManiSkill3 data of 7 tasks and 1,750 episodes.
On five held-out RLBench tasks with 20 trials each, XGenAct (RGB+D+N+S) averages 52% closed-loop success, versus 26% for the best baselines pi_0.5 and VLANeXt; it reaches 100% on close drawer, 60% on close microwave, 40% on close laptop, 40% on toilet seat down, and 20% on reach target. In a training-menu ablation, every perception stream improves over RGB-only 32.5% average; RGB+N+S is best at 61.25% (+28.75 points), while RGB+D+N+S is 58.75%. Direct generation of future depth gives AbsRel 0.28–0.33 versus 0.41–0.43 for RGB-then-expert, and segmentation mIoU 0.42–0.49 versus 0.22–0.24; normals are roughly equal at 0.76–0.77 cosine similarity, while RGB LPIPS changes only from 0.39 to 0.40–0.43. Over 100 paired trials, PAD-Depth and UWM time out on 88 and 86 trials, VLA-JEPA and MolmoAct hit IK failures on 48 and 22 trials, and XGenAct only records timeouts.
The authors note that the best training menu is task-dependent and more modalities are not always better: RGB+N+S outperforms RGB+D+N+S (61.25% vs 58.75%), and they call for broader evaluations and adaptive task balancing. The work is simulation-only on RLBench and ManiSkill3 with no real-robot validation, uses five held-out RLBench tasks for the external comparison, and does not mention code release. Fine-tuning a 5B video diffusion model is compute-heavy, and all results come from one action representation and one simulated manipulation setting.
Robot foundation-model developers—including Physical Intelligence, which produced the pi_0.5 baseline—as well as industrial manipulation vendors and humanoid robotics teams could use this approach to add geometry and semantics to world action models without custom heads. Simulation-only results make near-term use research or simulation-pipeline work; real-robot deployment is 1–3 years if the deterministic codecs transfer across cameras and embodiments and if real-robot closed-loop success is confirmed. Broad production use is 3+ years and would require larger multi-embodiment datasets and adaptive task-menu balancing.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.