World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories
This preprint, World Motion Models (WMMs), casts dynamic 3D scenes as collections of sparse SE(3) pose trajectories and trains a single flow-matching transformer to answer arbitrary known/unknown conditioning queries. Across separate checkpoints it reports 87.2% macro success on Language Table (vs 80.0% for LCB, the best prior), a 90.0% success on B2B, and faster/higher-success humanoid retargeting than offline PHUMA. It matters because the same architecture can serve as policy, forward dynamics model, planner, and cross-embodiment retargeter.
Can a single generative model over sparse SE(3) pose trajectories support any-to-any conditioning across future prediction, policy learning, model-predictive control, motion in-filling, and cross-embodiment retargeting?
Dynamic 4D world models are usually fragmented by entity type: humans, hands, objects, cameras, and robots use separate parameterizations and training pipelines, while video or dense point-track methods entangle 3D motion with appearance and scale poorly. As a result, no single prior cleanly supports the many inference directions needed in robotics, such as forecasting, inverse dynamics, planning, and retargeting.
Prior work includes morphology-specific generative models such as MDM, CHOIS, InterDiff, and OMOMO, dense point or video predictors such as TraceGen, and flexible sequence models such as Diffusion Forcing and masked diffusion. Diffusion Forcing supports per-token noise but focuses on temporal scheduling; class-specific models do not transfer across bodies, and dense observation methods are high-dimensional, pixel-tied, or fixed to specific morphologies.
WMM represents each moving entity as a trajectory of 9-D tokens encoding translation and rotation, with per-entity semantic and geometric features. It trains a flow-matching network under per-token noise levels so each pose token can be independently clean or noised; inference sets a mask to sample any conditional. A decoupled context-token timeline carries goals, exemplars, or identity labels, and the transformer uses five factored attention types with registers and AdaLN conditioning, scaling to 10K+ tokens.
On Language Table (50 rollouts per task, 200-step horizon), WMM achieves macro average 87.2% vs LCB 80.0%, LangTable original 77.0%, ActionVLA 76.8%, LAPA 62.0%; per-task B2B 90.0, B2AL 76.0, B2BRL 80.0, B2RL 90.0, Sep 100.0. MPC improves absolute tasks to 100% (B2B) and 80.0% (B2AL) from 90.0/76.0 plain. On TraceGen 3D trajectory prediction, WMM P=128 reports Droid MSE 0.151/MAE 1.129/endpoint 0.200 vs TraceGen 0.206/1.289/0.285, and EpicKitchen MSE 0.322 vs 0.445. On OMOMO human-object interaction, WMM gets MPJPE 12.07 vs CHOIS 15.30, foot sliding 0.84 vs 4.20, and C_F1 0.76 vs 0.67, though object translation error is worse (16.53 vs 12.53). On human-to-Unitree G1 retargeting, WMM reaches sequence success 0.8690 clean, 0.8525 in-fill, 0.8530 noisy; PHUMA 0.8255/0.6535/0.8060; GMR 0.7945/0.8095/0.7160, and WMM runs at 86.97 FPS vs PHUMA 2.27 and GMR 35.03. Ablations: removing per-token noise gives 0% B2B; plain DiT on TraceGen yields MAE 32.05 vs 2.167.
Authors state: trained one checkpoint per benchmark, no joint cross-domain model; SE(3) excludes scaling/shearing; requires geometry/depth/current-state input rather than direct RGB; no generalization to novel morphologies. Evident limitations include separate models and data splits per application, humanoid evaluation in physics simulation rather than physical hardware, slower inference than TraceGen on dense trajectories (4.18s vs 2.66s for 400 tracks/32 steps), and no reported code release.
Robot foundation-model startups and humanoid or teleoperation vendors could use a queryable motion prior to replace separate policy, planner, and retargeting stacks. Near-term (now/1–3 years), the approach is usable in research and simulation-based motion generation for manipulation and retargeting; production adoption requires joint multi-embodiment training, real-robot validation, and direct RGB or video input. In 3+ years, a cross-domain world motion model could become a component for general-purpose manipulation and humanoid control if trained on much larger trajectory data and verified safe on hardware.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.