DeltaWorld: Physically Consistent Interactive World Simulators via Action-Conditioned Latent Increment Learning
DeltaWorld is a preprint describing an interactive world simulator for robot manipulation that predicts action-conditioned latent increments instead of full future latent states. On a self-collected cross-robot dataset it reduces FVD by 46.6% and LPIPS by 31.1% relative to the IWS baseline, and on the IWS benchmark it improves FID from 44.16 to 36.04. This matters because more stable long-horizon action-conditioned video prediction could reduce real-robot rollouts for policy training and evaluation.
Can a world model predict robot-action consequences more faithfully by modeling latent state increments and interaction-focused supervision, rather than direct next-state prediction?
Existing action-conditioned world models predict entire next latent states; robot actions often change only small interaction regions while the background stays static, so models miss subtle object motion and accumulate errors over long rollouts, causing object drift, interpenetration, or excessive deformation.
Prior approaches such as IWS, LeWorld, and Cosmos3 learn action-conditioned dynamics by directly predicting future latent representations or pixels. This tends to emphasize largely unchanged scene content and underweight small robot-object changes, leading to drift and physical implausibility in autoregressive rollouts.
DeltaWorld uses a two-stage pipeline: first an autoencoder with a consistency decoder and Gaussian latent noise; then a Delta Latent Transition Model (Delta-LTM) predicts the difference between current and next latent state using 3D convolutions, FiLM action conditioning, spatial/temporal attention, and a residual update. Interaction-aware Latent Alignment constructs counterfactual scenes with the robot arm and object removed, uses the latent difference as a mask, and upweights prediction errors in interaction regions.
On the IWS benchmark averaged over 5 tasks and 2 camera views, DeltaWorld achieved FID 36.04, FVD 231.98, LPIPS 0.0490, PSNR 31.22, SSIM 0.9011, and MSE 0.0039, versus IWS baseline FID 44.16, FVD 264.52, LPIPS 0.0661, PSNR 28.80, SSIM 0.8537, and MSE 0.0054. On a cross-robot dataset of 4 tasks and 3 robot platforms, FVD fell from 168.98 to 90.31 (-46.6%) and LPIPS from 0.0440 to 0.0303 (-31.1%) relative to IWS. In ablations averaged over both datasets, removing Delta-LTM raised FVD from 191.50 to 228.03, and removing interaction-aware alignment raised FVD from 191.50 to 207.60.
The paper does not state explicit limitations. Evident constraints include evaluation only with offline video-prediction metrics, no demonstrated closed-loop policy training or real-robot policy evaluation, relatively small datasets (127–893 episodes per task), and no reported code or compute details.
Robotics companies building manipulation policies, world-model evaluators, or synthetic data pipelines could use this for simulation, checkpoint screening, and data augmentation. If validated on real-robot policy learning and evaluation, adoption could occur in 1–3 years for offline evaluation and data generation; production deployment for training policies would require larger real-world validation, faster inference, and higher-resolution multi-camera support.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.