Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

Can target articulated robot configurations rendered as fixed-view images provide a single shared action representation that supports forward dynamics, inverse dynamics, and joint generation across heterogeneous robot embodiments without learned embodiment-specific action decoders?

Joint-space action vectors vary in dimension across robots and lack explicit spatial structure, while end-effector visual representations encode only end-effector pose and gripper state, not full arm configuration, so execution needs embodiment-specific mappings to joint targets that can have singular or no solutions. Previous camera-aligned robot renderings require physical-camera extrinsic calibration. This makes it hard for one video-action model to directly control multiple robots with different kinematics and camera setups.

Prior families include unified video-action models such as UVA, UWM, Pelican-Unify, Ctrl-World, Motus, Fast-WAM, and LingBot-VA; standardized vector action interfaces like RT-style policies; latent and visual-motion actions; latent-frame actions such as Cosmos Policy; and end-effector image interfaces such as SpatialVAM and Action Images. Camera-aligned URDF conditioning appears in BridgeV2W and GeniWorld. Their shortcomings are per-embodiment learned heads or decoders, loss of full joint configuration, inverse-kinematics singularities, or dependency on calibrated observation camera geometry.

Dream4ACT converts current and target joint configurations into four fixed virtual-camera action views using URDF forward kinematics and rendering; camera presets are fixed across embodiments, giving a constant tensor shape. A frozen Qwen3.5-VL 9B vision-language model with a trainable adapter compresses head-camera image and instruction into semantic conditioning. A shared Wan2.2 VAE tokenizes RGB and action-view sequences, and a diffusion transformer with modality-specific noise levels and masked flow matching jointly models them, with masks selecting forward dynamics, inverse dynamics, or joint generation. Joint targets are recovered by training-free URDF-constrained Gauss-Newton optimization over rendered candidate configurations, using binarized/dilated views scored by IoU plus one-sided Chamfer distance and warm-started from the previous state.

On RoboTwin 2.0's full 50-task suite using a separately trained checkpoint, Dream4ACT achieves 90.50% clean and 87.46% randomized success, averaging 88.98%, above pi0.5's 79.80% (+9.18 avg) and Motus's 87.80%, but below LingBot-VA's 92.20% (-3.22) and Fast-WAM's 91.80% (-2.82). On TriWorldBench's 500-episode test set it scores 65.66 overall versus BWM's 65.54, with P3D 61.30, MQ 41.66, and TC 64.88 among listed methods highest, and VQ 31.42 tied with BWM. A jointly trained multi-embodiment checkpoint on 31 shared sim tasks averages 82.19% for Aloha-Agilex, 81.94% for Piper, 81.23% for ARX-X5, 63.48% for Franka-Panda, and 30.97% for UR5-Xsg. On real robots with 20 trials per task, two common tasks average 87.5% for Aloha-Agilex, 85.0% for TienYi2.5 Pro, 87.5% for Franka, and 90.0% for UR5e; bimanual four-task averages are 61.3% and 66.3%. A DROID ablation over 50 held-out trajectories gives action-view conditioning PSNR 24.33 dB vs 23.19 dB for camera-aligned skeleton, SSIM 0.906 vs 0.899, and LPIPS 0.093 vs 0.105.

The paper does not state an explicit limitations section. Evident limitations include low simulation success on UR5-Xsg (30.97%) and Franka-Panda (63.48%), weak real-world bimanual stack and storage performance (20–65%), and benchmark results that use separately trained checkpoints rather than a single unified checkpoint for all settings. Real-world evaluation is only 20 trials per task across four platforms and a few tasks, no code-release claim is made, recovery is an optimization loop whose latency is not reported, and the approach depends on having accurate URDF models for each embodiment.