Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation
This preprint introduces ViGAR, a hierarchical manipulation model that separates visual subgoal planning from subgoal-conditioned world-action generation, using a shared pretrained Cosmos3-Nano world model. On the RoboTwin Clean2Random benchmark it reaches 82.00% Clean and 67.02% Random success, 12.86 points above the best baseline average. It matters because image-based subtask goals allow the same policy to be steered to new task compositions via a global goal image without retraining.
Can world-action models be reorganized around explicit visual subgoal prediction to improve long-horizon compositional manipulation and enable inference-time task steering?
Long-horizon manipulation tasks require policies to sequence multiple coordinated subtasks, but existing world-action models mostly predict short-term visual futures and actions and do not expose an explicit task-level interface; language-only subtask plans can underspecify spatial outcomes.
Earlier methods either map instructions directly to actions with vision-language-action policies or add text subgoals from a vision-language model to world-action policies. Recent in-context manipulation approaches condition on demonstration videos. These approaches often treat subgoal information implicitly or as a separate planning output rather than as a pixel-level condition for the downstream action model, and language subgoals can be spatially ambiguous.
ViGAR uses two stages: a visual subgoal planner predicts the next target image from the current view and global instruction, optionally conditioned on a global goal image for in-context steering; a subgoal-guided world-action policy jointly generates future frames and action chunks conditioned on that image. Both components are initialized from Cosmos3-Nano, and training uses flow matching with a subgoal look-ahead rule and end-effector region-of-interest weighting. The subgoal image is injected as VAE latents into the generator branch rather than through semantic tokens.
On RoboTwin Clean2Random, ViGAR achieved 82.00% Clean and 67.02% Random success, average 74.51%, compared with the best baseline 4D-WAM at 81.50/41.80/61.65; a controlled Cosmos3-Nano-RoboTwin baseline without subgoal guidance scored 77.06/22.46/49.76. RoboTwin subgoal-quality metrics for ViGAR were LPIPS 0.13, DINO-cos 0.80, ΔIoU 0.65, and RetAcc 0.94, versus VISTA at 0.38/0.54/0.22/0.82 and RxBrain at 0.73/0.19/0.18/0.79. On five real-robot tasks with an AgiBot A2 (20 rollouts each), ViGAR outperformed π0.5 and Cosmos3-Nano-Policy in average task progress, but exact per-task values are not stated in the text. In two real-robot in-context tasks, average task progress was 77.5% for in-domain global goals and 47.5% for out-of-distribution goals. Ablations over 50 RoboTwin tasks showed generator-route goal injection reached 62.17 average success versus 50.97 for reasoner-route and 49.18 for none; removing both subgoal look-ahead and end-effector ROI supervision reduced ViGAR from 74.51 to 62.54 average success.
The paper does not provide an explicit limitations section. Evident constraints include simulation training on only 2500 Clean demonstrations and evaluation on one benchmark; real-robot evaluations have small trial counts of 20 per task on a single AgiBot A2 robot, and exact per-task progress scores are not fully reported; the method depends on a 16B-A8B pretrained Cosmos3-Nano backbone, implying significant compute; and out-of-distribution in-context steering reaches only 47.5% average progress. Peer review status is unconfirmed as an arXiv preprint.
Robotics companies building long-horizon manipulation systems for logistics, food service, or household robots could use explicit visual subgoal interfaces to improve robustness to scene changes and to let customers specify tasks by goal image instead of retraining. AgiBot-style bimanual manipulation providers and warehouse or kitchen automation vendors are plausible adopters. Near-term use within 1–3 years is possible if inference latency and compute cost are reduced on smaller world-model backbones; broader deployment in 3+ years would require evidence across more embodiments, larger real-world trials, and improved reliability and safety in unstructured settings.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.