TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
TS-WAM, a preprint from the Hong Kong University of Science and Technology (Guangzhou), lifted real-robot task success to 72.5% from 55.0% for Physical Intelligence's pi0.5 and held 62.9% against 35.0% when lighting, layout or table texture changed, across 20 trials per task on a Franka arm and a two-armed humanoid. The 1.2-billion-parameter policy learns partly from human videos that carry no robot actions by first predicting how the scene should change, then acting on that prediction.
Can a vision-language-action policy become more robust if it is trained to predict the visual effect of its next move, using human videos without action labels alongside robot demonstrations, in a single training run?
Most vision-language-action (VLA) policies map a camera image and an instruction straight to motor commands learned from robot demonstrations. They hold no explicit guess about what their action will do to the scene, cannot learn much from the far larger supply of human videos that lack robot actions, and tend to overfit the exact conditions of the demonstrations.
Latent-action methods such as LAPA and UniVLA compress the change between video frames into a compact code learned from video, and world-model VLAs such as WorldVLA and VLA-JEPA predict future states. Most of them pretrain the latent-action or world model in a separate stage and then fine-tune it into a robot policy, which adds compute and does not guarantee that the learned code helps the final action decoder.
TS-WAM trains everything end to end. A student vision-language model sees only the current image and instruction and predicts a latent action; a teacher copy, updated as a moving average, also sees the next frame and supplies the target, a joint-embedding setup the authors call vlm-jepa. A small decoder then predicts how frozen DINOv3 image features will shift under that latent action, and a flow-matching action head acts on the predicted future features. Human clips train the latent-action and prediction parts; robot clips also train the action head. The teacher is dropped at deployment.
In the real-world tests (20 trials per task and condition) TS-WAM averaged 72.5% against 55.0% for pi0.5 on two Franka Research 3 tasks and two tasks on a RevoTron bimanual humanoid with dexterous hands, and beat pi0.5 in all 12 out-of-distribution combinations (62.9% against 35.0% on average). Its success fell 9.6 points from normal to changed scenes, against 20.0 points for pi0.5. In simulation it reached 97.0% on LIBERO (pi0.5 96.9%), 63.8% on SimplerEnv (RoboVLMs 51.4%) and 75.7% zero-shot on LIBERO-Plus with 1.2 billion parameters, ahead of VLA-JEPA's 62.9%. Removing the human videos cut LIBERO-Plus to 64.8%; removing the teacher-student setup cut it to 61.0%. It runs at 30 Hz on one RTX 4090 after training on four A100s.
This is a preprint. The authors say the teacher branch raises training memory and that conditioning the action head on predicted features slows convergence. On standard LIBERO the gain over pi0.5 is 0.1 point, so the case rests on robustness tests. The real-robot comparison uses one baseline, pi0.5, at 20 trials per cell, and the paper does not state how many hours of human video were used or whether code will be released. The decoder predicts only one step ahead, so it is not a long-horizon simulator.
The result matters to anyone building VLA policies for factories and homes, where lighting and clutter never match the demonstration set: Physical Intelligence, Figure, Agility and Chinese humanoid makers all face it. A policy that tolerates scene changes with about a sixth of a 7B model's parameters and learns from cheap human video points to lower data and compute costs per deployed task. Expect the idea of predicting the next scene inside the policy to spread into commercial stacks within a year or two, once it is confirmed on more robots and longer tasks.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.