Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models
PEWAM is a preprint describing a 71.5M-parameter world-action model that compiles robot demonstrations into editable “effect programs”—3D object keypoint trajectories, grasp contact points, and final scene configuration—rather than actor-specific motions. On held-out LIBERO-Goal tasks, one program from a single demonstration completes 40 of 90 episodes, while the same backbone under goal-image or language conditioning and prior demonstration-conditioned methods complete at most 19. This matters because turning demonstrations into coordinate-based programs enables editing and cross-robot-arm reuse, a potential route to more flexible robot programming.
Can separating a demonstration's object-level effects from the actor's specific motion into a coordinate-based program make learned manipulation policies more editable and more transferable across tasks and robot arms?
Robot demonstrations bundle what happened to objects with how one particular arm made it happen, so systems that condition on raw demonstrations or goals inherit actor-specific behavior and are difficult for users to edit or transfer across embodiments.
Prior methods train policies conditioned on goal images, language instructions, or published demonstration-conditioned representations, without exposing a separable effect representation. In the abstract's LIBERO-Goal evaluation, these methods complete at most 19 of 90 held-out episodes.
PEWAM compiles each demonstration into an effect program containing 3D keypoint trajectories of moved objects, two contact points showing where each object was held, and the final scene configuration with the demonstrator removed. The 71.5M-parameter model generates four parallel streams—effect, robot execution, action, and terminal state—using independent flow-matching times; clamping the effect program and sampling execution makes inference a closed-loop planning step from the live scene. Because the program is a set of coordinates, a person can edit it by shifting terminal states or rotating contact points.
Held-out LIBERO-Goal: 40 of 90 episodes completed from one demonstration's effect program, versus at most 19 of 90 for the same backbone with goal images or language and for published demonstration-conditioned methods. On Meta-World, PEWAM exceeds the best published results on three held-out classes but not on the five-class mean. The abstract also reports the same program running on four robot arms.
The authors note Meta-World gains are partial: better than the best published results on three classes but not on the five-class mean. On LIBERO-Goal the system still succeeds in fewer than half of episodes (40/90). The abstract does not report real-robot trials, peer review, code release, training cost, or perception-noise robustness.
Robot-arm OEMs, warehouse and assembly automation vendors, and imitation-learning startups could use editable effect programs to let users reprogram a robot from one demonstration and reuse the same program across different arms. This is more likely a 1–3 year opportunity: real-robot validation, higher held-out success rates, and evidence that coordinate edits remain robust under perception noise would need to come first.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.