Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
This preprint builds one video model, P3-WAM, that can read 3D hand motion out of a first-person video, predict what a scene will look like after a given hand movement, and plan both the movement and the resulting video from a text instruction. Training the three skills together made each one better, and it beat specialised models on four public datasets, though robot use so far is two open-loop demonstrations.
Does training perception (recovering hand and camera motion), action-conditioned prediction and goal-directed planning in a single world action model improve all three compared with training each alone?
Robot learning from human video needs three abilities that are usually separate models: a hand-pose estimator, an action-conditioned video predictor and a planner. Separate models do not share what each learns about how hands and objects interact, and video generators without explicit 3D actions drift from the motion they are asked to show.
Hand reconstruction used per-frame or video estimators such as HaMeR and Dyn-HaMR. Action-conditioned world models such as DreamDojo, Hand2World and EgoSim, and general video models such as Wan2.2 and NVIDIA's Cosmos 3, predict or generate video but do not also output a structured 3D action plan.
P3-WAM adds a compact 3D representation of hand and camera trajectories to a pretrained video diffusion transformer (Wan2.2-Fun A14B-Control) through a mixture-of-transformers design with shared weights, modality-specific adaptation and two-way attention between video and actions. During training, video and actions are randomly hidden, so the same model learns to infer actions from video (Perceive), video from actions (Predict), or both from an image, an instruction and an optional goal image (Plan).
Against single-mode training at matched steps, unified training cut camera-space hand error by 23.5% and lowered prediction FVD from 1,106 to 1,010 averaged over EgoDex, ARCTIC, HOT3D and EgoLive. On EgoDex, camera-space hand error was 32.9 mm against 112.9 mm for HaMeR and 115.5 mm for Dyn-HaMR. In action-conditioned prediction it had the best full-frame results on all four datasets; average PSNR rose from 15.9 to 19.0 dB and FVD fell from 1,218 to 622 versus its own backbone. In planning, a blind Qwen VLM judge scored its videos 3.74 out of 5 for instruction adherence against 3.12 for the best Wan2.2 or Cosmos 3-Nano setting.
The authors note that global 3D hand position from a single camera is ambiguous and that robot execution is open-loop, so the robot cannot correct itself. The robot evidence is two qualitative examples after retargeting with no success rates. Planning quality is judged by a vision-language model rather than by task completion, and several baselines were run with their own conditioning protocols.
World action models are where NVIDIA (Cosmos), Alibaba (Wan) and humanoid labs are converging: a model that both imagines outcomes and outputs actions can generate training data and check plans before a robot moves. This paper shows that coupling explicit 3D hand actions to a large video model improves fidelity, which matters to companies building data engines from human video. It is research-stage; closed-loop robot results are the milestone to watch, likely 12 to 24 months out.