SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation

Can a sparse 3D skeleton representation of robot–object structure support effective world-action learning while substantially reducing model complexity?

Existing world-action models predict future videos or visual latents, which encode manipulation-relevant geometry only implicitly and may retain appearance information unrelated to control; VLA and world-model baselines often require billions of parameters and are sensitive to camera/viewpoint changes.

Prior methods include visuomotor policies such as ACT, Diffusion Policy, and DP3, VLA models such as RT-2, OpenVLA, π0, and π0.5, and world-action models such as WorldVLA, Fast-WAM, VLA-JEPA, GE-Act, and Cosmos-Policy. Their future-state modules typically reconstruct pixels or predict learned visual latents, which do not explicitly isolate the joint/object configuration governing contact and motion, and they are often large, around 2–7B parameters.

SkeleWAM builds a current skeleton online from RGB-D and robot proprioception: forward kinematics gives robot joint and end-effector keypoints, while a frozen pretrained perception network estimates object centers and interaction points. A two-expert mixture-of-transformers processes current skeleton tokens, noisy action tokens, and noisy future skeleton tokens with a block attention mask; both action and future-skeleton branches use flow matching, with future skeleton prediction as an auxiliary training objective only. At inference the future branch is dropped and actions are generated from the current skeleton plus language instruction; Medoid Action Consensus samples K=3 action chunks and selects the one with smallest mean pairwise distance over the first 10 motion steps. Training runs for 60K steps with effective batch size 48 on one RTX 4090, predicting 32-step action chunks and 8 future skeleton states; execution replans after 16 actions.

On LIBERO-Plus with 10,030 variants across seven perturbation categories, SkeleWAM scores 85.9% overall with 57.1M parameters versus Cosmos-Policy 82.2% at 2B params, π0.5 79.7%, GE-Act 80.3%, VLA-JEPA 77.9%, Fast-WAM 51.5%, and OpenVLA 15.6%. Under camera perturbation it scores 93.4% vs Cosmos-Policy 75.8%, a +17.6 point gap; it scores 89.1% on language, 96.0% on background, and 93.9% on noise perturbations. Layout is a weakness at 66.6% vs π0.5 84.1%. The privileged sim-state variant scores 87.7% overall with 51.4M params, leaving a 1.8-point gap to the RGB-D version. Real-robot evaluation on an ARX R5 with 20 trials per task: SkeleWAM averages 89% vs Cosmos-Policy 87%, Fast-WAM 83%, and π0.5 76%; per-task SkeleWAM results are open drawer 80%, close drawer 90%, stack blocks 90%, stack bowls 95%, and put block in drawer 90%. Ablations on LIBERO-Plus RGB-D: centers plus interaction points 85.9% vs centers only 82.3% and interaction points only 77.7%; action plus future skeleton 85.9% vs action only 80.1%; no cross-branch attention 85.9% vs S+→A 85.6% and A→S+ 84.8%; execution horizon 16 steps 85.9% vs 5 steps 80.7% and 32 steps 81.8%; MAC K=3 85.9% vs K=1 84.2%, K=5 85.2%, and K=10 84.1%; consensus window h=10 85.9% vs h=3 85.3%, h=5 85.4%, and h=16 85.7%.

The authors note layout generalization remains limited: 66.6% overall layout success with RGB-D skeletons and 69.0% even with simulator-derived coordinates, indicating sparse geometry alone does not fully capture large spatial layout changes. The paper is a preprint with no stated peer-reviewed venue. Real-world evaluation is limited to one ARX R5 arm, five tabletop tasks, and 20 trials per task; object landmark estimation relies on a fixed perception network, and no explicit code release is stated beyond the project page. Results may not transfer to clutter- or occlusion-heavy, high-mix environments.