UniWALM: Unified World Action Language Model for Embodied Robot Control
A robot policy that generates video, actions and its own short text notes in one model averaged 68.7% success on three real two-arm dexterous tasks, against 44.4% for MolmoBot, 43.7% for FastWAM, 33.7% for NVIDIA's GR00T N1.7 and 21.9% for pi0.5, researchers from Amazon, Brown and Northeastern report in a preprint. The written notes act as working memory: on a find-the-hidden-item task it scored 61.1% where the best baseline managed 33.3%.
If a video-action robot model also learns to write language, does that richer shared representation improve manipulation, and can the model use its own text as memory?
World-action models that predict future video and actions together are strong at short manipulation but have no explicit task state. Tasks that depend on what happened earlier (how many times a button was pressed, where an item was hidden) defeat them.
Prior systems bolt a separate vision-language model or planner onto a video-action backbone and run reasoning and acting as distinct stages, which adds latency and lets the stages disagree.
UniWALM puts video, action and language into one mixture-of-transformers backbone built on the Wan2.2 video model. Each modality has its own diffusion head and all three are denoised in the same pass, so the model writes the next subtask and a running memory string while producing the action. That text is fed back as input on the next step. A training trick called flow matching with correction teaches the action head to recover from perturbed actions, and LoRA keeps fine-tuning light. The language expert adds about 0.5B parameters; language and action parts are trained from scratch.
In simulation on 1,000 out-of-distribution episodes per MolmoSpaces task, UniWALM reached 68.0% on Pick&Place, 70.8% on Pick&Place-Color and 38.8% on Pick&Place-NextTo, ahead of MolmoBot with history (66.5%, 67.8%, 26.4%) and on average 14.8 points above FastWAM, which shares its backbone; it trailed MolmoBot on grasp-only Pick (49.2% vs 64.0%). Ablations on one Pick&Place split raised success from 31.0% (FastWAM-equivalent) to 35.9% with language, 38.9% with the correction training and 41.0% with LoRA. On RMBench's Press Button memory task it reached 51% after 100k steps where FastWAM scored 0%. On two Flexiv Rizon arms with Sharpa Wave or Paxini hands it scored 95.0% on cube pick-and-place (next best 80.0%), tied FastWAM at 50.0% on emptying a container, and reached 61.1% on the memory task over 18 trials.
Real-robot tests are small (20 rollouts per dexterous task, 18 for memory) with wide confidence intervals. The model is slower: a 32-step action chunk takes 123 ms with the language expert against 76 ms without, which may be too slow for high-rate control. It underperforms on short grasp-only tasks with tight step budgets. The authors list late or early task-completion signals, wrong subtask predictions on near-identical scenes and grasp precision on complex objects as failure modes. This is a preprint, not peer reviewed.
Amazon runs one of the largest robot fleets in warehouses and is testing dexterous picking; a single model that keeps its own task notes addresses multi-step jobs such as counting, checking and retrying that current VLA policies forget. For VLA developers such as Physical Intelligence and NVIDIA's GR00T team, the result argues that world-action models plus generated language can beat language-pretrained policies on real dexterous tasks. Commercial impact is one to three years out and depends on latency improvements.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.