Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation
Mem4Act, a preprint from UCLA and North Carolina State University, gave a two-armed robot a compact memory and lifted success on four real long-horizon block tasks to 81.5%, against 28.0% for the pi0.5 model it was built on and 42.5% for GPT-6 Astra used as a zero-shot planner, over 50 trials per task. It also led two memory benchmarks, with 49.9% on RoboMME and 61.0% on RMBench.
How should a vision-language-action policy remember what it saw minutes earlier, such as a block's colour before it was covered, without feeding an ever-longer video history into its model?
Most VLA policies act on the current camera frame. In long tasks the information that decides the next move often disappeared from view long before, for example which block sat under which cup. Feeding the whole history in makes the model attend over a growing pile of image tokens, which costs compute and can confuse it; compressing by picking key frames, dropping tokens or writing language summaries can throw away the detail that matters later.
Earlier memory methods keep raw past frames, compressed visual tokens or text summaries (MemER, MME-VLA variants such as FrameSamp and TokenDrop, Mem-0, NativeMEM, HAMLET). They either grow with episode length or risk losing timing, meaning or visual detail, and nothing forces the policy to actually rely on the memory it is given.
Mem4Act keeps a fixed-size latent memory of 512 tokens, updated through the episode. A three-layer encoder groups past frames at coarse to fine time scales (windows of 32, 16 and 8). Two training signals decide what memory keeps: questions and answers about past events, answered through the language model, and reconstruction of image features from key frames, for visual detail. Training runs memory first: the memory is pretrained with the policy frozen, then trained jointly with the action policy at a tenth of the learning rate, which the authors show makes the policy depend on memory more.
On YAM dual arms with 70 teleoperated demonstrations per task, Mem4Act scored 78%, 70%, 100% and 78% on covering, swapping, rearranging and putting back blocks, an 81.5% average, against 42.0% for MME-VLA, 28.0% for pi0.5 and 22.5% for Mem-0. GPT-6 Astra reached 42.5% but needed about 19 seconds per decision and 17 minutes per successful run, against 335 milliseconds and 40 seconds for Mem4Act. In simulation it led RoboMME at 49.88% (best baseline 44.51%) and RMBench at 61.0% (NativeMEM 48.7%), where on the tasks needing several past observations it scored 41.3% against 14.8%. Memory size matters less than structure: 1,024 tokens did worse (42.75%) than 512.
This is a preprint. The real-world tasks are block puzzles adapted from RMBench on one robot, and only head-camera history goes into memory. Each decision takes 335 milliseconds, about 2.6 times pi0.5's 131 milliseconds, on the same GPU. RoboMME success remains under 50% overall and 31.5% on spatial-permanence tasks, where MemER does better (53.2%). Some baselines are the authors' reimplementations, and GPT-6 Astra was run zero-shot with time limits, so it is not a like-for-like comparison with trained policies.
Memory is the gap between demo tasks and real work: assembling a kit, restocking a shelf or cleaning a room means remembering what was done and what was hidden. A bounded memory that bolts onto pi0.5 and nearly triples its real-robot success on such tasks matters to Physical Intelligence's partners and to humanoid makers chasing multi-step jobs. Expect memory modules of this kind in commercial VLA stacks within one to two years, provided latency falls and results hold beyond block tasks.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.