Memory before Goal: Role of Pre-History Memory for Humanoid Loco-Manipulation
Giving a humanoid's control model a short, well-chosen memory of what it saw earlier raised a Unitree G1's success on six real tasks from 36.7% to 88.3% (53 of 60 trials), in a preprint from MBZUAI. The robot had to act on cues, such as which can a person touched, that were no longer in its camera view.
Can a whole-body vision-language-action policy for a walking humanoid act correctly on events it saw before it turned towards its goal, and which past frames does it need to keep?
Humanoid loco-manipulation is non-Markovian: an object is hidden, moved or left behind as the robot turns and walks, so the cue that decides the right action is often gone from the head camera by the time the robot acts. Current whole-body VLA policies condition only on the present image and a short proprioceptive history.
Whole-body policies such as GR00T, pi0.5 adapted to whole-body actions, WholeBodyVLA and Psi-0 act on the current observation. Memory methods so far were built for tabletop arms that do not walk and usually keep frames by uniform sampling, retrieval or simple events, and existing humanoid benchmarks keep the goal in view.
WholeBodyMem uses the SAM3 segmentation model only as a tracker of the task objects. Object events, measured after removing the robot's own camera motion, select key frames within an adaptive budget; the action expert (six diffusion-transformer blocks) reads those frames through cross-attention together with object tokens that encode appearance, geometry and time. The authors also release HuMeGo, a memory benchmark with six simulated and six real Unitree G1 tasks that test what, where and when memories.
In simulation the base policy Psi-0 rose from 29.1% to 72.6% success on the memory tasks, and on the standard SIMPLE loco-manipulation suite from 70.2% to 86.9%, beating memoryless baselines (InternVLA-M1, Diffusion Policy, ACT, EgoVLA, pi0.5, GR00T) and a uniform-memory variant. On a real G1 with Dex3 hands, a head-mounted RealSense D435 and the SONIC whole-body controller, WholeBodyMem succeeded in 53 of 60 trials (88.3%) against 22 of 60 (36.7%) for Psi-0, after 15 teleoperated demonstrations per task. Inference ran on one RTX 5090.
The authors note that SAM3 misses a cue when the object is never detected, that online tracking adds inference cost, that rendering leaves a sim-to-real gap and that the simulated demonstrations are scripted. We add that the real-robot test is small (10 trials per task, 90 demonstrations in total), the policy runs off-board on a desktop GPU, and the tasks are short, staged scenes rather than long shifts in a factory or home.
Humanoid makers selling into logistics and retail need robots that remember what happened a few seconds ago, such as which tote was already emptied or where a person pointed. The result suggests that frame selection, not bigger context windows, is the cheap lever, which matters for Unitree G1 developers, NVIDIA GR00T users and anyone building on pi0.5. Expect this kind of memory module in research stacks within a year; on-board deployment depends on cutting the tracking cost.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.