Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems

When a vision-language-action (VLA) robot system is said to use memory, does the stored content actually drive its decisions, and how can that be tested?

Memory modules are being added to VLA policies so robots can handle long tasks, but the usual evidence is an ablation: switch memory off and compare success rates. That changes information and computation at the same time, and an aggregate rate can hide gains on some episodes offset by losses on others.

Earlier memory-augmented robot policies were judged mainly by task success with and without the memory bank, on benchmarks where the needed fact is often still visible or recoverable from the current observation.

The authors propose IMEP, an interventional protocol that pairs episodes by seed and edits memory at the retrieval interface: swapping the key fact, replacing it with a different record that keeps the fact, nudging a number by one unit in the last place, or inserting an exact copy. Decision correctness is scored separately from motor execution. They also build ChainBench, where a robot sees which drawer holds an object, then several distractor episodes, and must later choose between closed drawers that look identical; a frozen pi0.5 policy then executes the chosen drawer.

On ChainBench (202 chains, two choices, 50% chance), no memory gave 53.0% correct decisions and 39.1% task success; shuffled records gave 51.5% and 38.6%; full memory chose correctly on every chain, 100.0%, and reached 74.3% task success, the same as the oracle under shared recorded executions (plus 35.2 points, 1.90 times, MEKAWIRE calculation). In learned-bank experiments, a one-unit-in-the-last-place change, with its installation procedure, flipped 11.9% of seed-paired outcomes, while swapping the key fact produced no clear extra effect over a fact-preserving replacement. In Shell Game with the camera blacked out, memory reads added 4.0 points (73.5% vs 69.5%), below the authors' preregistered 10-point bar, and a trained memory plugin lowered success from 18.4% to 10.2% when its bank was kept.

This is a preprint and all experiments are in simulation, including SimplerEnv tasks. The authors say the learned-bank studies lack decision readouts and content positive controls, so small fact-specific effects stay unresolved, and results are not shown to generalise across checkpoints or implementations. ChainBench's full-memory reader uses a curated latest-write set and no decision-time language model, so the 74.3% shows that correct history can help rather than that any deployed memory system delivers it.