Evaluating memory in VLA robot policies: interventional protocol and ChainBenchExpert
Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems
When a robot had the right history to look up, a memory system lifted task success from 39.1% to 74.3% on a new benchmark that cannot be solved from the current camera view, matching an oracle that always picks correctly. The same preprint, from Hainan Bielefeld University of Applied Sciences, TU Munich and TU Berlin, shows that higher success rates alone do not prove a robot's memory is using the right facts.