Value as Memory: Online Value-Guided Post-training for General Robot Policies

Can a robot policy's own running estimate of task progress be used as an input, not only as a training score, to resolve situations that look the same but need different actions, and can online reinforcement learning on a real robot make post-training of large VLA policies faster and more sample-efficient?

Generalist robot policies are fine-tuned after deployment on data from their own attempts and from human corrections. Two problems slow this down. First, a camera frame alone often does not reveal how far a task has progressed, so a folded and a half-folded shirt can look alike while needing different next moves. Second, most real-world post-training runs offline in batches: collect, stop, retrain, redeploy. Feedback from new failures arrives late and each policy iteration is expensive.

Physical Intelligence's RECAP method (used for its pi0.6* model) learns a value function from demonstrations, rollouts and interventions and uses it to label good and bad actions for advantage-conditioned training. In that design the value estimate shapes training data but is never shown to the policy at run time, and the critic is refreshed only between offline training rounds.

VaM (Value as Memory) adds a value cache: the last few progress predictions of a separate value model are fed to the action model alongside camera images and joint states, so the policy can tell whether progress is rising, stalled or falling. Training moves through four stages, from demonstrations to offline RL warm-up, then online DAgger with human corrections, then autonomous learning from sparse success signals. Data collection, value training and policy training run concurrently. The action model is initialised from Physical Intelligence's open pi0.5 (a 2B-parameter PaliGemma backbone plus a 300M action transformer).

On box assembly, laundry folding and magnetic toy-bus assembly with one real robot, VaM reached 96.7%, 88.9% and 76.7% success over its final three checkpoints. Without the value cache the same pipeline scored 81.7%, 48.9% and 43.3%; RECAP scored 71.7%, 64.4% and 43.3%. Against matched offline RL, online training gained 32.8 points on average and cut wall-clock time per gradient step by 57.8%; on box assembly it ran 16 policy iterations versus 2 for the same gradient budget. Swapping in older value models at inference lowered success on all three tasks, showing the cache only helps when the value model tracks the current policy.

The authors state that the method needs a value model that tracks real progress closely, which makes the offline warm-up stage necessary, and that all experiments used a single robot, so fleet-scale behaviour is untested. We add that three tasks with a few dozen evaluation episodes each leave wide error bars (laundry folding without the cache varied by plus or minus 30 points), that human interventions remain part of the pipeline before the autonomous stage, and that the code is announced but not yet released.