ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

When a deployed robot must pick up new tasks one after another, can the reinforcement learning done for earlier tasks be reused to learn later ones faster, without the robot losing what it already learned, even as the underlying vision-language-action or world-action model is itself retrained with new demonstrations?

Robot foundation models such as VLAs and world action models (WAMs) work broadly but are imperfect on a specific line, and reinforcement learning (RL) after deployment can close that gap. Most RL post-training is a one-off for a single task. A cell that gets new parts and product variants would have to restart RL for each one and pay the full robot time again. Carrying a learner across tasks risks forgetting earlier skills, and keeping every past interaction becomes too expensive as tasks pile up.

Continual RL methods for VLAs such as Simple Recipe update the model with on-policy RL, which needs fresh rollouts for every update and gives little transfer to new tasks. Off-policy add-ons such as RL Token and EXPO-FT attach a small actor and critic to a frozen foundation model and are sample-efficient, but only within one task. Standard continual-learning tools, including experience replay, EWC weight penalties and parameter merging (RETAIN), either forget or slow down new learning.

ACCRUE keeps the foundation model as a proposer: at each stage it is fine-tuned on all demonstrations seen so far, then frozen, and it outputs a reference action chunk for each step. A lightweight actor refines that chunk and a critic scores it, both reading features from a fixed off-the-shelf vision encoder (DINOv3 in simulation) so their inputs do not change when the foundation model is retrained. Both networks are carried into the next task instead of being reset. The critic is trained only on the new task's transitions, because a small memory cannot keep its value estimates accurate on old tasks. Old behaviour is protected differently: the last N episodes of each finished task are kept (20 to 100 in the main runs), and the actor is distilled toward what the previous stage's actor output on those stored inputs. A penalty keeps the actor close to the foundation model's proposal, so RL performs a local correction rather than a search over the whole action space.

In simulation on NVIDIA's Isaac Lab FORGE tasks (peg insertion, gear meshing, nut threading) with GR00T N1.7 as the base model, ACCRUE reached 0.917 final accuracy across all tasks against 0.685 for plain sequential training and 0.626 for experience replay, with final forgetting of 0.068 against 0.409 and 0.516. Negative backward transfer fell to 0.065 from 0.217 for the best-retaining baseline, EWC, which in turn learned new tasks much worse (0.593 forward transfer against ACCRUE's 0.823). On five-stage AutoMate plug-and-socket sequences, final accuracy was 0.863 against 0.779 for replay, and forgetting 0.054 against 0.176. On real hardware (the DROID setup, with Physical Intelligence's pi0.5 as base model fine-tuned on 5 or 10 teleoperated demonstrations per task), inheriting the actor and critic cut robot interaction time to reach 95% success by 31.0% to 68.5% per stage across a three-stage NIST board sequence and a five-stage AutoMate sequence, and every earlier task still succeeded on at least 18 of 20 trials. Retaining 100 to 200 episodes per task, about 3% of the replay buffer, worked best; 10 to 20 were too few, and 200 was slightly worse than 100.

This is a preprint and has not been peer reviewed. The authors say sequences are at most five stages, memory is bounded per task rather than in total, and all tasks are closely related contact-rich assembly; transfer across embodiments or very different tasks is left open. On hardware, they compared only against single-task RL because a full baseline sweep was not feasible, and success was labelled by an operator. In our reading, the real-robot tasks start with the part already held in a custom 3D-printed gripper and use only joint positions as state, which removes grasping and object localisation from the problem; and the gains are relative to resetting RL, not to a stronger base model that might need no RL at all.