Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

Can an agent-guided loop that separates task execution from scene recovery in a digital twin and real-world deployment improve manipulation robustness under perturbations and lower human intervention?

Manipulation policies often fail when a single mistake leaves the scene in a state the task policy cannot continue from; most training data contain only successful trajectories, so policies lack corrective behaviors. Retrying the policy does not resolve underlying obstructions, and exploring failures on hardware is expensive, risky, and reset-intensive.

Prior code-as-policy agents such as CaP-Agent0, RATs, and ASPIRE generate control programs or skill libraries, while VLA policies such as π0, π0.5, and MolmoAct2 map observations and language to actions, and separate recovery controllers like Recovery RL, RecoveryChaining, FARL, RACER, FailSafe, and ReSYNC address recovery in specific ways. These methods do not combine digital-twin skill discovery, outcome verification, separate task and recovery policies, a reusable recovery program library, and human demonstrations in a single deployment-learning loop.

Recova uses a coding agent to reconstruct the workstation as a MuJoCo digital twin from real recordings, then explores failures and generates successful task rollouts, recovery rollouts, and programmatic recovery skills. A language-conditioned recovery policy is separate from the task policy and conditioned on short recovery instructions. During real-robot deployment, a Gemini 3.8 Flash monitor checks completion and scene intactness, invokes a learned or programmatic recovery, verifies restoration against initial views, and requests human help only when no recovery succeeds. Across parallel workstations, successful tasks, human task demos, human recovery demos, and verified recoveries are routed to the appropriate dataset and used for DAgger fine-tuning.

On LIBERO-Pro, Recova reaches 78.8% mean success across six settings versus 71.7% for ASPIRE, a gain of 7.1 percentage points, with largest gains of 22.0 and 23.0 points on Goal and Spatial task perturbations. On MolmoSpaces, it reaches 64.9% mean across four categories versus 38.0% for RATs, a gain of 26.9 points; category scores are Pick 78.7, pick-and-place 40.5, open 56.5, close 83.9. On four real-robot tasks, the base task policy achieves 23.8%, DAgger fine-tuning raises it to 77.5%, and recovery skills raise it to 87.5%. Over four DAgger rounds on drawing a mahjong tile, human intervention falls from 87.5% to 0%, and the fraction of episodes completed by the policy rises from 12.5% to 85.7%. In a 14-minute four-station session, the task policy completed 22 of 31 episodes and human control accounted for 11% of station time.

The authors note that newly learned recovery skills are not automatically registered or shared across tasks; real-robot evaluation covers only four tasks on one bimanual workstation type; simulation baselines are taken from prior publications rather than re-run; and the digital-twin construction still relies on recorded trajectories and task-object models. The paper does not report a code release.