UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning

Can an intervention agent learn a unified control-allocation policy that adapts as the RL policy improves during online reinforcement learning?

Real-world online reinforcement learning is sample-inefficient and can be unsafe or unproductive under sparse rewards, while human-in-the-loop methods demand continuous skilled supervision to time and execute corrections. Existing automated intervention methods rely on offline estimates or fixed decision rules, so their assistance can become mismatched to the policy's changing competence during training.

HIL-SERL uses a human operator for corrections during off-policy reinforcement learning; AutoSERL automates recovery from a single demonstration; UniIntervene learns intervention behavior offline and triggers goal-conditioned recovery when sustained value degradation is detected. These methods keep intervention timing or recovery modules static and often still leave routine corrections or monitoring to a human operator.

UniIntervene++ represents the evolving RL policy, demonstration-guided trajectory correction, and a task-structured CodePolicy as Options in a unified semi-Markov decision process and trains a Double DQN scheduler online with availability masks, duration-aware rewards, and costs for using assistance. An Adaptive RL Probe periodically forces unassisted RL execution to measure the policy's current competence and adjust the probe length, while coupled experience learning routes assisted and human-corrected transitions into the demonstration replay buffer so the RL policy improves from intervention data.

Across five real-world UR7e manipulation tasks, UniIntervene++ averaged 89.67% success versus 83.67% for HIL-SERL and UniIntervene, and 73.67% for AutoSERL. Human intervention rate was 0.77%, compared with 54.03% for HIL-SERL, 14.29% for AutoSERL, and 18.84% for UniIntervene, a 94.6% relative reduction from the best baseline. Per task, it outperformed all baselines on Ring Transfer by 3.33 points, Tube Insertion by 6.67 points, USB Insertion by 6.67 points, and Towel Folding by 5.00 points; on Wipe Whiteboard it reached 86.67%, 3.33 points below UniIntervene. Overall intervention rate was 56.04%, with 95.62-100% of intervention steps automated. Ablations on Ring Transfer and Towel Folding gave 0%/0% for SERL, 0%/0% without CodePolicy, 50%/47% without trajectory correction, 75%/80% for a fixed-prior scheduler, and 88.33%/85% for the full method. As Ring Transfer RL policy success rose from 0 to 88.33%, RL control share increased from 35.78% to 90.51%.

The authors state that the independent contribution of Adaptive RL Probe is not isolated in the ablations. Evaluation uses one UR7e arm, five tasks, and task-specific demonstrations, with no reported run-to-run variance. The overall intervention rate remains 56%, and CodePolicy depends on object pose estimation and task-specific primitives, so transfer to new tasks requires additional perception and skill engineering.