SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation
SPEAR, a preprint from the University of Hong Kong, uses an open vision-language model as the judge that teaches a robot by comparing pairs of its attempts, and reached 88.9% average final success on six Meta-World manipulation tasks against 65.1% for the strongest comparable method, RL-VLM-F. It asked 41.5% fewer questions with less than half the abstention rate, and learned three tasks from scratch on a real Mobile ALOHA arm.
Vision-language models can replace humans in telling a learning robot which of two attempts was better, removing hand-written reward functions. Can that be made cheap and reliable enough to train real robots online?
Every VLM query costs inference time, and the models often decline to answer when two camera frames look nearly the same, so a large share of the query budget is wasted. Early in training the robot also moves almost randomly, so the clips the VLM is shown carry little signal and learning stalls before it starts.
RL-VLM-F asks a VLM for preferences on randomly chosen pairs; VLM-RM scores frames by embedding similarity with the task text; Text2Reward has the model write reward code; PEBBLE uses preference learning with entropy-based selection and a scripted oracle.
SPEAR adds a small predictor that estimates, before asking, whether the VLM will be able to tell two clips apart, and combines that with the usual uncertainty score so it only asks questions that are both informative and answerable. For exploration it starts from a reward the VLM synthesises from the task description, then switches to the reward model learned from preferences, with relay exploration keeping trajectories diverse after the switch. InternVL3-78B served as the oracle in simulation and Qwen3-VL-8B on the real robot.
Final success averaged over six tasks and five seeds was 88.9%, against 65.1% for RL-VLM-F, 63.4% for Text2Reward, 51.4% for PEBBLE and 14.9% for VLM-RM; MEKAWIRE computes the gap to RL-VLM-F from the table as 23.8 points (the paper's headline says 23.3). The biggest gains came on contact-heavy tasks: drawer opening 86.4% against 34.8% for the best baseline. Text2Reward was better on window opening (87.7% vs 84.9%). To reach 30% success on drawer opening SPEAR needed 960 queries with 29.6% abstention against 1,640 queries and 55.5% for RL-VLM-F, and 9.1 hours instead of 13.6 (33.1% less). Across five open VLMs, conservative prompting gave 92.3% preference accuracy. On the real Mobile ALOHA left arm the policy converged in 18,200 steps for drawer opening, 16,600 for window opening and 12,600 for ball pushing, using 720, 660 and 500 comparisons. Ablations: query selection 80.7% vs 14.0% random; full exploration 88.3% vs 24.8% baseline.
This is a preprint. The six simulated tasks come from Meta-World, a simple tabletop benchmark, and the ablations use a simulated oracle fitted to VLM behaviour rather than the VLMs themselves. On the real robot, objects were reset by hand between episodes, the workspace was restricted for safety and only convergence steps are reported, not a success rate over test trials. At about one second per step, the drawer task alone is roughly five hours of arm motion by MEKAWIRE's arithmetic, before resets. The paper's headline margin does not match its own table exactly.
Teams that want robots to learn new tasks on site without reward engineering, from warehouse picking to service robots, need cheap automated feedback; SPEAR shows small open VLMs can supply it on a local workstation if the questions are chosen well. The query-budget savings matter most for companies paying for VLM inference at scale. Time scale: a research tool now; factory use needs automatic resets and safety limits, likely two years or more.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.