robustness benchmark for vision-language-action modelsIntermediate
LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models
Seventeen robot control models that score 63.8% to 98.5% on the popular LIBERO benchmark lost an average of 26.5 percentage points when objects, goals, cameras or the robot itself were changed mid-task, according to a preprint from Huazhong University of Science and Technology and Tsinghua. Fine-tuning Physical Intelligence's pi0.5 on just 80 robustness demonstrations lifted its success on unseen perturbed tasks to 70.1%, against 28.0% to 47.9% for existing policies.