Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks
This arXiv preprint introduces a benchmark-agnostic evaluation framework that measures how successful VLA-controlled trajectories change under input perturbations, going beyond task success rate. Testing three state-of-the-art VLA models across four LIBERO suites and seven perturbation conditions, it finds that perturbed trajectories can still succeed yet diverge substantially in motion smoothness, efficiency, and gripper behaviour. This matters because teams relying only on success rate may miss behavioural degradation that affects safety and reliability.
Can input perturbations alter the behaviour of successful VLA-controlled manipulation trajectories even when task success rate remains unchanged?
Current VLA robustness evaluation focuses on whether a task is completed successfully, but this ignores how the robot moves and interacts during successful attempts. As a result, models may pass standard benchmarks while exhibiting less smooth, less efficient, or differently controlled gripper behaviour under perturbations, which could indicate latent reliability or safety issues.
Prior work assessed VLA robustness primarily via Task Success Rate (TSR) on benchmarks like LIBERO and LIBERO-Plus, with some recent perturbation studies. This approach cannot detect changes in trajectory quality when success is preserved, so it gives an incomplete picture of robot behaviour under distribution shift.
The authors propose a benchmark-agnostic behavioural robustness framework that extends LIBERO and LIBERO-Plus to measure how successful trajectories are executed under perturbation. Instead of only TSR, they characterise typical successful behaviour and its variability using metrics of motion smoothness, efficiency, and gripper behaviour across four task suites, seven perturbation conditions, and three state-of-the-art VLA models.
The abstract does not report specific numerical metrics. It reports that all three state-of-the-art VLA models, across four LIBERO suites and seven perturbation conditions, can show comparable TSR under the same perturbation yet diverge substantially in motion smoothness, efficiency, and gripper behaviour on successful trajectories.
The abstract does not state explicit limitations. Evident constraints include evaluation on simulated LIBERO/LIBERO-Plus benchmarks without stated real-robot validation, and no quantitative effect sizes or thresholds are provided in the abstract. The framework's generalisability beyond tabletop manipulation and these specific models is not addressed.
Robotics companies deploying VLA models for tabletop manipulation—such as warehouse picking, lab automation, and service robotics—could use this framework to add behavioural robustness checks to their evaluation pipelines. If the approach proves predictive of real-world reliability, adoption for safety validation could occur within 1–3 years; first, the metrics must be validated on real robots and linked to failure modes.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.