Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models

Do vision-language-action (VLA) models act on instructions that specify how to carry out a task (grip here, go around there), not only on instructions that name the task, and if not, what in their training prevents it?

Robot makers pitch VLAs as robots you can coach in plain language, the way people correct a chatbot. Benchmarks mostly test whether a model can identify the task from paraphrased wording. Whether a spoken correction such as 'grasp the carrot by its green end' changes the robot's motion had not been tested systematically.

Prior work made VLAs robust to rephrased task instructions, or limited 'how' guidance to fixed command sets, or replaced language corrections with visual markers or externally generated recovery motions. Dual-system designs, in which a vision-language model watches the robot and sends sub-instructions, assume the low-level VLA will follow them.

The authors test five public VLAs, Physical Intelligence's pi0, InternVLA-M1, QwenGR00T, InstructVLA and LangForce, on the WidowX tasks of the SIMPLER simulator in two settings. In offline correction, failed episodes are rerun with a plain retry, a correction written by a Google DeepMind vision-language model, and an inverted correction that says the opposite. In online intervention, a supervising model sends corrections during execution. They then retrain the models on the same demonstrations relabelled with segment-level descriptions of the robot's movement, and build SIMPLER-IN, a benchmark in which success is only possible by following the 'how' instruction.

Pooled over 1,046 failed episodes, a plain retry recovered 13.1%, a correction 5.9% and an inverted correction 3.8%. pi0 recovered 25.1% on retry versus 12.3% with a correction. In online intervention, always-on guidance cut pooled success from 50.9% to 37.8%. With segment-level relabelling, actions moved in the direction the correction specified in about 84% of evaluated episodes, against about 53% for the original models and controls, close to the 50% chance level, and the always-on penalty shrank to 2 points; yet task success did not rise consistently, and no model reliably passed the demanding or out-of-distribution SIMPLER-IN tasks.

The authors note that all tests use one simulated robot (WidowX), one dataset (BridgeData V2) and short pick-and-place tasks, that some VLM-written corrections may be wrong, that relabelling changes both timing and content of instructions without separating the two, and that retraining used LoRA on a limited budget. We add that none of the newest commercial models (Figure Helix, Gemini Robotics, pi0.6) was tested, so the result describes open models.