RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

Does a single aggregate success rate tell a robot buyer how a vision-language-action policy breaks under changed conditions, and at which step of a long task the failure starts?

VLA robustness is usually reported as one success number before and after a perturbation. That hides whether the same episodes fail, whether some failures turn into successes and cancel out, and whether the robot fails at the first step or the last.

Robustness benchmarks such as LIBERO variants and RoboCasa perturb scenes and report aggregate success. They rarely pair each perturbed run with an identical clean run or track which sub-goal of a long-horizon task was reached.

RoboCasa-Shift, built on RoboCasa365, changes one factor at a time (illumination, external camera orientation, robot base starting pose, Gaussian blur) at three calibrated severities. Every perturbed rollout is paired with a clean rollout that shares all other settings, so each episode is classed as success-to-success, success-to-failure, failure-to-success or failure-to-failure. A stage tracker reads simulator state to record how far into the task the robot got.

Only GR00T N1.5 passed the authors' clean-competence gate, on four long-horizon kitchen tasks with 80 matched episodes per condition; clean macro success was 0.738. Gaussian blur cut success to 0.550, 0.225 and 0 at the three levels (drops of 0.188, 0.513 and 0.738), with 59 success-to-failure and no failure-to-success transitions at the top level. Shifting the base start pose produced drops of 0.075, 0.150 and 0.263; on RinseSinkBasin the drop reached 0.55. Camera orientation gave no change at level 1, 0.188 at level 2 and a smaller 0.138 at level 3. Illumination changed success by 0.012, minus 0.025 and 0.025, all within noise, yet 12 matched episodes flipped at level 2. At camera level 1, four successes became failures and four failures became successes, a net change of zero.

One policy, four tasks, simulation only: the authors make no cross-model claim and say real-robot transfer is untested. Stage labels come from audited simulator predicates, not official annotations. With 80 episodes per condition, small effects carry wide intervals. No tested policy other than GR00T N1.5 met the gate, which itself says something about current VLA reliability on long tasks.