When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models

Can VLA policies learn to bind instructions to scene-dependent actions, and does training with same-instruction, different-scene action alternatives remedy failures where models replay familiar but incorrect trajectories?

Fine-tuned VLA policies often exceed 90% success on in-distribution tasks and are robust to nuisance changes, but fail under counterfactual changes that require a different action, such as swapped object positions or a new target. Because demonstrations under one instruction typically contain a narrow trajectory family, the model can use language to retrieve a familiar behavior and use vision only for local tracking, never learning to choose the task-level action the current scene requires.

Standard flow-matching imitation learning on task-specific demonstrations (e.g., official pi0.5 and GR00T-N1.7 checkpoints) can fit instruction-keyed policies; prior augmentation methods such as CAST, RoCoDA, and MimicGen add demonstration diversity but do not pair same-instruction alternatives, and inference-time methods like CAG strengthen language conditioning without fixing the underlying supervision gap.

ECT has two parts. ECT data builds action-valid counterpart demonstrations (l, s, s', a, a') where the same instruction l requires different actions in visually distinguishable scenes, using geometric transforms and a replay controller to guarantee valid actions. The ECT loss samples matched pairs and sums the standard imitation losses of both counterparts in one update, sharing flow-matching noise and time; no auxiliary objective is added. The method keeps the policy architecture unchanged and was evaluated with constructed LIBERO pairs, naturally occurring CALVIN pairs, and physical UR5e demonstrations collected under a fixed budget.

In controlled Frozen-LM pi0.5 training on LIBERO-PRO, full ECT raises mean Swap success from 36% to 59% and mean Task success from 21% to 28%. Per suite, ECT data alone lifts Swap success from 52.7/38.3/41.3/12.7 to 72.1/71.1/51.8/29.5 across Spatial/Object/Goal/Long, and adding the ECT loss further changes these to 74.4/70.8/55.5/34.7. On CALVIN ABC to D, the ECT loss applied to original data raises five-task completion from 57.6% to 75.6%; constructed ECT data reaches 65.7%, and full ECT 70.6%. On a real UR5e with 400 demonstrations per model, unseen-position success improves from 3/40 with standard training to 30/40 with ECT data and 35/40 with full ECT; unseen object+position rises from 0/10 to 8/10, and swap from 4/20 to 20/20 under paired training. In behavioral probes, ECT increases correct/near-correct following from 27% to 48% and reduces retrieval-like behavior from 69% to 47%.

Authors note ECT only improves distinctions exposed by the added alternatives; in Spatial, object-name changes remain wrong because every training instruction keeps the bowl. A front-back mirror without visual separation can harm ID success (94% to 24% in a diagnostic run), so scene cues must distinguish alternatives. The Full-FT and GR00T comparisons use official checkpoints rather than matched retrained controls; GR00T results use ECT data but not the ECT loss. Real-robot tasks are simple pick-place scenes under a fixed budget, and CALVIN results are single-seed. The method depends on successful replay and task-specific transform families. As a preprint, the approach lacks independent peer review.