Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors
This preprint tunes action-expert vision-language-action models to accept non-language task cues: 8-channel EMG envelopes or visual segmentation annotations. On a real cube-picking task, EMG conditioning lifts cluttered success from 30% to 50% and cuts failure rate from 50% to 4%; on 16 simulated RoboCasa pick-and-place tasks, visual annotations lift cluttered success from 11% to 20% and cut failure from 47% to 25%. The findings suggest downstream VLA policies can usefully condition on continuous cues beyond fixed language prompts, especially in clutter.
Can VLA policies be specialized to use continuous EMG or visual segmentation signals as task descriptors instead of descriptive language, and does this improve robustness in cluttered or out-of-distribution scenes?
VLAs typically receive a fixed, human-written language prompt up front to specify the task; language can be ambiguous or cumbersome in cluttered scenes, and it does not update during execution. Existing systems lack a straightforward way to inject real-time or visual task intent into the state representation.
Prior work tuned VLAs with descriptive text prompts; shared-control research used EEG, EMG, or BCI signals mainly for assistive feedback or human-in-the-loop gating, and visual prompting methods such as RT-Trajectory, MOKA, MOO, and Gaze2Act used keypoints, trajectories, or masks often alongside language rather than replacing it. These approaches do not fully exploit continuous non-language signals as primary task conditioning in VLA state spaces.
The authors tune SmolVLA for a real SO-101 cube-picking task, concatenating an 8-dimensional EMG envelope from a supervising human's wrist gestures to the proprioceptive vector with a constant underspecified text prompt. They also tune π0.5 on 16 RoboCasa pick-and-place tasks, where target object and destination are highlighted in green and blue segmentation masks in the RGB images, with a constant underspecified text prompt. Both use the same flow-matching VLA objective and are evaluated against language-prompted baselines in uncluttered and cluttered scenes.
In Experiment 1 across three participant-specific models and 30 trials each, uncluttered success was 80% for both EC-VLA and baseline, while cluttered success was 50% vs 30% (+20 points), cluttered completion score 0.82 vs 0.43, and cluttered failure rate 4% vs 50%. In Experiment 2 over 800 total episodes on 16 simulated tasks, VA-VLA success was 31% vs 27% for the baseline; on eight cluttered tasks, success was 20% vs 11% (+9 points), completion score 0.51 vs 0.35, and failure rate 25% vs 47%.
Authors note future work must measure how much guidance is needed and account for new users or segmentation tracking systems; they did not test EMG cross-user generalization, and VA-VLA relies on simulator ground-truth masks. EC-VLA is real-robot but small (three participants, single cube task), while VA-VLA is simulation-only; no code-release statement or compute cost is reported.
Assistive-robot and shared-autonomy companies could use participant-calibrated EMG conditioning within 1–3 years to let users guide manipulation in cluttered or ambiguous settings; warehouse, bin-picking, and home-robot vendors could adopt visual mask conditioning in 1–3 years if robust real-time object detection and tracking is available. Broader deployment requires user-independent EMG models, low-latency closed-loop interfaces, and reliable segmentation outside simulation.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.