Enhancing Visual Task-Redirection Attacks on Vision-Language-Action Models via Closed-Loop Optimization
A printed image stuck on a wall made a real robot arm running a GR00T-based policy pick up a watch instead of the toy bee it was told to fetch in 8 of 15 trials, double the 4 of 15 of the earlier attack method, according to a preprint from Sun Yat-sen University and partners. In simulation the closed-loop attack, TRACO, redirected five VLA models in 85.2% of runs on seen layouts against 46.9% before.
How reliably can an attacker who only changes what a robot sees, with a perturbation or a printed patch, make a vision-language-action (VLA) model carry out a different task than the one its user asked for, and does optimising the attack in closed loop make it stronger?
VLA models follow a user's spoken or typed instruction, which makes their camera input a security surface. Earlier attacks tuned a visual perturbation to match a recorded target trajectory offline. Once the perturbation changes the robot's early moves, the robot reaches states the recording never contained, the fixed targets stop being useful and the attack often fails mid-task, so the real risk was hard to measure.
Action Attack and TRAP optimise global perturbations or adversarial patches against fixed target actions from a recorded trajectory, minimising average action error. Lower error on recorded frames did not guarantee that the robot completed the attacker's task when actually executed.
TRACO wraps existing targeted attacks in a loop: it runs the attacked robot, rebuilds the target supervision for the states actually reached, and concentrates further updates where task progress stalls while keeping supervision on the earlier steps that already worked. Printed patches add homography and colour calibration for physical robustness.
On LIBERO, averaged over five models from the StarVLA family and LangForce, global-perturbation attack success rose from 46.9% to 85.2% on seen layouts (+38.3 points, MEKAWIRE) and from 43.2% to 75.9% on unseen ones; for the GR00T policy it reached 96.4% against 63.2%. Patch attacks gained less, from 46.6% to 53.5% on average, and on the OFT model TRACO was weaker than the baseline (19.2% against 23.6%). On InstructVLA in SimplerEnv, TRACO lifted TRAP's patch success from 64.0% to 72.8%. On a real SO-101 arm with a GR00T policy, the clean robot did its task 15 of 15 times; a printed TRACO patch redirected it 8 of 15 times against 4 of 15 for Action Attack. Building the global attack took about 88.8 minutes against 396.3 for Action Attack (77.6% less, MEKAWIRE).
This is a preprint and an attack study, not a defence. The real-robot test is one low-cost arm, one task pair and 15 trials. The attack needs white-box access to the policy and repeated runs of the robot or simulator while it is built, which a real attacker may not have. It works less well on patches than on global perturbations and depends on the frozen policy being able to reach the target task from attacked states. Progress is estimated from end-effector positions, which do not show whether a grasp succeeded.
Any company deploying VLA-driven robots in places strangers can reach, such as retail, hospitals or warehouses with visitors, now has a measured example of a printed image hijacking which object a robot picks, while the user's instruction looks unchanged. NVIDIA's GR00T family and open VLA stacks were among the targets. Expect buyers and safety standards to start asking for adversarial testing of the vision input, task-level checks that compare the action with the instruction, and access control on the robot's view. For now the threat needs white-box access, which limits it to open-weight and leaked models.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.