STAR: Steering Action Representations during Inference
Vision-language-action models (VLAs) map language instructions and visual observations to robot actions, showing strong capabilities across robotic manipulation tasks. However, in our probing experiment, we find that action representations near the action output do not fully preserve task-relevant visual grounding.
From the abstract. Our summary is in progress.
Full paper
STAR: Steering Action Representations during Inference
Published under CC BY 4.0. Reproduced with attribution; original at arXiv:2610.41009 (PDF).
Abstract
Vision-language-action models (VLAs) map language instructions and visual observations to robot actions, showing strong capabilities across robotic manipulation tasks. However, in our probing experiment, we find that action representations near the action output do not fully preserve task-relevant visual grounding. Motivated by this late-layer grounding gap, we introduce STAR, for STeering Action Representations, a training-free inference method for pre-trained VLAs. STAR applies an exponential moving average over late-layer residual updates, allowing later action representations to inherit object-aware update directions from earlier layers without model updates or architectural changes. It further uses recent proprioceptive states to adapt the mixing strength, using conservative mixing when execution dynamics change rapidly. Experiments on robot manipulation benchmarks show that STAR improves late-layer object awareness and increases average task success rates by 4.0 percentage points. Our code is available at https://github.com/PKU-VaLuE-Lab/STAR.
1 Introduction
Despite the tremendous success of vision-language models (VLMs) [27, 7, 16, 26, 2, 1], previous work has shown that they may overly rely on language priors, producing responses that are not fully grounded in the visual input [25, 19, 15, 10, 51]. Meanwhile, recent studies also observe that vision-language-action models (VLAs) exhibit similar behavior patterns, such as insufficient use of task-relevant visual cues [45, 36, 48, 54, 21] or distraction from irrelevant visual tokens [20, 44, 58, 22]. Inspired by these works, we ask a key question: how strongly are action representations in VLAs grounded in task-relevant visual evidence?
To probe the dependence of action representations on visual evidence, we design a controlled experiment. As illustrated in Figure 1 (a), we mask the target object pixels by replacing them with the mean RGB values of the image. By masking the visual information important for action execution, we expect the action representations to differ from those produced without interference. To measure the representation change induced by object masking, we define an object awareness score, computed as \|h^{k}_{\mathrm{m}}-h^{k}_{\mathrm{o}}\|\ /\ \|h^{k}_{\mathrm{o}}\|. Here, h^{k} is the hidden state of the k-th layer in Figure 1(b), while \mathrm{o} and \mathrm{m} denote the original and masked observations, respectively. This metric provides a heuristic estimate of how strongly task-relevant visual evidence affects the action representation. A larger score indicates a larger difference in the representation after masking. The resulting object-awareness score for \pi_{0.5} [39] is reported in Figure 1 (c). We find that, during native inference, object awareness increases through early-to-middle layers and drops in late layers. This suggests that visual evidence is gradually aggregated into action representations from early to middle layers, but fades away in late layers. This probing experiment demonstrates a grounding gap in late layers during VLA inference.
h^{k} to h^{k+1} through a residual update. (c) Object awareness of \pi_{0.5} layer outputs across depth under native inference and our method STAR. (d) Success rates of native inference and our method STAR on LIBERO-PRO for \pi_{0.5} and GR00T-N1.5, respectively.To address the gap, we introduce STAR, for STeering Action Representations, a training-free method that regularizes the action representations in late layers without updating model weights. Intuitively, STAR steers late-layer action representations by carrying residual updates from earlier layers to later layers. Specifically, STAR applies an exponential moving average (EMA) [14] over late-layer updates. When action semantics change rapidly, such as during end-effector rotation or fine-grained adjustment, the intervention should be conservative. Conversely, when the robot does not rotate, such as when moving in a straight line, the intervention can be stronger. These differences in execution dynamics are reflected in recent proprioceptive trajectories. In short, STAR uses a proprioception-modulated EMA to smooth cross-layer residual updates, carrying object-aware directions from earlier layers.
This design is validated through our object awareness measurement and robotic manipulation benchmarks. In the test of object awareness, as shown in Figure 1 (c), STAR increases the object awareness of late-layer action representations. In robotic manipulation benchmarks, STAR enhances the performance of \pi_{0.5} [39] and GR00T-N1.5 [33] in LIBERO-PRO, as shown in Figure 1 (d), and yields an average success-rate improvement of 4.0 percentage points in our main experiments.
Our contributions are summarized as follows:
• We introduce STAR, a training-free VLA inference method that regularizes action representations toward better visual grounding.
• Our technical innovation is an EMA that combines residual updates across late layers, together with a modulation strategy that adjusts its coefficient using proprioceptive states.
• Extensive experiments validate that STAR can consistently improve pre-trained VLA models across benchmarks, with an average success-rate improvement of 4.0 percentage points.
2 Related Work
2.1 Vision-Language-Action Models
Vision-Language-Action models (VLAs) are robot policies adapted from pretrained vision-language models (VLMs) with strong multimodal understanding abilities [61, 55, 31, 17, 41]. RT-2 [61] established this paradigm by co-fine-tuning a VLM on vision-language data and robot trajectories, thereby aligning it with robotic control. This line was further developed by RoboFlamingo [23] and OpenVLA [18], which showed that VLM backbones can serve as strong and scalable policy priors. More recent works extend the action-token formulation [56] or the training pipeline while retaining this VLM-centric recipe. \pi_{0} [3] and \pi_{0.5} [39] adopt flow-based continuous action modeling. UniVLA [5] studies cross-embodiment policy learning through task-centric latent actions. The GR00T series [32, 33, 34, 35] balances efficiency and performance with a dual-system design. ReconVLA [45] strengthens the perceptual role of VLA models by using reconstructive supervision. As VLA foundation models improve, our work explores a complementary training-free direction that further enhances their inference-time performance without additional data collection or parameter updates.
2.2 Training-free Methods in Embodied Policies
A growing body of work improves embodied policies at test time without updating model weights, with existing approaches differing mainly in where the intervention is applied. One line of work modifies diffusion or flow trajectories. TNB [53], HAJL [30], and DemoDiffusion [38] re-noise and denoise alternative or retargeted action sequences, effectively blending them with the generative prior of a pre-trained policy. VLS [28] and DynaGuide [9] further steer denoising with guidance from additional vision foundation models. Another line targets action decoding. BID [29] restores reactivity via closed-loop resampling over candidate chunks. RTC [4] enables asynchronous real-time control by freezing committed actions and inpainting the remaining chunk. Recent test-time scaling methods use online search or candidate selection to improve action choice, including VLA-Reasoner [12] and anti-exploration steering [52]. A more recent line intervenes at the spatial visual-token level. IVRA [37] refines relations among visual tokens using encoder-derived affinity hints, CGVD [44] suppresses spatial distractors through concept-gated visual distillation, and DTP [20] prunes task-irrelevant visual tokens that mislead action generation. Overall, these results highlight that performance gains can be achieved purely through inference-time modifications. Most approaches operate on explicit input-output interfaces. In contrast, our work explores whether directly intervening on transformer hidden-state dynamics can improve visual grounding and downstream control.
2.3 Visual Hallucination
Visual hallucination is common in multimodal models, including VLMs [25, 50, 11, 49, 6] and embodied visuomotor policies, where it manifests as weak visual grounding or action-level hallucination [45, 54, 43]. For VLMs, DAMO [51] mitigates visual hallucination by accumulating activation momentum during decoding. Similarly, DCLA [46] and VASparse [60] suggest that inference-time control of hidden states or visual token flow can improve faithfulness without retraining. For embodied policies, the analogous failure is inaccurate visual grounding, such as attending to distractors or acting on the wrong object. Knowledge Insulation [8] shows that naively adapting VLM backbones to action can erode the transferred semantic knowledge that supports grounding. ReconVLA [45] strengthens grounding through a reconstructive objective over gaze regions, whereas DTP [20] offers a lightweight training-free alternative by pruning distracting tokens at inference time. Our method is closest in spirit to DAMO, as both methods use training-free control of layer-wise dynamics. However, DAMO targets VQA tasks in VLMs, while our work targets robotic action execution. This embodied setting introduces proprioceptive execution states as a key signal for controlling the intervention, a feature specific to robot policies.
3 STAR: Steering Action Representations during Inference
3.1 Overview
Vision-language-action models generate actions from visual observations, proprioceptive states, and language instructions. At each control step t, a VLA policy receives an observation o_{t}=(I_{t},s_{t}), where I_{t} denotes the visual input and s_{t} denotes the robot proprioceptive state, together with a language instruction x. The policy outputs an action chunk:
A_{t}=[a_{t,1},a_{t,2},\ldots,a_{t,H_{p}}],where H_{p} is the prediction horizon, and the first H_{e} actions a_{t,1:H_{e}} are executed. Each executed action is decoded from the model’s final action representations, making it important to preserve task-relevant visual evidence until the late layers.
Our key design principle is to preserve object-aware update directions from earlier layers and carry them to later layers using EMA. A fixed EMA strength, however, ignores the non-uniform dynamics of robotic manipulation. We characterize execution dynamics along a spectrum from progress, where the robot moves steadily along a consistent direction, to transition, where execution dynamics can change rapidly due to contact changes, end-effector reorientation, or fine-grained adjustment. Our heuristic is that transition phases require a more conservative EMA coefficient to limit the extent of inference-time editing, thereby preventing the policy from deviating too much from its original behavior. Following this principle, we propose STAR, a training-free VLA inference method that carries residual updates from earlier, object-aware layers to late layers via a proprioception-modulated EMA. An overview of our method is illustrated in Figure 2.
3.2 EMA-Based Layer-wise Residual Update
We assume the VLA backbone contains n Transformer layers L_{1},\ldots,L_{n}. We define the late-layer mixing region by a relative starting depth \rho:
p=\lfloor\rho n\rfloor,\qquad\mathcal{L}_{\mathrm{early}}=\{L_{1},\ldots,L_{p}\},\qquad\mathcal{L}_{\mathrm{mix}}=\{L_{p+1},\ldots,L_{n}\}.t. STAR estimates a step-level EMA coefficient \beta_{t} from proprioceptive history and applies it to layer-wise updates. Blue arrows denote the EMA coefficient \beta_{t} derived from proprioceptive history, while pink arrows denote the cross-layer update momentum used in the EMA mixing.In our experiments, we set \rho=2/3, so mixed updates are applied to the final third of the Transformer layers. This choice is motivated by the diagnostic study in Figure 1 and prior work on inference-time modification of multimodal models [51, 57]. The first p=\lfloor\rho n\rfloor layers are left unchanged, allowing the backbone to form language-conditioned visual evidence. For layers in \mathcal{L}_{\mathrm{mix}}, which are closer to action decoding and may become less object-aware, we apply proprioceptively modulated EMA updates.
Formally, we denote the direct output of the k-th Transformer block at robot time step t as \Delta h_{t}^{k}. This is the residual update of hidden state h_{t}^{k}:
h_{t}^{k}=h_{t}^{k-1}+\Delta h_{t}^{k}.STAR mixes residual updates across late layers using an EMA. Analogous to momentum in optimization, it maintains a residual-update trend (“momentum”) m_{t}^{k} across the late layers of each forward pass. At control step t, this trend is updated using coefficient \beta_{t}:
m_{t}^{k}=\beta_{t}m_{t}^{k-1}+(1-\beta_{t})\Delta h_{t}^{k},\qquad L_{k}\in\mathcal{L}_{\mathrm{mix}}.The hidden states of layers in \mathcal{L}_{\text{mix}} are updated using m_{t} rather than the original \Delta h_{t}:
\hat{h}_{t}^{k}=h_{t}^{k-1}+m_{t}^{k},\qquad h_{t}^{k}\leftarrow\hat{h}_{t}^{k},\quad L_{k}\in\mathcal{L}_{\mathrm{mix}}.This operation carries a smoothed update direction across depth, allowing the object-aware information formed in earlier layers to persist through later layers.
Another question is how to choose \beta_{t} at each control step t. A fixed coefficient \beta_{t} cannot account for how robot execution alternates between progress and transition phases. We therefore compute \beta_{t} from causal proprioceptive evidence.
3.3 Causal Proprioceptive Evidence
A VLA policy operates in a closed sensorimotor loop, so the robot’s own motion provides direct evidence of the current execution dynamics. Specifically, let p_{t}=\mathrm{pos}(s_{t})\in\mathbb{R}^{3} denote the end-effector position extracted from s_{t}, and let R_{t}=\mathrm{rot}(s_{t}) denote the corresponding orientation representation. We construct four causal quantities from the end-effector trajectory and orientation history.
First, we measure directional continuity:
c_{t}=\cos(p_{t}-p_{t-1},\,p_{t-1}-p_{t-2}).A high c_{t} indicates that the end-effector continues to move along a consistent direction, which often corresponds to steady progress.
Second, we measure the motion scale:
v_{t}=\|p_{t}-p_{t-1}\|_{2}.This quantity reflects whether the robot is still undergoing substantial translational motion.
Third, we measure an acceleration-like change in translation:
j_{t}=\|(p_{t}-p_{t-1})-(p_{t-1}-p_{t-2})\|_{2}.This quantity captures abrupt changes in motion, which often occur near transitions, contact changes, or local corrections.
Fourth, we measure orientation change:
\omega_{t}=d_{\mathrm{rot}}(R_{t},R_{t-1}),where d_{\mathrm{rot}}(\cdot,\cdot) denotes a rotation distance. Orientation changes are particularly informative for execution dynamics involving grasping, insertion, drawer interaction, or precise placement.
All quantities are computed only from observations available up to the current control step. Therefore, the modulation signal is strictly causal and does not use future states or oracle task information.
3.4 Causal Normalization
The raw magnitudes of proprioceptive quantities vary across embodiments, tasks, and simulators. We therefore normalize each scalar using a causal sliding window. For x_{t}\in\{c_{t},v_{t},j_{t},\omega_{t}\}, we define the window length and the corresponding history as
W=f_{\mathrm{ctrl}}T_{w},\qquad\mathcal{H}_{t}^{x}=\{x_{t-W},\ldots,x_{t-1}\},where f_{\mathrm{ctrl}} is the control frequency and T_{w} is the temporal span of the statistics. The normalized value is then computed as
z_{t}^{x}=\frac{x_{t}-\mu(\mathcal{H}_{t}^{x})}{\sigma(\mathcal{H}_{t}^{x})+\epsilon}.Since \mathcal{H}_{t}^{x} only contains values before the current control step, the normalization remains strictly causal. This makes the normalized proprioceptive evidence depend on recent execution context rather than on the entire episode history.
3.5 Proprioceptive Mixing Score
We compute a signed proprioceptive mixing score from the normalized proprioceptive indicators:
r_{t}=w_{c}z_{t}^{c}+w_{v}z_{t}^{v}+w_{j}z_{t}^{j}+w_{\omega}z_{t}^{\omega}.Positive r_{t} indicates that stronger mixed updates are preferred, while negative r_{t} indicates that late layers should adapt more strongly to the current update. The sign of each coefficient determines whether the corresponding signal increases or decreases the late-layer cross-layer mixing strength. In our implementation, directional continuity and motion scale provide positive evidence for steady progress and thus stronger mixed updates, whereas abrupt translational change and orientation change provide negative evidence when they indicate transitions or fine adjustments.
To make the scale of r_{t} stable across tasks, we further apply a robust causal normalization. Let \mathcal{R}_{t}=\{r_{t-W},\ldots,r_{t-1}\} be the recent history of proprioceptive mixing scores. We compute:
\hat{r}_{t}=\frac{r_{t}-\operatorname{median}(\mathcal{R}_{t})}{\operatorname{IQR}(\mathcal{R}_{t})+\epsilon}.The use of the median and interquartile range reduces sensitivity to transient spikes in proprioceptive measurements.
3.6 Proprioception-Conditioned Mixing Coefficient
The normalized proprioceptive score is mapped to the continuous coefficient used in the EMA mixed updates:
\beta_{t}=\beta_{\min}+(\beta_{\max}-\beta_{\min})\sigma(\kappa\hat{r}_{t}),where \sigma(\cdot) is the sigmoid function and \kappa controls the sharpness of the transition. A larger \beta_{t} preserves more accumulated update momentum in Eq. (4), helping late layers maintain object-aware update information during steady execution. A smaller \beta_{t} makes the cross-layer mixed update rely more on the current layer update, which is useful during transitions, contact changes, or fine adjustment.
This operation does not modify the attention map. Instead, it only changes how late-layer residual updates [13, 47, 40] are mixed before being propagated to the action head. Therefore, STAR directly targets the update stream where late-layer visual degradation occurs, while the proprioceptive coefficient \beta_{t} adapts the cross-layer mixing strength to the robot’s current execution dynamics.
4 Experiments and Results
We evaluate STAR on two manipulation benchmarks, SimplerEnv [24] and LIBERO-PRO [59]. These benchmarks cover complementary aspects of VLA control: SimplerEnv emphasizes Google Robot-style tabletop manipulation with diverse object and articulated-object interactions, while LIBERO-PRO focuses on long-horizon desktop manipulation with compositional task variations. Our experiments are designed to answer three questions: (i) whether STAR improves frozen VLA policies without additional training, (ii) whether the proposed cross-layer residual mixing mechanism transfers across different VLA backbones, and (iii) which design choices are responsible for the improvement, including the proprioceptive evidence, beta scheduling, and intervention start layer.
4.1 Implementation Details
Backbones. We evaluate STAR on three representative VLA backbones with different vision-language architectures and action-generation heads. GR00T-N1.5 [33] uses an NVIDIA Eagle VLM to encode language instructions and visual observations, and conditions a flow-based action head on the resulting vision-language embeddings together with robot state and action tokens. OpenVLA [18] is an open-source VLA model built on a Prismatic-style VLM. It combines DINOv2 and SigLIP visual encoders with a projection module and a Llama-2-based language backbone. We use OpenVLA-7B in our experiments. \mathbf{\pi_{0.5}} [39] follows a PaliGemma-style VLM plus action-expert design. Its VLM backbone uses a SigLIP vision encoder and a Gemma-based language transformer, and the action expert generates continuous action chunks through flow matching. For each backbone, we compare the original frozen policy with the same policy equipped with STAR. All model weights remain frozen during evaluation.
Evaluation protocol. We apply STAR only at inference time and follow the official evaluation protocol of each benchmark. We report task success rate as the primary metric, including both per-task success rates and average success rates.
4.2 Benchmarks
SimplerEnv. SimplerEnv [24] evaluates policies in simulated Google Robot manipulation environments. We use six Fractal/Google Robot tasks. Open Drawer requires the robot to grasp and pull an articulated drawer handle. Place in Drawer requires opening a drawer, placing an object inside, and completing the placement under occlusion and articulated-object constraints. Pick Coke Can evaluates object-centric grasping and lifting for a commonly used can object. Move Near tests spatial relation reasoning by asking the robot to move one object close to another target object. Close Drawer requires pushing or sliding the drawer back to a closed state after interaction. Pick Object evaluates generic object picking across randomized object instances and placements. These tasks jointly cover grasping, articulated-object interaction, spatial relocation, and relation-aware control.
LIBERO-PRO. LIBERO-PRO [59] evaluates long-horizon manipulation under multiple forms of generalization. We report five evaluation splits. LIBERO-10-lan paraphrases instructions while preserving task semantics. LIBERO-10-object modifies non-essential object attributes such as color, texture, or size. LIBERO-10-swap perturbs initial object placements and spatial configurations. LIBERO-10-task changes target objects, action requirements, or task goals. LIBERO-10-env varies visual context such as backgrounds, lighting, or textures. Each split contains ten tasks, and we report the success rate averaged over the tasks within each split. For STAR with \pi_{0.5}, we evaluate 50 episodes per task, totaling 500 episodes per split and 2,500 episodes across the five splits.
4.3 Main Results
We compare STAR with native inference and a training-free attention-side baseline, DTP [20]. DTP aims to improve VLA inference by identifying task-relevant visual regions and pruning the distraction image tokens according to action-token attention. It is therefore a natural baseline for testing whether improving visual attention allocation is sufficient for more reliable action generation. However, DTP requires an additional forward pass to obtain the attention signals used for pruning, whereas STAR performs single-pass inference and directly modulates the late residual update stream without changing tokens, attention maps, or model weights.
- Open Drawer
- Place in Drawer
- Pick Coke Can
- Move Near
- Close Drawer
- Pick Object
- No data
- No data
- No data
- GR00T-N1.5 [33]
- Native
- 26.7%
- 4.0%
- 69.3%
- 68.7%
- 45.0%
- 32.3%
- 41.0%
- DTP
- 26.7%
- 4.0%
- 59.3%
- 68.7%
- 41.7%
- 39.3%
- 40.0%
- No data
- STAR (Ours)
- 32.0%
- 4.7%
- 77.3%
- 76.7%
- 47.3%
- 40.0%
- 46.3%
- No data
- OpenVLA [18]
- Native
- 7.4%
- 0.0%
- 65.3%
- 56.7%
- 22.2%
- –
- 30.3%
- DTP
- 11.1%
- 0.0%
- 69.3%
- 69.0%
- 22.2%
- –
- 34.3%
- No data
- STAR (Ours)
- 18.5%
- 3.7%
- 70.0%
- 61.7%
- 25.9%
- –
- 36.0%
- No data
- lan
- object
- swap
- task
- env
- No data
- No data
- No data
- GR00T-N1.5 [33]
- Native
- 81.2%
- 40.4%
- 0.0%
- 6.0%
- 73.8%
- 40.3%
- DTP
- 82.0%
- 42.2%
- 0.0%
- 1.8%
- 75.6%
- 40.3%
- No data
- STAR (Ours)
- 82.8%
- 48.0%
- 2.2%
- 6.8%
- 76.4%
- 43.2%
- No data
- \pi_{0.5} [39]
- Native
- 92.8%
- 66.2%
- 9.2%
- 6.8%
- 65.8%
- 48.2%
- DTP
- 90.4%
- 64.6%
- 8.0%
- 7.8%
- 67.6%
- 47.7%
- No data
- STAR (Ours)
- 92.6%
- 67.0%
- 7.2%
- 18.8%
- 65.0%
- 50.1%
- No data
Table 1 summarizes the SimplerEnv results. STAR improves the average success rate from 41.0% to 46.3% on GR00T-N1.5 and from 30.3% to 36.0% on OpenVLA. The gains are consistent across all six GR00T-N1.5 tasks and are stronger overall than those of DTP, showing that STAR is effective across the two backbone-specific evaluation settings. Table 2 reports the LIBERO-PRO results. STAR improves the average success rate from 40.3% to 43.2% on GR00T-N1.5 and from 48.2% to 50.1% on \pi_{0.5}. GR00T-N1.5 gains most on the object split. For \pi_{0.5}, the largest gain is on the task split, from 6.8% to 18.8%, while the object split improves slightly and the language, swap, and environment splits decline. These results support the applicability of STAR across both autoregressive-style and continuous-action VLA backbones.
4.4 Ablation Studies
We conduct ablation studies with OpenVLA on SimplerEnv, and all ablations follow the same protocol as the main SimplerEnv evaluation. We focus on three factors: the proprioceptive evidence used to compute the proprioceptive mixing score, the beta scheduling strategy, and the layer region where modulation is applied. These ablations isolate the main design choices of STAR and test whether the observed gains come from meaningful proprioceptive modulation rather than from a generic late-layer perturbation.
4.4.1 Effect of Proprioceptive Evidence
- —
- c_{t}
- v_{t}
- j_{t}
- \omega_{t}
- Open Drawer
- Place in Drawer
- Pick Coke Can
- Move Near
- Close Drawer
- No data
- Full STAR (Ours)
- \checkmark
- \checkmark
- \checkmark
- \checkmark
- 18.5%
- 3.7%
- 70.0%
- 61.7%
- 25.9%
- 36.0%
- w/o direction consistency
- \times
- \checkmark
- \checkmark
- \checkmark
- 14.8%
- 0.0%
- 64.0%
- 56.7%
- 14.8%
- 30.1%
- w/o translation scale
- \checkmark
- \times
- \checkmark
- \checkmark
- 11.1%
- 0.0%
- 66.7%
- 55.0%
- 22.2%
- 31.0%
- w/o jerk
- \checkmark
- \checkmark
- \times
- \checkmark
- 14.8%
- 3.7%
- 65.3%
- 58.3%
- 14.8%
- 31.4%
- w/o rotation change
- \checkmark
- \checkmark
- \checkmark
- \times
- 14.8%
- 0.0%
- 66.7%
- 58.3%
- 18.5%
- 31.7%
- Positive evidence only
- \checkmark
- \checkmark
- \times
- \times
- 3.7%
- 0.0%
- 62.7%
- 61.7%
- 18.5%
- 29.3%
- Transition evidence only
- \times
- \times
- \checkmark
- \checkmark
- 11.1%
- 3.7%
- 70.7%
- 58.3%
- 22.2%
- 33.2%
- No proprioceptive evidence
- \times
- \times
- \times
- \times
- 11.1%
- 0.0%
- 56.0%
- 53.3%
- 18.5%
- 27.8%
Table 3 studies the role of proprioceptive evidence. Removing any single cue causes a 4.3–5.9 percentage point drop in average success rate, indicating that the four signals provide complementary information for characterizing execution dynamics across tasks. Direction consistency and translation scale help identify steady progress, while jerk and rotation change provide transition-sensitive evidence for contact changes and fine manipulation. The grouped ablations further support this interpretation: using only positive evidence reaches 29.3%, and using only transition evidence reaches 33.2%, both below the full model. The no-proprioception variant obtains 27.8%, confirming that the improvement is not simply caused by adding a late-layer mixing operation, but by conditioning the cross-layer mixing strength on meaningful robot-state dynamics.
4.4.2 Effect of Beta Scheduling
We report success rates averaged over eight evaluation runs for this ablation.
- —
- Open Drawer
- Place in Drawer
- Pick Coke Can
- Move Near
- Close Drawer
- No data
- Fixed-\beta=0.25
- 11.1%
- 0.0%
- 68.0%
- 56.7%
- 22.2%
- 31.6%
- Fixed-\beta=0.35
- 14.8%
- 0.0%
- 65.3%
- 56.7%
- 14.8%
- 30.3%
- Fixed-\beta=0.45
- 9.3%
- 1.9%
- 65.7%
- 58.3%
- 18.5%
- 30.7%
- Fixed-\beta=0.55
- 14.8%
- 0.0%
- 64.0%
- 60.0%
- 14.8%
- 30.7%
- Dynamic-\beta (0.15,0.55)
- 14.8%
- 0.0%
- 64.0%
- 55.0%
- 22.2%
- 31.2%
- Dynamic-\boldsymbol{\beta} \mathbf{(0.25,0.55)} (Ours)
- 18.5%
- 3.7%
- 70.0%
- 61.7%
- 25.9%
- 36.0%
- Dynamic-\beta (0.35,0.55)
- 7.4%
- 0.0%
- 70.7%
- 51.7%
- 14.8%
- 28.9%
- Dynamic-\beta (0.25,0.45)
- 11.1%
- 0.0%
- 64.0%
- 53.3%
- 25.9%
- 30.9%
- Dynamic-\boldsymbol{\beta} \mathbf{(0.25,0.55)} (Ours)
- 18.5%
- 3.7%
- 70.0%
- 61.7%
- 25.9%
- 36.0%
- Dynamic-\beta (0.25,0.65)
- 18.5%
- 3.7%
- 66.7%
- 63.3%
- 18.5%
- 34.1%
Table 4 compares fixed and dynamic choices of the cross-layer mixing coefficient. Fixed-\beta variants reach at most 31.6% average success, showing that a single cross-layer mixing strength cannot accommodate the different execution states in SimplerEnv. Dynamic scheduling improves performance when the range is properly calibrated: decreasing the lower or upper endpoint yields 31.2% with Dynamic-\beta(0.15,0.55) and 30.9% with Dynamic-\beta(0.25,0.45), respectively, while increasing them yields 28.9% with Dynamic-\beta(0.35,0.55) and 34.1% with Dynamic-\beta(0.25,0.65). The proposed range, Dynamic-\beta(0.25,0.55), achieves the best average success rate of 36.0%, showing that both adaptivity and range calibration are necessary for effective proprioception-conditioned mixed updates.
4.4.3 Effect of Start Layer
- —
- Open Drawer
- Place in Drawer
- Pick Coke Can
- Move Near
- Close Drawer
- No data
- \rho=0
- 14.8%
- 0.0%
- 49.3%
- 53.3%
- 11.1%
- 25.7%
- \rho=1/3
- 11.1%
- 0.0%
- 55.0%
- 68.3%
- 3.7%
- 27.6%
- \boldsymbol{\rho} \mathbf{=2/3} (Ours)
- 18.5%
- 3.7%
- 70.0%
- 61.7%
- 25.9%
- 36.0%
- \rho=1 (Native)
- 7.4%
- 0.0%
- 65.3%
- 56.7%
- 22.2%
- 30.3%
Table 5 evaluates which layer region should use mixed updates. Mixing from the first layer reduces the average success rate to 25.7%, suggesting that early visual-language computation should remain unchanged. Starting from the middle of the stack improves Move Near to 68.3%, but weakens object-centric and articulated-object manipulation, leading to a 27.6% average. Applying mixed updates only to the final third of layers achieves the best average success rate of 36.0% and the strongest performance on four out of five tasks. This supports the central design of STAR: intervene on the late residual update stream closest to action decoding, while preserving the backbone’s native early-to-middle multimodal processing.
4.5 Real-Robot Experiments
We further evaluate STAR with \pi_{0.5} on a real-robot task that requires picking up a pear and placing it in a bowl. We collect 100 episodes and fine-tune \pi_{0.5} for 15,000 steps. Both native inference and STAR use the same fine-tuned checkpoint, with STAR applied only at inference time. Table 6 reports the success rates over 40 evaluation trials.
- Native
- 17/40
- 42.5%
- STAR (Ours)
- 19/40
- 47.5%
On the real-robot system with a single NVIDIA RTX 3090, STAR introduces only 3.3% additional inference latency relative to native inference.
5 Conclusion
We introduced STAR, a training-free inference method that improves visual grounding in pre-trained VLA models by steering late-layer action representations. Based on the observation that object awareness weakens near the action output, STAR applies an EMA over late-layer residual updates to preserve object-aware update directions from earlier layers. It further adapts the mixing strength using recent proprioceptive states, making the intervention more conservative when action semantics change rapidly. Experiments show that STAR improves late-layer object awareness and consistently raises task success rates across robotic manipulation benchmarks. Our results highlight training-free residual steering as an effective and complementary way to enhance VLA foundation models without additional training.
Limitations. While STAR improves training-free VLA inference across multiple benchmarks and backbones, several limitations remain. First, STAR requires access to intermediate Transformer hidden states. It therefore cannot be directly applied to black-box policies that expose only action outputs. Second, the proprioceptive cues used by STAR are designed primarily for single-end-effector manipulation tasks. Extending this design to more complex embodiments remains to be explored. Finally, STAR is based on an empirical observation that sensitivity to task-relevant objects declines in later VLA Transformer layers. We observe this pattern in the models evaluated here, but it is not theoretically guaranteed for all pretrained VLA models. STAR may offer limited benefits when this pattern is absent.
References
Appendix A Internal Mechanism of STAR
We have shown in Section 4 that STAR improves VLA performance at inference time. This section reveals the internal effect of STAR.
A.1 Experiment Setup
All experiments in this section are conducted with the \pi_{0.5} VLA model on the LIBERO-PRO benchmark. We record two quantities at layer k. First, we record the hidden state of the action expert after the layer, denoted by h^{k}. Second, we record the action-to-image attention map, denoted by A^{k}. For masking diagnostics, we compare the original unmasked visual input with masked visual inputs obtained by replacing pixels inside a selected mask region with the mean RGB value of the image, while keeping all other inputs unchanged. This masking operation removes local visual evidence. Figure 3 summarizes the complete diagnostic protocol and the main empirical observations used in this section.
A.2 Object-awareness of Action Hidden State
A VLA policy ultimately predicts actions from action representations rather than from attention visualizations. Therefore, the diagnostic question we ask is not only whether the model visually attends to the target object, but whether the task-relevant object evidence actually affects the hidden state used for action generation. To measure this effect, we define an awareness score based on object masking.
For a mask region m, let h^{k}_{\varnothing} be the layer-k hidden state under the original visual input, and let h^{k}_{m} be the hidden state under the corresponding masked input. We define the layer-wise awareness induced by mask m as
\mathcal{A}^{k}(m)=\frac{\left\|h^{k}_{\varnothing}-h^{k}_{m}\right\|_{2}}{\left\|h^{k}_{\varnothing}\right\|_{2}}.When m is the task-relevant object mask m_{o}, the resulting quantity \mathcal{A}^{k}(m_{o}) is the object awareness of the layer. A larger value means that masking the task object produces a larger relative change in the action hidden state, indicating stronger sensitivity to task-relevant object evidence.
In Section 1, we introduce object awareness in its simplest form, using the task-object mask: \|h^{k}_{(\mathrm{m})}-h^{k}_{(\mathrm{o})}\|/\|h^{k}_{(\mathrm{o})}\|. We now generalize the same idea into a mask-dependent awareness function to conduct controlled experiments. Under this definition, \mathcal{A}^{k}(m_{o}) recovers the object-awareness score introduced in Section 1.
To verify that object awareness is not merely measuring generic sensitivity to image corruption, we construct two controls: the same number of pixels randomly scattered across the image (m_{r}), and a rectangular region of similar area randomly sampled from the background (m_{b}). We define control awareness as
\mathcal{A}^{k}_{\mathrm{ctrl}}=\max\left\{\mathcal{A}^{k}(m_{b}),\mathcal{A}^{k}(m_{r})\right\}.To summarize the masking comparison, we use two statistics: object-awareness gain and object-selectivity ratio. They quantify two complementary effects: whether STAR increases sensitivity to the task-relevant object compared with native inference, and whether the measured sensitivity is stronger for the task-relevant object than for same-area control regions.
The object-awareness gain compares STAR with native inference:
R^{k}_{\mathrm{gain}}=\frac{\mathcal{A}^{k}_{\text{STAR }}(m_{o})}{\mathcal{A}^{k}_{\mathrm{native}}(m_{o})}.The object-selectivity ratio compares object masking with same-area control masking under the same inference method u\in\{\mathrm{native},\text{STAR }\}:
R^{k}_{\mathrm{select}}(u)=\frac{\mathcal{A}^{k}_{u}(m_{o})}{\mathcal{A}^{k}_{u,\mathrm{ctrl}}}.Thus, R^{k}_{\mathrm{gain}}>1 indicates that STAR increases object awareness relative to native inference, while R^{k}_{\mathrm{select}}>1 indicates that the measured awareness is stronger for the task object than for same-area control regions.
Using the above metric, we compare native inference and STAR on the paired original and masked inputs. Figure 3(a) visualizes the three mask types used in the diagnostic, and Figure 3(b) reports the corresponding average awareness scores at layer 18 under native inference and STAR. Under the experimental setting with the \pi_{0.5} VLA model on the LIBERO-PRO benchmark, STAR increases object awareness from 0.056 to 0.087, giving R_{\mathrm{gain}}=1.554.
The selectivity statistics further show that the measured awareness is task-object specific. For native inference, R_{\mathrm{select}}(\mathrm{native})=1.806. For STAR, R_{\mathrm{select}}(\text{STAR })=1.977. Therefore, the increased awareness under STAR is not merely a response to arbitrary masking. It is stronger when the task-relevant object region is removed than when same-area control regions are removed.
These results show that object awareness is an effective diagnostic for the hidden-state intervention. It clearly captures that STAR makes late action representations more sensitive to task-relevant object evidence. As shown later in Section A.4, this object-awareness signal is much clearer than the corresponding attention-map change.
m_{o}, randomly scattered pixels m_{r} with the same pixel count, and a randomly sampled rectangular background region m_{b} of similar area. (b) Comparison of awareness scores for native inference and STAR under object masking and control masking at layer 18. (c) Attention-map visualization at layers 13, 16, and 18, showing that native inference and STAR produce similar attention patterns. (d) Attention mass on the task-object region across layers. Native inference and STAR exhibit close attention-mass curves, indicating that attention-based object focus does not clearly reveal the hidden-state intervention effect.A.3 Gradient-based Saliency Map
The masking analysis above measures whether the action hidden state is sensitive to the task object. We further ask a complementary, action-grounded question: does the final predicted action rely more strongly on pixels belonging to that object? To answer this question, we use input-gradient saliency [42], which identifies the image evidence that locally influences the action output.
For a predicted action chunk a_{1:K}, we first average the actions in the chunk,
u=\frac{1}{K}\sum_{k=1}^{K}a_{k},and construct a scalar action score by projecting u onto its stop-gradient direction:
s=u^{\top}\operatorname{sg}\!\left(\frac{u}{\lVert u\rVert_{2}+\epsilon}\right),where \operatorname{sg}(\cdot) denotes stop-gradient. Holding the action direction fixed during differentiation gives a scalar whose input gradient measures how each image region supports the action selected by the policy, without allowing the reference direction itself to change during backpropagation.
Let I be the input image and let g(p) denote the magnitude of \partial s/\partial I at pixel p, aggregated over the color channels. We spatially normalize this evidence and define the target-object gradient score as
\mathcal{S}_{\mathrm{grad}}=\frac{\sum_{p\in\Omega_{o}}g(p)}{\sum_{p\in\Omega_{\mathrm{img}}}g(p)+\epsilon},where \Omega_{o} is the task-object region and \Omega_{\mathrm{img}} contains all image pixels. Thus, a larger \mathcal{S}_{\mathrm{grad}} means that a greater fraction of the action-relevant visual evidence lies on the key object.
Figure 4(a) shows that STAR achieves a higher target-object gradient score than native inference on every LIBERO-PRO split. The improvement is therefore not restricted to a particular perturbation type: it appears under language, object-attribute, spatial-configuration, task, and visual-context variation. The paired differences in Figure 4(b) are also predominantly positive, indicating that the aggregate gains reflect a broadly consistent sample-level trend. These results are consistent with the final action depending more strongly on task-object pixels under STAR.
The qualitative maps in Figure 5 reveal the corresponding spatial effect. Under native inference, the action gradients are often weaker on the task object or dispersed over the robot and background. With STAR, stronger evidence is concentrated inside the cyan task-object regions, including articulated fixtures, manipulated objects, and interaction targets. Importantly, this result complements the attention analysis in Section A.4: STAR need not visibly reallocate attention mass to make the action more object-grounded. Instead, by carrying object-aware residual-update directions into the late action representations, it increases the local sensitivity of the final action to key-object pixels. Together, the masking and gradient diagnostics support the proposed mechanism: STAR preserves task-relevant visual evidence in the hidden-state update stream and makes that evidence more consequential for action generation.
A.4 Attention Maps Do Not Reveal the Intervention Effect
We next test whether attention maps can reveal the effect of the hidden-state intervention. We compare native inference and STAR using the same input and record the action-to-image attention maps at late action-expert layers. Figure 3(c) visualizes representative attention maps from native inference and STAR at layers 13, 16, and 18. Although STAR modifies the hidden-state stream, the corresponding attention maps remain visually similar across these layers.
To provide a more quantitative comparison, we compute the attention mass assigned to the task-object region. Let \Omega_{o} denote the image-token set covered by the task-object mask, \Omega_{\mathrm{img}} denote all image tokens, and A^{k} denote the attention map of layer k. For method u\in\{\mathrm{native},\text{STAR }\}, the layer-wise object attention mass is
\mathcal{M}^{k}_{u}=\frac{\sum_{p\in\Omega_{o}}A^{k}_{u}(p)}{\sum_{p\in\Omega_{\mathrm{img}}}A^{k}_{u}(p)}.Figure 3(d) reports \mathcal{M}^{k}_{u} across layers for native inference and STAR. These results show that the attention-based measure of object focus remains close to that of native inference and does not reliably expose the hidden-state intervention.
A.5 Diagnostics of STAR’s Internal Mechanism
The diagnostics above suggest that the internal effect of STAR is primarily reflected in the action hidden state and its dependence on visual evidence rather than in a visible reallocation of action-to-image attention. Object masking shows that STAR increases the sensitivity of late action representations to task-relevant object evidence, while same-area control masks induce smaller responses. Gradient saliency further shows that a larger fraction of the visual evidence influencing the final action is concentrated on the task object. In contrast, the attention maps and attention-mass curves remain close to native inference. Thus, STAR preserves task-object information in late action representations and makes it more consequential for action generation without necessarily producing a salient change in attention visualizations.
Appendix B Hyperparameter Settings
Table 7 reports the hyperparameters used for each backbone and benchmark. The coefficient notation follows Section 3, with w_{\omega} denoting the weight for orientation change.
- \pi_{0.5}
- LIBERO-PRO
- 0.25
- 0.55
- 0.25
- 1.0
- -0.125
- -0.25
- 2.30
- GR00T-N1.5
- LIBERO-PRO
- 0.20
- 0.50
- 1.0
- 1.0
- -0.375
- -2.0
- 3.75
- GR00T-N1.5
- SimplerEnv
- 0.20
- 0.50
- 1.0
- 1.0
- -0.375
- -2.0
- 4.00
- OpenVLA
- SimplerEnv
- 0.25
- 0.55
- 1.0
- 1.0
- -0.25
- -0.75
- 4.75
For the \pi_{0.5} success-rate evaluation on LIBERO-PRO, we apply STAR to the final six action-expert layers. The environment runs at 10 Hz, and the policy executes five actions before replanning.