FR-VLA: kraftbewusstes VLA von NTU und A*STAR erreicht 58,3 Prozent bei Kontaktaufgaben

Abstract

Integrating force awareness into Vision-Language-Action (VLA) models involves not merely encoding wrench measurements as an additional modality, but determining when and how strongly physical interaction should influence robot action generation. Existing VLAs that use force typically inject it through a fixed pathway, without explicitly determining when it is relevant to the current action. This calls for an intermediate force relevance mechanism inside the VLA, in which visual-language context estimates the relevance of physical feedback to the current task before force enters the action pathway. FR-VLA addresses this gap with a Vision-Language Force-Reasoning Model that uses learnable force query tokens to derive a force admission signal grounded in visual-language context, and uses this signal to modulate an external tokenized force pathway for action generation. In real-world experiments on five contact-rich tasks across two robotic platforms, FR-VLA achieves the highest mean success rate over six evaluated stages, and trajectory visualizations show that its force admission varies with the interaction phase.

Project page: https://fr-vla.pages.dev/

Keywords: Force-aware VLA, Contact-rich manipulation

1 Introduction

Modern VLAs have become effective generalist policies by grounding robot action generation in visual observations and language instructions through pretrained VLM backbones [1, 2, 3, 4]. However, contact-rich manipulation exposes a limitation of this visual-language foundation: interaction states in insertion, assembly, wiping, and pressing are often only partially observable from images, making force feedback essential for precision-oriented tasks [5, 6, 7, 8, 9]. The importance of force feedback for action generation varies across task phases, depending on the intended interaction and the evolving contact state [10, 11, 12]. This motivates policies that use visual-language context to determine when and to what extent force feedback should influence action generation. Consequently, force integration is not merely a problem of sensing or encoding wrench measurements, but one of dynamically grounding physical feedback in visual-language task context [13].

Existing force-aware VLA architectures mainly differ in the point at which force is injected, as illustrated in Figure 1(a). Injecting low-dimensional wrench signals into the VLM token stream risks diluting force information within high-dimensional visual-language representations [14, 15, 16, 17]. Relocating force conditioning to the action expert avoids this representational imbalance, but bypasses the VLM state where task progress and interaction phase are inferred [18, 19, 20, 21]. Both choices therefore provide force exposure rather than force grounding: the policy can react to measured wrench changes, but it is not explicitly supervised to decide when force should affect action generation under a task-dependent contact phase.

We frame force-aware VLA as a modality grounding problem, where grounding denotes estimating the task-conditioned admissibility of physical feedback from the VLA’s visual-language state before measured force is allowed to affect action generation. Recent advances in multimodal foundation models have shown that heterogeneous inputs can be grounded in pretrained backbones through intermediate interfaces such as latent queries, resamplers, and gated attention [22, 23, 24]. These interfaces transfer information into VLM representation spaces without requiring raw non-visual signals to be inserted as ordinary input tokens. For force-aware VLA, such an interface can use visual-language context to assess when physical feedback should influence action generation, taking task progress and contact phase into account.

This perspective motivates separating the estimation of force relevance from the encoding of force measurements. Such a separation calls for a mechanism that uses the VLA’s visual-language task state to modulate the contribution of measured wrench histories to action generation. FR-VLA instantiates this mechanism with force query states that infer a force admission signal from visual-language context, while measured wrench histories remain in a separate action-side force pathway. The resulting admission behavior varies with task progress rather than simply with wrench availability, as illustrated in Figure 1(b).

Real-world evaluations on two robotic platforms and five contact-rich tasks show improved performance and robustness over vision-only and directly force-conditioned VLA baselines. Our contributions are summarized as follows: (i) Force-aware VLA is formulated as a modality grounding problem for low-dimensional physical feedback, where force relevance is inferred from visual-language task context rather than assumed from sensor availability. (ii) FR-VLA introduces a force admission architecture in which visual-language states determine when force can influence action generation, while measured wrench histories are encoded separately for action-side control. (iii) A detailed analysis of grounded force admission shows that the learned admission behavior follows task phase interaction regimes rather than instantaneous force magnitude.

2 Related Work

Force-Aware Policy Learning and Contact Control. Classical contact control provides model-based approaches to compliant manipulation. Hybrid position/force control assigns position and force objectives to complementary directions in a task-related coordinate frame [25]. Impedance control specifies the dynamic relationship between motion and interaction force, shaping the robot’s mechanical response to its environment [26]. Force feedback has been studied in both policy learning and contact control [27, 28, 29, 30, 31]. ForceMimic combines force-aware demonstration collection with imitation learning of force–motion commands and hybrid force-position control [32]. Force Policy separates vision-guided global behavior from a local force feedback policy that estimates an interaction frame for hybrid force-position control [21]. Learning variable impedance provides another approach to adapting robot behavior under contact uncertainty [33, 34]. Beyond control design, effective multimodal learning requires preventing policies from neglecting force feedback. FACTR addresses this issue through a curriculum that progressively reduces visual corruption, encouraging the policy to use force information rather than overfit to vision [35]. CRAFT similarly addresses modality imbalance in VLAs, using a variational information bottleneck to regulate visual-language representations during curriculum fine-tuning [36]. FR-VLA addresses force utilization through an explicit architectural separation between context-dependent force admission and wrench history representation.

Force-Aware VLAs and Contact-Dependent Fusion. Recent approaches incorporate physical feedback into VLAs through different interfaces [16, 20, 37, 38]. ForceVLA introduces a force-aware mixture of experts for action generation, while ForceVLA2 combines force-based prompts in the VLM with cross-scale fusion in the action expert for hybrid force-position control [18, 14]. Other approaches exploit physical information during training while reducing deployment-time sensing requirements: FD-VLA distills force information into latent tokens, and HapticVLA transfers tactile-aware behavior to a student that operates without inference-time tactile sensing [15, 39]. Prediction-based designs anticipate future interaction: FoAR uses future contact prediction to guide visual–force fusion and dynamically adjust force use across contact and non-contact phases [10], and 3D world model policies forecast future scene states to guide action generation [40]. In FR-VLA, force query states infer interaction phases from visual-language context, and the resulting admission signal biases action-side attention to a separately represented wrench history. This phase-supervised interface connects task context to force utilization while keeping measured wrench inputs in a dedicated action-side pathway.

Cross-Modal Interfaces and Grounding. Latent queries and gated interfaces offer useful building blocks for connecting heterogeneous modalities. Perceiver IO uses latent arrays to process structured inputs, Flamingo introduces gated cross-attention between visual features and a language model, and BLIP-2 uses a query-based interface between frozen visual and language models [22, 23, 24]. FR-VLA adopts a task-specific query interface: its force query states receive motion phase supervision to ground force admission in visual-language context. Measured wrench histories remain outside the VLM token stream and condition the action model through a separate force pathway. This design distinguishes the inference of force relevance from the representation of measured physical feedback.

3 FR-VLA Model

3.1 Motivation

Direct force use exposes a behavioral ambiguity because making wrench measurements continuously available does not specify when force should become eligible to affect action generation. Force often becomes informative over temporal regimes in which wrench dynamics evolve before and during contact changes, not only at instants of large measured transients.

This ambiguity is evident in precision insertion, where a large contact force may indicate constraint or misalignment, while a subsequent reduction in force during corrective motion remains decision relevant because it indicates that the adjustment is releasing the constraint. A useful policy should therefore maintain force eligibility across such regimes, allowing the action model to observe evolving force patterns rather than react only to magnitude peaks, as shown in Figure 3. This distinction motivates treating force admission as a task-conditioned eligibility window over force dynamics. FR-VLA therefore separates force relevance from force representation, as shown in Figure 2: force query tokens estimate how strongly force should be admitted, and the Force Tokenizer represents the measured wrench on the action side.

3.2 Implicit Force Reasoning Module

Force query tokens provide a VLM-internal interface for force admission without carrying measured wrench values. They are optimized to infer an interaction phase state from visual-language context, rather than to reconstruct force measurements or predict actions directly. This allows the VLM to acquire force awareness while keeping raw physical feedback outside the token stream, as shown in Figure 2(b). The force query states h^{f}_{t} infer a motion phase distribution, from which a signed admission signal is derived as

\hat{p}_{t}=\mathrm{softmax}\!\left(P_{\phi}(h^{f}_{t})\right),

g^{f}_{t}=G_{\eta}(\hat{p}_{t})=s_{\mathrm{adm}}\left(1-\frac{\mathcal{H}(\hat{p}_{t})}{\log K_{p}}\right)\tanh\!\left(M_{g}(\hat{p}_{t})\right).

Here, P_{\phi} is the phase inference head, M_{g} is an MLP, and K_{p} is the number of task-specific phases. The \tanh output promotes or suppresses force attention, while the learnable positive scale s_{\mathrm{adm}} controls its magnitude; its parameterization is given in Appendix A.1. Normalized entropy measures phase uncertainty; greater uncertainty attenuates the adjustment toward zero, returning the logit offset toward its baseline.

Motion phase supervision. We generate phase labels offline from the robot’s 6D wrench measurements and gripper states. Gripper opening transitions partition demonstrations into manipulation segments. Within each segment, a shared detector identifies contact events from force deviations relative to an adaptive reference estimated from recent non-contact frames, using hysteresis and temporal persistence. Major events are selected based on both absolute and segment-relative peak deviations. The contact interval extends from the first major event’s onset to the final force release relative to its pre-contact reference, spanning intermediate retractions and brief force-free gaps; the remaining frames are labeled free. For tasks with extended pre-contact approach periods, we further distinguish an approach phase. These labels provide auxiliary supervision for phase prediction from visual-language context. The full labeling procedure is provided in Appendix A.3.

3.3 Force Tokenizer and Admission-Conditioned Attention

The force admission signal determines how strongly force is promoted or suppressed in action generation, but the promoted force still requires a stable action-side representation. Raw wrench histories are high-frequency and magnitude-sensitive; using them as continuous control features alone can make action generation overly responsive to transient contact changes. The Force Tokenizer Module addresses this by assigning short wrench histories to recurring contact-dynamic patterns, so local wrench variations that correspond to the same contact regime share a stable code embedding.

The Force Tokenizer is implemented as a VQ-VAE [41], pretrained on demonstration wrench histories and frozen during VLA training. Given a wrench history W_{t}, the VQ encoder selects the nearest codebook entry. Its embedding does not replace the continuous wrench representation; instead, it generates FiLM [42] scale and shift parameters that modulate the raw force tokens, so the action model captures recurring contact patterns while retaining detailed wrench information, as illustrated in Figure 2(c). Implementation details are given in Appendix A.2.

Force admission is implemented within an adapter-style action bridge following VLA-Adapter [43], where action query latents serve as the interface between VLM representations and action generation. FR-VLA incorporates force information into this bridge through admission-conditioned attention. Let S^{x,m}_{t} denote the scaled attention scores for source x in head m, with the force scores defined as

S^{w,m}_{t}=\frac{Q^{a,m}_{t}(K^{w,m}_{t})^{\top}}{\sqrt{d_{h}}}.

The force admission signal g^{f}_{t} and a learned per-head bias \beta_{m} jointly shift all force token scores:

\widetilde{S}^{w,m}_{t}=S^{w,m}_{t}+\beta_{m}+g^{f}_{t}.

The scores from all sources are then concatenated and jointly normalized:

A^{m}_{t}=\mathrm{softmax}\!\left([S^{\mathrm{self},m}_{t}\mid S^{\mathrm{adapter},m}_{t}\mid S^{\mathrm{vision},m}_{t}\mid\widetilde{S}^{w,m}_{t}]\right).

Here, Q^{a,m}_{t} and K^{w,m}_{t} are the action query and force key projections, and d_{h} is the head dimension. The bias \beta_{m} captures each head’s baseline preference for force information, while g^{f}_{t} provides a context-dependent adjustment. This shared offset preserves relative scores among force tokens while modulating their total attention mass.

3.4 Training Objective

Training proceeds in two stages. First, the task-specific Force Tokenizer is pretrained from demonstration wrench histories and then frozen. Second, FR-VLA is trained with behavior cloning and phase supervision. The policy predicts an H-step action chunk \hat{A}_{t}=(\hat{a}_{t},\ldots,\hat{a}_{t+H-1}) from the current observation, matched to the demonstrated chunk A_{t}=(a_{t},\ldots,a_{t+H-1}). The training objective is

\mathcal{L}_{\mathrm{FR}}=\left\|\hat{A}_{t}-A_{t}\right\|_{1}+\lambda_{p}\,\mathrm{CE}\!\left(\hat{p}_{t},y^{p}_{t}\right),

where the L_{1} term is summed over the action horizon, \hat{p}_{t} is the inferred motion phase distribution, and y^{p}_{t} is the offline phase label. During training, phase inputs to downstream modules are detached: the phase loss trains the force query phase inference module, while the action loss trains the admission gate and action expert without propagating gradients to the phase logits. At inference, the model infers the current contact phase from visual-language context, adjusts its attention to force accordingly, and generates the action chunk.

4 Experimental Results

Our experiments investigate four questions: (1) How does FR-VLA compare with other methods across different types of contact-rich manipulation tasks? (2) How does the learned force admission signal vary across interaction phases? (3) How do force admission and the Force Tokenizer contribute to performance? (4) How robust is FR-VLA to changes in objects and target positions? We address these questions through real-world comparisons, trajectory analysis, component ablations, and perturbation evaluations.

4.1 Experimental Setup

FR-VLA is evaluated on two Universal Robots platforms, UR5e and UR3e, using the robots’ built-in 6D force/torque sensing as the wrench feedback source. Each setup is observed by an Intel RealSense D455 third-view camera and a wrist-mounted Intel RealSense D405 camera. We build a benchmark with 70–90 demonstrations per task, as shown in Figure 4. FR-VLA is compared against four baseline methods: VLA-Adapter [43], VLA-Adapter w/F, ForceVLA [18] with force-aware MoE fusion during action decoding, and TA-VLA [19] with decoder-side torque-action joint diffusion. All methods use the same data and environmental conditions; detailed parameters are provided in Appendix B.1.

4.2 Real-World Contact-Rich Manipulation Results

As shown in Figure 5, FR-VLA achieves the highest success rate on all six evaluated stages across five tasks. Its mean success rate of 58.3% exceeds that of the strongest baseline by 20.8 percentage points. The main improvement comes from stages where force is informative only under the right task context, rather than from simply adding wrench feedback to the policy. In Switch Cable Connection, visual evidence becomes unreliable during the side power connector insertion, making contact feedback essential once insertion begins. However, the wrench rises sharply at the moment of contact, and policies that only react passively to this force signal are prone to jamming during insertion without recovering. FR-VLA instead admits force according to the inferred interaction phase, allowing transient contact cues to be exploited when they are informative without continuously driving the policy. In Screw Tightening, the useful signal is not the absolute wrench magnitude, but the transition between alignment, engagement, and tightening, where small exploratory contacts and transient changes indicate whether the screw has entered the correct regime. These results suggest that FR-VLA improves performance by deciding when wrench dynamics should affect action generation.

We further compare FR-VLA with a simpler wrench magnitude gating baseline on Gear Assembly, which regulates force admission by the measured wrench amplitude without phase supervision. FR-VLA achieves 90% success, compared with 70% for magnitude-based gating over 20 trials per method. This indicates that wrench magnitude alone does not adequately reflect the relevance of force to action selection: both high forces during constrained insertion and decreasing forces during corrective motion can be informative. A representative magnitude gating trajectory and further analysis are provided in Appendix B.2.

4.3 Analysis of Force Admission Behavior

As shown in Figure 6, the policy jams twice before completing the Ethernet cable insertion. Each time the plug jams and the wrench rises sharply, the policy promptly retracts instead of continuing to push, and then reinserts. During both retractions, the measured wrench returns close to zero, yet force admission remains positive, so attention to force is maintained throughout the recovery; admission becomes negative only after the insertion is completed and the inferred phase switches to free. This shows that grounding force admission in the interaction phase keeps force feedback available throughout contact-rich recovery, allowing the model to actively exploit force signals. Analyses for all five tasks are provided in Appendix B.3.3. The phase inference balanced accuracy reported in Table 1 illustrates the grounding ability of FR-VLA, and the evaluation protocol is described in Appendix B.3.2. Most phase prediction errors occur near contact onset and release, where gradual changes in visual observations may make phase boundaries difficult to identify.

4.4 Ablation Study

We evaluate the contributions of force admission and the Force Tokenizer through component ablations, as shown in Table 2. Removing force admission reduces the mean success rate from 58.3% to 35.0%, while removing the Force Tokenizer reduces it to 40.0%. Both ablations lower success rates across all six evaluated stages, supporting the complementary roles of context-conditioned force attention and contact-dynamic representations. To examine what the Force Tokenizer captures, we relate the selected codes to interaction events during contact. Within each task, individual codes are associated with specific events such as jamming, holding, and withdrawing; in Screw Tightening, one code is 3.3 times more likely than average to co-occur with jamming, while another is 5.0 times more likely to co-occur with withdrawing. The tokenizer therefore distinguishes interaction regimes within the contact phase rather than treating contact as a single state. Further analysis of force code usage is provided in Appendix B.4.

4.5 Robustness to Object and Position Perturbations

We evaluate robustness to object and position changes in Gear Assembly and Switch Cable Connection. For Gear Assembly, we test two unseen gears and two groups of unseen target peg positions, varying contact geometry and alignment conditions, respectively. For Switch Cable Connection, we test four groups of unseen initial Ethernet cable positions that alter the approach direction and contact angle during top port insertion, as illustrated in Appendix B.5.

As shown in Tables 3 and 4, FR-VLA outperforms VLA-Adapter w/F by 25 percentage points under both gear and target position changes, and completes the insertion in half of the trials across the unseen cable positions.

Cable position changes are more challenging because the cable is deformable: changing its initial position alters its bending configuration and tension, and thus the forces transmitted to the connector during insertion. The resulting wrench patterns are underrepresented in the demonstrations, making adaptation more demanding than compensating for rigid object pose changes.

5 Conclusion

This paper frames force integration in VLAs as a force grounding problem rather than a simple sensing or fusion problem. FR-VLA separates force admission from force representation, using visual-language context to decide when wrench feedback should influence action generation while keeping measured force in an action-side pathway. Real-world results show that this separation improves contact-rich manipulation and that learned force admission follows task phase interaction regimes rather than instantaneous force magnitude.

6 Limitations

Although FR-VLA improves force admission and representation, fine-grained force-conditioned corrections remain challenging in precision manipulation. Human demonstrations often use force changes to guide very small action adjustments, which can be diluted during behavior cloning because the corresponding action differences are small relative to the overall trajectory. Future work should introduce more structured action-side mechanisms, such as force-sensitive residual heads or transition-focused objectives, to increase sensitivity to small but decision-critical force variations. We will also evaluate, on larger-scale datasets, whether temporal changes across multiple visual frames can help anticipate more fine-grained force interaction states.

Acknowledgments

This research is supported by the National Robotics Programme (NRP) 2.0 funding initiative “Domain-specific Robotics Foundation Models for Manufacturing (DS-RFM)”.

References

1. [1] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning, 2024.
2. [2] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. \pi_{0}: A vision-language-action flow model for general robot control. In Proceedings of Robotics: Science and Systems, 2025.
3. [3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, 2023.
4. [4] Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. \pi_{0.5}: a vision-language-action model with open-world generalization. In Conference on Robot Learning, 2025.
5. [5] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 IEEE International Conference on Robotics and Automation (ICRA), pages 8943–8950. IEEE, 2019.
6. [6] H. Li, Y. Zhang, J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. In Conference on Robot Learning, 2022.
7. [7] S. Jiang, S. Zhao, Y. Fan, and P. Yin. GelFusion: Enhancing robotic manipulation under visual constraints via visuotactile fusion. arXiv preprint arXiv:2505.07455, 2025.
8. [8] B. Huang, Y. Wang, X. Yang, Y. Luo, and Y. Li. 3D-ViTac: Learning fine-grained manipulation with visuo-tactile sensing. In Conference on Robot Learning, 2024.
9. [9] B. Huang, J. Xu, I. Akinola, W. Yang, B. Sundaralingam, R. O’Flaherty, D. Fox, X. Wang, A. Mousavian, Y.-W. Chao, et al. VT-Refine: Learning bimanual assembly with visuo-tactile feedback via simulation fine-tuning. In Conference on Robot Learning, 2025.
10. [10] Z. He, H. Fang, J. Chen, H.-S. Fang, and C. Lu. FoAR: Force-aware reactive policy for contact-rich robotic manipulation. IEEE Robotics and Automation Letters, 10(6):5625–5632, 2025.
11. [11] T. Ablett, O. Limoyo, A. Sigal, A. Jilani, J. Kelly, K. Siddiqi, F. Hogan, and G. Dudek. Multimodal and force-matched imitation learning with a see-through visuotactile sensor. IEEE Transactions on Robotics, 41:946–959, 2025.
12. [12] X. Zhang, L. Sun, Z. Kuang, and M. Tomizuka. Learning variable impedance control via inverse reinforcement learning for force-related tasks. IEEE Robotics and Automation Letters, 6(2):2225–2232, 2021.
13. [13] P. Hao, C. Zhang, D. Li, X. Cao, X. Hao, S. Cui, and S. Wang. TLA: Tactile-language-action model for contact-rich manipulation. arXiv preprint arXiv:2503.08548, 2025.
14. [14] Y. Li, Zhaxizhuoma, H. Jiang, J. Xia, H. Zhang, J. Du, Y. Zhou, J. Zeng, C. Hao, J. Ren, Q. Yu, et al. ForceVLA2: Unleashing hybrid force-position control with force awareness for contact-rich manipulation. arXiv preprint arXiv:2603.15169, 2026.
15. [15] R. Zhao, W. Wang, Y. Ma, X. Li, F. E. Tay, M. H. Ang, Jr., and H. Zhu. FD-VLA: Force-distilled vision-language-action model for contact-rich manipulation. arXiv preprint arXiv:2602.02142, 2026.
16. [16] J. Huang, S. Wang, F. Lin, Y. Hu, C. Wen, and Y. Gao. Tactile-VLA: Unlocking vision-language-action model’s physical knowledge for tactile generalization. arXiv preprint arXiv:2507.09160, 2025.
17. [17] J. Bi, K. Y. Ma, C. Hao, M. Z. Shou, and H. Soh. VLA-Touch: Enhancing vision-language-action models with dual-level tactile feedback. IEEE Robotics and Automation Letters, 2026.
18. [18] J. Yu, H. Liu, Q. Yu, J. Ren, C. Hao, H. Ding, G. Huang, G. Huang, Y. Song, P. Cai, W. Zhang, and C. Lu. ForceVLA: Enhancing VLA models with a force-aware MoE for contact-rich manipulation. In Advances in Neural Information Processing Systems, volume 38, pages 93409–93439, 2025.
19. [19] Z. Zhang, H. Xu, Z. Yang, C. Yue, Z. Lin, H.-a. Gao, Z. Wang, and H. Zhao. TA-VLA: Elucidating the design space of torque-aware vision-language-action models. In Conference on Robot Learning, 2025.
20. [20] Y. Li, P. Tang, W. Zhang, C. Zhu, Y. Duan, W. Shi, X. Zhang, Z. Yang, J. Ji, and Y. Zhang. FAVLA: A force-adaptive fast-slow VLA model for contact-rich robotic manipulation. arXiv preprint arXiv:2602.23648, 2026.
21. [21] H. Fang, S. Tang, M. Mei, et al. Force policy: Learning hybrid force-position control policy under interaction frame for contact-rich manipulation. In Proceedings of Robotics: Science and Systems, 2026.
22. [22] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver IO: A general architecture for structured inputs & outputs. In International Conference on Learning Representations, 2022.
23. [23] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, volume 35, pages 23716–23736, 2022.
24. [24] J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730–19742. PMLR, 2023.
25. [25] M. H. Raibert and J. J. Craig. Hybrid position/force control of manipulators. Journal of Dynamic Systems, Measurement, and Control, 103(2):126–133, 1981.
26. [26] N. Hogan. Impedance control: An approach to manipulation: Part I—theory. Journal of Dynamic Systems, Measurement, and Control, 107(1):1–7, 1985.
27. [27] H. Zhu, X. Li, W. Chen, and C. Zhang. Flexure-based variable stiffness gripper for large-scale grasping force regulation with vision. In Intelligent Robotics and Applications (ICIRA), pages 346–357. Springer, 2019.
28. [28] H. Zhu, X. Li, W. Chen, X. Li, J. Ma, C. S. Teo, T. J. Teo, and W. Lin. Weight imprinting classification-based force grasping with a variable-stiffness robotic gripper. IEEE Transactions on Automation Science and Engineering, 19(2):969–981, 2022.
29. [29] H. Xue, J. Ren, W. Chen, G. Zhang, Y. Fang, G. Gu, H. Xu, and C. Lu. Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation. In Proceedings of Robotics: Science and Systems, 2025.
30. [30] A. Ranjbar, N. A. Vien, H. Ziesche, J. Boedecker, and G. Neumann. Residual feedback learning for contact-rich manipulation tasks with uncertainty. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2383–2390, 2021.
31. [31] X. Zhang, C. Wang, L. Sun, Z. Wu, X. Zhu, and M. Tomizuka. Efficient sim-to-real transfer of contact-rich manipulation skills with online admittance residual learning. In Conference on Robot Learning, 2023.
32. [32] W. Liu, J. Wang, Y. Wang, W. Wang, and C. Lu. ForceMimic: Force-centric imitation learning with force-motion capture system for contact-rich manipulation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 1105–1112. IEEE, 2025.
33. [33] M. Bogdanovic, M. Khadiv, and L. Righetti. Learning variable impedance control for contact sensitive tasks. IEEE Robotics and Automation Letters, 5(4), 2020.
34. [34] R. Martín-Martín, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
35. [35] J. J. Liu, Y. Li, K. Shaw, T. Tao, R. Salakhutdinov, and D. Pathak. FACTR: Force-attending curriculum training for contact-rich policy learning. In Proceedings of Robotics: Science and Systems, 2025.
36. [36] Y. Zhang, Y. Wang, X. Sun, K. Huang, Z. Xu, J. Ji, Z. Che, J. Tang, and J. Sun. CRAFT: Adapting VLA models to contact-rich manipulation via force-aware curriculum fine-tuning. arXiv preprint arXiv:2602.12532, 2026.
37. [37] C. Zhang, P. Hao, X. Cao, X. Hao, S. Cui, and S. Wang. VTLA: Vision-tactile-language-action model with preference learning for insertion manipulation. arXiv preprint arXiv:2505.09577, 2025.
38. [38] Z. Cheng, Y. Zhang, A. Tang, K. Wang, W. Zhang, H. Li, H. Zhang, and L. Song. OmniVTLA: Vision-tactile-language-action models with semantic-aligned tactile sensing. arXiv preprint arXiv:2508.08706, 2025.
39. [39] K. Gubernatorov, M. Sannikov, I. Mikhalchuk, E. Kuznetsov, M. Artemov, O. F. Oluwatobi, M. Fernando, A. Asanov, Z. Guo, and D. Tsetserukou. HapticVLA: Contact-rich manipulation via vision-language-action model without inference-time tactile sensing. arXiv preprint arXiv:2603.15257, 2026.
40. [40] W. Wang, R. Zhao, T. J. Teo, M. H. Ang, Jr., and H. Zhu. From dream to action: Hierarchical policy learning with 3D world imagination for robotic manipulation. In 2026 IEEE International Conference on Robotics and Automation (ICRA), pages 1155–1162. IEEE, 2026.
41. [41] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
42. [42] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
43. [43] Y. Wang, P. Ding, L. Li, C. Cui, Z. Ge, X. Tong, W. Song, H. Zhao, W. Zhao, P. Hou, et al. VLA-Adapter: An effective paradigm for tiny-scale vision-language-action model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 18638–18646, 2026.

Appendix A Method Details

A.1 Admission-Conditioned Attention

Eqs. 1–5 define phase inference, admission, and joint attention. We use K_{p}=3 for Gear Assembly and Switch Cable Connection, and 2 otherwise, as shown in Table 5. The gate map is initialized near the calibrated baseline, without an initial preference for promotion or suppression. The positive scale in Eq. 2 is parameterized as

s_{\mathrm{adm}}=\alpha_{g}\exp(\sigma_{\mathrm{adm}}),

where \alpha_{g} is the multiplier and \sigma_{\mathrm{adm}} is the learned log-scale. Before joint attention, force token features are normalized:

\bar{h}^{w}_{t}=\mathrm{LN}(P_{w}(h^{w}_{t})).

The projector P_{w} and LN align force features with normalized visual-language tokens.

A.2 Force Tokenizer Implementation

The force representation uses two coupled but distinct pathways. A continuous force history encoder produces the force tokens consumed by the action expert, while a pretrained VQ tokenizer maps a recent wrench window to a discrete contact-dynamic code.

The tokenizer operates on a history of L_{\tau} frames of 6-DoF wrench measurements. Each window is flattened into a 6L_{\tau}-dimensional vector and standardized using per-dimension training set statistics. Let W_{t-L_{\tau}+1:t}\in\mathbb{R}^{L_{\tau}\times 6} denote the wrench window. The tokenizer input is

x^{\tau}_{t}=\frac{\mathrm{vec}(W_{t-L_{\tau}+1:t})-\mu_{\tau}}{\max(\sigma_{\tau},0.1)}\in\mathbb{R}^{6L_{\tau}},

where division and the standard-deviation floor are applied elementwise, and \mu_{\tau} and \sigma_{\tau} are computed from the training set. The reconstruction target is this normalized 6L_{\tau}-dimensional window.

The VQ module is trained separately from the VLA policy with a reconstruction objective and a commitment term, while the codebook is maintained by an exponential-moving-average update. During VLA training, the tokenizer is kept frozen and is used only to provide the selected code embedding for each force history. The encoder maps x^{\tau}_{t} to a latent vector z_{t}=E_{\tau}(x^{\tau}_{t}). Given a codebook \mathcal{C}_{\tau}=\{c^{\tau}_{k}\}_{k=1}^{K_{\tau}}, the selected contact code is

k_{t}=\arg\min_{k}\|z_{t}-c^{\tau}_{k}\|_{2},\qquad z^{q}_{t}=z_{t}+(c^{\tau}_{k_{t}}-z_{t})_{\mathrm{sg}},

where (\cdot)_{\mathrm{sg}} denotes the stop-gradient straight-through operation. The decoder reconstructs the normalized wrench window vector, \hat{x}^{\tau}_{t}=D_{\tau}(z^{q}_{t}), and the tokenizer is pretrained with

\mathcal{L}_{\tau}=\|\hat{x}^{\tau}_{t}-x^{\tau}_{t}\|_{2}^{2}+\lambda_{\mathrm{com}}\|z_{t}-(c^{\tau}_{k_{t}})_{\mathrm{sg}}\|_{2}^{2}.

The encoder, decoder, and codebook configurations are given in Appendix B.1.2.

The selected code embedding conditions the continuous force token through FiLM modulation: a weighted code embedding predicts scale and shift terms that are applied to the projected and normalized force history token. The FiLM projection is a linear layer with zero-initialized weights and bias, so the initial FiLM transformation is the identity. The action expert initially receives the same continuous force representation and gradually learns to use the contact-dynamic code as a regime-dependent adjustment. This keeps fine wrench details in the continuous token while using the discrete code to provide a stable contact regime context. Let r_{t}=F_{w}(W_{t-L+1:t}) be the continuous force history representation and let \bar{r}_{t}=\mathrm{LN}(P_{w}(r_{t})) be its projected token. The selected code embedding is scaled as e_{t}=w_{t}c^{\tau}_{k_{t}}, where w_{t} is a scalar weight on the code embedding. The FiLM parameters are then computed as

[\gamma_{t},\beta_{t}]=\mathrm{Linear}_{\mathrm{film}}(e_{t}),\qquad h^{w}_{t}=\bar{r}_{t}\odot(1+s_{\mathrm{film}}\gamma_{t})+s_{\mathrm{film}}\beta_{t}.

We use s_{\mathrm{film}}=1, with no nonlinear activation applied to the predicted FiLM parameters. The resulting h^{w}_{t} is the force token passed to the action bridge and then regulated by the source-level admission mechanism in Eqs. 4–5.

A.3 Offline Motion Phase Label Generation

Phase labels are generated offline from demonstration force measurements and gripper states using a contact detection procedure shared across tasks. Task-specific configurations and parameter values are given in Appendix B.3.1.

Inputs and manipulation segments.

The detector uses the raw three-dimensional force F_{t}\in\mathbb{R}^{3} from the robot’s force/torque sensor, with no preliminary baseline subtraction. Gripper states are used to partition trajectories into manipulation segments. A transition of the gripper from closed to open ends a segment; closed corresponds to a raw reading greater than q_{\mathrm{raw}} or a normalized reading greater than q_{\mathrm{norm}}. A trajectory with unchanged gripper states forms one segment. Gripper values do not enter the contact event decision. The three-dimensional force used for labeling is distinct from the six-dimensional wrench input to the Force Tokenizer.

Adaptive reference and event detection.

At a sampling rate f_{s}, the free motion reference F_{\mathrm{ref}} is the componentwise median of the most recent N_{\mathrm{ref}} non-contact force samples. The detector computes

d_{t}=\|F_{t}-F_{\mathrm{ref}}\|_{2}.

An event starts when d_{t}\geq\theta_{\mathrm{on}} for N_{\mathrm{on}} consecutive frames and ends when d_{t}<\theta_{\mathrm{off}} for N_{\mathrm{off}} consecutive frames. The reference is held fixed during contact, except for a steady load update: when the standard deviation of \|F_{t}\|_{2} over the most recent N_{\mathrm{steady}} frames is below \sigma_{\mathrm{steady}}, the stable interval is incorporated into the reference. This update accommodates persistent loads such as cable or seat belt tension. All event detection parameters are shared across tasks.

Major event selection and contact intervals.

Within each manipulation segment, let m_{j} denote the peak force deviation of event j, and let m_{\max} be the largest event peak in that segment. An event is retained as a major event when

m_{j}\geq\theta_{\mathrm{major}},\qquad m_{j}\geq\rho m_{\max}.

The contact label begins at the onset of the first major event. Its endpoint is the last frame in the segment whose force deviation from that event’s pre-contact reference is at least \theta_{\mathrm{off}}. The resulting interval includes intermediate retractions, jamming, and brief force-free gaps. Thus, individual event boundaries and the full contact label interval serve different roles.

Phase assignment.

Frames within the resulting interval receive the contact label; the remaining frames receive the free label. For tasks with an approach phase, the N_{\mathrm{approach}} frames immediately preceding contact are instead labeled approach, clipped to the current manipulation segment. These tasks use labels 0, 1, and 2 for free, approach, and contact, respectively. The other tasks use free and contact as their two training categories. The phase head is supervised with these task-specific categories and predicts phases from visual-language context at inference time.

Appendix B Experimental Details and Analysis

B.1 Experimental Setup

B.1.1 Task Setup

We evaluate FR-VLA on five contact-rich manipulation tasks, using the platforms shown in Figure 7: Gear Assembly, Switch Cable Connection, Curved Surface Cleaning, Screw Tightening, and Seat Belt Buckle Insertion. Demonstrations are collected with a 3D mouse interface. The operator commands the desired end-effector motion, while the robot executes the command through a Cartesian impedance controller rather than a stiff position-tracking controller. In end-effector coordinates, the commanded interaction behavior can be written as

\tau=J(q)^{\top}\left[K_{x}(x_{d}-x)+D_{x}(\dot{x}_{d}-\dot{x})\right]+\tau_{\mathrm{bias}},

where x_{d} is the 3D-mouse-driven target pose, x is the measured end-effector pose, J(q) is the robot Jacobian, and K_{x},D_{x} define the Cartesian impedance. This compliant data collection mode encourages smooth contact behavior and reduces the effect of high-frequency human teleoperation noise, which is especially important in precision contact tasks where abrupt micro-motions can trigger protective stops. During data collection, images from both cameras are recorded at 10 Hz, while force, robot state, and actions are recorded at 50 Hz; for training, the force, state, and action streams are aligned to the image timestamps.

Gear Assembly requires the robot to align a gear with a constrained peg and complete insertion after contact. The task stresses the transition from visual approach to force-guided alignment, where small changes in wrench dynamics indicate whether the gear is seated correctly.

Switch Cable Connection contains two evaluated stages. In the top Ethernet cable insertion stage, the target port location is varied and the robot must localize the changed receptacle before insertion. In the side power connector insertion stage, the robot inserts a plug into a lower side connector where visual evidence is partially occluded and contact feedback is needed once the plug begins to engage.

Curved Surface Cleaning requires sustained contact while following a curved surface. Unlike insertion tasks, the informative force signal is not only the first contact event; the policy must maintain contact without overreacting to stable baseline forces.

Screw Tightening requires alignment, engagement, and rotational tightening. The task depends on recognizing the transition between light exploratory contact and successful thread engagement before applying the tightening motion.

Seat Belt Buckle Insertion requires pushing the buckle tongue into a latch under limited visual observability near the final engagement point. The task combines constrained insertion with a short contact transition at successful latch engagement.

In all tasks, a trial is counted as successful only when the specified stage objective is completed without human intervention.

B.1.2 Model and Training Parameters

FR-VLA predicts action chunks of H=8 steps, where each action is a 7-dimensional absolute end-effector pose with a gripper command, and the action expert receives a wrench history of L=16 frames. The phase loss weight in Eq. 6 is \lambda_{p}=0.5. The Force Tokenizer uses lightweight MLP encoder and decoder networks with a codebook of K_{\tau}=8 code embeddings. Its input is a flattened window of L_{\tau}=8 frames of 6D wrench measurements across all tasks. The tokenizer is pretrained per task on that task’s demonstration wrench histories and then frozen, while the action interface and phase loss weight are shared across tasks. Phase head category counts are specified in Table 5. At inference time, the frozen tokenizer and FiLM conditioning run as part of the policy forward pass, whereas the segmentation and contact event detection used for phase labeling are applied only offline. Training uses two NVIDIA RTX A6000 GPUs, and inference runs on a single NVIDIA RTX 3090 GPU.

B.2 Wrench Magnitude Gating Baseline

Figure 8 shows a simpler wrench magnitude gating baseline on Gear Assembly, in which the admission signal is computed from the measured wrench amplitude without phase supervision. In this design, the influence of force is determined by the measured wrench itself: it rises only after a large transient has occurred and falls back once the magnitude decreases. The policy can therefore only react passively to force feedback after contact events, and cannot adjust its reliance on force in anticipation of an upcoming interaction or maintain it when decision-relevant changes have small magnitude, such as the force reduction during corrective motion. In contrast, FR-VLA infers the interaction phase from visual-language context and uses it to set the admission signal. Force eligibility can thus be raised as the task enters a contact-relevant phase, sustained across the whole window, including intervals where the wrench magnitude is small, and suppressed once the phase no longer requires physical feedback. This allows FR-VLA to actively regulate how strongly force influences action generation, rather than only responding to the magnitude of the current measurement.

B.3 Analysis of Force Admission Behavior

B.3.1 Phase Supervision Configuration

Gear Assembly and Switch Cable Connection use three training classes (free, approach, and contact), while the remaining tasks use two (free and contact), as shown in Table 5. Switch Cable Connection contains two manipulation segments; each remaining task contains one.

Hyperparameter settings.

We use f_{s}=10\,\mathrm{Hz}, N_{\mathrm{ref}}=10, \theta_{\mathrm{on}}=5\,\mathrm{N}, and \theta_{\mathrm{off}}=2.5\,\mathrm{N}. The lower release threshold provides hysteresis, while N_{\mathrm{on}}=3 and N_{\mathrm{off}}=5 require persistence for 0.3 s and 0.5 s, respectively, to suppress short-lived fluctuations. The steady load update uses N_{\mathrm{steady}}=15 (1.5 s) and \sigma_{\mathrm{steady}}=1.5\,\mathrm{N}. Major events use \theta_{\mathrm{major}}=8\,\mathrm{N} and \rho=0.5, combining a minimum peak deviation with a segment-relative criterion. The approach window is N_{\mathrm{approach}}=10 (1 s). Gripper thresholds are q_{\mathrm{raw}}=100 and q_{\mathrm{norm}}=0.5. These settings are fixed across tasks; only the phase vocabulary differs, as shown in Table 5.

Table 6 reports the training label distribution after merging approach into free. Switch Cable Connection is reported as one task, aggregating its two manipulation segments.

B.3.2 Evaluation Protocol for Phase Inference

During each rollout, the phases predicted by the policy from visual-language context are logged online together with the recorded observations, force measurements, and gripper states. Ground-truth labels are then obtained post hoc by applying the offline labeling procedure in Appendix A.3 to the recorded rollout data, and phase inference accuracy is computed by comparing the logged predictions with these labels.

For consistent reporting across tasks, we merge approach into free when computing phase label distributions and phase inference accuracy. For three-class models, the predicted free and approach probabilities are summed to form the non-contact probability before computing binary balanced accuracy. Specifically, \hat{p}_{\mathrm{free}}^{\mathrm{eval}}=\hat{p}_{\mathrm{free}}+\hat{p}_{\mathrm{approach}} and \hat{p}_{\mathrm{contact}}^{\mathrm{eval}}=\hat{p}_{\mathrm{contact}}. The larger of these two probabilities determines the predicted evaluation class; ground-truth approach labels are mapped to free. Balanced accuracy is the mean recall of the resulting free and contact classes. This aggregation is used for reporting; phase supervision and gating retain the task-specific phase vocabulary.

B.3.3 Force Admission across Tasks

Figures 9 and 10 show the predicted phases, wrench measurements, selected force codes, and force admission signals on all five tasks. In Gear Assembly, Screw Tightening, and Seat Belt Buckle Insertion, admission rises at contact onset and remains high throughout the predicted contact phase, through alignment, engagement, and seating, whether the measured wrench is large or close to zero. In Gear Assembly, where the rollout ends in free motion, admission returns to a lower level once the phase switches back to free. Switch Cable Connection shows the same pattern in each of its two stages, with admission dropping during the transfer between the Ethernet cable and power connector insertions. Curved Surface Cleaning exhibits the opposite behavior: admission is suppressed throughout the long period of stable contact and recovers once contact is released, since the steady wrench confirms ongoing contact but carries little new information for action selection. Across tasks, changes in admission coincide with predicted phase boundaries rather than with wrench peaks, while the selected code typically remains stable during free motion and switches more often during contact, reflecting the distinct contact regimes encountered during interaction.

B.4 Force Code Usage

Figure 11 relates the selected force codes to interaction events in each task. The events describe end-effector motion during contact: advance moves along the insertion direction; slide moves tangentially under load, as in wiping or alignment; jam pushes under high load without progressing and is followed by a retraction; hold stays nearly still under high load, as when pressing, tightening, or seating; and withdraw retracts while contact is maintained. Each cell reports the lift P(\mathrm{event}\mid\mathrm{code})/P(\mathrm{event}): values above 1 indicate that the event occurs more often than average when the code is selected, and values below 1 indicate that it occurs less often.

Because the Force Tokenizer is trained separately for each task, the same code index carries a task-specific meaning. Within each task, individual codes are concentrated on particular interaction events. In Switch Cable Connection, code 7 is enriched for hold and depleted for slide. In Screw Tightening, code 2 is enriched for jam, whereas codes 5 and 7 are enriched for withdraw. In Seat Belt Buckle Insertion, code 4 is enriched for hold, and in Curved Surface Cleaning, code 6 is strongly enriched for advance. The association is weaker in Gear Assembly, where code 1 is mainly associated with withdraw. Codes therefore distinguish interaction events within the contact phase, such as jamming, holding, and withdrawing, rather than representing contact as a single state.

B.5 Robustness to Object and Position Perturbations

Figure 12 illustrates the perturbation settings. Figure 12(a) shows the two groups of unseen target peg positions in Gear Assembly, which shift where the gear must be aligned before contact-guided insertion. Figure 12(b) shows the two unseen gears, whose different geometry changes the contact conditions and shifts the force baseline during insertion. Figure 12(c) shows the four groups of unseen initial Ethernet cable positions in Switch Cable Connection, which change the approach direction and contact angle during top port insertion.

These perturbations affect force feedback in different ways. Unseen gears mainly alter the wrench observed during engagement and seating, so a policy that conditions directly on raw wrench values encounters force patterns outside its training distribution. Target position changes mainly affect the visual approach and the timing of first contact. Because admission is tied to the inferred phase rather than to a fixed point in the trajectory, force is still promoted once contact begins at the new location. Cable position changes alter both the approach geometry and, through the deformable cable, the forces transmitted to the connector, producing visual and force conditions not seen during training. FR-VLA succeeds in every cable position group, demonstrating that phase-grounded force admission generalizes to unseen initial configurations. Although the cable configuration changes the appearance of the approach as well as the magnitude and direction of contact forces, the transition from approach to plug–port contact remains recognizable from visual-language context. The policy therefore still promotes force at contact onset and uses the measured wrench to complete the final alignment and insertion under contact conditions that differ from the demonstrations.