TacEx, 촉각 불확실성 가중으로 로봇 탐색을 접촉 중심으로 유도

Abstract

Mastering robot manipulation skills via reinforcement learning (RL) remains largely sample-inefficient. The most common RL algorithms rely on random action sampling to discover new strategies, resulting in agents that allocate most of their training budget to motions in free space, away from the contacts from which manipulation skills emerge. Existing intrinsic motivation methods based on model disagreement or epistemic uncertainty improve on isotropic noise, but they can also reward uncertainty in functionally irrelevant transitions, such as erratic motions in free space. In this work, we argue that tactile feedback provides a natural signal for exploration, and introduce TacEx, a framework that incorporates touch into epistemic uncertainty-driven exploration by decomposing model uncertainty across sensory modalities and directing curiosity toward the tactile channel. By anchoring curiosity to the sense of touch, TacEx drives the robot to discover complex contact dynamics, learning to manipulate and grasp objects without task rewards or expert demonstrations during exploration. The interaction-dense dataset collected through this tactile-driven curiosity supports offline learning of downstream pick-and-place policies without additional environment interaction. We further use tactile-driven exploration to post-train vision-language-action (VLA) models. Although the VLAs are initially pre-trained without tactile feedback, post-training with TacEx substantially improves downstream performance while remaining highly sample-efficient.

Keywords: Tactile sensing, exploration, post-training, reinforcement learning

1 Introduction

Human babies learn to manipulate the world through touch [14]. Long before they can name an object or reason about its dynamics, they grasp, press, and reorient, and it is through these contacts that complex manipulation skills emerge. In fact, contact is the central piece of manipulation physics, and any agent that hopes to learn dexterous behavior must learn to seek out and understand contact.

By contrast, reinforcement learning typically discovers behavior by sampling actions isotropically. The consequence is an agent that allocates most of its training budget to motions in free space, far from the contacts that matter. In manipulation, the informative region of the state space is vanishingly small relative to its volume: a gripper a centimeter above an object and a gripper pressing into it occupy nearby states but induce entirely different dynamics, and only the latter teaches the agent anything about grasping. Random exploration is poorly equipped to discover this region, and once found, poorly rewarded to stay there.

Intrinsic motivation [2] methods based on forward dynamics disagreement or epistemic uncertainty improve substantially on undirected noise [31, 22, 27, 32, 33], but their curiosity remains agnostic to contact. The agent is rewarded for resolving uncertainty wherever it happens to lie, and is just as content hovering a manipulator through unexplored free space as it is dwelling on a contact to explore its dynamics. Worse, because interaction signals are sparse and arise only upon contact, the states an agent most needs to be curious about are precisely the ones it visits least.

We argue that tactile feedback is the natural prior for where curiosity should point. The agent should not seek one-off contact, nor should it seek uncertainty uniformly across the state space; it should seek uncertainty in the tactile channel, and across the contact modes that touch exposes. This reframes exploration around a simple inductive bias: the value of resolving uncertainty is not uniform across the senses, and the sense of touch is the one whose uncertainty most reliably indicates that something worth learning is taking place.

We introduce TacEx, a framework that builds on this idea (fig. 1). Rather than treating epistemic uncertainty as a single scalar over the full state, TacEx decomposes it across sensory modalities and introduces a per-modality weighting on the resulting uncertainty terms. The key inductive bias is that the tactile component must be represented and weighted in the bonus, rather than, as in standard uncertainty-driven exploration, folding touch into an undifferentiated quantity where its contribution is diluted. With touch given non-negligible weight, the exploration bonus drives the agent toward contact-rich regions of the state space and rewards it for resolving the dynamics it finds there. The result is curiosity anchored to the sense of touch: the robot is drawn to contact, and it learns to grasp and manipulate objects without task rewards or expert demonstrations during exploration.

This formulation yields three contributions. First, we introduce a modality-weighted exploration objective, motivated by information-gain exploration, that explicitly prioritizes tactile predictive uncertainty. Second, a pure tactile-driven exploration phase learns emergent grasping behaviors entirely from scratch, and the interaction-dense dataset it collects supports offline learning of downstream pick-and-place policies without additional environment interaction. Third, the same principle makes the post-training of vision-language-action (VLA) models interaction-driven and sample-efficient: although these models are pre-trained without any tactile feedback, post-training with TacEx substantially improves downstream performance while remaining highly sample-efficient. We evaluate these claims in two simulation settings: reward-free exploration from scratch and sample-efficient post-training of frozen VLAs.

2 Background

In the following, we summarize a few key exploration strategies for RL and prior works that leverage tactile sensing for robot learning. A more detailed list of related work is also provided in appendix A in the appendix.

2.1 The Exploration-Exploitation Dilemma in RL

The exploration-exploitation trade-off represents a crucial challenge in RL. Should the agent exploit its current knowledge to greedily maximize the rewards or explore new policies in the hope of discovering a better solution? Common exploration strategies in RL are \epsilon-greedy, Boltzmann exploration, and maximizing information gain. We discuss these in the following.

Problem Setting

We consider an infinite-horizon Markov decision process [24, MDP,], defined by the tuple ({\mathcal{S}},{\mathcal{A}},p,\gamma,r,\rho). The state and action spaces are assumed to be continuous, i.e., {\mathcal{S}}\subset\mathbb{R}^{d_{s}},{\mathcal{A}}\subset\mathbb{R}^{d_{a}}, and the unknown transition kernel p:{\mathcal{S}}\times{\mathcal{S}}\times{\mathcal{A}}\to[0,\infty) represents the probability density of the next state {\bm{s}}_{t+1}\in{\mathcal{S}} given the current state {\bm{s}}_{t}\in{\mathcal{S}} and action {\bm{a}}_{t}\in{\mathcal{A}}. At step t, the agent observes the state {\bm{s}}_{t} and samples an action {\bm{a}}_{t} from the policy {\bm{\pi}}:{\mathcal{A}}\times{\mathcal{S}}\to[0,\infty), {\bm{a}}_{t}\sim{\bm{\pi}}({\bm{a}}|{\bm{s}}_{t}). Following the execution of the action {\bm{a}}_{t} in the environment, the agent receives a reward r:{\mathcal{S}}\times{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R}_{+} and the goal of the agent is to maximize the discounted sum of rewards.

\max_{{\bm{\pi}}\in\Pi}J({\bm{\pi}})=\max_{{\bm{\pi}}\in\Pi}\mathbb{E}_{{\bm{s}}_{0},{\bm{a}}_{0},\dots}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right].

The state-action critic Q^{{\bm{\pi}}} and the value function V^{\pi} are defined as:

Q^{{\bm{\pi}}}({\bm{s}}_{t},{\bm{a}}_{t})=\mathbb{E}_{{\bm{s}}_{t+1},{\bm{a}}_{t+1}\sim{\bm{\pi}},\dots}\left[\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}\right],\;V^{{\bm{\pi}}}({\bm{s}}_{t})=\mathbb{E}_{{\bm{a}}_{t}\sim{\bm{\pi}},{\bm{s}}_{t+1},{\bm{a}}_{t+1}\sim{\bm{\pi}},\dots}\left[\sum_{l=0}^{\infty}\gamma^{l}r_{t+l}\right].

\epsilon–greedy exploration

A common strategy to balance exploration and exploitation is \epsilon-greedy [16, 21, 38], where with probability \epsilon, the agent selects a random action {\bm{a}}_{t}\sim\text{Unif}({\mathcal{A}}) to encourage exploration and otherwise acts greedily, i.e. a=\arg\max_{a\in{\mathcal{A}}}\;Q^{*}({\bm{s}}_{t},{\bm{a}}). Here Q^{*} is the estimate of the optimal state-action value function. For continuous action spaces, Lillicrap et al. [19], Fujimoto et al. [8] use a deterministic policy {\bm{\pi}}_{\theta}, which maximizes the value function, in combination with random Gaussian noise to encourage exploration.

Boltzmann Exploration

Many RL algorithms are based on Boltzmann exploration [34, 35]. Here the policy {\bm{\pi}} follows the distribution below

{\bm{\pi}}({\bm{a}}|{\bm{s}})=Z^{-1}({\bm{s}})\exp\left(\alpha^{-1}Q^{{\bm{\pi}}}({\bm{s}},{\bm{a}})\right),

where Z^{-1}({\bm{s}}) is a normalization term, \alpha is the temperature parameter that regulates exploration and Q^{{\bm{\pi}}} is the soft-Q function. Haarnoja et al. [9] show that eq. 2 corresponds to maximizing the entropy weighted reward \tilde{r}({\bm{s}},{\bm{a}},{\bm{s}}^{\prime})=r({\bm{s}},{\bm{a}},{\bm{s}}^{\prime})+\alpha\mathcal{H}\left({\bm{\pi}}({\bm{a}}|{\bm{s}})\right).

For \alpha\to 0, the policy acts greedily and maximizes Q^{{\bm{\pi}}}({\bm{s}},{\bm{a}}), and for \alpha\to\infty the policy explores uniformly, adding equal mass to all actions in {\mathcal{A}}. Intuitively, Boltzmann exploration can be interpreted as a smoother alternative to \epsilon–greedy, with \alpha serving a similar role to \epsilon in controlling the degree of exploration.

Both \epsilon–greedy and Boltzmann exploration fail to account for the agent’s “lack of knowledge” and do not encourage directed exploration strategic for the task. The agent explores by sampling random action sequences, which leads to suboptimal performance, particularly in challenging exploration tasks with continuous state-action spaces.

Maximizing Information Gain

Sukhija et al. [31] propose learning an uncertainty-aware dynamics model of the MDP and utilize the epistemic uncertainty or model disagreement {\bm{\sigma}}_{n}({\bm{s}},{\bm{a}})\in\mathbb{R}_{+}^{d_{{\bm{s}}}} as an intrinsic reward bonus for directing exploration. They propose the following modification to the Boltzmann distribution above

{\bm{\pi}}({\bm{a}}|{\bm{s}})\propto\exp\left(\alpha^{-1}\left(Q^{{\bm{\pi}}}({\bm{s}},{\bm{a}})+\lambda\left\|{\bm{\sigma}}_{n}({\bm{s}},{\bm{a}})\right\|\right)\right).

Effectively, the agent explores by not only applying random actions but also visiting new state-action tuples that exhibit high epistemic uncertainty. This exploration strategy enables principled and directed exploration, enjoying strong theoretical guarantees on the agent’s convergence [31, 33, 3]. Similar to eq. 2, eq. 3 can be viewed as maximizing the reward \tilde{r}({\bm{s}},{\bm{a}},{\bm{s}}^{\prime})=r({\bm{s}},{\bm{a}},{\bm{s}}^{\prime})+\alpha\mathcal{H}\left({\bm{\pi}}({\bm{a}}|{\bm{s}})\right)+\lambda\left\|{\bm{\sigma}}_{n}({\bm{s}},{\bm{a}})\right\|. Sukhija et al. [31] also propose how to automatically select the temperature parameters (\alpha,\lambda) and show that eq. 3 outperforms state-of-the-art RL algorithms for both state-based and visual control tasks.

2.2 Tactile Sensing

Tactile feedback can be available in many forms, varying widely in the richness of the signal they provide. At the lowest end of the spectrum are binary contact sensors [42], which report only whether contact is present, and force-torque sensors, which measure aggregate wrench at a single point but discard any spatial information about where contact occurs. In this work, we focus on high-dimensional tactile feedback, which, much like human skin, resolves contact spatially and reports the distribution of pressure and shear across the sensing surface.

The most common forms of high-dimensional tactile feedback are tactile images [43] and force maps [30], both of which are naturally interpretable as standard images and can therefore be processed with off-the-shelf vision architectures. Here we use force maps: a spatial binning of the contact patch in which each cell records the local pressure and shear forces, represented as a three-dimensional force vector. Because both representations share this image-like structure, force maps can be readily swapped for tactile images from vision-based sensors with little change to the rest of the pipeline, although its effectiveness with tactile images remains to be evaluated. We adopt force maps because they are convenient in practice: they are computable from real-world tactile sensors [30, 28] and readily available in common simulation frameworks [29, 1], which makes them well suited for studying tactile-driven exploration in both settings.

2.3 Touch-guided exploration

Prior work already uses touch to guide exploration. Vulin et al. [39] reward crossing a cumulative contact-force threshold and prioritize contact-rich replay; Huang et al. [11] combine impact penalties with surprise from an impact-penalty predictor; and Rajeswar et al. [25] combine visual-to-haptic prediction error with visual forward-model error. Our contribution is to explicitly weight modality-specific ensemble disagreement, including uncertainty over spatial tactile representations, and study its effect on exploration and downstream learning. Spatial normal/shear force maps retain contact-location and force-distribution information lost by aggregation.

3 TacEx: Tactile Exploration Drives Efficient Learning

A key limitation of the objective in eq. 3 is that it aggregates predictive uncertainty across the state representation without explicitly controlling each modality’s contribution. Since {\bm{\sigma}}_{n}({\bm{s}},{\bm{a}})\in\mathbb{R}_{+}^{d_{{\bm{s}}}}, maximizing the norm \left\|{\bm{\sigma}}_{n}({\bm{s}},{\bm{a}})\right\| implicitly encourages exploration uniformly in every direction of the state space. However, in many robotic systems, effective exploration is modality-dependent and may be concentrated in only a subset of the state space.

For instance, in robotic manipulation tasks, exploration of states associated with physical interaction between the manipulator and the object is substantially more informative than exploration in regions where the manipulator remains in free space. To incorporate this inductive bias, we augment the state representation with tactile information and introduce a weighting vector that prioritizes uncertainty in task-relevant dimensions. In particular, let {\bm{v}}\in\mathbb{R}^{d_{{{\bm{v}}}}} be the visual state representation of the robot extracted from images, \bm{\iota}\in\mathbb{R}^{d_{\bm{\iota}}} the tactile state representation extracted from the tactile sensors, and {\bm{z}}\in\mathbb{R}^{d_{{\bm{z}}}} a latent representation of the robot’s state. The full state is defined as {\bm{s}}=[{\bm{z}}^{\top},\bm{\iota}^{\top},{\bm{v}}^{\top}]^{\top}. We then decompose epistemic uncertainty across modalities and modify the exploration strategy from eq. 3 as follows:

{\bm{\pi}}({\bm{a}}|{\bm{s}})\propto\exp\left(\alpha^{-1}\left(Q^{{\bm{\pi}}}({\bm{s}},{\bm{a}})+\lambda\left(\omega_{{\bm{v}}}\left\|{\bm{\sigma}}^{{\bm{v}}}_{n}({\bm{s}},{\bm{a}})\right\|+\omega_{\bm{\iota}}\left\|{\bm{\sigma}}^{\bm{\iota}}_{n}({\bm{s}},{\bm{a}})\right\|+\omega_{{\bm{z}}}\left\|{\bm{\sigma}}^{{\bm{z}}}_{n}({\bm{s}},{\bm{a}})\right\|\right)\right)\right).

Here, {\bm{\sigma}}^{k}_{n}({\bm{s}},{\bm{a}}) denotes modality-specific epistemic uncertainty, and \omega_{\bm{k}}\geq 0 controls the relative contribution of each sensing channel. Crucially, this formulation enables tactile-centric exploration, which can be enforced by assigning dominant weight to \bm{\iota}. For instance, setting \omega_{\bm{\iota}}\gg\omega_{{\bm{v}}},\omega_{{\bm{z}}} biases exploration toward contact-rich regions, and \omega_{{\bm{v}}}=\omega_{{\bm{z}}}=0 yields purely tactile-driven information gain. Changing these weights changes the exploration bonus, rather than specifying which observations the policy receives. Purely tactile weighting is not assumed to be optimal for every task, since visual or latent disagreement can encourage complementary exploration.

4 Experiments

In the following, we present our experimental evaluation. The central hypothesis behind TacEx is that tactile feedback provides a crucial signal for enabling meaningful and directed exploration. To investigate this hypothesis, we first study whether touch encourages contact-rich exploration in simulation. We report means and standard errors over five seeds for reward-free exploration and ten seeds for diffusion steering.

4.1 Tactile-Driven Reward-Free Exploration

To isolate the effect of tactile information during learning, we consider a reward-free exploration setting in which the agent optimizes only the uncertainty-driven intrinsic reward defined from eq. 4, without access to task rewards. All modality-weighted exploration runs use zero extrinsic reward. We refer to this variant as TacEx (intrinsic-only) and systematically ablate different weighting configurations (\omega_{{\bm{v}}},\omega_{\bm{\iota}},\omega_{{\bm{z}}}) to analyze the relative contribution of visual, tactile, and latent uncertainty to exploration behavior.

Does touch lead to contact-rich and meaningful exploration?

In fig. 2, we evaluate TacEx in a simulated environment with a robotic gripper and a single object. Successful interaction with the object depends entirely on the agent’s exploration behavior. During the exploration phase, the agent receives no extrinsic reward r, and exploration is therefore driven solely by the uncertainty-based intrinsic objective in eq. 4. We compare several configurations of the weighting vector, namely: (i) tactile-only exploration (\omega_{{\bm{v}}}=0,\omega_{\bm{\iota}}=1,\omega_{{\bm{z}}}=0), (ii) combined tactile and visual exploration (\omega_{{\bm{v}}}=1,\omega_{\bm{\iota}}=1,\omega_{{\bm{z}}}=0), (iii) vision-only exploration (\omega_{{\bm{v}}}=1,\omega_{\bm{\iota}}=0,\omega_{{\bm{z}}}=0), and (iv) latent-state-only exploration (\omega_{{\bm{v}}}=0,\omega_{\bm{\iota}}=0,\omega_{{\bm{z}}}=1).

To further assess the benefits of contact-rich exploration, we additionally consider a downstream sparse-reward pick-and-place task, where the objective is to grasp the object and lift it to a target location. The reward uses a finite-support tolerance margin around the spatial target, following the DeepMind Control Suite [36]. After reward-free collection, we freeze the replay buffer, relabel its transitions with this reward, and train SAC entirely offline, without further environment interaction. Importantly, during data collection, the agent is driven purely by the uncertainty-based exploration objective, without access to task rewards. Finally, we compare against an online RL baseline, DrQ [17], trained directly on the sparse-reward task.

From fig. 2, we draw several conclusions. First, incorporating tactile-driven disagreement increases the frequency with which the agent interacts with and attempts to grasp the object. In contrast, without tactile disagreement, the agent rarely engages with the object at all, suggesting that uniform uncertainty over the state space does not necessarily translate into meaningful exploration. This highlights the role of tactile sensing as an effective inductive bias that promotes contact-rich behavior in manipulation tasks.

Interestingly, variants that prioritize tactile disagreement also induce substantially more object interaction than the sparse-reward online RL baseline and yield strong offline downstream performance on the pick-and-place task. In particular, policies trained on data collected with tactile-driven exploration achieve better offline returns, underscoring the importance of contact-rich data collection for subsequent reinforcement learning. This suggests that guiding exploration toward physically informative, contact-based experiences is critical for learning effective manipulation policies.

Finally, we investigate how the exploration agent behaves in a more complex environment with multiple objects. To this end, we introduce three additional objects, resulting in four objects in total, and report the corresponding results in fig. 3. To analyze how the agent distributes its interactions across these objects, we quantify the effective number of objects being grasped during exploration. Specifically, we compute the entropy H=-\sum_{i=1}^{4}p_{i}\log p_{i}, where p_{i} denotes the proportion of grasps attributed to object i, and report the effective number of objects as 2^{H}.

From fig. 3, we further observe that tactile disagreement plays a crucial role in enabling contact-rich manipulation and strong offline pick-and-place performance. Interestingly, augmenting tactile disagreement with visual disagreement increases exploration diversity, leading the agent to interact with multiple objects (effective number of objects \approx 2), whereas purely tactile-driven exploration results in more focused behavior, with an effective number of objects close to 1. While tactile disagreement yields slightly better performance on the downstream pick-and-place task—where only a single object must be manipulated—incorporating visual disagreement encourages broader exploration and maintains curiosity across multiple objects in the scene.

4.2 Tactile Diffusion Steering

The previous experiments show that tactile uncertainty can guide exploration when learning from scratch. We now ask whether the same idea can improve larger pretrained policies without updating them. We use diffusion steering [40], where a lightweight MLP predicts a noise action {\bm{w}}_{t}\in{\mathcal{W}} and a frozen VLA policy \pi_{0} maps this noise to robot actions. We build on the Libero-90 suite [20] and add tactile sensing [29] to the gripper.

The base policy \pi_{0} receives the same visual observations, proprioceptive state, and language command as during pre-training. In the tactile variants, the learned noise-prediction policy and critics additionally receive tactile observations \bm{\iota}_{t} from the gripper and for TacEx the uncertainty-based exploration reward from eq. 4 is used to direct exploration.

Baseline and method

DSRL-SAC trains an MLP policy \pi^{\bm{w}} with SAC to predict the latent diffusion noise {\bm{w}}_{t} from the current learner state {\bm{s}}_{t}, while the frozen VLA policy \pi_{0} maps this noise to robot actions. This baseline uses visual and proprioceptive observations, but no tactile input, and optimizes only the sparse task reward.

DSRL-TacEx augments the learner state with tactile observations \bm{\iota}_{t} from the gripper and adds the intrinsic reward (4). We train an ensemble dynamics model on online transitions ({\bm{s}}_{t},{\bm{w}}_{t},r_{t},{\bm{s}}_{t+1}), using the current learner state and predicted noise action as inputs. The ensemble predicts the next transition in selected observation modalities and the task reward; disagreement between ensemble heads serves as an epistemic uncertainty estimate. The policy is therefore encouraged to choose latent noise actions whose outcomes are informative under the learned latent-noise MDP. In the main experiments, we evaluate tactile-only disagreement, (\omega_{\bm{v}},\omega_{\bm{\iota}},\omega_{\bm{z}})=(0,1,0), and two mixed variants, (1,1,0) and (0,1,1).

Experimental setup

We evaluate on eight contact-rich tasks from the libero-90 suite. These tasks require the robot to establish and exploit physical contact with objects, containers, or articulated scene elements, making them a natural testbed for tactile diffusion steering. All methods use the same pretrained \pi_{0} checkpoint, the same language command for each task, and the same online interaction budget within each task. We report success rate, shown as average evaluation return, as a function of online environment interaction and average over random seeds.

Results

Figure 4 shows that tactile MaxInfo diffusion steering can improve over DSRL-SAC across the evaluated contact-rich tasks. The effect is task dependent: when the pretrained \pi_{0} prior already explores sufficiently, additional intrinsic reward is less important and can sometimes be distracting, which may reflect differences in the pretrained policy and the interactions required for completion. Still, across the selected tasks, the best-performing curves typically include tactile disagreement. These results suggest that touch is useful not only as an observation modality, but also as a way to shape exploration during post-training of frozen VLAs.

Isolating the role of tactile uncertainty

In Appendix C, we ablate the effect of adding touch purely as a state modality versus using it to direct exploration. Additionally, App. Fig. 5 separates tactile observations, information-gain exploration, and tactile disagreement in the bonus. On this task, tactile observations improve performance, information-gain exploration improves over SAC, and the highest-performing exploration variants also include tactile disagreement.

The five-task comparison in App. Fig. 6 extends this analysis. We compare tactile-only, tactile+vision, and tactile+latent bonuses with bonuses computed over visual, latent, or state predictions without an explicit tactile target. Tactile+vision achieves the highest reported mean final return on all five tasks, while other prediction targets remain competitive on several tasks. Together, these results support tactile uncertainty as a useful, task-dependent exploration signal, with visual uncertainty providing complementary benefits.

Additional baselines in App. Fig. 6 examine the tactile representation and the post-training algorithm. Two summed-taxel controls use aggregate touch either directly as an intrinsic reward or as a prediction target for ensemble disagreement, testing the value of retaining spatial tactile information. DSRL-PPO replaces SAC in the noise-steering policy, while DPPO [26] fine-tunes the diffusion policy itself. Across the five evaluated tasks, the tactile+vision variant achieves higher mean final return than both summed-taxel controls and the PPO-based baselines.

5 Conclusion

In this work, we introduced TacEx, a framework that establishes tactile feedback as a core prior for directed exploration in robotic manipulation. By prioritizing tactile uncertainty, TacEx avoids the inefficiency of traditional reinforcement learning methods that spend large portions of training on uninformative free-space exploration.

Our results show that tactile-driven curiosity enables the discovery of meaningful contact dynamics and emergent grasping behaviors without external rewards or demonstrations and the interaction-rich data collected through TacEx supports offline learning of downstream pick-and-place policies without additional environment interaction. Furthermore, TacEx improves the sample efficiency of post-training Vision-Language-Action (VLA) models, despite the VLA itself being pre-trained with datasets without tactile observations. Finally, we find that tactile and visual uncertainty encourages both focused contact exploration and broader interaction diversity.

Overall, these findings demonstrate that touch is not merely an observation modality, but a critical signal for efficient exploration and skill acquisition in robotic manipulation.

Limitations and future work

Our study has several limitations that open avenues for future work. First, while we adopt force maps precisely because they transfer readily between simulated and real tactile sensors [28], validating tactile-driven exploration on physical hardware—where contact is noisier and resets are costly—remains an important next step. We do not evaluate sensitivity to tactile noise, calibration drift, or sensor wear, and the present experiments do not establish whether these effects can be separated from uncertainty about contact dynamics. Moreover, contact-seeking curiosity does not itself enforce safe contact forces or prevent unintended interactions with distractor objects. Offline initialization and explicit safety constraints are possible directions for physical deployment, but their effectiveness with TacEx remains to be evaluated.

Second, our experiments use a single parallel-jaw gripper and a single tactile representation; whether tactile curiosity scales to dexterous multi-fingered hands, richer contact modes, and vision-based tactile images is left open, though our pipeline is designed to accommodate such sensors with minimal modification. The evaluated tasks do not test fragile or deformable objects, or establish force regulation in tightly constrained manipulation. Third, the per-modality weights (\omega_{v},\omega_{\iota},\omega_{z}) are fixed hyperparameters. Our multi-object results suggest a tradeoff between focused contact exploration and interaction diversity that these weights control, and learning or adapting them online is a natural direction. Finally, while we show that tactile uncertainty improves the post-training of frozen VLAs, we do not explore whether the contact-rich data collected through TacEx could be fed back into pre-training, closing the loop between tactile exploration and large-scale policy learning.

Acknowledgments

We thank Manuel Wendl, Jasmine Bayrooti, and Bruce D. Lee for their valuable feedback on this work. This work was supported as a part of NCCR Automation, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (grant number 51NF40_225155). B. Sukhija was supported by ELSA (European Lighthouse on Secure and Safe AI) funded by the European Union under grant agreement No. 101070617.

Numerical simulations were performed on the ETH Zürich Euler cluster. Parts of this text were revised with the assistance of a large language model to assist with writing and editing as well as for code development throughout this work; the authors remain responsible for all content.

References

1. [1] I. Akinola, J. Xu, J. Carius, D. Fox, and Y. Narang (2025) Tacsl: a library for visuotactile sensor simulation and learning. IEEE Transactions on Robotics.
2. [2] A. Aubret, L. Matignon, and S. Hassas (2023) An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey. Entropy.
3. [3] M. Bartos, B. D. Lee, L. Treven, A. Krause, F. Dörfler, and M. N. Zeilinger (2026) Optimistic online lqr via intrinsic rewards. arXiv preprint arXiv:2603.28938.
4. [4] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky (2026) \pi_{0}: A vision-language-action flow model for general robot control. External Links: 2410.24164, Link
5. [5] Y. Burda, H. Edwards, A. Storkey, and O. Klimov (2018) Exploration by random network distillation. arXiv preprint arXiv:1810.12894.
6. [6] R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine (2018) More than a feeling: learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters.
7. [7] S. Dong, D. K. Jha, D. Romeres, S. Kim, D. Nikovski, and A. Rodriguez (2021) Tactile-rl for insertion: generalization to objects of unknown geometry. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6437–6443.
8. [8] S. Fujimoto, H. Hoof, and D. Meger (2018) Addressing function approximation error in actor-critic methods. In ICML,
9. [9] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. (2018) Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905.
10. [10] J. Huang, S. Wang, F. Lin, Y. Hu, C. Wen, and Y. Gao (2025) Tactile-vla: unlocking vision-language-action model’s physical knowledge for tactile generalization. arXiv preprint arXiv:2507.09160.
11. [11] S. H. Huang, M. Zambelli, J. Kay, M. F. Martins, Y. Tassa, P. M. Pilarski, and R. Hadsell (2019) Learning gentle object manipulation with curiosity-driven deep reinforcement learning. External Links: 1903.08542, Link
12. [12] K. Iten, B. Lee, C. Li, L. Treven, A. Krause, and B. Sukhija (2026) Model-based reinforcement learning for control under time-varying dynamics. External Links: 2604.02260, Link
13. [13] K. Iten, L. Treven, B. Sukhija, F. Dörfler, and A. Krause (2026) Sample-efficient and scalable exploration in continuous-time rl. External Links: 2510.24482, Link
14. [14] R. S. Johansson and J. R. Flanagan (2009) Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience 10 (5), pp. 345–359.
15. [15] J. Jones, O. Mees, C. Sferrazza, K. Stachowicz, P. Abbeel, and S. Levine (2025) Beyond sight: finetuning generalist robot policies with heterogeneous sensors via language grounding. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Atlanta, USA.
16. [16] M. Kearns and S. Singh (2002) Near-optimal reinforcement learning in polynomial time. Machine learning.
17. [17] I. Kostrikov, D. Yarats, and R. Fergus (2021) Image augmentation is all you need: regularizing deep reinforcement learning from pixels. External Links: 2004.13649, Link
18. [18] K. Lee, Y. Qin, X. Wang, and S. Lim (2024) DexTouch: learning to seek and manipulate objects with tactile dexterity. arXiv preprint: arXiv 2401.12496.
19. [19] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra (2015) Continuous control with deep reinforcement learning. External Links: 1509.02971, Link
20. [20] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310.
21. [21] V. Mnih (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
22. [22] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell (2017) Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778–2787.
23. [23] D. Pathak, D. Gandhi, and A. Gupta (2019) Self-supervised exploration via disagreement. In ICML, pp. 5062–5071.
24. [24] M. L. Puterman (2014) Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
25. [25] S. Rajeswar, C. Ibrahim, N. Surya, F. Golemo, D. Vazquez, A. Courville, and P. O. Pinheiro (2021) Touch-based curiosity for sparse-reward tasks.
26. [26] A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz (2024) Diffusion policy policy optimization. External Links: 2409.00588, Link
27. [27] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak (2020) Planning to explore via self-supervised world models. In ICML, pp. 8583–8592.
28. [28] C. Sferrazza and R. D’Andrea (2022) Sim-to-real for high-resolution optical tactile sensing: from images to three-dimensional contact force distributions. Soft Robotics 9 (5), pp. 926–937.
29. [29] C. Sferrazza, Y. Seo, H. Liu, Y. Lee, and P. Abbeel (2023) The power of the senses: generalizable manipulation from vision and touch through masked multimodal learning.
30. [30] C. Sferrazza, A. Wahlsten, C. Trueeb, and R. D’Andrea (2019) Ground truth force distribution for learning-based tactile sensing: a finite element approach. IEEE Access 7, pp. 173438–173449.
31. [31] B. Sukhija, S. Coros, A. Krause, P. Abbeel, and C. Sferrazza (2025) Maxinforl: boosting exploration in reinforcement learning through information gain maximization. ICLR.
32. [32] B. Sukhija, L. Treven, C. Sancaktar, S. Blaes, S. Coros, and A. Krause (2024) Optimistic active exploration of dynamical systems. NeurIPS.
33. [33] B. Sukhija, L. Treven, C. Sferrazza, F. Dörfler, P. Abbeel, and A. Krause (2025) SOMBRL: scalable and optimistic model-based rl. NeurIPS.
34. [34] R. S. Sutton and A. G. Barto (2018) Reinforcement learning: an introduction. MIT press.
35. [35] C. Szepesvári (2022) Algorithms for reinforcement learning. Springer nature.
36. [36] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. (2018) Deepmind control suite. arXiv preprint arXiv:1801.00690.
37. [37] S. Tian, F. Ebert, D. Jayaraman, M. Mudigonda, C. Finn, R. Calandra, and S. Levine (2019) Manipulation by feel: touch-based control with deep predictive models. arXiv preprint: arXiv 1903.04128.
38. [38] H. Van Hasselt, A. Guez, and D. Silver (2016) Deep reinforcement learning with double q-learning. In AAAI,
39. [39] N. Vulin, S. Christen, S. Stevšić, and O. Hilliges (2021) Improved learning of robot manipulation tasks via tactile intrinsic motivation. IEEE Robotics and Automation Letters 6 (2), pp. 2194–2201.
40. [40] A. Wagenmaker, M. Nakamoto, Y. Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine (2025) Steering your diffusion policy with latent space reinforcement learning. Conference on Robot Learning (CoRL).
41. [41] J. Xu, Y. Jia, D. Yang, P. Meng, X. Zhu, Z. Guo, S. Song, and M. Ciocarlie (2024) Tactile-based object retrieval from granular media. arXiv preprint: arXiv 2402.04536.
42. [42] Z. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang (2023) Rotating without seeing: towards in-hand dexterity through touch. arXiv preprint arXiv:2303.10880.
43. [43] W. Yuan, S. Dong, and E. H. Adelson (2017) Gelsight: high-resolution robot tactile sensors for estimating geometry and force. Sensors 17 (12), pp. 2762.
44. [44] H. Zheng, B. Sukhija, C. Li, K. Iten, A. Krause, and R. K. Katzschmann (2025) Learning soft robotic dynamics with active exploration. External Links: 2510.27428, Link

Appendix A Extended Related Work

Learning to Manipulate With Touch

A growing body of work leverages tactile sensing for robotic manipulation [37, 7, 18, 41]. Prior works have shown that combining vision and touch can improve manipulation performance relative to using visual observations alone [6, 29]. Beyond its role as an observation modality, tactile feedback has also been used explicitly to guide exploration during learning. Prior methods reward force magnitude [39] instead of epistemic uncertainty, curiosity over a scalar impact-force predictor [11], or cross-modal prediction error that can conflate reducible epistemic with aleatoric uncertainty and uses aggregate force/torque with touch only in the reward [25]. TacEx instead rewards modality-specific epistemic uncertainty, which decays as dynamics are learned, and uses spatial normal/shear force maps that preserve contact location, multiple contacts, rotation, and slip cues. Since tactile signals are inherently sparse and only available upon contact, exploration and learning with touch are especially challenging.

Because informative contact signals arise only when the robot interacts with its surroundings, collecting useful tactile experience remains an exploration challenge. In this work, we study this problem and investigate how tactile feedback can be used to direct exploration in reinforcement learning. More specifically, we investigate the benefits of touch-driven exploration in visually guided manipulation tasks.

Exploration Strategies in Reinforcement Learning

Efficient exploration remains one of the central challenges in reinforcement learning, particularly in high-dimensional continuous control problems. Classical approaches such as \epsilon–greedy exploration [16, 21, 38] and Boltzmann exploration [34, 35] encourage exploration through randomized action selection or entropy maximization. While effective in simple settings, these methods explore indiscriminately and fail to account for the agent’s lack of knowledge about the environment. To address this limitation, several works propose uncertainty-aware exploration strategies that explicitly direct exploration toward poorly understood regions of the state-action space [31, 33, 22, 5].

In particular, Sukhija et al. [31] formulate exploration as maximizing information gain by augmenting the policy objective with epistemic uncertainty estimates derived from learned dynamics models. Such approaches have demonstrated strong empirical performance [44, 13, 12] and improved sample efficiency in continuous control and visual RL tasks. An aggregate uncertainty objective, however, does not explicitly control the contributions of different sensory modalities. In robotic manipulation, informative interactions are often concentrated around contact-rich regions of the state space, suggesting that exploration should preferentially focus on tactile interaction. Our work builds on uncertainty-driven exploration and introduces a modality-aware exploration bonus that weights tactile uncertainty as a distinct term to guide exploration more effectively in manipulation tasks.

Role of Touch in Post-Training of VLAs

Vision-language-action (VLA) models, such as \pi_{0} [4], can leverage large robot demonstration datasets that do not include tactile observations. Incorporating tactile feedback into such models at scale remains challenging, since it requires additional sensing hardware, calibration, and large multimodal datasets with aligned tactile observations and robot actions. FuSe [15] addresses this challenge by finetuning pretrained generalist robot policies on smaller heterogeneous-sensor datasets, using language as a common grounding space between modalities such as vision and touch. Tactile-VLA [10] incorporates tactile inputs into the VLA backbone and finetunes the model end-to-end to predict both motion and contact-force targets. In contrast, we ask whether touch can improve RL-based post-training while keeping the pretrained VLA and its input modalities unchanged.

We study this question in the diffusion-steering [40] setting. Rather than finetuning the pretrained VLA, we keep it fixed and train a lightweight MLP policy that predicts the diffusion noise used by the frozen policy. The resulting policy chain is {\bm{s}}_{t}\xrightarrow{\pi^{\bm{w}}}{\bm{w}}_{t}\xrightarrow{\pi_{0}}{\bm{a}}_{t}, where \pi^{\bm{w}} predicts the latent-noise action {\bm{w}}_{t} and the frozen VLA \pi_{0} maps this noise to the robot action {\bm{a}}_{t}, conditioned on its original visual, proprioceptive, and language inputs.

We show that tactile feedback can help this post-training procedure in two complementary ways: first, by enriching the state representation available to the noise-prediction policy, and second, by providing an exploration signal through ensemble disagreement over tactile predictions. Concretely, we train a tactile diffusion-steering policy using the modality-weighted MaxInfoRL objective from eq. 4, and evaluate whether incorporating tactile observations and tactile disagreement improves performance on contact-rich tasks.

Appendix B Implementation Details and Hyperparameters

We use the same SAC and MaxInfoSAC hyperparameter choices as [31, 40] unless explicitly noted. Within the SAC/MaxInfoSAC modality ablations, we hold the optimization settings, model capacity, update schedule, and evaluation protocol fixed while varying tactile observations and disagreement targets. This matching does not extend to PPO and DPPO, which use different optimization proce- dures. This makes the ablations isolate the role of tactile sensing and tactile uncertainty as outlined in appendix C. Hyperparameters for the LIBERO experiments using DSRL are outlined in table 1.

Appendix C Ablations

We compare four diffusion-steering variants. DSRL-SAC is the standard diffusion-steering baseline: an MLP policy \pi^{\bm{w}} is trained with SAC to predict the latent noise {\bm{w}}_{t} from the current state {\bm{s}}_{t}, while the frozen VLA policy \pi_{0} maps this noise to actions. This method uses visual and proprioceptive observations, but no tactile input. DSRL-Tac-SAC uses the same SAC learner in the latent-noise action space, but augments the learner state with tactile observations \bm{\iota}_{t} from the gripper. This baseline tests whether tactile information improves diffusion steering as an additional observation modality, without changing the exploration objective.

We then add exploration that explicitly seeks informative transitions in the latent-noise MDP. For this, DSRL-MaxinfoSac and DSRL-TacEx train an uncertainty-aware statistical dynamics model on the online replay buffer ({\bm{s}}_{t},{\bm{w}}_{t},r_{t},{\bm{s}}_{t+1}). The model takes the current learner observation and the predicted noise action {\bm{w}}_{t} as input, and predicts the resulting transition in selected observation modalities, as well as the task reward. We represent this model as an ensemble, so that epistemic uncertainty can be estimated from the disagreement between ensemble members [23]. Intuitively, if different ensemble heads make different predictions about what will happen after applying a noise action {\bm{w}}_{t}, then executing this noise action is expected to provide information about the latent-noise MDP.

DSRL-SAC trains the noise-prediction policy using only the sparse task reward. In contrast, DSRL-MaxinfoSac augments this reward with an intrinsic bonus proportional to ensemble disagreement, encouraging the policy to choose noise actions whose outcomes are informative under the learned latent-noise dynamics model. In DSRL-MaxinfoSac, this disagreement is computed only over non-tactile modalities, i.e., {\omega}_{\bm{\iota}}=0 since tactile observations are not present. DSRL-TacEx uses a similar maximum information objective, but additionally receives tactile observations whose embeddings \bm{\iota} are included in the state {\bm{s}} and may include the tactile modality when computing disagreement with different weights. This tests whether tactile uncertainty provides a useful exploration signal for discovering contact-rich behavior.

The results in Figure 5 support two conclusions. First, tactile observations improve the diffusion-steering state representation: for the same disagreement modality, solid DSRL-TacEx curves consistently outperform their dashed DSRL-MaxinfoSac counterparts, which do not receive tactile input. Second, exploration itself is important, as the colored DSRL-TacEx variants improve over the black SAC baselines. However, the strongest performance is obtained when tactile information is used not only as an observation, but also as part of the disagreement signal. When DSRL receives tactile observations but computes disagreement only over non-tactile modalities, performance degrades relative to tactile-disagreement variants. This indicates that touch is most useful when it directly guides epistemic exploration toward contact-rich transitions.

We include the explore-then-exploit variant as a schedule control, shown by the dotted curve in Figure 5. This method trains without an extrinsic reward for the first 100k environment steps, and then greedily maximizes w.r.t. the task. Since it changes the training schedule rather than the sensing modality, we isolate it in a separate panel and do not use it for the main modality comparison.

Figure 6 compares five LIBERO tasks: ketchup in tray (58), soup in tray (55), butter in tray (56), book to back caddy (77), and stove and frying pan (21). The added “summed taxels” baselines either use the aggregate signal directly as a diffusion-steering reward, approximating force-magnitude intrinsic rewards, or use the ensemble disagreement when predicting the same signal as a MaxInfoRL bonus [31]. This isolates the value of spatially resolved tactile uncertainty. DSRL-PPO and DPPO test post-training; DSRL-PPO replaces SAC with PPO while keeping the setup and interaction budget fixed. This provides the matched on-policy comparison, although DSRL-PPO does not achieve meaningful learning progress on these five tasks within the evaluated interaction budget. We additionally include DPPO [26], a structurally different baseline that fine-tunes the diffusion policy itself using policy gradients.