韩国ETRI的HuTact用人手视频把触觉机器人手任务成功率从33.8%提升至50%

Abstract

Tactile representation learning for dexterous manipulation often requires substantial robot-side data collection, limiting its scalability across different hands and sensors. We explore whether large-scale human hand-object interaction (HOI) data can provide a useful prior for robotic tactile learning. To this end, we propose HuTact, a framework that pretrains tactile-aware hand representations from human HOI data without requiring real tactile signals during pretraining. HuTact transfers the interaction structure learned from human hand motions to robotic tactile observations and can be further adapted with a small amount of unlabeled robot interaction data. We evaluate HuTact on tactile perception and manipulation tasks, including grasp prediction, slip detection, object classification, and tactile-conditioned policy learning. Across these tasks, human HOI pretraining improves data efficiency and provides a strong initialization for downstream learning. Notably, with HOI pretraining followed by midtraining on unlabeled robot play data, HuTact achieves 88.26% Macro-F1 on grasp prediction using only 25% of the labeled downstream data, comparable to training from scratch on the full labeled dataset (88.23%). These results suggest that human interaction data can serve as a scalable complementary source for robotic tactile representation learning.

I Introduction

Tactile sensing enables humans to perceive and interact with the physical world, providing direct information about contact, force, texture, and motion that is difficult to obtain through vision alone [1]. Inspired by this remarkable human capability, tactile sensing has emerged as a critical modality for robotic manipulation, enabling robots to perceive subtle contact interactions, infer local physical properties, and regulate contact forces during complex manipulation tasks that demand precise and adaptive physical interaction [2, 3, 4].

Accordingly, a wide range of tactile sensors have been developed to capture rich information arising from physical contact, including vision-based sensors (e.g., GelSight [2], DIGIT [5]) and magnetic-based sensors (e.g., AnySkin [4], ReSkin [6]). Beyond advances in tactile sensing hardware, the growing availability of large-scale tactile interaction data has further motivated learning-based approaches to extract generalizable representations from raw tactile observations [7]. In particular, recent works have explored self-supervised pretraining using large-scale tactile interaction data collected with respective tactile sensors, learning representations that can be transferred to downstream manipulation tasks [8, 9, 10, 11].

Learning generalizable tactile representations can reduce the amount of task-specific data required for downstream tasks. For vision-based tactile sensing, Feng et al. propose AnyTouch [12], which learns transferable representations from a sensor-aligned dataset collected across multiple visuo-tactile sensors. For non-vision-based tactile sensing, Sharma et al. [8] apply DINO-based self-distillation to masked tactile sequences collected from a dexterous hand, while Guzey et al. [9] pretrain tactile representations using task-agnostic robotic play data. However, existing approaches for non-vision-based tactile sensors still largely require real tactile interaction data collected directly with the target robotic hand and sensing hardware. As a result, adapting representation learning to a new robotic platform or tactile sensor requires additional data collection, introducing substantial time and cost for pretraining.

As an alternative, human hand-object interaction (HOI) data provide a promising source of pretraining supervision, where they contain large-scale and diverse interactions across objects, hand configurations, and manipulation behaviors [13, 14, 15, 16]. These datasets are particularly attractive for learning transferable hand representations because they provide (1) interaction patterns that can be shared across different hand embodiments, (2) large-scale interaction data that would be costly to collect on robotic platforms, and (3) an opportunity to reduce embodiment-specific data collection when adapting to a new robotic hand. Many of these datasets also provide temporally aligned 3D reconstructions of human hands and manipulated objects, allowing fingertip configurations and their evolution during interaction to be recovered without collecting data on a robotic platform. Although HOI datasets do not contain real tactile signals, their hand–object trajectories encode spatial and temporal structure associated with physical interaction. We hypothesize that this structure can serve as a transferable prior for robotic tactile representation learning and reduce the amount of robot-side tactile data required for adaptation.

In this work, our research question is: Can human hand-object interaction (HOI) data provide a transferable prior for robotic tactile representation learning? To answer this question, we propose HuTact, a framework for learning tactile-aware robot hand representations from human HOI data. Rather than attempting to reconstruct real tactile signals from human data, HuTact learns the spatial and temporal structure of hand interactions from wrist-local fingertip trajectories. To bridge human HOI pretraining and robotic tactile sensing, we separately tokenize fingertip coordinates and tactile observations, allowing the coordinate representation and shared Transformer backbone learned from HOI data to be retained while adapting the sensor-specific tactile input embedding. We further incorporate geometry-derived pseudo-contact signals as an auxiliary cue, encouraging the model to capture the relationship between fingertip coordinates and their corresponding tactile-like signals. This framework enables large-scale human interaction data to provide a transferable initialization for tactile-aware robot learning while reducing the amount of robot-side tactile data required for adaptation.

To validate the effectiveness and transferability of HuTact, we evaluate it on tactile-sensitive downstream tasks, including grasp prediction, slip detection, object classification, and policy learning on tactile-sensitive tasks, and analyze the factors that enable transfer from human HOI data to robotic tactile sensing. In summary, our contributions are as follows:

• We propose HuTact, a framework that uses large-scale human hand-object interaction data to pretrain tactile-aware robot hand representations without requiring real tactile signals during the initial pretraining stage.
• We show that representations learned from HOI data transfer to robotic tactile perception and manipulation, improving grasp prediction, slip detection, object classification, and tactile-conditioned policy learning.
• We provide systematic analyses of the transfer mechanism through architectural ablations and data-efficiency experiments, studying the roles of temporal modeling, token separation, pseudo-contact cues, and robot-side midtraining. Our results show that HOI pretraining can substantially reduce the amount of task-specific robotic tactile data required for downstream adaptation.

II Related Work

Tactile Sensing: Tactile sensors for robotic manipulation broadly include vision-based and non-optical sensing modalities. Vision-based sensors [2, 5] capture fine-grained contact geometry and have been applied to object identification [17], geometry sensing [18, 3], pose estimation [19, 20], and policy learning [21, 22]. However, their optical components can result in a relatively large form factor, making integration across multiple fingers challenging. Non-optical sensors instead use compact magnetic [4, 23] or capacitive sensing elements [24] to enable distributed tactile sensing. In this work, we use the BrainCo Revo 2 hand [25], which is equipped with fingertip capacitive sensors that measure normal force, tangential force, tangential-force direction, and proximity.

Tactile Representation Learning: Recent work has explored transferable tactile representations through large-scale self-supervised pretraining. For vision-based tactile sensors, objectives developed for visual representation learning [26, 27, 28] have been adapted to tactile observations [29, 10, 12, 11]. For skin-like tactile sensors, Sharma et al. [8] apply self-distillation to masked tactile sequences from a dexterous hand, while Guzey et al. [9] pretrain tactile representations using task-agnostic robotic play data. While these approaches demonstrate the benefits of tactile pretraining, they generally require interaction data collected with the target robot and tactile sensor. In contrast, HuTact initializes tactile-aware representations from human hand–object interaction data without real tactile signals. We pretrain on the spatial and temporal structure of wrist-local fingertip interactions, augmented with geometry-derived pseudo-contact cues, and subsequently adapt the representation to the target robotic tactile sensor.

Robot Learning from Human Data: Human interaction data have been increasingly used to reduce robot-specific supervision for manipulation learning [30, 31, 32, 33, 34], by extracting hand poses, object motions, and interaction trajectories from large-scale demonstrations. HOI data have also been applied directly to dexterous manipulation: Zhao et al. [35] retarget human hand-object interactions to dexterous robot hands and learn a residual policy for embodiment mismatch, while Zeng et al. [36] use HOI data for vision-language-action learning. Large-scale datasets such as OakInk2 [16], TACO [37], ARCTIC [38], and HOT3D [39] provide scalable supervision for learning hand–object interaction structure. HuTact differs in objective: rather than transferring human motion directly to a robot policy, we use HOI data to pretrain a tactile-aware hand representation that is subsequently adapted to real robotic tactile sensing. To our knowledge, prior work has not systematically studied large-scale HOI data as a pretraining source for robotic tactile representation learning.

III Methodology

HuTact is designed to transfer interaction structure learned from human HOI data to robotic tactile sensing. The key challenge is that the source HOI datasets provide hand and object geometry but no real tactile signals, whereas the target robot provides sensor-specific tactile observations. We address this mismatch by factorizing the input representation into two streams: a shared wrist-local fingertip coordinate stream and a sensor-dependent contact stream. The shared Transformer backbone is pretrained on human interactions, while the sensor-specific tactile input embedding is adapted when transferring to real robotic tactile observations.

III-A Problem setup

We consider two dexterous robot hands (BrainCo Revo2 Hand) with N{=}10 fingertip tactile sensors. At each timestep t, sensor i captures a signal \mathbf{s}_{i}^{t}\in\mathbb{R}^{C} with C{=}4 channels—normal force, tangential force, tangential direction, and proximity—and forward kinematics provides the corresponding fingertip coordinate \mathbf{c}_{i}^{t}\in\mathbb{R}^{3} in the wrist-local frame as in [33]. The model operates on a temporal window

\mathbf{X}=\{(\mathbf{c}_{i}^{t},\,\mathbf{s}_{i}^{t})\}_{i=1,\dots,N}^{t=1,\dots,T},

with T{=}15 frames sampled at 30 Hz, spanning 0.5 s of contact. Then, we pretrain a tactile encoder f_{\theta} with human-object interaction (HOI) data and subsequently transfer it to robotic downstream tasks. To maintain a consistent input structure across the human and robot domains, human hand samples are represented using fingertip coordinates paired with a one-dimensional geometry-derived pseudo-contact signal (i.e., C=1), whereas robot observations use the corresponding fingertip coordinates paired with four-dimensional tactile signals.

III-B HOI Data Processing

The pretraining corpus is assembled from HOI dataset. We obtain its 3D coordinate from the MANO hand model [40] and extract a one-dimensional pseudo-contact signal from the geometric relationship between the hand and object meshes. First, we convert the MANO hand representation [40] into a common 21-joint skeleton by augmenting the 16 MANO kinematic joints with five fingertip vertices and reordering them into a unified joint convention. From this representation, we extract wrist-local fingertip coordinates and pseudo-contact signals for HOI pretraining.

Fingertip coordinates. For each hand and frame, we construct a wrist-local coordinate frame from the hand geometry. The wrist defines the origin, while the palm-normal and finger directions are derived from the pinky–thumb and wrist–middle-finger axes and orthogonalized to form a right-handed basis. We transform all joints into this frame and retain the five fingertips per hand, yielding N{=}10 coordinates \mathbf{c}_{i}^{t}\in\mathbb{R}^{3}. During the experiment, robot fingertip coordinates from forward kinematics use the same local convention.

Pseudo-contact signals. We define a surface patch \mathcal{P}_{i}^{t} over each fingertip pad rather than estimating contact from a single fingertip point. MANO vertices are assigned to joints using dominant skinning weights, and each fingertip is augmented with the 20 nearest rest-pose vertices around its fingertip vertex. At each frame, we retain only vertices on the palmar side using the local pad direction, preventing dorsal-side vertices from producing spurious contact responses.

For each retained vertex v\in\mathcal{P}_{i}^{t}, we compute the shortest point-to-triangle distance d_{v}^{t} to the object surface. Given a contact threshold of d_{\max}=10 mm, we define

u_{v}^{t}=\frac{\max(d_{\max}-d_{v}^{t},\,0)}{d_{\max}},\qquad r_{v}^{t}=(u_{v}^{t})^{2}(3-2u_{v}^{t}),

where r_{v}^{t}\in[0,1] provides a smooth proximity response.

We then aggregate the vertex responses using their associated surface areas:

s_{i}^{t}=\frac{\sum_{v\in\mathcal{P}_{i}^{t}}a_{v}^{t}r_{v}^{t}}{\sum_{v\in\mathcal{P}_{i}^{t}}a_{v}^{t}},

where a_{v}^{t} is the representative surface area of vertex v, computed from its incident triangles. This area-weighted formulation reduces sensitivity to mesh tessellation and yields a bounded proximity-like signal s_{i}^{t}\in[0,1], which increases as a larger portion of the fingertip pad approaches the object surface. When multiple objects are present, we select the object yielding the largest s_{i}^{t} for each fingertip. For HOI pretraining, we use s_{i}^{t} as a one-dimensional contact observation (C=1).

III-C Architecture

As shown in Fig. 2, HuTact consists of separate coordinate and tactile tokenizers, a frame encoder, and a temporal encoder.

Tokenizers. Within a frame t, fingertip coordinates c^{t} and tactile signals s^{t} are tokenized as two separate streams: \{\mathbf{c}_{i}^{t}\} and \{\mathbf{s}_{i}^{t}\} are projected by distinct input embeddings into N coordinate tokens and N tactile tokens, each stream normalized with its own statistics. Specifically, the tokenizing process can be formalized as follows:

z_{c,i}^{t}=\mathrm{LN}_{c}\!\left(\mathbf{W}_{c}\mathbf{c}^{t}_{i}+b_{c}\right),\qquad z_{s,i}^{t}=\mathrm{LN}_{s}\!\left(\mathbf{W}_{s}\mathbf{s}^{t}_{i}+b_{s}\right),

where the z_{c,i}^{t}\in\mathbb{R}^{D} and z_{s,i}^{t}\in\mathbb{R}^{D} denote the coordinate tokens and tactile tokens for the i-th finger, respectively. Also, \mathbf{LN}, \mathbf{W}\in\mathbb{R}^{D\times m} (m is the dimension of input), and b\in\mathbb{R}^{D} refer to the Layer Norm [41], linear weight, and bias, respectively. Finally, we concatenate these tokens with [CLS] token as:

\mathbf{Z}^{t}=\left[\mathbf{z}^{t}_{\mathrm{CLS}},\{\mathbf{z}_{c,i}^{t}\}_{i=1}^{N},\{\mathbf{z}_{s,i}^{t}\}_{i=1}^{N}\right].

As shown in this Eq 4 and Eq 5, we separate the token input for the tactile signals and fingertip coordinates unlike previous approaches [8]. This token separation is particularly important for transferring from HOI pretraining to real tactile sensing: while the contact signal used during pretraining and the real tactile signal have different dimensionalities (i.e., 1 dim vs. 4 dim), the coordinate tokenizer remains unchanged and can be directly reused after pretraining.

Frame encoder. Given the concatenated token sequence \mathbf{Z}^{t} at each timestep t\in\{1,2,\ldots,T\}, we apply 2D multimodal RoPE [42] using hand and finger identities as the two positional dimensions. We then process the tokens with a lightweight 4-layer Transformer encoder, following the remaining architectural hyperparameters of ViT-Tiny architecture [43]. Through self-attention, the encoder captures interactions between tactile observations and fingertip coordinates, producing a unified representation of the hand-object contact state at each timestep. The frame-level representations are then passed to the temporal encoder to model their relationship over time. Specifically, we take the [CLS] token from each frame encoder output and concatenate them along the token dimension to construct a sequence of frame-level representations. Given T=15 input frames, we obtain

\mathbf{H}=\left[\mathbf{h}_{\mathrm{CLS}},\hat{\mathbf{z}}^{1}_{\mathrm{CLS}},\hat{\mathbf{z}}^{2}_{\mathrm{CLS}},\ldots,\hat{\mathbf{z}}^{15}_{\mathrm{CLS}}\right],

where \mathbf{h}_{\mathrm{CLS}} and \hat{\mathbf{z}}^{i}_{\mathrm{CLS}} are new CLS token for temporal encoder and output CLS token for i-th frame with frame encoder, respectively.

Temporal encoder. Given the frame-level sequence \mathbf{H}, the temporal encoder applies RoPE based on the absolute frame index and models temporal dependencies across the T=15 frame-level [CLS] tokens. It uses the same Transformer architecture as the frame encoder with independent parameters. The output temporal [CLS] token \hat{\mathbf{h}}_{\mathrm{CLS}} is used as the final window-level representation. This representation summarizes the spatiotemporal hand-object interaction over the entire T=15 frame window and is subsequently used by the downstream prediction head.

III-D Training objectives and stages

Objective. We pretrain the model using temporal self-distillation following the DINOv2 [27] as [8] in the pretraining and midtraining stages. A student network is trained to match an exponential-moving-average (EMA) teacher by combining a distillation loss on the temporal [CLS] embedding with a masked-token prediction loss on the frame-level tokens.

Stage 1: Pretraining on HOI data. In the first stage, the encoder is pretrained for 200 epochs using the objective described above on the HOI data. The data consists of contact signals and fingertip coordinates derived from public HOI data. Note that, no robot data are used at any point during this stage.

Stage 2 (Optional): Midtraining on robotic play data. In the second stage, we optionally continue self-supervised training using the same objective and optimization recipe on a small unlabeled robotic play data. Because the real robot provides C=4 tactile sensor channels, whereas Stage 1 uses a different pseudo-tactile channel configuration, we re-initialize the tactile input tokenizer at the transition between the two stages while retaining all remaining pretrained parameters. Each epoch repeats the robot corpus 32 times, preserving approximately the same number of training samples per epoch as in Stage 1 (\approx 505k samples per epoch). We train for 100 epochs using the same warmup-cosine learning-rate schedule. Hand poses are min-max normalized to the unit box during this stage.

Stage 3: Downstream transfer. For downstream transfer, we fine-tune the full encoder jointly with an attentive-pooling task head. When Stage 2 midtraining is not conducted, we re-initialize the tactile input tokenizer before downstream fine-tuning; otherwise, the tokenizer adapted during Stage 2 is used. For the given downstream data, we use AdamW with a constant learning rate of 1\times 10^{-4} for 100 epochs.

Optimization. We use AdamW with a peak learning rate of 1\times 10^{-4} and a batch size of 512. The learning rate is linearly warmed up for the first 10 epochs and subsequently decayed using a cosine schedule to 1\times 10^{-6}. The teacher temperature is increased from 0.04 to 0.07 during the first 2 epochs. The EMA momentum of the teacher is increased from 0.994 to 1.0 throughout training, while the weight decay follows a cosine schedule ending at 0.04.

IV Experiments

IV-A Experimental setup

We use a Unitree G1 humanoid equipped with a BrainCo Revo2 dexterous hand [25], which has capacitive tactile sensors at each fingertip, as shown in Fig. 3.

Pretraining datasets. For HOI pretraining, we use HOT3D [39], OakInk-v2 [16], TACO [37], and ARCTIC [38], extracting fingertip coordinates and pseudo-contact signals from 2.57M frames (\sim24 hours). We additionally collect 80K frames (\sim44 minutes) of Revo2 play data containing fingertip coordinates and tactile signals through Meta Quest 3 teleoperation [44] with six objects (e.g., plastic bottle, bubble wrap, driver, pot, sponge, and towel).

Downstream datasets. For grasp prediction, we collect 590 teleoperated grasp episodes over five objects, totaling 18K frames (\sim10 minutes), with each episode labeled by grasp success. For slip detection, we hold the robot hand fixed and manually move the grasped object while tracking an attached marker; marker motion is labeled as slip, yielding 18 episodes over six object conditions and 51K frames (\sim28 minutes). For object classification, we reuse successful grasp episodes, extract the result-phase frames, and label the grasped object identity, yielding 252 episodes and 43K frames (\sim24 minutes).

For ID evaluation, we perform episode-level cross-validation before constructing temporal windows, ensuring that all windows from the same episode remain in the same split. We use stratified 4-fold cross-validation for grasp prediction and object classification, with 75% of the episodes for training and 25% for evaluation in each fold, and 3-fold cross-validation for slip detection. Reported ID performance is averaged across the corresponding folds.

For policy learning, we collect two datasets using a scripted controller. The cup-grasping and card-sliding datasets contain 40 and 55 episodes, respectively, each totaling approximately 30K frames (\sim17 minutes). For each policy configuration, success rates are computed over 40 evaluation trials per task.

IV-B Performance on downstream tasks

In Table I, we evaluate HuTact on three downstream perception tasks: grasp prediction, slip detection, and object classification. Grasp prediction and slip detection are binary classification tasks that predict grasp success from observations collected 1 second after grasp execution and whether the current grasp is slipping or stable, respectively. Object classification is a five-way classification task that predicts the identity of the grasped object from observations collected during the post-grasp phase. Also, in the ID setting, all object categories are included in both the training and evaluation sets. In the OOD setting, we use a leave-one-object-out protocol, where one object is excluded from training and used only for evaluation, while the remaining objects are used exclusively for training (except object classification). We compare HuTact against two baselines: (i) From Scratch, where the model is randomly initialized, and (ii) Revo2 Pretraining, where the encoder is pretrained on Revo2 play data using the same self-supervised objective as Stage 1.

HOI pretraining (Stage 1 only) consistently improves grasp prediction over both training from scratch and Revo2-only pretraining, with particularly clear gains in the OOD setting. For slip detection, HOI pretraining provides smaller but consistent gains in ID Avg and OOD performance, while achieving comparable ID Last performance to training from scratch. Revo2-only pretraining performs worse on slip detection, indicating that a small amount of robot play data alone is insufficient to learn transferable features for this task. For object classification, however, Revo2 pretraining outperforms HOI pretraining, suggesting that real tactile observations contain additional object-specific information. Across all three tasks, HOI pretraining followed by Revo2 midtraining achieves the best overall performance, demonstrating that large-scale HOI data and robot-side tactile data provide complementary benefits for downstream tactile representation learning.

IV-C Performance with action policy

In Table II, we evaluate the learned tactile representation on two contact-sensitive manipulation tasks using ACT [45] and Diffusion Policy (DP) [46]. Cup Destacking requires the robot to separate a cup from a stack while maintaining a stable grasp, whereas Single-Card Sliding requires the robot to slide a single card without displacing the remaining cards. We compare three tactile configurations: (i) no tactile input, (ii) a tactile encoder trained from scratch, and (iii) our HOI-pretrained encoder followed by Revo2 midtraining. We modify ResNet18-based ACT and DP with our tactile encoder. For ACT, the tactile representation is linearly projected to the Transformer embedding dimension and appended as a single token to the encoder input sequence. The tactile representation is used only as a conditioning input to the policy and is not provided to the VAE encoder. For Diffusion Policy, we concatenate the tactile representation directly with the visual observation features in the global conditioning vector.

Our tactile representation achieves the best overall performance for both policy architectures. With ACT, HOI + Revo2 midtraining improves the average success rate to 50.0%, compared with 33.8% without tactile input and 32.5% with a tactile encoder trained from scratch. With Diffusion Policy, the improvement is more consistent across both tasks: our method achieves 47.5% on Cup Destacking and 52.5% on Single-Card Sliding, yielding an average success rate of 50.0%, compared with 22.5% without tactile input and 38.8% from scratch. We observe that ACT with a tactile encoder trained from scratch often grasps multiple cups during Cup Destacking, which is counted as a failure and contributes to its low success rate of 5.0%. These results indicate that the pretrained tactile representation provides useful contact-sensitive features for downstream policy learning beyond either omitting tactile input or learning the tactile encoder solely from task-specific demonstrations.

IV-D Transfer performance comparison

Table III compares HuTact with Sparsh-Skin [8] to evaluate cross-sensor transfer across different pretraining sources and encoder architectures. We initialize Sparsh-Skin from the authors’ publicly released checkpoint, pretrained on an Allegro hand equipped with Xela uSkin sensors, and fine-tune it on the Revo2 downstream tasks. Because the two platforms differ in the number of sensors and tactile signal dimensions, we reinitialize the patch and positional embeddings while retaining the remaining pretrained parameters. HuTact consistently outperforms Sparsh-Skin across all three downstream tasks. This performance gap is consistent with the benefits of token separation for transfer across sensing configurations and temporal modeling for capturing contact dynamics over time. These results indicate that HuTact provides an effective initialization for robotic tactile representation learning.

IV-E Ablation Studies

In Table IV-(a), we analyze the effects of the temporal encoder, token separation, and HOI pretraining on the three downstream tasks. Configurations A and B use a non-temporal, unified-token architecture similar to prior tactile representation learning approaches [8], serving as a reference for our architectural modifications. (i) The temporal encoder provides the largest gain for slip detection: without HOI pretraining, adding temporal modeling improves Macro-F1 from 61.8% to 73.1% (C\rightarrowE), highlighting the importance of modeling contact dynamics over time. (ii) Token separation is particularly effective when combined with HOI pretraining, improving grasp prediction from 87.7% to 91.0% and object classification from 83.7% to 87.9% (B\rightarrowD). Separating fingertip coordinates from sensor-specific tactile inputs also allows the coordinate tokenizer learned from HOI data to be retained during transfer to the robot domain, and HOI pretraining yields larger gains when separate tokenizers are used. (iii) The complete configuration (F) achieves the best grasp prediction and slip detection performance among the architecture variants.

Table IV-(b) examines the contribution of pseudo-contact signals during Stage 1 pretraining. (i) Pseudo-contact pretraining provides its clearest benefit on slip detection, a task that relies strongly on tactile contact information, improving Macro-F1 from 72.6% to 73.8% without Revo2 midtraining (F{}^{\prime}\rightarrowF), while the gains in grasp prediction and object classification are negligible. (ii) This benefit becomes more evident after Revo2 midtraining, where pseudo-contact improves slip detection from 73.0% to 76.4% (G{}^{\prime}\rightarrowG). These results suggest that pseudo-contact pretraining helps the model capture the relationship between fingertip-coordinate and tactile tokens, providing a more transferable initialization when adapting to real robotic tactile signals.

IV-F Downstream Data Efficiency

We further evaluate the data efficiency of HuTact by varying the amount of downstream training data from 5% to 100%. As shown in Fig. 4, HOI + Revo2 midtraining consistently reduces the amount of task-specific data required across all three tasks. It matches or exceeds the full-data scratch baseline using only 25% of the data for grasp prediction (88.3% vs. 88.2%), 75% for slip detection (75.9% vs. 75.7%), and 50% for object classification (86.2% vs. 85.1%). These results demonstrate that HuTact substantially improves downstream data efficiency.

V Limitations

Our work has two main limitations. First, geometry-derived pseudo-contact signals do not capture physical contact properties and provide only modest additional gains beyond fingertip coordinates for grasp prediction and object classification. They also require reconstructed hand and object meshes, limiting applicability to HOI datasets with 3D geometry. Second, our evaluation is limited to a Revo2-based bimanual setup, leaving transfer to other robotic hands and tactile sensing modalities unexplored. Future work will investigate richer contact cues from visual and hand-motion observations and extend HuTact to broader robotic embodiments and sensor layouts.

VI Conclusion

In this work, we presented HuTact, a framework for pretraining tactile-aware robot hand representations from large-scale human hand-object interaction (HOI) data without requiring real tactile signals during initial pretraining. HuTact leverages fingertip coordinates and pseudo-contact signals, with separate coordinate and tactile tokenization to support transfer to robotic tactile sensing. Across perception and policy-learning tasks, HOI pretraining provides a useful initialization that is further improved by Revo2 midtraining, while also substantially improving downstream data efficiency. These results show that human interaction data can serve as an effective pretraining prior for robotic tactile learning and reduce the need for robot-side tactile data collection.

References

1. [1] R. S. Dahiya, G. Metta, M. Valle, and G. Sandini (2009) Tactile sensing—from humans to humanoids. IEEE transactions on robotics 26 (1), pp. 1–20.
2. [2] W. Yuan, S. Dong, and E. H. Adelson (2017) Gelsight: high-resolution robot tactile sensors for estimating geometry and force. Sensors 17 (12), pp. 2762.
3. [3] I. H. Taylor, S. Dong, and A. Rodriguez (2022) Gelslim 3.0: high-resolution measurement of shape, force and slip in a compact tactile-sensing finger. In 2022 international conference on robotics and automation (ICRA), pp. 10781–10787.
4. [4] R. Bhirangi, V. Pattabiraman, E. Erciyes, Y. Cao, T. Hellebrekers, and L. Pinto (2025) Anyskin: plug-and-play skin sensing for robotic touch. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 16563–16570.
5. [5] M. Lambeta, P. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al. (2020) Digit: a novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters 5 (3), pp. 3838–3845.
6. [6] R. Bhirangi, T. Hellebrekers, C. Majidi, and A. Gupta (2022) ReSkin: versatile, replaceable, lasting tactile skins. In Proceedings of the 5th Conference on Robot Learning, A. Faust, D. Hsu, and G. Neumann (Eds.), Proceedings of Machine Learning Research, Vol. 164, pp. 587–597.
7. [7] F. Yang, C. Ma, J. Zhang, J. Zhu, W. Yuan, and A. Owens (2022) Touch and go: learning from human-collected vision and touch. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link
8. [8] A. Sharma, C. Higuera, C. K. Bodduluri, Z. Liu, T. Fan, T. Hellebrekers, M. Lambeta, B. Boots, M. Kaess, T. Wu, F. R. Hogan, and M. Mukadam (2025) Self-supervised perception for tactile skin covered dexterous hands. In 9th Annual Conference on Robot Learning, External Links: Link
9. [9] I. Guzey, B. Evans, S. Chintala, and L. Pinto (2023) Dexterity from touch: self-supervised pre-training of tactile representations with robotic play. In 7th Annual Conference on Robot Learning, External Links: Link
10. [10] Q. Liu, Q. Ye, Z. Sun, Y. Cui, G. Li, and J. Chen (2024) Masked visual-tactile pre-training for robot manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 13859–13875.
11. [11] R. Feng, Y. Zhou, S. Mei, D. Zhou, P. Wang, S. Cui, B. Fang, G. Yao, and D. Hu (2026) AnyTouch 2: general optical tactile representation learning for dynamic tactile perception. In The Fourteenth International Conference on Learning Representations,
12. [12] R. Feng, J. Hu, W. Xia, A. Shen, Y. Sun, B. Fang, D. Hu, et al. (2025) AnyTouch: learning unified static-dynamic representation across multiple visuo-tactile sensors. In The Thirteenth International Conference on Learning Representations,
13. [13] Y. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, J. Kautz, and D. Fox (2021) DexYCB: a benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9044–9053.
14. [14] L. Yang, K. Li, X. Zhan, F. Wu, A. Xu, L. Liu, and C. Lu (2022) OakInk: a large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20953–20962.
15. [15] S. Hampali, M. Rad, M. Oberweger, and V. Lepetit (2020) HOnnotate: a method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
16. [16] X. Zhan, L. Yang, Y. Zhao, K. Mao, H. Xu, Z. Lin, K. Li, and C. Lu (2024) Oakink2: a dataset of bimanual hands-object manipulation in complex task completion. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 445–456.
17. [17] R. Patel, R. Ouyang, B. Romero, and E. Adelson (2020) Digger finger: gelsight tactile sensor for object identification inside granular media. In International Symposium on Experimental Robotics, pp. 105–115.
18. [18] S. Dong, W. Yuan, and E. H. Adelson (2017) Improved gelsight tactile sensor for measuring geometry and slip. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 137–144.
19. [19] M. B. Villalonga, A. Rodriguez, B. Lim, E. Valls, and T. Sechopoulos (2021) Tactile object pose estimation from the first touch with geometric contact rendering. In Conference on robot learning, pp. 1015–1029.
20. [20] M. Bauza, A. Bronars, and A. Rodriguez (2023) Tac2pose: tactile object pose estimation from the first touch. The International Journal of Robotics Research 42 (13), pp. 1185–1209.
21. [21] H. Xue, J. Ren, W. Chen, G. Zhang, Y. Fang, G. Gu, H. Xu, and C. Lu (2025) Reactive diffusion policy: slow-fast visual-tactile policy learning for contact-rich manipulation. In Proceedings of Robotics: Science and Systems (RSS),
22. [22] W. Chen, H. Xue, Y. Wang, F. Zhou, J. Lv, Y. Jin, S. Tang, C. Wen, and C. Lu (2026) Implicitrdp: an end-to-end visual-force diffusion policy with structural slow-fast learning. IEEE Robotics and Automation Letters.
23. [23] T. P. Tomo, M. Regoli, A. Schmitz, L. Natale, H. Kristanto, S. Somlor, L. Jamone, G. Metta, and S. Sugano (2018) A new silicone structure for uskin—a soft, distributed, digital 3-axis skin sensor and its integration on the humanoid robot icub. IEEE Robotics and Automation Letters 3 (3), pp. 2584–2591. External Links: Document
24. [24] Y. Song, J. Wang, Z. Li, W. Hu, Y. Qiu, Y. Tian, P. Zhao, A. Liu, and H. Wu (2026) Fingertip-scale six-axis tactile interface with high-precision force sensing and position localization for dexterous human–machine interactions. Microsystems & Nanoengineering 12 (1), pp. 193.
25. [25] BrainCo Revo 2 dexterous hand documentation. Note: https://www.brainco-hz.com/docs/revolimb-hand/en/revo2/parameters.htmlAccessed: 2026-08-16
26. [26] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick (2022) Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 15979–15988.
27. [27] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P. Huang, H. Xu, V. Sharma, S. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski (2023) DINOv2: learning robust visual features without supervision.
28. [28] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763.
29. [29] F. Yang, C. Feng, Z. Chen, H. Park, D. Wang, Y. Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens, et al. (2024) Binding touch to everything: learning unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26340–26353.
30. [30] R. Hoque, P. Huang, D. Yoon, J. Zhang, et al. (2026) Egodex: learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations, Vol. 2026, pp. 4218–4237.
31. [31] R. Punamiya, S. Kareer, Z. Liu, J. Citron, R. Qiu, X. Cai, A. Gavryushin, J. Chen, D. Liconti, L. Y. Zhu, et al. (2026) Egoverse: an egocentric human dataset for robot learning from around the world. arXiv preprint arXiv:2604.07607.
32. [32] H. Luo, Y. Feng, W. Zhang, S. Zheng, Y. Wang, H. Yuan, J. Liu, C. Xu, Q. Jin, and Z. Lu (2026) Being-h0: vision-language-action pretraining from large-scale human videos. In International Conference on Machine Learning,
33. [33] H. Bi, L. Wu, T. Lin, H. Tan, Z. Su, H. Su, and J. Zhu (2025) H-rdt: human manipulation enhanced bimanual robotic manipulation. External Links: 2507.23523, Link
34. [34] Y. Wi, J. Yin, E. Xiang, A. Sharma, J. Malik, M. Mukadam, N. Fazeli, and T. Hellebrekers (2026) TactAlign: human-to-robot policy transfer via tactile alignment. External Links: 2602.13579, Link
35. [35] S. Zhao, X. Zhu, Y. Chen, C. Li, Y. Xie, X. Zhang, M. Ding, and M. Tomizuka (2025) Dexh2r: task-oriented dexterous manipulation from human to robots. IEEE/ASME Transactions on Mechatronics.
36. [36] H. Zeng, L. Chen, J. Yang, Y. Zhang, F. Shi, P. Liu, and X. Zuo (2026) FlowHOI: flow-based semantics-grounded generation of hand-object interactions for dexterous robot manipulation. arXiv preprint arXiv:2602.13444.
37. [37] Y. Liu, H. Yang, X. Si, L. Liu, Z. Li, Y. Zhang, Y. Liu, and L. Yi (2024) Taco: benchmarking generalizable bimanual tool-action-object understanding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21740–21751.
38. [38] Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges (2023) ARCTIC: a dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
39. [39] P. Banerjee, S. Shkodrani, P. Moulon, S. Hampali, S. Han, F. Zhang, L. Zhang, J. Fountain, E. Miller, S. Basol, R. Newcombe, R. Wang, J. J. Engel, and T. Hodan (2025) HOT3D: hand and object tracking in 3D from egocentric multi-view videos. CVPR.
40. [40] J. Romero, D. Tzionas, and M. J. Black (2017) Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36 (6).
41. [41] J. L. Ba, J. R. Kiros, and G. E. Hinton (2016) Layer normalization. arXiv preprint arXiv:1607.06450.
42. [42] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
43. [43] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou (2021) Training data-efficient image transformer & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, pp. 10347–10357. External Links: Link
44. [44] Unitree Robotics (2024) XR-Teleoperate: an open-source teleoperation framework and data collection toolkit for embodied intelligence. Note: https://github.com/unitreerobotics/xr_teleoperateAccessed: 2026-02
45. [45] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea. External Links: Document
46. [46] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), pp. 1684–1704.