ETRI 'HuTact', 사람 손 영상으로 촉각 로봇 손 작업 성공률 33.8%에서 50%로
실제 촉각 데이터 없이 사람이 물체를 다루는 24시간 분량의 기록으로 로봇 손의 촉각 인코더를 사전학습하자 촉각에 민감한 두 작업의 평균 성공률이 촉각 입력이 없을 때 33.8%에서 50.0%로 올랐다고 한국전자통신연구원(ETRI) 연구진이 프리프린트에서 밝혔다. 실험은 브레인코 레보2 손을 단 유니트리 G1 휴머노이드에서 진행했고, 파지 예측에서는 데이터 4분의 1만으로 전체 데이터를 쓴 기준 모델과 같은 성능을 냈다.
손과 물체의 형상은 담겼지만 촉각 신호는 없는 대규모 공개 데이터가, 로봇 손이 스스로 촉각 데이터를 많이 모으기 전에 쓸모 있는 촉각 감각을 가르칠 수 있는지를 묻는다.
다지 로봇 손의 촉각 학습은 로봇에서 직접 모은 데이터가 많이 필요하고, 그 데이터는 특정 손과 센서 배치에 묶여 있어 하드웨어가 바뀌면 거의 옮겨 쓰지 못한다. 실제 장비로 모으는 데 시간과 비용도 크다.
기존 연구는 특정 센서로 모은 대규모 촉각 데이터로 인코더를 자기지도 사전학습하거나 여러 시각 촉각 센서의 데이터를 정렬했다. 사람 손과 물체 상호작용 데이터는 VLA 학습에는 쓰였지만 로봇 촉각의 사전 지식으로는 쓰이지 않았다.
HuTact는 HOT3D, OakInk-v2, TACO, ARCTIC 등 공개 데이터 4종의 257만 프레임(약 24시간)에서 손끝 좌표와 형상 기반 의사 접촉 신호를 뽑아, 좌표 토큰과 촉각 토큰을 분리하고 15프레임 구간을 묶는 트랜스포머를 사전학습한다. 이어 메타 퀘스트3 원격조종으로 모은 레보2 손 데이터 8만 프레임(약 44분)으로 중간 학습을 한다.
컵 분리와 카드 한 장 밀기 작업(작업당 40회)에서 ACT 정책은 HuTact 적용 시 평균 50.0%로, 촉각 입력 없음 33.8%, 처음부터 학습한 촉각 인코더 32.5%를 앞섰다. 디퓨전 정책은 22.5%와 38.8%에서 50.0%로 올랐다. 파지 예측 Macro-F1은 처음부터 학습한 88.23에서 93.29가 됐다. 전체 데이터를 쓴 기준 모델과 같은 성능에 필요한 데이터는 파지 예측 25%(88.3 대 88.2), 물체 분류 50%(86.2 대 85.1), 미끄러짐 감지 75%(75.9 대 75.7)였다.
프리프린트다. 연구진은 의사 접촉 신호가 실제 접촉의 물리적 성질을 담지 못하고 세 작업 중 두 작업에서 손끝 좌표 대비 개선 폭이 작으며, 손과 물체의 3D 메시가 필요하다고 밝혔다. 실험은 레보2 양손 구성 하나에 한정됐고, 50% 성공률은 개선이지만 비교적 단순한 작업에서도 절반은 실패한다는 뜻이다.
테솔로, 원익로보틱스 같은 국내 로봇 손 기업과 외부 손을 쓰는 휴머노이드 업체가 새 손마다 모아야 하는 촉각 데이터를 줄일 수 있다. 공개된 사람 데이터에 기대므로 연구 현장에서는 바로 쓸 수 있고, 센서 간 이식이 확인되면 1~2년 안에 제품 학습 파이프라인에 들어갈 수 있다. 정부출연연이 휴머노이드 손의 데이터 부족 문제를 직접 다룬 연구라는 점에서 국내 손 업체와의 공동 검증이 다음 단계다.
논문 전문
Learning Tactile-Aware Hand Representations from Human Hand-Object Interaction Data
CC BY 4.0 라이선스로 공개된 논문입니다. 출처를 밝혀 전재하며, 원문은 arXiv:2610.27995(PDF)에서 볼 수 있습니다.
Abstract
Tactile representation learning for dexterous manipulation often requires substantial robot-side data collection, limiting its scalability across different hands and sensors. We explore whether large-scale human hand-object interaction (HOI) data can provide a useful prior for robotic tactile learning. To this end, we propose HuTact, a framework that pretrains tactile-aware hand representations from human HOI data without requiring real tactile signals during pretraining. HuTact transfers the interaction structure learned from human hand motions to robotic tactile observations and can be further adapted with a small amount of unlabeled robot interaction data. We evaluate HuTact on tactile perception and manipulation tasks, including grasp prediction, slip detection, object classification, and tactile-conditioned policy learning. Across these tasks, human HOI pretraining improves data efficiency and provides a strong initialization for downstream learning. Notably, with HOI pretraining followed by midtraining on unlabeled robot play data, HuTact achieves 88.26% Macro-F1 on grasp prediction using only 25% of the labeled downstream data, comparable to training from scratch on the full labeled dataset (88.23%). These results suggest that human interaction data can serve as a scalable complementary source for robotic tactile representation learning.
I Introduction
Tactile sensing enables humans to perceive and interact with the physical world, providing direct information about contact, force, texture, and motion that is difficult to obtain through vision alone [1]. Inspired by this remarkable human capability, tactile sensing has emerged as a critical modality for robotic manipulation, enabling robots to perceive subtle contact interactions, infer local physical properties, and regulate contact forces during complex manipulation tasks that demand precise and adaptive physical interaction [2, 3, 4].
Accordingly, a wide range of tactile sensors have been developed to capture rich information arising from physical contact, including vision-based sensors (e.g., GelSight [2], DIGIT [5]) and magnetic-based sensors (e.g., AnySkin [4], ReSkin [6]). Beyond advances in tactile sensing hardware, the growing availability of large-scale tactile interaction data has further motivated learning-based approaches to extract generalizable representations from raw tactile observations [7]. In particular, recent works have explored self-supervised pretraining using large-scale tactile interaction data collected with respective tactile sensors, learning representations that can be transferred to downstream manipulation tasks [8, 9, 10, 11].
Learning generalizable tactile representations can reduce the amount of task-specific data required for downstream tasks. For vision-based tactile sensing, Feng et al. propose AnyTouch [12], which learns transferable representations from a sensor-aligned dataset collected across multiple visuo-tactile sensors. For non-vision-based tactile sensing, Sharma et al. [8] apply DINO-based self-distillation to masked tactile sequences collected from a dexterous hand, while Guzey et al. [9] pretrain tactile representations using task-agnostic robotic play data. However, existing approaches for non-vision-based tactile sensors still largely require real tactile interaction data collected directly with the target robotic hand and sensing hardware. As a result, adapting representation learning to a new robotic platform or tactile sensor requires additional data collection, introducing substantial time and cost for pretraining.
As an alternative, human hand-object interaction (HOI) data provide a promising source of pretraining supervision, where they contain large-scale and diverse interactions across objects, hand configurations, and manipulation behaviors [13, 14, 15, 16]. These datasets are particularly attractive for learning transferable hand representations because they provide (1) interaction patterns that can be shared across different hand embodiments, (2) large-scale interaction data that would be costly to collect on robotic platforms, and (3) an opportunity to reduce embodiment-specific data collection when adapting to a new robotic hand. Many of these datasets also provide temporally aligned 3D reconstructions of human hands and manipulated objects, allowing fingertip configurations and their evolution during interaction to be recovered without collecting data on a robotic platform. Although HOI datasets do not contain real tactile signals, their hand–object trajectories encode spatial and temporal structure associated with physical interaction. We hypothesize that this structure can serve as a transferable prior for robotic tactile representation learning and reduce the amount of robot-side tactile data required for adaptation.
In this work, our research question is: Can human hand-object interaction (HOI) data provide a transferable prior for robotic tactile representation learning? To answer this question, we propose HuTact, a framework for learning tactile-aware robot hand representations from human HOI data. Rather than attempting to reconstruct real tactile signals from human data, HuTact learns the spatial and temporal structure of hand interactions from wrist-local fingertip trajectories. To bridge human HOI pretraining and robotic tactile sensing, we separately tokenize fingertip coordinates and tactile observations, allowing the coordinate representation and shared Transformer backbone learned from HOI data to be retained while adapting the sensor-specific tactile input embedding. We further incorporate geometry-derived pseudo-contact signals as an auxiliary cue, encouraging the model to capture the relationship between fingertip coordinates and their corresponding tactile-like signals. This framework enables large-scale human interaction data to provide a transferable initialization for tactile-aware robot learning while reducing the amount of robot-side tactile data required for adaptation.
To validate the effectiveness and transferability of HuTact, we evaluate it on tactile-sensitive downstream tasks, including grasp prediction, slip detection, object classification, and policy learning on tactile-sensitive tasks, and analyze the factors that enable transfer from human HOI data to robotic tactile sensing. In summary, our contributions are as follows:
- We propose HuTact, a framework that uses large-scale human hand-object interaction data to pretrain tactile-aware robot hand representations without requiring real tactile signals during the initial pretraining stage.
- We show that representations learned from HOI data transfer to robotic tactile perception and manipulation, improving grasp prediction, slip detection, object classification, and tactile-conditioned policy learning.
- We provide systematic analyses of the transfer mechanism through architectural ablations and data-efficiency experiments, studying the roles of temporal modeling, token separation, pseudo-contact cues, and robot-side midtraining. Our results show that HOI pretraining can substantially reduce the amount of task-specific robotic tactile data required for downstream adaptation.
II Related Work
Tactile Sensing: Tactile sensors for robotic manipulation broadly include vision-based and non-optical sensing modalities. Vision-based sensors [2, 5] capture fine-grained contact geometry and have been applied to object identification [17], geometry sensing [18, 3], pose estimation [19, 20], and policy learning [21, 22]. However, their optical components can result in a relatively large form factor, making integration across multiple fingers challenging. Non-optical sensors instead use compact magnetic [4, 23] or capacitive sensing elements [24] to enable distributed tactile sensing. In this work, we use the BrainCo Revo 2 hand [25], which is equipped with fingertip capacitive sensors that measure normal force, tangential force, tangential-force direction, and proximity.
Tactile Representation Learning: Recent work has explored transferable tactile representations through large-scale self-supervised pretraining. For vision-based tactile sensors, objectives developed for visual representation learning [26, 27, 28] have been adapted to tactile observations [29, 10, 12, 11]. For skin-like tactile sensors, Sharma et al. [8] apply self-distillation to masked tactile sequences from a dexterous hand, while Guzey et al. [9] pretrain tactile representations using task-agnostic robotic play data. While these approaches demonstrate the benefits of tactile pretraining, they generally require interaction data collected with the target robot and tactile sensor. In contrast, HuTact initializes tactile-aware representations from human hand–object interaction data without real tactile signals. We pretrain on the spatial and temporal structure of wrist-local fingertip interactions, augmented with geometry-derived pseudo-contact cues, and subsequently adapt the representation to the target robotic tactile sensor.
Robot Learning from Human Data: Human interaction data have been increasingly used to reduce robot-specific supervision for manipulation learning [30, 31, 32, 33, 34], by extracting hand poses, object motions, and interaction trajectories from large-scale demonstrations. HOI data have also been applied directly to dexterous manipulation: Zhao et al. [35] retarget human hand-object interactions to dexterous robot hands and learn a residual policy for embodiment mismatch, while Zeng et al. [36] use HOI data for vision-language-action learning. Large-scale datasets such as OakInk2 [16], TACO [37], ARCTIC [38], and HOT3D [39] provide scalable supervision for learning hand–object interaction structure. HuTact differs in objective: rather than transferring human motion directly to a robot policy, we use HOI data to pretrain a tactile-aware hand representation that is subsequently adapted to real robotic tactile sensing. To our knowledge, prior work has not systematically studied large-scale HOI data as a pretraining source for robotic tactile representation learning.
III Methodology
HuTact is designed to transfer interaction structure learned from human HOI data to robotic tactile sensing. The key challenge is that the source HOI datasets provide hand and object geometry but no real tactile signals, whereas the target robot provides sensor-specific tactile observations. We address this mismatch by factorizing the input representation into two streams: a shared wrist-local fingertip coordinate stream and a sensor-dependent contact stream. The shared Transformer backbone is pretrained on human interactions, while the sensor-specific tactile input embedding is adapted when transferring to real robotic tactile observations.
III-A Problem setup
We consider two dexterous robot hands (BrainCo Revo2 Hand) with N{=}10 fingertip tactile sensors. At each timestep t, sensor i captures a signal \mathbf{s}_{i}^{t}\in\mathbb{R}^{C} with C{=}4 channels—normal force, tangential force, tangential direction, and proximity—and forward kinematics provides the corresponding fingertip coordinate \mathbf{c}_{i}^{t}\in\mathbb{R}^{3} in the wrist-local frame as in [33]. The model operates on a temporal window
\mathbf{X}=\{(\mathbf{c}_{i}^{t},\,\mathbf{s}_{i}^{t})\}_{i=1,\dots,N}^{t=1,\dots,T},with T{=}15 frames sampled at 30 Hz, spanning 0.5 s of contact. Then, we pretrain a tactile encoder f_{\theta} with human-object interaction (HOI) data and subsequently transfer it to robotic downstream tasks. To maintain a consistent input structure across the human and robot domains, human hand samples are represented using fingertip coordinates paired with a one-dimensional geometry-derived pseudo-contact signal (i.e., C=1), whereas robot observations use the corresponding fingertip coordinates paired with four-dimensional tactile signals.
III-B HOI Data Processing
The pretraining corpus is assembled from HOI dataset. We obtain its 3D coordinate from the MANO hand model [40] and extract a one-dimensional pseudo-contact signal from the geometric relationship between the hand and object meshes. First, we convert the MANO hand representation [40] into a common 21-joint skeleton by augmenting the 16 MANO kinematic joints with five fingertip vertices and reordering them into a unified joint convention. From this representation, we extract wrist-local fingertip coordinates and pseudo-contact signals for HOI pretraining.
Fingertip coordinates. For each hand and frame, we construct a wrist-local coordinate frame from the hand geometry. The wrist defines the origin, while the palm-normal and finger directions are derived from the pinky–thumb and wrist–middle-finger axes and orthogonalized to form a right-handed basis. We transform all joints into this frame and retain the five fingertips per hand, yielding N{=}10 coordinates \mathbf{c}_{i}^{t}\in\mathbb{R}^{3}. During the experiment, robot fingertip coordinates from forward kinematics use the same local convention.
Pseudo-contact signals. We define a surface patch \mathcal{P}_{i}^{t} over each fingertip pad rather than estimating contact from a single fingertip point. MANO vertices are assigned to joints using dominant skinning weights, and each fingertip is augmented with the 20 nearest rest-pose vertices around its fingertip vertex. At each frame, we retain only vertices on the palmar side using the local pad direction, preventing dorsal-side vertices from producing spurious contact responses.
For each retained vertex v\in\mathcal{P}_{i}^{t}, we compute the shortest point-to-triangle distance d_{v}^{t} to the object surface. Given a contact threshold of d_{\max}=10 mm, we define
u_{v}^{t}=\frac{\max(d_{\max}-d_{v}^{t},\,0)}{d_{\max}},\qquad r_{v}^{t}=(u_{v}^{t})^{2}(3-2u_{v}^{t}),where r_{v}^{t}\in[0,1] provides a smooth proximity response.
We then aggregate the vertex responses using their associated surface areas:
s_{i}^{t}=\frac{\sum_{v\in\mathcal{P}_{i}^{t}}a_{v}^{t}r_{v}^{t}}{\sum_{v\in\mathcal{P}_{i}^{t}}a_{v}^{t}},where a_{v}^{t} is the representative surface area of vertex v, computed from its incident triangles. This area-weighted formulation reduces sensitivity to mesh tessellation and yields a bounded proximity-like signal s_{i}^{t}\in[0,1], which increases as a larger portion of the fingertip pad approaches the object surface. When multiple objects are present, we select the object yielding the largest s_{i}^{t} for each fingertip. For HOI pretraining, we use s_{i}^{t} as a one-dimensional contact observation (C=1).
III-C Architecture
As shown in Fig. 2, HuTact consists of separate coordinate and tactile tokenizers, a frame encoder, and a temporal encoder.
Tokenizers. Within a frame t, fingertip coordinates c^{t} and tactile signals s^{t} are tokenized as two separate streams: \{\mathbf{c}_{i}^{t}\} and \{\mathbf{s}_{i}^{t}\} are projected by distinct input embeddings into N coordinate tokens and N tactile tokens, each stream normalized with its own statistics. Specifically, the tokenizing process can be formalized as follows:
z_{c,i}^{t}=\mathrm{LN}_{c}\!\left(\mathbf{W}_{c}\mathbf{c}^{t}_{i}+b_{c}\right),\qquad z_{s,i}^{t}=\mathrm{LN}_{s}\!\left(\mathbf{W}_{s}\mathbf{s}^{t}_{i}+b_{s}\right),where the z_{c,i}^{t}\in\mathbb{R}^{D} and z_{s,i}^{t}\in\mathbb{R}^{D} denote the coordinate tokens and tactile tokens for the i-th finger, respectively. Also, \mathbf{LN}, \mathbf{W}\in\mathbb{R}^{D\times m} (m is the dimension of input), and b\in\mathbb{R}^{D} refer to the Layer Norm [41], linear weight, and bias, respectively. Finally, we concatenate these tokens with [CLS] token as:
\mathbf{Z}^{t}=\left[\mathbf{z}^{t}_{\mathrm{CLS}},\{\mathbf{z}_{c,i}^{t}\}_{i=1}^{N},\{\mathbf{z}_{s,i}^{t}\}_{i=1}^{N}\right].As shown in this Eq 4 and Eq 5, we separate the token input for the tactile signals and fingertip coordinates unlike previous approaches [8]. This token separation is particularly important for transferring from HOI pretraining to real tactile sensing: while the contact signal used during pretraining and the real tactile signal have different dimensionalities (i.e., 1 dim vs. 4 dim), the coordinate tokenizer remains unchanged and can be directly reused after pretraining.
Frame encoder. Given the concatenated token sequence \mathbf{Z}^{t} at each timestep t\in\{1,2,\ldots,T\}, we apply 2D multimodal RoPE [42] using hand and finger identities as the two positional dimensions. We then process the tokens with a lightweight 4-layer Transformer encoder, following the remaining architectural hyperparameters of ViT-Tiny architecture [43]. Through self-attention, the encoder captures interactions between tactile observations and fingertip coordinates, producing a unified representation of the hand-object contact state at each timestep. The frame-level representations are then passed to the temporal encoder to model their relationship over time. Specifically, we take the [CLS] token from each frame encoder output and concatenate them along the token dimension to construct a sequence of frame-level representations. Given T=15 input frames, we obtain
\mathbf{H}=\left[\mathbf{h}_{\mathrm{CLS}},\hat{\mathbf{z}}^{1}_{\mathrm{CLS}},\hat{\mathbf{z}}^{2}_{\mathrm{CLS}},\ldots,\hat{\mathbf{z}}^{15}_{\mathrm{CLS}}\right],where \mathbf{h}_{\mathrm{CLS}} and \hat{\mathbf{z}}^{i}_{\mathrm{CLS}} are new CLS token for temporal encoder and output CLS token for i-th frame with frame encoder, respectively.
Temporal encoder. Given the frame-level sequence \mathbf{H}, the temporal encoder applies RoPE based on the absolute frame index and models temporal dependencies across the T=15 frame-level [CLS] tokens. It uses the same Transformer architecture as the frame encoder with independent parameters. The output temporal [CLS] token \hat{\mathbf{h}}_{\mathrm{CLS}} is used as the final window-level representation. This representation summarizes the spatiotemporal hand-object interaction over the entire T=15 frame window and is subsequently used by the downstream prediction head.
III-D Training objectives and stages
Objective. We pretrain the model using temporal self-distillation following the DINOv2 [27] as [8] in the pretraining and midtraining stages. A student network is trained to match an exponential-moving-average (EMA) teacher by combining a distillation loss on the temporal [CLS] embedding with a masked-token prediction loss on the frame-level tokens.
Stage 1: Pretraining on HOI data. In the first stage, the encoder is pretrained for 200 epochs using the objective described above on the HOI data. The data consists of contact signals and fingertip coordinates derived from public HOI data. Note that, no robot data are used at any point during this stage.
Stage 2 (Optional): Midtraining on robotic play data. In the second stage, we optionally continue self-supervised training using the same objective and optimization recipe on a small unlabeled robotic play data. Because the real robot provides C=4 tactile sensor channels, whereas Stage 1 uses a different pseudo-tactile channel configuration, we re-initialize the tactile input tokenizer at the transition between the two stages while retaining all remaining pretrained parameters. Each epoch repeats the robot corpus 32 times, preserving approximately the same number of training samples per epoch as in Stage 1 (\approx 505k samples per epoch). We train for 100 epochs using the same warmup-cosine learning-rate schedule. Hand poses are min-max normalized to the unit box during this stage.
Stage 3: Downstream transfer. For downstream transfer, we fine-tune the full encoder jointly with an attentive-pooling task head. When Stage 2 midtraining is not conducted, we re-initialize the tactile input tokenizer before downstream fine-tuning; otherwise, the tokenizer adapted during Stage 2 is used. For the given downstream data, we use AdamW with a constant learning rate of 1\times 10^{-4} for 100 epochs.
Optimization. We use AdamW with a peak learning rate of 1\times 10^{-4} and a batch size of 512. The learning rate is linearly warmed up for the first 10 epochs and subsequently decayed using a cosine schedule to 1\times 10^{-6}. The teacher temperature is increased from 0.04 to 0.07 during the first 2 epochs. The EMA momentum of the teacher is increased from 0.994 to 1.0 throughout training, while the weight decay follows a cosine schedule ending at 0.04.
IV Experiments
IV-A Experimental setup
We use a Unitree G1 humanoid equipped with a BrainCo Revo2 dexterous hand [25], which has capacitive tactile sensors at each fingertip, as shown in Fig. 3.
Pretraining datasets. For HOI pretraining, we use HOT3D [39], OakInk-v2 [16], TACO [37], and ARCTIC [38], extracting fingertip coordinates and pseudo-contact signals from 2.57M frames (\sim24 hours). We additionally collect 80K frames (\sim44 minutes) of Revo2 play data containing fingertip coordinates and tactile signals through Meta Quest 3 teleoperation [44] with six objects (e.g., plastic bottle, bubble wrap, driver, pot, sponge, and towel).
Downstream datasets. For grasp prediction, we collect 590 teleoperated grasp episodes over five objects, totaling 18K frames (\sim10 minutes), with each episode labeled by grasp success. For slip detection, we hold the robot hand fixed and manually move the grasped object while tracking an attached marker; marker motion is labeled as slip, yielding 18 episodes over six object conditions and 51K frames (\sim28 minutes). For object classification, we reuse successful grasp episodes, extract the result-phase frames, and label the grasped object identity, yielding 252 episodes and 43K frames (\sim24 minutes).
For ID evaluation, we perform episode-level cross-validation before constructing temporal windows, ensuring that all windows from the same episode remain in the same split. We use stratified 4-fold cross-validation for grasp prediction and object classification, with 75% of the episodes for training and 25% for evaluation in each fold, and 3-fold cross-validation for slip detection. Reported ID performance is averaged across the corresponding folds.
For policy learning, we collect two datasets using a scripted controller. The cup-grasping and card-sliding datasets contain 40 and 55 episodes, respectively, each totaling approximately 30K frames (\sim17 minutes). For each policy configuration, success rates are computed over 40 evaluation trials per task.
IV-B Performance on downstream tasks
- Grasp Prediction
- From Scratch
- 88.23 \pm 0.78
- 86.39 \pm 0.38
- 74.92 \pm 9.78
- 74.12 \pm 9.78
- Revo2 Pretraining
- 92.15 \pm 0.40
- 92.09 \pm 0.35
- 77.84 \pm 9.31
- 77.51 \pm 9.15
- 자료 없음
- HOI Pretraining (Ours)
- 92.61 \pm 0.74
- 92.18 \pm 0.75
- 79.21 \pm 10.06
- 78.46 \pm 10.05
- 자료 없음
- HOI + Revo2 Midtraining (Ours)
- 93.29 \pm 0.53
- 93.24 \pm 0.54
- 80.27 \pm 9.52
- 80.02 \pm 9.49
- 자료 없음
- Slip Detection
- From Scratch
- 75.68 \pm 1.73
- 73.14 \pm 0.73
- 69.28 \pm 4.55
- 67.39 \pm 3.24
- Revo2 Pretraining
- 72.37 \pm 1.69
- 71.55 \pm 1.96
- 65.75 \pm 4.38
- 65.80 \pm 3.87
- 자료 없음
- HOI Pretraining (Ours)
- 75.28 \pm 1.00
- 73.78 \pm 0.80
- 69.56 \pm 4.38
- 68.86 \pm 3.14
- 자료 없음
- HOI + Revo2 Midtraining (Ours)
- 76.88 \pm 1.03
- 76.41 \pm 0.85
- 69.79 \pm 4.24
- 69.83 \pm 3.75
- 자료 없음
- Object Classification
- From Scratch
- 85.11 \pm 1.35
- 82.96 \pm 0.71
- N/A
- N/A
- Revo2 Pretraining
- 89.07 \pm 1.31
- 88.85 \pm 1.23
- N/A
- N/A
- 자료 없음
- HOI Pretraining (Ours)
- 87.64 \pm 0.82
- 87.31 \pm 1.08
- N/A
- N/A
- 자료 없음
- HOI + Revo2 Midtraining (Ours)
- 89.89 \pm 1.52
- 89.70 \pm 1.48
- N/A
- N/A
- 자료 없음
- ACT
- None
- 30.0
- 37.5
- 33.8
- From Scratch
- 5.0
- 60.0
- 32.5
- 자료 없음
- HOI + Revo2 Midtraining (Ours)
- 37.5
- 62.5
- 50.0
- 자료 없음
- Diffusion Policy
- None
- 27.5
- 17.5
- 22.5
- From Scratch
- 37.5
- 40.0
- 38.8
- 자료 없음
- HOI + Revo2 Midtraining (Ours)
- 47.5
- 52.5
- 50.0
- 자료 없음
In Table I, we evaluate HuTact on three downstream perception tasks: grasp prediction, slip detection, and object classification. Grasp prediction and slip detection are binary classification tasks that predict grasp success from observations collected 1 second after grasp execution and whether the current grasp is slipping or stable, respectively. Object classification is a five-way classification task that predicts the identity of the grasped object from observations collected during the post-grasp phase. Also, in the ID setting, all object categories are included in both the training and evaluation sets. In the OOD setting, we use a leave-one-object-out protocol, where one object is excluded from training and used only for evaluation, while the remaining objects are used exclusively for training (except object classification). We compare HuTact against two baselines: (i) From Scratch, where the model is randomly initialized, and (ii) Revo2 Pretraining, where the encoder is pretrained on Revo2 play data using the same self-supervised objective as Stage 1.
HOI pretraining (Stage 1 only) consistently improves grasp prediction over both training from scratch and Revo2-only pretraining, with particularly clear gains in the OOD setting. For slip detection, HOI pretraining provides smaller but consistent gains in ID Avg and OOD performance, while achieving comparable ID Last performance to training from scratch. Revo2-only pretraining performs worse on slip detection, indicating that a small amount of robot play data alone is insufficient to learn transferable features for this task. For object classification, however, Revo2 pretraining outperforms HOI pretraining, suggesting that real tactile observations contain additional object-specific information. Across all three tasks, HOI pretraining followed by Revo2 midtraining achieves the best overall performance, demonstrating that large-scale HOI data and robot-side tactile data provide complementary benefits for downstream tactile representation learning.
IV-C Performance with action policy
In Table II, we evaluate the learned tactile representation on two contact-sensitive manipulation tasks using ACT [45] and Diffusion Policy (DP) [46]. Cup Destacking requires the robot to separate a cup from a stack while maintaining a stable grasp, whereas Single-Card Sliding requires the robot to slide a single card without displacing the remaining cards. We compare three tactile configurations: (i) no tactile input, (ii) a tactile encoder trained from scratch, and (iii) our HOI-pretrained encoder followed by Revo2 midtraining. We modify ResNet18-based ACT and DP with our tactile encoder. For ACT, the tactile representation is linearly projected to the Transformer embedding dimension and appended as a single token to the encoder input sequence. The tactile representation is used only as a conditioning input to the policy and is not provided to the VAE encoder. For Diffusion Policy, we concatenate the tactile representation directly with the visual observation features in the global conditioning vector.
Our tactile representation achieves the best overall performance for both policy architectures. With ACT, HOI + Revo2 midtraining improves the average success rate to 50.0%, compared with 33.8% without tactile input and 32.5% with a tactile encoder trained from scratch. With Diffusion Policy, the improvement is more consistent across both tasks: our method achieves 47.5% on Cup Destacking and 52.5% on Single-Card Sliding, yielding an average success rate of 50.0%, compared with 22.5% without tactile input and 38.8% from scratch. We observe that ACT with a tactile encoder trained from scratch often grasps multiple cups during Cup Destacking, which is counted as a failure and contributes to its low success rate of 5.0%. These results indicate that the pretrained tactile representation provides useful contact-sensitive features for downstream policy learning beyond either omitting tactile input or learning the tactile encoder solely from task-specific demonstrations.
IV-D Transfer performance comparison
- Sparsh-Skin [8]
- Allegro + Xela uSkin
- 87.8
- 70.6
- 81.0
- HuTact (Ours)
- HOI
- 92.2
- 73.8
- 87.3
Table III compares HuTact with Sparsh-Skin [8] to evaluate cross-sensor transfer across different pretraining sources and encoder architectures. We initialize Sparsh-Skin from the authors’ publicly released checkpoint, pretrained on an Allegro hand equipped with Xela uSkin sensors, and fine-tune it on the Revo2 downstream tasks. Because the two platforms differ in the number of sensors and tactile signal dimensions, we reinitialize the patch and positional embeddings while retaining the remaining pretrained parameters. HuTact consistently outperforms Sparsh-Skin across all three downstream tasks. This performance gap is consistent with the benefits of token separation for transfer across sensing configurations and temporal modeling for capturing contact dynamics over time. These results indicate that HuTact provides an effective initialization for robotic tactile representation learning.
IV-E Ablation Studies
- —
- Temp. Encoder
- Token Sep.
- 자료 없음
- HOI Pretrain.
- Grasp Pred.
- Slip Det.
- Object Cls.
- A
- –
- –
- 자료 없음
- –
- 84.8
- 60.8
- 81.8
- B
- –
- –
- 자료 없음
- \checkmark
- 87.7
- 60.0
- 83.7
- C
- –
- \checkmark
- 자료 없음
- –
- 84.3
- 61.8
- 83.7
- D
- –
- \checkmark
- 자료 없음
- \checkmark
- 91.0
- 60.1
- 87.9
- E
- \checkmark
- \checkmark
- 자료 없음
- –
- 86.4
- 73.1
- 83.0
- F
- \checkmark
- \checkmark
- 자료 없음
- \checkmark
- 92.2
- 73.8
- 87.3
- (b) Pseudo-contact ablation
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- —
- Pseudo-Contact Signals
- 자료 없음
- Revo2 Midtrain.
- 자료 없음
- Grasp Pred.
- Slip Det.
- Object Cls.
- F′
- –
- 자료 없음
- –
- 자료 없음
- 92.0
- 72.6
- 87.3
- F
- \checkmark
- 자료 없음
- –
- 자료 없음
- 92.2
- 73.8
- 87.3
- G′
- –
- 자료 없음
- \checkmark
- 자료 없음
- 92.9
- 73.0
- 89.5
- G
- \checkmark
- 자료 없음
- \checkmark
- 자료 없음
- 93.2
- 76.4
- 89.7
In Table IV-(a), we analyze the effects of the temporal encoder, token separation, and HOI pretraining on the three downstream tasks. Configurations A and B use a non-temporal, unified-token architecture similar to prior tactile representation learning approaches [8], serving as a reference for our architectural modifications. (i) The temporal encoder provides the largest gain for slip detection: without HOI pretraining, adding temporal modeling improves Macro-F1 from 61.8% to 73.1% (C\rightarrowE), highlighting the importance of modeling contact dynamics over time. (ii) Token separation is particularly effective when combined with HOI pretraining, improving grasp prediction from 87.7% to 91.0% and object classification from 83.7% to 87.9% (B\rightarrowD). Separating fingertip coordinates from sensor-specific tactile inputs also allows the coordinate tokenizer learned from HOI data to be retained during transfer to the robot domain, and HOI pretraining yields larger gains when separate tokenizers are used. (iii) The complete configuration (F) achieves the best grasp prediction and slip detection performance among the architecture variants.
Table IV-(b) examines the contribution of pseudo-contact signals during Stage 1 pretraining. (i) Pseudo-contact pretraining provides its clearest benefit on slip detection, a task that relies strongly on tactile contact information, improving Macro-F1 from 72.6% to 73.8% without Revo2 midtraining (F{}^{\prime}\rightarrowF), while the gains in grasp prediction and object classification are negligible. (ii) This benefit becomes more evident after Revo2 midtraining, where pseudo-contact improves slip detection from 73.0% to 76.4% (G{}^{\prime}\rightarrowG). These results suggest that pseudo-contact pretraining helps the model capture the relationship between fingertip-coordinate and tactile tokens, providing a more transferable initialization when adapting to real robotic tactile signals.
IV-F Downstream Data Efficiency
We further evaluate the data efficiency of HuTact by varying the amount of downstream training data from 5% to 100%. As shown in Fig. 4, HOI + Revo2 midtraining consistently reduces the amount of task-specific data required across all three tasks. It matches or exceeds the full-data scratch baseline using only 25% of the data for grasp prediction (88.3% vs. 88.2%), 75% for slip detection (75.9% vs. 75.7%), and 50% for object classification (86.2% vs. 85.1%). These results demonstrate that HuTact substantially improves downstream data efficiency.
V Limitations
Our work has two main limitations. First, geometry-derived pseudo-contact signals do not capture physical contact properties and provide only modest additional gains beyond fingertip coordinates for grasp prediction and object classification. They also require reconstructed hand and object meshes, limiting applicability to HOI datasets with 3D geometry. Second, our evaluation is limited to a Revo2-based bimanual setup, leaving transfer to other robotic hands and tactile sensing modalities unexplored. Future work will investigate richer contact cues from visual and hand-motion observations and extend HuTact to broader robotic embodiments and sensor layouts.
VI Conclusion
In this work, we presented HuTact, a framework for pretraining tactile-aware robot hand representations from large-scale human hand-object interaction (HOI) data without requiring real tactile signals during initial pretraining. HuTact leverages fingertip coordinates and pseudo-contact signals, with separate coordinate and tactile tokenization to support transfer to robotic tactile sensing. Across perception and policy-learning tasks, HOI pretraining provides a useful initialization that is further improved by Revo2 midtraining, while also substantially improving downstream data efficiency. These results show that human interaction data can serve as an effective pretraining prior for robotic tactile learning and reduce the need for robot-side tactile data collection.