Ego2Contact: Predicting Touch from Egocentric Human Video for Contact-Rich Robot Manipulation
A touch predictor trained on human first-person video with sensing gloves raised a simulated robot hand's bottle-cap turning success from 64.1% to 71.0%, close to the 74.1% reached with real tactile sensors, in a Texas A&M-led preprint. The idea is to give robots a sense of touch from cameras alone, because tactile hardware is costly and robot touch data is scarce.
Can a model that learns to infer hand pressure from human egocentric video be moved onto robot hands with different sensor layouts and then feed useful touch signals to robot control policies?
Where a hand touches an object and how hard it presses are often hidden from cameras, yet they decide whether a grasp slips or a cap turns. Robot tactile skins are expensive, tied to one gripper design and rarely recorded, so large cross-robot datasets carry almost no touch data. Human video is plentiful but has no touch channel either.
Earlier predictors such as TouchAnything learned pressure maps from glove-equipped human video but were only tested on human hands. Robot-side methods like TacImag and ViTacWorld imagine touch from robot data alone, and approaches such as VTDexManip and H-Tac tie the tactile estimate to one policy architecture or action space.
Ego2Contact fuses scene features from a frozen DINOv2 encoder over eight frames with hand-crop features from a frozen WiLoR backbone and uses a flow-matching diffusion transformer to generate a 21 by 21 pressure grid per hand (217 valid cells). Anatomical region mappings translate any robot's sensors, from 17 simulated links to an 880-taxel glove, into that grid, so robot readings become fine-tuning targets and predictions become policy inputs. The predictor stays separate from the policy and can serve reinforcement learning or vision-language-action post-training.
On the EgoTouch human benchmark, contact overlap (C.IoU) reached 0.5735 on seen objects against 0.5064 for the TouchAnything reproduction (13.3% higher by MEKAWIRE's calculation) and 0.4812 against 0.4720 on unseen objects, though unseen volumetric IoU was slightly lower (0.3417 vs 0.3457). After fine-tuning on 8,741 simulated robot clips, grid contact IoU rose from 0.008 to 0.476 (TouchAnything 0.415) and link-level centre-of-pressure error fell to 0.224 from 0.300, 25.3% lower. In four NVIDIA Isaac Gym tasks trained with PPO, predicted touch lifted success over no touch on lever sliding (84.0% to 88.8%, sensors 98.2%), bottle-cap turning (64.1% to 71.0%, sensors 74.1%) and bimanual handover (29.3% to 33.1%, sensors 49.1%); MEKAWIRE calculates it recovered 69%, 34% and 19% of the sensor benefit. Pen-cap removal stayed at 1.0% against 95.0% with real sensors.
The control gains are shown only in simulation: the real xArm6 and BrainCo Revo 2 hand setup (about 123k frames, two tasks with roughly 50 demonstrations each) is presented through camera views with no success rates. The pen-cap result shows predicted touch can fail completely where real sensing matters most. The robot mapping is hand-designed per robot, the real rollouts ran at about 24.5 Hz against a requested 30 Hz, and the authors say they have not yet measured how much robot data adaptation needs.
Makers of dexterous hands and humanoids that cannot afford dense tactile skins on every unit could train touch-aware policies from cameras, with a small calibration set from a gloved or sensorised reference hand. Hand suppliers such as BrainCo and tactile-glove makers gain a role as data sources rather than only as per-robot hardware. Expect this in research and pilot cells within one to two years; production use needs the real-robot success numbers this preprint does not yet give.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.