TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering
TouchTherm is a preprint framework that turns real objects into simulation-ready digital twins combining a coarse visual mesh, registered tactile micro-height fields, and a physics-regularized dynamic thermal field. On 20 objects it predicts held-out surface temperature with MAE 0.465°C at 30 s and 0.592°C at 45 s, and improves synthetic-to-real tactile object recognition from 20.0% to 34.0% Top-1 accuracy over coarse geometry. The work matters because it closes part of the simulation-to-real gap for contact-rich robotics and VR by making tactile and thermal cues queryable in simulation.
Can a unified object-asset pipeline reconstruct and represent both contact-scale tactile microgeometry and transient object-level temperature fields from real-world scans so they can be rendered in robotics simulation and VR?
Existing object assets and reconstruction pipelines are vision-centric: they capture metric geometry and appearance but not the micro-scale surface relief needed for optical tactile rendering nor the spatially varying temperature dynamics that depend on material and geometry. Microgeometry is either lost in coarse collision meshes or too expensive to encode as dense geometry, and thermal data remains local sensor observations rather than an object-level field.
Prior work includes structured-light scanning, multi-view stereo, NeRF, and Gaussian splatting for geometry/appearance; object datasets like YCB, Google Scanned Objects, and Objaverse; multisensory datasets such as ObjectFolder and X-Capture; and separate tactile simulators or thermal-material-recognition systems. They generally provide coarse collision geometry or partial multisensory observations, not registered tactile micro-height fields plus an object-specific dynamic thermal field.
TouchTherm uses an EinScan Pro 2X V2 handheld structured-light scanner, smartphone photometric stereo with SDM-UniPS normal estimation, and HIKMICRO P09 thermal cameras. Object-to-camera poses are solved with PnP; normal maps are transformed into tangent space and integrated by a Poisson/DCT solver to recover micro-height fields stored separately from the coarse mesh. A contact-conditioned renderer composites coarse indentation with calibrated micro-height. For thermal reconstruction, observations are projected onto a point cloud, a graph diffusion operator is built over k-nearest neighbors, and an MLP is jointly optimized with a physics loss to identify a surface-diffusion coefficient alpha and ambient-relaxation coefficient h; runtime uses explicit Euler rollout.
Across 20 objects, Real–Real tactile similarity G-SSIM/HF-NCC was 0.0954±0.0471/0.1622±0.0917, and TouchTherm Real–Sim was 0.0712±0.0217/0.0775±0.0575. In ablation over 3 contacts per object, TouchTherm reached G-SSIM/HF-NCC 0.0701±0.0246/0.0911±0.0648 vs Coarse Geometry 0.0480±0.0283/0.0116±0.0328 and Image-space Height 0.0629±0.0242/0.0081±0.0208. For thermal prediction on 300 held-out windows per horizon (20 objects × 15 windows), full model surface MAE at 30 s and 45 s was 0.465±0.240°C and 0.592±0.266°C, with Hotspot F1 0.793±0.179 and 0.708±0.246; removing h raised MAE to 0.820±0.411 and 1.083±0.493°C. Synthetic-to-real tactile recognition with a simulated-only ResNet-18: TouchTherm Top-1/Top-3/Macro-F1 34.0±2.7%/63.0±1.7%/29.3±2.5% vs coarse geometry 20.0±4.4%/35.0±2.3%/13.4±3.7%. In a 10-person VR study, visual–thermal consistency, cooling consistency, response timeliness, realism, and comfort scored 6.30, 6.20, 5.40, 6.10, and 6.40 on a 7-point scale.
Authors state microgeometry is used only for tactile rendering and not for contact mechanics, and the thermal model covers natural cooling only, not bidirectional contact heat transfer. The evaluation is a preprint with 20 objects, manual pose/correspondence annotation, per-object tactile calibration, and a single GelSight Mini sensor; thermal validation uses held-out natural-cooling videos, not robotic contact; no code release is described.
Robot simulator vendors such as NVIDIA Isaac Sim, tactile sensor makers such as GelSight, and VR/haptics glove developers could use this asset format for tactile and thermal rendering. Near-term (now): small object sets for simulation and demos. 1–3 years: scalable automated acquisition and validation on broader objects, integration with compliant contact/friction models, and contact-dependent thermal transfer would be required for production use in robot learning and embodied-AI datasets.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.