visuo-tactile world modelsExpert
ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
A world model trained only on 4.9 hours of people pushing and pulling objects with handheld GelSight touch sensors predicts the next half-second of a scene 0.7 to 2.0 dB PSNR better than two visuo-tactile models trained from scratch on the same data, researchers from MIT, Harvard, UW, Berkeley and Stanford report in a preprint. Run on a robot without fine-tuning, its predicted touch imprints still appear and vanish with real contact, while its camera predictions transfer unevenly.