Equivariant Visual-Tactile Diffusion Policy for Contact-Rich Manipulation
This preprint introduces VISTA, a diffusion-policy architecture that fuses wrist camera and tactile images into a rotation-equivariant spherical representation for contact-rich manipulation. In simulation with 50 demonstrations per task it reaches 80.9% mean success, outperforming an equivariant tactile baseline by 15.9 points, and on three real-robot tasks it succeeds in 7.33/10 trials versus 1.33/10 for that baseline. The design could reduce the demonstration data needed for tight-insertion and surface-contact tasks by encoding workspace rotation symmetry into visuotactile perception.
Can a workspace-level equivariant visuotactile diffusion policy improve data efficiency and robustness for contact-rich manipulation compared with non-equivariant and generic visuotactile or vision-only baselines?
Contact-rich tasks such as USB insertion, pencil insertion, and erasing require precise tactile feedback, but expert demonstrations are expensive and limited, and small pose deviations cause failures. Existing imitation policies treat vision and touch as generic features, so they require many demonstrations to cover object and gripper poses and struggle under limited data; vision-only policies miss critical contact cues.
Prior work includes non-equivariant visuotactile diffusion and action-chunking methods such as ManiFeel, VITAL, and UniVTAC; image-only equivariant policies such as Image-to-Sphere Policy (ISP); and equivariant tactile approaches such as EquiDiff+Tac and EquiTac. These either lack workspace-level spatial symmetry for tactile observations, treat tactile features as trivial group representations, or provide local fingertip-level equivariance rather than jointly transforming vision, touch, and 6-DoF actions in the workspace.
VISTA encodes egocentric RGB and tactile images using an equivariant ResNet, projects both into spherical tokens, and uses permutation-equivariant cross-attention so each visual spherical direction queries tactile tokens before harmonic projection. The fused tokens are projected onto spherical harmonics, rotated into the workspace frame using the end-effector orientation, and then condition a finite-group-equivariant diffusion policy that denoises 10-D action chunks containing translation, two rotation axes, and a gripper command. The system uses I60 for full 3D rotational tasks and C8 for yaw-dominant tasks.
In simulation across eight contact-rich tasks with 50 demonstrations per task and 40 rollouts each, VISTA achieves 80.9% mean success, compared with 48.8% for ManiFeel, 43.8% for VITAL, 21.3% for UniVTAC, and 65.0% for EquiDiff+Tac, corresponding to gains of 32.1, 37.1, 59.6, and 15.9 percentage points respectively. An image-only ISP ablation reaches 68.1%, so the tactile pathway adds 12.8 points. Alternative tactile fusion variants reach 69.4%, 72.8%, and 74.1%, making VISTA 6.8 points higher than the strongest alternative. With 100 demonstrations, VISTA reaches 84.7%, while EquiDiff+Tac with 100 demonstrations reaches 75.6%. On three real-robot tasks with 10 trials each, VISTA averages 7.33/10 successes versus 1.33/10 for EquiDiff+Tac; per-task results are 9/10 vs 1/10 for erase, 5/10 vs 3/10 for insert USB, and 8/10 vs 0/10 for insert pencil.
The equivariance guarantees are exact only for the selected finite group and do not cover arbitrary in-hand object rotations. The approach assumes rigidly mounted sensors and joint robot-object-sensor rotations with unchanged local observations; background or illumination changes, gravity-dependent contact, slip, calibration errors, external cameras, and changing extrinsics can violate this. The deployed policy replans at 10 Hz, so new tactile observations affect decisions only every 0.6-0.8 seconds, missing transient slip or high-rate contact changes. Real-robot evaluation is small, with 10 trials per task on three tasks. The paper does not report code or data release, and training compute cost is not reported.
Robot manipulator vendors, warehouse and logistics automation firms, and electronics assembly or kitting system builders could use this to reduce demonstration collection for tight-insertion and surface-contact tasks. In 1-3 years, pilot cells using existing wrist cameras and tactile sensors such as GelSight or XELA may benefit if the model is validated on more objects and real-robot data. Broader deployment in 3+ years would require faster tactile loops or residual feedback, robust extrinsic calibration under wear and lighting changes, and larger multi-task real datasets.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.