SARI: Phase-Split Sim-Real Co-Training for Contact-Rich Manipulation

Can separating a contact-rich manipulation task into simulated approach and real interaction phases let a single policy generalize to unseen object placements with less real-world data collection?

Adapting VLA models to contact-rich tasks usually requires many full-task demonstrations because policies overfit to object placements; full-task simulation is unreliable for contact physics, and mixing full-task simulated and real data lets the narrow real distribution dilute the spatial coverage available from simulation.

Earlier synthetic-data pipelines such as MimicGen, DemoGen, and Real2Render2Real, as well as full-task sim-real co-training, synthesize or mix complete demonstrations, assuming simulation is reliable throughout contact or requiring full-task real trajectories. They fail to transfer spatial generalization or waste real-world effort on free-space approaches.

SARI builds a Gaussian-splatting digital twin and renders only free-space approaches from home to a pre-contact pose across all placements; real contact segments are teleoperated from a few transferred pre-contact poses. It aligns simulated and real images with degree-3 spherical harmonics and per-camera color calibration, re-expresses all actions in a wrist-camera-relative frame, and post-trains one π0.5 policy by behavior cloning with no phase labels or switching.

On the lamp task, SARI achieved 100% approach success at seen and unseen placements, 50% completion at seen placements, and 27.5% completion at unseen placements; Full Sim, Full Real, and Full-task Co-training baselines all had 0% unseen completion, with Full-task Co-training at 55% seen completion. Real contact data collection fell from 28.03 to 18.42 minutes (a 34.3% reduction) while using 60 contact demos instead of 60 full-task demos. Across five tasks, unseen-placement success was 81.25% for book, 75.0% for button, 43.75% for tissue, 27.5% for lamp, and 25.0% for drawer; seen-placement success ranged from 50% to 87.5%. Appearance ablations showed best lamp success of 35.0% with high-fidelity rendering, color correction, and correct GS scene, versus 8.3% for low-fidelity without color correction. Camera-relative actions achieved 50% seen and 27.5% unseen lamp success, compared with 35% seen and 20% unseen for EEF-relative GR00T N1.7 and 0% for joint-space π0.5 variants.

Stated limitations include fixed tabletop scenes, one Franka arm, two-stage approach–interaction tasks, manual scene reconstruction, camera calibration, pre-contact-pose selection, and photometric reference selection; dynamic scenes, moving objects, multi-stage interactions, more embodiments, and automated preparation remain future work. Additional evident limits are a small seen-placement count (two per task), modest unseen success rates (e.g., 25–27.5% on lamp and drawer), single-robot evaluation, and reliance on a manual pipeline.