3DROID: A Renderable 3D Gaussian Dataset with Measured Per-Scene Reliability
This preprint introduces 3DROID, a DROID-derived dataset that turns 114 robot manipulation episodes into renderable 3D Gaussian scenes anchored to the robot's metric workspace, with per-scene calibration and reconstruction-reliability metadata. Combining refined camera extrinsics with pose conditioning improves held-out view PSNR from 12.57 to 15.81 dB for YoNoSplat and from 12.75 to 16.27 dB for ZipSplat. The work matters because view-consistent synthetic robot data are useful for manipulation only if they preserve real-world scale and robot-frame alignment.
How can renderable 3D Gaussian reconstruction for robot manipulation be produced and evaluated so that photometric fidelity is separated from metric workspace consistency, and how do camera extrinsic reliability and pose conditioning affect that consistency?
Robot manipulation models often reason from 2D observations while acting in 3D, and renderable 3D scenes could help, but standard 3DGS optimizes for photorealistic appearance and does not by itself preserve metric scale or robot-frame alignment. Real-world robot camera extrinsics are often noisy, so shipped calibrations cannot be treated as ground truth, and feed-forward 3DGS methods typically assume reliable poses or ignore pose information altogether.
Earlier robot 3D annotations added point clouds, depth, or trajectories to DROID-style data, including DP3, FP3, DROID-3D, 4D-VLA, Lift3D-VLA, PointWorld, and Robo3R. Renderable approaches such as RoboSnap build scenes from single images but do not provide measured per-scene reconstruction reliability. Feed-forward Gaussian splatting methods such as PixelSplat, MVSplat, DepthSplat, NoPoSplat, AnySplat, YoNoSplat, ZipSplat, and GenSplat either assume trustworthy camera poses or operate pose-free, but do not validate metric workspace consistency.
The authors first apply a calibration gate before reconstruction: they project kinematically known robot-surface points into each camera using candidate extrinsics, compare against classical stereo depth, and keep only scenes whose median stereo-to-projected-depth ratio lies in 0.85–1.15 with at least 30 valid projections. They then reconstruct scenes with pose-conditioned feed-forward 3DGS using PointWorld-refined extrinsics and release renderable Gaussians, rendered metric depth, coverage masks, camera parameters, calibration-gate verdicts, and per-scene reliability envelopes.
On 110 scenes with usable intrinsics, three-view reconstruction evaluated on the held-out ext2R view gives YoNoSplat PSNR/SSIM/LPIPS of 12.57/0.3582/0.5588 for the pose-free baseline versus 15.81/0.5105/0.2592 for pose-conditioned refined extrinsics; ZipSplat improves from 12.75/0.4233/0.5467 to 16.27/0.5242/0.2979. Ablations on up to 110 scenes show pose conditioning at fixed extrinsics gives median per-scene PSNR gains of +0.73 to +2.73 dB, while switching to refined extrinsics at fixed pose setting gives +0.29 to +2.13 dB. AnySplat on 107 scenes gains +0.39 dB from refined extrinsics. Geometric checks with refined extrinsics show YoNoSplat robot error stays nearly flat at 7.52 vs 7.54 cm, while ZipSplat robot error improves from 8.99 to 8.04 cm; stereo disagreement with refined extrinsics is 7.08 vs 6.92 cm for YoNoSplat and 8.85 vs 8.07 cm for ZipSplat. With shipped extrinsics, pose conditioning has mixed effects: ZipSplat aggregate robot error rises from 19.71 to 30.58 cm, but the paired median change is not significant. Under shipped extrinsics without pose conditioning, robot-depth ratios are 1.34, 1.31, and 1.26, placing the arm too far. The gate selected 114 scenes from 225 eligible scenes using refined extrinsics, versus 80 with shipped extrinsics.
The authors state that scenes are static snapshots reconstructed from four exterior views, geometry is measured only where robot-surface or stereo support exists, additional well-calibrated views are not isolated as a factor, the wrist stream is excluded, and no full-scene geometric ground truth or downstream manipulation improvement is claimed. The dataset is small at 114 gated scenes, with 110 usable for photometric evaluation and 109 for shipped-extrinsic comparisons, all from DROID. The paper is a preprint and only the dataset is described as released; no code or training-pipeline release is stated, and compute cost is not reported.
Robotics data-platform teams, manipulation foundation-model developers, and warehouse or robot-arm vendors that collect demonstration data could use 3DROID as a benchmark for metric 3D scene reconstruction and as a source of calibrated synthetic views for training visuomotor policies. The calibration-gate idea could be adopted now by teams curating real-robot data. Production use in policy training likely needs 1–3 years: downstream policy gains would need to be demonstrated, coverage expanded, dynamic 4D scenes added, and integration with existing robot-learning stacks standardized.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.