EmbPASS: Towards Cross-Embodiment Open Panoramic Segmentation
The authors introduce EmbPASS, a 1,000-image panoramic semantic segmentation benchmark spanning vehicle, drone, wearable and quadruped platforms, and EPONet, a CLIP-based open-vocabulary segmentation network for cross-embodiment panoramic perception. On EmbPASS, EPONet reaches 35.82% mIoU, 1.10 points above the strongest baseline, and also leads DensePASS. This is a preprint.
Can a single open-vocabulary panoramic segmentation model be trained only on pinhole source images and then segment 360° images from multiple unseen embodied platforms without target-platform annotations?
Panoramic cameras on vehicles, drones, wearables and quadrupeds observe scenes from different heights, orientations and latitude ranges, causing cross-embodiment observation shifts and latitude-dependent equirectangular projection distortion. Existing panoramic benchmarks use a single acquisition platform, and acquiring dense annotations for every new embodied platform is costly, so models trained on one platform or on pinhole images do not generalize well.
Open-vocabulary segmentation methods such as CAT-Seg, S2-Corr and SAN target perspective images, while panoramic open-vocabulary work such as OOOPS/OPS extends CLIP-based segmentation to 360° images but does not explicitly address heterogeneous embodied platforms. Deformable convolutions and MetricConv provide adaptive spatial sampling, but rely only on content features for local geometry prediction.
EmbPASS contains 250 annotated equirectangular images per platform, or 1,000 total, with 19 Cityscapes classes; vehicle, wearable and quadruped images are curated from existing datasets, and the drone subset is newly collected. EPONet freezes a CLIP ViT-B/16 encoder and trains a lightweight side adapter network on COCOStuff-164k pinhole images. RAMA predicts a Randers metric from content features plus directional neighbor similarities to constrain structured spatial sampling under varying distortion. CAST transfers intermediate CLIP features to the side branch using a content-dependent reassembly kernel and a spatial modulation gate.
On EmbPASS, EPONet achieves 35.82 mIoU and 42.30 open mIoU averaged over the four platforms, compared with 34.72/41.14 for OOOPS with RERP and 33.82/40.37 for SAN, giving +2.00/+1.93 over SAN and +1.10/+1.16 over the strongest baseline. Per-platform mIoU is 40.78 for vehicle, 25.90 for drone, 38.87 for wearable and 37.74 for quadruped. On Stanford2D3D, EPONet reaches 43.87 mIoU and 48.96 open mIoU; on DensePASS it reaches 49.15 mIoU and 55.22 open mIoU. Ablations show base SAN at 33.82 mIoU, +RAMA at 34.99, +CAST at 34.78, and both at 35.82. EPONet uses 8.80M learnable parameters versus 9.17M for OOOPS and 8.37M for SAN. Training uses 118k COCOStuff pinhole images; evaluation is on 1,000 EmbPASS images, 1,413 Stanford2D3D images and 100 DensePASS images, all real imagery without physical robot experiments.
The paper does not include a dedicated limitations section. Evident limitations include: only 1,000 annotated panoramic images across four platforms; the drone subset remains much harder at 25.90 mIoU; evaluation is limited to segmentation, with no downstream navigation or manipulation task; only a single frozen CLIP ViT-B/16 backbone is tested; no real-robot deployment is reported; and the code was not yet publicly released at the time of writing. The work is a preprint and has not been peer reviewed.
Autonomous vehicle suppliers, delivery and inspection drone makers, AR/wearable device companies, and quadruped robot manufacturers could use cross-embodiment panoramic segmentation to reduce per-platform annotation costs and improve 360° scene understanding. Near term, 1–3 years, it is suitable for research and perception prototypes; production deployment is 3+ years away and would require substantially higher mIoU, larger embodied datasets, real-time embedded inference, and validation on downstream navigation or planning tasks.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.