OmniAct3D, 단일 ERP 파노라마 3D 검출을 위한 VFM 적응 프레임워크

Abstract

Accurate 3D detection is essential for mobile embodied agents, while Vision Foundation Models (VFMs) offer transferable visual and geometric priors. Yet existing VFM-based 3D detectors rely on narrow-view monocular images or discrete perspective views, limiting coherent surround perception; equirectangular projection (ERP) instead encodes a continuous 360^{\circ} scene in a single image. Direct transfer remains difficult because ERP organizes geometry and visual information differently, making object-relevant cues hard to model, localize, and preserve. We propose OmniAct3D, a framework that adapts perspective-trained VFM detectors to ERP while preserving transferable VFM priors. To resolve geometric mismatch, the ERP-Ray Geometry Adapter (ERGA-Ray) models spherical viewing rays and periodic spatial structure. To localize evidence in scene-wide context, the Visual-Action Reasoning Chain (VARC) grounds each hypothesis in relevant panoramic evidence and converts it into a structured geometric action. To recover local cues lost under fixed token budgets, the Appearance-Guided Heading Expert (AGHE) re-encodes object regions at higher resolution for heading estimation. Experiments show that OmniAct3D improves over the previous best 3D detector by 2.96 NDS points on Spheriverse and over the unadapted VFM baseline by 24.87 mAP points on PanoMMOcc. With target-specific geometry adaptation, VARC retains 95–98\% of the same-configuration mAP, indicating reusable object-level 3D reasoning across sensing configurations. The source code will be made publicly available at https://github.com/FeiT-FeiTeng/OmniAct3D.

I Introduction

Accurate and robust 3D detection is essential for mobile embodied agents to understand and interact with the physical world. From autonomous vehicles to mobile robots, agents must estimate the 3D locations, dimensions, and orientations of surrounding objects to support navigation, motion planning, and embodied decision-making [1, 2].

Recent VFM-based 3D detectors [5, 6] leverage transferable visual and geometric priors for object-level 3D perception. Despite this progress, existing methods primarily follow two perspective-image paradigms. As shown in Fig. 1, monocular methods [3, 7] infer 3D boxes from a single perspective image but observe only a limited field of view; Multi-view methods [8, 9] extend the surrounding coverage using discrete perspective observations. In contrast, equirectangular projection (ERP) encodes the surrounding spherical field of view within a single image, providing a unified and spatially continuous representation for coherent 360^{\circ} perception [10, 11].

However, directly transferring perspective-trained VFM-based 3D detectors to ERP observations remains challenging because ERP organizes geometry and visual information differently from perspective imagery. This representation mismatch makes object-relevant cues difficult to model, localize, and preserve, giving rise to three challenges. First, the geometric priors of VFMs are learned predominantly from locally planar and non-periodic perspective imagery, whereas ERP exhibits spherical viewing directions, latitude-dependent distortion, and horizontal periodicity. Reusing these priors without adaptation therefore produces geometric representations misaligned with ERP structure, particularly at high latitudes and across the horizontal boundary. Second, although a single ERP image captures scene-wide context, the evidence relevant to each object occupies only a small portion of the panorama. Directly decoding 3D hypotheses from global features makes it difficult to localize and associate the supporting evidence for each object, especially when objects are small or occluded. Third, encoding an ultra-wide panorama under a fixed token budget compresses individual objects and discards fine-grained local appearance cues. The loss of features such as front–rear asymmetry impairs directed heading estimation.

To address these challenges, we propose OmniAct3D, a framework that adapts perspective-trained VFM detectors to ERP observations while preserving transferable VFM priors. To resolve the geometric mismatch, we introduce the ERP-Ray Geometry Adapter (ERGA-Ray), which models spherical viewing rays and ERP-aware spatial interactions to align pretrained foundation geometry with ERP-specific spherical and periodic structure. To localize object-relevant evidence in scene-wide context, we introduce the Visual-Action Reasoning Chain (VARC), which grounds each 3D hypothesis in relevant panoramic evidence and converts that evidence into a structured geometric action for object-level 3D reasoning. To recover local cues lost under a fixed token budget, we introduce the AGHE, which re-encodes object regions at higher resolution for directed heading estimation.

We evaluate OmniAct3D on Spheriverse [12] and PanoMMOcc [13]. On Spheriverse, OmniAct3D achieves 0.1887/0.1825 mAP/NDS, improving the previous best NDS by 2.96 points. On PanoMMOcc, it achieves 0.5897/0.4032 mAP/NDS and improves the unadapted VFM baseline by 24.87 mAP points. Moreover, with target-specific geometry adaptation via ERGA-Ray, VARC retains 95\% and 98\% of same-configuration mAP on Spheriverse and PanoMMOcc, respectively, indicating reusable object-level 3D reasoning across sensing configurations.

In summary, our contributions are as follows:

• We propose OmniAct3D, a framework that adapts perspective-trained VFM detectors to ERP observations while preserving transferable VFM priors.
• We introduce ERGA-Ray to align foundation geometry with ERP spherical and periodic structure.
• We introduce VARC to ground 3D hypotheses in relevant panoramic evidence and convert that evidence into a structured geometric action, and AGHE to recover fine-grained local cues for directed heading estimation.
• Experiments show that OmniAct3D improves NDS by 2.96 points over the previous best detector on Spheriverse and mAP by 24.87 points over the unadapted VFM baseline on PanoMMOcc; with target-specific geometry adaptation, VARC retains 95–98\% of same-configuration mAP across sensing configurations.

II Related Work

II-A Vision-Based 3D Object Detection

Vision-based 3D object detection estimates object locations, dimensions, and orientations from images. Monocular methods infer 3D states from a single perspective view using geometric constraints, depth estimation, or object-centric representations [14, 15, 16]. Multi-view methods aggregate complementary observations through geometric projection, query-based aggregation, or BEV representations [17, 18, 19], while temporal approaches further exploit cross-frame information [20]. These methods are developed primarily for perspective imagery. Monocular systems have limited fields of view, whereas surround-view systems rely on multiple discrete camera views and cross-view integration. In contrast, our work studies object-level 3D detection directly from a panoramic representation for continuous 360^{\circ} perception.

II-B Omnidirectional Vision and ERP Perception

Equirectangular projection (ERP) represents a spherical field of view within a single image and naturally provides spatially continuous horizontal 360^{\circ} coverage. ERP imagery has been studied for panoramic depth estimation, scene understanding, 3D detection, and omnidirectional occupancy prediction [21, 22, 23, 24]. However, ERP geometry differs fundamentally from perspective imaging due to latitude-dependent distortion, horizontal periodicity, and spherical viewing rays. Prior work addresses these properties through spherical convolutions and distortion sampling [25, 26], projection fusion [27], and panoramic attention [28]. These works highlight the need for ERP-specific geometric modeling for coherent panoramic perception.

II-C Foundation Models for Panoramic 3D Perception

Large-scale VFMs provide transferable priors for semantic and geometric perception. Visual models such as DINO [29, 30] and SAM [31, 32] learn reusable visual representations, while geometry foundation models such as UniDepth [33], MoGe [34], and VGGT [4] provide transferable priors for depth and 3D structure. We use the term VFM-based 3D detector for a 3D detector built on such pretrained models [6, 35, 36, 37]. Related work also develops camera-general geometry [38, 39, 40, 41, 42, 43, 44, 45]. Despite their transferability, adapting VFM-based 3D detection from perspective imagery to ERP remains challenging. We address this perspective-to-ERP adaptation through ERP geometry alignment and evidence-grounded object-level 3D reasoning.

III Methodology

Given a full equirectangular projection (ERP) image I\in\mathbb{R}^{H_{I}\times W_{I}\times 3}, our goal is to directly infer object-level 3D states from a panoramic observation. For each object, the state comprises its 3D center (X\mathord{\mathchar 59\relax}Y\mathord{\mathchar 59\relax}Z), dimensions (W\mathord{\mathchar 59\relax}L\mathord{\mathchar 59\relax}H), orientation \theta, category, and detection confidence.

III-A Overview of OmniAct3D

Given an ERP image I, the frozen VFM-based detector extracts two complementary feature representations:

\left(F_{\mathrm{vis}}\mathord{\mathchar 59\relax}M_{0}\right)=\mathcal{E}_{\mathrm{VFM}}(I)\mathchar 59\relax\qquad M_{0}\in\mathbb{R}^{N\times d}\mathchar 59\relax

where F_{\mathrm{vis}} denotes the visual features extracted by the SAM3 branch [32], and M_{0} denotes the pretrained geometry memory produced by the DINOv2-based LingBot branch [46].

As illustrated in Fig. 2, the overall pipeline consists of four successive transformations:

\begin{array}[]{@{}l@{}}M=\mathcal{G}_{\mathrm{ERGA}}\left(M_{0}\mathord{\mathchar 59\relax}G_{\mathrm{ERP}}\right)\mathchar 59\relax\quad(B\mathord{\mathchar 59\relax}Q\mathord{\mathchar 59\relax}C)=\mathcal{D}\left(F_{\mathrm{vis}}\mathord{\mathchar 59\relax}M\right)\mathchar 59\relax\\[2.0pt]
(V\mathord{\mathchar 59\relax}A\mathord{\mathchar 59\relax}B^{\prime})=\mathcal{R}_{\mathrm{VARC}}\left(B\mathord{\mathchar 59\relax}Q\mathord{\mathchar 59\relax}C\mathord{\mathchar 59\relax}M\right)\mathchar 59\relax\quad\hat{B}=\mathcal{H}_{\mathrm{AGHE}}\left(I\mathord{\mathchar 59\relax}B^{\prime}\right).\end{array}

ERP geometry adaptation. G_{\mathrm{ERP}} encodes the spherical viewing rays and periodic positions of the ERP token grid. The ERGA-Ray operator \mathcal{G}_{\mathrm{ERGA}} uses this encoding to transform the pretrained geometry memory M_{0} into the detection-oriented ERP geometry memory M.

Initial hypothesis generation. Conditioned on the visual features F_{\mathrm{vis}} and adapted geometry memory M, the detector \mathcal{D} produces an initial 3D box hypothesis B, its object query Q, and candidate-quality cues C=[s_{\mathrm{2D}}\mathord{\mathchar 59\relax}s_{\mathrm{3D}}\mathord{\mathchar 59\relax}s_{\mathrm{comb}}].

Evidence-grounded reasoning. Given (B\mathord{\mathchar 59\relax}Q\mathord{\mathchar 59\relax}C\mathord{\mathchar 59\relax}M), VARC retrieves hypothesis-relevant panoramic evidence V, reasons jointly over B and V to infer a structured geometric action A, and applies A to obtain the resulting box B^{\prime}. This explicit hypothesis–evidence–action chain grounds each geometric action in relevant panoramic evidence.

Appearance-guided heading estimation. The AGHE operator \mathcal{H}_{\mathrm{AGHE}} re-encodes the object region projected onto I at higher resolution and predicts its directed heading from fine-grained local appearance cues, producing the output \hat{B}.

The VFM-based detector remains frozen throughout training, whereas ERGA-Ray, VARC, and AGHE constitute the trainable components for ERP geometry adaptation, evidence-grounded object-level reasoning, and appearance-guided heading estimation, respectively.

III-B ERP-Ray Geometry Adapter

To align pretrained foundation geometry with ERP-specific spherical and periodic structure, we introduce ERGA-Ray, achieving detection-oriented ERP geometry while preserving transferable priors.

ERP ray geometry. Let (u\mathord{\mathchar 59\relax}v) denote normalized ERP coordinates. Longitude and latitude are obtained as \lambda=\lambda(u) and \phi=\phi(v), where the exact vertical mapping depends on the ERP field of view. In the camera coordinate system, spherical ray and lifted 3D points are defined as follows:

\displaystyle\mathbf{d}(\lambda\mathord{\mathchar 59\relax}\phi)=\begin{bmatrix}-\cos\phi\sin\lambda\\
-\sin\phi\\
\cos\phi\cos\lambda\end{bmatrix}\mathord{\mathchar 59\relax}\;\mathbf{P}=R\mathbf{d}\mathord{\mathchar 59\relax}\;R>0.

Here, R denotes the Euclidean viewing-ray range. Conversely, for a known 3D point \mathbf{P}, we obtain R=\|\mathbf{P}\|_{2} and \mathbf{d}=\mathbf{P}/R in turn. Projection and back-projection wrap across the ERP seam.

Ray-conditioned panoramic adaptation. Foundation geometry does not explicitly encode ERP-specific viewing directions, latitude-dependent distortion, or horizontal periodicity. ERGA-Ray therefore associates each token M_{0\mathord{\mathchar 59\relax}i} with:

\mathbf{g}_{i}=\big[d_{x}\mathord{\mathchar 59\relax}d_{y}\mathord{\mathchar 59\relax}d_{z}\mathord{\mathchar 59\relax}\,\sin\lambda\mathord{\mathchar 59\relax}\cos\lambda\mathord{\mathchar 59\relax}\,\sin\phi\mathord{\mathchar 59\relax}\cos\phi\big].

The ray components encode the physical viewing direction, while the trigonometric angular terms provide a smooth spherical positional encoding. In particular, \cos\phi also reflects the latitude-dependent area distortion of the ERP parameterization and therefore serves as a distortion cue.

A lightweight geometry encoder maps \mathbf{g}_{i} to \mathbf{e}_{i}, which modulates a low-dimensional projection of the foundation token through FiLM:

\displaystyle\begin{gathered}\tilde{\mathbf{c}}_{i}=\left(1+\eta\tanh\bm{\gamma}_{i}\right)\odot W_{d}\operatorname{LN}(M_{0\mathord{\mathchar 59\relax}i})+\bm{\beta}_{i}+\mathbf{e}_{i}\mathord{\mathchar 59\relax}\\
(\bm{\gamma}_{i}\mathord{\mathchar 59\relax}\bm{\beta}_{i})=f_{\mathrm{film}}(\mathbf{e}_{i}).\end{gathered}

The ray-conditioned tokens are rearranged into their ERP feature layout and processed by an asymmetric spatial mixer. Horizontal interactions use circular padding to preserve left–right continuity, whereas vertical interactions remain non-periodic. ERGA-Ray predicts a gated residual over the pretrained geometry memory:

\displaystyle M={} \\ \displaystyle M_{0}+\mathbf{g}_{m}\odot W_{u}\operatorname{Mix}_{\mathrm{ERP}}(\tilde{\mathbf{C}})\mathord{\mathchar 59\relax} \\ \displaystyle\mathbf{g}_{m}={} \\ \displaystyle\sigma\!\left(f_{\mathrm{gate}}(\tilde{\mathbf{C}})\right).

Here, W_{d} and W_{u} are learned down- and up-projection matrices,\eta bounds the multiplicative FiLM modulation, and \tilde{\mathbf{C}}=[\tilde{\mathbf{c}}_{1}\mathord{\mathchar 59\relax}\ldots\mathord{\mathchar 59\relax}\tilde{\mathbf{c}}_{N}]denotes the stacked ray-conditioned tokens. This residual formulation preserves transferable foundation priors while introducing the spherical and periodic geometry required by panoramic 3D object detection.

III-C Visual-Action Reasoning Chain

To ground 3D hypotheses in panoramic evidence, we introduce VARC, enabling evidence-grounded object-level 3D reasoning through structured geometric actions.

Hypothesis representation. VARC takes an explicit box hypothesis B and its object query Q. We parameterize B in panoramic coordinates:

\displaystyle s={} \\ \displaystyle[\log\rho\mathord{\mathchar 59\relax}\lambda\mathord{\mathchar 59\relax}\phi\mathord{\mathchar 59\relax}\log W\mathchar 59\relax\log L\mathord{\mathchar 59\relax}\log H\mathord{\mathchar 59\relax}\theta_{\mathrm{obs}}]\mathord{\mathchar 59\relax} \\ \displaystyle\rho={} \\ \displaystyle\sqrt{X^{2}+Z^{2}}\mathord{\mathchar 59\relax}\qquad\theta_{\mathrm{obs}}=\operatorname{wrap}(\theta-\lambda).

with \lambda=\operatorname{atan2}(-X\mathord{\mathchar 59\relax}Z) and \phi=\operatorname{atan2}(-Y\mathord{\mathchar 59\relax}\rho). Here, \rho is the horizontal range used by VARC, while R is the Euclidean viewing-ray range used by ERP lifting and ERGA-Ray supervision; they satisfy \rho=R\cos\phi. We use trigonometric embeddings \psi(s) for angular variables before feeding the state to the Reasoner. The box represents the geometric hypothesis, while Q supplies object-level context.

Hypothesis-grounded panoramic evidence. The evidence reader grounds each 3D hypothesis in target-relevant panoramic observations. For each B, we construct K=13 object-centric support points comprising the box center, eight corners, and four side-face centers. Query-conditioned offsets adapt this geometric template while remaining bounded, i.e., \|\Delta P\|_{\infty}\leq\delta_{\max}. The resulting points are projected to ERP coordinates and sampled from the fixed geometry memory using circular interpolation. For the k-th support point:

\displaystyle m^{k}={} \\ \displaystyle\mathcal{S}_{\mathrm{circ}}\!\left(M\mathord{\mathchar 59\relax}\Pi_{\mathrm{ERP}}(P^{k};B)\right)\mathord{\mathchar 59\relax} \\ \displaystyle a^{k}={} \\ \displaystyle\operatorname{softmax}_{k}\!\left(\frac{f_{k}(m^{k})^{\top}f_{q}(Q)}{\sqrt{d}}\right).

We further define the projection descriptor p=[w^{\mathrm{ang}}\mathord{\mathchar 59\relax}h^{\mathrm{ang}}\mathord{\mathchar 59\relax}s^{\mathrm{seam}}\mathord{\mathchar 59\relax}r^{\mathrm{vis}}], which encodes angular width, angular height, seam crossing, and the fraction of visible support points. The target-relevant panoramic evidence is then represented as:

V=f_{\mathrm{read}}\!\left(\left[\begin{aligned} &\sum_{k=1}^{K}a^{k}f_{v}(m^{k})\mathord{\mathchar 59\relax}\,f_{q}(Q)\mathord{\mathchar 59\relax}\\
&f_{s}(\psi(s))\mathord{\mathchar 59\relax}\,f_{p}(p)\end{aligned}\right]\right).

Thus, V provides hypothesis-grounded object-level evidence from the panoramic representation.

Evidence-driven geometric action. Given V, the Visual-Action Reasoner combines the evidence with object query, spherical state, quality cues, and projection descriptor:

h=f_{\mathrm{reason}}\!\left(\left[\begin{aligned} &V\mathord{\mathchar 59\relax}\,Q\mathord{\mathchar 59\relax}\,f_{s}(\psi(s))\mathord{\mathchar 59\relax}\\
&f_{c}(C)\mathord{\mathchar 59\relax}\,f_{p}(p)\end{aligned}\right]\right).

From h, the model predicts a structured action A and gate g=\sigma(f_{\mathrm{gate}}(h)), which controls the action strength. We factorize the horizontal-range action into a discrete direction d\in\{-1\mathord{\mathchar 59\relax}0\mathord{\mathchar 59\relax}+1\} and continuous magnitude \mu\in[0\mathord{\mathchar 59\relax}1]:

\Delta\log\rho=g\,d\,\mu\,\log(1.15).

Here, “radial” refers to the horizontal range \rho, rather than the Euclidean viewing-ray range R used for ERP lifting. During training, d is represented by the expectation under the three-class distribution, whereas inference uses argmax.

Longitude, latitude, dimensions, and observation-relative yaw use bounded actions:

A^{\mathrm{cont}}=g\,\tanh\left(f_{\mathrm{act}}(h)\right)\odot\bm{\kappa}\mathchar 59\relax

where \bm{\kappa} provides variable-dependent geometric bounds. These terms form A, yielding:

s^{\prime}=\operatorname{WrapClamp}(s+A).

Here, \operatorname{WrapClamp} wraps longitude and observation-relative yaw to their periodic intervals, clamps latitude to the calibrated vertical field of view, and preserves the log-parameterization of range and dimensions. Inverse parameterization yields the resulting 3D box B^{\prime}.

III-D Appearance-Guided Heading Expert

To recover fine-grained heading cues weakened by global ERP encoding, we introduce the AGHE for directed heading estimation via high-resolution object-centric re-encoding.

The AGHE operates on the resulting 3D boxes. Rather than rerunning the complete VFM-based 3D detector, it re-encodes only the projected object regions at a higher object-centric resolution using the pretrained DINOv2 encoder [30] from the LingBot branch. For each box B_{i}^{\prime}, we project it back to the original ERP image, crop the corresponding object region with contextual margins, and reuse the pretrained DINOv2 encoder [30]:

\begin{gathered}I_{i}^{\mathrm{crop}}=\operatorname{Crop}\!\left(I\mathord{\mathchar 59\relax}\Pi_{\mathrm{ERP}}(B_{i}^{\prime})\right)\mathord{\mathchar 59\relax}\\
F_{i}^{\mathrm{local}}=\mathcal{F}_{\mathrm{DINO}}(I_{i}^{\mathrm{crop}}).\end{gathered}

All crops from each ERP image are batched; SAM3 [32], the depth decoder, and the detection head are not re-executed.

The local appearance is combined with a category embedding and an orientation-free geometric encoding:

\mathbf{g}_{i}^{\mathrm{head}}=\left[\begin{aligned} &\log\rho_{i}\mathord{\mathchar 59\relax}\,\sin\lambda_{i}\mathord{\mathchar 59\relax}\cos\lambda_{i}\mathord{\mathchar 59\relax}\\
&\log W_{i}\mathord{\mathchar 59\relax}\log L_{i}\mathord{\mathchar 59\relax}\log H_{i}\end{aligned}\right].

The AGHE predicts periodic observation-relative heading and converts it to the camera-frame direction:

\displaystyle(\hat{s}_{i}\mathord{\mathchar 59\relax}\hat{c}_{i}) \\ \displaystyle=f_{\mathrm{head}}\!\left(F_{i}^{\mathrm{local}}\mathord{\mathchar 59\relax}\mathbf{e}_{i}^{\mathrm{cls}}\mathchar 59\relax\mathbf{g}_{i}^{\mathrm{head}}\right)\mathord{\mathchar 59\relax} \\ \displaystyle\hat{\theta}_{i} \\ \displaystyle=\operatorname{wrap}\!\left(\operatorname{atan2}(\hat{s}_{i}\mathord{\mathchar 59\relax}\hat{c}_{i})+\lambda_{i}\right).

The expert replaces the orientation while preserving the center, dimensions, category, and detection score.

III-E Training Objectives

ERGA-Ray is supervised by a multi-target loss:

\displaystyle\mathcal{L}_{\mathrm{ERGA}}={} \\ \displaystyle\mathcal{L}_{\mathrm{target}}+\lambda_{\mathrm{ori}}\mathcal{L}_{\mathrm{ori}}+\lambda_{\mathrm{pre}}\mathcal{L}_{\mathrm{pre}} \\ \displaystyle+\lambda_{\mathrm{conf}}\mathcal{L}_{\mathrm{conf}}+\lambda_{\mathrm{res}}\mathcal{L}_{\mathrm{res}}.

Here, \mathcal{L}_{\mathrm{target}} jointly regresses the seam-aware image-center residual, log-range \log R, and log-dimensions. The remaining terms preserve orientation, pretrained regression, and confidence predictions, while constraining the magnitude of the learned adaptation residual. VARC is supervised by the following objective:

\displaystyle\mathcal{L}_{\mathrm{VARC}}={} \\ \displaystyle\mathcal{L}_{b}+\lambda_{\mathrm{id}}\mathcal{L}_{\mathrm{id}}+\lambda_{g}\mathcal{L}_{g}+\lambda_{r}\mathcal{L}_{r}.

Here, \mathcal{L}_{b} supervises the resulting spherical box state, \mathcal{L}_{\mathrm{id}} encourages identity actions for already accurate hypotheses, and \mathcal{L}_{g} supervises action-gate activation. \mathcal{L}_{r} denotes the radial loss, combining direction classification, action ranking, and action-magnitude regression. The AGHE is supervised on the sine-cosine encoding of the periodic observation-relative heading in Eq. (16) using a cosine loss.

IV Experiments

We evaluate OmniAct3D on two panoramic 3D detection benchmarks. We then analyze the effects of ERGA-Ray, VARC, and AGHE, and test whether VARC transfers between different sensing configurations.

IV-A Experimental Setup

Datasets and metrics. We use Spheriverse [12] and PanoMMOcc [13], which were originally developed for 3D occupancy prediction. Both datasets provide LiDAR observations, 3D bounding boxes, and camera calibration. We transform the boxes from the LiDAR coordinate system to the panoramic camera coordinate system to construct the two ERP 3D detection benchmarks. The datasets differ in camera height, vertical field of view, and object-distance distribution. Following the nuScenes-style protocol [54], we report mAP, NDS, mATE, mASE, and mAOE as evaluation metrics. The two benchmarks do not provide the attribute and velocity annotations required for mAAE and mAVE. We therefore report a three-TP variant of NDS.

Baselines and evaluation protocol. We compare against representative 3D detectors spanning Cartesian-grid BEV architectures [18, 20, 47], geometry-oriented BEV architectures [50, 51], polar-coordinate BEV architectures [48], sparse query-based architectures [17, 19], configuration-aware architectures [49], monocular detectors [52], and VFM-based detectors [36, 37, 53]. In Table I, WildDet3D uses its original square preprocessing scheme, while 3D-MOOD and DetAny3D decompose each panorama into perspective views. The remaining methods operate directly on the full rectangular equirectangular projection (ERP) panorama. All ablations use the same ERP protocol. Unless otherwise noted, all methods are evaluated on both benchmarks with only minimal dataset-specific adaptations.

Implementation details. All experiments are conducted on four NVIDIA GeForce RTX 3090 GPUs. The VFM-based 3D detector remains frozen throughout training, and all input images are resized to 2480\times 512 pixels. We optimize ERGA-Ray, VARC, and AGHE separately, updating one component at a time while keeping the others fixed. ERGA-Ray and VARC together introduce approximately 17.07M trainable parameters, corresponding to only a 0.15\% increase in the total parameter count, while incurring a measured inference-latency overhead of 3.3\%–5.0\%. For VARC transfer, we use a target-specific ERGA-Ray module.

IV-B Main Results

To evaluate the overall detection performance of OmniAct3D, we compare it with existing 3D detectors on both benchmarks. Table I reports the results. On Spheriverse, OmniAct3D improves the previous best mAP and NDS by 1.98 and 2.96 points, respectively (0.1887 vs. 0.1689 and 0.1825 vs. 0.1529). On PanoMMOcc, it improves the previous best mAP and NDS by 1.39 and 2.19 points, respectively (0.5897 vs. 0.5758 and 0.4032 vs. 0.3813). It also gives the lowest mASE and mAOE on Spheriverse. The consistent gains across both benchmarks show the effectiveness of OmniAct3D under different panoramic sensing configurations. Fig. 3 qualitatively compares OmniAct3D without VARC (W.V), full OmniAct3D (Ours), and representative baselines across diverse Spheriverse and PanoMMOcc scenes.

V Analysis and Ablation Studies

We analyze equirectangular projection (ERP) geometry adaptation, hypothesis-evidence-action reasoning, fine-grained heading recovery, and reasoning reuse.

V-A Component Ablation Analysis

Table II reports the component ablation. Adding VARC to ERGA-Ray improves mAP by 6.96 points on Spheriverse (0.1887 vs. 0.1191) and by 15.44 points on PanoMMOcc (0.5897 vs. 0.4353). Adding AGHE further reduces mAOE while leaving mAP unchanged. Thus, VARC mainly improves object localization, whereas AGHE improves heading estimation in full OmniAct3D.

V-B ERP Geometry Adaptation

ERP-specific geometry modeling. To investigate whether existing adaptation methods are sufficient for ERP 3D detection, Table III compares several methods using the same LingBot backbone [46]. No Adapt. changes only the projection to ERP and directly uses the original LingBot features. MLP and AdaptFormer [55] apply feature adaptation, while Circular Conv [56], Ray-FiLM [57], and the PanoSwin-based [28] model horizontal periodicity, spherical viewing rays, and panoramic spatial interactions, respectively. ERGA-Ray combines spherical ray conditioning with periodic spatial modeling. Overall, ERGA-Ray improves mAP/NDS over the strongest alternatives by 21.5\%/9.1\% on Spheriverse and by 16.7\%/11.1\% on PanoMMOcc.

Geometry backbone comparison. To test whether ERGA-Ray is effective with different geometry backbones, Table III also evaluates UniK3D [39], DA2 [40], and DAP [41]. UniK3D is designed for general camera models, while DA2 and DAP are designed for panoramic images. Although DAP with ERGA-Ray has lower mAP than LingBot with ERGA-Ray, adding ERGA-Ray still improves DAP by 44.4\% on Spheriverse (0.0986 vs. 0.0683) and by 13.1\% on PanoMMOcc (0.2097 vs. 0.1854). This shows that ERGA-Ray is also effective with a different backbone.

V-C Hypothesis-Evidence-Action Reasoning

To examine how VARC improves object-level predictions, we analyze difficult targets, its three main components, and the resulting localization changes.

Evidence-challenging targets. To test VARC on evidence-challenging targets, we use a subset containing objects in the lowest area quartile that are at least 75\% occluded by a closer object. The complement contains all remaining objects, and the main results use the full validation sets. Center@8 is the one-to-one center error capped at 8\,\mathrm{m}, while S@2m and S@4m measure localization success at the corresponding thresholds. Diagnostic mAP is computed on this subset. Table IV compares the frozen detector with ERGA-Ray only on the subset and its complement. The subset has higher center error and lower S@2m on both datasets. The reported differences are the subset values minus the complement values, with all 95\% frame-cluster bootstrap confidence intervals excluding zero.

Reasoning-chain components. Table V ablates the key VARC designs by replacing hypothesis-conditioned evidence with fixed evidence, structured geometric actions with unstructured regression, and object-query context with box-state-only conditioning. Full VARC achieves 0.1768 diagnostic mAP on PanoMMOcc. Removing evidence localization, structured action, or object-query context reduces it to 0.1257, 0.1400, and 0.1311, respectively. All three components therefore contribute to the final result.

Geometric gains on evidence-challenging targets. To measure the localization changes produced by VARC, we compare the same targets with and without VARC. Center Gain is the reduction in center error, \DeltaS is the change in localization success, and positive-change precision is the percentage of changed predictions that improve. As shown in Table VI, VARC reduces center error by 0.80\,\mathrm{m} on Spheriverse and 0.54\,\mathrm{m} on PanoMMOcc, with positive-change precision above 86\%.

We further examine its effectiveness across different object ranges and scene-overlap levels. Figure 4 shows that adding VARC improves mAP and NDS across different object ranges and overlap levels. The gains are larger for nearby objects and scenes with more overlap.

V-D Fine-Grained Heading Recovery

We evaluate whether object-centric re-encoding recovers local appearance cues weakened by global ERP encoding. Table II shows that the AGHE leaves mAP unchanged while reducing mAOE from 1.5585 to 0.8455 on Spheriverse and from 2.0781 to 0.7902 on PanoMMOcc, increasing NDS to 0.1825 and 0.4032. Thus, AGHE improves heading estimation without changing object localization.

V-E Cross-Configuration Reasoning Transfer

To investigate whether VARC remains reusable across sensing configurations, we transfer it between the two datasets without updating its parameters, using the target-specific ERGA-Ray. As shown in Table VII, the transferred VARC retains 95\% and 98\% of same-configuration mAP on Spheriverse and PanoMMOcc, respectively (0.1800 vs. 0.1887 and 0.5803 vs. 0.5897). This shows that object-level reasoning is preserved after transfer.

VI Conclusion

We present OmniAct3D for perspective-to-ERP adaptation of VFM-based 3D detection. The framework addresses the three requirements of this central challenge: ERGA-Ray aligns pretrained foundation geometry with ERP-specific structure, VARC establishes an explicit hypothesis-evidence-action reasoning chain for evidence-grounded object-level 3D reasoning, and an AGHE preserves local cues for directed heading estimation. OmniAct3D achieves the best mAP and NDS on two ERP benchmarks with heterogeneous sensing configurations. VARC also retains strong performance across configurations when paired with target-specific ERGA-Ray. These results highlight the effectiveness of OmniAct3D for mobile embodied agents. Future work will explore extending evidence-grounded object-level 3D reasoning to temporal panoramic observations and more complex embodied perception tasks.

References

1. [1] Y. Hu et al. (2023) Planning-oriented autonomous driving. In CVPR,
2. [2] W. Sun et al. (2025) SparseDrive: End-to-end autonomous driving via sparse scene representation. In ICRA,
3. [3] Z. Zhang et al. (2026) Towards intrinsic-aware monocular 3D object detection. In CVPR,
4. [4] J. Wang et al. (2025) VGGT: Visual geometry grounded transformer. In CVPR,
5. [5] A. J. Yang et al. (2025) FOMO-3D: Using vision foundation models for long-tailed 3D object detection. In CoRL,
6. [6] J. Kim et al. (2025) MonoDINO-DETR: Depth-enhanced monocular 3D object detection using a vision foundation model. In IV,
7. [7] A. Kumar et al. (2025) CHARM3R: Towards unseen camera height robust monocular 3D detector. In ICCV,
8. [8] F. Teng et al. (2026) Hallucinating 360°: panoramic street-view generation via local scenes diffusion and probabilistic prompting. In ICRA,
9. [9] J. Wei et al. (2026) OneBEV++: Towards unifying bird’s-eye-view semantic mapping with panoramas. TPAMI.
10. [10] S. Wu et al. (2025) QuaDreamer: controllable panoramic video generation for quadruped robots. In CoRL,
11. [11] F. Teng et al. (2026) Hallucinating 360°: panoramic street-view generation via local scenes diffusion and probabilistic prompting. In ICRA,
12. [12] F. Teng et al. (2026) Spheriverse: 3D scene understanding from spherical observations in the wild. Note: arXiv:2609.09012
13. [13] G. Zhao et al. (2026) Panoramic multimodal semantic occupancy prediction for quadruped robots. Note: arXiv:2603.13108
14. [14] Z. Liu et al. (2020) SMOKE: Single-stage monocular 3D object detection via keypoint estimation. In CVPRW,
15. [15] T. Wang et al. (2021) FCOS3D: Fully convolutional one-stage monocular 3D object detection. In ICCVW,
16. [16] X. Ma et al. (2021) Delving into localization errors for monocular 3D object detection. In CVPR,
17. [17] Y. Wang et al. (2021) DETR3D: 3D object detection from multi-view images via 3D-to-2D queries. In CoRL,
18. [18] Z. Li et al. (2022) BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV,
19. [19] H. Liu et al. (2023) SparseBEV: High-performance sparse 3D object detection from multi-camera videos. In ICCV,
20. [20] J. Park et al. (2023) Time will tell: New outlooks and a baseline for temporal multi-view 3D object detection. In ICLR,
21. [21] N. Zioulis et al. (2018) OmniDepth: Dense depth estimation for indoors spherical panoramas. In ECCV,
22. [22] G. Payen de La Garanderie et al. (2018) Eliminating the blind spot: adapting 3D object detection and monocular depth estimation to 360° panoramic imagery. In ECCV,
23. [23] H. Shi et al. (2026) OneOcc: Semantic occupancy prediction for legged robots with a single panoramic camera. In CVPR,
24. [24] M. Duan et al. (2026) O3N: Omnidirectional open-vocabulary occupancy prediction for embodied intelligent robotics. Note: arXiv:2603.12144
25. [25] B. Coors et al. (2018) SphereNet: Learning spherical representations for detection and classification in omnidirectional images. In ECCV,
26. [26] Y. Su and K. Grauman (2017) Learning spherical convolution for fast features from 360° imagery. In NeurIPS,
27. [27] H. Jiang et al. (2021) UniFuse: Unidirectional fusion for 360° panorama depth estimation. RA-L.
28. [28] Z. Ling et al. (2023) PanoSwin: a pano-style swin transformer for panorama understanding. In CVPR,
29. [29] M. Caron et al. (2021) Emerging properties in self-supervised vision transformers. In ICCV,
30. [30] M. Oquab et al. (2024) DINOv2: Learning robust visual features without supervision. TMLR.
31. [31] A. Kirillov et al. (2023) Segment anything. In ICCV,
32. [32] N. Carion et al. (2026) SAM 3: Segment anything with concepts. Note: arXiv:2511.16719
33. [33] L. Piccinelli et al. (2024) UniDepth: Universal monocular metric depth estimation. In CVPR,
34. [34] R. Wang et al. (2025) MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In CVPR,
35. [35] J. Yao et al. (2026) Open vocabulary monocular 3D object detection. In 3DV,
36. [36] H. Zhang et al. (2025) Detect anything 3D in the wild. In ICCV,
37. [37] W. Huang et al. (2026) WildDet3D: Scaling promptable 3D detection in the wild. Note: arXiv:2604.08626
38. [38] Y. Guo et al. (2025) Depth any camera: zero-shot metric depth estimation from any camera. In CVPR,
39. [39] L. Piccinelli et al. (2025) UniK3D: Universal camera monocular 3D estimation. In CVPR,
40. [40] H. Li et al. (2026) DA{}^{2}: Depth anything in any direction. In ICLR,
41. [41] X. Lin et al. (2026) Depth any panoramas: a foundation model for panoramic depth estimation. In CVPR,
42. [42] G. C. Ganesan et al. (2026) UniDAC: Universal metric depth estimation for any camera. In CVPR,
43. [43] Y. Guo et al. (2026) PanoVGGT: Feed-forward 3D reconstruction from panoramic imagery. In CVPR,
44. [44] J. Yuan et al. (2026) VGGT-360: Geometry-consistent zero-shot panoramic depth estimation. In CVPR,
45. [45] H. Peng et al. (2026) OmniVGGT: Omni-modality driven visual geometry grounded transformer. In CVPR,
46. [46] B. Tan et al. (2026) Masked depth modeling for spatial perception. Note: arXiv:2601.17895
47. [47] M. Dähling et al. (2026) DenseBEV: Transforming BEV grid cells into 3D objects. In WACV,
48. [48] Z. Yu et al. (2024) PolarBEVDet: Exploring polar representation for multi-view 3D object detection in bird’s-eye-view. Note: arXiv:2408.16200
49. [49] Z. Kuang et al. (2026) CoIn3D: Revisiting configuration-invariant multi-camera 3D object detection. In CVPR,
50. [50] H. Lu et al. (2025) Towards generalizable multi-camera 3D object detection via perspective rendering. In AAAI,
51. [51] J. Zhang et al. (2025) GeoBEV: Learning geometric BEV representation for multi-view 3D object detection. In AAAI,
52. [52] Z. Zhang et al. (2026) Unleashing the power of chain-of-prediction for monocular 3D object detection. In CVPR,
53. [53] Y. Yang et al. (2025) 3D-MOOD: Lifting 2D to 3D for monocular open-set object detection. In ICCV,
54. [54] H. Caesar et al. (2020) nuScenes: A multimodal dataset for autonomous driving. In CVPR,
55. [55] S. Chen et al. (2022) AdaptFormer: Adapting vision transformers for scalable visual recognition. In NeurIPS,
56. [56] S. Schubert et al. (2019) Circular convolutional neural networks for panoramic images and laser data. In IV,
57. [57] E. Perez et al. (2018) FiLM: Visual reasoning with a general conditioning layer. In AAAI,