Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis

Abstract

This paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack of supervisory signal, but rather due to inconspicuous architectural choices: spatially expressive decoders that dilute representational capabilities of the scene encoder, and low-level pixel-space targets that hinder feature learning. We present SNAP, a self-supervised encoder-decoder transformer that addresses both through a pose-conditioned local decoder and a latent-space reconstruction objective. SNAP is task agnostic, and we show that it is competitive with special-purpose geometry-supervised methods. SNAP also performs competitively against self-supervised representations across five tasks: visual localization, pose estimation, point correspondence, depth estimation, and robot manipulation. Remarkably, SNAP’s patch features exhibit emergent viewpoint invariance that approaches heavily supervised models despite lower compute and data budgets. Under camera shifts where standard 2D representations collapse, SNAP degrades more gracefully, revealing that restricting decoder expressivity actively prevents the suppression of transferable geometric structure. https://snap-nvs.github.io

1 Introduction

Geometric understanding from images is a foundation of computer vision. Methods like DUSt3R [1], MASt3R [2], and VGGT [3] demonstrate that transformers can produce rich, transferable geometric representations, when trained on large-scale annotated 3D data. This supervision requirement is a fundamental bottleneck since dense 3D ground truth is expensive to collect at scale.

Novel View Synthesis (NVS) offers a compelling alternative [4, 5, 6, 7]. These models must reason about 3D scene structure to predict unseen viewpoints, providing a rich geometric supervision signal only from posed images, which are abundantly available across videos, robotics platforms, and structure-from-motion pipelines. However, representations produced from feed-forward NVS methods transfer poorly to downstream geometric tasks. We argue this poor transfer is not a signal failure, but rather an architectural failure, driven by two identifiable and independently addressable choices that dilute signal.

First, spatially expressive decoders tend to absorb the representational load of the task. When a decoder has sufficient capacity to reconstruct a target view from any encoder output, the encoder is not pressured to maintain structured, locally discriminative features. While the reconstruction task is solved, the latent representations are not geometrically grounded. Second, pixel-space reconstruction targets require modeling high-frequency aliasing statistics that are weakly related to 3D structure and semantics. Consequently, the reconstruction loss produces gradients that encourage the representation encoder to preserve fine-grained, image-specific statistics that are not directly informative about the underlying 3D structure or semantics.

We present SNAP, which addresses both failure axes with two targeted interventions. First, a pose-conditioned local decoder built on PRoPE [8] restricts the decoder’s receptive field via an explicit BlockNN mask applied to queried-view self-attention. This constraint forces the geometric structure to reside within the encoder’s scene representation rather than being an effect of the generative capacity of the decoder. Second, SNAP operates directly on frozen DINOv3 ViT-B patch features to provide a richer supervision signal than RGB reconstruction. This structured grounding is jointly aware of appearance, geometry, and semantics and produces higher quality representations.

Our main contributions are:

1. SNAP, a self-supervised multi-view spatial reasoning backbone that builds upon frozen 2D semantic latent spaces with an asymmetric NVS prediction objective, and latent feature targets to learn geometrically transferable representations without dense 3D supervision.
2. Mechanistic analyses isolating the independent impact of decoder expressivity and prediction space on geometric transfer, via receptive field sweeps and prediction space landscape experiments.
3. Empirical evaluation across five tasks matching or exceeding self-supervised baselines and competitive with geometry-supervised methods that require large-scale annotated 3D data.

2 Related Work

SNAP sits at the intersection of self-supervised representation learning, novel view synthesis, and geometric foundation models. Detailed discussions are provided in Appendix A.

Self-Supervised Representation Learning. Masked Image Modeling (MIM) has evolved from pixel-reconstruction (MAE [9], SimMIM [10]) toward latent-space prediction (I-JEPA [11], V-JEPA [12]). In multi-view regimes, CroCo [13] and Muskie [14] leverage cross-view completion to learn implicit 3D priors. SNAP extends this trajectory by utilizing frozen DINOv3 [15] features as opposed to pixel space targets used in Muskie [14], which act as semantically structured latent distillation targets to provide a cleaner supervision signal for optimization.

Novel View Synthesis (NVS). Feed-forward NVS was established by architectures such as SRT [16] and RUST [17], recently scaling to large-view models such as LVSM [4], RayZer [6], with contemporaneous works LagerNVS [18] and SVSM [7] continuing to extend the paradigm. However, most of these variants utilize expressive decoders that absorb the representational load of the task, leading to ungrounded latents. By constraining the decoder’s receptive field via a BlockNN mask, adopting PRoPE [8], and targeting a latent feature space, SNAP ensures its representations prioritize 3D structural awareness over reconstruction statistics.

Geometry-Supervised Models. Recent foundation models, including DUSt3R [1], MASt3R [2], VGGT [3], and D4RT [19], achieve state-of-the-art performance but rely on massive 3D-annotated datasets. Similarly, \pi^{3} [20] and Depth Anything 3 [21] provide robust spatial priors via explicit 3D supervision. Conversely, SNAP approaches the geometric transferability of these models while remaining self-supervised and training on significantly less data and compute.

3 Method

SNAP operates in two stages, as illustrated in Figure 1. First, a pose-free multi-view encoder aggregates frozen DINOv3 patch features from N reference views into scene tokens \mathbf{Z} via full cross-view self-attention. Notably, no camera parameters are provided at this stage. Second, a pose-conditioned patch decoder then queries \mathbf{Z} using PRoPE-rotated cross-attention to synthesize the target view’s feature map. The target and context camera intrinsics and extrinsics are fed into PRoPE at this stage. Two design choices jointly force geometric structure into the encoder: a BlockNN mask restricting the decoder’s spatial receptive field (Sec. 3.3), and a latent reconstruction objective against frozen DINOv3 teacher features rather than RGB pixels (Sec. 3.4). We train SNAP on scenes from RealEstate10K [22], DL3DV [23], and Co3Dv2 [24]; comprehensive implementation and architectural details are deferred to Appendix B

3.1 Notation

We use \{\mathbf{I}_{k},T_{k},\mathbf{K}_{k}\}_{k=1}^{N} to denote N reference views, where \mathbf{I}_{k}\in\mathbb{R}^{3\times H\times W} is an RGB image, T_{k}\in\mathrm{SE}(3) its camera-to-world pose with respect to T_{1} (with T_{1} set as the identity) and \mathbf{K}_{k}\in\mathbb{R}^{3\times 3} its intrinsic matrix. During inference, a set of target poses \{T^{*}_{j}\} and intrinsics \{\mathbf{K}^{*}_{j}\} are provided for one or more target views. L=(H/p)\times(W/p) denotes the number of p{\times}p patches per image, \varphi a frozen ViT featurizer (DINOv3 [15]) with output dimension d, and the projection matrix \mathbf{P}_{k}=\hat{\mathbf{K}}_{k}\cdot T_{k}^{-1} maps world-space 3D coordinates to image-space patch coordinates with normalized intrinsics \hat{\mathbf{K}}_{k}.

3.2 Pose-Free Multi-View Encoder as a Spatial Backbone

SNAP’s pose-free multi-view encoder aggregates reference features \mathbf{F}_{k}=\varphi(\mathbf{I}_{k})\in\mathbb{R}^{L\times d} into unified scene tokens \mathbf{Z}=\mathcal{E}_{\mathrm{RoPE2D}}\!\bigl(\{F_{k}\}\bigr)\in\mathbb{R}^{NL\times E}. Each reference image \mathbf{I}_{k} is patchified by the frozen DINOv3 encoder into \mathbf{F}_{k}\in\mathbb{R}^{L\times d}, then projected to the transformer working dimension E via a learned linear layer to yield X_{k}\in\mathbb{R}^{L\times E}. RoPE2D [25] is applied, to queries and keys within self-attention layers, encoding the intra-image 2D patch grid position (i,j). The projected multi-view sequence [X_{1};\,\ldots;\,X_{N}]\in\mathbb{R}^{NL\times E} is processed by a multi-view transformer with L_{e} cross-view self-attention blocks. In full cross-view self-attention, each patch attends to all patches across all reference views. The output \mathbf{Z} is a pose-free scene representation with implicit geometry.

3.3 Pose-Conditioned Patch Decoder as a Feature Predictor

For the second stage of SNAP, a pose-conditioned patch decoder predicts the target feature map:

\hat{\mathbf{F}}=\mathcal{D}_{\textsc{PRoPE}}\!\bigl(\mathbf{Z};\,\{\mathbf{P}^{*}_{j}\},\{\mathbf{P}_{k}\}\bigr)\in\mathbb{R}^{L\times d}.

Our decoder \mathcal{D} is a latent novel-view predictor, which predicts the patchified latent features \mathbf{F}^{*} from the camera parameters of the target and context views. We employ PRoPE [8] for pose-conditioning in the decoder, which consists of alternating blocks of PRoPE-rotated cross-attention against \mathbf{Z} and block-masked self-attention among target queries which is motivated in Sec. 4.1. A single learned feature query \mathbf{q}\in\mathbb{R}^{1\times E} is broadcast into a sequence of L identical target patch queries \mathbf{Q}\in\mathbb{R}^{L\times E}. Spatial symmetry is broken entirely by PRoPE, which differentiates the learned queries across viewpoints using the target intrinsics \mathbf{K}^{*}_{j} and pose T^{*}_{j}. During cross-attention, the target queries \mathbf{Q} serve as q, while the encoder’s scene tokens \mathbf{Z} provide k and v. PRoPE constructs projection matrices \mathbf{P}_{k}=\hat{\mathbf{K}}_{k}\cdot T_{k}^{-1} and rotates queries and keys such that:

\mathrm{Attn}(q,\,k,\,v)=\mathrm{softmax}\!\left(\frac{(\mathbf{P}^{*}_{j}{}^{\!\top}q)\,(\mathbf{P}_{k}^{-1}k)^{\!\top}}{\sqrt{E/H}}\right)\!v,

where H is the number of attention heads and \mathbf{P}^{*}_{j}=\hat{\mathbf{K}}^{*}_{j}\cdot T^{*-1}_{j} is the projection matrix for target view j. A token pair scores high attention when their world-space rays are co-aligned emulating a soft epipolar search.

Following cross-attention, lateral communication among target queries is restricted by a non-overlapping BlockNN mask. Every patch is assigned to an exclusive spatial block of size \text{RF}{\times}\text{RF}. Self-attention is only permitted between queries falling within the same block, explicitly bottlenecking the spatial receptive field of the decoder. As demonstrated in Sec. 4.1, isolating spatial communication forces geometric reasoning into the encoder; therefore, in our optimal SNAP architecture, we strictly enforce \text{RF}=1, reducing target-query self-attention to a pointwise residual MLP.

3.4 Latent Reconstruction Objective

Training minimizes

\mathcal{L}_{\text{recon}}=\underbrace{\frac{1}{Ld}\sum_{p,d}\bigl|\hat{\mathbf{F}}_{p,d}-\mathbf{F}^{*}_{p,d}\bigr|}_{\mathcal{L}_{\mathrm{feat}}}+\frac{1}{2}\underbrace{\vphantom{\frac{1}{Ld}\sum_{p,d}}\bigl(\|\nabla_{x}\hat{\mathbf{F}}-\nabla_{x}\mathbf{F}^{*}\|_{1}+\|\nabla_{y}\hat{\mathbf{F}}-\nabla_{y}\mathbf{F}^{*}\|_{1}\bigr)}_{\mathcal{L}_{\mathrm{grad}}},

where \nabla_{x},\nabla_{y} are first-order spatial finite-differences over the 2D token grid. We apply an L_{1} penalty rather than cosine similarity to the normalized features to promote sharper, sparsity-inducing gradients at structural boundaries. Both \hat{\mathbf{F}} and the ground-truth target map \mathbf{F}^{*}=\varphi(\mathbf{I}^{*}) are \ell_{2}-normalised onto the unit hypersphere before supervision.

\mathcal{L}_{\mathrm{grad}} is not auxiliary: without it, \mathcal{D} can satisfy \mathcal{L}_{\mathrm{feat}} with spatially-uniform predictions that average out local structure, collapsing the geometric signal the encoder is required to encode. Penalising finite-difference residuals enforces spatial fidelity precisely at object boundaries and texture edges, where patch-wise \ell_{1} is least discriminative.

By routing supervision through DINOv3 features rather than RGB, we replace a weak, low-level training signal with more structured guidance. Unlike raw pixels, which contain nuisance variation and little inherent bias toward semantics, DINOv3 features reside on a structured manifold and provide dense, spatially localized embeddings. Section 4.2 motivates this choice and quantifies its effect via a systematic sweep over DINOv3 layers.

3.5 Multitask Evaluation of Representation

We probe the geometric content of frozen \mathbf{Z} on five tasks spanning correspondence, localization, and pose estimation. In every case \mathcal{E} is held frozen; only a lightweight task-specific head is trained.

(T1) Point Correspondence

Given views A and B, a point correspondence head PC(\mathbf{Z}_{A},\,\mathbf{Z}_{B},\,\mathbf{c}_{A}) predicts the normalized coordinate \hat{\mathbf{c}}_{B}\in[0,1]^{2} in view B matching query \mathbf{c}_{A}\in[0,1]^{2} in view A. Both views are encoded under identity poses; pixel-unit intrinsics (f_{x},f_{y},c_{x},c_{y}) preserve field-of-view. Ground-truth correspondences are supplied at training time by the frozen RoMa v2 matcher [26]; pairs with fewer than \tau_{\min} confident matches are discarded. The objective is \mathcal{L}_{\mathrm{corr}}=\|\hat{\mathbf{c}}_{B}-\mathbf{c}_{B}^{*}\|_{1}. Evaluation: \mathrm{pct}@\{5,10\}\,\mathrm{px} in pixels and Average Pixel Error (APE).

(T2) Visual Localization.

Given N reference views with known poses and a single unposed query image, a visual localization head VL(\mathbf{Z}_{\mathrm{ref}},\,\mathbf{Z}_{\mathrm{tgt}}) regresses the absolute target pose as a 6D rotation [27] and 3D translation, all tokens extracted under identity poses. Let \mathcal{H}_{\beta}(a,b)=\mathrm{SmoothL1}(a,b,\beta); the loss is

\mathcal{L}_{\mathrm{loc}}=\,\mathcal{H}_{0.05}(R_{\mathrm{pred}},\,R^{*})+0.5\mathcal{H}_{0.05}(t_{\mathrm{pred}},\,t^{*}).

Evaluation: AUC at \{20^{\circ},\,30^{\circ}\}, where joint AUC integrates \min(\mathrm{recall}_{R},\,\mathrm{recall}_{t}) normalized to the threshold range.

(T3) Multi-View Pose Estimation.

Given N images in an unknown coordinate frame, a pose estimation sequence head PES(\mathbf{Z}) uses N learned view queries to simultaneously regress all poses in a single forward pass. Per-view validity indicators m_{k}\in\{0,1\} mask padded entries:

\mathcal{L}_{\mathrm{seq}}=\frac{1}{\sum_{k}m_{k}}\sum_{k=1}^{N}m_{k}\bigl(\,\mathcal{H}_{0.05}(R_{k},\,R_{k}^{*})+0.5\mathcal{H}_{0.05}(t_{k},\,t_{k}^{*})\bigr).

Metrics are identical to T2, marginalized over all valid views.

(T4) Pointmap and Depth Estimation.

To test dense 3D scene structure, a DPT [28] readout head predicts relative depth for each patch token. Evaluated on Hypersim [29] and 7Scenes [30] using L_{1} relative depth error (AbsRel) and \delta_{1}, the percentage of pixels satisfying \max(\hat{d}/d,d/\hat{d})<1.25.

(T5) Robot Manipulation.

To evaluate viewpoint robustness in embodied settings, frozen encoder tokens serve as the visual state for an imitation learning policy. Evaluated on MimicGen [31] under systematic camera shifts (0.0 to 1.0 radians), measuring task success rate.

4 Analysis: Decoder Expressivity and Prediction Target

Our objective is to learn scene representations that transfer across geometric tasks, not simply to optimize novel-view reconstruction. These goals can diverge: an encoder-decoder NVS model [4, 18] may reconstruct target views accurately while placing little transferable spatial structure in the multi-view encoder. We identify two key design factors that influence whether the learned multi-view representation encodes transferable spatial structure: decoder expressivity and the prediction target.

4.1 Axis 1: Decoder Expressivity Degrades Geometric Transfer

Decoder expressivity. In encoder-decoder NVS models, the decoder’s capacity dictates the balance between its own learned 2D priors and the encoded conditioning signal. A highly expressive decoder acts as a strong unconditional prior, utilizing local spatial context to reconstruct or smooth images independently. While this improves raw synthesis quality, it creates an optimization shortcut: the strong prior weakens the model’s reliance on the conditioning signal, relieving the encoder of the need to capture precise, multi-view geometry. Consequently, the downstream utility of the learned representation degrades. In transformer-based architectures, the strength of this prior is primarily controlled by the number of decoder layers and the receptive field of target-view self-attention.

Prior NVS models commonly use highly expressive decoders with full self-attention in the reconstruction of the target-view [4, 18]. We study the influence of decoder expressivity by varying the receptive field of these self-attention blocks.

Experiment To isolate this factor, we train SNAP variants with varying decoder receptive fields, \text{RF}\in\{1{\times}1,2{\times}2,4{\times}4,8{\times}8,16{\times}16\}. We control the receptive field using a BlockNN mask, which restricts the decoder’s self-attention to non-overlapping k{\times}k spatial windows. Expanding the mask increases the amount of target-view spatial context the decoder can aggregate, while leaving the decoder architecture otherwise unchanged. At the strictest setting, \text{RF}=1{\times}1, spatial communication between target queries is eliminated, reducing the decoder to a patchwise residual MLP. At the broadest setting, \text{RF}=16{\times}16, the window spans the full feature map, giving the decoder full self-attention in target-view, as in [4, 18].

Figure 2 shows that increasing decoder receptive field improves the reconstruction objective but degrades downstream geometric transfer. As RF grows, the decoder is able to use more target-view spatial context, leading to lower feature reconstruction error. However, the frozen encoder representation becomes less useful for geometry: point correspondence error, visual localization, and pose estimation all degrade. This inverse relationship suggests that reconstruction quality alone is a poor proxy for representation quality in encoder-decoder NVS models. The qualitative results in Figure 3 show the same trend. With \text{RF}=1{\times}1, similarity maps are sharply localized around the corresponding point and remain stable across viewpoints, while with complete self-attention (\text{RF}=16{\times}16), the maps do not always track the ground truth correspondence. These results demonstrate limiting target-view spatial communication prevents the decoder from absorbing reconstruction priors, encouraging transferable spatial structure into the multi-view encoder, motivating our patch decoder.

4.2 Axis 2: Prediction Space Determines Representation Quality

The prediction target determines what information the model is pressured to encode. Recent NVS models typically supervise reconstruction in pixel space [4, 18], which can emphasize low-level appearance, texture, and illumination rather than transferable geometry. We instead use latent feature prediction, where earlier ViT layers remain closer to RGB while deeper layers yield high-level structural abstractions that are robust to non-geometric artifacts like lighting and aliasing.

We sweep both the DINOv3 layer that is used as the featurized input and the layer used as the prediction target, asking which feature spaces are optimal for novel-view prediction, both at the input and at the target. We evaluate the resulting encoders on RE10K and DL3DV (val subset), reporting feature reconstruction error, nearest-neighbor point correspondence ([email protected]), patch retrieval (MRR), and intrinsic dimensionality using the TwoNN estimator [32].

Figure 4 shows that the easiest reconstruction targets do not necessarily produce the best spatial representations. Lower-layer targets are closer to RGB and are easier to predict, with the best reconstruction error obtained by the (input: 9, target: 2) pairing. However, downstream geometry improves when predicting higher-layer DINOv3 features, suggesting that more semantic latent targets provide a better supervisory signal for transferable spatial structure. The sweep also reveals a trade-off in the input feature space: input layer 4 gives the best in-distribution PCK and MRR, improving over layer 11 by roughly 5.8\% and 5.2\%, respectively, but its representation has \sim 53\% higher intrinsic dimensionality. We therefore use layer 11 as input, accepting a small loss in in-distribution accuracy for a more compressed and invariant multi-view representation. A comprehensive analysis of these trade-offs across all layers and metrics is provided in Appendix G.

5 Experiments

We evaluate SNAP across five tasks using frozen encoder features with lightweight task-specific decoder heads. This protocol isolates the intrinsic spatial structure learned during pretraining, testing out-of-the-box generalization without confounding effects of task-specific fine-tuning data.

Baselines. We compare to different kinds of methods: feed-forward NVS models without any intermediate geometry prediction (LVSM [4], LagerNVS [18], RayZer [6]), feed-forward 3DGS prediction models (E-RayZer [5]), and feed-forward models trained for geometry prediction itself (VGGT [3]). We also compare to recent work on learning multi-view representations using masked modeling methods (Muskie [14]).

To evaluate domain transfer, we split our protocol: point correspondence and relative pose are evaluated on the validation splits of our pretraining datasets (RealEstate10K [22], DL3DV [23]), while depth estimation and robot manipulation are evaluated on domains unseen by the vision backbone (Hypersim, 7Scenes, MimicGen) to evaluate out-of-domain generalization.

For perceiver-style NVS methods whose latents are not spatially organized (LVSM, RayZer), we cross-attend their latents to a shared H{\times}W spatial grid and compute all downstream features from the resulting spatial grid tokens, ensuring a fair comparison with spatially indexed encoder methods.

Importantly, despite employing full cross-view attention, SNAP maintains a significantly smaller memory footprint than alternative models (Appendix E). By operating on 256{\times}256 images with a patch size of 16, our encoder processes a sequence length of only 256N+8 tokens (including registers). In contrast, VGGT processes 518{\times}518 images at a patch size of 14, resulting in 1369N+5 tokens; it would require more than 12 views for our model to remotely approach the attention cost of these baselines. Combined with Grouped Query Attention (GQA) configured with 16 heads and 8 groups, SNAP alleviates KV costs, operating efficiently even under limited compute budgets.

5.1 Point Correspondence

Point correspondence tests spatial discrimination at the pixel level: a match is correct only if the feature at one pixel is more similar to its true correspondent than to all other image pixels. Ground-truth correspondences for evaluation are supplied by a state-of-the-art dense matcher (RoMa v2 [26]) acting as a correspondence oracle. As Table 1 shows, SNAP outperforms self-supervised baselines on RealEstate10K across all pixel thresholds. Remarkably, SNAP achieves this higher sub-patch precision despite processing inputs at half the resolution of VGGT (256{\times}256 vs. 518{\times}518), underscoring the spatial sharpness of our learned representations. On DL3DV, where scene complexity is higher, SNAP performs competitively with VGGT despite utilizing strictly no dense 3D supervision, approaching its performance within fractional margins across all distance thresholds. NVS baselines (ERayZer, LVSM, RayZer) consistently underperform, confirming that high reconstruction fidelity does not imply spatial feature discriminability.

5.2 Visual Localization

Visual localization requires matching a query image against a reference map and estimating absolute camera pose, thereby combining per-patch discriminability with global geometric consistency under large viewpoint change. We evaluate by training a lightweight regression head on frozen encoder features to directly output a 6D rotation matrix and 3D translation. As Table 2 shows, SNAP establishes a new frontier among self-supervised models on most metrics, confirming that the structured features induced by receptive field restriction generalize beyond local correspondence to global pose inference.

5.3 Relative Pose Estimation

Relative pose estimation uses a direct regression head over the full encoder latent, including register tokens, with early-stage cross-view fusion. On RealEstate10K (Table 2), SNAP achieves high rotation accuracy among NVS-based methods and is competitive with other self-supervised and NVS methods on translation accuracy. The result confirms that SNAP’s encoder retains global geometric reasoning simultaneously with the local discriminative structure demonstrated in Table 1 — the local decoder bottleneck does not degrade large-scale geometric inference.

5.4 Robot Manipulation Under Viewpoint Shift

Beyond static benchmarks, we evaluate whether SNAP can transfer effectively to online decision-making. We compare SNAP against a representative suite of baselines, including semantic backbones (DINOv3, ResNet), geometry-supervised encoders (VGGT), and special-purpose pipelines (Adapt3R [34]) on robotic manipulation under out-of-distribution viewpoint shifts. We evaluate on three MimicGen tasks under camera shifts of 0.0 to 1.0 radians using frozen encoder backbones. Despite not being trained on any robotics-related data, SNAP is competitive across the board.

As Figure 5 shows, semantic encoders (DINOv3, ResNet) collapse rapidly beyond 0.2 radians. Arguably, 2D contrastive training suppresses the cross-view spatial variations which the policy requires to infer 3D configuration. Conversely, SNAP’s multi-view latent reconstruction objective induces strong viewpoint invariance out-of-the-box. Notably, SNAP outperforms its own frozen DINOv3 teacher, generally outperforms task-specific baseline Adapt3R, and tracks closely with the fully geometry-supervised VGGT baseline across all tasks and shift magnitudes.

5.5 Relative Depth Estimation

Relative depth estimation requires predicting scale-invariant depth for each image patch, a direct test of whether an encoder has internalized 3D scene structure. We train a depth head on Hypersim and evaluate performance on 7Scenes and Hypersim validation split. Notably, while LagerNVS was pre-trained on Hypersim and operates at a 518 resolution, LVSM and SNAP are evaluated zero-shot. Furthermore, LVSM utilizes a smaller p=8 patch size, providing a higher-resolution latent grid than SNAP. As Table 3 shows, SNAP outperforms NVS-based encoders. Using explicit 3D supervision (VGGT) performs better in this task. While exposure to Hypersim during pre-training yields high synthetic \delta_{1} for LagerNVS, SNAP demonstrates strong generalization on the real-world 7-Scenes dataset. We supply qualitative results in Appendix F.

6 Conclusion

While our current pretraining relies on posed video sequences and static, textured scenes, and our findings are bound by a fixed compute and data budget that has yet to reveal the full scaling laws of the architecture, SNAP demonstrates that feed-forward novel view synthesis (NVS) is a legitimate powerhouse for self-supervised geometric pretraining. Extending this paradigm to dynamic video remains a critical next step in establishing this approach as a foundation model. More broadly, our results show that NVS’s rich geometric signal is most available when two architectural pathologies are corrected: decoders expressive enough to absorb spatial reasoning that should belong to the encoder, and pixel-space targets that contaminate gradients with low-level statistics.

Restricting decoder receptive field via a BlockNN mask biases spatial reasoning back into the scene tokens; replacing pixel targets with last-layer DINOv3 features further redirects optimization toward a structured, geometry-privileged manifold. The resulting backbone rivals geometry-supervised models across five tasks without dense 3D annotations, and transfers robustly to robot manipulation, where viewpoint shifts break standard 2D semantic models. Moreover Appendix D suggests that these improvements are not restricted to a single teacher, improving correspondence among DINOv3, SAM, and MAE alike. Most strikingly, SNAP demonstrates NVS’s geometric signal is best leveraged by restricting decoder capacity, not by optimizing reconstruction, underscoring that representation quality does not always correspond with reconstruction fidelity.

References

1. [1] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud (2024) DUSt3R: geometric 3d vision made easy. In Conference on Computer Vision and Pattern Recognition (CVPR),
2. [2] V. Leroy, Y. Cabon, and J. Revaud (2024) Grounding image matching in 3d with mast3r. In European conference on computer vision, pp. 71–91.
3. [3] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny (2025) VGGT: visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
4. [4] H. Jin, H. Jiang, H. Tan, K. Zhang, S. Bi, T. Zhang, F. Luan, N. Snavely, and Z. Xu (2025) LVSM: a large view synthesis model with minimal 3d inductive bias. In The Thirteenth International Conference on Learning Representations, External Links: Link
5. [5] Q. Zhao, H. Tan, Q. Wang, S. Bi, K. Zhang, K. Sunkavalli, S. Tulsiani, and H. Jiang (2026) E-rayzer: self-supervised 3d reconstruction as spatial visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7525–7535.
6. [6] H. Jiang, H. Tan, P. Wang, H. Jin, Y. Zhao, S. Bi, K. Zhang, F. Luan, K. Sunkavalli, Q. Huang, and G. Pavlakos (2025) RayZer: a self-supervised large view synthesis model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4918–4929.
7. [7] E. Kim, H. Ryu, T. W. Mitchel, and V. Sitzmann (2026) Scaling view synthesis transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
8. [8] R. Li, B. Yi, J. Liu, H. Gao, Y. Ma, and A. Kanazawa (2025) Cameras as relative positional encoding. In Advances in Neural Information Processing Systems,
9. [9] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick (2022) Masked autoencoders are scalable vision learners. In Conference on Computer Vision and Pattern Recognition (CVPR),
10. [10] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu (2022) SimMIM: a simple framework for masked image modeling. In Conference on Computer Vision and Pattern Recognition (CVPR),
11. [11] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas (2023) Self-supervised learning from images with a joint-embedding predictive architecture. In Conference on Computer Vision and Pattern Recognition (CVPR),
12. [12] A. Bardes, Q. Garrido, J. Ponce, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas (2024) V-JEPA: latent video prediction for visual representation learning. arXiv preprint arXiv:2404.08471.
13. [13] P. Weinzaepfel, V. Leroy, T. Lucas, R. Brégier, Y. Cabon, V. Arora, L. Antsfeld, B. Chidlovskii, G. Csurka, and J. Revaud (2022) Croco: self-supervised pre-training for 3d vision tasks by cross-view completion. Advances in Neural Information Processing Systems 35, pp. 3502–3516.
14. [14] W. Li, S. Liu, P. Qiao, Y. Dou, and T. Hu (2025) Muskie: multi-view masked image modeling for 3d vision pre-training. External Links: 2511.18115, Link
15. [15] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H. Jégou, P. Labatut, and P. Bojanowski (2025) DINOv3. External Links: 2508.10104, Link
16. [16] M. S. M. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, S. Vora, M. Lučić, D. Duckworth, A. Dosovitskiy, J. Uszkoreit, T. Funkhouser, and A. Tagliasacchi (2022) Scene representation transformer: geometry-free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6229–6238.
17. [17] M. S. M. Sajjadi, A. Mahendran, T. Kipf, E. Pot, D. Duckworth, M. Lučić, and K. Greff (2023) RUST: latent neural scene representations from unposed imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17297–17306.
18. [18] S. Szymanowicz, M. Chen, J. Wang, C. Rupprecht, and A. Vedaldi (2026) LagerNVS: latent geometry for fully neural real-time novel view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15443–15453.
19. [19] C. Zhang, G. Le Moing, S. Koppula, I. Rocco, L. Momeni, J. Xie, S. Sun, R. Sukthankar, J. K. Barral, R. Hadsell, Z. Ghahramani, A. Zisserman, J. Zhang, and M. S. M. Sajjadi (2026) Efficiently reconstructing dynamic scenes one d4rt at a time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7382–7392.
20. [20] Y. Wang, J. Zhou, H. Zhu, W. Chang, Y. Zhou, Z. Li, J. Chen, J. Pang, C. Shen, and T. He (2026) \pi^{3}: Permutation-equivariant visual geometry learning. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, pp. 10481–10497. External Links: Link
21. [21] H. Lin, S. Chen, J. H. Liew, D. Y. Chen, Z. Li, Y. Zhao, S. Peng, H. Guo, X. Zhou, G. Shi, J. Feng, and B. Kang (2026) Depth Anything 3: recovering the visual space from any views. In The Fourteenth International Conference on Learning Representations, External Links: Link
22. [22] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely (2018) Stereo magnification: learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817.
23. [23] L. Ling, Y. Sheng, Z. Tu, W. Zhao, C. Xin, K. Wan, L. Yu, Q. Guo, Z. Yu, Y. Lu, et al. (2024) Dl3dv-10k: a large-scale scene dataset for deep learning-based 3d vision. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22160–22169.
24. [24] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny (2021) Common objects in 3d: large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision,
25. [25] B. Heo, S. Park, D. Han, and S. Yun (2024) Rotary position embedding for vision transformer. In European Conference on Computer Vision, pp. 289–305.
26. [26] J. Edstedt, D. Nordstöm, Y. Zhang, G. Bökman, J. Astermark, V. Larsson, A. Heyden, F. Kahl, M. Wadenbäck, and M. Felsberg (2026) RoMa v2: Harder Better Faster Denser Feature Matching. In Proceedings of the European Conference on Computer Vision (ECCV), External Links: Link
27. [27] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li (2019) On the continuity of rotation representations in neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR),
28. [28] R. Ranftl, A. Bochkovskiy, and V. Koltun (2021) Vision transformers for dense prediction. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12159–12168.
29. [29] M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind (2021) Hypersim: a photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10912–10922.
30. [30] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon (2013) Scene coordinate regression forests for camera relocalization in RGB-D images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2930–2937.
31. [31] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox (2023) MimicGen: a data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning (CoRL),
32. [32] E. Facco, M. d’Errico, A. Rodriguez, and A. Laio (2017) Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports 7 (1). External Links: ISSN 2045-2322, Link, Document
33. [33] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
34. [34] A. Wilcox, M. Ghanem, M. Moghani, P. Barroso, B. Joffe, and A. Garg (2025) Adapt3R: adaptive 3d scene representation for domain transfer in imitation learning. In Proceedings of The 9th Conference on Robot Learning, J. Lim, S. Song, and H. Park (Eds.), Proceedings of Machine Learning Research, Vol. 305, pp. 1486–1514. External Links: Link
35. [35] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng (2020) NeRF: representing scenes as neural radiance fields for view synthesis. In ECCV,
36. [36] T. Müller, A. Evans, C. Schied, and A. Keller (2022) Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41 (4), pp. 102:1–102:15. External Links: Link, Document
37. [37] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis (2023) 3D gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42 (4). External Links: Link
38. [38] A. Yu, V. Ye, M. Tancik, and A. Kanazawa (2021) PixelNeRF: neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
39. [39] Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser (2021) IBRNet: learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
40. [40] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su (2021) MVSNeRF: fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14124–14133.
41. [41] D. Charatan, S. L. Li, A. Tagliasacchi, and V. Sitzmann (2024) PixelSplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19457–19467.
42. [42] Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T. Cham, and J. Cai (2024) MVSplat: efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision (ECCV),
43. [43] S. Szymanowicz, C. Rupprecht, and A. Vedaldi (2024) Splatter image: ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10208–10217.
44. [44] T. Mitchel, H. Ryu, and V. Sitzmann (2026) True self-supervised novel view synthesis is transferable. In The Fourteenth International Conference on Learning Representations, External Links: Link
45. [45] Q. Wang, Y. Zhang, A. Holynski, A. A. Efros, and A. Kanazawa (2025) Continuous 3D perception model with persistent state. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
46. [46] N. Keetha, N. Müller, J. Schönberger, L. Porzi, Y. Zhang, T. Fischer, A. Knapitsch, D. Zauss, E. Weber, N. Antunes, J. Luiten, M. Lopez-Antequera, S. R. Bulò, C. Richardt, D. Ramanan, S. Scherer, and P. Kontschieder (2026) MapAnything: universal feed-forward metric 3D reconstruction. In International Conference on 3D Vision (3DV),
47. [47] Z. Cong, Q. Zhao, M. Jeon, and S. Tulsiani (2026) Flow3r: factored flow prediction for scalable visual geometry learning. In CVPR,
48. [48] D. Nordström, J. Edstedt, F. Kahl, and G. Bökman (2025) MuM: multi-view masked image modeling for 3d vision. arXiv preprint arXiv:2511.17309.
49. [49] M. El Banani, A. Raj, K. Maninis, A. Kar, Y. Li, M. Rubinstein, D. Sun, L. Guibas, J. Johnson, and V. Jampani (2024) Probing the 3d awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21795–21806.
50. [50] Y. Chen, X. Chen, A. Chen, G. Pons-Moll, and Y. Xiu (2025) Feat2GS: probing visual foundation models with gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
51. [51] Z. Huang, X. Li, Z. Lv, and J. M. Rehg (2026) How much 3d do video foundation models encode?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 384–394.
52. [52] J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai (2023) Gqa: training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 4895–4901.
53. [53] A. Abouzeid, M. Mansour, Q. Sun, Z. Sun, and D. Song (2025) Geoaware-vla: implicit geometry aware vision-language-action model. arXiv preprint arXiv:2509.14117.
54. [54] J. Song, C. Meng, and S. Ermon (2021) Denoising diffusion implicit models. In International Conference on Learning Representations, External Links: Link
55. [55] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollar, and R. Girshick (2023) Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015–4026.
56. [56] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) ImageNet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255.
57. [57] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba (2017) Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
58. [58] L. Mur-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, N. Ballas, and A. Bardes (2026) V-JEPA 2.1: unlocking dense features in video self-supervised learning. External Links: 2603.14482, Link

Supplementary Material

Appendix A Related Work

A.1 Feed-forward novel-view synthesis

Novel-view synthesis has traditionally been addressed through per-scene optimization of explicit or implicit 3D representations, including neural radiance fields and 3D Gaussian splatting [35, 36, 37]. These methods produce high-quality renderings, but require optimizing a new scene representation at test time. A parallel line of work instead relies on reconstruction priors learned from data, using feed-forward models for generalizable view synthesis. Early examples condition radiance fields or image-based rendering networks on sparse input views [38, 39, 40], while recent methods directly predict feed-forward 3D Gaussian scene representations [41, 42, 43]. More recently, large transformer-based NVS models have moved further away from 3D representations, synthesizing target views directly from learned multi-view representations or scene tokens [4, 6, 5, 44, 18, 7]. These models are typically evaluated by image reconstruction quality, but their learned intermediate representations have also been proposed as general-purpose multi-view scene representations. Our work investigates aspects of NVS pretraining to improve the learned representation for downstream tasks, rather than only for reconstructing novel views.

A.2 Multi-view representation learning

Beyond novel-view synthesis, several works learn multi-view representations explicitly for geometric reasoning. CroCo introduced cross-view completion as a self-supervised pretraining objective for two-view 3D vision [13]. This representation was later used as the foundation for DUSt3R and MASt3R, which used pointmap prediction to solve many diverse tasks [1, 2]. Recent multi-view generalist models such as VGGT, CUT3R, \pi^{3}, MapAnything, D4RT, and Flow3R further extend this direction, predicting camera parameters, depth, pointmaps, tracks, or dynamic scene structure from arbitrary image collections [3, 45, 20, 46, 19, 47]. Many of the strongest geometric foundation models rely on explicit 3D supervision, such as depth, pointmaps, camera poses, or tracks, which limits the scale and diversity of usable training data. Self-supervised alternatives such as Muskie and MuM [14, 48] revisit masked modeling in the multi-view setting. Our work is complementary: rather than using masked reconstruction as the pretext task, we study novel-view prediction as a scalable self-supervised signal, and identify how decoder expressivity and prediction target determine whether NVS produces transferable spatial representations.

A.3 3D awareness in video and multi-view models

Recent work studies whether pretrained 2D, video, and multi-view foundation models encode useful 3D structure. [49] probes frozen 2D visual foundation models for single-view geometry and cross-view surface consistency, finding that some self-supervised image features encode non-trivial 3D information. Feat2GS [50] further probes visual foundation models by reading out 3D Gaussian attributes from frozen features, using novel-view synthesis to evaluate geometry and texture awareness. VidFM3D [51] shows that video foundation model features also contain 3D-aware information extractable with lightweight geometry heads. Our work asks a complementary question: what makes NVS pretraining produce transferable spatial representations? Rather than only probing for 3D information after pretraining, we isolate two design factors that shape the learned encoder representation: decoder expressivity and the prediction target. We also evaluate our representation on a range of geometric and robotics tasks.

Among these, the architectures closest to ours are SVSM [7], LagerNVS [18], and D4RT [19], all concurrent works with distinct objectives. SVSM and LagerNVS both follow prior work in pairing the scene representation with an expressive decoder, which is the main architectural difference from our approach; as we show in Sec. 4.1, this design choice likely limits how well their learned representations transfer to downstream multi-view tasks. Within this pair, SVSM focuses on the data scale required for pretraining, while LagerNVS instead focuses on distilling priors from geometry-pretrained models such as VGGT. D4RT, in contrast, adopts a lightweight cross-attention decoder similar to ours, but trains it with supervised geometry targets, whereas we rely on a purely self-supervised NVS objective.

Appendix B Implementation Details

B.1 Pretraining Setup

Datasets and Splits. We train SNAP on a large-scale mixture of multiview video datasets totaling approximately 117,000 sequences. This training set consists of the RealEstate10K training partition (\sim67,000 sequences), DL3DV-10K (\sim10,000 sequences), and Co3Dv2 (\sim40,000 sequences). For the DL3DV and Co3Dv2 datasets, we hold out 20% of the available sequences for validation, and we use the preprovided test split for validation on RE10K. During training, we randomly sample up to N=6 reference views and one target view per sequence. All images are center-cropped and resized to 256\times 256 pixels. We use 16 layers of self attention for the multi-view encoder with embedding dimension of 1024 and 8 blocks of alternating PRoPE [8] cross-attention and masked self attention. We leverage grouped-query attention [52] operating with 16 query heads and 8 KV-groups.

Optimization. Training was conducted for 40,000 steps using an effective batch size of 256 across multiple classes of GPU hardware using Distributed Data Parallel (DDP). We employ the AdamW optimizer with a peak learning rate of 1\times 10^{-4} and a weight decay of 0.02. The learning rate follows a linear warmup for the first 2,000 steps, followed by a cosine annealing schedule that decays toward zero. We apply gradient clipping with a maximum norm of 0.5 to maintain training stability.

Exponential Moving Average (EMA). To stabilize the learned representations, we maintain an Exponential Moving Average of all trainable parameters with a decay rate of 0.999. The EMA scope encompasses every component of the architecture, including the pose-free multi-view encoder and the pose-conditioned local decoder, which serves as the latent feature reconstruction predictor. All empirical evaluations and downstream geometric probes reported in the main text utilize these EMA weights.

B.2 Point Correspondence

Label Generation. Training labels are generated using image pairs from the train splits of RealEstate 10K, DL3DV-10K, and Hypersim. We employ two distinct strategies depending on the dataset:

• RE10K and DL3DV: We derive pseudo-ground-truth correspondences from a frozen RoMa v2 matcher. To ensure high-quality supervision, we filter for image pairs containing at least \tau_{min}=8 confident matches.
• Hypersim: We compute exact ground-truth correspondences via cross-projection using provided depth maps and camera poses.

Architecture. The point correspondence head employs a three-stage mechanism: (1) symmetric cross-attention layers that map the encoder’s latent tokens to a canonical 16\times 16 spatial grid; (2) a source-view cross-attention block to localize the discriminative feature at query coordinate c_{A}; and (3) a matching stage that computes angular similarity between the query and target-view features via L_{2} normalized dot products.

Specifically, the decode stage (1) consists of 3 cross-attention layers, and the localization stage (2) consists of 3 cross-attention layers. In the final matching stage (3), independent linear projections are applied to the queries and keys to prevent gradient interference. The L_{2} normalized dot product is scaled by a learnable, log-parameterized temperature scalar (initialized to 1.0) before calculating the similarity heatmap. A differentiable spatial expectation over the resulting similarity heatmap produces the final predicted coordinate.

Optimization. The correspondence head is trained for 20,000 steps with an effective batch size of 128. We utilize the AdamW optimizer with a peak learning rate of 2\times 10^{-4}, a 2000-step linear warmup, and a cosine annealing schedule. As specified in our evaluation protocol, all views are encoded under identity camera-to-world poses with pixel-unit intrinsics to isolate the encoder’s intrinsic geometric awareness from external pose signals. Evaluation occurs on the held-out validation splits on all datasets.

B.3 Visual Localization

Architecture. The visual localization head is designed to regress an absolute target pose by localizing an unposed query image against a reference multi-view context. The target pose is parameterized using 9 parameters: a continuous 6D rotation representation and a 3D translation.

The regression head utilizes alternating transformer layers of self-attention (processing the target sequence) and cross-attention (reading from the reference context). A learned CLS token is prepended to the target image sequence, and 1D RoPE is applied to encode sequence positions. After processing the tokens through the transformer blocks, the refined CLS token is extracted and passed through a final multi-layer perceptron to output the 9 pose parameters. To ensure stable early training, the final linear layer’s biases are initialized to represent an identity transformation.

Optimization. We train this head using the RealEstate 10K and DL3DV train splits, and evaluation is conducted on their respective held-out validation splits. Optimization uses a peak learning rate of 2\times 10^{-5} with a cosine annealing schedule and 2,500 steps of linear warmup. The network minimizes a 9D pose regression loss using a SmoothL1 penalty on both the predicted rotation R_{pred} and translation t_{pred}.

B.4 Pose Estimation

Architecture. The relative pose estimation head predicts the pose sequence for all N input views simultaneously in a single forward pass. Similar to the visual localization head, this module uses transformer layers alternating between self-attention and cross-attention. It queries the multi-view context using N learned view queries (one corresponding to each view) equipped with 1D RoPE. The output tokens are processed by a regression MLP (Linear \rightarrow GELU \rightarrow Linear) to predict the 9D pose (6D rotation and 3D translation) for each respective view. Per-view validity indicators mask out padded entries during the loss computation.

Optimization. The sequence head is trained on the RealEstate 10K and DL3DV train splits and evaluated on their held-out validation splits. Training utilizes a peak learning rate of 2\times 10^{-5}, governed by a cosine annealing schedule with 2,500 warmup steps. The model is optimized using a 9D pose regression loss based on the Smooth-L1 error of the rotation and translation predictions.

Appendix C Robot Manipulation

Here we discuss details about the sim experiments, including detailed information about the inputs to the policies, hyperparameter choices, etc.

Tasks.

We evaluate on three MimicGen manipulation tasks: Stack Three-D1, Square-D1, and Coffee-D1. Each task is solved by a single 7-DoF Franka Panda arm with a parallel-jaw gripper. For every task we train on the full MimicGen demonstration set of 1{,}000 demonstrations generated from the source human demos. Observations are RGB images from two cameras: a fixed third-person camera and a wrist-mounted camera on the robot. The policy is only conditioned on these visual features.

Vision Encoders.

We benchmark six vision backbones with otherwise identical training and policy hyperparameters. The encoders we benchmark are the following: (i) ImageNet-pretrained ResNet-18; (ii) Adapt3R, a 3D-aware encoder built on a CLIP RN50 backbone with a point-cloud attention extractor and pyramid feature network; (iii) DINOv3 ViT-B/16; (iv) Muskie, a multi-view backbone with alternating attention; (v) VGGT, a geometry-aware transformer aggregator; (vi) SNAP (ours), a multi-view backbone built on a frozen DINOv3 featurizer. For VGGT and SNAP we are using their patch, register, and camera (only applicable to VGGT) tokens. Note that all multi-view encoders see both cameras as input views (V{=}2), whereas single-view encoders (ResNet, Adapt3R, DINOv3) process each camera independently and their tokens are concatenated downstream. As for the policy, most encoders (Muskie, DINOv3, ResNet, SNAP) we use their last layer features to condition the policy. For VGGT, we use the element-wise mean of all four intermediate aggregator-layer features, preserving the per-token shape, which has been shown to an extent to provide better viewpoint robustness [53].

Policy.

For all encoders we use the same diffusion policy, a Diffusion-Transformer (DiT) that predicts a chunk of H_{a}{=}8 future actions in delta pose representation, of which the first H_{e}{=}4 are executed (action-horizon) before re-planning. The policy uses 1 encoder block and 3 transformer decoder blocks with hidden dimension d_{\text{embed}}{=}512, 4 attention heads, GELU activations, and dropout 0.1. The diffusion process uses 100 training steps and 8 DDIM [54] sampling steps at evaluation. We train all models using AdamW (\beta_{1}{=}0.95, \beta_{2}{=}0.999) with a learning rate of 1\times 10^{-4} and weight decay of 1\times 10^{-6}, a cosine schedule with 1{,}000 warm-up steps decaying to 1\% of the peak, batch size 128, and gradient clipping at 100 for 350 epochs. Models use embedding dimension d_{e}{=}512, chunk size 8, and action horizon 4. Results are reported over 400 rollouts per task per policy.

Image Resolution and Augmentation.

We use a common training resolution of 256{\times}256 for DINOv3, SNAP. For VGGT and Muskie we used a resolution of 224{\times}224. ResNet and Adapt3R are trained at 128{\times}128 resolution (following a similar spirit as Adapt3r’s training resolution). The same lightweight augmentation is applied to every encoder: per-batch color jitter (brightness, contrast, saturation, hue) followed by a random translation crop of \pm 8 pixels.

Appendix D Comparison with Teachers

D.1 Comparison with Frozen Teachers

SNAP uses a frozen visual teacher to provide feature targets for its feature-space reconstruction objective. While our main experiments use DINOv3 [15], the proposed multi-view pretraining objective is not specific to this teacher. We therefore compare SNAP using frozen DINOv3 [15], SAM [55], and MAE [9] features.

Point Correspondence.

As shown in Table 4, SNAP substantially improves point correspondence for every teacher, dataset and threshold. 1

On held-out Hypersim, [email protected] improves from 29.29 to 55.35 with DINOv3 [15], from 40.82 to 52.27 with SAM [55], and from 11.01 to 50.67 with MAE [9]. The gains are largest at tight thresholds and shrink at [email protected] for the stronger teachers (DINOv3: 84.22 to 88.35; SAM: 78.74 to 83.64), which approach saturation, while MAE still improves by 23.35 points (60.81 to 84.16). The same trends hold on RealEstate10K and DL3DV. Thus, the geometric gains are not specific to the DINOv3 feature space. Instead, the multi-view, pose-conditioned reconstruction objective consistently improves the spatial consistency and geometric utility of the underlying teacher representation.

Among the base teachers, DINOv3 is strongest at [email protected], whereas SAM is stronger (or tied) at [email protected] and [email protected] on all three datasets (e.g., Hypersim [email protected]: 40.82 vs. 29.29). After SNAP, the spread between teachers narrows considerably (Hypersim [email protected] ranges from 50.67 to 55.35, versus 11.01 to 40.82 for the base features), and DINOv3 yields the strongest SNAP representation in all but one setting (DL3DV [email protected]). DINOv3 is therefore a good choice but not a requirement for SNAP. The substantial improvements obtained with SAM [55] and MAE [9] demonstrate that the gains arise from combining high-order abstract representations with the view synthesis objective rather than simply inheriting the capabilities of the DINOv3 teacher. The remaining differences between teachers show that the quality of the underlying feature space is still significant.

Relative Depth Estimation.

We further compare SNAP against its DINOv3 [15] teacher in Table 5. SNAP outperforms its teacher on Hypersim, maintaining more accurate relative depth along with a better \delta_{1}. This pattern holds in 7Scenes where SNAP displays a 15.8% improvement in accuracy in depth estimation while displaying a comparable \delta_{1}. The improvements demonstrate that SNAP exceeds its teacher at dense prediction tasks as well as point correspondence.

Single-View Semantics.

We additionally evaluate whether the multi-view transformation preserves single-view semantic information. As shown in Table 6, the DINOv3 [15] variant improves slightly on ImageNet [56] k-NN, from 68.45 to 68.81 Top-1 accuracy. The SAM [55] variant also improves, from 18.06 to 21.46 Top-1 accuracy, while the MAE [9] variant decreases from 30.78 to 28.64. On ADE20K [57] linear-probe segmentation, DINOv3 [15] obtains 51.92 mIoU and 87.27 pixel accuracy, compared with 49.87 mIoU and 84.31 pixel accuracy for SNAP (Table 7). These results show that SNAP largely retains the semantic discriminability of the teacher while deliberately shifting the representation toward multi-view geometric transfer.

D.2 Comparison with V-JEPA

SNAP and V-JEPA share the use of feature-space prediction rather than direct pixel reconstruction, but they differ fundamentally in the type of representation they are designed to learn. V-JEPA [12] is a video representation learner that predicts latent features across video observations, while V-JEPA 2.1 [58] extends this paradigm toward dense video features. In contrast, SNAP is explicitly a multi-view encoder: it learns a shared scene representation from multiple posed images and trains that representation to predict features at arbitrary target viewpoints. Thus, the comparison separates the benefit of feature-space prediction from the additional geometric constraint introduced by explicit multi-view representation learning.

Hypersim, RealEstate10K and DL3DV.

We evaluate a V-JEPA 2.1 ViT-L model using the same frozen-feature point-correspondence protocol as SNAP, i.e., global soft-argmax matching with a low temperature. As shown in Table 8, SNAP outperforms both V-JEPA 2.1-L and DINOv3 on RealEstate10K, DL3DV and Hypersim at every threshold, with the largest margins at tight thresholds.

On Hypersim, SNAP achieves 55.35, 77.90 and 88.35 PCK at thresholds of 0.1, 0.2 and 0.3, respectively, compared with 21.45, 50.21 and 72.44 for V-JEPA 2.1-L. DINOv3 obtains 29.29, 63.67 and 84.22. On RealEstate10K, SNAP achieves 78.21, 91.32 and 95.62, compared with 35.85, 67.05 and 85.05 for V-JEPA 2.1-L and 45.83, 80.13 and 93.71 for DINOv3. DL3DV shows the same pattern ([email protected]: 58.74 for SNAP vs. 24.55 and 28.00). The advantage narrows at [email protected], where all methods approach saturation; on RealEstate10K the SNAP–DINOv3 gap there is under 2 points.

The consistent advantage across all three benchmarks indicates that feature-space prediction alone is insufficient to obtain SNAP’s level of geometric transfer. The explicit multi-view constraint requires the representation to remain useful across viewpoint changes, rather than only across temporally related video observations.

Overall, the comparison highlights a fundamental distinction between video/image representation learning and explicit multi-view representation learning. Although both V-JEPA [12] and SNAP operate in feature space, their training constraints are different. V-JEPA learns representations through latent prediction between video observations, whereas SNAP learns a shared scene representation from multiple posed images and requires that representation to support feature prediction across viewpoints. The stronger correspondence performance of SNAP across Hypersim, RealEstate10K, and DL3DV indicates that this explicit multi-view geometric constraint provides a distinct advantage beyond feature-space prediction alone.

Appendix E Inference Speed

We evaluate the inference efficiency of SNAP against Muskie [14] and VGGT [3] to characterize the computational overhead of our architecture. As shown in Table 9, SNAP demonstrates superior scalability and memory efficiency as the number of input views (V) increases.

Memory and Parameter Efficiency.

SNAP is significantly more memory-efficient than both baselines. At a high-resolution regime of V=100, SNAP requires only 1.03 GB of peak GPU memory, representing a 40.5% reduction compared to Muskie (1.73 GB) and a 69.8% reduction compared to VGGT (3.41 GB). This efficiency is underpinned by SNAP’s compact parameterization; with only 233.38M parameters including the DINOv3 backbone, SNAP is approximately 1.3\times smaller than Muskie and 3.9\times smaller than the geometry-grounded VGGT.

Temporal Performance.

While SNAP consistently outperforms VGGT, achieving a 2.9\times speedup at V=100, it is roughly 36% slower than Muskie in the high-view regime (0.245s vs. 0.180s). We attribute this latency gap to Muskie’s architecture, which utilizes fewer multi-view self-attention blocks compared to our design. However, for practical deployment, SNAP offers a more accessible memory profile: at V=100, SNAP requires only 1.03 GB of VRAM, whereas Muskie consumes 67.9\% more memory. For practitioners working with limited GPU memory or ultra-dense view counts, SNAP provides a viable alternative that prioritizes a smaller memory footprint over raw inference throughput. Notably, SNAP provides a different tradeoff of performance, speed, and memory.

Appendix F Qualitative Depth Estimation Results

We provide qualitative relative depth estimation results from our trained probe (see Figure 7). Given a sparse collection of four input images, we use a frozen backbone with trained DPT layers to predict relative depth. Overall, SNAP produces coherent depth predictions that are comparable to those of LagerNVS (geometry pretrained). In contrast, LVSM exhibits noisier predictions and less consistent depth structure across the examples.

The comparison with LagerNVS is not resolution-matched: LVSM and SNAP are decoded at 256px, whereas LagerNVS is decoded at 518px. The higher decoding resolution of LagerNVS likely contributes to its stronger depth estimation.

Appendix G Landscape Plots

We expand the sweep methodology from Section 4.2 to evaluate a broader set of metrics: mAP/MRR (reflecting bag-of-words correspondence retrieval), cycle consistency, and variance ratio (the ratio of within-scene to across-scene variance, indicating how well features cluster by scene) (Figure 8). Interestingly, we observe a distinct trade-off. While pairing mid-layer inputs with last-layer prediction targets yields higher raw correspondence, this configuration results in lower cycle consistency compared to initializing with the last layer of DINO. This performance valley for mid-layer inputs is consistently prominent across mAP/MRR, cycle consistency, and the variance ratio. Ultimately, this highlights that optimizing for raw local correspondence does not necessarily yield the most structurally robust, 3D-grounded representations, justifying our use of deeper semantic layers to enforce global multi-view consistency.

Appendix H Qualitative Feature Visualizations

To qualitatively assess what these models learn, we visualize their high-dimensional feature maps projected into RGB space. As illustrated in Figure 9, standard 2D priors like DINO group scenes into flat, semantic regions but lack fine-grained 3D geometric structure. In contrast, SNAP clearly learns a representation distinct from its DINOv3 teacher: it introduces smooth, continuous spatial gradients across physical surfaces (e.g., ground planes), indicating an emergent understanding of 3D geometry while preserving fine-grained, discriminative object boundaries. Alternative multi-view baselines fail to achieve this geometric abstraction. Muskie heavily encodes raw appearance and texture rather than underlying structure, resulting in representations that simply mimic the RGB input. Meanwhile, LagerNVS suffers from severe, high-frequency grid artifacts, disrupting the continuous spatial coherence required for robust multi-view reasoning along with feature anisotropy.