GeoScaffold: Learning Compact Geometric Latents via Reconstruction for Efficient Vision-Language Navigation
This preprint presents GeoScaffold, a training-only geometric supervision framework for streaming vision-language navigation. A 4B policy fine-tuned with eight geometry query tokens reaches 64.8% success rate on R2R-CE and 59.6% on RxR-CE while using only RGB at inference, and runs at 81 ms per frame on an RTX 4090. The approach avoids resident depth sensors or 3D encoders, making spatially aware navigation cheaper to deploy on edge robots.
The authors ask whether geometry can be injected into an end-to-end vision-language-action navigator through training-time reconstruction objectives such that it needs only RGB images and language instructions at deployment.
Streaming Video-LLM navigation policies learn from 2D-pretrained backbones and therefore lack metric and topological spatial reasoning, causing failures on depth-sensitive decisions like judging traversability or when to stop. Existing fixes add depth sensors, 3D encoders, BEV maps, or multi-module planners that remain active at inference and increase hardware cost, memory, and latency.
Earlier methods used geometry-enhanced policies that keep a frozen 3D encoder such as VGGT resident (JanusVLN), lift RGB-D into geometry-aware BEV maps (GA-VLN), or run dual-system planners and zero-shot perception pipelines. These approaches improve spatial understanding but retain sensors, encoders, maps, or tool calls at deployment, complicating edge integration and slowing control loops.
GeoScaffold adds a removable scaffold during training. Stage I trains a 0.74M-parameter depth VQ-VAE with 100 tokens and a 128-entry codebook on training-trajectory depth maps and freezes it. Stage II fine-tunes a streaming policy, using Qwen3-VL-4B in the main results, with eight learnable geometry query tokens—four for depth, two for connectivity, two for traversability—and a hybrid attention mask where each turn's query group attends bidirectionally while other positions remain causal. Hidden states at these tokens reconstruct discrete depth codes, a rendered depth map, and binary connectivity/traversability masks generated offline from RGB with Grounding-DINO and SAM. The reconstruction heads and tokenizer are removed after training; only the learned query-token embeddings and original RGB-text action path remain.
On R2R-CE Val-Unseen, GeoScaffold-4B reaches 64.8% SR and 60.5% SPL with large-scale additional navigation data, outperforming JanusVLN-7B by 4.3 SR and 3.7 SPL, and outperforming the RGB-D method GA-VLN-7B by 3.8 SR and 5.3 SPL while observing only RGB. On RxR-CE Val-Unseen it reaches 59.6% SR, 52.2% SPL, and 66.3 nDTW, improving over NavFoM-7B by 2.2 SR and 2.8 SPL. Trained only on R2R-CE and RxR-CE, the 4B model reaches 55.3% SR and 51.0% SPL on R2R-CE. Across three backbones (LLaVA-Video-7B, Qwen2.5-VL-3B, Qwen3-VL-4B), the same recipe increases SR by 5.5–7.9 points and SPL by 6.4–8.7 points. On an RTX 4090, GeoScaffold-4B runs at 81 ms/frame versus 750 ms for GA-VLN and 1,160 ms for JanusVLN, about 9x–14x faster; the VGGT encoder alone is 400 ms. Ablations show removing depth supervision drops SR from 55.3 to 51.1, removing traversability to 53.1, and removing connectivity to 52.7. Real-robot tests on a Unitree Go2 Edu with RGB-only input complete two zero-shot geometry-demanding episodes, but no quantitative real-world success rate is reported.
The paper does not include an explicit limitations section. Evident limits: the main benchmarks are simulated Matterport3D scenes; real-world evidence is limited to two zero-shot episodes and no quantitative real-world success rate; training requires generated depth, connectivity, and traversability targets and roughly 1,500 A100 GPU-hours; geometry annotations depend on Grounding-DINO and SAM segmentations; and only one real robot platform is shown. No code-release information is given in the paper.
This could help mobile-robot makers—service robots, last-mile or warehouse robots, and quadrupeds such as those from Unitree—to deploy vision-language navigation on edge devices like NVIDIA Jetson without depth sensors or a resident 3D encoder. Training is expensive, but inference is cheap and portable. Commercial use is likely 1–3 years away; product teams would first need broader quantitative real-world validation across environments, robot-specific low-level action policies, and confidence or safety checks for navigation decisions.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.