FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs

Abstract

We present FoveaVLA, a Vision-Language-Action (VLA) model that replaces the wrist camera with learned foveation actions. Wrist cameras provide close-up detail but restrict the view to the end-effector’s surroundings. External cameras see the whole scene, but the policy receives them at low resolution, so the fine detail needed for manipulation is lost. FoveaVLA recovers this detail by predicting, alongside each action chunk, the interaction region to observe at the next policy query. The policy then receives crops of these regions, taken from the original-resolution images, together with the full views. We build FoveaVLA on \pi_{0.5} in two configurations, Single and Dual, using one or two external stereo cameras with full and foveal RGB and depth inputs, and post-train both on DROID with RoboInter interaction annotations. Across four real-world task suites, FoveaVLA-Single achieves 70.8% success with one external RGB-D view, and FoveaVLA-Dual achieves 77.1%, versus 52.1% for the wrist-equipped \pi_{0.5}. Both match the baseline on plain pick-and-place and improve success with distractors, cloth manipulation, and far-away objects.

†

I Introduction

Recent progress in robot manipulation has been driven by large models trained on diverse robot datasets [1, 2, 3]. Most of these models take a wrist camera as input [4, 5, 6], which provides close-up observations that learned policies often depend on [7, 8, 9]. This close-up view comes at the cost of scene coverage: the camera moves with the end-effector, so distant targets and surrounding context can lie outside its view, more so when manipulation spans a larger workspace or involves a mobile robot. The wrist camera also adds hardware and creates an observation mismatch with human demonstrations recorded without a wrist view [10, 11]. Can a policy get the close-up detail it needs from external observations alone, so that the robot no longer needs a wrist camera?

External cameras record high-resolution images, but resizing a whole scene to the policy’s low input resolution reduces the detail available for the gripper and object, and uniformly increasing the input resolution spends computation on background regions. External views also lack a consistent geometric frame: a wrist camera maintains a fixed pose relative to the end-effector, whereas an external view depends on the camera’s placement relative to the robot. Prior work reports lower success for policies learned without wrist observations [12, 7]. We observe a similar gap on our four real-world task suites: \pi_{0.5} post-trained on two external RGB views alone achieves 25.0% success, compared with 52.1% with the wrist view.

Existing approaches obtain close-up observations through additional sensing or constructed viewpoints. Generative methods synthesize wrist images from external observations [13, 14]. Others render virtual views from 3D scene representations [15, 16], or move a camera on a dedicated arm or neck [17, 18]. Human-gaze-guided foveation is closer to our goal of selecting detail within an existing image [19, 20], but relies on gaze recorded during data collection. We instead learn where to crop from existing robot demonstrations, supervised by their interaction annotations.

Human vision allocates detail selectively. Peripheral vision provides scene context while central vision resolves the detail needed to grasp an object [21], and gaze reaches the next contact point before the hand does [22]. We take the same approach in FoveaVLA, which replaces the wrist camera with a learned foveation action that selects where to look next. Cropping the predicted interaction region from the original-resolution image and resizing it to the same 224\times 224 encoder input devotes many more pixels to the gripper and object than the resized full view does. Feeding this crop alongside the full view gives the policy both the close-up detail it needs for manipulation and the layout of the whole scene.

We build FoveaVLA on \pi_{0.5} [23]. From the current observation, it generates a robot action chunk and predicts one bounding box per external view for the interaction region at the next policy query. The predicted boxes crop the newly captured images, and the crops are added to the next policy input alongside the full views. We derive supervision from RoboInter annotations [24] on DROID demonstrations [5], constructing a bounding box around the components involved in each annotated skill, such as the gripper, object, and target region. Annotations at the next query provide targets for future-box prediction. We instantiate the method as FoveaVLA-Single and FoveaVLA-Dual, using one and two external stereo cameras, respectively, with full and foveal RGB images and aligned depth estimated from each stereo pair. Both operate without a wrist camera; Dual adds a second external viewpoint to the same foveation mechanism.

We post-train both configurations on DROID and evaluate them on the four real-world task suites. FoveaVLA-Single achieves 70.8% success using one external RGB-D view, exceeding the wrist-equipped \pi_{0.5} at 52.1%, and FoveaVLA-Dual achieves 77.1%. The largest gains are on pick-and-place with distractors, where Single raises success from 25.0% to 58.3%, and on moving far objects, from 41.7% to 75.0%. On MolmoSpaces [25], FoveaVLA-Dual achieves 22.4% average zero-shot success versus 17.8% for \pi_{0.5}-DROID [26], outperforming it on seven of nine tasks.

II Related Work

Wrist cameras in robot manipulation. Eye-in-hand cameras date back to visual servoing [27] and were adopted in learning-based manipulation after empirical studies showed their benefit [12]. Hsu et al. argued that the partial observability of the hand-centric view is an inductive bias that aids generalization [7]. Wrist cameras are now standard in bimanual hardware [4], handheld collection devices [6], datasets and benchmarks [5, 28, 25], and generalist policies [23, 29, 30]. Wrist-only observations can lack the scene context needed for manipulation, motivating a wide-angle fisheye lens in UMI [6] and an additional external view in MV-UMI [31]. Policies also come to depend on this view: LIBERO-Plus finds that multi-view VLAs retain much of their performance with the external view masked [8], and other studies find policies including \pi_{0.5} more sensitive to masking or perturbing the wrist view [9, 32, 33]. FoveaVLA keeps the close-up that makes the wrist view valuable, without mounting a camera on the hand.

Constructed viewpoints and active vision. WristWorld generates wrist-view videos from anchor views [13], while inference-time view synthesis supplies in-hand images to a trained policy [14]. WARPED renders wrist-aligned training observations from egocentric human videos but retains a physical wrist camera during execution [34]. Others re-render the scene from virtual viewpoints using 3D scene representations, zooming coarse to fine on point clouds [16] or selecting virtual views and applying 3D zoom within a VLA [15]. Physical active-vision systems instead control a camera using demonstrations [17, 18, 35] or task reward [36]. EyeVLA predicts pan, tilt, and zoom for a physical PTZ camera [37]. Such systems require camera actuation and supervision or rewards for camera control. FoveaVLA uses fixed cameras and selects regions within the images they already capture.

Learned visual attention and foveation. Human vision resolves fine detail only in a small central region, and the eyes move this region to the grasp point before the hand arrives [38, 21, 22]. Attention models learn glimpse actions [39], and foveation also aids recognition [40] and gaze-conditioned generation [41]. VLMs retrieve local evidence through crops [42, 43, 44, 45] to answer questions about a given image, whereas we use predicted regions to select the next robot observation. In manipulation, foveation has been supervised by human gaze, as a foveal crop fed to the policy [19] or gaze co-predicted with actions [20], a signal recorded with eye trackers during collection and not provided by DROID. AtVLA corrects visual attention with register tokens and, when uncertain, crops a region chosen by attention rollout [46]. Another approach trains a policy that chooses which regions a sensor captures for frozen downstream models, including a manipulation policy [47]. FoveaVLA instead decodes a future interaction box alongside each action chunk. Prior VLAs also use localization supervision [23], spatial traces [48, 49], or visual and textual affordances [50], but only as auxiliary targets that help action prediction. In FoveaVLA, the predicted box determines the crop the policy receives next, and its supervision comes from existing large-scale interaction annotations [24].

III Preliminaries

Vision-Language-Action models. A VLA is a robot policy \pi_{\theta}(A_{t}\mid o_{t},\ell) that conditions on the current observation o_{t}, comprising camera images and the robot state q_{t}, and a language instruction \ell. It outputs an action chunk A_{t}=(a_{t+1},\ldots,a_{t+H}) spanning a prediction horizon of H control steps. In receding-horizon execution, the controller can execute only the first w\leq H actions before replanning. A common recipe starts from a pretrained vision-language model, adds an action head, and trains the resulting policy by imitation on robot demonstrations [2, 3]. The visual encoder converts camera images into patch tokens, which the language-model backbone processes together with the instruction and robot state to form the representation used for action generation.

Backbone and action generation. We build on \pi_{0.5} [23]. Its PaliGemma backbone [51] combines a SigLIP visual encoder with a Gemma language model, while a separate action expert generates continuous action chunks through flow matching, conditioned on the observation representation. PaliGemma’s vocabulary also includes location tokens for image coordinates in detection and grounding. The \pi_{0.5} base checkpoint does not emit these tokens at inference; we reuse them to represent foveation actions in Sec. IV-B.

IV Method

FoveaVLA learns where to look alongside how to act (Fig. 2). From the full and foveal RGB-D observations, the robot state, and a task instruction, it predicts an action chunk and the interaction region for the next observation. The predicted regions determine which crops of the next external images the policy receives. We post-train it from the \pi_{0.5} base checkpoint on DROID with RoboInter annotations. FoveaVLA-Single uses one external stereo camera, and FoveaVLA-Dual uses two.

IV-A Full and Foveal Observations

Let N\in\{1,2\} be the number of external views for Single and Dual, respectively. Let I_{t}^{v} be the source RGB image, at the camera’s original resolution, from external view v\in V=\{1,\ldots,N\} at time t, and let b_{t}^{v} specify the region selected for detailed observation. The full view G_{t}^{v}=\mathcal{R}(I_{t}^{v}) is the source image resized and padded to the encoder’s low input resolution (224\times 224 in our case). The foveal crop F_{t}^{v}=\mathcal{C}(I_{t}^{v},b_{t}^{v}) is a square crop around b_{t}^{v}, extracted at the source resolution and resized to the same low resolution. The full view preserves the scene layout, while the foveal crop spends the same number of pixels on the interaction region.

Each external camera is a stereo camera. We use Fast-FoundationStereo [52] to estimate disparity from its rectified image pair and convert it to metric depth using the focal length and stereo baseline. The depth map D_{t}^{v} is aligned with I_{t}^{v} and provides an explicit distance cue. We apply the same resizing and cropping to depth as to RGB, giving D_{G,t}^{v} and D_{F,t}^{v}. Depth is mapped through a fixed logarithmic scale and colorized for the visual encoder; patches with insufficient valid depth are masked. The resulting observation is o_{t}=\left(q_{t},\left\{G_{t}^{v},F_{t}^{v},D_{G,t}^{v},D_{F,t}^{v}\right\}_{v=1}^{N}\right), where q_{t} contains the robot’s joint positions and gripper state. RGB and colorized depth use the same visual encoder and enter the transformer as separate sequences of visual tokens. The task instruction \ell and discretized robot state complete the shared input prefix.

IV-B Foveation Actions

We extend the VLA policy defined in Sec. III to predict one bounding box per external view alongside the robot action chunk. Each box describes the interaction region at t+w, when the policy is queried next after executing the first w actions. We predict H=16 actions and execute w=8 per query (Appendix B). We target t+w because the gripper and objects move during the executed actions, and apply the predicted box to the image captured at that time. Denoting B_{t+w}=\{b_{t+w}^{v}\}_{v\in V} for the foveation actions, the joint policy factorizes as

\pi_{\theta}(A_{t},B_{t+w}\mid o_{t},\ell)=\pi_{\theta}(A_{t}\mid o_{t},\ell)\,p_{\theta}(B_{t+w}\mid o_{t},\ell).

The two outputs share the observation representation, but are predicted by two independent heads: the action expert generates A_{t}, and the language-model head decodes B_{t+w}.

We initialize FoveaVLA from the \pi_{0.5} base checkpoint, retaining its backbone and flow-matching action expert. To predict B_{t+w}, we reuse the language-model output head to autoregressively decode the existing location tokens introduced in Sec. III.

Each box is represented by four coordinates on the resized and padded full view, quantized into 1{,}024 bins and serialized in (y_{\min},x_{\min},y_{\max},x_{\max}) order. The boxes form a sequence of 4N location tokens: four for Single and eight for Dual. The output vocabulary is restricted to location tokens. The action expert attends to the shared input prefix, while each box token also attends to the preceding box tokens. The two output branches are masked from each other.

Let \hat{b}_{t+w}^{v} denote the predicted box in source-image coordinates. It determines the next foveal observation through F_{t+w}^{v}=\mathcal{C}\!\left(I_{t+w}^{v},\hat{b}_{t+w}^{v}\right), with the same crop applied to the aligned depth map.

IV-C Learning from Interaction Annotations

We derive foveation supervision from the RoboInter annotations associated with DROID demonstrations [24, 5]. For each annotated frame and external view, we construct one interaction box as the smallest axis-aligned rectangle enclosing the components required by the annotated primitive skill. Figure 3 illustrates transfer and fold; both use the gripper, manipulated object, and annotated target region. Appendix A specifies the complete role mapping and preprocessing procedure. The interaction box defines the supervision target; the foveal crop is a square region enclosing this box, extracted as described in Sec. IV-A.

A training example pairs the observation and demonstrated action chunk at t with the interaction boxes at t+w. To improve robustness to imperfect predictions, we augment ground-truth crops at t by perturbing their centers and scales while retaining the annotated region. Some crops are replaced by full views to match the initial query at inference. RGB and depth share the same crop geometry. Training examples require valid current and future annotations in the selected view for Single and in both views for Dual.

During post-training, we jointly optimize action generation and foveation prediction. For the action objective, we interpolate between a demonstrated chunk A_{t} and Gaussian noise \epsilon\sim\mathcal{N}(0,I) as A_{t}^{\tau}=(1-\tau)A_{t}+\tau\epsilon, where \tau\in[0,1] is the flow time. The action expert v_{\theta} learns the corresponding velocity field:

\mathcal{L}_{\mathrm{flow}}=\mathbb{E}_{\mathcal{D},\tau,\epsilon}\left[\left\|v_{\theta}(A_{t}^{\tau},o_{t},\ell,\tau)-(\epsilon-A_{t})\right\|^{2}\right],

where the squared error is averaged over action entries.

For foveation prediction, let z_{1:4N}^{*} be the location tokens for the N annotated future boxes. We train their prediction with teacher forcing:

\mathcal{L}_{\mathrm{foveation}}=-\mathbb{E}_{\mathcal{D},j}\left[\log p_{\theta}(z_{j}^{*}\mid o_{t},\ell,z_{<j}^{*})\right],

where the expectation averages over training examples and uniformly over the 4N token positions. The total loss is

\mathcal{L}=\mathcal{L}_{\mathrm{flow}}+\lambda\mathcal{L}_{\mathrm{foveation}},

with \lambda=0.1. Both losses update the shared backbone. We additionally apply dropout to the depth streams to train with partial depth availability.

IV-D Closed-Loop Inference

The initial foveal inputs are the full views. Subsequent foveal inputs use the preceding box predictions to crop the current RGB images and aligned depth maps, as described in Sec. IV-B. Each policy query encodes the observation prefix once for both outputs. The action expert generates the chunk by integrating the learned flow from noise to actions, and the chunk is returned immediately; the bounding box location tokens are then decoded greedily from the cached prefix while the robot executes the first w actions, so the boxes are ready when the next frames arrive at t+w.

V Experiments

We evaluate learned foveation through five questions:

• (Q1) Can learned foveation replace the wrist camera?
• (Q2) What do foveal crops add beyond box supervision?
• (Q3) What do depth and a second view add?
• (Q4) Does the policy transfer to unseen environments?
• (Q5) Can the policy predict where to look next?

V-A Experimental Setup

Data and training

We post-train FoveaVLA from the \pi_{0.5} base checkpoint on DROID [5], comprising \sim\!75K teleoperated Franka Panda episodes with a Robotiq 2F-85 gripper, two external stereo cameras, and a wrist camera. Our models use only external views and RoboInter interaction-box supervision [24] (Sec. IV-C), following the open-source training recipe [26]. All models are trained on eight NVIDIA H100 GPUs with a global batch size of 256. Appendix A details the data construction, and Appendix B gives the training and inference settings for Single and Dual.

Tasks and protocol

We evaluate on the real DROID setup across four suites: uncluttered, near-object Pick-and-Place; Pick-and-Place with Distractors; Fold/Unfold with near and far cloth placements; and Move with far objects. All methods share the same 12 randomized object, placement, and prompt instances per suite (48 trials total). The external cameras are placed by hand without extrinsic calibration. Appendix D lists the complete task instances and success criteria. Trials run at a target 15 Hz for at most 900 control steps, stopping early if continued execution risks robot damage. Success requires full task completion; we report mean success \pm standard error of the mean (SEM), in percent, over 12 trials per suite and pool all 48 for Overall.

Models and comparisons

FoveaVLA-Single and FoveaVLA-Dual use one and two external RGB-D views, respectively, with full and foveal RGB and depth inputs (four images per view). We compare against wrist-equipped \pi_{0.5}, which shares our base checkpoint and training data, and its deployment-time wrist blackout, \pi_{0.5}-Blackout; \pi_{0.5}-Single-External and \pi_{0.5}-Dual-External variants, which were post-trained with one or two full external RGB views without a wrist view, box prediction, or crops; and \pi_{0.5}-Single-External-RGB-D and \pi_{0.5}-Dual-External-RGB-D, which add depth to the full views. ECoT-inspired [49] uses two external RGB views and our box supervision but no crops. FoveaVLA-Single-RGB and FoveaVLA-Dual-RGB retain foveation while removing depth from Single and Dual.

The blackout tests how much \pi_{0.5} relies on the wrist view at deployment, and the external-only baselines test how well a policy learns from external views alone.

V-B Replacing the Wrist Camera (Q1)

FoveaVLA-Single achieves 70.8% overall success using one external RGB-D view and FoveaVLA-Dual 77.1%, compared with 52.1% for wrist-equipped \pi_{0.5}. Both match the baseline’s 100.0% success on plain Pick-and-Place and achieve higher success on the other three suites (Fig. 4). Table I reports all baselines and ablations.

Close-up detail across the workspace

With a single external RGB-D view, success on Distractors rises from the wrist-equipped baseline’s 25.0% to 58.3%, and success on Move rises from 41.7% to 75.0%. Dual achieves the strongest Fold/Unfold and Move results (75.0% and 83.3%). Figure 5 suggests why: \pi_{0.5} places almost all image attention on the wrist view and almost none on the external view, so it can miss a target outside the wrist view. FoveaVLA keeps attending to the full external views, which cover the workspace, while the crops add close-up detail.

Inference latency

On an RTX 5080, FoveaVLA-Single takes approximately 125 ms per policy query (prefix encoding and action generation) and 50 ms for cached-prefix box decoding. FoveaVLA-Dual takes roughly twice as long. Box timing includes all camera views and coordinate conversion. At our control rate, box decoding fits within action-chunk execution, so the predicted boxes are ready when the next observation arrives.

V-C Local Detail and Geometric Cues

The ablations in Table I show that replacing the wrist view needs two things: a detailed view of the interaction and geometric cues relating the gripper to its target.

Observing the predicted region (Q2)

Foveal crops improve success over full views alone, with both RGB and RGB-D. With RGB-D inputs, the single- and dual-view baselines without crops achieve 6.3% and 41.7% overall, compared with 70.8% and 77.1% with foveal crops, with gains in all four suites. With two RGB views, we can separate the effect of box supervision from that of the crops: \pi_{0.5}-Dual-External reaches 25.0%, box supervision without crops (ECoT-inspired) reaches 31.3%, and FoveaVLA-Dual-RGB reaches 66.7%. Box supervision alone adds 6.3 percentage points, and the crops add a further 35.4.

Combining detail with depth (Q3)

With a single view, neither cue suffices on its own. Foveal crops without depth achieve 20.8% overall success, and depth without crops achieves 6.3%; bringing foveal detail and depth together raises success to 70.8%. Depth also improves the dual-view models, from 25.0% to 41.7% with full views and from 66.7% to 77.1% with foveation.

A complementary viewpoint (Q3)

With foveal crops but no depth, a second external view raises success from 20.8% to 66.7%, approaching the 70.8% of Single with RGB-D. This suggests that a second view supplies part of the geometric information that depth provides: adding depth yields a larger gain with one view than with two (50.0 versus 10.4 percentage points). With RGB-D, the second view raises overall success from 70.8% to 77.1%, with gains on Fold/Unfold and Move, while Single remains best on Distractors (58.3% versus 50.0%).

V-D Zero-Shot Generalization (Q4)

We deploy the FoveaVLA-Dual zero-shot on MolmoSpaces [25], a photorealistic simulation benchmark with a DROID-matched Franka setup and procedurally generated scenes unseen in training. Our evaluation uses two adapted external camera views and simulator-provided depth; Table II lists published \pi_{0.5}-DROID leaderboard scores for reference. Appendix C specifies the observation and evaluation protocols.

Over 8{,}915 trials, FoveaVLA-Dual achieves 22.4% average zero-shot success, versus the 17.8% published reference, with higher scores on seven of nine tasks. The largest differences are on Pick-v1.5 (33.7% versus 18.1%) and Pick&Place-Color-v2 (20.8% versus 10.4%).

V-E Foveation Prediction (Q5)

We evaluate how well FoveaVLA predicts the next interaction region on held-out DROID interaction boxes, alongside PaliGemma [51], \pi_{0.5}-DROID, SAM3 [53], and fine-tuned YOLO11s [54] (Table III). Our model predicts the region at t+w from frame t. PaliGemma, SAM3, and YOLO11s receive the target frame and annotation-derived role information; PaliGemma and SAM3 use role-specific text prompts, while YOLO11s uses the annotated primitive to select the detected roles.

Our future-box predictions achieve 0.465 mIoU and 0.483 [email protected], comparable to YOLO11s at 0.492 and 0.488 despite predicting before the target frame is available. They also exceed \pi_{0.5}-DROID, PaliGemma + PG, and SAM3 + PG. The increase from \pi_{0.5}-DROID’s 0.009 mIoU shows the value of interaction-box supervision. Because FoveaVLA decodes its box during action execution, it adds no wait when the new observation arrives, whereas YOLO11s, which needs the target frame, adds 29 ms.

VI Conclusion

FoveaVLA makes the selection of visual detail part of the robot policy: alongside each action chunk, it predicts the interaction regions that form its next foveal observations. On four real-world task suites, FoveaVLA-Single achieves 70.8% success with one external RGB-D view, versus 52.1% for the wrist-equipped baseline, and FoveaVLA-Dual reaches 77.1%. With a single view, both foveal detail and depth are needed. FoveaVLA-Dual also transfers zero-shot to simulated MolmoSpaces scenes. Because it needs no wrist view, FoveaVLA could also learn from demonstrations recorded without wrist cameras, including human videos.

Acknowledgment

This work was supported by NSF:FRR 2153854, the Stanford Institute for Human-Centered AI (HAI), and Amazon. This work used Marlowe [55], Stanford University’s GPU-based Computational Instrument, supported by Stanford HAI and Stanford Research Computing. We thank Michael Piseno, Haoquan Fang, Chongkai Gao, Zhi (Leo) Wang, and Baifeng Shi for their insightful discussions.

References

1. [1] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al., “Open X-Embodiment: Robotic learning datasets and RT-X models,” in IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 6892–6903.
2. [2] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn, “OpenVLA: An open-source vision-language-action model,” in Proceedings of The 8th Conference on Robot Learning, vol. 270. PMLR, 2025, pp. 2679–2713.
3. [3] K. Black, N. Brown, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al., “\pi_{0}: A vision-language-action flow model for general robot control,” in Proceedings of Robotics: Science and Systems, 2025.
4. [4] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proceedings of Robotics: Science and Systems, 2023.
5. [5] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al., “DROID: A large-scale in-the-wild robot manipulation dataset,” in Proceedings of Robotics: Science and Systems, 2024.
6. [6] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song, “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” in Proceedings of Robotics: Science and Systems, 2024.
7. [7] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn, “Vision-based manipulators need to also see from their hands,” in International Conference on Learning Representations, 2022.
8. [8] S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, et al., “LIBERO-Plus: A progressive robustness benchmark for visual-language-action models,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 38 574–38 583.
9. [9] B. Huang, B. Wei, Y. Cai, and Z. Wang, “Selective cross-view consistency for world action models: Held-out viewpoint robustness without test-time camera information,” arXiv preprint arXiv:2608.21402, 2026.
10. [10] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu, “EgoMimic: Scaling imitation learning via egocentric video,” in IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 13 226–13 233.
11. [11] M. Lepert, J. Fang, and J. Bohg, “Phantom: Training robots without robots using only human videos,” in Proceedings of The 9th Conference on Robot Learning, vol. 305. PMLR, 2025, pp. 4545–4565.
12. [12] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín, “What matters in learning from offline human demonstrations for robot manipulation,” in Proceedings of the 5th Conference on Robot Learning, vol. 164. PMLR, 2022, pp. 1678–1690.
13. [13] Z. Qian, X. Chi, Y. Li, S. Wang, Z. Qin, X. Ju, S. Han, and S. Zhang, “WristWorld: Generating wrist-views via 4D world models for robotic manipulation,” arXiv preprint arXiv:2510.07313, 2025.
14. [14] H. Ding, A. Duan, Z. Sun, D. Song, and Y. Nakamura, “Imagination at inference: Synthesizing in-hand views for robust visuomotor policy inference,” arXiv preprint arXiv:2509.15717, 2025.
15. [15] Z. Liu, Y. Gu, Y. Wang, X. Xue, and Y. Fu, “ActiveVLA: Injecting active perception into vision-language-action models for precise 3D robotic manipulation,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 8141–8151.
16. [16] A. Goyal, V. Blukis, J. Xu, Y. Guo, Y.-W. Chao, and D. Fox, “RVT-2: Learning precise manipulation from few demonstrations,” in Proceedings of Robotics: Science and Systems, 2024.
17. [17] I. Chuang, A. Lee, D. Gao, M.-M. Naddaf-Sh, and I. Soltani, “Active vision might be all you need: Exploring active vision in bimanual robotic manipulation,” in IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 7952–7959.
18. [18] H. Xiong, X. Xu, J. Wu, Y. Hou, J. Bohg, and S. Song, “Vision in action: Learning active perception from human demonstrations,” in Proceedings of The 9th Conference on Robot Learning, vol. 305. PMLR, 2025, pp. 5450–5463.
19. [19] H. Kim, Y. Ohmura, and Y. Kuniyoshi, “Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1630–1637, 2021.
20. [20] I. Chuang, J. Zou, A. Lee, D. Gao, and I. Soltani, “Look, focus, act: Efficient and robust robot learning via human gaze and foveated vision transformers,” arXiv preprint arXiv:2507.15833, 2025.
21. [21] B. Sivak and C. L. MacKenzie, “Integration of visual information and motor output in reaching and grasping: the contributions of peripheral and central vision,” Neuropsychologia, vol. 28, no. 10, pp. 1095–1116, 1990.
22. [22] R. S. Johansson, G. Westling, A. Bäckström, and J. R. Flanagan, “Eye–hand coordination in object manipulation,” The Journal of Neuroscience, vol. 21, no. 17, pp. 6917–6932, 2001.
23. [23] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, et al., “\pi_{0.5}: A vision-language-action model with open-world generalization,” in Proceedings of The 9th Conference on Robot Learning, vol. 305. PMLR, 2025, pp. 17–40.
24. [24] H. Li, Z. Wang, Z.-h. Ding, S. Yang, Y. Chen, Y. Tian, X. Hu, T. Wang, D. Lin, F. Zhao, S. Liu, and J. Pang, “RoboInter: A holistic intermediate representation suite towards robotic manipulation,” in International Conference on Learning Representations, 2026, pp. 113 314–113 381.
25. [25] Y. Kim, W. Pumacay, O. Rayyan, M. Argus, W. Han, E. VanderBilt, J. Salvador, A. Deshpande, R. Hendrix, S. Jauhri, et al., “MolmoSpaces: A large-scale open ecosystem for robot navigation and manipulation,” arXiv preprint arXiv:2602.11337, 2026.
26. [26] Physical Intelligence, “openpi: Open-source models and packages for robotics,” https://github.com/Physical-Intelligence/openpi, 2025, GitHub repository.
27. [27] F. Chaumette and S. Hutchinson, “Visual servo control. I. Basic approaches,” IEEE Robotics & Automation Magazine, vol. 13, no. 4, pp. 82–90, 2006.
28. [28] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “LIBERO: Benchmarking knowledge transfer for lifelong robot learning,” in Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 44 776–44 791.
29. [29] S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu, “RDT-1B: A diffusion foundation model for bimanual manipulation,” in International Conference on Learning Representations, 2025, pp. 29 982–30 009.
30. [30] M. J. Kim, C. Finn, and P. Liang, “Fine-tuning vision-language-action models: Optimizing speed and success,” in Proceedings of Robotics: Science and Systems, 2025.
31. [31] O. Rayyan, J. Abanes, M. Hafez, A. Tzes, and F. J. Abu-Dakka, “MV-UMI: A scalable multi-view interface for cross-embodiment learning,” IEEE Access, vol. 14, pp. 104 545–104 555, 2026.
32. [32] H. Heo, S. Woo, S. M. Kim, J. Kim, J. Lee, Y. Lee, and Y. M. Kim, “AnyCamVLA: Zero-shot camera adaptation for viewpoint robust vision-language-action models,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.
33. [33] J. X. Yang, R. Dagli, A. Zook, H. Hadfield, A. Goyal, S. Birchfield, F. Ramos, and J. Tremblay, “RoboLab: A high-fidelity simulation benchmark for analysis of task generalist policies,” in Proceedings of Robotics: Science and Systems, 2026.
34. [34] H. Freeman, C. H. Kim, and G. Kantor, “WARPED: Wrist-aligned rendering for robot policy learning from egocentric human demonstrations,” arXiv preprint arXiv:2604.10809, 2026.
35. [35] J. Yu, Y. Shentu, D. Wu, P. Abbeel, K. Goldberg, and P. Wu, “EgoMI: Learning active vision and whole-body manipulation from egocentric human demonstrations,” in IEEE International Conference on Robotics and Automation (ICRA), 2026.
36. [36] J. Kerr, K. Hari, E. Weber, C. M. Kim, B. Yi, T. Bonnen, K. Goldberg, and A. Kanazawa, “Eye, Robot: Learning to look to act with a BC-RL perception-action loop,” in Proceedings of The 9th Conference on Robot Learning, vol. 305. PMLR, 2025, pp. 3647–3664.
37. [37] J. Yang, Y. Han, Y. Xie, N. Guo, and W. Lian, “Look, zoom, understand: The robotic eyeball for embodied perception,” arXiv preprint arXiv:2511.15279, 2025.
38. [38] H. Strasburger, I. Rentschler, and M. Jüttner, “Peripheral vision and pattern recognition: A review,” Journal of Vision, vol. 11, no. 5, p. 13, 2011.
39. [39] V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu, “Recurrent models of visual attention,” in Advances in Neural Information Processing Systems, vol. 27, 2014.
40. [40] N. M. Blauch, G. A. Alvarez, and T. Konkle, “FOVI: A biologically-inspired foveated interface for deep vision models,” in Forty-third International Conference on Machine Learning, 2026.
41. [41] B. Chao, L. Yariv, H. Xiao, and G. Wetzstein, “Foveated diffusion: Efficient spatially adaptive image and video generation,” arXiv preprint arXiv:2603.23491, 2026.
42. [42] P. Wu and S. Xie, “V*: Guided visual search as a core mechanism in multimodal LLMs,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 13 084–13 094.
43. [43] Z. Zheng, M. Yang, J. Hong, C. Zhao, G. Xu, L. Yang, C. Shen, and X. Yu, “DeepEyes: Incentivizing ‘thinking with images’ via reinforcement learning,” in International Conference on Learning Representations, 2026, pp. 126 775–126 798.
44. [44] N. Shabtay, M. Kimhi, A. Spector, S. Haray, E. Rivlin, C. Baskin, R. Giryes, and E. Schwartz, “Look where it matters: High-resolution crops retrieval for efficient VLMs,” arXiv preprint arXiv:2603.16932, 2026.
45. [45] J. Min, L. Valkov, V. Petsiuk, H. Souri, and D. D. Mohan, “Foveated reasoning: Stateful, action-based visual focusing for vision-language models,” in European Conference on Computer Vision (ECCV), 2026.
46. [46] J. Cui, Y. Hu, X. Long, L. Li, B. Zhao, and P. Ren, “Look where it matters: Adaptive visual refinement for vision-language-action models,” arXiv preprint arXiv:2608.02197, 2026.
47. [47] H. Xiao, J. Ackermann, B. Deng, and G. Wetzstein, “Policy-based foveated imaging and perception,” in ACM SIGGRAPH Conference Papers. New York, NY, USA: Association for Computing Machinery, 2026, pp. 87:1–87:11.
48. [48] J. Lee, J. Duan, H. Fang, Y. Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y. R. Wang, S. Lee, et al., “MolmoAct: Action reasoning models that can reason in space,” in IEEE International Conference on Robotics and Automation (ICRA), 2026.
49. [49] M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,” in Proceedings of The 8th Conference on Robot Learning, vol. 270. PMLR, 2025, pp. 3157–3181.
50. [50] J. Li, Y. Zhu, Z. Tang, J. Wen, M. Zhu, X. Liu, C. Li, R. Cheng, Y. Peng, Y. Peng, et al., “CoA-VLA: Improving vision-language-action models via visual-text chain-of-affordance,” in IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 9759–9769.
51. [51] L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al., “PaliGemma: A versatile 3B VLM for transfer,” arXiv preprint arXiv:2407.07726, 2024.
52. [52] B. Wen, S. Dewan, and S. Birchfield, “Fast-FoundationStereo: Real-time zero-shot stereo matching,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 7513–7524.
53. [53] N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Surís Coll-Vinent, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al., “SAM 3: Segment anything with concepts,” in International Conference on Learning Representations, 2026, pp. 138 846–138 923.
54. [54] G. Jocher and J. Qiu, “Ultralytics YOLO11,” https://docs.ultralytics.com/models/yolo11/, 2024.
55. [55] C. Kapfer, K. Stine, B. Narasimhan, C. Mentzel, and E. Candès, “Marlowe: Stanford’s GPU-based computational instrument,” 2025. [Online]. Available: https://doi.org/10.5281/zenodo.14751899
56. [56] M. Piseno, G. Tevet, and C. K. Liu, “Cloak: Zero-shot cross-embodiment manipulation by masking the end-effector from the VLA,” arXiv preprint arXiv:2606.22836, 2026.

\useRomanappendicesfalse

Appendix A Dataset Construction

A-A Matching demonstrations and annotations

We construct training examples by joining DROID demonstrations with RoboInter’s frame-level interaction annotations [5, 24]. DROID’s RLDS records provide the robot state, action sequence, and task instruction; the corresponding raw recordings provide external-camera images at 1280\times 720 resolution and stereo recordings for depth estimation. We match the two by episode, camera view, and frame index, so each crop, box target, and robot action come from the same timestep. We use the rectified left imager of each external stereo camera as the RGB reference. The wrist stream is not used to construct FoveaVLA’s observations.

A-B Interaction-box construction

An interaction box encloses the visual components required by the annotated primitive skill. We use three annotation roles: the gripper, the manipulated object or tool, and the target region, which RoboInter labels as a placement proposal. Table IV gives the role mapping used in preprocessing.

We normalize each required component box by image width and height, discarding boxes with invalid coordinates. If \mathcal{S}(s_{t}) is the set of roles required by primitive s_{t} and c_{r,t}^{v} is the box for role r in view v, the target is

b_{t}^{v}=\operatorname{BBox}\!\left(\bigcup_{r\in\mathcal{S}(s_{t})}c_{r,t}^{v}\right),

where \operatorname{BBox} returns the smallest enclosing axis-aligned rectangle. Frames with a missing or invalid required component, or an unsupported primitive, are excluded. Primitive labels are used to construct supervision and are not provided to FoveaVLA at inference.

A-C Temporal pairing and sample validity

A sample at t uses b_{t}^{v} to construct its foveal observation and b_{t+8}^{v} as the prediction target (w=8). Both annotations must be valid, and the future frame must exist in the same demonstration. The future box is constructed using the primitive annotated at the future frame, which allows the target to change across a skill transition. We intersect these eligible samples with the DROID non-idle timestep ranges used by the OpenPI training pipeline [26] and with the availability of aligned depth. Single uses the physical-left external view; Dual additionally requires valid inputs and targets for the physical-right view, paired with the left view at the same timestep.

A-D Foveal crops and augmentation

The prediction target is the tight interaction box; the visual encoder receives a larger square crop around it. Let m be the larger of the box’s pixel width and height. The nominal crop is centered on the box and has side length 1.5m. We adjust its position and size at image boundaries while retaining the interaction box. When a square containing the box cannot fit inside the image, the out-of-image area is padded with zeros. The crop is extracted from the 1280\times 720 source image and then resized to 224\times 224. Full views preserve aspect ratio and are padded to the same encoder resolution.

The prepared training crops use a mixture of 45\% nominal crops, 45\% perturbed crops, and 10\% full views. For a perturbed crop, each center coordinate is shifted uniformly by up to 0.25m, and the nominal side length is multiplied by a log-uniform factor in [1,4/3]. The resulting square is constrained to retain the annotated interaction region. The full views match the initial inference query, before any predicted box is available. The prediction target remains the unperturbed future box.

A-E Aligned metric depth

For each external camera, we recover its rectified stereo pair and camera calibration from the raw DROID recording. Fast-FoundationStereo [52] estimates disparity from each camera’s own stereo pair at 640\times 360 resolution, using eight refinement iterations and a maximum disparity of 192. After accounting for the resize scale, metric optical-axis depth is computed as z=f_{x}B/d, where f_{x} is the horizontal focal length, B is the stereo baseline in meters, and d is disparity in pixels at the corresponding resolution.

We mark non-finite or non-positive disparity, correspondences outside the right image, and depths outside [0.2,3.0] m as invalid. The depth map is aligned to the reference RGB image, and the same padding and stored foveal crop coordinates are applied to depth and its validity mask. Real-robot deployment uses the same disparity conversion and crop geometry.

Appendix B Implementation and Training Details

B-A Model and optimization

Single and Dual are separately post-trained from the \pi_{0.5} base checkpoint using the shared recipe in Table V. Each retains the SigLIP visual encoder, Gemma language backbone, and flow-matching action expert. RGB images and colorized depth maps use the same visual encoder but occupy separate token sequences. Single supplies four image streams (full and foveal RGB and depth), and Dual supplies eight. We update all model parameters; both the flow and foveation losses propagate into the shared backbone.

The learning rate warms up to 5\times 10^{-5} over the first 1{,}000 updates and remains constant thereafter. We use the DROID normalization statistics supplied with the base checkpoint.

The seven arm-joint action coordinates are represented as offsets from the current joint positions during training; the gripper command remains absolute. State and action normalization follow the base policy’s quantile convention, and inverse transformations recover absolute joint-position commands for execution. The physical state and action each have eight dimensions and are padded to the model’s 32-dimensional interface. The normalized robot state is discretized into 256 bins for the language prefix.

B-B Depth representation and visual augmentation

Depth is encoded on a fixed metric scale shared across all frames. For valid depth z, we compute

r(z)=\frac{\log(z/0.2)}{\log(3.0/0.2)}

and apply a Turbo color map to 1-r(z), so near points have warm colors and far points have cool colors. Invalid pixels are black. A 14\times 14 depth patch is masked from attention if fewer than 25\% of its pixels are valid. During training, each depth stream is independently dropped with probability 0.1. RGB receives brightness, contrast, and saturation jitter with magnitudes 0.3, 0.4, and 0.5, respectively. We apply color jitter to RGB only and use no additional geometric augmentation, which would break the alignment between crops, depth, and box coordinates.

B-C Box coordinates and decoding

The predicted box describes the tight interaction region in the full view, before crop expansion. We transform normalized source-image coordinates to the resized and padded 224\times 224 full view, then quantize each coordinate to the nearest of 1{,}024 bins. The sequence order is (y_{\min},x_{\min},y_{\max},x_{\max}), with the physical-left box preceding the physical-right box in Dual. At inference, location-token decoding is greedy. We invert the resize and padding of the full view, clip coordinates to the source-image bounds, and construct the square foveal crop. If a decoded box is invalid, we reuse that view’s last valid box, or the full view if none exists.

B-D Prediction and execution horizons

The action expert predicts H=16 actions per query, and the controller executes the first w=8 before querying again (Sec. IV-B). The flow objective in Eq. (2) supervises the full predicted chunk. Following the base implementation, flow time is sampled as \tau=0.999u+0.001 with u\sim\operatorname{Beta}(1.5,1), and inference integrates the flow from noise to actions with ten steps.

Appendix C MolmoSpaces Evaluation Protocol

C-A Tasks and zero-shot transfer

We evaluate the FoveaVLA-Dual checkpoint without any training on MolmoSpaces data [25]. The nine tasks cover picking, pick-and-place with spatial or color constraints, and opening or closing articulated objects. Task instances, scene initializations, and success definitions follow the official MolmoSpaces benchmarks.

C-B External-camera observations

We render external images at 1280\times 720 so that crops can be extracted at the same source resolution used in training. The first view is fed as the left external camera and the second as the right; full views and predicted foveal crops are resized to 224\times 224. No wrist image is provided.

Table VI summarizes the camera adaptations. For Pick-v1.5, Open-v1, and Close-v1, we retain the original external camera and add a fixed right-shoulder view. For the shoulder-camera tasks, we preserve the recorded left-camera pose and add a right-camera pose with independent position perturbations of up to 5 cm and orientation perturbations of up to 8^{\circ} around the mirrored nominal pose. For Pick-v2-RandCam, the two randomized ZED2-like views are constrained to opposite sides of the robot base with a 5 cm lateral margin.

C-C Depth and action interfaces

The simulation evaluation supplies metric depth directly from the simulator, aligned with each external RGB image; real-robot deployment estimates depth with Fast-FoundationStereo. The policy receives full and foveal depth using the same crop geometry and fixed metric color encoding as in training.

The policy receives the simulated Franka joint positions, gripper state, and task instruction in DROID format; its joint-position and gripper commands run in simulation.

As on the real robot, eight actions are executed per query before new images are captured and cropped with the predicted boxes. The control period is 66 ms. The seven picking and pick-and-place tasks use their benchmark-defined time horizons, while Open-v1 and Close-v1 have a 450-step limit (29.7 s). Episodes stop early on benchmark success.

C-D Scoring and published reference

The evaluation covers 1{,}000 trials for each task except Close-v1, whose source benchmark contains 915 trials. We score episodes with the official MolmoSpaces evaluator, and report the unweighted mean of the nine task success rates.

The \pi_{0.5}-DROID values in Table II come from the official MolmoSpaces leaderboard. Its camera layout, input resolution, and depth availability differ from our evaluation, so these scores serve as a published reference rather than a controlled comparison; the real-world ablations (Sec. V-C) isolate the effects of foveation, depth, and view count.

Appendix D Real-World Benchmark Details

D-A Task instances and execution

The real-world benchmark contains four task suites with 12 instances each. An instance specifies the objects, their initial placement, and a language instruction; the same instances are used across methods. Our tasks adapt the tabletop tasks of Cloak [56], with changes to the object set and towel setups and a Pick-and-Place with Distractors suite in place of Cloak’s Remove task. Tables VII–X list all 48 instances. Figure 6 shows representative initial and completed states from all four suites. Figure 7 illustrates the full views and foveal crops during a successful distractor trial.

We use the DROID robot setup described in Sec. V-A. Each trial runs at a target 15 Hz for at most 900 control steps and can be stopped earlier to avoid robot damage. The object lists and positions are used only to set up the scene and are not given to the policy. Positions are named tabletop slots: FL, FR, BL, and BR denote front-left, front-right, back-left, and back-right, respectively, and FM and BM denote the middle position in the front and back rows.

D-B Task definitions and success criteria

Pick-and-Place

The policy must move the specified source object into the specified container. The instances vary the source among markers, cubes, and towels and the container among bowls, a box, and a large pot. Success requires grasping the specified source and releasing it inside the correct container.

Move

The policy must move the source object next to a reference object. Success requires grasping the source and releasing it within one source-object width of the reference without moving the reference. Instances use cubes, cups, and towels as sources and cups, bowls, and a large pot as references.

Fold/Unfold

The 12 trials comprise four groups of three: folding one towel, folding a target towel with a distractor present, unfolding one towel, and unfolding a target towel with a distractor present. Each group targets blue, grey, and pink towels once each. For folding, every towel in the scene starts flat; for unfolding, every towel starts with one half-fold. Thus, in the two-towel scenes, initial fold state does not reveal the target. The instruction specifies the target color. Folding requires one half-fold with opposite edges aligned; unfolding requires a flat towel with no folded-over section. Success requires completing the requested manipulation on the specified towel and releasing it.

Pick-and-Place with Distractors

The four scene compositions are two markers with one bowl, two markers with two bowls, three markers with two bowls, and three markers with three bowls, with three trials per composition. Markers occupy the back row and empty bowls the front row. A two-object row uses its left and right positions; a three-object row additionally uses the middle position, which is occupied by a distractor. A single bowl is placed at front-middle. The instruction names both the marker and bowl colors, and each group uses all three marker colors and all three bowl colors once as targets. Success requires placing the instructed marker inside the instructed bowl.

D-C Outcome annotation

An operator assigns binary success after each attempt according to the task-specific criteria above. In scenes with distractors, disturbing a distractor is allowed as long as the instructed manipulation is completed. Per-suite success is averaged over its 12 trials, and Overall pools all 48 trials, with SEM reported as in the main text.

D-D Complete trial specifications

The tables list the exact instructions given to the policy, which sometimes use alternative words such as “pen” for a marker and “cloth” for a towel. Within the Fold/Unfold and Distractors suites, only the objects listed for an instance are placed on the table. The Fold/Unfold initial-state rules above apply to both the target and any distractor towel.