EyeRobot 2.0、単一固定ステレオカメラの能動注視で手首カメラ不要の精密バイマニュアル操作を実現

Abstract

Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human’s fixation sequence while performing the task (Fig. 1). EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper proprioception and actions into a rotating fixation-relative \text{SE}(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52\% to 27\%. EyeRobot 2.0 closes this gap without any wrist cameras, outperforming passive stereo by 40\% in real and 20\% in sim. It matches ego + wrist policies when their wrist views are clear (69\% vs. 64\%), and more than doubles their success when grasped objects occlude the wrist cameras (48\% vs. 22\%). These results suggest active gaze as a single-camera alternative to wrist cameras for fine-grained manipulation.

1 Introduction

Consider the task of picking up a cup with a lid, grasping a straw, and inserting the straw into a small opening in the lid (Fig. 1). As humans carry out this task, our eyes flit around the scene, filtering information with coordinated look-ahead movements that anticipate hand motion [1]. This task-relevant visual sampling, called fixation, directs high-resolution foveal vision to relevant parts of the visual field. Fixation also provides 3D anchor points for manipulation, simplifying the challenge of spatially reasoning about the scene [2] and implicitly breaking larger tasks into concrete sub-goals as gaze switches between targets.

We hypothesize that this behavior could also benefit robot manipulation. Fixating on a dynamic task-relevant 3D point acts as a form of physical attention which centers input observations, compacts the data distribution, filters irrelevant visual regions, and allocates additional resolution where it matters for fine-grained tasks without the cost of processing the full panorama at high resolution. Some of these benefits can be offered by wrist-mounted cameras, which are used by nearly all existing bimanual manipulation systems [3, 4, 5, 6]. However, colocating vision with the gripper ties precise visual input to the manipulator, which is limiting in tool use, whole-body manipulation, or when grasped objects occlude the camera. In addition, these systems constrain manipulator design because of their direct mounting to the gripper, which adds complexity, often occludes the ego camera, introduces motion blur, and prevents sleeker gripper designs. A purely ego-centric behavior cloning approach has the potential to ease constraints on end effector design, make better use of ego-centric human data, and improve tasks which benefit from vision-manipulator decoupling like fast motion, reaching into tight spaces, or searching large spaces [7].

We present EyeRobot 2.0, a robot learning framework which introduces Active Visual Fixation (AVF). Using input only from one fixed stereo camera, EyeRobot 2.0 can carry out precise manipulation tasks like inserting a straw, capping a marker, zipping a bag, or extracting a wrench from a toolkit, and multi-stage tasks like opening a toaster oven, pulling the tray out, extracting a pan, pushing the tray back, and closing the door again.

EyeRobot 2.0 couples a fixation policy with a fixation-centric gripper policy. The fixation policy has two modules: a low-level gaze policy that coordinates two eye viewpoints to rapidly find and track 3D objects in the scene, and a high-level target selector to choose what object to look at and when to look at it during the task. Visual inputs are represented as foveated stereo multi-crops centered at the fixation point, which allocate more resolution towards the direction of interest. This physical attention not only increases visual acuity, but also reduces computation allocated to distractors. In addition, EyeRobot 2.0 trains a fixation-centric gripper policy via behavior cloning, by representing SE(3) proprioception and action chunks canonicalized into the rotated fixation frame. This effectively constrains the input and output space for the gripper model with respect to task-relevant local regions. The gripper policy additionally receives the distance of the 3D fixation point as an observation, which acts as an anchor point for the policy to understand depth. Both the gaze policy and target selector are trained with reinforcement learning (RL), with gaze emerging from a BC-RL loop as in Kerr et al. [7].

We evaluate EyeRobot 2.0 on 7 physical tasks and 6 simulated MuJoCo tasks, collecting data and training single-task policies from scratch. Across over 1000 physical trials, it outperforms static stereo baselines by 40%, and by 20% on simulated pick-place tasks, with a larger performance improvement on fine-precision tasks. Compared to the widely adopted SOTA ego + wrist setup, EyeRobot 2.0 without wrist cameras matches task performance in physical trials, and on tasks with heavy occlusion of wrist cameras maintains strong performance while wrist baselines regress to static stereo performance. In ablations we find that fixation-centric actions account for a significant portion of EyeRobot 2.0’s improvement, causing a 20% performance boost across sim tasks, and foveation contributes strongly to real-world success on fine-grained tasks, boosting by 20% overall.

2 Related Work

2.1 Learning manipulation from human demonstrations

Behavior Cloning (BC) allows robots to learn complex behaviors from teleoperated demonstrations without requiring handcrafted motion primitives and reward functions [8, 9, 10, 11]. BC has been extended to work in complex settings including mobile manipulation [12, 6] and bimanual manipulation [3, 4], and several recent efforts have been directed towards scaling teleoperation datasets [13, 14, 5, 15, 16]. Another enticing direction for scaling is learning from ego-centric human data, either by instrumenting humans or scraping internet videos [17, 18, 19, 20, 21]. However, learning from human videos presents a significant domain gap between ego-centric sensors in first-person datasets and the wrist cameras that our robot policies so frequently rely on for fine-grained performance. EyeRobot 2.0 aims to provide a route towards lessening this gap by leveraging human-like fixation for precise bimanual manipulation. We are also inspired by task conditioning approaches in BC which condition a low-level policy on the output of a higher-level task sequencer [22, 23, 24, 25]. Similarly, EyeRobot 2.0 learns a target selector to control gaze, but does not require human gaze supervision and instead trains gaze with RL via a BC accuracy reward.

2.2 Active vision for manipulation

Active vision deliberately moves sensors to improve observability, filter information, and reduce occlusions for better environment understanding [26, 27, 28, 29, 30, 31, 32, 33]. Previous systems have used active vision to help with tasks such as visual search [34, 35, 36, 37, 38], object detection and tracking [39, 40, 41, 42], and 3D reconstruction [43, 44, 45, 46]. Similarly, active vision has proven beneficial for robot manipulation with a “robot neck” moving the camera based on VR headset movement during task teleoperation to reveal better viewpoints [47, 48, 49, 50, 51, 52]. Active visual search can also benefit from training on ego-centric human data [53, 54, 55]. EyeRobot 2.0 in contrast does not attempt to address viewpoint selection which has been explored in prior works, but rather investigates how coordinated fine-grained eye movements can boost performance even with a well-positioned viewpoint.

Related to learning more precise gaze, some papers collect human eye tracking data during teleoperation to predict a gaze point as an intermediate output to condition a gripper policy with [56, 57] or crop a high-resolution finger-tip image [58]. Others [59] learn gaze in simulation through on-policy task-based rewards; however, these methods struggle with sim-to-real transfer. EyeRobot [7] overcomes the need for gaze demonstrations by training a gaze policy on top of real-world behavior cloning data from 360^{\circ} video by co-training the eye and arm policies with an action feedback loop for reward. Building on this, EyeRobot 2.0 uses a similar co-training loop but tackles much more fine-grained manipulation tasks by utilizing active stereo fixation, a fixation-centric action representation, and a hierarchical target selector to precisely coordinate gaze throughout execution.

3 Manipulation with Active Visual Fixation

Problem Statement: We study visuomotor policy learning from human teleoperation demonstrations for bimanual manipulation with parallel-jaw grippers. Given a set of demonstrations specifying a task, our goal is to train task-specific policies that can reproduce the demonstrated behavior with bounded variations in pose. In particular, EyeRobot 2.0 considers an ego-centric setting in which observations are provided solely through an approximately head-mounted stereo camera. The policy is evaluated using task-specific success criteria.

Overview: AVF trains separate gripper and fixation policies that work together to achieve a task. The two eyes’ viewpoints actively pivot to converge on a shared 3D fixation point, whose location is updated at each timestep by the fixation policy. The resulting fixation-centered observations are fed into the downstream gripper policy, which produces action chunks for robot execution. These SE(3) action chunks are canonicalized into a dynamically shifting fixation frame pointing towards the current fixation point (Fig. 4). Each policy uses a foveated transformer-decoder architecture similar to [7], which we fully describe in the Appendix.

3.1 Learning Gaze Patterns with RL

We train AVF in two phases. First, we train a goal-conditioned fixation policy capable of fixating on any task-relevant objects. Next, we train a hierarchical target selection policy that sequences fixations to optimize task performance. In the first phase, the fixation policy is trained with RL using a geometric reward from detected object positions, enabling accurate 3D tracking of prompted objects. In the second phase, the target selector is trained with RL in a BC-RL loop as in EyeRobot [7]: a gripper BC policy is co-trained on the resulting fixated views, and the selector is rewarded for fixations that maximize the BC policy’s action prediction accuracy. This hierarchical formulation allows task-dependent gaze behavior to naturally arise without the need for manual human specification.

To learn gaze, we need a way to counterfactually render it on top of existing robot demonstrations that were collected before any gaze policy exists. We therefore need to render what the robot would have seen had it looked somewhere. EyeRobot [7] addressed this by collecting demonstrations with a 360^{\circ} camera and rendering novel views from it. We transfer this idea to stereo, using a calibrated stereo camera to synthesize fixation: rotating a camera about its optical center only requires warping its image, without any 3D reconstruction. This lets AVF use the same stereo camera for data collection, training, and deployment. We simulate fixation by rendering viewpoints from a stereo camera to replicate the appearance of two eyes converging their gaze on a common 3D point in the scene.

Goal-Conditioned Stereo Fixation

First, we train a standalone goal-conditioned fixation policy. This policy takes as input a language prompt specifying the goal, the current fixated observations, and the current fixation point in polar coordinates. Though in principle this language prompt could be anything, in practice we limit it to a task-specific annotated set of target objects which the robot interacts with in the scene. The gaze policy outputs local fixation deltas for azimuth and elevation \Delta\theta,\Delta\phi, and fixation distance \Delta d represented as a discretized distribution.

We train this policy with PPO [60] on rollouts from randomly selected stereo still-frames from our teleop demonstration set. To compute a goal-conditioned reward r for each rollout, we first detect candidate objects by querying SAM3 [61] with the task’s language prompts, then randomly select one detected prompt. To localize the object in 3D, we extract depth from the calibrated stereo pair with FoundationStereo [62] and set the ground truth position to be the 3D centroid of depth intersected with the object mask, p_{gt}. At each step of RL, the policy is conditioned on the current prompt and rewarded based on the current fixation point p_{cur} with r=-||p_{cur}-p_{gt}||_{2}. The result is a lightweight policy that can rapidly fixate on all task-relevant objects at 30 Hz.

Learning to Sequence Fixations via a BC-RL Loop

While the first phase enables low-level gaze behavior capable of fixating on any object, we now address the high-level decision of where and when to focus. As in EyeRobot [7], to circumvent the burden of hand-annotated task stages, we learn this decision process via RL by jointly training gaze alongside the BC policy itself. This naturally encourages gaze sequences that optimize task performance. To do this, we freeze the goal-conditioned fixation policy, and train a lightweight target selector with RL, which is conditioned on robot proprioception and outputs a categorical distribution over per-task prompts (Fig. 4, top left). The resulting sampled prompt is used to condition the fixation policy, whose behavior is used during PPO rollout as observations to compute gripper action chunk predictions, which are then used to compute reward for the target selector to optimize (Fig. 4, middle). During training, the target selector emits a new target once every 15 frames to allow gaze to settle before sampling a new target.

During train rollouts we uniformly sample a start frame from the demonstration, with special 10% weighting given to the first frame in the demo which often contains the most critical decision for gaze (what to reach for first). The first 30 frames of the agent’s observations operate on a paused frame to allow gaze to settle before being forced to act. We adopt a similar per-step reward as [7], which extracts a task performance proxy from the BC policy’s prediction error. Given predicted and ground truth bimanual action chunks transformed into the same frame A_{pred},~A_{gt}, we first identify an “active arm” whose end effector moves a greater distance in A_{gt}, then compute the arc-length Fréchet distance between A_{gt} and A_{pred} for this arm. The negative of this distance is used as reward to optimize the target selector, encouraging fixation which aids BC prediction accuracy. Limiting reward computation to the active arm avoids sensitivity to spurious correlations in bimanual demonstrations, and Fréchet distance provides a more stable velocity-invariant metric than L2 error.

3.2 Fixation-Centric Action Chunking

Knowing where to focus not only reduces perceptual complexity but also improves action prediction. Specifically, we represent the robot’s proprioception and action chunks relative to the fixation frame. Such a representation standardizes the input about the fixation direction, making the training distribution more compact. Model input proprioception and actions are represented as SE(3) end effector poses transformed into the fixation frame, defined as the cyclopean (virtual center-eye) rotation towards the fixation point in 3D. This canonicalized action representation moves dynamically with fixation, which results in a significant performance improvement over a fixed world frame. We jointly train the BC policy alongside gaze post-training on perspectives sampled from the fixations induced by RL policy rollout. We maintain a ring buffer of the past 100 rollouts seen, and sample BC train batches independently from this buffer to maintain diversity across demonstrations. The BC architecture is a transformer decoder resembling that of [7]; see the Appendix for architecture and hyperparameter details.

4 Experiments

EyeRobot 2.0 is implemented using a bimanual, 14-DoF I2RT YAM manipulator with parallel-jaw grippers. We mount a static stereo camera between the arms 35 cm above the base joint, angled 45∘ downwards for full observability of the table. Each eye captures a 120∘ diagonal FOV at 1600\times 1200 resolution. We use a custom leader arm modified from GELLO [63] and TRLC [64] to teleop the robot.

Simulation Environment

We build a simulation environment in MuJoCo [65] to conduct repeatable, large-volume experiments. Note that our objective is not sim-to-real transfer, nor do we propose to use physics simulation to learn eye gaze, but rather we strive for a digital twin with which to conduct careful method comparisons using the same training pipeline as in real. We replicate the dimensions of the YAM manipulators, and use the public calibrated PD and mass-inertial constants in MJ-Playground [66]. We implement non-pinhole camera rendering in MuJoCo by rendering pinhole images and warping them to exactly match the camera characteristics of each camera. We collect teleoperation with GELLOs through a custom VR interface which streams visuals to the operator in a Quest 3 headset, similar to prior VR teleop systems. All simulation code will be made public.

Tasks and Data

In total we collected teleoperation data for 7 physical tasks and 6 simulated tasks as shown in Figure 5. We focus on 5 multi-stage real tasks which involve bimanual coordination (e.g. object handover, coordinated grasping), fine-grained tolerance requirements (e.g. marker capping, zipping, boba straw insertion), and long-horizon behavior (e.g. opening a toaster oven, taking out a pan, pushing the shelf back in, closing the door). Due to the challenges of simulating contact-rich physical interactions in real-time, our simulated tasks aim to evaluate a range of basic pick and place behavior. We collect between 10 and 53 minutes of data per task. Full task definitions, stage visualizations, train distribution visualizations, and dataset size details are provided in the Appendix.

5 Results

We evaluate how well EyeRobot 2.0 performs compared to SOTA baselines. To construct carefully controlled comparisons, we choose to collect all data ourselves and train all policies from scratch on the exact same demonstrations. This avoids obfuscation from unknown pretraining distributions of off-the-shelf models, reduces compute demand, and increases the tractable volume of experiments by an order of magnitude.

In real, we train every policy with 3 seeds and randomly select one for evaluation. For each task, we evaluate all policies on 25 scene configurations selected by placing objects within the training data region, but not exactly matching train datapoints. During evaluation, we overlay the live camera stream on the test positions and manually align objects so all policies share the exact same test positions. See the Appendix for a visualization of these test distributions. Across all real-world physical trials of baselines and ablations, we evaluate over 1000 policy rollouts.

In sim, every baseline and EyeRobot 2.0 policy is trained across 3 seeds, of which we report the maximum performance to prune failed seeds. See Appendix for full statistics. All sim evaluations are run on the same 100 random configurations, all unseen during training but drawn from the same distribution. Each policy is trained from scratch on single-task data, and executed using the same temporal ensembling weights and inference code. The gaze policy is given 1 second to allow gaze to settle before acting. Per-stage success is programmatically judged using manually specified MuJoCo physics requirements, and each task has a failure timeout based on the average length of demonstrations.

Baselines

We collect all our data with stereo ego and wrist cameras, meaning EyeRobot 2.0 and baselines share the same training data. No Gaze takes in the raw stereo input stream, which has the exact same observability but without fixation, and Ego + Wrist mimics the prevailing bimanual setup using two wrist cameras and the left image from the stereo pair. All baselines are trained with ACT [3] and share the same architecture and training hyperparameters as EyeRobot 2.0. We augment baseline input images with 95% center-crop, and 10% translation and rotation randomization to make them less sensitive to placement error, as in [5]. To select the best action space, we test baselines with end-effector (EE) relative SE(3), world-absolute SE(3), joint chunk-relative, and joint chunk-absolute. Since EE-relative SE(3) performs the best in sim (see Appendix), we use that for real evaluations. Baselines are trained with a chunk size of 1 second as suggested in ACT [3] and EyeRobot 2.0 is trained with 1.5 seconds. We found a slight performance gain from 1.5s chunks with fixation but no such gain for baselines when swept in simulated tasks.

How does EyeRobot 2.0 compare to gaze-free stereo?

This baseline corresponds to training standard policies with static ego-only stereo inputs without wrist cameras. EyeRobot 2.0 outperforms static stereo policies significantly across all 13 tasks, on average by +20\% in sim and +40\% in real (Fig. 6). Ego-only policies have a strong tendency to overfit to the training positions, an effect which can particularly be seen in the tape_handover task, as data was collected in a regular 8\times 8 grid, or the tea task which requires servoing around the entire workspace. In addition, ego policies lack the visual acuity to competently accomplish precise tasks like marker insertion (4%) and straw insertion (8%). Despite having access to the exact same stereo stream, by foveating on relevant objects EyeRobot 2.0 achieves 68% and 44% end-to-end success on these two tasks. The failure modes of static stereo policies tend to cluster around task stages that require precision (see Sankey diagrams in Appendix), as well as significantly more overfitting behavior in tasks with wide spatial distribution (tape handover).

How does EyeRobot 2.0 compare against SOTA ego + wrist policies?

On precise physical tasks where the wrist camera maintains full visibility (marker, tea, toaster, wrench), both perform similarly (Fig. 6). In tasks where grasped objects occlude the wrist cameras (boba, pot), ego+wrist performance collapses to near the gaze-free stereo baseline, while EyeRobot 2.0 maintains strong performance, overall a 2\times improvement (Fig. 7). In the tape_handover task, wrist camera policies tend to overfit to the grid training distribution similarly to ego-only policies because the wrist camera on the gripper holding the yellow tape is occluded until just before placement, by which point the tape is already in an unrecoverable position.

Does fixation improve robustness to visual distractors?

We evaluate all policies in real on tea, wrench, and tape with colorful distractor objects randomly scattered on the table and report their end-to-end success along with first stage success in Table 1. All comparisons use the exact same object positions to ensure comparability. The wrist camera policy often reaches for the largest or most distinct object in its view. Once it begins reaching, the policy cannot recover since the wrist camera loses sight of the target. In contrast, EyeRobot 2.0 reaches for the correct object when its gaze is correctly directed at it, and mostly misses grasps when it is looking in the wrong place. This shows most clearly in first-grasp success, which is significantly higher with active gaze. Decoupling vision and action could improve robustness in cluttered scenes by literally ignoring irrelevant information.

Gaze analysis

See Fig. 2 for a visualization of target selector probabilities over the full course of the marker task. EyeRobot 2.0 first fixates on the marker cap, then switches to the marker just before picking it up and keeps its gaze there throughout capping. Overall, it learns intuitive gaze switching, for example looking at an object until just after it has been grasped. Interestingly, it sometimes uses static objects as a sort of anchor during manipulation, for example during spans of the marker task it gazes at the pencil pouch which during data collection was always in a roughly straight-ahead position.

5.1 Ablations

No foveated image architecture

We ablate the multi-scale image pyramid and train both gaze and gripper policies on one peripheral view for each eye (Tab. 2). Overall this harms performance by 20%, with the largest gap on precise physical tasks which require detailed image understanding. Interestingly, in simulation this ablation performs the same as EyeRobot 2.0. This may be because robot execution in simulation has little to no variance, whereas real-world execution varies from trial to trial and so demands more closed-loop visual servoing.

Fixation-centric actions

We test an ablation of EyeRobot 2.0 in simulation which uses SE(3) actions and proprioception in the world frame rather than a fixation-relative frame. This ablation performs on par with the gaze-free ego baseline (-21\%). Including gaze while still allowing a global output space for the policy seems to still afford overfitting to specific trajectory positions in this low-data regime.

Stereo vs. Mono

We train an ablation which only has one input image and no input fixation depth. This ablation performs on average 18\% worse than EyeRobot 2.0 across all real-world tasks, indicating that the model benefits from access to binocular information.

6 Limitations and Future Work

EyeRobot 2.0 currently only trains task-specific gaze and gripper policies. An interesting direction for future extension is to investigate how to adapt the framework to maintain multi-task performance. The usage of a target selector means that gaze would need to be trained on a sufficiently large corpus to allow generic prompts, which may necessitate different architectures for generating prompts such as a vision-language model. Separately, in this work we do not control head or neck movement, and only propose how to coordinate fine-grained eye movements provided a good head position is already available. Finally, while object-level pretraining is sufficient for many manipulation tasks, it cannot yet train sub-object-level gaze which may provide more benefits for manipulation.

7 Conclusion

This work presents EyeRobot 2.0, a robot learning framework for fine-grained bimanual manipulation using only a fixed stereo camera that leverages active visual fixation to direct attention towards task-relevant features. Our experiments suggest that active fixation more effectively leverages stereo data than non-active stereo baselines, strongly outperforming them across a range of high-precision physical tasks and matching or exceeding the performance of wrist cameras. Ablating any of its components results in a performance drop, indicating it utilizes multi-resolution image inputs, fixation-centric actions, and stereo information to effectively facilitate fine-grained real-world manipulation. This suggests exciting potential for systems which use active visual fixation for fine-grained manipulation instead of wrist cameras.

Appendix

Appendix A Additional Results

Failure modes analysis

Please see the included Sankey diagrams (Figs. 9, 10, 11) for a visualization of per-stage success breakdown for AVF and baselines across all physical trials. The primary failure cases align with the most difficult, precision-requiring stages of each task, for example inserting the boba straw or capping the marker. In comparison, the ego-only baseline struggles more with these precise stages, and also has considerably more failures in “easy” portions of each task, for example picking up objects initially. With only a fixed input view the policy struggles to achieve the precision required to succeed, and even struggles more at basic picking, for example in the tea task only grasping the small teabag in 32% of trials, while AVF always grasped the teabag. The ego+wrist baseline recovers this grasping precision, but fails severely on task stages (pot on lid, boba insertion) where wrists become occluded by the grasped object. Even in tasks with unoccluded views from the wrist camera, wrist cameras have similar conditional success rates to AVF, indicating that failures stem from a common modeling or data insufficiency which wrist cameras provide no advantage for.

How well do policy rankings in sim and real correlate?

To measure sim-real correlation we collect 2 similar tasks (pot and tape) in sim and in real. While the absolute success rate in sim is not predictive of real-world success, policy rankings are maintained on these tasks between all baselines, and results indicate that sim may favor wrist cameras over physical trials, perhaps because EE-relative chunks are more sensitive to compounding action errors with imperfect real physics than simulated. Pot real-world success is notably lower overall because simulation has more forgiving physics for pot lid closure.

What fixation patterns does AVF learn?

Probabilities while inserting the marker into the bag are more ambiguous as all objects are in the same location. Quantitatively, we analyze per-stage gaze in simulation and report full task results in the Appendix. During the first task stage, 100% of policies learned to look at the first object the robot grasps. In tape_handover, dish_rack, and tray the gaze switches to the second object, while in the pot and spoon tasks it remains on the first object.

How accurately can AVF fixate in 3D?

Overall, the average fixation error after gaze pre-training is 3.5 cm computed over 800 random frames from all trajectories. Fixation quality is best seen in our included policy rollout videos, which visualize real-time fixation from the policy during execution. Despite being trained only on still frames, the policy can reliably track moving objects during robot execution.

Appendix B Model Architecture and Training Details

Both the fixation and gripper policies use a transformer architecture which decodes actions from input image, proprioception, and goal prompt observations. The input proprioception includes the current fixation point encoded as a unit vector direction and distance. Image tokens are extracted using a frozen DINOv3 [67] ViT-S/16 backbone in a foveated manner: a pyramid of centered crops is independently passed through the encoder and concatenated along the sequence dimension, allowing the model to attend to tokens from multiple resolutions as it desires.

Each input token is given a RoPE [68] (x,y,t) position. For image tokens, x,y corresponds to the location of the token within the global image, and all other tokens are given a value of the image center (\frac{h}{2},\frac{w}{2}). Each token is assigned t corresponding to the current observation timestep, except action chunks which are assigned t consistent with their future time. We use a \theta value of 1,000 for sinusoidal frequencies, lower than the default 10,000 for RoPE to adjust for the shorter sequence length. The BC gripper policy takes in history-free single-frame observations, while the gaze policy is allowed to attend to the gaze direction of the previous 3 frames, which stabilizes gaze.

The transformer uses bi-directional attention between observation and query tokens, to improve compute efficiency over full self-attention. Each input and output modality is projected to the same token stream and decoded into outputs with a 3-layer MLP. The fixation policy is an actor-critic agent which outputs a categorical distribution over actions alongside a continuous value function estimate v. The target selector is a 3-layer, 256-wide MLP with GELU [69] activations which takes in proprioceptive input and outputs a categorical distribution over target prompts along with a critic estimate. We apply 10% dropout to the hidden layers of the MLP to prevent overfitting to proprioceptive data, along with 10% proprioceptive token drop during BC batch sampling. The attention and FFN hidden layers of the transformer MLPs also have 10% dropout. Each transformer uses 6 transformer blocks, with any learnable query tokens initialized as standard normal embedding vectors with a standard deviation of 0.02 in each dimension. The token dimension of the transformer is 384, and it has an MLP expansion ratio of 4.0, with 12 attention heads. Of each head’s 128 dimensions, 32 are used for temporal RoPE, 48 for x, and 48 for y.

B.1 Hyperparameters

All training code will be released for reproducibility, but we report selected important hyperparameters here.

General RL parameters

We utilize the standard PPO update rules with value bootstrapping and discount factor \gamma=0.95. We utilize generalized advantage estimation [70] with \lambda=0.95. We do on-policy RL updates for 10 epochs, early-terminating if KL-divergence on the discrete policy output reaches more than 0.02.

The gaze policy discretizes its action space and outputs a cross-product categorical distribution. Azimuth and elevation are binned into 8 polar directions, with 2 magnitudes for each direction (10∘ and 3∘) along with a zero action. Depth deltas are binned with 3 magnitudes (10 cm, 5 cm, 1 cm) and a zero action. The resulting cross-producted action space has 119 bins for the policy output. We prefer discretized action spaces as they help break ties in ambiguous actions (i.e., duplicate objects).

Gaze pre-training

Each policy rollout consists of 120 timesteps, all over a paused stereo frame. The policy is optimized with the AdamW [71] optimizer, with a learning rate of 10^{-4}, \beta_{1}=0.9,\beta_{2}=0.95, weight decay of 0.1, with a linear ramp over the first roughly 5% of train steps and cosine decay over the last 30\%. Each rollout uses 64 parallel environments, with a minibatch size of 16 environment rollouts. Policies are optimized for 15M total rollout steps, which corresponds to 1,950 iterations, or 125,000 unique rollouts total. We use an entropy regularization term of 0.01 during training, which decays to 0.005 alongside cosine decay.

Gaze post-training and behavior cloning

During BC-RL training, the behavior cloning policy is completely isolated from RL gradients by having its own set of projection and decoder MLPs. Both the target selector and BC policy are optimized with Muon [72], with the learning rate for the BC policy set to 5\times 10^{-4} and the target selector 10^{-4}. The target selector entropy is regularized with a SAC-style entropy regularizer [73] which tunes the regularization to reach a target average entropy of 70\% of the theoretical maximum policy entropy. During post-training, we maintain a buffer of the past 100 episodes of experience, from which we IID sample minibatches of 256 frames for the BC policy to use for SGD. We perform 4 minibatches of BC optimization for each update of the rolling experience buffer. During post-training we roll out 64 parallel environments for 30 steps with paused demonstration time, and 170 steps with time playing forward. During post-training, the video is played at 2x speed (skipping every other frame) so that each episode provides more coverage of the demonstration. In total, we roll out 10,000,000 environment steps, which corresponds to 50,000 unique episode rollouts, 781 iterations, or 3,124 BC gradient steps.

Training details

Each train run uses 8 NVIDIA L40S GPUs, training with data-parallel process coordination with environments evenly split among GPUs. All reported batch sizes are for the total number, not per-GPU. Training is done in bfloat16 mixed precision, except RoPE which is computed at float32 resolution.

Baseline details

To decide the action space to use for all baselines, we considered the classically known options of joint-space and end-effector SE(3) space, both chunk-relative and chunk-absolute. While community wisdom has recently drifted towards increasingly using chunk-relative representations, we wanted to ensure that in our data-frugal scenario we thoroughly calibrated the performance of these. Testing such a large grid-search is infeasible in physical trials, so we ran a subset of our simulated tasks with all action spaces (Fig. 12). Across the board, we found that end-effector SE(3) relative to the first EE frame in the chunk (“EE-relative”) performed the best, so we use this action space for all baseline experiments. To doubly verify this, we ran full baseline evaluations with world-relative SE(3) actions on the wrench and toaster tasks, and observed no difference in performance. Baselines are trained at 256 image resolution.

Appendix C Robot inference

Action chunks are predicted as 30 Hz open-loop trajectories, which we cubically interpolate at 200 Hz to obtain the commanded robot positions. We apply temporal ensembling during inference with a k value of 0.01. We additionally compute the temporally-ensembled velocity across predicted action chunks and provide this as a feedforward term for the YAM follower arms, which increases fluidity of the robot movements. See included videos for unedited videos of robot execution.

All models are queried as quickly as compute will allow, with end-to-end throughput of around 40 Hz for baselines and 20 Hz for AVF policies.

Appendix D Data Details

For stage-by-stage visualizations of all task stages, see Figs. 13 and 14. Figure 15 shows the exact training distribution for all sim and real tasks, as calculated by detecting each object’s starting position in 3D and plotting them projected onto the tabletop surface. As described in the Results section, notice how the tape task is collected in a grid, while other tasks are randomly initialized within a certain region for each object. Baseline policies are surprisingly sensitive to this positional distribution, with a large gap in performance between AVF and wrist policies on this grid-collected task. This gap is lessened by collecting randomly distributed initial states.

See Tab. 4 for dataset size statistics for all tasks.

Appendix E Simulation Details

We use MuJoCo’s Newton solver with an elliptic friction cone, implicit-fast integration at a 0.5 ms timestep, and 10 noslip iterations to ensure stable contact-rich grasping behavior in simulation. Collision meshes are generated with a convex decomposition from CoACD [74], and we use the default YAM collision meshes and physical properties from MuJoCo Playground [66]. Simulation environments have hand-coded initial object distributions and physical success measures, and the same parameters are used across all models. The same seed is used for environment initializations to ensure direct comparability.

Appendix F Evaluation Details

F.1 Test Distribution

See Fig. 16 for a visualization of the test distributions that we use for policy evaluation. All policies are evaluated on the exact same positions, within human placement error, by aligning object positions with an overlay of the current position vs test position.

F.2 Distractor objects

We follow the same procedure for distractor experiments as full evals, by recording 25 positions of test objects and distractor objects and exactly repeating them for all models. See Fig. 17 for a random subsample of initial positions of distractor objects in the wrench task.

F.3 Per-stage success criteria for real tasks

For all stages, we allow 3 retries before counting a stage as a failure, and count a failure if the robot stops moving for more than 10 seconds.

Tape

Pick yellow: the robot must grasp the yellow tape and lift it fully off the table without dropping it before the next stage. Handover: the robot must successfully transfer the yellow tape to the right gripper without dropping it. Place: the robot must place the yellow tape on top of the grey tape without it falling off or tilting. The static position must be stable with the center of gravity fully supported by the grey tape roll.

Pot

Pick lid: the robot must fully grasp and lift the pot lid off the table without dropping it. Place lid: the robot must place the pot lid on the pot with no gaps visible. This success criterion is quite strict, as simply resting the lid on top of the pot is insufficient. The lid must fully seal the rim of the pot, which is partially why the real success of this task is significantly lower than the simulated version of the task.

Toaster

Open door: the robot must fully open the door to expose the internal oven. Pull rack: the robot must pull out the internal wire rack from the toaster oven, and success is only marked if the rack is fully extended to reveal the pan inside, and the tray is not over-extended so far as to physically prevent pushing it back in. Pick up tray: the robot must grab the tray inside the toaster oven with both hands (failure if only one grasps), and gently place it on the table (failure counted if the tray is dropped from more than about 3 inches or twisted during placement). Push rack in: the robot must push the wire rack back into the oven, with failure marked if it insufficiently pushes the rack to allow door closure. Close oven door: the robot must fully close the oven door and end without touching the door.

Marker

Pick cap: the robot must pick up the cap and reach the capping position without dropping it (mid-air drop counts as a cap failure). Pick marker: the robot must pick up the marker from the table and reach the cap position without dropping it. Cap marker: the robot must insert the marker into the cap such that the cap reaches a friction-fit over the marker. If the cap is partially inserted and loosely dangling, this counts as a failure. Grab bag: the robot needs to grasp the pencil bag and place it upright close to the robot base (failure if not grasped or if the bag falls over). Insert marker: the robot must fully insert the marker into the pencil pouch zipper enclosure, with no part of the marker sticking out. Zip bag: the robot must fully zip the bag and let go with both grippers (zipping is considered reached when the robot fully closes the bag, with the zipper within 1cm of the edge of the bag).

Boba

Pick cup: the robot must lift the boba cup off the workspace without dropping it. Pick straw: the robot must pick up the straw without dropping it. Insert straw: the robot must insert the straw into the hole in the cup, with success counted if the straw is fully seated in the hole after releasing (partially inside and tilting outside is counted as a failure). Place on coaster: the robot must place the cup on the coaster, with the cup resting flat fully supported by the coaster at the end.

Teabag

Pick teabag: the robot must grasp and lift the teabag off the table without dropping it. Place teabag in cup: after releasing, the teabag must be fully contained inside the cup; if it hits the teacup wall and bounces out this is a failure.

Wrench

Grasp box tab: the robot must grasp the black box tab with the left hand securely without slippage during the lid movement. Open toolbox: the toolbox lid must be fully open more than 90∘. Extract wrench: the robot must fully grasp and lift the wrench out of the toolbox, and reach a steady state above the wrench without dropping the wrench. If the wrench falls before reaching steady state, this is considered a failure.

References

1. [1] M. F. Land and M. Hayhoe. In what ways do eye movements contribute to everyday activities? Vision Research, 41(25):3559–3565, 2001. ISSN 0042-6989. doi:https://doi.org/10.1016/S0042-6989(01)00102-X. URL https://www.sciencedirect.com/science/article/pii/S004269890100102X.
2. [2] J. A. Feldman. Four frames suffice: A provisional model of vision and space. Behavioral and Brain Sciences, 8(2):265–289, 1985. doi:10.1017/S0140525X00020707.
3. [3] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
4. [4] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329, 2024.
5. [5] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. \pi0: A vision-language-action flow model for general robot control, 2024. URL https://arxiv. org/abs/2410.24164.
6. [6] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024.
7. [7] J. Kerr, K. Hari, E. Weber, C. M. Kim, B. Yi, T. Bonnen, K. Goldberg, and A. Kanazawa. Eye, robot: Learning to look to act with a bc-rl perception-action loop. arXiv preprint arXiv:2506.10968, 2025.
8. [8] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
9. [9] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35, 2017.
10. [10] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3(1):297–330, 2020.
11. [11] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025.
12. [12] Y. Du, D. Ho, A. Alemi, E. Jang, and M. Khansari. Bayesian imitation learning for end-to-end mobile manipulation. In International Conference on Machine Learning, pages 5531–5546. PMLR, 2022.
13. [13] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
14. [14] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, V. Guizilini, D. A. Herrera, M. Heo, K. Hsu, J. Hu, M. Z. Irshad, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O’Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Martín-Martín, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
15. [15] O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang, G. Wang, H. Su, H.-S. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Vakil, J. Bohg, J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang, J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. J. Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, M. Z. Irshad, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart’in-Mart’in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, V. Guizilini, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. https://arxiv.org/abs/2310.08864, 2023.
16. [16] T. Z. Zhao, J. Tompson, D. Driess, P. Florence, K. Ghasemipour, C. Finn, and A. Wahid. Aloha unleashed: A simple recipe for robot dexterity, 2024. URL https://arxiv.org/abs/2410.13126.
17. [17] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik. Ego4d: Around the world in 3,000 hours of egocentric video, 2022. URL https://arxiv.org/abs/2110.07058.
18. [18] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024.
19. [19] Z. Lv, N. Charron, P. Moulon, A. Gamino, C. Peng, C. Sweeney, E. Miller, H. Tang, J. Meissner, J. Dong, K. Somasundaram, L. Pesqueira, M. Schwesinger, O. Parkhi, Q. Gu, R. D. Nardi, S. Cheng, S. Saarinen, V. Baiyya, Y. Zou, R. Newcombe, J. J. Engel, X. Pan, and C. Ren. Aria everyday activities dataset, 2024. URL https://arxiv.org/abs/2402.13349.
20. [20] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022.
21. [21] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W.-C. Tseng, Y. Dong, K. Mo, C.-H. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. R. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M.-Y. Liu, Y. Zhu, J. Jang, and L. J. Fan. Dreamdojo: A generalist robot world model from large-scale human videos, 2026. URL https://arxiv.org/abs/2602.06949.
22. [22] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113–1132. Pmlr, 2020.
23. [23] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422, 2023.
24. [24] L. X. Shi, Z. Hu, T. Z. Zhao, A. Sharma, K. Pertsch, J. Luo, S. Levine, and C. Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv:2403.12910, 2024.
25. [25] S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. In https://arxiv.org/abs/2403.01823, 2024.
26. [26] K. Y. Goldberg and R. Bajcsy. Active touch and robot perception. Cognition and Brain Theory, 7(2):199–214, 1984.
27. [27] R. Bajcsy. Active perception. Proceedings of the IEEE, 76(8):966–1005, 1988.
28. [28] J. Aloimonos, I. Weiss, and A. Bandyopadhyay. Active vision. International journal of computer vision, 1(4):333–356, 1988.
29. [29] R. Bajcsy, Y. Aloimonos, and J. K. Tsotsos. Revisiting active perception. Autonomous Robots, 42(2):177–196, 2018.
30. [30] D. Jayaraman and K. Grauman. Learning to look around: Intelligently exploring unseen environments for unknown tasks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1238–1247, 2018.
31. [31] T. Van de Maele, T. Verbelen, O. Çatal, C. De Boom, and B. Dhoedt. Active vision for robot manipulators using the free energy principle. Frontiers in neurorobotics, 15:642780, 2021.
32. [32] D. H. Ballard. Animate vision. Artificial intelligence, 48(1):57–86, 1991.
33. [33] J. K. Tsotsos. On the relative complexity of active vs. passive visual search. International journal of computer vision, 7(2):127–141, 1992.
34. [34] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. ieee, 2017.
35. [35] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.
36. [36] X. Ye, Z. Lin, H. Li, S. Zheng, and Y. Yang. Active object perceiver: Recognition-guided policy learning for object searching on mobile robots. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6857–6863. IEEE, 2018.
37. [37] A. Rasouli, P. Lanillos, G. Cheng, and J. K. Tsotsos. Attention-based active visual search for mobile robots. Autonomous Robots, 44(2):131–146, 2020.
38. [38] S. Dass, J. Hu, B. Abbatematteo, P. Stone, and R. Martín-Martín. Learning to look: Seeking information for decision making via policy factorization, 2024. URL https://arxiv.org/abs/2410.18964.
39. [39] N. P. Papanikolopoulos, P. K. Khosla, and T. Kanade. Visual tracking of a moving target by a camera mounted on a robot: A combination of control and vision. IEEE transactions on robotics and automation, 9(1):14–35, 1993.
40. [40] D. Wilkes and J. K. Tsotsos. Active object recognition, volume 3. University of Toronto, 1994.
41. [41] S. Gould, J. Arfvidsson, A. Kaehler, B. Sapp, M. Messner, G. R. Bradski, P. Baumstarck, S. Chung, A. Y. Ng, et al. Peripheral-foveal vision for real-time object recognition and tracking in video. In Ijcai, volume 7, pages 2115–2121, 2007.
42. [42] A. Jonnalagadda, W. Y. Wang, B. Manjunath, and M. P. Eckstein. Foveater: Foveated transformer for image classification. arXiv preprint arXiv:2105.14173, 2021.
43. [43] J. E. Banta, Y. Zhien, X. Z. Wang, G. Zhang, M. Smith, and M. A. Abidi. Best-next-view algorithm for three-dimensional scene reconstruction using range images. In Intelligent Robots and Computer Vision XIV: Algorithms, Techniques, Active Vision, and Materials Handling, volume 2588, pages 418–429. SPIE, 1995.
44. [44] S. Isler, R. Sabzevari, J. Delmerico, and D. Scaramuzza. An information gain formulation for active volumetric 3d reconstruction. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3477–3484. IEEE, 2016.
45. [45] M. Mendoza, J. I. Vasquez-Gomez, H. Taud, L. E. Sucar, and C. Reta. Supervised learning of the next-best-view for 3d object reconstruction. Pattern Recognition Letters, 133:224–231, 2020.
46. [46] R. Zeng, Y. Wen, W. Zhao, and Y.-J. Liu. View planning in robot active vision: A survey of systems, algorithms, and applications. Computational Visual Media, 6(3):225–245, 2020.
47. [47] X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-television: Teleoperation with immersive active visual feedback. arXiv preprint arXiv:2407.01512, 2024.
48. [48] I. Chuang, A. Lee, D. Gao, M.-M. Naddaf-Sh, and I. Soltani. Active vision might be all you need: Exploring active vision in bimanual robotic manipulation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 7952–7959. IEEE, 2025.
49. [49] Q. Zeng, C. Li, J. S. John, Z. Zhou, J. Wen, G. Feng, Y. Zhu, and Y. Xu. Activeumi: Robotic manipulation with active perception from robot-free human demonstrations. arXiv preprint arXiv:2510.01607, 2025.
50. [50] H. Xiong, X. Xu, J. Wu, Y. Hou, J. Bohg, and S. Song. Vision in action: Learning active perception from human demonstrations. arXiv preprint arXiv:2506.15666, 2025.
51. [51] J. Yu, Y. Shentu, D. Wu, P. Abbeel, K. Goldberg, and P. Wu. Egomi: Learning active vision and whole-body manipulation from egocentric human demonstrations. arXiv preprint arXiv:2511.00153, 2025.
52. [52] G. Spigler. Tavis: A benchmark for egocentric active vision and anticipatory gaze in imitation learning. arXiv preprint arXiv:2605.07943, 2026.
53. [53] Y. Li, M. Liu, and J. M. Rehg. In the eye of the beholder: Gaze and actions in first person video. IEEE transactions on pattern analysis and machine intelligence, 45(6):6731–6747, 2021.
54. [54] A. Pani and Y. Yang. Gaze-regularized vision-language-action models for robotic manipulation. arXiv preprint arXiv:2603.23202, 2026.
55. [55] J. Li, Y. Qiao, Y. Guo, C. Chen, and W. Lian. Act, sense, act: Learning non-markovian active perception strategies from large-scale egocentric human data, 2026. URL https://arxiv.org/abs/2602.04600.
56. [56] I. Chuang, J. Zou, A. Lee, D. Gao, and I. Soltani. Look, focus, act: Efficient and robust robot learning via human gaze and foveated vision transformers. arXiv preprint arXiv:2507.15833, 2025.
57. [57] C. Li, K. Xiong, Y. Xu, L. Qian, Y. Wang, and W. Zhu. Gazevla: Learning human intention for robotic manipulation. arXiv preprint arXiv:2604.22615, 2026.
58. [58] H. Kim, Y. Ohmura, and Y. Kuniyoshi. Multi-task robot data for dual-arm fine manipulation. arXiv preprint arXiv:2401.07603, 2024.
59. [59] A. Lee, I. Chuang, D. Gao, K. Fukazawa, and I. Soltani. Gaze on the prize: Shaping visual attention with return-guided contrastive learning. arXiv preprint arXiv:2510.08442, 2025.
60. [60] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347.
61. [61] N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. Rädle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y. Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. K. Cheng, P. Dollár, N. Ravi, K. Saenko, P. Zhang, and C. Feichtenhofer. Sam 3: Segment anything with concepts, 2025. URL https://arxiv.org/abs/2511.16719.
62. [62] B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo, and S. Birchfield. Foundationstereo: Zero-shot stereo matching. CVPR, 2025.
63. [63] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators, 2023.
64. [64] The Robot Learning Company. TRLC-DK1: An Open Source Dev Kit for AI-native Robotics. https://www.robot-learning.co/, 2026. Accessed: 2026-05-26.
65. [65] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109.
66. [66] K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y. Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs, C. Sferrazza, Y. Tassa, and P. Abbeel. Mujoco playground, 2025. URL https://arxiv.org/abs/2502.08844.
67. [67] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H. Jégou, P. Labatut, and P. Bojanowski. DINOv3, 2025. URL https://arxiv.org/abs/2508.10104.
68. [68] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. ISSN 0925-2312. doi:https://doi.org/10.1016/j.neucom.2023.127063. URL https://www.sciencedirect.com/science/article/pii/S0925231223011864.
69. [69] D. Hendrycks and K. Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016. URL http://arxiv.org/abs/1606.08415.
70. [70] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
71. [71] I. Loshchilov and F. Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101.
72. [72] K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
73. [73] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
74. [74] X. Wei, M. Liu, Z. Ling, and H. Su. Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search. ACM Transactions on Graphics (TOG), 41(4):1–18, 2022.