카메라 먼저 움직이는 로봇, 가려진 물체 작업 성공 90회 중 35회에서 83회로
푸단대 프리프린트에 따르면 로봇이 손을 뻗기 전에 손목 카메라를 먼저 움직이게 하고, 그 카메라 이동이 끝나는 지점에서 시작하는 데이터로 정책을 다시 학습시키자 실제 가림 상황 작업 성공이 90회 중 35회에서 83회(92.2%)로 늘었다. 학습 데이터를 바꾸지 않고 카메라 이동만 붙이자 pi0.5의 시뮬레이션 성공률은 오히려 46.3%에서 11.5%로 떨어졌다.
로봇이 먼저 카메라를 움직여 가려진 물체를 보게 하면 왜 시각언어행동(VLA) 정책 성능이 오히려 나빠지는가, 그리고 학습 데이터를 어떻게 다시 만들어야 더 나은 시야가 실제로 도움이 되는가를 묻는다.
물건이 어지럽게 놓인 곳에서는 기본 시점으로 목표물이 안 보이는 경우가 많아 손목 카메라로 먼저 둘러보는 방식이 당연해 보인다. 하지만 카메라를 움직이면 팔 자세와 정책이 처음 보는 영상이 함께 바뀌어 시연 데이터에 없던 출발 상태가 된다. 연구진은 이를 첫 프레임 분포 민감성이라 불렀다. 출발 자세를 흔든 LIBERO 실험에서 pi0.5 앞에 카메라 이동만 붙이자 평균 성공률이 46.3%에서 11.5%로, pi0은 30.7%에서 7.0%로 내려갔다.
능동 지각과 다음 최적 시점 기법은 정보가 많은 시점을 고르지만 주로 인식 성능으로 평가됐고, 그다음 행동하는 정책까지 따지지 않았다. Vision in Action이나 ActiveGlasses처럼 보기와 행동을 함께 배우는 시스템은 6자유도 목이나 별도 카메라 팔 같은 전용 하드웨어가 필요하다. 데이터 증강이나 무작위 시점 흔들기는 정책이 보는 영상의 폭을 넓히지만 새 팔 자세에서 이어갈 행동 정답은 주지 못한다.
ActiveAlign은 VLA 구조는 그대로 두고 미세조정 데이터를 바꾼다. 학습된 모듈이 장면 RGB-D와 지시문으로 손목 카메라 궤적을 만든다. 오프라인에서 초기화가 가능한 시뮬레이터나 실제 작업 셀에서 그 궤적을 실행하고, 기존 시연 앞부분에서 가장 가까운 말단 자세를 찾아 폐루프 제어로 차이를 메운 뒤 나머지 조작 동작을 재생한다. 재생에 성공한 궤적만 미세조정에 쓴다. 실제 운용 때는 같은 시점 모듈이 한 번 돈 뒤 적응된 정책이 새 영상을 보며 작업한다.
출발 자세를 흔든 LIBERO 40개 과제(과제당 50회)에서 ActiveAlign은 pi0 70.7%, pi0.5 72.7%로 표준 학습보다 각각 40.0%포인트, 26.4%포인트 높았다. OpenVLA는 56.4%에서 69.6%(+13.2), 엔비디아 GR00T N1은 51.8%에서 67.8%(+16.0)로 올랐다. LIBERO Spatial에서 데이터 양은 같게 두고 기본 시점, 무작위 시점, 시점 흔들기를 쓴 대조군은 54.6%, 58.0%, 61.4%에 그쳐 ActiveAlign의 78.2%에 못 미쳤다. RealSense D435 카메라를 단 실제 로봇팔 두 종(ARX X5, ARCHER-Y6)으로 가림막 뒤 사과를 다루는 과제는 90회 중 83회 성공해 표준 학습 35회, 단순 카메라 이동 17회를 크게 앞섰다. 표준 대비 약 2.37배(MEKAWIRE 계산)다. 컵 집기, 퍼즐 꺼내기, 관찰 후 열고 집어 놓기 과제는 각 20회 중 17회, 15회, 13회 성공했다. 실제 데이터 구축 시도 100회 중 86회가 채택됐다.
프리프린트다. 고정구로 물체를 1cm 이내로 되돌릴 수 있는 작업 셀이나 초기화 가능한 시뮬레이터가 필요해 데이터 수집 중 장면을 되돌릴 수 없는 가정 환경에는 바로 적용하기 어렵다. 재생에 성공한 경우만 남기므로 어려운 시점은 데이터에 덜 담긴다. 카메라는 접촉 전에 한 번만 움직이고 움직이는 가림막에는 대응하지 않는다. 벤치마크 기본 출발 상태에서는 이득이 없었고(96.6% 대 96.8%) 출발 자세를 흔들수록 격차가 커졌다. 실제 실험은 조건당 20회나 30회라 신뢰구간이 넓다.
피지컬 인텔리전스의 pi 계열이나 엔비디아 GR00T 같은 파운데이션 정책을 상자, 선반, 키팅 공정 피킹에 맞춰 미세조정하는 기업에 바로 와닿는 결과다. 가림이 일상인 현장에서 학습 데이터를 다시 만들지 않고 능동 시각 단계만 붙이면 오히려 성능이 떨어질 수 있다는 점을 보여준다. 고정구가 있는 작업 셀을 운영하는 국내 제조와 물류 자동화 업체라면 새 모델 없이 시연 데이터만 재구성하면 되는 저비용 방법이다. 공장과 물류 셀에 먼저 쓰이고, 가정용 로봇에는 장면 초기화 없이 작동하는 변형이 필요하다.
논문 전문
Look Before You Act: Bridging Active Perception and Vision-Language-Action Models via Observation-Aligned Data Adaptation
CC BY 4.0 라이선스로 공개된 논문입니다. 출처를 밝혀 전재하며, 원문은 arXiv:2610.23378(PDF)에서 볼 수 있습니다.
Abstract
Robots manipulating objects in clutter must decide not only how to act, but also where to look. Although active camera motion can expose an occluded target, it may move the robot into an initial observation–state distribution that differs from the manipulation policy’s training distribution. We identify this problem as first-frame distribution sensitivity: under perturbed starts in LIBERO, naively prepending camera repositioning to \pi_{0.5} reduces success from 46.3% to 11.5%. We introduce ActiveAlign, a data-adaptation framework that trains a Vision-Language-Action (VLA) policy from the states at which active perception terminates. A learned viewpoint module first generates a camera trajectory. Offline, ActiveAlign uses closed-loop alignment to connect the trajectory endpoint to an existing demonstration, replays the remaining manipulation trajectory, and retains successful continuations for fine-tuning. At deployment, the same viewpoint module precedes the adapted VLA without modifying its architecture. Across 40 LIBERO tasks, ActiveAlign achieves 70.7% success with \pi_{0} and 72.7% with \pi_{0.5}, outperforming their standard-training baselines by 40.0% and 26.4% respectively. Matched-state and data-matched controls support gains beyond prefix shortening and generic viewpoint augmentation; OpenVLA and GR00T N1 also show the same trend. Success rate under physical occlusion in real world increases from 35/90 to 83/90. These results show that active perception benefits manipulation when policy training is aligned with the observation and robot-state distribution induced by viewpoint selection.
I INTRODUCTION
Effective manipulation begins with an observation that supports the intended action [1, 2]. A robot asked to retrieve fruit from a container or grasp a cup behind clutter may need to move its camera before it can see a useful approach direction. Vision-Language-Action (VLA) models [3, 4, 5, 6, 7] provide a powerful interface from visual observations and language to robot actions, but their input is still determined by where the robot looks. Repositioning an existing wrist camera is an appealing way to expose the target before manipulation. The resulting system must therefore solve two connected problems: selecting an informative view and continuing the task from the state produced by that selection.
The most direct composition is to let an active-perception (AP) module improve the view, then hand control to a VLA trained on conventional demonstrations. Our experiments reveal a counterintuitive outcome: this composition can substantially reduce task success. Under a shared perturbed-start LIBERO protocol, \pi_{0.5} falls from 46.3% to 11.5% average success after camera repositioning, and \pi_{0} falls from 30.7% to 7.0%. The qualitative examples in Fig. 1 illustrate the tension: a policy can act from a familiar but occluded view, yet struggle to exploit a view selected to reveal the target. Improving the observation and making it usable by the action policy are distinct requirements.
We study this interface through first-frame distribution sensitivity. Here, the first frame is the observation at the handoff from active perception to manipulation. Wrist-camera motion changes image location, scale, and occlusion together with arm configuration and the relationship to the remaining task. A policy must consequently infer a suitable continuation from a different combination of visual and proprioceptive cues. Action chunking [8, 9] can propagate an early error through the actions executed before the next observation, even when later decisions use fresh images. This motivates examining the joint observation–action interface; a difference between first-frame image distributions alone does not establish the cause of every failure. Our experiments therefore pair the failure observation with controls that separate viewpoint information from robot-state progress and data volume.
The central design insight is that the deployment viewpoint policy is available during data construction. Rather than requiring a manipulation learner to accommodate arbitrary sensing motions, we can expose it to successful continuations from the endpoints of the very module it will encounter at inference. This turns the composition problem into a question about supervision: how can existing demonstrations be made to begin where active perception stops? Demonstration-generation methods such as MimicGen and DemoGen [10, 11] motivate reusing recorded behavior, while the present setting adds a specific constraint: the starting observation and state are chosen by a task-conditioned camera policy. Visual augmentation alone does not supply the actions needed to continue from that state.
ActiveAlign constructs these continuations offline. Given scene RGBD and a language instruction, a learned module predicts a wrist-camera trajectory. In a restored environment, we execute the trajectory and match its endpoint to a nearby end-effector state in an early demonstration prefix. Closed-loop alignment bridges the residual gap, after which the recorded manipulation suffix is replayed. Successful alignment-and-replay trajectories become training data for the VLA. The active approach replaces part of the original demonstration, while the recorded continuation teaches the policy how to proceed after that approach. At deployment, the same viewpoint module runs once before object contact; the adapted VLA then manipulates using fresh observations. The procedure changes training data and policy weights while preserving the VLA architecture.
Replay feasibility is essential to this construction. End-effector proximity does not ensure that object and contact states agree, so nearest-state matching alone cannot validate a continuation. Successful replay filters the candidate endpoints and limits the training distribution to states for which usable supervision can be reconstructed. ActiveAlign consequently applies to resettable simulators and reconstructable physical workcells. This requirement belongs to offline data collection: deployment requires neither scene reset nor expert replay. The resulting scope is pre-manipulation view selection, with longer-horizon tasks testing the lasting value of that initial view rather than continuous response to moving occluders.
Our evaluation follows this reasoning from composition failure to controlled explanation and physical validation. On LIBERO Spatial, a matched-state camera-extrinsics control holds the demonstration step and manipulation suffix fixed and reaches 54.6%, compared with 78.2% for ActiveAlign. Data-matched random views and broad viewpoint jitter reach 58.0% and 61.4%, respectively. These comparisons support a benefit from the informative, deployment-matched observations beyond shorter demonstrations and generic coverage. Across four suites, ActiveAlign improves \pi_{0} and \pi_{0.5} by 40.0 and 26.4 percentage points; OpenVLA and GR00T N1 extend the evidence to different action formulations. Initial-occlusion simulation tasks, diverse physical tasks, and endpoint perturbations further characterize when the construction is useful and where it breaks down.
Our contributions are threefold:
- We characterize the failure of naive composition of active perception and VLAs, and test competing explanations through matched-state and data-matched controls.
- We introduce observation-aligned demonstration adaptation, connecting active-view endpoints to successful manipulation through prefix matching, closed-loop alignment, and replay filtering.
- We evaluate the construction across four VLA backbones, 40 LIBERO tasks, initial occlusion, and physical manipulation, with sensitivity studies that quantify the feasibility–diversity trade-off and endpoint error tolerance.
II RELATED WORK
Vision-language-action policies. Large-scale language-conditioned robot policies such as RT-1 and RT-2 established a general interface from visual observations and language instructions to robot actions [12, 13]. Recent VLA models instantiate this interface with different action representations: \pi_{0} and \pi_{0.5} use flow matching [3, 5], OpenVLA predicts discrete action tokens [4], and GR00T N1 uses a diffusion-transformer action module [14]. Diffusion policies and alternative action tokenizations provide additional formulations for visuomotor control [9, 15, 16, 17]. Despite these architectural differences, each policy depends on the observation–state distribution represented in its training demonstrations. ActiveAlign is orthogonal to these action parameterizations: it preserves the policy architecture while adapting the fine-tuning distribution to the states induced by a deployment-time viewpoint module which offers a more informative view.
Active and multi-view perception. Active perception [18, 19] and next-best-view methods [20, 21] select sensing actions that improve downstream decisions. Multi-view manipulation methods aggregate or render observations from multiple viewpoints to improve spatial reasoning [22, 23, 24, 25], while SAM2Act, VISO-Grasp, and ViewActive explore related mechanisms for visual representation and viewpoint selection [26, 27, 28]. Our approach, instead, physically repositions a wrist camera before invoking a separately adapted VLA, thus studying physical wrist-camera motion, which changes not only the visual input but also the robot configuration from which manipulation begins. Its focus is therefore the compatibility between a selected better viewpoint and the downstream policy’s manipulation distribution.
Recent robotic systems learn physical sensing motion together with manipulation. Vision in Action couples sensing and bimanual behavior using a 6-DoF neck and VR teleoperation [29]; HoMMI combines egocentric human demonstrations, hand–eye policies, and whole-body control [30]; and ActiveGlasses uses a dedicated perception arm to coordinate camera and manipulation motion [31]. These systems collect or learn paired sensing and manipulation behavior for specialized active-vision embodiments. In contrast, ActiveAlign uses an existing wrist camera for a single pre-manipulation viewpoint trajectory and connects it to an independently trained VLA through offline demonstration adaptation, without altering the VLA architecture or requiring continuous gaze control. GenDoP [32] studies related autoregressive RGBD-and-language camera trajectory generation [33]; our contribution concerns the robot-data adaptation interface.
Demonstration adaptation and distribution shift. DAgger [34] addresses policy-induced state mismatch through iterative collection. Domain randomization and augmentation broaden visual coverage [35, 36, 37], including through generated images [38, 39, 40]. ActiveAlign targets a different source of shift: the handoff states produced by a separate viewpoint policy used at deployment. Demonstration-generation methods reuse recorded behavior to expand the training distribution. MimicGen transforms demonstrations across object configurations [10], while DemoGen adapts trajectories and point-cloud observations for spatial generalization [11]. ActiveAlign instead adapts demonstrations to the physical endpoints induced by a task-conditioned viewpoint policy. Because camera motion changes both arm configuration and visual observations, geometric endpoint matching alone does not guarantee a valid manipulation continuation. ActiveAlign therefore uses closed-loop alignment and successful replay to validate supervision. Our matched-state and data-matched controls further compare this targeted, deployment-matched adaptation with shorter manipulation suffixes and broader random-view or viewpoint-jitter coverage.
III METHOD
III-A Problem Formulation
Let x_{t} denote the physical scene and robot state and let o_{t} denote the complete policy observation, including scene RGB, wrist RGB, and proprioception where supported by the backbone. A policy \pi_{\theta} conditions on o_{t} and instruction l. Demonstrations contain state-observation-action trajectories; denote the observation distribution at original demonstration starts by p_{\mathrm{demo},0}.
An active perception module \mathcal{A} generates a camera trajectory from a scene observation s_{0} and instruction l. Execution induces a new state \tilde{x}_{0} and observation \tilde{o}_{0}. The corresponding initial distribution p_{\mathcal{A},0} may differ from p_{\mathrm{demo},0}. Because the camera is wrist-mounted, this shift generally involves both vision and robot state. A mismatch of initial-image marginals alone neither proves that the input is outside the full training support nor establishes the cause of a policy failure.
Our objective is to improve manipulation from the states induced by \mathcal{A}. We construct adapted demonstrations \widetilde{\mathcal{D}} and fine-tune with the backbone’s supervised learning objective:
\theta^{*}=\arg\min_{\theta}\mathbb{E}_{(o,l,a)\sim\mathcal{D}_{\mathrm{train}}}\bigl[\mathcal{L}_{\mathrm{VLA}}(\theta;o,l,a)\bigr].Here \mathcal{D}_{\mathrm{train}} denotes the condition’s fine-tuning data, including aligned demonstrations for ActiveAlign; data-matched controls use the same mixing ratio. The target a is an action or chunk, and \mathcal{L}_{\mathrm{VLA}} is its token-prediction or continuous-action loss. Task success is evaluated through rollouts, not optimized directly by this loss.
The construction must preserve more than the appearance of the active view. Each observation needs an action target that is executable from the corresponding robot and scene state. We therefore use the deployed perception module to generate candidate starting states, and use recorded manipulation to supply their continuations. Prefix matching identifies a nearby continuation, alignment connects to it, and replay checks whether the resulting episode remains successful. These three operations jointly determine which candidate views can become useful VLA supervision.
III-B Active Perception Module
View selection is conditioned on both the task and the scene: the instruction identifies what needs to be observed, while the surrounding geometry determines which camera motion may expose it. The module combines RGB features, depth features, and language features to predict a sequence of camera poses. RGB and depth use separate CLIP branches, and a conditional autoregressive decoder generates quantized pose tokens. The decoded extrinsics are passed to the robot controller, which turns the visual prediction into a candidate sensing motion.
Formally, the input is a scene RGBD observation s_{0}=(I_{\mathrm{rgb}},D) and instruction l, and the output is \{T_{1},\ldots,T_{K}\}. The default uses K=30 poses (Fig. 2), represented in SE(3) and transformed into the calibrated robot/world frame. The generated sequence can be interpolated to 120 poses for smooth execution. Training uses scene-instruction-camera-trajectory pairs: 20,000 in simulation and 1,200 for the physical setting, separate from the VLA demonstrations.
Predicted paths are candidate sensing motions, without certified optimality or collision freedom. Simulation uses IK repositioning; the physical robot moves continuously through its controller. Adaptation retains endpoints permitting alignment and successful manipulation replay.
III-C Observation-Aligned Data Adaptation
In Algorithm 1, the environment is restored to each demonstration’s starting state. After executing \mathcal{A}, we identify the nearest end-effector state within a prefix window [\alpha T,\beta T]. Closed-loop Cartesian alignment closes the residual gap, after which the recorded manipulation suffix is replayed. Only the alignment segment and suffix of successful attempts are retained; the active approach is not a VLA training target.
Prefix matching. The default window is [0.05T,0.30T]. Earlier matching retains more of the original approach; later matching shortens manipulation and may require unreproduced scene changes. End-effector proximity does not ensure matching object/contact states, making successful replay a necessary filter. Our window study measures retention, diversity, and success.
Closed-loop alignment. A recorded adjustment connects the active endpoint to the matched suffix using operational space control (OSC) in simulation and Cartesian control on the robot. The default simulation adjustment budget is 20 steps. Directly replaying the suffix from the unmatched endpoint can accumulate control error, so this transition is included in the adapted trajectory. The VLA therefore receives supervision for leaving the active endpoint as well as following the retained manipulation. Unlike IK repositioning, the alignment segment consists of executable VLA training actions.
Filtering and retries. Simulation retries vary the active trajectory budget and alignment steps. The reported adaptation counts are 1,736 retained out of 2,000 attempted in simulation (86.8%) and 86 retained out of 100 attempted on the physical platform (86.0%).
Consequently, \widetilde{\mathcal{D}} represents a feasibility-filtered active-view distribution: filtering and retries can make p_{\widetilde{\mathcal{D}},0} differ from p_{\mathcal{A},0}. Sharing the viewpoint module targets nominal alignment, without guaranteeing distribution equality or optimal task success. The control and perturbation studies test its practical benefit and sensitivity.
III-D Deployment and Applicability
Deployment consists of one active approach followed by manipulation. The viewpoint module is queried once before object contact. Thereafter, the VLA obtains fresh images and state inputs and predicts actions. For the chunked \pi_{0} and \pi_{0.5} implementations,
\{a_{t},\ldots,a_{t+H-1}\}=\pi_{\theta^{*}}(o_{t},l),\quad H=10,\quad R=5,where the first R actions are executed before replanning. Thus errors can accumulate within the executed portion, but the policy does not commit to all H actions without correction. Other backbones retain their native action-generation interfaces.
The wrist camera moves with the arm during manipulation, but no further viewpoint-selection phase is invoked. Offline adaptation needs simulation reset or a reconstructable physical workcell; deployment needs neither reset nor demonstration replay. Replacing \mathcal{A} may require regenerating data and fine-tuning.
IV EXPERIMENTS
IV-A Experimental Setup
Environments and data. LIBERO [41] provides four suites of 10 tasks, with 50 original demonstrations per task (2,000 total). LIBERO-Occ [42] uses the same suites but occludes the target in the default initial view. The physical evaluation runs on two arm settings (an ARX X5 arm and an ARCHER-Y6 arm), both with scene-level and wrist-mounted Intel RealSense D435 cameras, and 100 original demonstrations. RLBench [43] is used only to visualize zero-shot transfer of the perception module.
Backbones and training. We evaluate \pi_{0}, \pi_{0.5}, OpenVLA, and GR00T N1. We follow the model-specific recommended training recipes, with hyperparameters selected using the validation set. All models share the LIBERO Franka end-effector delta-pose and gripper interface.
Perturbed-start protocol. Each LIBERO result uses 50 rollouts per task, with horizons of 220, 280, 300, and 520 steps for Spatial, Object, Goal, and Long. Initial end-effector pose coordinates receive zero-mean Gaussian perturbations with standard deviation 0.1 (translation in meters and rotation in radians). This perturbed-start protocol differs from the benchmark’s prescribed initial states and is held fixed across all method conditions.
Conditions and statistical reporting. Standard trains on original demonstrations and uses no AP; Naive Active invokes AP only at inference; ActiveAlign trains on observation-aligned adaptations and invokes the same AP module. Data-matched controls fix retained episode count, filtering, suffix length, optimization, and mixing. LIBERO uses the environment completion checker. Physical results report binary counts and Wilson 95% confidence intervals.
IV-B Simulation Results and Analysis
Perturbed-start performance. Table I shows the handoff failure directly: AP without aligned training reduces average success from 30.7% to 7.0% for \pi_{0} and from 46.3% to 11.5% for \pi_{0.5}. ActiveAlign instead reaches 70.7% and 72.7%, gains of 40.0 and 26.4 percentage points (pp) over Standard, with improvements on every suite. These values are comparisons within our perturbed-start protocol.
- \pi_{0}
- Standard
- 34.0
- 32.2
- 28.4
- 28.0
- 30.7
- Naive Active
- 6.6
- 0.0
- 13.6
- 7.6
- 7.0
- 자료 없음
- ActiveAlign
- 73.8
- 78.2
- 66.6
- 64.2
- 70.7
- 자료 없음
- \pi_{0.5}
- Standard
- 48.0
- 39.0
- 49.4
- 48.8
- 46.3
- Naive Active
- 22.0
- 2.2
- 18.6
- 3.0
- 11.5
- 자료 없음
- ActiveAlign
- 78.2
- 78.0
- 68.8
- 65.8
- 72.7
- 자료 없음
The gains also extend beyond recovering the loss from naive composition. On Object, for example, ActiveAlign exceeds Standard by 46.0 pp for \pi_{0} and 39.0 pp for \pi_{0.5}, while Naive Active reaches only 0.0% and 2.2%. Thus, the informative starting condition becomes useful when paired with adapted supervision. The two adapted backbones finish within 2.0 pp of each other on average; this motivates studying the data interface alongside model choice, without inferring that model capacity is generally irrelevant.
- \pi_{0}
- 30.7
- 7.0
- 70.7
- \pi_{0.5}
- 46.3
- 11.5
- 72.7
- OpenVLA
- 56.4
- 24.8
- 69.6
- GR00T N1
- 51.8
- 21.6
- 67.8
Table II tests whether the adaptation is tied to the two flow-matching \pi policies. ActiveAlign improves OpenVLA by 13.2 pp and GR00T N1 by 16.0 pp over their Standard conditions. The consistent trend across flow-matching, autoregressive-token, and diffusion-transformer policies supports compatibility with the tested action formulations, without implying universal backbone independence.
Table III shows that under the benchmark-prescribed initial states, Standard and ActiveAlign are nearly matched; as the start-state shift increases, the gap widens.
- Standard
- 96.8
- 85.2
- 70.4
- 46.3
- Naive Active
- 48.0
- 38.2
- 23.1
- 11.5
- ActiveAlign
- 96.6
- 90.4
- 83.0
- 72.7
- ActiveAlign - Standard
- -0.2
- +5.2
- +12.6
- +26.4
Viewpoint and data controls. Table IV isolates two competing explanations. The matched-state control fixes k^{*}, suffix length, retained count, training schedule, and mixing, while independently changing camera extrinsics to provide a default view instead of an active informative observation. Its 54.6% success, versus 78.2% for ActiveAlign, leaves a 23.6 pp gap after controlling for prefix shortening. As an auxiliary diagnostic, the target is visible in 43.7% of the matched-state control observations and 74.8% of the corresponding ActiveAlign observations.This visibility difference confirms that the two conditions provide substantially different target information; together with the controlled success comparison, it is consistent with the benefit of training on informative views induced by the deployment viewpoint module. Random-view adaptation and broad viewpoint jitter adaptation use the same data and optimization budgets and reach 58.0% and 61.4%. Generic viewpoint coverage therefore improves over Standard, but remains 20.2 pp and 16.8 pp below ActiveAlign. These results support the value of deployment-matched viewpoint adaptation beyond additional data, shorter continuations, and untargeted viewpoint diversity.
- Standard
- 48.0
- Naive Active
- 22.0
- Matched-state control
- 54.6
- Random-view adaptation
- 58.0
- Broad viewpoint jitter
- 61.4
- ActiveAlign
- 78.2
Initial occlusion and perception transfer. Table V evaluates targets hidden in the default first frame. ActiveAlign exceeds Standard by 28.6, 24.6, 22.4, and 34.6 pp on Spatial, Object, Goal, and Long. Naive Active remains below Standard. The Long result shows that a better initial observation can remain useful over a longer manipulation horizon; it does not test occluders that move during execution.
- Standard
- 20.0
- 22.6
- 21.4
- 7.8
- Naive Active
- 5.8
- 2.2
- 8.6
- 0.2
- ActiveAlign
- 48.6
- 47.2
- 43.8
- 42.4
Fig. 3 shows the selected views around clutter. RLBench is a qualitative test of perception transfer only; end-to-end manipulation is evaluated in LIBERO and on the physical robot.
In Fig. 4, adapted demonstrations and AP-conditioned inference occupy overlapping feature regions. This visualization is descriptive; the controlled rollouts provide the causal evidence.
\pi_{0.5}. Each panel varies one factor at a time.- Grasp a cup on a cluttered tabletop
- Target identity and clutter
- 9/20 [25.8,65.8]
- 4/20 [8.1,41.6]
- 17/20 [64.0,94.8]
- Retrieve puzzles from a container
- Object and container geometry
- 6/20 [14.5,51.9]
- 3/20 [5.2,36.0]
- 15/20 [53.1,88.8]
- Observe, open, grasp, and place
- Longer manipulation horizon
- 5/20 [11.2,46.9]
- 3/20 [5.2,36.0]
- 13/20 [43.3,81.9]
Sensitivity analysis. Table VI Panel (a) exposes the feasibility–diversity trade-off: later windows retain more adaptations but reduce endpoint diversity (measured as mean pairwise terminal-camera translation after scene normalization), and the middle window gives the highest success. Panel (b) perturbs only the predicted terminal camera position at inference. This perturbation is separate from the base protocol’s initial robot-pose noise and tests sensitivity to errors in the AP endpoint. Success falls by 6.0 pp at 0.02 m and 9.0 pp at 0.04 m, then drops below Standard at 0.10 m. The method therefore tolerates moderate translational endpoint error but degrades when the deployment handoff moves substantially beyond the aligned training distribution. Panel (c) shows consistent gains as the trajectory budget grows from 5 to 30. This trend shows that performance benefits from additional viewpoint-trajectory capacity.
IV-C Physical Evaluation
Collection and reset. Marked fixtures restore object and container states within 1 cm and 3^{\circ}. Predicted camera paths are executed continuously; the recorded end-effector prefix is matched, closed-loop Cartesian control reduces the residual, and the manipulation suffix is replayed. An operator provides safety supervision only: near collision or force limits, the operator can abort but cannot correct a motion, teleoperate, or finish the task. The procedure retains 86 of 100 attempts. Reset and replay are used only for offline data construction.
Evaluation protocols. We conduct two separate physical evaluations. The first evaluation measures task diversity using cup grasping, puzzle retrieval, and a longer observe–open–grasp–place task. Each task is evaluated for 20 trials, yielding 60 additional trials per method. The second measures robustness to initial occlusion using three apple-manipulation tasks and three occlusion severities. Each task–severity combination is evaluated for 10 trials, yielding 3\ \text{tasks}\times 3\ \text{severities}\times 10\ \text{trials}=90\ \text{trials per method}.
- Partial
- 16/30 (53.3)
- 8/30 (26.7)
- 29/30 (96.7)
- Medium
- 11/30 (36.7)
- 5/30 (16.7)
- 28/30 (93.3)
- Severe
- 8/30 (26.7)
- 4/30 (13.3)
- 26/30 (86.7)
- Overall
- 35/90 (38.9)
- 17/90 (18.9)
- 83/90 (92.2)
Physical result interpretation. The 60 trials per method in Table VII are a separate evaluation and are not merged with the occlusion aggregate. ActiveAlign improves over both baselines on every task, with 17/20 successes for cup grasping, 15/20 for container retrieval, and 13/20 for observe–open–grasp–place. Their per-task gains over Standard are 40, 45, and 40 pp, but the 20-trial intervals remain wide. We therefore use these results as evidence of broader task coverage rather than a precise estimate of performance over arbitrary physical tasks. Besides, the lower success on the longer task illustrates the limits of an informative initial view over an extended manipulation horizon. Fig. 5 shows the representative rollouts.
Table VIII aggregates three apple-and-occluder tasks at each severity. ActiveAlign succeeds in 83/90 trials (92.2%), compared with 35/90 (38.9%) for Standard and 17/90 (18.9%) for Naive Active. Their Wilson intervals are [84.8,96.2]%, [29.5,49.2]%, and [12.1,28.2]%, respectively. Relative to Standard, ActiveAlign gains 43.3, 56.7, and 60.0 pp under partial, medium, and severe occlusion.
\pi_{0.5} + ActiveAlign. > Rows show different tasks at varying occlusion levels, with active observation preceding manipulation.V LIMITATIONS
ActiveAlign is currently limited by its offline data-construction assumptions. Constructing aligned continuations requires a restorable simulator or a reconstructable physical workcell, as well as demonstration prefixes that remain replayable from the endpoints generated by the viewpoint module. Because training trajectories are retained only after successful alignment and replay, the adapted dataset represents a feasibility-conditioned subset of the deployment endpoint distribution. Difficult endpoints may therefore be underrepresented even though they can still occur at deployment. RLBench evidence is perception-only. The current system performs a single viewpoint-selection phase before manipulation. It does not reselect viewpoints in response to moving occluders. Extending observation-aligned adaptation to non-resettable environments, continuous viewpoint selection, and broader physical deployments remains an important direction.
VI CONCLUSION
ActiveAlign connects the decision of where to observe with the learned behavior needed to act from that observation. By constructing successful continuations from active-view endpoints, it enables existing VLA architectures to benefit from a pre-manipulation sensing phase. LIBERO and physical results show consistent gains, while matched-state and data-matched controls support benefits beyond prefix shortening and generic viewpoint coverage. The resulting lesson is that viewpoint selection and manipulation supervision should be designed together, even when the two policies are trained separately. Extending this construction beyond replayable prefixes and interleaving observation selection with manipulation are promising next steps.