ReEnvision: Future-Guided Selective Action Repair for Long-Horizon Manipulation
Long-horizon robotic manipulation requires robust execution over sequences of interdependent actions. Existing methods typically handle execution deviations by replanning the remaining action chunk or correcting actions step by step as new observations arrive.
From the abstract. Our summary is in progress.
Full paper
ReEnvision: Future-Guided Selective Action Repair for Long-Horizon Manipulation
Published under CC BY 4.0. Reproduced with attribution; original at arXiv:2610.17977 (PDF).
R E ENVISION : FUTURE-GUIDED SELECTIVE ACTION REPAIR FOR LONG-HORIZON MANIPULATION Xiyin Zeng*, Xuewei Huang* & Yi Lu Information Hub The Hong Kong University of Science and Technology (Guangzhou)
arXiv:2610.17977v1 [cs.RO] 13 Oct 2026
Weihao Yuan† School of Intelligence Science and Technology Nanjing University
Hao Wang† Information Hub The Hong Kong University of Science and Technology (Guangzhou)
ABSTRACT
Long-horizon robotic manipulation requires robust execution over sequences of interdependent actions. Existing methods typically handle execution deviations by replanning the remaining action chunk or correcting actions step by step as new observations arrive. However, a local deviation may affect only a few actions under distribution shift, yet replanning regenerates the entire chunk from an unfamiliar state, potentially introducing errors into otherwise valid actions. Stepwise correction may react too late, as it does not explicitly anticipate how the deviation affects subsequent actions. We propose ReEnvision, a future-guided selective action repair framework that updates a joint future-action belief using execution feedback. It then learns how the resulting belief revision affects each unexecuted action. The resulting belief revision enables selective repair of only the affected actions within a cached action chunk, preserving valid actions and reserving full replanning for severe deviations. On CALVIN, L-CALVIN, and RLBench, ReEnvision achieves 86.6% 5-step success, improves 10-step success from 56% to 63%, and reaches an average success rate of 88.8%, respectively. On five real-world manipulation tasks, it achieves an average success rate of 82.8%. Controlled repair experiments further demonstrate the benefits of localizing affected actions and preserving valid ones during execution. Our project page is available at https://reenvision.pages.dev.
INTRODUCTION
Long-horizon robotic manipulation requires robots to coordinate multiple dependent actions over an extended execution horizon, while preserving consistency across stages and recovering from intermediate deviations (Zhou et al., 2024; 2026). Such capability is important for real-world manipulation, where many tasks involve sustained, multi-stage interactions rather than isolated actions (Chen et al., 2024; Chai et al., 2026). However, reliable long-horizon execution remains challenging: an error at one stage can alter the state encountered by subsequent actions, causing later actions to become inappropriate and allowing errors to accumulate until the overall task fails (Tan et al., 2026). Long-horizon manipulation has been approached from multiple perspectives, including hierarchical planning, predictive world modeling, and adaptive execution (Chai et al., 2025; Ma et al., 2026). Hierarchical planning and control methods decompose long tasks into subtasks, while world models anticipate future states and use them to guide actions (Sridhar et al., 2026; Li et al., 2026a). More recently, adaptive execution methods incorporate new observations during execution to adjust planned behavior: Future Forward Dynamics Causal Attention (FFDC) (Wang et al., 2026b) verifies whether the current rollout should continue or be replanned, while Asynchronous Action Chunk Correction (A2C2) (Sendai et al., 2025) uses each newly observed frame to predict a residual update to the actions being executed. In contrast, Real-Time Robot Execution with Masked Action Chunking (REMAC) (Wang et al., 2026a) preserves actions that are already committed and resamples only the remaining portion of the chunk. Despite these advances, existing methods primarily 1
Initial Belief (Before execution) t Future latent z f
Observation & Task
post
Updated future latent z f,t 1 …
Action intent z
t a
ht
Execute actions
Executed
Stack the wine bottle
Updated Belief
Predict belief
Update belief
New Observation
htpost 1
post
Updated Action intent z a,t 1
Expected belief ht prior 1
Select affected actions and repair Revised action chunk
Mismatch !
Original action chunk
VS Expected future
at
at 1
Executed
at 2
at 3
at 4
Affected subset
htpost 1
New Observation
revise
htprior 1
Update the action sequence and continue execution.
at
at 1
Executed
at 2
at 3
at 4
Repaired actions
Figure 1: Overview of ReEnvision. ReEnvision maintains a joint future-action belief ht and revises it with new observations. The revision identifies affected actions in the cached chunk, enabling selective repair. Unlike methods that rely on full replanning, ReEnvision repairs only affected actions within the current chunk while preserving unaffected ones.
address execution deviations through rollout-level replanning, step-wise correction, or resampling the editable portion of an action chunk. They do not explicitly estimate which unexecuted actions are affected by an observed deviation. Under out-of-distribution (OOD) deviations, regenerating the entire remaining chunk from an unfamiliar intermediate state may disrupt established alignment and contact, amplifying local errors into downstream failures (Chen et al., 2026). Our analysis identifies pronounced replanning failures in approximately 11% of OOD sequences, with 55% occurring in direction-sensitive manipulation and 36% in contact- or pose-sensitive tasks. Meanwhile, step-wise correction may fail to anticipate how current deviations affect subsequent actions. Addressing this limitation requires reasoning about the expected future and planned actions together: execution errors change what happens next, and these changes can in turn make some planned actions no longer suitable (Ye et al., 2026). By jointly modeling these two signals, the robot can reason about how an observed deviation changes what should happen next and how that change affects subsequent actions. This makes it possible to determine whether the remaining action chunk requires a global update or only a local adjustment. When only part of the chunk is affected, selectively repairing those actions preserves unaffected ones without regenerating the entire sequence. Building on this idea, we propose ReEnvision, a future-guided selective action repair framework that revises a joint future-action belief and anticipates how the revision affects individual actions within a cached chunk as illustrated in Fig. 1. Given the current observation and task, a pretrained future generator predicts visual evolution, which is combined with the observed state to infer the corresponding action intent. Together, they form a joint future-action belief that conditions a diffusion policy to generate a multi-step action chunk. As new observations arrive, a latent dynamics model predicts how the joint belief should evolve following the executed action and updates the prediction using the observed outcome. The difference between the predicted and updated beliefs defines the belief revision. A temporal impact model then estimates how this belief revision affects each remaining action in terms of correction, validity, and uncertainty. These predictions enable selective repair of the affected actions while preserving unaffected ones, with full replanning reserved for broadly affected or highly uncertain suffixes. In experiments, ReEnvision achieves relative improvements of 2.5%, 12.5%, and 2.5% over the strongest reported baselines on CALVIN (Mees et al., 2022) 5-step success, L-CALVIN (Fan et al., 2025) 10-step success, and real-world manipulation, respectively, while achieving 88.8% average success on RLBench (James et al., 2019). Our contributions are summarized as follows: (i) We find that changes in the expected future typically affect only a local subset of actions within the remaining chunk; selectively repairing these actions is more effective than rewriting the entire chunk and enables correction before they are executed. 2
(ii) We propose ReEnvision, which learns how revisions in a joint future-action belief affect each unexecuted action and uses the predicted action-level impact to determine the scope of intervention, preserving unaffected actions while selectively repairing affected segments. (iii) Extensive experiments demonstrate that ReEnvision achieves 86.6% 5-step success on CALVIN, 88.8% average success on RLBench, and 82.8% average success on five real-world manipulation tasks, with a 12.5% relative improvement in 10-step success on L-CALVIN.
RELATED WORK
2.1
HIERARCHICAL PLANNING AND CONTROL
Hierarchical planning and control methods decompose long-horizon manipulation into subgoals or execution phases, thereby reducing the effective decision horizon and improving temporal consistency (Zhao et al., 2023). Hierarchical Action Models for Open-World Robot Manipulation (HAMSTER) (Li et al., 2025b) first predicts a coarse end-effector trajectory, then conditions a low-level policy on this trajectory to generate precise 3D actions. Similarly, Long-VLA (Fan et al., 2025) instead decomposes each subtask into moving and interaction phases, and generates actions conditioned on the current phase to improve transitions between subtasks. Going beyond static decomposition, Reflective Contrastive Alignment and Planning Architecture (ReCAPA) (Zeng et al., 2026) further introduces predictive correction across action, subgoal, and trajectory levels, allowing local deviations to be corrected before they propagate into cascading failures. Despite these advances, such methods primarily operate over action-centric representations, without explicitly modeling how the physical world should evolve toward the task goal.
2.2
PREDICTIVE WORLD MODELS
Predictive world models forecast how the environment will evolve under candidate actions and use these predicted futures to guide subsequent control (Feng et al., 2025). Latent Diffusion World Model (LaDi-WM) (Huang et al., 2025) uses a diffusion-based world model to predict future states in a learned latent space, then conditions a diffusion policy on these predictions to refine actions. More recently, Mixture of Latent Actions (MoLA) (Li et al., 2026c) focuses on grounding imagined futures into executable control, converting future visual transitions into latent actions through inverse dynamics before decoding them into robot commands. In contrast, Decoupled Visual Forward and Inverse Dynamics Pretraining (DeFI) (Zhang et al., 2026) decouples the two learning objectives, separately training forward dynamics to predict futures and inverse dynamics to infer actions. However, these methods typically use predicted futures to guide action generation, so execution deviations may render the remaining actions inconsistent with the updated future (Gao et al., 2026).
2.3
ADAPTIVE ACTION EXECUTION
Adaptive action execution mitigates the mismatch between the updated future and the remaining committed actions by using execution feedback for online correction or replanning (Liang et al., 2026; Yao et al., 2026). FFDC (Wang et al., 2026b) compares predicted visual futures and actions with real observations to decide whether to continue the current rollout or replan, while A2C2 (Sendai et al., 2025) uses visual feedback at each step to predict a residual correction to the currently executed chunk action. More selectively, REMAC (Wang et al., 2026a) preserves already committed actions while resampling the remaining editable portion of the action chunk to maintain consistency during execution. These methods improve adaptive execution through rollout-level replanning decisions, step-wise residual correction, or masked action generation. However, they do not explicitly characterize how changes in the expected future affect individual unexecuted actions within a cached chunk (Bi et al., 2026). Consequently, these corrections may unnecessarily modify unaffected actions or fail to anticipate downstream effects on subsequent actions. 3
GRU
rt
Ot 1
rt 1
Observation Encoder
htprior 1
prior t 1
Temporal Convolutional Network 1D Conv + FiLM + ReLU 1D Conv + FiLM + ReLU …
prior ht 1 htpost 1 ht 1
at 4
5 Selective repair
ht 1
1D Conv + FiLM + ReLU
Correction Head Validity Head Uncertainty Head
Cached actions At
Atrem
^ rem
htpost 1
at 3
rem
New observation Ot 1 …
at 2
K steps (e.g., 5)
(ht, at )
at 1
Affected Action Detection
at
Belief Revision via RSSM
Diffusion Policy
ht
At
v u
Affected mask
Repaired actions Atrem '
Action Intent z a
Task: Stack the wine bottle
Action Chunk Generation At:t K 1
Observation & Task Encoding Future Latent z f Stable Video Diffusion
Figure 2: In ReEnvision, joint future-action belief ht conditions action-chunk generation and is revised after each new observation. The resulting belief revision is used to predict per-action correction, validity, and uncertainty, enabling selective repair of the affected cached actions.
METHODOLOGY
3.1
FRAMEWORK OVERVIEW
To avoid both broad replanning from unfamiliar intermediate states and delayed step-wise correction, execution feedback should be used to anticipate how the expected future has changed and where its effects will appear in the remaining action chunk. This motivates localizing the affected actions in advance and revising only that portion while preserving the rest. To this end, we propose ReEnvision as shown in Fig. 2, which uses Stable Video Diffusion (SVD) (Blattmann et al., 2023) to predict visual evolution from the current observation and task instruction. Conditioned on the current state and anticipated future, an intent generator produces an action representation. Together with the future latent, it forms a joint belief that conditions a diffusion-based policy to generate an action chunk. As new observations arrive, a Recurrent State-Space Model (RSSM) (Hafner et al., 2019) updates the joint belief and measures its deviation from the expected belief transition. A Temporal Convolutional Network (TCN) (Bai et al., 2018) then uses this belief revision to estimate how each action in the remaining chunk is affected, leading to keep, selective correction, or replanning. 3.2
JOINT FUTURE-ACTION BELIEF
Given the current observation ot and language instruction l, a pretrained SVD model extracts a base future representation zft ≡ zfbase of the expected task-relevant evolution. During training, a target future representation zftarget is constructed from successful trajectories to represent the desired execution stage. A Multilayer Perceptron (MLP)-based action-intent generator predicts an action-intent representation zat, conditioned on the current observation, the base and target future representations, their difference ∆zf = zftarget − zft, and the task instruction. The generator is trained using an L1 loss against action-intent targets derived from successful action chunks. During training, positive, identity, and negated revision pairs are sampled equally, with identity pairs mitigating the training–inference mismatch under zero revision. At inference, the initial action intent is generated using a zero future revision, with zftarget = zft and ∆zf = 0, without requiring a ground-truth target future. The future and action-intent representations are combined into a joint future-action belief, ht = (zft, zat ),
(1)
which captures both the expected future and the corresponding action intent. To generate executable actions, zat is introduced as an additional conditioning signal to the pretrained diffusion policy. The original conditioning (ot, l, zft ) is extended to (ot, l, zft, zat ), while the training 4
objective remains the original diffusion action-prediction objective on ground-truth action chunks. The resulting policy produces a K-step action chunk At = [at, . . ., at+K−1 ]. 3.3
BELIEF REVISION
After generating an action chunk, the robot executes at and observes ot+1 as feedback on the resulting environment transition. Since the belief naturally evolves during execution, a change from ht to the next belief does not by itself indicate an execution deviation. An RSSM therefore models the expected belief transition and incorporates execution feedback to capture discrepancies between the predicted and observation-updated beliefs. Given the current belief ht and executed action at, the prior branch predicts the expected next belief hprior t+1 . The posterior branch additionally incorporates the new observation ot+1 to infer the updated belief hpost t+1 . During training, paired future–action latent representations are collected from consecutive execution steps of the same rollout. The current belief ht and target belief htarget correspond to the current and next execution steps, respectively. Their difference defines the teacher latent displacement, ∆h∗ = htarget − ht .
(2)
Both RSSM branches are supervised to predict this displacement: post ∗ ∗ LRSSM = ℓ(hprior t+1 − ht, ∆h ) + ℓ(ht+1 − ht, ∆h ),
(3)
where ℓ denotes the Smooth L1 loss. The prior learns the expected belief transition without access to the new observation, while the posterior incorporates the observed outcome to refine the predicted update. During execution, the belief revision is defined as the difference between the posterior and prior prior beliefs, ∆ht+1 = hpost t+1 − ht+1 . This revision captures the observation-induced update to the expected belief transition and guides selective action repair. The posterior belief is then maintained for subsequent execution, ht+1 ← hpost t+1 . 3.4 3.4.1
SELECTIVE ACTION REPAIR BELIEF-GUIDED SUFFIX REPAIR
= [at+1, . . ., at+K−1 ]. Since the After executing at, the remaining actions are cached as Arem t belief revision ∆ht+1 may affect only part of this suffix, a TCN Sψ predicts an action correction ∆âk, validity score vk, and uncertainty uk for each remaining action: rem ∆Ârem , v, u = Sψ ot+1, hprior , A , ∆h , (4) t+1 t t t+1 where k = 1, . . ., K − 1 indexes the cached actions, larger vk indicates higher action validity, and larger uk indicates greater uncertainty in determining whether the action requires correction. The affected actions are selected by optimizing a binary mask m ∈ {0, 1}K−1 : m̂ = arg min m
λv
K−1 X
(1 − mk )(1 − vk ) + λm
k=1
+ λc
K−2 X
K−1 X
mk
k=1
|mk+1 − mk | + λu
k=1
K−1 X
(5)
m k uk .
k=1
The objective penalizes leaving low-validity actions unrepaired, modifying unnecessary actions, fragmented repair regions, and repairing highly uncertain actions. The corresponding weights are set to λv = 1, λm = 0.5, λc = 0.15, and λu = 0.15, favoring compact and reliable repair regions. The optimized mask determines whether the cached suffix should be kept, selectively repaired, or PK−1 1 discarded. The repair ratio r = K−1 k=1 m̂k measures the fraction of affected actions, s denotes the number of disconnected affected segments, and umax = max1≤k≤K−1 uk measures the maximum uncertainty across the entire suffix. Full replanning is triggered when umax ≥ η. Otherwise, the suffix is retained when no action is selected, and selective repair is applied when r ≤ ρ and 5
Table 1: CALVIN (Mees et al., 2022) ABC→D results. We report the average success rate over 1,000 rollouts and the average number of consecutively completed tasks (Avg. Len.). ReEnvision demonstrates clear improvements over the baselines, particularly on longer-horizon sequences.
Method
∗ π0.5 (Intelligence et al., 2025)
VPP (Hu et al., 2025) REMAC (Wang et al., 2026a) PALM (Liu et al., 2026) DeFI (Zhang et al., 2026) VITA (Ma et al., 2026) ReEnvision
Avg. Len. ↑
94.8 96.5 96.0 96.9 97.9 99.1 97.8
87.4 90.9 91.4 93.8 94.2 94.9 94.5
78.2 86.6 88.3 89.3 90.7 91.2 93.0
71.7 82.0 84.9 85.9 87.0 87.8 90.1
64.3 76.9 78.4 82.0 81.2 84.5 86.6
3.97 4.33 4.39 4.48 4.51 4.57 4.62
s ≤ smax . If either condition is violated, the cached suffix is discarded and fully replanned. Here, ρ, smax, and η control the maximum repair extent, fragmentation, and uncertainty, respectively. For selective repair, only the selected actions are updated: Arem′ = Arem + m̂ ⊙ ∆Ârem , t t t
(6)
with the correction magnitude clipped to a bounded range and smoothing applied within the selected repair region to maintain trajectory continuity. Under full replanning, the cached suffix is discarded, and a new action chunk is generated by the diffusion policy conditioned on the posterior belief hpost t+1 . 3.4.2
ACTION IMPACT MODEL TRAINING
Sψ is trained on outcome-guided action pairs to approximate belief-induced corrections. Each pair consists of lower- and higher-return action chunks, A− and A+, generated by the same policy under matched observation, task, diffusion noise, and cache conditions. A− = Aneg,
A+ = Apos .
(7)
Both A− and A+ are full action chunks generated by the same frozen policy. Their first K − 1 rem + . Their actions, denoted by A− rem and Arem, are temporally aligned with the cached suffix At − + ∗ difference, ∆Arem = Arem − Arem, provides an outcome-guided correction from lower- to higher. return actions. At inference, pre-revision actions are retrieved directly from the cached suffix Arem t For each remaining action k, the validity target indicates whether its outcome-guided correction magnitude remains within a predefined threshold: vk∗ = I [∥∆a∗k ∥2 ≤ τ ], (8) where k = 1, . . ., K − 1 indexes the cached suffix, and vk∗ = 1 indicates that the outcome-guided correction magnitude does not exceed the threshold τ . An uncertainty target u∗k is further constructed from the distance between ∥∆a∗k ∥2 and τ, assigning higher uncertainty to actions closer to the validity decision boundary and lower uncertainty to those farther away. The threshold τ is used only to construct offline supervision targets. During inference, predicted validity and uncertainty guide structured mask selection without directly thresholding predicted corrections. Sψ is jointly trained to predict outcome-guided action corrections, validity, and uncertainty: LTCN = ∆Ârem − ∆A∗rem t
+ λval BCE (v, v∗ ) + λunc ∥u − u∗ ∥1 .
(9)
The loss weights are set to λval = 1.0 and λunc = 0.1. This outcome-guided supervision is used to approximate the action changes induced by belief revision. During execution, predicted corrections are applied only to the selected actions in Eq. 6, while unaffected actions are preserved. 3.5 3.5.1
TRAINING AND INFERENCE TRAINING
The framework is trained in stages. Future representations from the pretrained SVD are first used to train the action-intent generator, after which the diffusion policy is further trained to generate action 6
Table 2: Comparison with prior methods on the L-CALVIN (Fan et al., 2025), using 100 evaluation sequences of 10 consecutive instructions under the ABC→D setting.
Method
Tasks Completed in Sequence
VLAS (Zhao et al., 2025) RoboVLMs (Li et al., 2026b) REMAC (Wang et al., 2026a) DeFI (Zhang et al., 2026) Long-VLA (Fan et al., 2025) ReEnvision
Avg. Len.
10
0.88 0.90 0.94 0.98 1.00 0.99
0.80 0.80 0.90 0.96 1.00 0.98
0.72 0.78 0.82 0.92 0.98 0.95
0.58 0.70 0.74 0.86 0.91 0.92
0.49 0.64 0.71 0.75 0.85 0.89
0.46 0.60 0.58 0.71 0.82 0.83
0.34 0.51 0.54 0.59 0.79 0.77
0.28 0.41 0.50 0.57 0.70 0.72
0.19 0.36 0.47 0.52 0.63 0.68
0.16 0.34 0.41 0.44 0.56 0.63
4.90 6.04 6.61 7.30 8.24 8.36
Table 3: Performance comparison on 18 RLBench (James et al., 2019) manipulation tasks, with 50 evaluation episodes per task. We report task success rates (%) and the average success rate (Avg. SR) across all 900 episodes. Best results are shown in bold. Close Insert Light Meat off Open Place Shape Place Avg. SR Jar Peg Bulb Grill Drawer Cups Sorter Wine
Method
HDFlow (Gireesh et al., 2026) TVVE (Bai et al., 2026) SpatialActor (Shi et al., 2026) BridgeVLA (Li et al., 2025a) ReEnvision
87.5 86.6 87.4 85.7 88.8
96.0 100.0 94.0 98.0 98.0
93.3 98.0 93.3 80.0 82.0
88.7 86.0 88.7 86.0 90.0
98.7 94.0 98.7 100.0 100.0
82.0 90.0 82.0 98.0 90.0
56.7 54.0 56.7 48.0 60.0
73.3 62.0 73.3 38.0 50.0
94.7 92.0 94.7 90.0 96.0
Method
Put Push Put in Put in Reach & Slide Stack Stack Sweep Turn Buttons Groceries Drawer Safe Drag Block Blocks Cups Dustpan Tap
HDFlow TVVE SpatialActor BridgeVLA ReEnvision
100.0 100.0 100.0 100.0 90.0
72.0 74.0 72.0 80.0 92.0
98.7 100.0 98.7 88.0 94.0
96.7 78.0 96.7 100.0 100.0
100.0 100.0 100.0 100.0 94.0
91.3 100.0 91.3 98.0 82.0
56.0 74.0 56.0 76.0 94.0
81.3 64.0 81.3 80.0 90.0
100.0 92.0 100.0 90.0 96.0
95.3 100.0 95.3 92.0 100.0
chunks conditioned on both future and action-intent representations. Rollouts from the resulting policy are then used to train the RSSM, with both prior and posterior branches supervised by the teacher latent displacement ∆h∗ = htarget − ht . Finally, matched lower- and higher-return action chunks from the frozen diffusion policy provide outcome-guided correction targets for training Sψ to predict per-action corrections, validity, and uncertainty. 3.5.2
INFERENCE
At initialization, the current observation and task instruction are used to construct the initial joint belief and generate an action chunk. After each action is executed, the RSSM predicts the next belief and incorporates the new observation to obtain the posterior belief. The pre-revision actions are retrieved directly from the cached suffix Arem , without regeneration. Given the belief revision t ∆ht+1 and cached suffix, Sψ predicts per-action corrections, validity, and uncertainty. The suffix is retained when no action is selected and uncertainty is low, and selectively repaired when the affected region is sufficiently localized. Full replanning is triggered when the affected region is too large, fragmented, or uncertain, generating a new action chunk conditioned on hpost t+1 .
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
We evaluate ReEnvision on CALVIN (Mees et al., 2022), L-CALVIN (Fan et al., 2025), and RLBench (James et al., 2019), using DeFI (Zhang et al., 2026) as the backbone for CALVIN/L-CALVIN and BridgeVLA (Li et al., 2025a) for RLBench. On CALVIN, we follow the ABC→D protocol and evaluate 1,000 five-task sequences using 10-step, 7-DoF relative action chunks from RGB observa7
Table 4: Real-world manipulation performance on five tasks using an ARX X5 robot. Each method is evaluated over 50 trials per task. Success Rate (%)
Method
Diffusion Policy (Chi et al., 2024) ∗ π0.5 (Intelligence et al., 2025) BridgeVLA (Li et al., 2025a) DeFI (Zhang et al., 2026) ReEnvision
Pen
Color
Drawer
Towel
Flower
Average
78 90 88 86 90
56 70 70 68 76
78 80 86 78 84
80 94 74 82 88
52 70 68 66 76
68.8 80.8 77.2 76.0 82.8
Predict
Flower dropped!
future
ot
Action index
at t
at 1
t 1
t2
t 3
at 3
at 4 t4
ak*
0.03
0.05
0.04
0.28
0.31
0.73
0.84
0.78
0.30
0.36
0.12
0.15
0.11
0.46
0.38
(teacher)
Validity (pred.) Uncertainty (pred.) Teacher mask m
Predicted mask m
at 2
ot 2 Revise belief Stop inserting Pick up the flower Revise actions
at 3
t 3
at 4 t4
0.09
0.06
0.77
0.82
0.17
0.14
Figure 3: Visualization on a flower placement task. ReEnvision detects affected actions following an unexpected flower drop and selectively repairs the remaining action chunk through belief revision. For visualization, the teacher correction magnitudes ∥∆a∗k ∥2 are rescaled for readability, with actions exceeding 0.10 on the rescaled scale marked as affected. tions, proprioception, and language instructions. We report success rates for 1–5 consecutive tasks and average sequence length. L-CALVIN extends the same policy to ten-task sequences, reporting SR1–SR10 and average length. On RLBench, we evaluate 18 tasks with 50 episodes each and report per-task and mean success rates. All results are averaged over three random seeds. For real-world evaluation, we use an ARX X5 robot on five tasks: pen placement, color sorting, sequential drawer placement, towel folding and placement, and sequential flower placement. We collect 150 kinesthetic demonstrations per task. Two Intel RealSense D405 cameras capture RGB observations at 30 Hz and 640 × 480 resolution. Each policy receives both RGB views, a language instruction, and a 7-D proprioceptive state. 4.2
ABLATION STUDIES
We use DeFI and BridgeVLA as the baselines. For belief representation, the diffusion-policy conditioning is varied while the rest of ReEnvision is kept fixed: zf only for DeFI, za only for action-only, and ht = (zft, zat ) for joint belief. Belief revision ∆h, uncertainty uk, and the full replanning are further ablated, with selective repair replaced by full replanning in the latter. To isolate the effect of repair location, we fix the observation, task condition, and diffusion noise post while changing the belief from hprior t+1 to ht+1 . Their paired policy responses define the reference ∗ correction ∆Arem, with actions above threshold τ forming the reference mask m∗ . No Repair leaves the cached suffix unchanged without applying ∆A∗rem . For the remaining reference-mask controls, the same ∆A∗rem is used while only the repair mask is varied: Selective w/ m∗ repairs the affected positions, Random Mask repairs the same number of randomly selected positions, Complement Mask repairs positions outside m∗, and Whole Suffix repairs the entire remaining suffix. ReEnvision applies the TCN-predicted mask m̂ together with the predicted correction ∆Ârem . t 4.3
ANALYSIS
As shown in Table 1, ReEnvision achieves the best performance from 3/5 onward and the highest Avg. Len. of 4.62, with larger gains at longer horizons. This trend is clearer on L-CALVIN (Ta8
Table 5: Ablation study of ReEnvision on RLBench, CALVIN, and L-CALVIN.
Methods
RLBench
CALVIN
L-CALVIN
Avg. SR ↑
5-Step ↑
Avg. Len. ↑
Avg. Len. ↑
5-Step ↑
10-Step ↑
Baseline
85.7
81.2
4.51
7.30
0.75
0.44
zf only za only w/o belief revision w/o uncertainty w/ full replanning ReEnvision (zf, za )
84.6 88.4 87.4 88.0 88.9 88.8
84.7 85.2 82.6 85.2 82.1 86.6
4.58 4.57 4.54 4.60 4.53 4.62
8.22 8.26 7.96 8.32 7.74 8.36
0.85 0.80 0.83 0.90 0.77 0.89
0.55 0.56 0.50 0.61 0.51 0.63
Table 6: Repair-location control study on RLBench, CALVIN, and L-CALVIN.
Methods
No Repair Random Mask Complement Mask Selective w/ m∗ Whole Suffix ReEnvision
RLBench Avg. SR ↑ 85.7 83.1 77.4 88.8 81.2 88.8
CALVIN 5-Step ↑ Avg. Len. ↑ 81.2 81.6 78.8 85.9 80.7 86.6
4.51 4.48 4.39 4.65 4.50 4.62
L-CALVIN Avg. Len. ↑ 5-Step ↑ 10-Step ↑ 7.30 7.59 6.85 8.45 7.15 8.36
0.75 0.80 0.69 0.91 0.80 0.89
0.44 0.49 0.38 0.66 0.43 0.63
ble 2), where ReEnvision reaches an Avg. Len. of 8.36 and improves the later stages. On RLBench (Table 3), it improves the reproduced BridgeVLA baseline from 85.7% to 88.8% average success rate. In real-world experiments (Table 4), ReEnvision achieves the highest average success rate of 82.8% across five manipulation tasks, with leading or tied-leading results on Pen, Color, and Flower. Overall, selective repair reduces error accumulation by preserving valid actions: 86.8% of cached suffixes are kept unchanged, 11.1% selectively patched, and only 2.1% fully replanned. Table 5 shows consistent gains over the corresponding backbone baselines across all three benchmarks, especially in longer-horizon settings. Joint (zf, za ) conditioning also outperforms either representation alone. Full replanning is slightly better on RLBench (88.9 vs. 88.8), while removing uncertainty gives a marginally higher L-CALVIN 5-step success (0.90 vs. 0.89); ReEnvision remains stronger on the longer-horizon metrics. In addition, Table 6 further highlights the importance of repair localization. On L-CALVIN, Selective w/ m∗ reaches an Avg. Len. of 8.45, compared with 7.59 for Random Mask and 6.85 for Complement Mask, showing that repairing the affected actions is more effective than modifying random or unaffected ones. Fig. 3 illustrates how ReEnvision selectively repairs affected actions following an unexpected flower drop. 4.4
DISCUSSION
Our results suggest that belief changes usually affect only a local part of the cached action suffix rather than making the whole suffix invalid. The repair-location study further shows that repair location is critical: Selective w/ m∗ outperforms Random Mask, Complement Mask, and Whole Suffix, supporting selective modification of affected actions while preserving valid ones. The ablations also show complementary roles for future prediction and action intent: the former captures expected task evolution, while the latter represents how the policy realizes it. Their combination therefore provides a stronger basis for deciding which remaining actions require revision. Full replanning remains competitive on RLBench, where tasks are shorter and rebuilding the suffix can correct errors with less disruption. In longer-horizon settings, repeated replanning may discard useful actions and introduce unnecessary changes, making ReEnvision increasingly advantageous as the execution horizon grows. The learned ReEnvision policy also performs close to Selective w/ m∗, suggesting that these useful repair locations can be recovered without access to the reference mask. ReEnvision estimates belief-induced action changes with a lightweight TCN and invokes full replanning only under severe deviations. Despite these optimizations, ReEnvision still introduces about 30% additional inference latency over the baseline. In addition, the outcome-guided action pairs 9
provide action-level supervision, but they do not establish whether each individual repair improves task success. Task-level success is difficult to use directly because it gives only a final outcome, making action-level credit assignment hard. Future work could first localize a small repair region using belief changes, then use task outcomes to refine which actions should be modified.
CONCLUSION
We presented ReEnvision, which combines the expected future and action intent into a joint belief, updates this belief with new observations, and estimates how the resulting revision affects the remaining action sequence. This enables the policy to preserve valid actions, selectively repair affected ones, and reserve full replanning for severe deviations. Experiments on CALVIN, L-CALVIN, and RLBench, together with real-robot evaluations, show consistent improvements over the corresponding backbone policies, with larger gains over longer horizons. The repair-location study further shows that performance gains depend on correctly localizing belief-induced changes within the action chunk, so that affected actions are corrected while valid actions are preserved. Future work may improve affected-action estimation and extend selective repair to more diverse real-world settings.
AI USE STATEMENT In this work, generative AI tools were used to assist with literature search and synthesis and to improve the clarity, readability, and wording of the manuscript. All AI-assisted content, including literature-related information and textual revisions, was reviewed and verified by the authors against the original sources where applicable. The authors take full responsibility for the final content, claims, and presentation of this work.
REPRODUCIBILITY STATEMENT We provide the model architecture, training objectives, and staged training and inference procedures in Section 3. Section 4 details the benchmark protocols, evaluation metrics, and real-world experimental setup. Appendix A includes additional experimental analyses, runtime measurements, and hyperparameter sensitivity studies. All source code and pretrained model weights will be publicly released upon acceptance to facilitate reproducibility.