Back on Track：用世界模型规划在部署阶段纠正黑盒机器人策略

Abstract

State-of-the-art visuomotor policies are increasingly capable of executing complex tasks, yet can fail during deployment in ways they cannot predict or recover from on their own. Improving their performance and robustness typically requires collecting more data, retraining, or modifying the policy architecture, all of which are time- and resource-intensive in practice. We instead propose an offline-trained monitoring-and-intervention framework that improves a policy during deployment using only black-box query access. From a limited set of labeled on-policy rollouts, we learn a policy-conditioned liveness value that measures the nominal policy’s progress toward task completion; we additionally train a compact latent world model using these rollouts together with play data. At deployment, the liveness value continuously monitors execution. When it detects a loss of progress, a short-horizon planner searches the world model for corrective actions using the same value as its planning objective, after which control returns to the nominal policy. We evaluate our method on the LIBERO-Goal task suite and on a Franka Panda manipulator, using the vision-language-action model \pi_{0.5} and a diffusion policy as nominal policies, and demonstrate significant improvements in task success rates, while supporting real-time control. Additional results and videos are available on the project website.

†

I Introduction

Visuomotor policies are increasingly capable of complex tasks from rich sensory observations [1, 2, 3], yet they remain brittle at deployment and can fail in ways they cannot recover from on their own.

The dominant response is to scale policy learning itself, with more data, more compute, and larger architectures [4, 5].

While effective, these approaches are expensive and require access to the policy’s parameters and training pipeline — access a practitioner increasingly does not have, since strong policies are increasingly served as inference APIs or shipped with closed weights. When the weights are out of reach, deployment-time correction is a promising avenue.

We therefore ask how a policy that has already been trained and deployed can be improved without modifying the policy itself. We assume only black-box access: the parameters, architecture, and action-generation procedure are unknown, and the policy can only be queried for actions. Fine-tuning and methods that steer the policy’s internal sampling are therefore inapplicable, and improvement must come from an external wrapper that monitors execution and intervenes through actions alone. Two requirements make this demanding: policy rollouts are costly, so the wrapper must be learned from few labeled on-policy executions, and all monitoring and planning must keep pace with the control loop while the policy executes.

Our approach is based on monitoring-and-intervention, or filtering [6, 7], which in control theory most often enforces safety: a filter watches the nominal policy and overrides it only when a constraint would be violated [8, 9]. We instead filter for task completion. Rather than characterizing and avoiding every failure mode, an intervention seeks to return the system to a state from which the nominal policy can complete the task. The intuition is that a capable policy need not be replaced or retrained when it falters; a short corrective action is often enough to put it back in its basin of success.

The framework has two components. A monitor tracks the nominal policy’s execution and detects when it is no longer progressing toward task completion; we instantiate it with a policy-conditioned liveness value function [10], learned from on-policy rollouts with sparse episode-level outcome labels, that estimates how close the policy is to completing the task. When the value indicates a loss of progress, a planner briefly overrides the policy and searches a compact latent world model — trained on the same rollouts plus auxiliary play data — for a short corrective action sequence, then hands control back.

Critically, the same value that triggers intervention also scores the futures the world model imagines, directing the planner toward states from which the nominal policy can recover.

Methods that generate video futures report seconds-scale intervention times [11, 12], forcing either a pause or a control rate too low for responsive manipulation. We therefore design around deployment latency: the world model is compact, candidate futures are evaluated in parallel, and planning emits an action chunk rather than a single action. In our hardware experiment a planning cycle takes 108 ms, which, amortized over the chunk, keeps the monitor–planner loop above 10 Hz.

We summarize our contributions as follows. First, we formulate deployment-time improvement of visuomotor manipulation policies under a strict black-box access model and cast it as liveness filtering: an external wrapper intervenes briefly to return a fixed nominal policy to its basin of success. Second, we present a real-time instantiation in which a single policy-conditioned liveness value both monitors execution and guides corrective planning in a compact latent world model, with a data-informed action proposal that makes the search efficient. Third, we validate it across policy classes: on the ten-task LIBERO-Goal suite our method improves \pi_{0.5} [13] from 72.6\% to 81.5\%, outperforming offline-RL and safety-filtering baselines, and on a Franka Panda manipulator it improves a diffusion policy [14] from 40\% to 60\% while operating within the policy’s control loop.

II Related Work

We group prior work by the access and supervision it assumes, noting in each case how our problem setting differs. One line of work intervenes inside the policy’s action-generation process, requiring white-box access to the policy. Latent Policy Barrier [15] treats the latent embeddings of expert demonstrations as a barrier separating in- from out-of-distribution states, using a learned dynamics model to steer a diffusion policy’s denoising noise back toward the expert distribution at inference, equating in-distribution behavior with task success. VLS [16] steers a pretrained policy’s denoising process with guidance from a vision-language model; and [17] replaces a diffusion policy’s Gaussian sampling noise with a distribution learned via reinforcement learning (RL). These methods intervene directly in action generation, but rely on the policy’s internal representations or sampling procedure and so do not apply when the nominal policy is only a black box.

A second class of methods treats the nominal policy as a black box and intervenes only through its inputs or outputs. RoboMonkey [18] and Steer Your Generalists [19] rerank action candidates sampled from the policy. This avoids modifying the policy’s weights, but requires multiple policy queries at every decision point, which is prohibitive for real-time control: a single diffusion-policy query alone takes 180 ms in our hardware setting. We instead query \pi_{b} once per intervention and obtain candidate diversity from our own proposal distribution rather than from the policy’s sampler.

Most closely related to our work are methods that use a learned world model to improve a fixed policy at deployment. Latent Policy Steering [20] continuously searches a latent world model for improved action sequences. In contrast, our monitor invokes planning selectively and returns control to the nominal policy after a short correction, limiting unnecessary computation and intervention when the nominal policy is progressing successfully.

DreamAvoid [12] ranks futures from a pretrained video world model at annotated critical phases, and VLA-in-the-Loop [11] specializes the paradigm to grasping, rolling back to an earlier state on a predicted grasp failure. Both are scoped to predefined critical events and to supervision we do not assume: access to the VLA’s sampling procedure plus critical-phase, progress, and failure-boundary labels for the former, grasp-feasibility labels and state rollback for the latter. We assume only black-box access and learn a policy-conditioned liveness value from sparse episode-level outcomes; the same value both detects loss of progress and guides planning, so intervention is not confined to predefined events, such as the wrong-task behavior in Fig. 2. Their video models are also slower at deployment, reporting 2 s (DreamAvoid) and 20 s (VLA-in-the-Loop) per intervention against our 108 ms planning cycle.

Monitoring-and-intervention filters have a principled basis in reachability analysis [8], which formalizes intervention under known dynamics: a filter overrides the policy only when needed to keep the system within a desired set of states. This machinery supports the dual objectives of safety (avoiding a failure set) and liveness (reaching a goal set [7]). We focus on the latter.

Classical liveness filtering computes the reachable set exactly via dynamic programming [7], but this is intractable for high-dimensional visual observations and the contact-rich dynamics of manipulation. We therefore build on the learned policy-conditioned liveness value of [10], which can be estimated from limited policy-rollout data with sparse, episode-level outcome labels. Whereas that prior work uses the liveness value for offline policy evaluation, we use it here both to detect loss of progress during execution and, critically, to guide corrective planning in a learned world model.

World models differ in what they predict, and that choice sets the deployment cost. Video-generation models synthesize pixels, which is expensive enough to put them at seconds-scale latency; DreamerV3 predicts distributions over discrete latent codes [21]; feature-prediction models forecast the patch-level features of a pretrained visual encoder, as in DINO-WM [22]. We take the last approach and keep the model compact, since each intervention must evaluate many imagined action sequences within a real-time budget.

III Problem Formulation

We consider the problem of deployment-time policy improvement. Given an episodic robotic task \mathcal{I} and a fixed visuomotor policy \pi_{b} trained to perform \mathcal{I}, our goal is to increase the task success rate of \pi_{b} during execution without modifying or retraining the policy itself. Our formulation applies to episodic tasks with a measurable completion outcome; in this work, we focus on visuomotor manipulation.

We assume black-box access to \pi_{b}: we do not know its parameters, architecture, or internal representations. The only information available is a dataset \mathcal{D}=\{(s^{i}_{1:T_{i}},\,a^{i}_{1:T_{i}-1},\,y^{i})\}_{i=1}^{N} of N rollouts generated by executing \pi_{b} on \mathcal{I}. Each rollout i consists of a state-action trajectory of length T_{i} and an episode-level binary outcome y^{i}\in\{0,1\} indicating whether the task was completed. These sparse outcome labels are the only task-specific supervision assumed by our method. To learn a dynamics model, we may additionally use an auxiliary dataset \mathcal{D}_{\mathrm{play}} of state-action transitions, without requiring outcome labels or any specific generation policy.

We formulate deployment-time improvement as a monitoring-and-intervention problem. During execution, the nominal policy \pi_{b} acts by default. An external monitor \mathcal{M} observes the execution of \pi_{b} and determines whether intervention is necessary. When \mathcal{M} triggers, a fallback policy {\pi_{f}} temporarily overrides \pi_{b} and applies a short corrective intervention, after which control is returned to \pi_{b}. Neither component accesses or modifies the internals of the nominal policy. Our objective is to design \mathcal{M} and {\pi_{f}} from \mathcal{D} and optionally \mathcal{D}_{\mathrm{play}} to maximize task completion probability within the latency budget of the deployment control loop.

IV Method

We now present a concrete instantiation of the monitoring-and-intervention framework. A shared, policy-conditioned liveness value couples its two components: a monitor \mathcal{M}, which detects when the nominal policy \pi_{b} has lost progress toward task completion, and a short-horizon planner, acting as the fallback policy {\pi_{f}}, which searches a learned world model \phi for a corrective action sequence under that same value. On trigger, the planner overrides \pi_{b}, executes a short correction, and returns control to the nominal policy.

IV-A Monitor

Let o denote the image observation and q the proprioceptive state provided to \pi_{b}. We define the state as s=[\psi(o),\ q], where \psi(\cdot) is an image encoder (e.g., SigLIP2 [23]). As the monitor, we employ the liveness value function V_{\pi_{b}} of the nominal policy \pi_{b}, introduced in [10]. This value is learned from the policy-rollout dataset \mathcal{D} using the Bellman update

V_{\pi_{b}}(s)=(1-\gamma)+\gamma\min\{l(s),V_{\pi_{b}}(s^{\prime})\}

where s^{\prime} is the next state, \gamma is the discount factor, and l(s) is a sparse goal function that is -1 at a goal state and +1 otherwise. In practice, we construct l from the episode-level outcome labels in \mathcal{D}: the final state of a successful episode is labeled as a goal, while all other states receive +1. Unlike a standard value function that sums discounted reward along a trajectory, Eq. (1) uses the \min operator to propagate goal attainment backward through the nominal policy’s rollouts. The resulting value V_{\pi_{b}} is policy-conditioned: it characterizes progress toward task completion under the future behavior of \pi_{b}, rather than progress under an arbitrary controller. It takes values in [-1,1], falling toward -1 as \pi_{b} approaches completion and rising toward +1 otherwise, so the monitor triggers on high values and the planner minimizes V_{\pi_{b}}.

Because the learned value is noisy — especially when \mathcal{D} is small — we build the monitor from an ensemble of value networks, whose mean prediction we denote \overline{V}_{\pi_{b}}(s), together with temporal smoothing over a window of size W. Specifically, at time t, we average the current prediction with the previous W-1 predictions,

\widetilde{V}(s_{t})=\frac{1}{W}\sum_{j=0}^{W-1}\overline{V}_{\pi_{b}}(s_{t-j}),

using W=3 in our experiments. The monitor triggers whenever \widetilde{V}(s_{t})>\eta_{t}, where \eta_{t} is a time-varying threshold. We decay \eta_{t} from +1 to a terminal value \hat{\eta}=0 according to a sigmoid schedule. Concretely \eta_{t}=[1+\exp((t-t_{c})/\sigma)]^{-1} with t the frame index within the episode and t_{c}\!=\!60, \sigma\!=\!10, so \eta_{t} falls from 0.998 at the first frame to below 0.01 by frame 120 to allow roughly three seconds for the manipulator to leave its initial configuration, during which the liveness value is high even on successful executions and an intervention would be a false alarm. The same schedule is used for all ten tasks. We ablate the ensemble, temporal smoothing, and threshold schedule in Tab. IV.

IV-B Planner

The planner optimizes an action sequence over a horizon H and executes only its first h\leq H actions before returning control. This short horizon matches the fallback policy’s role: it seeks a corrective intervention rather than attempting to complete the task itself.

IV-B1 Dynamics Model

Planning must run inside the control loop, so we learn a world model \phi over the compact state representation s rather than in pixel space: given a state–action pair (s_{t},a_{t}) it predicts the next state, \hat{s}_{t+1}=\phi(s_{t},a_{t}). We parameterize \phi as a 29.6M-parameter, four-layer ViT with sinusoidal positional encodings [22]. This prioritizes inference latency over capacity: each planning cycle evaluates many imagined trajectories while the robot executes.

We train \phi using two sources of data. The policy-rollout dataset \mathcal{D} provides transitions from regions of the state-action space visited by \pi_{b}. We supplement these transitions with auxiliary play data \mathcal{D}_{\mathrm{play}}, which broadens action coverage beyond the nominal policy’s behavior, a common practice in world model training [24, 25].

To improve multi-step prediction with a compact model, we train \phi autoregressively under a rollout-length curriculum. At curriculum length \kappa the model is applied to its own predictions from \hat{s}_{t}=s_{t}, minimizing \mathcal{L}_{\mathrm{WM}}=\sum_{j=1}^{\kappa}\left|\hat{s}_{t+j}-s_{t+j}\right|_{2}^{2}. The curriculum gradually increases \kappa from 1 to 16, first learning accurate local transitions and then exposing the model to its own prediction errors over longer rollouts.

IV-B2 Planning Objective

The planning objective J(\cdot) encodes the desired effect of an intervention: returning the system to states from which the nominal policy can complete the task. Given the current state s_{0} and a candidate action sequence a_{0:H-1}, the world model generates the imagined trajectory \hat{s}_{1}=\phi(s_{0},a_{0}) and \hat{s}_{i}=\phi(\hat{s}_{i-1},a_{i-1}) for i>1. We score the candidate by summing the ensemble-mean liveness value along this trajectory:

J(a_{0:H-1})=\sum_{i=1}^{H}\overline{V}_{\pi_{b}}(\hat{s}_{i})

The planner minimizes Eq. (3). Using the cumulative value, rather than only the terminal prediction, favors action sequences that make sustained progress throughout the imagined rollout. Critically, \overline{V}_{\pi_{b}} is the same policy-conditioned value used by the monitor. The planner therefore does not seek to complete the task on its own; it searches for states that are favorable under the subsequent behavior of \pi_{b}.

IV-B3 Planning Algorithm

We minimize Eq. (3) with the Cross-Entropy Method (CEM), a parallelizable sampling-based optimizer suited to the non-smooth, non-convex objective a learned value function induces. Candidates must be diverse enough to find a correction, yet implausible actions push the world model off its training distribution and yield exploitable prediction errors; we therefore sample from a data-informed proposal centered on the nominal policy’s action.

Vanilla CEM perturbs each action coordinate independently with isotropic Gaussian noise, which for action chunks yields temporally incoherent candidates; iCEM [26] instead samples colored noise with a 1/f^{\beta} spectrum, imposing smoothness as a generic prior. However, this prior is not informed by the temporal and cross-dimensional correlations present in task-relevant actions.

We instead estimate these correlations directly from the rollout data. We flatten each training-time action chunk of horizon H and action dimension D into a vector in \mathbb{R}^{HD} and compute its empirical covariance \Sigma. Let V_{r} and \Lambda_{r} contain the leading r eigenvectors and eigenvalues of \Sigma. We sample an action perturbation as

\delta=s\,V_{k}\Lambda_{k}^{1/2}\varepsilon,\qquad\varepsilon\sim\mathcal{N}(0,I_{k}),

where we use the k of the HD eigenvectors that explain 98\% of the measured variance (e.g., k\!\!=\!\!13 for H\!\!=\!\!10,D\!\!=\!\!6 for our \pi_{0.5} LIBERO case), and set the exploration scale to s\!=\!0.6. The gripper dimension is near-binary, so a continuous Gaussian subspace models it poorly; we leave it unperturbed and use the nominal policy’s gripper command. This proposal biases the search toward temporally coherent directions supported by the collected actions, reducing—though not eliminating—the world-model exploitations that arise from unconstrained action search.

When the monitor triggers, we query \pi_{b} once for its nominal action chunk a^{\pi_{b}}_{t:t+H-1} and initialize the CEM population around it using Eq. (4). Each candidate is rolled out through \phi and scored by Eq. (3); CEM then refits the proposal to the top-scoring elites and repeats for a fixed number of iterations. The planner executes the first h\leq H actions of the best sequence and returns control to the monitor, which re-invokes it if \pi_{b} has still not recovered.

V Experimental Results

V-A Experimental Setup

We evaluate our method on the ten-task LIBERO-Goal [27] suite, using \pi_{0.5} [13]’s LIBERO checkpoint as the nominal policy \pi_{b} throughout. All ten tasks share the same kitchen tabletop scene with a Franka Panda manipulator and differ only in their language-based task goals. For detailed comparisons against computationally intensive baselines, we use a four-task representative subset — Tasks t0 (open the middle drawer), t3 (open the top drawer and put the bowl inside), t4 (put the bowl on the top of the cabinet) and t9 (put the wine bottle on the rack), spanning \pi_{b}’s competence range and covering drawer opening, pick-and-place, and a multi-step task.

The \pi_{0.5}-LIBERO checkpoint reaches 98.0\% success on LIBERO-Goal under the default benchmark setup, as reported by the official openpi implementation [28]. Because this near-saturated performance provides few failures on which to evaluate intervention, we construct a perturbed evaluation setting by varying the initial manipulator configuration, which also reorients the wrist camera, and the initial x,y locations of objects in the scene. We tune the perturbation magnitudes by task difficulty: up to \pm 0.1\chi radians on joints 4 and 5, \pm 0.2\pi\chi radians on the wrist roll joint, and [x\pm\Upsilon,y\pm\Upsilon] meters for object locations. Under this protocol, \pi_{b} achieves an average success rate of 72.6\% (Table I). These perturbations define the deployment distribution, and training rollouts and test episodes use the same perturbation parameters.

For each task, we collect 200 rollouts of \pi_{b} under the same perturbation parameters used at test time, each capped at 300 steps, and labeled success or failure (timeout) by the simulator. We train the liveness value function V_{\pi_{b}} and world model \phi on these rollouts on a single NVIDIA GeForce RTX 4090 GPU. We additionally supplement the world model \phi with 53 minutes of the LIBERO-Goal teleoperated demonstrations used as play data, pooled across all ten tasks. For any given task, 90\% of these demonstrations correspond to other tasks and therefore provide primarily image–action coverage rather than demonstrations of task-correct behavior.This additional play data need not come from expert demonstrations: trained on the 200 policy rollouts alone, our method still gains 14.7pp on Task 0 (Sec. V-D).

We run 300 paired episodes per task, where all methods receive the same initial condition, and \pi_{0.5}’s sampling is fixed to the same seed across methods. Without intervention, the methods therefore reproduce the nominal policy’s trajectory exactly; trajectories diverge only when a monitor triggers, attributing differences in outcomes to the interventions alone.

V-B Deployment-time Improvement Across the Suite

Across all ten tasks and 3{,}000 paired trials, our method improves \pi_{b} from 72.6\% to 81.5\% (+8.9pp; Table I). The monitor reliably identifies executions requiring intervention: it fires on 94\% of the trials that \pi_{b} would have failed and on only 16\% of those that \pi_{b} would have succeeded. On the other hand, intervening on trials that \pi_{b} would otherwise have completed creates an opportunity to disrupt successful behavior. Across the suite, the filter rescues 369 nominal-policy failures but breaks 102 nominal-policy successes, for a net gain of 267 successful trials. This rescue–breakage tradeoff is a core tension when filtering with learned models.

The monitor fires in proportion to the competency of \pi_{b} on that task. On Task 7 (turn on the stove), where \pi_{b} already succeeds 98.0\% of the time, it fires on 5 of 300 trials and nets one additional success. On Task 9, the hardest task, it fires on 211 and nets 62 additional successes. Task 0 shows the largest gain (+27.0pp), where the task’s progression is visually legible to \overline{V}_{\pi_{b}} through the configuration of the cabinet drawers, and \phi models the relevant drawer dynamics sufficiently well for planning. Here, \pi_{b} often times out after opening the wrong drawer and getting stuck; our method instead plans toward the correct drawer. Fig. 2 illustrates one such Task 0 episode. The nominal policy begins by moving correctly toward the cabinet but then reaches for the handle of the top drawer. The monitor registers the drift as a threshold exceedance, and the planner steers the end effector away from the top drawer and toward the lower drawers. \pi_{b} then resumes control, selects the middle drawer, and opens it. The two trajectories are identical until the monitor triggers. Without intervention, the policy not only opens the wrong drawer but continues executing a different task, placing the bowl inside it. This failure involves neither an unsafe state nor a local control error, legible only as a loss of progress toward the commanded goal.

The aggregate improvement on the task suite is limited by both ceiling effects and occasional harmful interventions. Task 7 is already near ceiling (98.0\!\rightarrow\!98.3), leaving little room for improvement. Task 2 (put the wine bottle on top of the cabinet) is the only task on which performance regresses (-1.0pp). Here, \phi hallucinates grasps on the thin wine bottle, sometimes causing the robot to tip over the bottle. In contrast, Task 9 presents more recoverable failures involving the same object and improves by +20.7pp.

V-C Comparison Against Other Policy Improvement Methods

Table II compares our method against policy-improvement approaches spanning two access regimes. Our primary comparisons are methods that, like ours, can be constructed using only rollout data and black-box policy queries. TD3+BC [29] and IQL [30] are offline-RL baselines that train replacement policies from the collected rollouts. In contrast, LatentSafe [31] is a filtering baseline that adopts the same monitoring-and-intervention structure as our method. It learns an unsafe set and a safety-preserving policy via latent-space reachability analysis. We adapt it to our liveness (task-completion) formulation, using its learned policy as the fallback {\pi_{f}} in place of our planner and our same sigmoid trigger schedule. We additionally compare against Filtered-BC: it fine-tunes all 3.3B parameters of the \pi_{0.5} checkpoint on the successful rollouts our monitor and world model see. It therefore requires white-box access to the policy, its training pipeline, and H100-scale training compute. Because these requirements violate our assumed black-box access model, we treat Filtered-BC as a white-box reference rather than a directly competing baseline.

Among the black-box methods, ours achieves the highest mean success rate at 80.9\%, compared with 59.2\% for LatentSafe, 42.2\% for IQL, and 39.6\% for TD3+BC. Although TD3+BC and IQL require no access to the parameters of \pi_{b}, they discard the capabilities encoded in the original policy. Their performance is also limited by the narrow action coverage of the rollout data: \pi_{b} is nearly deterministic and rarely revisits the same state, providing little within-state action variation from which to infer better actions. Consequently, IQL’s policy-extraction step approaches behavior cloning on potentially suboptimal rollouts. TD3+BC, which we augment with transitions generated by the learned world model, is additionally exposed to model error and performs particularly poorly on Task 3 (7.3\%), where accurately modeling the drawer state is necessary for the subsequent bowl placement. Similarly, LatentSafe rarely intervenes, and its interventions do not reliably recover progress. On Task 9, for example, it breaks 80 nominal-policy successes while rescuing only 7 failures, leaving \pi_{b} worse off overall (59.2\% versus 65.2\% mean success). Its fallback is a liveness-preserving controller trained to keep the system within the set of states from which the goal remains reachable. Such an objective is well suited to maintaining invariance, but provides limited guidance once task progress has already been lost: returning to the reachable set requires recovery, rather than simply remaining within it.

Finally, our black-box method achieves comparable mean performance to the white-box Filtered-BC baseline (80.9\% versus 78.2\%), without accessing or modifying the nominal policy’s weights. Filtering and fine-tuning are also complementary rather than mutually exclusive. For example, applying our filter on top of Filtered-BC, without additional data or further access to its weights, raises its mean success rate from 78.2\% to 79.8\%. Task 0 improves from 81.0\% to 95.3\%, whereas Task 3 decreases from 75.3\% to 66.0\%. The Task 3 monitor remains similarly effective after fine-tuning, flagging 87.8\% of failures under Filtered-BC compared with 86.8\% under \pi_{b}. The degradation therefore occurs downstream of detection, consistent with this task’s world-model limitations.

V-D Robustness Analysis and Component Ablations

Perturbation-severity stress test. We conduct a limited stress test by increasing each perturbation magnitude by 50\%, one at a time, leaving V_{\pi_{b}} and \phi unchanged (n\!=\!100 per condition; Table III). This experiment characterizes sensitivity to more severe perturbations along the same axes used in our main evaluation. Increasing the configuration perturbation degrades \pi_{b} most severely, to 44.5\%, and yields our largest mean improvement (+18.0pp). Increasing the object perturbation yields an improvement of +11.5pp. Our method improves over \pi_{b} in seven of the eight conditions; the exception is Task 4 at the largest \pm 15 cm object displacement, where it underperforms \pi_{b} by 1.0pp.

Component ablations. We next ablate the principal design choices in the monitor and planner (Table IV). Monitor ablations vary one component at a time (n\!=\!100 per task), averaged across the subset. Removing the sigmoid threshold schedule and replacing it with a fixed zero threshold reduces mean success by 8.5pp; collapsing the five-member value ensemble to a single network costs 8.3pp; and removing temporal smoothing costs 4.3pp. The threshold schedule and value ensemble therefore have the largest effects among the monitor design choices, although even the weakest variant still improves over \pi_{b} by 6.0pp.

The liveness value is the component around which both monitoring and planning are organized, so we test it within the complete intervention pipeline. We replace it with the distributional Monte Carlo value function of [3], which discretizes trajectory progress into timestep classes and predicts a distribution over them, while leaving the world model, planner, trigger schedule, and threshold unchanged. This substitution reduces mean success by 13.1pp, the largest effect measured in our ablations. The two objectives carry different inductive biases: the liveness recursion propagates the non-goal value until goal attainment is observed, whereas the classifier tends toward frequently represented classes mid-trajectory. Such predictions can still detect failures well but provide a less effective planning objective. Consistent with this explanation, MC-D rescues 152 nominal-policy failures, compared with 211 for our liveness value, while breaking 120 nominal-policy successes, compared with 38 for ours.

On the planner side, replacing our learned action basis with iCEM-colored noise (\beta=2, Brownian) [26] costs 4.7 pp, demonstrating the efficacy of the proposed planning algorithm. We also vary the replanning frequency, executing only 2 of the planned actions rather than 5 leads to a modest cost of -1.3pp.

Finally, we ask how much of the result depends on privileged demonstration data. Refitting both the dynamics model and the planner’s action basis on the 200 policy rollouts alone — removing play and demonstration data from every learned component — still improves Task 0 from 66.7\% to 81.3\% (+14.7pp), retaining 54\% of the full gain and still exceeding our strongest baseline. We attribute the gap to state–action support rather than task-specific knowledge: without play data the rollout actions need 23 principal components to capture 99\% of their variance against 15 with it, so our fixed 13-component basis spans materially less.

V-E Hardware Experiment

We deploy our method on a lunchbox-opening task (Fig. ) using a Franka Panda 7-DoF manipulator and a visuomotor diffusion policy [14], with inference performed on an RTX 5090 GPU. The nominal policy \pi_{b} is trained on 105 expert demonstrations to grasp the lip of a deformable cloth lid and open the lunchbox without spilling its contents, in under 30 seconds; all collected at the same nominal initial configuration.

At test time, we perturb the initial manipulator configuration by an angle in [0,9]^{\circ} and use the same 30-second wall-clock limit. Under this test distribution, the nominal policy achieves a success rate of 40\% over 120 trials. Next, we use these 120 on-policy rollouts to train the liveness value \overline{V}_{\pi_{b}} and world model \phi. The world-model data are supplemented with the 105 expert demonstrations, 15 minutes of teleoperator practice, and 30 minutes of targeted play consisting of smooth motions around the lunchbox workspace.

Our method, run for 50 episodes under the same conditions, recovers 60.0\% (+20pp). The improvement is not confined to small perturbations (Table V). The largest observed difference occurs in the most difficult band, [6.5,9]^{\circ}, where the nominal policy succeeds only 22.7\%, compared with 70.6\% with our filter in the loop.

We further decompose the 50 filtered trials according to whether the monitor intervened. The monitor remained inactive in 16 trials, all of which succeeded. It triggered at least once in the remaining 34 trials, of which 14 succeeded. Thus, every observed failure followed at least one monitor trigger and detection is not what limits the result.

Fig. (a) illustrates one such intervention. At t_{1}\approx 1\,\text{s}, the monitor observes that \pi_{b} has erroneously closed the gripper without grasping the lid. The smoothed liveness value exceeds the intervention threshold, \overline{V}_{\pi_{b}}(s_{t_{1}})\!>\!\eta_{t_{1}}, and triggers the planner. The planning traces and decoded world-model predictions show the imagined futures of the best candidates whose actions are executed. Control alternates between \pi_{b} and {\pi_{f}} across successive interventions until t_{4}\approx 6.5\,\text{s}, when the planner predicts a value-minimizing plan for a complete grasp-and-open maneuver and the system executes it, after which \pi_{b} resumes to finish opening the lid.

V-F Deployment Cost

A full planning cycle takes 108\!\pm\!3 ms in addition to 180\!\pm\!3 ms for a single query to the nominal policy: intervening costs less than a single policy query, and far less than the n queries per step that re-ranking approaches pay. Both inference costs are amortized over the four-step action chunk they produce, so planning adds 27 ms per executed control step against the policy’s own 45 ms. This corresponds to an effective control rate of approximately 13.9 Hz during intervention, while planning incurs no overhead when the monitor remains inactive.

Evaluating many candidate futures is affordable within this budget because the world model processes them in parallel. A single 16-step imagined rollout takes 9.5 ms, while a batch of 256 such rollouts takes 44.2 ms. This sublinear scaling with our compact 29.6M-parameter world model and five-1.85M-parameter value ensemble makes sample-based planning viable within the control loop. The additional inference nevertheless reduces the number of actions executed within the fixed 30\,\text{s} episode limit. Across the hardware evaluation, the filtered system executes 20\% fewer control steps than the unfiltered, yet still achieves a higher success rate (Table V).

For context, VLA-in-the-Loop [11] and DreamAvoid [12] report approximately 20 s and 2 s per video-based intervention, respectively, while RoboMonkey [18] operates at approximately 1.5 Hz when sampling and verifying multiple policy proposals. These use different policies and hardware, but indicate the computational scale of existing test-time correction approaches. In their respective reported settings, our 108 ms planning cycle is approximately 6–185\times shorter, enabling corrective planning without pausing execution.

VI Limitations and Conclusion

We present a monitoring-and-intervention framework for improving visuomotor manipulation policies at deployment time using only black-box access. A policy-conditioned liveness value monitors progress and, when needed, guides a short-horizon planner toward states from which the nominal policy can recover; planning, evaluating, and executing within the real-time control loop. Across the LIBERO-Goal suite, our method improves the success rate of \pi_{0.5} from 72.6\% to 81.5\%, outperforming offline-RL and alternative filtering baselines. We further demonstrate the framework with a diffusion policy on a Franka Panda manipulator, where it improves success from 40\% to 60\%.

The method does not, however, guarantee that every intervention improves the execution. Table I reveals two distinct failure mechanisms. First, an inaccurate liveness value can trigger on an execution that the nominal policy would have completed, creating an opportunity for the planner to disrupt an otherwise successful trajectory. Second, even when intervention is warranted, errors in the learned world model compound over autoregressive rollouts and can be exploited by the planner: an action sequence may score well under the imagined dynamics but fail when executed. Moreover, the framework requires labeled rollouts of the nominal policy and auxiliary transition data; it is not a zero-shot improvement method. Finally, extending the framework to longer-horizon tasks, partially observable progress, and shared models that transfer across tasks remains an important direction.

References

1. [1] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246.
2. [2] G. R. Team, A. Abdolmaleki, S. Abeyruwan, J. Ainslie, J. Alayrac, M. G. Arenas, A. Balakrishna, N. Batchelor, A. Bewley, J. Bingham, et al. (2025) Gemini robotics 1.5: pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer. arXiv preprint arXiv:2510.03342. External Links: Link
3. [3] P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, et al. (2025) \pi_{0.6}^{*}: a vla that learns from experience. arXiv preprint arXiv:2511.14759.
4. [4] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892–6903.
5. [5] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. (2024) Droid: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. External Links: Link
6. [6] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada (2016) Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control 62 (8), pp. 3861–3876.
7. [7] J. Borquez, K. Chakraborty, H. Wang, and S. Bansal (2024) On safety and liveness filtering using hamilton–jacobi reachability analysis. IEEE Transactions on Robotics 40, pp. 4235–4251.
8. [8] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin (2017) Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances. arXiv. Note: arXiv:1709.07523 [eess]Comment: This paper summarizes the recent major research efforts in reachability. The paper will be presented in the proceedings of CDC 2017 External Links: Link, Document
9. [9] H. Wang, A. Dhande, and S. Bansal (2024) Cooptimizing Safety and Performance With a Control-Constrained Formulation. IEEE Control Systems Letters.
10. [10] H. Wang, J. Bowden, C. Crosby, and S. Bansal (2026) Offline policy evaluation for manipulation policies via discounted liveness formulation. arXiv preprint arXiv:2605.11479.
11. [11] S. Xu, F. Li, Z. Duan, Y. Yang, T. Xie, and Z. Yang (2026) VLA-in-the-loop: online policy correction with world models for robust robotic grasping. External Links: Link
12. [12] X. Fan, Y. Lu, S. Gao, X. Wu, R. Han, M. Li, and H. Zhao (2026) DreamAvoid: critical-phase test-time dreaming to avoid failures in vla policies. arXiv preprint arXiv:2605.11750.
13. [13] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (2025) \pi_{0.5}: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054.
14. [14] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv. Note: arXiv:2303.04137 [cs] version: 4Comment: Porject website: https://diffusion-policy.cs.columbia.edu External Links: Link, Document
15. [15] Z. Sun and S. Song (2025) Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution. arXiv. Note: arXiv:2508.05941 [cs] External Links: Link, Document
16. [16] S. Liu, I. S. Singh, Y. Xu, J. Duan, and R. Krishna (2026) VLS: steering pretrained robot policies via vision-language models. arXiv preprint arXiv:2602.03973.
17. [17] A. Wagenmaker, M. Nakamoto, Y. Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine (2025) Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.15799.
18. [18] J. Kwok, C. Agia, R. Sinha, M. Foutter, S. Li, I. Stoica, A. Mirhoseini, and M. Pavone (2025) Robomonkey: scaling test-time sampling and verification for vision-language-action models. arXiv preprint arXiv:2506.17811.
19. [19] M. Nakamoto, O. Mees, A. Kumar, and S. Levine (2024) Steering your generalists: improving robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816.
20. [20] Y. Wang, M. Verghese, and J. Schneider (2025) Latent policy steering with embodiment-agnostic pretrained world models. arXiv preprint arXiv:2507.13340.
21. [21] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2024) Mastering Diverse Domains through World Models. arXiv. Note: arXiv:2301.04104 [cs]Comment: Website: https://danijar.com/dreamerv3 External Links: Link, Document
22. [22] G. Zhou, H. Pan, Y. LeCun, and L. Pinto (2025) DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. arXiv. Note: arXiv:2411.04983 [cs] External Links: Link, Document
23. [23] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. (2025) Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. External Links: Link
24. [24] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet (2020) Learning latent plans from play. In Conference on robot learning, pp. 1113–1132.
25. [25] S. Chen, T. Zhu, Z. Wang, J. Zhang, K. Wang, S. Gao, T. Xiao, Y. W. Teh, J. He, and M. Li (2025) Internalizing world models via self-play finetuning for agentic rl. External Links: 2510.15047, Link
26. [26] C. Pinneri, S. Sawant, S. Blaes, J. Achterhold, J. Stueckler, M. Rolinek, and G. Martius (2021) Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning, pp. 1049–1065.
27. [27] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, pp. 44776–44791. External Links: Link
28. [28] Physical Intelligence (2025) openpi: open-source models and packages for robotics. Note: LIBERO benchmark results reported for the pi05_libero checkpointhttps://github.com/Physical-Intelligence/openpi/blob/main/examples/libero/README.md
29. [29] S. Fujimoto and S. S. Gu (2021) A minimalist approach to offline reinforcement learning. Advances in neural information processing systems 34, pp. 20132–20145.
30. [30] I. Kostrikov, A. Nair, and S. Levine (2021) Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.
31. [31] K. Nakamura, L. Peters, and A. Bajcsy (2025) Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis. In Proceedings of Robotics: Science and Systems, LosAngeles, CA, USA. External Links: Document