CarryOn, 고정된 pi0.5 위에 감독 계층 얹어 안전 위반 52.6%에서 6.1%로

Abstract

While generalist vision-language-action (VLA) models have enabled diverse open-world manipulation, real-world robotic assistance requires two critical capabilities they currently lack: personalization and reliable execution. First, executing an instruction like ‘make me a sandwich’ requires grounding subtask decomposition in persistent user preferences rather than relying on a single generic solution. Second, reliable deployment demands explainable safety assurance that proactively prevents hazards and gracefully recovers from occasional physical failures. To address these gaps, we present CarryOn, a dual-loop harness designed to augment a frozen VLA policy. The planning loop maintains persistent user-level and scene-level context that is continually grounded against new observations and revised through interaction, enabling personalized task selection and execution grounding across conversations and changing scenes. The execution loop actively supervises policy rollouts, shielding actions from predicted hazardous contact. When execution fails, the system invokes reusable control-based recovery skills that restore task progress and are refined through reflection on earlier failures. On a real robot and in simulation, CarryOn dispatches preference-consistent subtasks across users and scenes, reduces the violation rate of an already safety-finetuned VLA from 52.6% to 6.1%, and increases recovery success from 13% to 80% when refined skills are reused.

1 Introduction

Robot assistance ultimately aims to serve people in their everyday lives. Generalist vision-language-action (VLA) models have advanced this goal by learning to execute diverse manipulation instructions across tasks and environments [1, 2, 3]. Large-scale robot datasets further support learning across tasks and embodiments [4, 5, 6]. However, progress in instruction-conditioned manipulation alone does not bridge the gap to everyday assistance. A robot must know both what to do to meet the user’s needs in the current environment and how to carry it out reliably as physical conditions change. Bridging this gap requires a system that connects context-sensitive task decisions with reliable execution.

A harness is the system surrounding a base model that orchestrates execution and decides how the model plans, perceives and manages context, stores artifacts, and evaluates results [7]. Language-agent systems illustrate how memory and reflection can support decisions across interactions [8, 9]. In the field of robotics, hierarchical VLA systems connect high-level requests to manipulation instructions [10]. Recent concurrent works on embodied harnesses use memory to steer frozen VLAs and coordinate heterogeneous policies [11, 12], or evolve runtime critics and recovery skills from execution experience [13]. These works mainly focus on how to improve VLA models’ performance in out-of-distribution (OOD) scenarios. However, for everyday assistance, reliable execution must also be coupled with deciding what is appropriate for the user in the current situation.

We study this connection through personalized robot assistance, which exposes two challenges. First, task decisions must remain aligned with the user’s needs as interactions and scenes change, and the same request can require different actions. Learning preferences or identifying personal objects [14, 15] provides context, while in reality the context can be dynamically changed and thus requires revision in stored memory. Second, the resulting plan must be supported by reliable physical execution: impending hazardous contact calls for intervention [16] to ensure safety, while a failure that has already occurred calls for repair [17] to ensure robustness. The harness must connect these responses to the intended task and retain failure experience to improve later recovery.

Consider a user who asks a robot to “serve me food” (Fig. 1). The robot should respect an earlier vegetarian preference without requiring it to be repeated in the query. It may look beneath a remembered cover for a carrot, but finding meat instead must change its original plan. Even when the ingredient is suitable, an obstacle or a lost grasp can interrupt delivery to the bowl. The harness must preserve the user’s dietary requirement when observations change the plan, restore progress when a grasp fails, and use that failure to improve later recovery.

We present CarryOn, a dual-loop harness that connects adaptive task decisions with reliable execution for personalized robot assistance. Its central design makes task context and recovery behavior persistent, revisable parts of the runtime around a frozen VLA. The planning loop determines what to do for the user and scene, and the execution loop protects and repairs the active subtask. Visual feedback connects them, informing plan revision and determining whether execution should advance or invoke recovery while retaining the task goal.

The planning loop supports contextual adaptation through memories. User context expresses needs and preferences, while scene context helps determine where and how they can be fulfilled. The planner combines both with the current multimodal request and available observations to select an endpoint and supported subtasks. Dialogue revises user context, and observations trigger explicit scene-memory updates before subsequent plans proceed. Memory thus participates in a feedback loop: it guides action, while the resulting evidence updates what later plans can rely on.

The execution loop supports physical reliability through fast proactive shielding and slower reactive recovery. A fast proactive shield predicts gripper and carried-object motion and filters commands against geometric contact constraints. After a failure, slower, event-triggered recovery skills repair the physical state so the interrupted task can continue. Offline reflection then uses failures of the current recovery version to revise its parameters and strategies, retaining successive versions in a skill library for later attempts. Immediate protection and reactive repair operate within an episode, and accumulated experience changes recovery behavior across attempts. This design keeps the VLA fixed while allowing the recovery pathway to improve through repeated reflection.

Contributions. In this paper, we present a dual-loop harness, called CarryOn, with the features of personalization and reliable execution. The planning loop of the harness connects persistent user and scene contexts with observation-driven revision, supporting personalized task decisions across interactions, scenes, and query modalities, while the execution loop combines proactive geometric shielding with reusable and iteratively refined recovery skills. Real-world studies demonstrate the effectiveness of our harness in planning preference-oriented tasks across diverse users and scenes. Notably, our method strongly increases recovery success rates from 13% to 80% using refined skills, and reduces the constraint-violation rate from 52.6% to 6.1%.

2 Related Work

Personalization through persistent context. TidyBot learns placement preferences that generalize to later requests [14], while Bring My Cup personalizes through reference-image memory and visual prompting [15]. Embodied personalization studies memory across user interactions [18, 19], and language agents retrieve and consolidate experience for subsequent decisions [8, 9]. Unlike previous works, CarryOn connects persistent preferences with revisable scene knowledge: user memory constrains task selection, while the scene memory guides later actions.

Harnesses for robot policy execution. Code as Policies generates programs over perception and control APIs [20]. Language-model planners ground instructions through affordances, feedback, and multimodal observations [21, 22, 23, 24, 25, 26], while uncertainty-aware planners decide when to act or seek information [27]. Hierarchical VLA systems connect instructions to visuomotor execution [10]; RoboHarness uses memory to orchestrate heterogeneous policies [12]. Harness VLA combines a frozen contact-rich primitive with analytic operations and memory-guided retries [11]. In contrast, CarryOn retains the VLA to execute manipulation and couples persistent personalization with action filtering and physical repair. Both loops share the active task, preserving the user-conditioned goal through recovery and using observed outcomes to govern task progression.

Recovery and adaptation from failure. Vision-language models provide semantic failure feedback [28], while FailSafe trains failure reasoning and recovery from generated failure recovery cases [29]. RecoveryChaining learns local policies that return control to nominal controllers [17], and Zetta evolves runtime critics and recovery skills around a frozen policy [13]. Language agents accumulate experience through memory and reusable skills [30, 31, 32]. In our method, CarryOn confines refinement to control-based recovery: offline reflection revises supported parameters and strategies, retaining successive skills for later attempts. This separates immediate repair from experience accumulation while leaving the VLA and geometric shield fixed.

Geometric protection during manipulation. Latent-space safety filters use reachability analysis in learned world models [33]; AnySafe adapts such filters to parameterized constraints [34]. VLSA adds a barrier-function safety layer [16], and Code-as-Monitor generates executable constraints for failure monitoring [35]. In our design, the proactive shield predicts the spatial response to policy-predicted actions and filters actions according to minimum-distance constraints between the gripper or carried object and task-specific obstacles. This provides a check before hazardous contact, complementing recovery after failure without online learning.

3 Method

CarryOn uses a dual-loop harness to adapt task decisions to the user and environment and to support reliable physical execution. The planning loop turns a request into subtasks using persistent user and scene knowledge, then updates that knowledge through dialogue and observation. The execution loop runs a pretrained VLA, checks its actions for collision risk, and invokes control-based recovery skills when manipulation fails. Experience is retained in two forms: context for future planning and recovery experience that supports successive revisions of a versioned skill library. Fig. 2 summarizes the runtime, and the following sections describe how each form of experience is used and revised.

3.1 Overview of CarryOn

Given a request c and an optional reference image, the planner combines user memory M^{u}, scene memory M^{s}, and descriptions of the available skills to produce a plan. Each planned subtask specifies an instruction \ell_{i}, an endpoint that provides the requested skill, and optional dependencies on earlier subtasks. Endpoints include manipulation stations and information-gathering tools, so a plan can request an observation or external information before deciding what to manipulate. At a manipulation endpoint, the frozen VLA predicts action chunks U_{t}=\pi_{\theta}(o_{t},\ell_{i}) from observation o_{t} and the subtask instruction \ell_{i}. A geometric shield filters these chunks using task-dependent contact constraints before execution. The Visual Judge compares recent camera images with the scene at the start of the subtask to assess progress toward \ell_{i}. It reports running, success, or failed: success advances the plan, while failure invokes recovery on the same subtask. Observation results return to the planner for plan and memory updates.

3.2 The Planning Loop: Personalization across Interactions and Scenes

3.2.1 Planning with persistent, revisable context

A personalized plan must satisfy the user’s needs and account for the environment in which the task will be carried out. Since the current request may leave both kinds of information unstated, we maintain user and scene memories as complementary planning context. User memory supplies preferences and object references established in earlier conversations. We store preferences, habits, and reference images of the user’s objects in user-specific profiles, which enter the planner context with each request. The planner is instructed to apply the requesting user’s profile when selecting subtasks. The profile is combined with the current instruction, so planning accounts for both retained preferences and the user’s present request.

Scene memory provides planning context for deciding where and how to carry out the task in the current environment. We store object inventories and descriptions by place. The planner combines these records with the available skill descriptions to select an endpoint and decide whether an observation is needed before manipulation. If the target is hidden or its location needs checking, the planner requests an observation. The returned facts enter the planning context and are combined with the user context to revise subsequent subtasks.

3.2.2 Updating context from dialogue and observation

Personalized assistance must adapt as user needs and the surrounding environment change. This requires the robot to distinguish lasting changes in user preferences from temporary requests, and to revise its understanding of the scene as new evidence becomes available. We update user memory from dialogue and scene memory from observations to keep this planning context current. For user memory, the planner is instructed to store lasting preferences during the interaction and keep temporary instructions in the current conversation. After a plan terminates, a background summarizer reviews the dialogue using the same criterion to retain user facts not yet recorded. When the user revises or retracts a preference, the corresponding profile entry is replaced or removed, so later plans use the updated requirements.

The environment can also change independently of the user’s instructions. New observations must therefore revise the robot’s understanding of the scene, both for the current task and for later requests. We require a scene-memory update whenever an observation reveals the target or finds its remembered location empty or stale. The runtime retains the observation and records the need for an update. The planner uses this evidence to specify an update_scene operation in its next plan, revising the scene record and removing outdated information. If the planner omits this required operation, a plan validator returns the plan to the planner for revision.

3.3 The Execution Loop: Proactive Protection and Reactive Recovery

Reliable physical assistance requires both safety and robustness: the robot must intervene before hazardous contact and restore progress after disturbances interrupt a task. A policy trained to complete the task may still fall short of either requirement. Two observations motivate intervention around the VLA: The safety-finetuned \pi_{0.5} checkpoint, trained on 19,664 safe demonstrations [36], still violates constraints in 52.6% of our 900 evaluation episodes, compared with 71.1% for the original checkpoint (Section 4.2). In a separate LIBERO-Object policy-only probe, frozen StarVLA [37] completes 26/200 episodes (13.0%) after a grasped object is displaced by 12 cm (Fig. 3).

We therefore divide execution support into fast proactive shielding and slower, event-triggered reactive recovery. The shield checks actions predicted by the VLA before execution and filters them according to predicted geometric contact risk. Shielding reduces risks in the proposed motion, but surrounding environment may still introduce disturbance and change the physical state. Reactive recovery acts after the Judge detects a subtask failure, suspending the VLA and invoking a control-based skill to attempt repair before continue the same subtask. Both interventions operate within an episode, at action and subtask levels respectively. Offline reflection subsequently uses unresolved failures to refine the current recovery version for later attempts, while the VLA remains fixed.

3.3.1 Proactive shielding within an episode

A VLA policy may predict actions that cause unintended contact even when the task instruction itself is appropriate. Safe execution therefore requires checking the physical consequences of these actions before execution. We implement proactive protection with a geometric action filter, the shield, between the policy and controller. For each action chunk, it uses current RGB-D observations and the target and destination descriptions to construct obstacle sets, predict gripper and carried-object motion, and filter the actions. The output is an executable action prefix or, if the first action is blocked, a short retreat.

Grasping and placement require contact with the target and destination, so treating every scene point as an obstacle would prevent the task itself. We construct separate obstacle sets for the gripper and carried object by identifying which observed geometry belongs to these task roles. A VLM infers descriptions of the target and destination from the task instruction. The scene builder then uses text-prompted segmentation to associate these descriptions with image masks and projects the masked depth pixels into three-dimensional point sets. It removes robot self-geometry using the robot model and current state and identifies the target’s local supporting surface from geometry beneath it. Contact rules exclude permitted target, support, and destination geometry from each body’s obstacle set. Other points, including unassigned geometry, remain obstacles. Appendix C describes the grounding and contact rules used in the safety evaluation.

Motion prediction then estimates the end-effector trajectory resulting from the VLA’s predicted actions. Proposed action increments alone do not describe the realized robot displacement: the controller response and ongoing motion also affect where the gripper moves. For the next H actions U=(u_{0},\ldots,u_{H-1}), the shield predicts end-effector poses using executed-motion history \mathcal{H}_{t} and a short-horizon response model. Translation is approximated by critically damped dynamics,

\ddot{p}=\omega_{n}^{2}(p^{\mathrm{goal}}-p)-2\omega_{n}\dot{p},

where p denotes end-effector position, with velocity \dot{p} and acceleration \ddot{p}. The controller target p^{\mathrm{goal}} is obtained by adding the scaled translational component of the VLA-predicted action to the end-effector position at the start of the control step. The natural frequency \omega_{n} sets the response rate. Starting from the observed motion state, we predict position and, through an analogous rotation model, orientation over the action chunk. The shield then propagates the gripper and any inferred carried-object geometry with these poses, so the next stage checks distances from the moving bodies to obstacles.

Action filtering uses these predicted body positions to decide how much of the action chunk can execute. At step k, the shield computes the distance d_{a,k} between each body a (the gripper or carried object) and its obstacle set. It releases only the actions before the first predicted step at which any distance reaches or falls below its threshold \delta_{a}. The resulting prefix length is

g=\min\Bigl(\{k\in\{0,\ldots,H-1\}:\min_{a}(d_{a,k}-\delta_{a})\leq 0\}\cup\{H\}\Bigr).

When g>0, it releases (u_{0},\ldots,u_{g-1}), and g=H permits the entire checked chunk. When g=0, it withholds the candidate actions and commands a short retreat upward and away from the forbidden point nearest the end effector. It then reassesses using fresh observations. History is updated only from actions actually executed and their observed motion, so withheld actions are not treated as changes to the physical state.

3.3.2 Reactive recovery with reusable skills

Robust task completion requires recovering from disturbances that have already interrupted manipulation. At this point, filtering the next action may reduce further contact risk but cannot by itself restore task progress. We therefore implement reactive recovery with control-based skills that attempt to repair the physical state before the VLA retries the interrupted subtask. The recovery skill library initially contains one shared base skill for all targets. When the Judge detects a subtask failure, the runtime suspends the VLA and invokes the applicable recovery behavior from the current library version, using retained revisions where available or base version otherwise.

The base skill proceeds through observation, retrieval, verification, and resumption. For a dropped object, it first moves the arm to an observation configuration and re-localizes the target. It generates grasp candidates, executes a grasp, then lifts and checks holding from the measured gripper width. A passing check allows the arm to carry the object to a stand-off pose near the destination, where a second check assesses whether the grasp was retained during transport. If this check passes, the VLA resumes the original instruction from that pose. If retrieval fails or the second holding check does not pass, the recovery attempt is marked as failed. The runtime then attempts to return to the available grasp candidate’s pre-grasp pose, or to the recorded pre-failure pose if no candidate is available, and opens the gripper on arrival. Both paths continue in the current scene within the remaining episode budget, and the Judge assesses task completion separately from recovery and resumption.

3.3.3 Refining recovery skills across attempts

Failures can recur when recovery repeats an unsuitable grasp configuration or an ineffective sequence of operations. Improving recovery can therefore require changing an operation’s numerical settings or the procedure in which it is used. We use offline reflection to refine the current skill version from unresolved failures. Between evaluation rounds, reflectors inspect selected failed attempts using scene and target images, grasp-stage measurements, and the version used. Parameter memory (P) records changes to numerical settings, for example, a revised contact tool-center-point height is applied when constructing a grasp command. Strategy memory (S) records changes to the recovery control flow, such as adding retries or VLM-assisted re-grounding. With re-grounding enabled, the runtime segments current object candidates and asks a VLM to compare them with the target reference image. The selected mask is then passed to the grasp-generation routine.

After reflection produces proposed revisions, an admission check rejects unsupported formats and out-of-range numerical values. Accepted P/S updates are incorporated into the current recovery procedure and saved as the next library version. The following round evaluates that version, and its unresolved failures guide further reflection and refinement. Successive rounds build on the preceding version while the VLA weights remain fixed. Appendix D.2 details how the records are used, and Section 4.3 specifies failure selection and the two-round protocol.

4 Experiments

We evaluate the planning, shielding, and recovery mechanisms. Real-robot studies test their use in physical assistance (Fig. 5); controlled simulation studies extend the evaluation across scenes, safety constraints, and disturbances (Fig. 4).

4.1 Personalization across Interactions and Scenes

We test whether persistent context supports appropriate task decisions, remains useful after updates, and reduces scene search. Real-robot studies report subtask dispatch accuracy: the fraction of dispatched subtasks judged correct under task requirements and applicable user preferences (Table 1). RoboLab complements these studies with controlled endpoint selection and memory revision.

Sandwich preparation: retaining and revising preferences. This experiment tests whether user memory preserves user preferences across conversations and supports plan changes when those preferences are revised. The robot prepares sandwiches from toast, lettuce, and ham across four ingredient layouts. We compare three conditions: Textual specifies how to make the sandwich directly in the query; Memory introduces the user’s preference in an earlier conversation, restarts the program, and then asks only “Make me a sandwich”; Preference update uses another conversation to revise the user’s dietary preference before the same request. Conversations also contain unrelated remarks, requiring only relevant preferences to be retained. Dispatch accuracy is 100% (24/24) for both Textual and Memory, and 83.3% (20/24) after updates. These results show that the system retains user preferences across conversations and follows revised preferences in 20 of 24 dispatched subtasks.

Activity-dependent packing: combining profiles, intent, and online information. This experiment tests whether the planner can model the user’s profile with more complex information to make personalized decisions in a multi-user setting. The robot selects from a cap, cup, umbrella, charger, and towel for different activities such as running, visiting the gym, going to the park, or commuting to work. The planner queries the current weather at the user’s residence through online search and combines it with the requested activity and personal habits. Two users have distinct residences and habits, including preferences that differ from generic packing choices. Memory supplies the user’s profile through conversation before restarting the program; No-habit supplies the same information without personal habits and relies on commonsense reasoning for planning; Isolation registers both profiles before one user’s request to test profile isolation. Required and forbidden items are judged against recorded weather and preferences. Dispatch accuracies are 100% (16/16), 80.0% (16/20), and 88.9% (16/18), respectively. These results support combining persistent user profiles, current intent, and online information for personalized decisions in a multi-user setting.

Covered ingredients: retrieving and correcting scene memory. This experiment tests whether scene memory guides the search for hidden targets and is revised when observations contradict remembered locations. Lettuce, toast, and ham are hidden beneath colored lids, and the robot is asked to transfer a named ingredient to a plate. Write and retrieve provides its location before a restart, followed by a request without a location hint. Search without memory requires successive lid opening and observation before finding the target. Revise stale memory swaps ingredients after memory formation to test if memory can be revised. Dispatch accuracies are 100% (12/12), 80.0% (8/10), and 75.0% (9/12), respectively. Remembered locations support retrieval, while fresh observations can correct the stale context used for planning. Recorded failure cases include misidentification and omitted subtasks.

Multimodal requests and memory revision across scenes. We test whether memory enables efficient scene selection for named and visually specified personal objects across four endpoints in RoboLab [38]: garage, dining table, kitchen, and study desk. The planner must select where to act for a target specified by name or photograph, allowing users to refer to personal objects without a precise verbal description. With no initial scene view, the planner requests endpoint images or consults stored scene memory. Each text/image condition contains 88 requests. Image queries never name the object: without memory, the photograph accompanies the instruction go pick up this; with memory, go pick up mine refers to a photograph supplied in an earlier conversation as this is mine. Memory raises correct selections in both query modes and reduces latency (Table 2a). These configurations support both named and remembered visual references while reducing the search required to select a scene.

We additionally conduct a revision experiment to test whether new observations correct persistent scene records and whether later requests can use the revised memory for endpoint selection. We relocate targets while keeping their old locations in memory. A valid memory correction must write the new location and remove the stale association. Corrections succeed in 76/88 text and 77/88 image trials; follow-up dispatch is correct in 83/88 and 85/88, respectively (Table 2b). The two measures support observation-driven revision and later reuse of scene knowledge.

Analysis. The above studies evaluate the lifecycle of personalized context: acquiring requirements, retaining them across interactions, and revising them after change. The results support connecting memory to planning and observation, rather than treating stored profiles and locations as permanently valid instructions.

4.2 Safety during Execution

Geometric checks beyond safety fine-tuning. We test whether runtime shielding reduces unsafe contact beyond improvements from training with safe demonstrations. LIBERO-Safety [36] provides tabletop spatial avoidance (TSA) and free-space hand-object avoidance (FSHOA), with 450 episodes per suite. We evaluate the original and safety-finetuned (SFT) \pi_{0.5}, with and without our shield, alongside adapted VLSA [16], and extend the comparison to OpenVLA-OFT [39] and VLAct [40] to test transfer across policy backbones (Table 3). The official SFT \pi_{0.5} checkpoint was released by [36] and was trained on 19,664 safe demonstrations for collision-free execution. We report violation-free task success rate (VF) and violation rate (VIOL.).

Despite safety fine-tuning, 52.6% of episodes incur violations. Shielding reduces this to 6.1%, compared with 37.7% for adapted VLSA, while VF remains similar at 25.3% without shielding and 25.6% with it. For the original checkpoint, violations decrease from 71.1% to 22.7%, compared with 49.8% for adapted VLSA, while VF increases from 6.7% to 14.6%. The same shield also reduces violations with OpenVLA-OFT, from 42.0% under VLSA to 17.2%, while increasing VF from 12.3% to 14.8%. On VLAct, our shield reduces violations from 44.8% under VLSA to 26.1%, with a smaller VF increase from 13.0% to 13.9%, supporting cross-backbone violation reduction.

For deployment cost, measured mean per-invocation times are 21.06 ms for geometric filtering, 47.82 ms for VLA inference, and 170.82 ms for perception. Perception dominates these measurements, while filtering contributes a smaller computational component.

Real-world safe execution. The physical study tests whether the proactive shield reduces contact during real-robot execution when an obstacle is introduced. The real-robot policy is trained on carrot relocation without obstacles. With an inserted obstacle, contact-free completion increases from 0/20 for the raw policy to 20/20 with shielding (Table 1), showing repeatable intervention against collision hazards.

4.3 Recovery and Improvement from Failure Experience

Recovery from perturbation This experiment tests whether the recovery skill restores task progress after disturbance and whether using successively refined skills in reflection-and-retry rounds increases cumulative completion. The disturbance-response results (Fig. 3) motivate control-based recovery when perturbations undermine VLA execution.

The experiments are conducted on the LIBERO-Object task suite with a 12 cm positive-x hold-loss perturbation. R0 uses the shared base recovery skill with empty strategy (S) and parameter (P) memories. After each round, reflection uses up to three failed episodes with recovery evidence per object. Baselines and our reflection process use Qwen3-VL-235B-A22B-Instruct. Harness VLA uses the same Qwen3-VL-4B-OFT VLA checkpoints as our system, while CaP-X does not require a VLA.

With base recovery, R0 completes 58/200 episodes (29.0%). R1 resolves 83/142 previous failures, raising cumulative completion to 70.5%; R2 resolves another 19/59, reaching 80.0% (Table 5). The second reflection-and-retry round extends recovery beyond the first revision, and the full iterative system achieves the highest completion rate.

To assess the VLM-query cost of recovery-focused refinement, we measure billed API tokens including both reflection rounds and all retries, amortized over the original episodes. Our recorded usage is 41,109 tokens per episode, including 2,609 for reflection, compared with the recorded baseline totals in Table 5. Higher cumulative completion together with lower recorded VLM-token usage supports recovery-focused refinement as a more efficient method.

Recovery from recorded failure states. We further evaluate recovery from natural execution-induced failure states using LIBERO-Recovery [43]. The evaluation covers 77 Object scenes with five initially closed-gripper and five initially open-gripper trials each, yielding 770 cases without additional runtime disturbances. Since failure states are supplied, we disable the Judge and invoke recovery after scene initialization. Pure VLA and R0 share the same Qwen3-VL-4B-OFT VLA checkpoint.

Base recovery raises completion from 32.08% to 50.00%, a gain of 17.92 percentage points under the same rollout budget. R1 resolves 184/385 previous failures, increasing cumulative completion to 73.90%; R2 resolves another 17/201, reaching 76.10% (Table 5). Final cumulative API usage is 7,158.77 tokens per original case, including reflection and retries. The above results support iterative recovery from recorded natural failure states.

Real-world recovery. This experiment tests whether base recovery and retained skill revisions improve real-robot task completion after object removal. An operator takes the lifted object and places it at one of five locations, with two orientations and two repetitions (20 episodes per condition). We compare the raw policy, base recovery, and a revised skill combining a -0.02 m grasp-height offset with up to three attempts and renewed perception. Task completion rises from 3/20 to 9/20 and 18/20, respectively (Table 1). External repair improves completion over the raw policy, and retained revisions improve it further across the tested placements.

Analysis. The execution studies support complementary interventions around a frozen VLA: shielding reduces contact violations, while recovery restores progress after disruption. Reflection-and-retry rounds extend cumulative completion, and revisions improve real-robot repair. Together, the two paths combine immediate protection with reusable failure recovery skills.

4.4 Integrated Assistance on the Real Robot

Setup. This study examines whether personalized planning, proactive shielding, and reactive recovery operate as one system in a long-horizon assistance task, where each component acts on what the others produce. In the food-serving scenario of Fig. 1, a user with a stored vegetarian preference asks the robot to serve lunch from a covered pan. The robot must look under the lid, decide whether the food may be served, and carry it past an obstacle while an operator disturbs the transport. A second trial places meat under the lid while scene memory still records a carrot. All real-robot experiments use a Qwen3-VL-4B-OFT policy [37]; the policy for this task was trained on lid opening and food pickup without obstacles.

Results. In the carrot run (Fig. 6a–h), the robot uncovers the pan guided by scene memory and starts the transfer. The shield avoids potential collisions with the rice cooker obstacle during transport. After the carrot is dropped into the pan as a perturbation, the VLA is not able to pick it up again. The Judge triggers the recovery procedure, which re-grasps the carrot and hands control back to the VLA to place it in the bowl. In the meat run (Fig. 6i–l), the robot uncovers the pan and finds a contradiction with scene memory, then replaces the stale carrot entry with the observed meat entry. The system then declines to serve anything as the food currently on the table does not meet the user’s dietary requirements. The accompanying video shows both runs.

Analysis. These runs illustrate how CarryOn preserves the user’s intent while adapting both the plan and its execution. Scene memory guides where to look, while fresh visual evidence and the stored dietary preference determine whether serving should proceed. In the carrot run, the obstacle and lost grasp disrupt execution without invalidating the goal: shielding adjusts the transfer motion, and Judge-triggered recovery restores the grasp before returning control to the VLA to finish serving. In the meat run, the newly revealed food instead invalidates the planned transfer under the same dietary constraint. Visual feedback therefore changes the planning decision and corrects the persistent scene record, rather than merely checking whether a manipulation succeeded. The benefit of the dual-loop design is this distinction between repairing an interrupted, appropriate task and stopping a task that new evidence makes inappropriate. Together, the runs provide evidence that contextual decisions, action-level protection, and physical repair can support a common assistance goal around a frozen VLA.

5 Conclusion

We presented CarryOn, a dual-loop harness for personalized and reliable robot assistance. The planning loop maintains revisable user and scene memories to guide task decisions across interactions, query modalities, and execution endpoints. The execution loop combines proactive geometric shielding with reactive recovery, using offline reflection to refine recovery skills while keeping the underlying VLA policy frozen. Real-robot and simulation studies evaluate these complementary capabilities, from preference-conditioned assistance and scene-memory revision to runtime protection and recovery after disturbance. Together, they show that persistent context and physical feedback extend robot assistance beyond isolated task execution. Future work will evaluate longer-term adaptation and the transfer of recovery experience across tasks, while improving the balance between protective intervention and task completion.

References

1. [1] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) \pi_{0}: a vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164.
2. [2] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246.
3. [3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. (2023) Rt-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818.
4. [4] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892–6903.
5. [5] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. (2024) Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213.
6. [6] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davchev, Y. Zhou, A. Gupta, A. Raju, et al. (2023) Robocat: a self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706 1 (8).
7. [7] L. Weng (2026) Harness engineering for self-improvement. lilianweng.github.io. External Links: Link
8. [8] C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez (2023) Memgpt: towards llms as operating systems. arXiv preprint arXiv:2310.08560.
9. [9] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein (2023) Generative agents: interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1–22.
10. [10] L. X. Shi, B. Ichter, M. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, et al. (2025) Hi robot: open-ended instruction following with hierarchical vision-language-action models. arXiv preprint arXiv:2502.19417.
11. [11] Y. Zhang, H. Zhang, F. Gao, X. Li, Z. Liu, C. Zhu, J. Qiu, Y. Yan, J. Liu, W. Tang, et al. (2026) Harness vla: steering frozen vlas into reliable manipulation primitives via memory-guided agents. arXiv preprint arXiv:2607.08448.
12. [12] J. Huang, Y. Hu, Z. Li, R. Qi, Y. Xiao, Z. Zhang, M. Coates, T. Cao, and Y. Zhang (2026) RoboHarness: memory-driven orchestration of heterogeneous robot policies for long-horizon planning. arXiv preprint arXiv:2607.18060.
13. [13] X. Ding, L. Mi, M. Huang, Z. Wang, C. Zhang, Z. Hao, F. Chen, X. Li, Y. Zheng, Y. Guo, et al. (2026) Zetta \zeta: an efficient closed-loop embodied harness for self-evolving physical intelligence. arXiv preprint arXiv:2608.16590.
14. [14] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser (2023) Tidybot: personalized robot assistance with large language models. Autonomous Robots 47 (8), pp. 1087–1102.
15. [15] S. Lee, S. Mo, and W. Han (2025) Bring my cup! personalizing vision-language-action models with visual attentive prompting. arXiv preprint arXiv:2512.20014.
16. [16] S. Hu, Z. Liu, S. Liu, J. Cen, Z. Meng, S. Wang, X. Li, and X. He (2025) Vlsa: vision-language-action models with plug-and-play safety constraint layer. arXiv preprint arXiv:2512.11891.
17. [17] S. Vats, D. K. Jha, M. Likhachev, O. Kroemer, and D. Romeres (2025) Recoverychaining: learning local recovery policies for robust manipulation. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9776–9783.
18. [18] T. Kwon, D. Choi, H. Kim, S. Kim, S. Moon, B. Kwak, K. Huang, and J. Yeo (2026) Embodied agents meet personalization: investigating challenges and solutions through the lens of memory utilization. In International Conference on Learning Representations, Vol. 2026, pp. 4790–4835.
19. [19] J. Lee, C. Park, and D. Lee (2026) Personalizing embodied multimodal large language model agents over long-term user interactions. arXiv preprint arXiv:2605.26256.
20. [20] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng (2023) Code as policies: language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA), pp. 9493–9500.
21. [21] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. (2022) Do as i can, not as i say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
22. [22] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg (2022) Progprompt: generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302.
23. [23] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. (2022) Inner monologue: embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608.
24. [24] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei (2023) Voxposer: composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973.
25. [25] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. (2023) Palm-e: an embodied multimodal language model. arXiv preprint arXiv:2303.03378.
26. [26] Y. Zhu et al. (2023) Vima: general robot manipulation with multimodal prompts. In International Conference on Learning Representations (ICLR),
27. [27] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al. (2023) Robots that ask for help: uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928.
28. [28] B. Zhang, Y. Zhang, J. Ji, Y. Lei, J. Dai, Y. Chen, and Y. Yang (2026) Safevla: towards safety alignment of vision-language-action model via constrained learning. Advances in Neural Information Processing Systems 38, pp. 153335–153373.
29. [29] Z. Lin, J. Duan, H. Fang, D. Fox, R. Krishna, C. Tan, and B. Wen (2025) Failsafe: reasoning and recovery from failures in vision-language-action models. arXiv preprint arXiv:2510.01642.
30. [30] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, pp. 8634–8652.
31. [31] A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang (2024) Expel: llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, pp. 19632–19642.
32. [32] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023) Voyager: an open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291.
33. [33] K. Nakamura, L. Peters, and A. Bajcsy (2025) Generalizing safety beyond collision-avoidance via latent-space reachability analysis. arXiv preprint arXiv:2502.00935.
34. [34] S. Agrawal, J. Seo, K. Nakamura, R. Tian, and A. Bajcsy (2025) Anysafe: adapting latent safety filters at runtime via safety constraint parameterization in the latent space. arXiv preprint arXiv:2509.19555.
35. [35] E. Zhou, Q. Su, C. Chi, Z. Zhang, Z. Wang, T. Huang, L. Sheng, and H. Wang (2025) Code-as-monitor: constraint-aware visual programming for reactive and proactive robotic failure detection. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6919–6929.
36. [36] R. Cui, Z. Zhang, J. Pang, H. Chi, J. Guo, S. Zhang, S. Xie, X. Jin, Y. Mu, J. Yang, et al. (2026) LIBERO-safety: a comprehensive benchmark for physical and semantic safety in vision-language-action models. arXiv preprint arXiv:2606.23686.
37. [37] S. Community (2026) StarVLA: a lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014.
38. [38] J. X. Yang, R. Dagli, A. Zook, H. Hadfield, A. Goyal, S. Birchfield, F. Ramos, and J. Tremblay (2026) Robolab: a high-fidelity simulation benchmark for analysis of task generalist policies. arXiv preprint arXiv:2604.09860.
39. [39] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. arXiv preprint arXiv:2502.19645.
40. [40] S. Yang, C. Wang, Y. Chen, Z. Wang, L. Tang, H. Gui, J. Ye, C. Lu, X. Wu, M. Zhu, et al. (2026) Beyond data scaling: representation-centric continued pre-training for vision-language-action models. arXiv preprint arXiv:2608.27550.
41. [41] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (2025) \pi_{0.5}: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054.
42. [42] L. Fu, J. Yu, K. El-Refai, E. Kou, H. Xue, H. Huang, W. Xiao, G. Wang, D. Niu, F. Li, et al. (2026) CaP-x: a framework for benchmarking and improving coding agents for robot manipulation. arXiv preprint arXiv:2603.22435.
43. [43] L. Liu, Z. Bao, L. Zhang, Z. Song, W. Yang, S. Tao, W. Liu, and H. Lu (2026) LIBERO-recover: beyond task success towards failure recovery in robotic manipulation models. arXiv preprint arXiv:2609.05178.

Appendix

Appendix A System Implementation and Runtime Coordination

This appendix specifies how control passes between the planner, the VLA, the Judge, the shield, and recovery while a request is handled (Fig. 2). Each planner output is a plan: a goal and a set of subtasks, each of which sends an instruction to an endpoint. Invalid plans, including those that omit a required update_scene (Appendix B.3), are returned to the planner for repair. Algorithm 1 summarizes one request. A subtask without dependencies starts immediately, and a subtask that depends on earlier subtasks starts in sequence.

Appendix B Persistent User and Scene Memory

This appendix gives the storage format of both memories, their update paths, and logged examples showing how a revision changes the next plan.

B.1 Storage and Planner Injection

Both memories are Markdown files of ## <section> blocks with - key: value lines (Table 6). Both are rendered into the planner context on every request. All user files are rendered under per-user headings, and the planner prompt instructs it to apply only the requesting user’s facts.

B.2 User Memory from Dialogue

User memory has two write paths. During a turn, the planner can record a preference the user states explicitly. In addition, after each plan finishes, a background summarizer reviews the recent conversation and writes durable facts that are not yet stored. Both follow the same rule for separating durable facts from temporary statements. A new value for a key replaces the old one, and a superseded preference is retracted.

For example, when Bob says “I take my charger with me everywhere, even to the gym. Also, I need to leave by five today.”, the profile stores profile:habit/charger and nothing about leaving at five.

We show an example below to demonstrate how preference change is reflected in the user profile. In the sandwich experiment, the same request yields different plans before and after the stored diet changes.

B.3 Scene Memory from Observation

An observation that reveals new contents creates an outstanding record, and the validator rejects the next plan unless it contains an update_scene subtask. The RoboLab example below shows a stale location being revised and reused. Memory lists a Rubik’s cube at the garage bench, but the cube is on the dining table. The first request checks the remembered station, rewrites it, finds the cube elsewhere, and records the new location; the follow-up dispatches directly.

B.4 Remembered Reference Images

A reference image is stored as an entity attribute: the file is copied into the memory store, and a reference_image fact pointing to it is written under entity:<name>, either by the summarizer when the model tags an entity fact with an attached image or by an explicit /remember_image <path> <entity> command. A later request can then refer to the entity (“go pick up mine”), and the stored image is available to the planner through the user-memory block.

Appendix C Proactive Shielding

The LIBERO-Safety scene builder assigns task roles to observed geometry before applying contact rules. It receives target and destination names from the instruction. Text-prompted SAM3 segmentation associates these names with image masks; the corresponding depth pixels are back-projected into the world frame. When text grounding returns no object mask, the scene builder can use a VLM point prompt followed by point-prompted segmentation. The self-mask removes robot and mount pixels. The target’s source support is obtained as a local geometric patch beneath the grounded target.

The contact rule exempts intended target, source-support, and destination contact for the bodies to which those permissions apply. Remaining scene obstacles are checked geometrically. Gripper geometry is propagated from each predicted end-effector pose. A holding-state latch uses measured gripper width and the executed gripper command to determine when carried-object geometry should be included. The inferred held-object point set is transformed using its attachment pose, so carried-object contact is checked in addition to gripper contact. If the first candidate action is blocked, the controller executes a three-control-step retreat with an away-from-obstacle component and an upward component. The retreat direction uses the obstacle point nearest the current end-effector reference point, and the filtering distance uses the surfaces of the predicted bodies, as distinguished in Fig. 7.

Table 7 gives the parameters of the response model and decision rule on LIBERO-Safety. Figure 7 illustrates action filtering and retreat.

Appendix D Recovery Skills and Reflection

This appendix describes additional details of the recovery and reflection mechanism.

D.1 Base Recovery Skill

The base skill attempts to restore the physical state needed to continue the interrupted subtask (Algorithm 2). After suspending the VLA, it moves the arm to an observation configuration, locates the target, and generates grasp candidates for retrieval. A check of measured gripper width after lifting determines whether to proceed with transport. If this check passes, the skill locates the destination, moves to a stand-off pose near it, and checks holding again. Passing the second check allows the VLA to resume the original instruction from that pose.

Recovery fails if the skill cannot locate the target or destination, its grasp attempts do not establish a hold, a required motion cannot be completed, or the second holding check does not pass. In these cases, the skill stops and the runtime attempts to return the arm to the available grasp candidate’s pre-grasp pose, or to the recorded pre-failure pose if no candidate exists. It opens the gripper after arrival, then requests VLA resumption with the original instruction and remaining episode budget. Resumption is still requested if the fallback motion fails. The Judge continues to assess task completion; a successful recovery only enables the task to continue.

D.2 Applying a Stored Recovery Version

Each recovery version preserves the recovery procedure and the strategy (S) and parameter (P) revisions from offline reflection for use in later attempts. S changes the sequence or conditions of recovery operations, while P adjusts their numerical settings, such as grasp height. Recovery uses the revisions stored in the current library version, with the base skill supplying operations or settings that have not yet been revised. Between rounds, reflection builds on this version using the remaining failures and saves further revisions for subsequent attempts.

The strategy revisions address different failure sources in simulation and on the real robot. In simulation, we found target grounding to be a recovery bottleneck in scenes with visually unrealistic object appearances. The evaluated S revision therefore enables VLM-assisted re-grounding only for objects that are hard to ground and lead to recovery failure. During recovery, a VLM receives the target name and reference image together with a fresh scene image showing numbered candidate masks, and selects the mask used for grasp generation. Images from earlier failed attempts inform the offline reflector’s decision to enable this step. On the real robot, S also adds retries with renewed perception, allowing another grasp attempt to use an updated target location.

D.3 A Complete Revision Chain

We provide a BBQ sauce example across reflection rounds in the LIBERO perturbation study. Both memories start empty. After each round, the reflector reads up to three failed recoveries of the object and decides whether to write an S entry and whether to update the P entry. The next round reruns the failed episodes with the new memory version.

• R0 (empty memory): 3/20. In the three failures selected for M1, all six grasp attempts were planned at a contact height of 66.0 mm and all descents jammed at 109.0–113.2 mm.
• M1. S write: “Algorithm-selected crop shows tomato ketchup bottle, not BBQ sauce reference”, obtained by the reflector’s visual judgment. P update to 110.0 mm: “raising z to 110.0 mm tests a shallower approach.”
• R1: 3/17. Descents now reached the planned height instead of jamming, but the object often slipped during the lift.
• M2. S skip (“crops match reference BBQ sauce bottle”); P update to 105.0 mm (“a testable hypothesis to improve grasp stability”).
• R2: 11/14. Cumulative completion for the task rises from 3 to 17 of 20.

Appendix E Experimental Configuration

Table 8 lists the models used for each role.

Appendix F Agent Prompts

F.1 Agent Inputs and Outputs

Table 9 summarizes what each language-model module receives and returns.

F.2 Agent Prompts

This section lists the prompts of the core modules. The planner and Judge prompts are instantiated from the real-robot studies, and the reflector prompts from the LIBERO perturbation study; the summarizer’s durability rule is excerpted in Appendix B.2. Other studies use the same prompts except for the available endpoints and benchmark-specific inputs. Runtime content, such as memories, endpoint descriptions, and images, is replaced by <<SLOT: ...>> placeholders. [SYSTEM MESSAGE] and [USER MESSAGE] separate the messages of a request.