RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation

Abstract

Pre-trained robot policies, such as Vision-Language-Action (VLA) models, provide a strong foundation for generalist robotic manipulation, but adapting them to new tasks with only a few demonstrations remains challenging. Directly fine-tuning a pre-trained policy often achieves strong in-distribution performance, yet it can distort the representations learned during pre-training and weaken the model’s generalization ability under out-of-distribution shifts. We propose Retrieval-Anchored Representation Alignment (RARA), a two-stage adaptation method, to address this issue. First, RARA retrieves pre-training demonstrations with similar task intent and motion, and aligns the policy’s representations of the target demonstrations with the pre-trained representations of the retrieved demonstrations. Second, RARA fine-tunes the policy on the target demonstrations while limiting how much its representations of the retrieved demonstrations change during fine-tuning. By reducing the representation mismatch before action learning and preserving pre-trained knowledge during adaptation, RARA limits unnecessary representation drift during fine-tuning. Experiments with Diffusion Policy and GR00T on 13 tasks in RoboTwin, RoboCasa, and on a real robot show that RARA improves the average success of fine-tuning by 9.5 points in-distribution and 13.6 points out-of-distribution, and outperforms all compared baselines. Project page: https://shua-kang.github.io/RARA_robot/

1 Introduction

Pre-training on large-scale robot demonstrations followed by fine-tuning on a small target dataset has become a standard practice in learning robot manipulation policies (O’Neill et al., 2024; Khazatsky et al., 2024; Black et al., 2025; Bjorck et al., 2025). By leveraging supervision from large and diverse robot demonstration datasets, pre-training provides useful initial representations that help adaptation to a new task. However, fine-tuning on limited target data can still overfit the policy to the narrow distribution seen during adaptation: A policy may succeed on the target task under the in-distribution conditions, but fail when the camera viewpoint, background, or object properties change (Fei et al., 2026). This vulnerability arises because standard fine-tuning on limited data can severely distort the representations learned during pre-training (Kumar et al., 2022; Kachaev et al., 2026; Yadav et al., 2026). Therefore, robust policy fine-tuning, which requires learning a target task while maintaining generalization to out-of-distribution shifts, remains a critical challenge when adapting pre-trained robotic policies to new tasks.

Existing methods have explored several ways to preserve generalizable representations when adapting policies to new tasks. One line of work constrains the adapted policy toward its pre-trained initialization, for example by regularizing fine-tuned weights toward the pre-trained weights (Huang et al., 2026), merging pre-trained and fine-tuned weights (Yadav et al., 2026), or aligning policy representations with those of a frozen pre-trained visual encoder (Kachaev et al., 2026). Another line of work distills knowledge from multiple pre-trained models into a shared representation for downstream robot learning (Shang et al., 2024). While these approaches preserve broad pre-trained knowledge, they do not provide a principled way to focus alignment to pre-training demonstrations that are most relevant to the target task.

We propose Retrieval-Anchored Representation Alignment (RARA), which uses task-relevant pre-training demonstrations as anchors for fine-tuning (Fig. 1). RARA retrieves relevant demonstrations from the pre-training data by matching both task instructions and robot motions to the target demonstrations. For example, demonstrations of removing a pot lid may be retrieved for target tasks like opening a blender and remove a bottle cap. The retrieved demonstrations support adaptation in two complementary ways. Before fine-tuning, RARA aligns every target frame with the retrieved frames whose upcoming actions are most similar, so each frame is matched to the same stage of the motion in the retrieved demonstrations (e.g., a frame in which the gripper closes around the blender lid is matched to frames in which it closes around a pot lid). The pre-trained policy has seen these stages under many visual conditions, so alignment lets the target frames reuse this robust representation instead of fitting the target demonstrations from scratch. During fine-tuning, an anchor loss keeps the representations of the retrieved demonstrations close to their pre-trained values, so the small target dataset does not overwrite this robust structure. Unlike a standard drift regularizer, the anchor acts only on the retrieved demonstrations. It therefore does not pull the target representation back, and it preserves only the pre-trained structure that is related to the target task.

We fine-tune a pre-trained Diffusion Policy on four RoboTwin tasks, and a DROID-pre-trained GR00T policy on five RoboCasa tasks and four real-robot tasks. We evaluate each policy in the conditions covered by its demonstrations (in-distribution, ID) and under new camera poses, backgrounds, distractors, objects or object positions (out-of-distribution, OOD). Across these 13 tasks, RARA improves the average success rates of fine-tuned policies by 9.5 % points in ID and 13.6 % points in OOD settings, consistently outperforming the baseline methods.

Our contributions are threefold: (1) we introduce RARA, a framework for robust policy fine-tuning that uses task-relevant pre-training experience to guide representation learning; (2) we evaluate RARA across different policies, simulated environments, and real-world manipulation tasks, measuring performance of target tasks and robustness to unseen conditions; and (3) we conduct extensive studies that separate initial alignment from representation constraints during fine-tuning, and examine how the choice of source data and network layer affects adaptation.

2 Related Work

Robust fine-tuning. The goal of robust policy fine-tuning is to learn a downstream task without erasing the generalization acquired during pre-training (Yadav et al., 2026; Grover et al., 2025; Hancock et al., 2026; Yang et al., 2026; Driess et al., 2025). In transfer learning, Kumar et al. (2022) show that full fine-tuning can distort pre-trained features and hurt OOD robustness, motivating staged or constrained updates. Other methods keep the fine-tuned features close to those of the pre-trained model (Mukhoti et al., 2024), or use auxiliary data with rich semantics as anchors that preserve OOD generalization (Han et al., 2024). In robot learning, one line of work freezes parts of the model and adds lightweight trainable modules, such as adapters or task-specific heads, to preserve existing capabilities while specializing to new tasks (Liang et al., 2022; Sharma et al., 2023; Liu et al., 2024; Xu et al., 2023). Another line preserves the capabilities of the pre-trained policy more directly. Weight-space methods merge the pre-trained and fine-tuned weights (Yadav et al., 2026; Dey et al., 2025) or constrain how far the weights move (Huang et al., 2026). Other methods keep a frozen copy of the pre-trained model next to the adapted one, as a frozen encoder with a trainable branch (Grover et al., 2025), a frozen prior in an ensemble (Bagatella et al., 2025), or a frozen expert that guides an adaptation expert (Guo et al., 2026). Representation-space methods align the features of the adapted policy with those of a frozen copy or teacher on the target data (Dalal et al., 2026; Lin et al., 2026). These methods preserve the pre-trained policy through its architecture, its weights, or the representation of the target data. RARA instead anchors the representation of retrieved pre-training demonstrations: it aligns the target representation with the pre-trained representation of these demonstrations, and then limits how much it changes during fine-tuning, without restricting the architecture or which weights are updated.

Domain adaptation. Robot policies often face domain gaps between simulation and the real world, different embodiments, or pre-training and downstream deployment environments. Domain adaptation methods therefore seek task-relevant representations or data that transfer across such shifts. For VLA models, Kachaev et al. (2026) identify degradation of visual representations after action fine-tuning and preserve visual grounding through alignment with a frozen vision teacher, while Luo et al. (2026) show that action generation can become insufficiently sensitive to relevant visual tokens in deeper layers, and Kim et al. (2026) regularize VLA representations with a contrastive loss whose positives are defined by robot states. In sim-and-real co-training, Maddukuri et al. (2025) show that mixing simulated and real data improves real-world success, Cheng et al. (2025) align joint observation-action distributions with optimal-transport-inspired objectives, while Lei et al. (2026) combine adversarial alignment with explicit domain conditioning, which keeps the sim and real domains distinguishable in the learned representation space. Data selection offers a complementary route: Behavior Retrieval and FlowRetrieval retrieve prior data that are similar to the target demonstrations in state-action or motion space (Du et al., 2023; Lin et al., 2024), and Dass et al. (2026) select useful data from a pre-training dataset for a target imitation-learning task, improving the adapted policy with task-relevant source examples. Similarly, RARA selects task-relevant data from the pre-training dataset, but it uses the selected demonstrations as anchors for representation alignment rather than as additional training data: it aligns the target representation with that of the selected demonstrations and limits how much their representation changes during fine-tuning.

3 Problem Definition

We consider adapting a visuomotor policy \pi_{\theta_{\mathrm{pre}}}, pre-trained on a large and diverse set of robot demonstrations D_{\mathrm{pre}}, to a downstream target task with a limited target demonstration set D_{\mathrm{tgt}}=\{(o_{i},a_{i})\}_{i=1}^{N}. Here o_{i} denotes the full observation available to the policy, including visual inputs and any other language or state information, and a_{i} denotes the demonstrated action. Standard target-only fine-tuning updates the policy by behavior cloning,

\theta_{\mathrm{ft}}=\arg\min_{\theta}\mathbb{E}_{(o,a)\sim D_{\mathrm{tgt}}}\left[\mathcal{L}_{BC}\big(\pi_{\theta}(o),a\big)\right],

where \mathcal{L}_{BC} is the behavior cloning loss. While this objective directly optimizes performance on the in-distribution (ID) target data, a robust policy must also remain reliable under out-of-distribution (OOD) shifts after adaptation. Therefore, robust fine-tuning aims to learn a policy that achieves high expected task success under both ID and OOD evaluation distributions:

\max_{\theta}\;J_{\mathrm{ID}}(\pi_{\theta})+J_{\mathrm{OOD}}(\pi_{\theta}),\qquad J_{\mathcal{E}}(\pi_{\theta})=\mathbb{E}_{\tau\sim p_{\mathcal{E}}(\pi_{\theta})}\left[S(\tau)\right],\quad\mathcal{E}\in\{\mathrm{ID},\mathrm{OOD}\},

where S(\tau)\in\{0,1\} indicates task success for a trajectory \tau, and p_{\mathcal{E}} denotes the trajectory distribution induced by the policy \pi_{\theta} in evaluation setting \mathcal{E}.

4 RARA: Retrieval-Anchored Representation Alignment

RARA adapts a pre-trained policy to a target task in two stages. It first aligns the target representation to the pre-trained representation to obtain a good initial representation, and then fine-tunes the policy on the target task while constraining representation shifts of pretraining tasks. Both stages use demonstrations retrieved from the pre-training data (§4.1). In Stage 1, the target representation is aligned with their pre-trained representation, and in both stages their own representation is anchored.

Stage 1: Aligning to the pre-trained representation.

Before fully fine-tuning the policy, Stage 1 obtains an initial representation of the target demonstrations by aligning it to the pre-trained representation. Specifically, it aligns the representation of each target frame with the pre-trained representation of retrieved frames at the matched steps of the task (Fig. 2). An anchor loss limits how much this alignment changes the representation of the retrieved demonstrations. Only the layers that produce the representation and a projector that maps pooled representation into an embedding space are trained in this stage.

Stage 2: Fine-tuning on the target task.

Stage 2 fine-tunes the policy on the target demonstrations with its behavior cloning loss. The anchor loss stays active, so fine-tuning preserves the pre-trained representation on the retrieved demonstrations while the policy adapts to the target task. The projector is discarded, and only the fine-tuned policy is used at inference.

Formally, the two stages solve

\displaystyle\min_{\theta,\,\psi}\;\mathcal{L}_{\mathrm{contrastive}}+\lambda_{1}\,\mathcal{L}_{\mathrm{anchor}}, \\ \displaystyle\min_{\theta}\;\mathcal{L}_{\mathrm{BC}}+\lambda_{2}\,\mathcal{L}_{\mathrm{anchor}},

where \mathcal{L}_{\mathrm{BC}} is the behavior-cloning loss on the target demonstrations (Eq. 1), \mathcal{L}_{\mathrm{contrastive}} aligns target and retrieved representations, \mathcal{L}_{\mathrm{anchor}} limits the change of the retrieved representations, and \psi are the projector parameters. In Stage 1, \theta includes only the layers that produce the anchored representation; in Stage 2, the trainable parameters follow the standard fine-tuning recipe of the policy. The rest of this section describes retrieval (§4.1) and the two losses (§4.2).

4.1 Retrieving Relevant Pre-training Demonstrations

RARA uses pre-training demonstrations relevant to the target task as a reference for the policy’s representation update. We define relevance at two levels. A relevant demonstration performs the same type of task as the target, even though the objects and scenes may differ; we find such demonstrations by instruction similarity. A relevant frame is at the matched step of the task as a target frame; we find such frames by comparing action chunks (i.e., short sequences of consecutive actions). Both levels are necessary. Fig. 3 shows an example. Removing a pot lid and unstacking tissue rolls both lift an object upward, so their action chunks are similar even though the tasks are different. Action similarity alone would pair a target frame with a frame of a different task.

Retrieving demonstrations.

We embed task instructions with a frozen CLIP text encoder (Radford et al., 2021). Because a target task can have several instructions, we average their normalized embeddings. We then rank the pre-training demonstrations by the cosine similarity between their instruction embedding and this average, and keep the top K demonstrations.

Pairing frames by action chunks.

We split every demonstration in D_{\mathrm{ret}} and D_{\mathrm{tgt}} into overlapping action chunks A_{t}=[a_{t},\ldots,a_{t+H-1}] and represent each chunk by the observation o_{t} at its first step. Let \delta(A,A^{\prime}) be the dynamic-time-warping (DTW) distance between two action chunks, which matches similar actions executed at different speeds. Defining positives by action similarity follows CLASS (Lee et al., 2025); here the pairs connect target frames with retrieved pre-training frames. For a target frame o_{i}^{\mathrm{tgt}} with action chunk A_{i}^{\mathrm{tgt}}, we compute the DTW distance \delta(A,A_{i}^{\mathrm{tgt}}) from every retrieved chunk A to A_{i}^{\mathrm{tgt}}. The positives \mathcal{P}(i) are the frames of the m_{+} retrieved chunks with the smallest distance. The negatives \mathcal{N}(i) are frames sampled from the retrieved chunks whose distance is above the median of these distances. Using the median as the threshold is a practical choice that keeps the negatives clearly different from the target chunk while leaving many chunks to sample from. RARA uses the actions of the retrieved demonstrations only to form these pairs, not as behavior-cloning targets.

4.2 Aligning an Intermediate Representation

Let h_{\theta}^{\ell}(o) denote the representation at layer \ell of policy \pi_{\theta}, and let \bar{h}^{\ell}(o)=h_{\theta_{\mathrm{pre}}}^{\ell}(o) be the corresponding representation from a frozen copy of the pre-trained policy. The observation o includes the policy’s image, language and state inputs, as applicable.

Choosing the anchored layer.

We call \ell the anchored layer and h_{\theta}^{\ell} the anchored representation. Both the contrastive loss and the anchor loss act on this representation. The anchored layer can be any layer of the policy, from an early perception layer to a layer inside the action network, so RARA does not depend on a particular policy architecture. In practice, the target data usually differ from the pre-training data mainly in visual appearance. In this case, any layer that follows the visual input can serve as the anchored layer. We keep \ell general below; §5.1 gives the layer used in our experiments, and RQ3 (§5.4) shows that the gain is not tied to one layer.

Matching target representations to source references.

For a target observation o_{i}^{\mathrm{tgt}}, let \mathcal{P}(i) and \mathcal{N}(i) be its positive and negative frames from D_{\mathrm{ret}} (§4.1). We compare representations in a contrastive embedding space. The representation is pooled over tokens or cameras as needed and mapped to an \ell_{2}-normalized embedding by a trainable two-layer MLP projector g_{\psi}. Following common practice in contrastive learning, we compute the loss after this projector rather than directly on the representation, which has been shown to improve the learned representation and to reduce its dimensional collapse (Chen et al., 2020; Jing et al., 2022) (see Appx. F for details). The embedding of the target frame is computed with the adapting policy, z_{i}^{\mathrm{tgt}}=g_{\psi}(h_{\theta}^{\ell}(o_{i}^{\mathrm{tgt}})), and the embedding of a retrieved frame o_{j} is computed with the frozen pre-trained policy, \bar{z}_{j}=g_{\psi}(\bar{h}^{\ell}(o_{j})). A multi-positive contrastive loss (Oord et al., 2018; Khosla et al., 2020; Miech et al., 2020) then aligns each target embedding with the embeddings of its positive frames (Fig. 2):

\mathcal{L}_{\mathrm{contrastive}}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\sum_{p\in\mathcal{P}(i)}\exp\big(\mathrm{sim}(z_{i}^{\mathrm{tgt}},\bar{z}_{p})/\tau\big)}{\sum_{p\in\mathcal{P}(i)}\exp\big(\mathrm{sim}(z_{i}^{\mathrm{tgt}},\bar{z}_{p})/\tau\big)+\sum_{n\in\mathcal{N}(i)}\exp\big(\mathrm{sim}(z_{i}^{\mathrm{tgt}},\bar{z}_{n})/\tau\big)},

where B is the target batch size, \mathrm{sim}(u,v)=u^{\top}v/(\|u\|\,\|v\|) is cosine similarity, and \tau is the temperature. The pre-trained policy is frozen, so the representations \bar{h}^{\ell} of the retrieved frames stay fixed and serve as the reference for the anchor loss below; only their embeddings \bar{z}_{j} change as the projector is trained. The projector is used only for the contrastive loss and is discarded after Stage 1.

Anchoring source representations.

Updating the policy to represent the target also changes how it represents source observations. We constrain these changes with

\mathcal{L}_{\mathrm{anchor}}^{\ell}(\theta)=\mathbb{E}_{o\sim D_{\mathrm{ret}}}\left[d\!\left(h_{\theta}^{\ell}(o),\bar{h}^{\ell}(o)\right)\right],

where d measures the discrepancy between the two representations (see Appx. C for details). The loss compares the adapting and the pre-trained policy on the same frame of a retrieved demonstration but uses no action labels. It is computed only on the retrieved demonstrations, so it leaves the target representation free to change under the contrastive loss and, later, the task loss; it only limits how much these changes also alter the representation of the retrieved demonstrations. Together with the contrastive loss, it forms the Stage-1 objective in Eq. 3, and it stays active in Stage 2 (Eq. 4).

5 Experiments

We evaluate whether RARA learns a target task while generalizing beyond the conditions covered by its demonstrations, by fine-tuning two pre-trained policies on 13 tasks in simulation and on a real robot (§5.1). We aim to answer the following four research questions:

• RQ1: How does RARA change the representation compared with standard fine-tuning? (§5.2)
• RQ2: Does RARA improve ID and OOD success over fine-tuning and prior methods? (§5.3)
• RQ3: Does the gain depend on the choice of the anchored layer? (§5.4)
• RQ4: Which components and data contribute to the gain? (§5.5)

5.1 Setup

We evaluate RARA in two simulated environments and on a real robot (Fig. 4). In each setting, a pre-trained policy is fine-tuned on a small set of target demonstrations and evaluated both in-distribution (ID) and out-of-distribution (OOD). See Appx. D for the full details of setup.

• RoboTwin 2.0 (Chen et al., 2026). We use four single-arm target tasks, such as Place Container and Move Pill Bottle. ID uses the narrow layout of the target demonstrations; OOD widens the object layout, the backgrounds and the distractors. For pre-training, we collect about 2k demonstrations of various skills in RoboTwin, such as pick-and-place and pressing a bell, holding out the four target tasks. The pre-training data cover a larger workspace and more distractors, objects and background textures than the target demonstrations. Each target task has 50 demonstrations.
• RoboCasa (Nasiriany et al., 2026). We use five target tasks, such as Open Blender Lid and Close Fridge Drawer. ID uses the demonstration kitchen with randomized object poses; OOD changes the camera pose, background, distractors and object instances. Pre-training uses DROID (Khazatsky et al., 2024), which contains 76k demonstrations, and each target task has 100 demonstrations collected in one kitchen.
• Real Franka robot. We use four target tasks, such as Remove Pot Lid and Load Bread in Toaster. ID and OOD are defined as in RoboCasa. Pre-training uses DROID, and each target task has 50 to 80 demonstrations collected by VR teleoperation.

Policies.

We evaluate two common visuomotor policies:

• Diffusion Policy (Chi et al., 2023), a widely used visuomotor policy, which we pre-train on the RoboTwin pre-training data.
• GR00T N1.7 (Bjorck et al., 2025), a large vision-language-action (VLA) model pre-trained on DROID, which we use for RoboCasa and the real robot.

Since the target tasks differ from the pre-training data mainly in visual appearance, the anchored layer of both policies is the output of the layers that encode the visual input. All methods in a setting use the same optimizer settings and fine-tuning budget, and RARA additionally runs a short Stage 1 before fine-tuning. See Appx. C for details.

Evaluation Protocol.

We evaluate each policy on 150 rollouts per task and setting in RoboTwin, 50 in RoboCasa and 10 trials on the real robot, and repeat each experiment three times (see Appx. D for details). On the real robot, all methods are tested on the same recorded scene randomizations. We report the mean and standard deviation of the success rate.

Baselines.

We compare RARA with five baselines. They cover the two common ways to reuse pre-training knowledge during fine-tuning: training on pre-training data (Co-FT and STRAP) and keeping the policy close to the pre-trained one (RETAIN and BlindVLA). FT fine-tunes the policy on the target demonstrations only. Co-FT also trains on the retrieved pre-training demonstrations D_{\mathrm{ret}} (§4.1) with the behavior-cloning loss. STRAP (Memmel et al., 2025) instead retrieves pre-training sub-trajectories that match the target demonstrations under dynamic time warping, and co-trains on them. RETAIN (Yadav et al., 2026) merges the weights of the pre-trained and fine-tuned policies. BlindVLA (Kachaev et al., 2026) aligns the policy’s visual representations with those of a frozen pre-trained visual encoder during fine-tuning. See Appx. D.2 for details.

5.2 RQ1: How Does RARA Change the Representation?

Anchoring changes the representation of the retrieved pre-training frames as little as possible by design. RQ1 asks whether RARA also keeps the representation of target-task frames, on which the anchor is never applied, close to that of the pre-trained policy P, and whether it keeps the structure that P learned in pre-training. We compare the anchored representation of each adapted policy with that of P on Move Pill Bottle and Place Container in RoboTwin, whose objects do not appear in the pre-training data (Table 1; see Appx. D.3 for definitions). The left part of Table 1 is computed on held-out pre-training frames, and higher values mean that more of the pre-trained structure remains: effective rank (Garrido et al., 2023) measures how many directions the representation uses, CKA (Kornblith et al., 2019) measures its similarity to that of P, and the position-probe R^{2} (Davari et al., 2022) measures how well object positions can be read from it. The right part reports feature distortion (Kumar et al., 2022) on frames from held-out episodes of the target task, in the ID and OOD conditions; it measures how far adaptation moves the representation from that of P, and lower values mean smaller changes.

RARA keeps the representation of target-task frames close to that of the pre-trained policy P.

On both tasks and in both ID and OOD conditions, FT and Co-FT change the representation of target-task frames by 0.47–0.63, whereas RARA changes it by only 0.23–0.33, although the anchor is never applied to these frames. On pre-training frames, FT lowers the rank, CKA and probe R^{2} on both tasks, whereas RARA stays close to P (CKA 0.91–0.92, R^{2} 0.70–0.71). Co-FT, which trains on the retrieved demonstrations with their actions, also keeps a high CKA (0.93–0.94), but it changes the representation of target-task frames the most, and its success is similar to or lower than that of FT (Table 2). Keeping the structure of the pre-training frames is therefore not sufficient; RARA also keeps the target-task representation close to that of P, while improving success on both tasks.

Stage 1 aligns the representation and the Stage-2 anchor keeps it close to P.

Right after Stage 1, the representation of target-task frames is about as far from that of P as after FT (distortion 0.46–0.56, row “RARA, after Stage 1”). Stage 1 does not keep the representation close to P; it aligns the target-task frames with the retrieved frames. Fine-tuning with the anchor in Stage 2 then reduces the distortion to 0.23–0.33. RQ4 shows that both stages are needed for the full gain.

5.3 RQ2: Does RARA Improve ID and OOD Success?

This experiment tests the main claim of the paper: keeping the representation close to retrieved pre-training data should improve success under distribution shift, not only on the configuration seen in the demonstrations. We therefore report ID and OOD success separately (Table 2 for simulation and Fig. 5 for the real robot).

RARA achieves the highest task-average success in ID and in OOD in all three settings. Over FT, it gains 8.6 (ID) and 11.0 (OOD) points on RoboTwin, 4.8 and 11.2 points on RoboCasa, and 16.4 and 19.2 points on the real robot. In all three settings, the gain is larger in OOD than in ID, which is the goal of robust fine-tuning. The baselines, which reuse pre-training knowledge in other ways, improve less. Co-FT and STRAP, which train on pre-training demonstrations with their actions, gain at most 5.5 points in OOD, and Co-FT does not consistently improve over FT in simulation. RETAIN, which merges weights, falls well below FT in simulation (see Appx. H for a discussion), and BlindVLA, which aligns with a frozen visual encoder, gains at most 2.5 points in simulation and 8.6 points on the real robot. Since these baselines differ from RARA in more than one respect, we evaluate the contribution of each component of RARA separately in RQ4.

5.4 RQ3: Does the Choice of Anchored Layer Matter?

RARA aligns and anchors the representation at a single anchored layer, and the method does not fix which layer this is (§4.2). This experiment asks whether the choice matters. If only a few layers worked, a user would first have to search for the right layer before applying RARA.

We place the anchored layer at eight locations in the Diffusion Policy on Move Pill Bottle: the four ResNet stages, the encoder output (our default), and three places in the action head (the FiLM conditioning, the middle block and the output).

Every location improves over FT (Fig. 6, left): success rises from 21.2\% to between 32.0\% and 39.7\%, a gain of 11 to 18 points. Some locations work better than others, but none of them comes close to FT. On this task, the gain therefore does not depend on one particular layer. GR00T N1.7 shows a similar trend (Fig. 6, right): four of the five locations improve over FT by 4 to 10 points, and only an early layer of the vision encoder stays at the level of FT (Appx. E).

5.5 RQ4: Which Components and Which Data Produce the Gain?

We run an ablation study on RoboTwin, averaged over its four tasks, and on Open Blender Lid in RoboCasa (Table 3). We compare RARA with five variants. Stage 1 only fine-tunes without the anchor after Stage 1, and Stage 2 only skips Stage 1. Co-FT in Stage 2 adds behavior cloning on the retrieved demonstrations to Stage 2. Random pre-training data replaces the retrieved demonstrations with a random subset of the pre-training data of the same size. Random Stage-1 pairs pairs target and retrieved frames at random in Stage 1, instead of by action-chunk similarity.

Both stages are needed. On RoboTwin, Stage 1 only is no better than FT, and Stage 2 only recovers most of the ID gain (73.0\% vs. 76.0\%) but none of the OOD gain (46.2\%, vs. 47.3\% for FT and 58.3\% for RARA). On Open Blender Lid, each stage alone recovers part of the OOD gain, and only the full method reaches 56.7\%. A likely explanation is that each stage needs the other. Without the anchor (Stage 1 only), fine-tuning is free to move the representation again, so the alignment from Stage 1 is not preserved and the policy behaves like FT. Without Stage 1 (Stage 2 only), the anchor preserves the pre-trained representation of the retrieved demonstrations, but the target frames are not aligned with it, so the policy cannot reuse this robust representation in OOD conditions. Adding behavior cloning on the retrieved demonstrations (Co-FT in Stage 2) lowers success in all four columns, so the retrieved demonstrations are most useful as an anchor for the representation.

The choice of pre-training data and how its frames are paired also matter. With random pre-training data, RARA performs worse than FT on both RoboTwin and Open Blender Lid, by up to 17.1 points on RoboTwin. With random Stage-1 pairs, RARA falls below FT on RoboTwin (65.3\%/40.6\% vs. 67.4\%/47.3\%) and is only on par with FT on Open Blender Lid, so the action-aware pairing is needed. Appx. G further shows that the anchor helps when the pre-training data contain the same skill as the target task, and can hurt when the skill differs.

6 Conclusion

We presented RARA, a two-stage method for robust policy fine-tuning. RARA retrieves pre-training demonstrations of the same type of task as the target, first aligns the target representation to their pre-trained representation, and then fine-tunes the policy while anchoring the representation of these demonstrations. Across 13 tasks in simulation and on a real robot, RARA improves the average success of fine-tuning by 9.5 points in ID and 13.6 points in OOD. Our analysis shows that it also keeps the representation of target-task frames closer to that of the pre-trained policy, although the anchor is applied only to the retrieved demonstrations. While being effective, RARA relies on pre-training data that contain the same type of task as the target, and our experiments focus on visual changes with the same robot and action space. Future work includes retrieving useful references when such data are scarce, and extending RARA to new embodiments and action spaces.

Acknowledgments

This work was partly supported by Delta Electronics Inc. and Toyota Research Institute. This work used the Delta system at the National Center for Supercomputing Applications [award OAC 2005572] through allocation CIS261341 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program (Boerner et al., 2023), which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

References

1. Bagatella et al. (2025) Marco Bagatella, Jonas Hübotter, Georg Martius, and Andreas Krause. Active fine-tuning of multi-task policies. In International Conference on Machine Learning (ICML), 2025.
2. Bardes et al. (2022) Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations (ICLR), 2022.
3. Bjorck et al. (2025) Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GR00T N1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.
4. Black et al. (2025) Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. \pi_{0}: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025. doi: 10.15607/RSS.2025.XXI.010.
5. Boerner et al. (2023) Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. ACCESS: Advancing innovation: NSF’s advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing (PEARC ’23), 2023. doi: 10.1145/3569951.3597559.
6. Chen et al. (2026) Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In International Conference on Machine Learning (ICML), 2026.
7. Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597–1607, 2020.
8. Cheng et al. (2025) Shuo Cheng, Liqian Ma, Zhenyang Chen, Ajay Mandlekar, Caelan Reed Garrett, and Danfei Xu. Generalizable domain adaptation for sim-and-real policy co-training. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
9. Chi et al. (2023) Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023.
10. Dalal et al. (2026) Dwip Dalal, Shivansh Patel, Chahit Jain, Jeonghwan Kim, Utkarsh Mishra, Alex Baratian, Hyeonjeong Ha, Heng Ji, Svetlana Lazebnik, and Unnat Jain. Generalizable VLA finetuning via representation anchoring and language-action alignment. arXiv preprint arXiv:2607.13429, 2026.
11. Dass et al. (2026) Shivin Dass, Alaa Khaddaj, Logan Engstrom, Aleksander Madry, Andrew Ilyas, and Roberto Martín-Martín. DataMIL: Selecting data for robot imitation learning with datamodels. In International Conference on Learning Representations (ICLR), 2026.
12. Davari et al. (2022) MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing representation forgetting in supervised and unsupervised continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
13. Davari et al. (2023) MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, and Eugene Belilovsky. Reliability of CKA as a similarity measure in deep learning. In International Conference on Learning Representations (ICLR), 2023.
14. Dey et al. (2025) Sombit Dey, Jan-Nico Zaech, Nikolay Nikolov, Luc Van Gool, and Danda Pani Paudel. ReVLA: Reverting visual domain limitation of robotic foundation models. In IEEE International Conference on Robotics and Automation (ICRA), pp. 8679–8686, 2025.
15. Driess et al. (2025) Danny Driess, Jost Tobias Springenberg, Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z. Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, and Sergey Levine. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
16. Du et al. (2023) Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. In Robotics: Science and Systems (RSS), 2023.
17. Fei et al. (2026) Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. LIBERO-Plus: A progressive robustness benchmark for visual-language-action models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 38574–38583, 2026.
18. Garrido et al. (2023) Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning (ICML), 2023.
19. Grover et al. (2025) Shresth Grover, Akshay Gopalkrishnan, Bo Ai, Henrik I Christensen, Hao Su, and Xuanlin Li. Enhancing generalization in vision-language-action models by preserving pretrained representations. arXiv preprint arXiv:2509.11417, 2025.
20. Guo et al. (2026) Xinyu Guo, Bin Xie, Wei Chai, Xianchi Deng, Tiancai Wang, Zhengxing Wu, and Xingyu Chen. PriorVLA: Prior-preserving adaptation for vision-language-action models. arXiv preprint arXiv:2605.10925, 2026.
21. Han et al. (2024) Jinwei Han, Zhiwen Lin, Zhongyisun Sun, Yingguo Gao, Ke Yan, Shouhong Ding, Yuan Gao, and Gui-Song Xia. Anchor-based robust finetuning of vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
22. Hancock et al. (2026) Asher J Hancock, Xindi Wu, Lihan Zha, Olga Russakovsky, and Anirudha Majumdar. Actions as language: Fine-tuning VLMs into VLAs without catastrophic forgetting. In International Conference on Learning Representations (ICLR), 2026.
23. Huang et al. (2026) Chengyue Huang, Mellon M Zhang, Robert Azarcon, Glen Chou, and Zsolt Kira. MAPS: Preserving vision-language representations via module-wise proximity scheduling for better vision-language-action generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 32451–32462, 2026.
24. Jing et al. (2022) Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations (ICLR), 2022.
25. Kachaev et al. (2026) Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, and Aleksandr I. Panov. Don’t blind your VLA: Aligning visual representations for OOD generalization. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1864–1873, 2026. doi: 10.65109/PPER9186.
26. Khazatsky et al. (2024) Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems (RSS), 2024.
27. Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 18661–18673, 2020.
28. Kim et al. (2026) Taeyoung Kim, Jimin Lee, Myungkyu Koo, Dongyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, and Jinwoo Shin. Contrastive representation regularization for vision-language-action models. In International Conference on Machine Learning (ICML), 2026.
29. Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML), 2019.
30. Kumar et al. (2022) Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations (ICLR), 2022.
31. Lee et al. (2025) Sung-Wook Lee, Xuhui Kang, Brandon Yang, and Yen-Ling Kuo. CLASS: Contrastive learning via action sequence supervision for robot manipulation. In Conference on Robot Learning (CoRL), 2025.
32. Lei et al. (2026) Yu Lei, Minghuan Liu, Abhiram Maddukuri, Zhenyu Jiang, and Yuke Zhu. A mechanistic analysis of sim-and-real co-training in generative robot policies. arXiv preprint arXiv:2604.13645, 2026.
33. Liang et al. (2022) Anthony Liang, Ishika Singh, Karl Pertsch, and Jesse Thomason. Transformer adapters for robot learning. In CoRL 2022 Workshop on Pre-training Robot Learning, 2022. URL https://openreview.net/forum?id=H--wvRYBmF.
34. Lin et al. (2026) Haihao Lin, Xiangsheng Huang, Xiao Yang, Weibang Zhou, Yiqi Zhang, Bo Yang, Simin Zeng, Jiawei Yang, Zhengyang Wang, and Jiahui Du. FiberTune: Preserving action-fiber visual residuals in vision-language-action fine-tuning. arXiv preprint arXiv:2606.08653, 2026.
35. Lin et al. (2024) Li-Heng Lin, Yuchen Cui, Amber Xie, Tianyu Hua, and Dorsa Sadigh. FlowRetrieval: Flow-guided data retrieval for few-shot imitation learning. In Conference on Robot Learning (CoRL), 2024.
36. Liu et al. (2024) Zuxin Liu, Jesse Zhang, Kavosh Asadi, Yao Liu, Ding Zhao, Shoham Sabach, and Rasool Fakoor. TAIL: Task-specific adapters for imitation learning with large pretrained models. In International Conference on Learning Representations (ICLR), 2024.
37. Luo et al. (2026) Yulin Luo, Hao Chen, Zhuangzhe Wu, Bowen Sui, Jiaming Liu, Chenyang Gu, Zhuoyang Liu, Qiuxuan Feng, Jiale Yu, Shuo Gu, Peng Jia, Pheng-Ann Heng, and Shanghang Zhang. Look before acting: Enhancing vision foundation representations for vision-language-action models. arXiv preprint arXiv:2603.15618, 2026.
38. Maddukuri et al. (2025) Abhiram Maddukuri, Zhenyu Jiang, Lawrence Yunliang Chen, Soroush Nasiriany, Yuqi Xie, Yu Fang, Wenqi Huang, Zu Wang, Zhenjia Xu, Nikita Chernyadev, Scott Reed, Ken Goldberg, Ajay Mandlekar, Linxi Fan, and Yuke Zhu. Sim-and-real co-training: A simple recipe for vision-based robotic manipulation. In Robotics: Science and Systems (RSS), 2025.
39. Memmel et al. (2025) Marius Memmel, Jacob Berg, Bingqing Chen, Abhishek Gupta, and Jonathan Francis. STRAP: Robot sub-trajectory retrieval for augmented policy learning. In International Conference on Learning Representations (ICLR), 2025.
40. Miech et al. (2020) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9879–9889, 2020.
41. Mukhoti et al. (2024) Jishnu Mukhoti, Yarin Gal, Philip H. S. Torr, and Puneet K. Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. Transactions on Machine Learning Research (TMLR), 2024.
42. Nasiriany et al. (2026) Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, and Yuke Zhu. RoboCasa365: A large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations (ICLR), 2026.
43. O’Neill et al. (2024) Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), pp. 6892–6903, 2024.
44. Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
45. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 8748–8763, 2021.
46. Shang et al. (2024) Jinghuan Shang, Karl Schmeckpeper, Brandon B. May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. In Conference on Robot Learning (CoRL), 2024.
47. Sharma et al. (2023) Mohit Sharma, Claudio Fantacci, Yuxiang Zhou, Skanda Koppula, Nicolas Heess, Jon Scholz, and Yusuf Aytar. Lossless adaptation of pretrained vision models for robotic manipulation. In International Conference on Learning Representations (ICLR), 2023.
48. Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
49. XPolicyLab Community et al. (2026) XPolicyLab Community, Tianxing Chen, et al. XPolicyLab: A unified standard and open ecosystem for robot policy evaluation and deployment. arXiv preprint arXiv:2608.09892, 2026.
50. Xu et al. (2023) Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper-decision transformer for efficient online policy adaptation. In International Conference on Learning Representations (ICLR), 2023.
51. Yadav et al. (2026) Yajat Yadav, Zhiyuan Zhou, Andrew Wagenmaker, Karl Pertsch, and Sergey Levine. Robust fine-tuning of vision-language-action robot policies via parameter merging. In International Conference on Learning Representations (ICLR), 2026.
52. Yang et al. (2026) Shuai Yang, Hao Li, Bin Wang, Yilun Chen, Yang Tian, Tai Wang, Hanqing Wang, Feng Zhao, Yiyi Liao, and Jiangmiao Pang. Vision-language-action instruction tuning: From understanding to manipulation. In International Conference on Learning Representations (ICLR), 2026.

Appendix A Analysis of Visual Feature Shift

We discuss visual feature alignment for VLA adaptation. A pre-trained VLA visual encoder can separate diverse source tasks, but downstream image from a new simulator or real setup can be different. Even when the action space, language instruction are similar, such image shifts can leave target features poorly organized for action prediction.

Fig. 7 illustrates this issue. We extract features from a DROID-pretrained \pi_{0} policy and visualize them with PCA, using five DROID tasks as ID source tasks and five RoboCasa tasks as OOD downstream tasks (Nasiriany et al., 2026); details are in §A.1. DROID features are well separated, while unseen RoboCasa features collapse into a compact region despite representing different manipulation tasks. Thus, the DROID-pretrained visual encoder is not sufficiently task-discriminative in the unseen environment.

Such feature collapse is problematic for policy adaptation. If the visual encoder maps different target states or tasks into a collapsed region, the action model receives weak state information and can easily overfit the few available demonstrations. This motivates visual feature alignment: before learning target actions, we first move target-domain visual features toward a task-relevant region of the pre-trained feature space, making the new observations easier for the pre-trained policy to interpret and providing a better starting point for adaptation.

A.1 PCA Visualization Details

For the \pi_{0} feature visualizations, we extract image features from the SigLIP visual encoder before any pooling operation. For each image and each camera view, the encoder outputs 196 patch tokens with feature dimension 1152. We average over the 196 patch tokens to obtain one 1152-dimensional feature vector per camera. The three camera-specific vectors are then concatenated, giving a 3456-dimensional visual feature for each multi-view observation. Stacking features over all selected observations yields a matrix in \mathbb{R}^{N\times 3456}. We fit PCA to this feature matrix using SVD and project each observation onto the top two principal components for visualization. When comparing multiple data sources or adaptation checkpoints, we fit PCA once on the combined feature set and use the same projection for all points.

Appendix B Retrieval and Pair Construction Details

This appendix details how RARA retrieves pre-training demonstrations and pairs their frames with target frames (§4.1). The same procedure is used for DROID and for RoboTwin.

Retrieval by instruction.

Each demonstration comes with a task instruction, and a target task usually has several instructions; on RoboTwin, they are generated from templates. Let e(g) be the embedding of an instruction g from a frozen CLIP text encoder (Radford et al., 2021). For a target task with instruction set \mathcal{G}_{\mathrm{tgt}}, we average the normalized instruction embeddings,

\bar{e}_{\mathrm{tgt}}=\frac{1}{|\mathcal{G}_{\mathrm{tgt}}|}\sum_{g\in\mathcal{G}_{\mathrm{tgt}}}\frac{e(g)}{\|e(g)\|_{2}}.

We score each pre-training demonstration by the cosine similarity between the embedding of its instruction and \bar{e}_{\mathrm{tgt}}, and keep the top K=100 demonstrations as D_{\mathrm{ret}}. Filtering by instruction separates tasks with different intent, such as opening, closing, placing, wiping and insertion, even when their action chunks contain similar end-effector displacements.

Action chunks.

We represent each demonstration as \tau=\left(g,\{o_{t},q_{t},a_{t}\}_{t=1}^{T}\right), where g is the instruction, o_{t} the image observation, q_{t} the proprioceptive state, and a_{t} the action in the action space of the policy. Each demonstration is split into overlapping action chunks

A_{t}=[a_{t},a_{t+1},\ldots,a_{t+H-1}],\qquad t\in\{1,1+s,1+2s,\ldots\},

with horizon H=16 and stride s, and each chunk is paired with the observation at its first step. Continuous action dimensions are normalized with statistics of the pre-training data, and the gripper dimension is scaled separately so that binary open/close changes do not dominate the translational and rotational dimensions. Target chunks are normalized in the same way.

DTW distance.

For two action chunks A and B, we use the dynamic-time-warping distance

\delta(A,B)=\min_{\omega\in\Omega(A,B)}\frac{1}{|\omega|}\sum_{(u,v)\in\omega}\left\|A_{u}-B_{v}\right\|_{2},

where \Omega(A,B) is the set of monotone temporal alignments between the two chunks. DTW matches similar actions that are executed at different speeds.

Positive and negative pairs.

For each target chunk A_{i}^{\mathrm{tgt}}, we compute \delta to every chunk of the demonstrations in D_{\mathrm{ret}}. The m_{+} closest chunks are the positives \mathcal{P}(i); they can come from any retrieved demonstration. The negatives \mathcal{N}(i) are sampled from the retrieved chunks whose distance to A_{i}^{\mathrm{tgt}} is larger than the median of these distances. Each chunk is represented by the observation at its first step. Because pairs are drawn only from D_{\mathrm{ret}}, a target frame is not paired with a frame of an unrelated task that happens to have similar actions, such as a pushing or placing segment for a lid-opening target task.

Figure 8 shows additional examples of retrieval with and without the instruction filter.

Appendix C Implementation Details

Anchored representation.

Our target tasks differ from the pre-training data mainly in visual appearance, so we anchor the representation computed from the visual input. For the Diffusion Policy, this is the pooled output of the ResNet-18 encoders of all cameras, concatenated into a 1024-dimensional feature. For GR00T, it is the output tokens of the vision tower. Stage 1 trains the layers that produce this representation, together with the projector (Appx. F), for 3 000 steps, and keeps the rest of the policy fixed.

Anchor distance.

The distance d in the anchor loss (Eq. 6) is the mean squared error for the Diffusion Policy. For GR00T, we divide the squared error by the batch variance of the reference representation \bar{h}^{\ell}, so that the same weight can be used for representations of different scales.

Retrieval and training budget.

In all settings, we retrieve K=100 demonstrations and use action chunks of H=16 steps (Appx. B). All methods in a setting use the same optimizer settings and training budget. Appx. D.2 lists the full settings of the Diffusion Policy on RoboTwin.

Appendix D Evaluation Environments

This appendix describes the RoboCasa and real-robot settings (§D.1), the RoboTwin setting (§D.2), and the representation metrics used in RQ1 (§D.3).

D.1 RoboCasa and Real Robot

Figures 9 and 10 show the training and OOD evaluation environments of all RoboCasa and real-robot tasks.

Evaluation protocol.

In RoboCasa, each policy is evaluated for 50 episodes per task, and each experiment is repeated with three training seeds. For GR00T-based policies, the model predicts an action chunk of 16 steps and executes all 16 steps before replanning. In the real-world Franka experiments, each trained policy is evaluated for 10 trials per task, and this evaluation is repeated three times. For Put 3 Toys in Basket, each trial is scored by the number of toys placed in the basket, so the success rate is the fraction of the three toys placed. To improve fairness across real-world comparisons, we record the randomization setting for each trial, including scene layout, object placement, distractors, and other visual variations. We then evaluate the next model under the same recorded setting whenever possible before switching to a new randomized setting. For real-world GR00T-based policies, the model predicts a 16-step action chunk but executes only the first 8 steps before replanning.

D.2 RoboTwin

RoboTwin 2.0 (Chen et al., 2026) makes it easy to collect demonstrations, and its randomization of object instances, backgrounds, clutter and object positions gives enough diversity to build both a broad pre-training set and narrower target tasks in the same environment. We use the right arm of the Aloha-AgileX embodiment with the head and wrist RGB cameras.

Pre-training and target data.

We collect about 2,000 pre-training demonstrations in total, in two groups, each paired with two target tasks.

The first group contains pick-and-place demonstrations (place_a2b_right), in which the right arm picks up an object and places it next to a second object. The first object is drawn from 11 categories (playing cards, mouse, bell, stapler, soap, toy car, coffee box, phone, Rubik’s cube, bread and tea box), and the second from six (Rubik’s cube, wooden block, phone, bread, bell and soap). This group serves Place Container and Move Pill Bottle. These targets use the same skill, but their objects do not appear in the pre-training data: Move Pill Bottle uses a pill bottle and a blue pad, and Place Container uses a plate together with a bowl or a cup. The novelty of these two targets is therefore at the object level.

The second group contains four skills: pressing a bell (click_bell), shaking a bottle vertically (shake_bottle), opening a laptop (open_laptop) and rotating a QR code (rotate_qrcode). This group serves Click Alarm Clock and Shake Bottle. Click Alarm Clock uses the same pressing skill as click_bell on a new object, and Shake Bottle shakes the same kind of object horizontally instead of vertically; neither target task appears in the pre-training data. Opening a laptop and rotating a QR code are unrelated to both targets, and retrieval is meant to exclude them.

The pre-training demonstrations are collected with wide randomization: 100 table textures and backgrounds, cluttered distractors, and object positions over the whole reachable table. Each target task has 50 demonstrations, collected with a narrow layout, 10 backgrounds and light clutter. ID evaluation uses the narrow layout of the demonstrations; OOD evaluation widens the object layout, the backgrounds and the distractors. Figure 11 shows the training and OOD evaluation environments of the four target tasks.

Policy and training.

We use the Diffusion Policy (Chi et al., 2023) implementation in XPolicyLab (XPolicyLab Community et al., 2026): a ResNet-18 encoder per camera with group normalization and spatial soft-argmax pooling (1024-dimensional visual feature over both cameras), a 1-D UNet denoiser, and 16-step action chunks. It is pre-trained for 120 epochs on the pre-training set. Every adaptation method then trains for 300 epochs on the 50 target demonstrations with a batch of 128. Co-FT fills each batch with 64 target chunks and 64 chunks of the retrieved pre-training demonstrations D_{\mathrm{ret}}, and applies the behavior-cloning loss to both. In Stage 2 of RARA, each batch holds 64 target chunks and 64 pre-training frames; the behavior-cloning loss is applied to the target chunks and \mathcal{L}_{\mathrm{anchor}} (mean squared error, \lambda_{2}=1) to the pre-training frames, without their actions. The frozen reference in RQ1 is FT with the layers that produce the anchored representation fixed. Table 4 lists the remaining settings.

Anchored layer.

By default, the anchored representation h^{\ell} is the pooled output of the two ResNet-18 encoders, concatenated into a 1024-dimensional feature. Stage 1 updates these encoders and the projector. Appx. E describes the other locations used in RQ3.

Stage 1.

Retrieval and pair construction follow §4.1 and Appx. B: the top K=100 pre-training demonstrations are retrieved by instruction similarity, and the positives and negatives are defined by the DTW distance between action chunks, with m_{+}=4. Stage 1 converges on both targets: the contrastive loss falls from 2.73 to 0.22 on Place Container (92\%) and from 2.62 to 0.87 on Move Pill Bottle (67\%).

Baselines.

All baselines start from the same pre-trained policy and use the same 50 target demonstrations and 300-epoch budget as FT. For weight-space preservation (RETAIN; Yadav et al., 2026), we interpolate the pre-trained and fine-tuned weights as \theta(\alpha)=(1-\alpha)\theta_{\mathrm{pre}}+\alpha\theta_{\mathrm{ft}} with \alpha\in\{0.25,0.5,0.75\}. For same-image teacher alignment (BlindVLA; Kachaev et al., 2026), we add a feature-distillation term toward frozen DINOv2 features of each target image, with weight \lambda\in\{1,0.1,0.01\}. For retrieval and co-training (STRAP; Memmel et al., 2025), we follow the original method: the image observations are encoded with DINOv2, target sub-trajectories are matched to pre-training sub-trajectories by subsequence DTW over these visual features, and we keep the top 50 matches and train with the Co-FT loss and the same 1:1 batch mixture.

Evaluation.

Rollouts use the benchmark’s success check with a 400-step horizon. Each result uses the last three checkpoints (epochs 250, 275 and 300), each evaluated on 50 rollouts, so that checkpoint-to-checkpoint noise is averaged rather than selected. Each experiment is repeated with three training seeds, and we report the mean and standard deviation over the seeds.

D.3 RQ1 Metrics

All metrics use the anchored representation, the pooled outputs of the two ResNet-18 encoders concatenated into a 1024-dimensional feature, on frames the policy was not trained on: 100 held-out target episodes in the ID region, 100 in the OOD region, and pre-grasp frames of held-out pre-training episodes. f_{\theta} denotes the adapted encoder and f_{P} the pre-trained one.

Feature distortion (Kumar et al., 2022) is \mathbb{E}_{o}\,\|f_{\theta}(o)-f_{P}(o)\|_{2}/\|f_{P}(o)\|_{2} over the frames of a set; 0 means the features did not move. We divide by the norm of the pre-trained feature so that a change of scale does not dominate.

Effective rank (Garrido et al., 2023) is \exp(-\sum_{i}p_{i}\log p_{i}) with p_{i}=\sigma_{i}/\sum_{j}\sigma_{j}, where \sigma_{i} are the singular values of the centered feature matrix of a set.

Linear CKA (Kornblith et al., 2019) is \|X^{\top}Y\|_{F}^{2}/(\|X^{\top}X\|_{F}\,\|Y^{\top}Y\|_{F}) for the centered feature matrices X of the adapted and Y of the pre-trained encoder on the same frames; 1 means the two representations agree up to rotation and scale. Because CKA can change without a change in function (Davari et al., 2023), we do not rely on it alone.

Position-probe R^{2} (Davari et al., 2022) is the score of a ridge regression from the features to the object position at the grasp of the same episode (32 PCA components, \alpha=10, five-fold GroupKFold by episode). Frames after the grasp are excluded, because the object then moves with the hand. A drop relative to P indicates forgetting.

Appendix E Choice of Anchored Layer

This section gives the setup of the layer study in RQ3 (§5.4, Fig. 6). Both policies are evaluated on Move Pill Bottle in RoboTwin, and each arm changes only the anchored layer.

Diffusion Policy.

Besides the default location, the pooled output of the two ResNet-18 encoders, we anchor seven other locations: the outputs of the four ResNet stages (spatially pooled), the FiLM conditioning vector of the UNet, the output of the UNet middle block, and the predicted action. All arms use \lambda_{1}=\lambda_{2}=1. When the anchored layer is inside the action network, the diffusion timestep and the noisy action input are shared between the trainable and the reference policy, so the two differ only in their weights. All other settings follow Appx. D.2.

GR00T.

We use GR00T N1.7 trained on the RoboTwin pre-training data. As in §4, Stage 1 trains only the layers that produce the anchored representation. In Stage 2, all anchor arms train the vision encoder and the action head with the LLM frozen, the same trainable parameters as the FT reference. Any difference from FT therefore comes from where the anchor is applied, not from which parameters are fine-tuned. The LLM is frozen in Stage 2 because the official GR00T fine-tuning recipe keeps it frozen. To study anchored layers inside the LLM, such as the output of the vision-language self-attention, Stage 1 also trains the LLM layers that produce this representation. In our default setting, which anchors the output of the visual encoder, RARA improves over FT whether the LLM is frozen or trained. We anchor five locations: the output of layer 6 of the 24-layer ViT, the output of the visual encoder (our default), the output of the vision-language self-attention, the output of the DiT action head, and the predicted action. All arms use \lambda_{1}=\lambda_{2}=0.1.

Evaluation.

For both policies, each point in Fig. 6 averages the last three checkpoints, and the error bars show the standard deviation over these checkpoints. Each GR00T checkpoint is evaluated on 100 ID and 100 OOD rollouts.

Appendix F Projector Analysis

Stage 1 computes the contrastive loss in a projected space. A shared two-layer MLP projector (1024\!\to\!512\!\to\!256, GELU, \ell_{2}-normalized output) maps the anchored representation of both the target frames and the retrieved frames to the contrastive embedding, following common practice in contrastive learning (Chen et al., 2020; Bardes et al., 2022). The projector is trained together with the policy (learning rate 10^{-4}, compared with 10^{-5} for the policy layers) and is discarded after Stage 1, so Stage 2 starts from the updated policy without it. We compare this default with two alternatives on RoboTwin, using the policy pre-trained with one seed: no projector, where the loss is applied directly to the normalized representation, and a randomly initialized frozen projector, which Kachaev et al. (2026) use when aligning with a frozen teacher. Each success rate is computed over 150 rollouts.

Removing the projector does not change the result of Stage 1 followed by fine-tuning without the anchor: 86.7\%/60.7\% vs. 87.3\%/60.0\% (ID/OOD) on Place Container, and 30.7\% vs. 32.0\% (ID) on Move Pill Bottle. A frozen projector changes Stage 1 itself. The representation aligns more strongly: on held-out Move Pill Bottle frames, the AUC of ranking a positive frame above the negatives improves by 8.1 points over the pre-trained policy, compared with 5.1 points with the trainable projector. The representation also moves further from that of the pre-trained policy (CKA on pre-training frames 0.86 vs. 0.95), while the contrastive loss stays two to three times higher. A trainable projector therefore absorbs part of the alignment that would otherwise change the representation. These differences do not carry over to success. With either projector, Stage 1 followed by fine-tuning reaches 32.0\% ID on Move Pill Bottle and 85.3\% vs. 87.3\% on Place Container, and full RARA reaches 49.3\% vs. 53.3\% and 82.0\% vs. 84.7\% (frozen vs. trainable projector, ID). We therefore use the trainable projector by default.

Appendix G Choice of Pre-training Data

This section studies which pre-training data RARA should align to. To test whether the skill shown in the data matters, we swap the pre-training data while keeping everything else fixed: episode count, randomization, cameras and acting arm (Table 5). With pick-and-place data, RARA gains 18.8 and 4.9 points on the two pick-and-place targets, but only 0.2 points on Click Alarm Clock. With Click Bell data, the pattern reverses: RARA gains 10.3 points on Click Alarm Clock, a pressing task like Click Bell, and loses 6.7 and 20.0 points on the pick-and-place targets. The anchor therefore tends to help when the pre-training data shows the same skill as the target, and it can hurt when the skill differs. This is why RARA retrieves the pre-training data instead of using all of it.

Appendix H RETAIN Under a Large Pre-training–Target Gap

RETAIN (Yadav et al., 2026) keeps the adapted policy close to the pre-trained one by interpolating their weights, \theta(\alpha)=(1-\alpha)\theta_{\mathrm{pre}}+\alpha\theta_{\mathrm{ft}} (Appx. D.2). In our experiments, how well this works depends on how far the target task is from the pre-training data. Table 6 compares RETAIN with FT in the two simulated settings, and Table 7 gives the per-task RoboCasa results that are left out of Table 2.

Observation.

On RoboTwin, the pre-training and target data come from the same simulator, robot and cameras. RETAIN stays below FT on average but keeps about 70% of its success (47.0% vs. 67.4% ID, 34.2% vs. 47.3% OOD), and on Move Pill Bottle it even improves OOD success over FT (13.3% vs. 9.6%). On RoboCasa, the policy is pre-trained on real-world DROID data and adapted to simulated kitchens. Here RETAIN keeps less than half of the success of FT (31.1% vs. 69.2% ID, 14.8% vs. 44.3% OOD), and it falls below FT on all five tasks in both ID and OOD, down to 3.3% OOD success on Pick-Place to Drawer.

Analysis.

Weight interpolation tends to work when the pre-trained and fine-tuned weights lie in one connected region of low loss, which is typical when fine-tuning stays close to the pre-training distribution (Wortsman et al., 2022). When the target is close to the pre-training data, fine-tuning needs only a small change of the weights, so moving part of the way back toward the pre-trained weights keeps most of the target skill. A likely explanation for the RoboCasa result is that the gap from real-world DROID images to simulated kitchens requires a much larger change of the weights. The interpolated weights then lie between two solutions, and neither the pre-trained nor the fine-tuned behavior is preserved, so both ID and OOD success drop. RARA does not constrain the weights. It constrains the representation of the retrieved demonstrations, so the policy can still change as much as the target task needs, which is consistent with its gains on RoboCasa (Table 2).