Evo-RL: Iterative Robot Reinforcement Learning from Selective Experience

Abstract

Existing real-world reinforcement learning methods improve robot policies by combining human corrections with autonomous rollouts. However, accumulated experience varies in its value for policy improvement, and even successful trajectories can contain inefficient or redundant behavior, motivating more selective experience reuse. To this end, we propose Evo-RL, an iterative reinforcement learning method that combines human interventions with advantage-guided experience selection. Evo-RL uses human corrections and experience with positive estimated advantages from successful trajectories to iteratively refine the policy. To support learning from these different experience sources, we introduce RW-RL-Dataset, which spans over 1,000 hours across four robot embodiments. On a real-world high-precision manipulation task, Evo-RL improves attempt success rate over RECAP by 30.46%. Our code, model checkpoints, and dataset are publicly available.

I Introduction

Vision-language-action (VLA) models generate robot actions by conditioning on visual observations and language instructions [1]. Recent advances have extended their applicability to a wider range of tasks and robot embodiments [2, 3, 4] and improved deployment efficiency [5]. Yet policies trained on demonstrations can still encounter unfamiliar states during deployment, where errors can accumulate and recovery may be difficult [6]. These limitations motivate continued policy improvement using experience collected during execution.

Existing approaches support this improvement through human corrections and reinforcement learning. Human interventions supply corrective behavior at states visited by the deployed policy [7, 8], while autonomous rollouts provide feedback beyond the initial demonstrations. RECAP [9] combines demonstrations, autonomous rollouts, and human interventions, using value-estimated advantages to condition policy training and iteratively update the policy with deployment data.

The value of accumulated experience for policy improvement varies across and within trajectories. Even successful trajectories may contain hesitation, repeated attempts, or other inefficient behavior, so the contribution of experience to policy improvement must be assessed at a finer level. Human corrections target deployment difficulties, but policy actions in the same trajectories may include behavior that prompted the intervention. These observations motivate Evo-RL’s selective update procedure: estimated advantages guide the selection of experience for policy updates, while only human-controlled portions of successful intervention trajectories are retained for corrective fine-tuning.

We propose Evo-RL, an iterative real-world reinforcement learning method that combines human corrections with advantage-guided experience selection. Starting from a demonstration-trained policy, Evo-RL collects human interventions and autonomous rollouts during deployment. It trains a value model on demonstrations, intervention data, and autonomous experience, using rewards that account for execution time and human takeovers, and then estimates advantages to guide experience selection. For policy fine-tuning, Evo-RL retains only the human-controlled portions of successful intervention trajectories, keeps the original demonstrations, and adds selected positive-advantage experience from demonstrations and successful autonomous rollouts. RECAP trains the policy on accumulated experience with a binary advantage label as an additional input. Evo-RL instead uses estimated advantages to select experience for policy updates and increase the sampling weight of selected experience. The updated policy is then redeployed for the next iteration.

To advance the real-world RL research, we further introduce RW-RL-Dataset which organizes demonstrations, human interventions, and autonomous experience with action-source metadata and semantic annotations. It spans over 1,000 hours across four diverse robot embodiments and includes the task-specific training experience used in our experiments.

Our main contributions are as follows:

• We propose Evo-RL, an iterative method that combines corrective supervision with selective reuse of experience guided by positive estimated advantages. Under our evaluation protocol, first-round Evo-RL achieves an attempt success rate (ASR) 30.46% higher than RECAP on a real-world manipulation task requiring high-precision alignment.
• We present RW-RL-Dataset, a collection of over 1,000 hours of real-world data from four robot embodiments, pairing synchronized camera observations with robot states and actions. Its annotations specify task instructions and scene context, execution quality and error counts, subtask intervals, and whether each frame records human or policy control.

II Related Work

II-A Vision-Language-Action Model and Robot Datasets

Large-scale robot datasets collectively cover diverse household, office, and industrial manipulation tasks across single-arm, dual-arm, and humanoid platforms [10, 11, 12, 13, 14]. This diversity provides a foundation for VLA pretraining, enabling policies to learn reusable manipulation skills across tasks and robots [14]. Building on these skills, models such as OpenVLA can adapt to new tasks and robot setups through fine-tuning on additional demonstrations [2]. However, even after fine-tuning, policies may still fail in situations poorly covered by datasets. Human interventions provide targeted corrections in these situations, yet the resulting corrective experience is less systematically captured and organized.

II-B Human Interventions and Data Aggregation

Out-of-distribution (OOD) situations refer to scenarios that differ from those encountered during training and may therefore challenge the policy’s ability to generalize. During deployment, human interventions can be introduced in such OOD situations, not necessarily only after an execution error occurs. These interventions provide corrective demonstrations that can be incorporated into the training data, helping the policy improve its responses when similar situations arise in the future. Dataset Aggregation (DAgger) formalizes this process by querying expert actions for states visited by the current policy and aggregating the resulting state–action pairs into the training set [6].

Building on this paradigm, subsequent research focuses on two related questions: when human interventions should be triggered and how the resulting correction data should be incorporated into policy learning. Some methods let human operators decide when to intervene [7], whereas others use state novelty or failure risk to determine when to request human assistance [15]. Because intervention samples often constitute only a small fraction of the collected data, their influence on policy updates can be limited. IWR addresses this imbalance by increasing the relative weight of intervention samples, while using non-intervention samples to regularize policy updates and help retain previously learned skills [8]. However, corrective action labels alone do not explicitly indicate which states are likely to lead to failure or which trajectories are of higher quality. Incorporating such evaluations can provide additional supervisory signals for further policy improvement.

II-C Real-World RL and Iterative Policy Improvement

Reinforcement learning enables robots to improve their policies through interaction with the environment. Early work on RL for VLA models focused on offline preference alignment[16, 17], such as GRAPE [17], which employs Direct Preference Optimization (DPO) [18] to align robot trajectories with human intent. More recent approaches incorporate experience collected during deployment into VLA post-training. RECAP trains a value function on demonstrations, human interventions and autonomous rollouts then uses advantage-based indicators to condition policy training and guide iterative improvement[9]. To reduce the cost of online adaptation, RLT compresses latent representations into an RL token and trains a lightweight actor–critic to refine action chunks while keeping the VLA frozen[19]. Beyond action-level optimization, LaST-R1 extends reinforcement learning to the latent space, jointly optimizing reasoning and execution[20]. Our Evo-RL is motivated by RECAP, as both methods combine human corrections with autonomous rollouts to realize value-guided iterative policy improvement.

III Method

Evo-RL organizes policy improvement into a data loop with demonstration-based initialization followed by two complementary stages: fine-tuning with human interventions and advantage-guided policy refinement. Starting from demonstration data, we initialize the policy by fine-tuning a pretrained VLA model. Next, human interventions collected during deployment are used for policy fine-tuning to improve performance in challenging scenarios. Following intervention-based refinement, we use accumulated experience for advantage-guided policy updates. Algorithm 1 summarizes this loop and the policy updates within each iteration.

III-A Demonstration-Based Policy Initialization

At time step t, the policy receives an observation o_{t}=(I_{t},s_{t}) and a task instruction \ell, where I_{t} denotes the camera images and s_{t} the robot’s proprioceptive state. The policy models a conditional distribution over action chunks:

\mathbf{a}_{t}=(a_{t},\ldots,a_{t+H-1})\sim\pi_{\theta}(\cdot\mid o_{t},\ell),

where H is the chunk length and \theta denotes the policy parameters. Following the architecture of \pi_{0.5} [21], our policy combines a PaliGemma vision-language backbone with a flow-matching action expert to generate action chunks conditioned on visual observations, proprioceptive state, and task instructions. We train the policy on teleoperated demonstrations D_{demo} using the flow-matching objective.

For each demonstration sample (o_{t},\ell,\mathbf{a}_{t})\sim D_{demo}, we sample standard Gaussian noise \epsilon with the same shape as \mathbf{a}_{t}, and a flow time \tau\in[0,1]. The noisy action chunk is constructed as

x_{t}^{\tau}=(1-\tau)\mathbf{a}_{t}+\tau\epsilon.

The action expert v_{\theta} predicts the corresponding velocity \epsilon-\mathbf{a}_{t}, conditioned on o_{t} and \ell. The objective function can be written as

\mathcal{L}_{\mathrm{demo}}(\theta)=\mathbb{E}\!\left[\operatorname{MSE}\!\left(v_{\theta}(x_{t}^{\tau},o_{t},\ell,\tau),\epsilon-\mathbf{a}_{t}\right)\right],

While this objective enables effective behavioral cloning, the model’s capability and generalization remain inherently bounded by the quality, scale, and diversity of the demonstration data. This fine-tuning produces the initial task policy \pi_{0}, which is subsequently deployed to collect experience for further improvement.

III-B Policy Refinement with Human Interventions

The initial policy may still struggle in challenging scenarios. We therefore use human-supervised rollouts to collect corrective actions. An operator monitors task execution and intervenes when the policy encounters states poorly covered by the demonstrations and struggles to recover or make further progress. After each correction, the robot resumes autonomous operation. For an intervention trajectory indexed by i, we represent the information collected at each frame t as

S_{i,t}=(\ell_{i},o_{i,t},a_{i,t},m_{i,t}),

where \ell_{i} is the task instruction, o_{i,t} contains images and proprioceptive state, and a_{i,t} is the executed action. We introduce the control-source label m_{i,t} to distinguish human corrections from policy actions, with 1 indicating human control and 0 indicating policy control. A trajectory of length T_{i} is written as

\xi_{i}=(S_{i,1},\ldots,S_{i,T_{i}}),

with y_{i}\in{0,1} indicating whether the task succeeds. We retain human-controlled data from successful intervention trajectories to construct the dataset D_{\mathrm{interv}}:

D_{\mathrm{interv}}=\left\{S_{i,t}\mid y_{i}=1,\ m_{i,t}=1\right\}.

We preserve the temporal order and segment boundaries of these data and sample training action chunks only within contiguous human-controlled segments. Policy-controlled portions are excluded from D_{\mathrm{interv}} to avoid learning from actions that are likely to cause failures or prolong task completion. During training, we fine-tune the current policy with the same loss function, sampling from D_{\mathrm{interv}} and D_{demo} with probabilities p\in(0,1) and 1-p, respectively. The updated policy is then redeployed to identify remaining difficulties and collect further corrections. This cycle continually incorporates human corrections into training, allowing the policy to learn from its mistakes and improve task performance over successive updates.

III-C Reinforcement Learning for Policy Refinement

Demonstration training and intervention-based refinement provide an initial policy and corrective supervision. Further improvement requires finer-grained signals beyond episode-level success or failure to identify promising action segments within the accumulated experience. We use rewards that account for execution time and human takeovers to learn a value model and estimate segment advantages. These estimates guide experience selection and sampling for policy fine-tuning in each round.

III-C1 Reward Design with Time and Intervention Costs

To account for both execution time and reliance on human corrections, we construct rewards from per-frame costs and takeover events. Using the label m_{t} introduced in the previous subsection, we identify a takeover as

b_{t+1}=\mathbf{1}[m_{t}=0,\ m_{t+1}=1].

For a successful trajectory, let T denote the index of the terminal observation and set b_{T}=0. The frame-level reward is

r_{t}=\begin{cases}0,&t=T,\\
-1-C_{\mathrm{int}}b_{t+1},&t<T,\end{cases}

where C_{\mathrm{int}}\geq 0 is the additional cost of a human takeover.

A takeover typically indicates that the policy’s behavior requires correction. We therefore treat it as negative feedback and assign the penalty to the policy transition immediately preceding the takeover, rather than to the subsequent human corrections [22]. The unit cost at each nonterminal frame encourages faster task completion. Together, these terms favor trajectories with lower execution time and less dependence on human assistance.

III-C2 Value Function Learning from Trajectory Returns

For each task, we train a vision-only value model with parameters \phi independently of the policy. Given camera images I_{t}, it predicts a categorical distribution p_{\phi}(c\mid I_{t}) over 201 bins whose centers v_{c} are evenly spaced in [-1,0].

The return of frame t is defined as G_{t}=\sum_{u=t}^{T}r_{u}. We normalize it by a fixed, task-specific scale Z_{\ell}>0, giving a training target G_{t}/Z_{\ell}\in[-1,0]. Returns closer to zero indicate a lower combined cost of remaining execution time and human takeovers.

Each normalized return is encoded as a two-hot target q_{t}(c), with weights determined by linear interpolation between adjacent bin centers. We train the model on sampled trajectory frames using the cross-entropy loss

\mathcal{L}_{V}(\phi)=-\mathbb{E}_{\xi,t}\left[\sum_{c=1}^{201}q_{t}(c)\log p_{\phi}(c\mid I_{t})\right].

The scalar value estimate is the probability-weighted mean of the bin centers:

V_{\phi}(I_{t})=\sum_{c=1}^{201}p_{\phi}(c\mid I_{t})v_{c}.

III-C3 Advantage-Guided Policy Fine-Tuning

Using the trained value model, we estimate the advantage of an action sequence as

\widehat{A}_{t}=\sum_{j=0}^{n_{t}-1}\tilde{r}_{t+j}+V_{\phi}(o_{t+n_{t}},\ell)-V_{\phi}(o_{t},\ell).

Here, \tilde{r}_{t}=r_{t}/Z_{\ell} is the normalized reward, and n_{t}=\min(n,T-t) limits the lookahead window of at most n\leq H frames to the remaining trajectory. The terminal value is fixed to V_{\phi}(o_{T},\ell)=0.

In each training round, we use estimated advantages to select promising segments from D_{demo} and available D_{auto}. These segments are mixed with human corrections to fine-tune the policy with the same flow-matching loss. Selection and sampling details are given in Section V-E.

IV RW-RL-Dataset

RW-RL-Dataset organizes demonstrations, human interventions, and autonomous executions to support research on iterative robot policy improvement. Figure 2 summarizes four key dimensions of RW-RL-Dataset: robot embodiment diversity, long-horizon real-world executions, frame-aligned semantic annotations, and multi-source collection regimes for iterative policy learning.

IV-A Embodiments, Modalities, and Collection Regimes

The dataset contains more than 1,000 hours of real-world interaction collected on four robot embodiments: AgiBot G2, Galaxea R1Lite, AgileX PiperX, and SO-101. These platforms span industrial mobile manipulation, modular dual-arm systems, and low-cost tabletop learning.

G2 provides eight RGB video streams (top, top-back, paired fisheye and stereo views, and two wrist views), together with joint states, end-effector poses and wrenches, gripper/head/waist/base states, and actions. R1Lite provides four RGB streams (top, top-right, and two wrist views), together with arm joint positions and velocities, gripper/torso/chassis states, chassis IMU readings, end-effector poses, and actions. The PiperX and SO-101 platforms are dual-arm systems. Both are fitted with an external RGB camera and a pair of wrist cameras. PiperX uses 14-dimensional state and action vectors, while SO-101 employs 12-dimensional counterparts.

Beyond embodiment diversity, RW-RL-Dataset also covers a broad range of real-world task settings. The data span nine scene domains: store shelves, industrial production, commercial kitchens, home spaces, care environments, laboratories, warehouses, hotels, and education or training settings. Across these domains, the dataset instantiates more than 30 task templates, including object sorting, shelf replenishment, scanning and bagging, precision insertion, garment folding, and box stacking.

RW-RL-Dataset further distinguishes episodes by how the actions are generated during collection. Demonstrations record human execution; autonomous episodes record policy execution; and intervention episodes combine both, with frame-level controller labels identifying human correction segments.

IV-B Data Organization and Hierarchical Semantic Annotations

Each episode in the dataset links visual observations, robot states, and executed actions through episode identifiers and timestamps. These records are accompanied by semantic annotations at three levels of granularity: dataset, episode, and frame (Fig. 3). Dataset-level fields specify the task, embodiment, difficulty, objects, and coordination or precision requirements. Episode-level fields describe instructions, scenes, outcomes, execution quality, and mistakes. They also include action-source metadata, which categorizes episodes into three types: teleoperated demonstrations, autonomous policy rollouts, and human-intervention episodes. Frame-level annotations consist of subtask intervals, along with frame-wise controller labels that identify human and policy actions. A subtask interval records its start and end frames, local instruction, manipulated objects, and active arm, and marks operation attributes such as bimanual coordination. All these annotations are synchronized under a unified timestamp system for straightforward cross-reference.

V Experiments

We first assess whether intervention data are more effective than additional demonstrations under the same data budget (Table I). Beyond this data-efficiency comparison, task-specific metrics are used to evaluate policy improvements across successive training stages (Tables II). Representative sequences from the dataset illustrate the progression of each task in Figure 4. We also describe our value-model choice and advantage-guided training-data mixture.

V-A Experimental Setup and Task Definitions

We conduct experiments on two robot platforms: PiperX and SO-101. Policies are initialized from \pi_{0.5} and deployed with real-time chunking (RTC) [23].

Each trial has a five-minute time limit. A trial is considered a failure if the task is not completed within this limit. The setups for the two manipulation tasks are described below.

PiperX insertion. Five screw–sleeve pairs are placed on the tabletop. The robot selects, grasps, aligns, and inserts each screw. Either arm can insert the screw into the sleeve held by the other. Each unit U_{j} corresponds to inserting one screw into a sleeve.

SO-101 folding. Five T-shirts are loosely piled on the desk. The robot retrieves and folds them sequentially. Units U_{1}–U_{5} denote retrieval, spreading, folding in half, folding the sleeves, and folding the hem toward the collar. If the garment enters an erroneous state, an operator manually restores it to the state before the failed stage without fully smoothing it out.

V-B Evaluation Metrics

For PiperX insertion and SO-101 folding, we evaluate the trained models using two metrics. ASR is the total number of successful attempts divided by the total number of attempts, aggregated across U_{1}–U_{5} and all trials, reported in %. Throughput (TP) is the number of successful tasks per hour. Each U_{j} cell represents one individual stage: either a PiperX insertion unit or one SO-101 folding stage. It reports the number of successful completions for that stage and the number of attempts made for that stage. An attempt count of 0 indicates that the trial timed out before reaching this stage. N is the success target per unit or stage, at which evaluation stops: N=10 for PiperX and N=20 for SO-101.

V-C Data Efficiency of Human Interventions

To evaluate whether targeted human corrections yield greater performance gains than equivalently scaled teleoperated demonstrations under a fixed data-collection budget, we compare two data collection regimes with the same time budget: one collects demonstrations only, while the other reserves part of the budget (an hour) for human corrections. If interventions are more efficient, the same budget should translate into higher policy performance than additional demonstrations alone. We benchmark a demonstrations-only baseline (SFT) against SFT + Int., which substitutes one hour of teleoperated demonstrations with human-intervention data. The two regimes are evaluated as paired comparisons under identical total budgets, scaled from 2 to 16 hours and trained from the same pretrained checkpoint. With a four-hour budget, adding one hour of human-intervention data raises ASR from 27.03% to 51.52%, and TP from 12.00 to 20.40, compared with the baseline (Table I). The eight-hour mixture achieves an ASR of 78.00%, comparable to 16-hour SFT (78.26%), while reaching a similar throughput of 53.04 versus 53.27 completed stages per hour.

These gains can be attributed to a fundamental difference between demonstrations and interventions in how they cover the state space. Successful teleoperation may rarely cover states induced by policy errors, whereas interventions supply corrective actions in these states. Their value lies in relevance to the current policy’s failures. Since the policy executes autonomously for most of the episode, human supervision intervenes only at a few critical moments, supplying precisely the data the model most needs and thereby increasing the value of the collected training data.

V-D Intervention and Real-Robot RL

Beyond the data-efficiency comparison, we evaluate how human interventions and reinforcement learning jointly support continued policy improvement. We conduct ablation experiments on PiperX to assess the contribution of each training stage (Table II). All variants start from the same policy trained on demonstrations for 2.5 epochs (SFT). To evaluate human interventions alone, we fine-tune the policy on demonstrations mixed with one hour of human interventions, sampling interventions with probability p=0.5 (SFT + Int.). Alternatively, we apply advantage-guided selection to the baseline policy using only demonstration data (SFT + RL): advantages computed from the demonstrations identify high-advantage segments, which are then sampled with elevated probability during fine-tuning. The combined setting (Evo-RL) integrates intervention-based fine-tuning with advantage-guided sampling: it mixes original demonstrations, high-advantage demonstration segments sampled with elevated probability, and human-intervention data for policy fine-tuning. High-advantage segments from successful autonomous rollouts are also included in this round. Finally, we repeat the data collection and training cycle to evaluate the benefits of a second iteration (Evo-RL R2).

PiperX insertion. The ablations in Table II examine how human corrections and reinforcement learning contribute to policy improvement. Both SFT + Int. and SFT + RL improve over SFT, with human corrections providing the larger gain. Combining the two in Evo-RL yields the highest ASR and TP among the first-round variants, raising ASR from 78.72% for SFT + Int. to 88.24% and improving TP by 26.3%. These results support complementary roles for corrective experience and advantage-guided refinement: corrections supply behavior for difficulties encountered during deployment, while advantage-guided sampling directs further training toward promising behavior in the available data.

Beyond these ablations, first-round Evo-RL outperforms RECAP on PiperX, achieving 88.24% versus 57.78% ASR and 60.61 versus 35.54 TP. Policy fine-tuning takes approximately 4 h 52 min for Evo-RL (1,511 steps), compared with an estimated 64 h for RECAP (102,111 steps, 2.5 epochs), both using eight A800 80 GB GPUs.

Statistics show that the human-control time fraction decreases from 15.30% for SFT to 9.56% for Evo-RL. Together with the higher ASR and TP, this indicates that improved task performance coincides with a smaller share of recorded execution under direct human control. A second collection-and-training round further raises ASR from 88.24% to 92.16% and TP from 60.61 to 67.14, with U_{5} completions increasing from five to eight. Thus, another round can improve execution beyond the first update.

SO-101 folding. The complete Evo-RL method also improves over SFT on folding, raising ASR from 55.56% to 76.67% and TP from 75.78 to 90.00. At the spreading stage (U_{2}), successful completions increase from 17 to 19 while attempts decrease from 76 to 36. This substantial reduction in repeated attempts helps explain the higher aggregate ASR and accompanies improved throughput, despite a more modest increase in successful stage completions.

V-E Value Learning and Advantage-Guided Sampling

Our value models are task-specific and vision-only. Each model consists of a SigLIP visual encoder followed by an MLP output head, and is trained by randomly sampling frames from demonstrations \mathcal{D}_{\mathrm{demo}}, human-intervention episodes \mathcal{D}_{\mathrm{interv}}, and autonomous rollouts \mathcal{D}_{\mathrm{auto}}. Value training takes approximately 2 h 19 min on three A100 GPUs for Evo-RL, compared with 3 h 46 min on eight A800 80 GB GPUs for RECAP.

The same value model and advantage computation apply to demonstrations and successful autonomous rollouts. We jointly select nonterminal starts with positive advantages in the top 10% across both sources, merging overlapping or adjacent retained chunks. Selected autonomous segments join the original demonstrations and human corrections for fine-tuning; selected demonstration segments receive increased sampling weight.

Figure 5 plots the value and advantage estimates over a screw-insertion episode. Gold dots mark the selected starts, and the overlaid images show the corresponding observations. Which segments are retained for training is determined by the advantage at each start, not by high state value alone. The selected states consistently correspond to critical transition phases of the task: grasping the target object, manipulating it (inserting the screw into the sleeve), and placing it at the target location. These are precisely the points where small errors most affect downstream execution, and where the policy most needs fine-grained supervision. Notably, states near task completion often exhibit high value yet low advantage, confirming that value alone is insufficient for selecting informative segments.

VI Conclusion

We presented Evo-RL, which combines human corrections with advantage-guided experience selection in an iterative policy improvement loop. Corrections address deployment difficulties, while estimated advantages guide subsequent experience selection and sampling. Experiments support their complementary roles: interventions outperform additional demonstrations under matched human-control budgets, and Evo-RL improves attempt success rate and throughput on high-precision and deformable-object manipulation tasks. On PiperX, it outperforms RECAP and either update stage alone, with further gains in a second iteration. To support research on this learning process, RW-RL-Dataset brings together demonstrations, interventions, and autonomous experience, including the training data used in our evaluation.

References

1. [1] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V. Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y. Lu, S. Levine, L. Lee, T. E. Lee, I. Leal, Y. Kuang, D. Kalashnikov, R. Julian, N. J. Joshi, A. Irpan, B. Ichter, J. Hsu, A. Herzog, K. Hausman, K. Gopalakrishnan, C. Fu, P. Florence, C. Finn, K. A. Dubey, D. Driess, T. Ding, K. M. Choromanski, X. Chen, Y. Chebotar, J. Carbajal, N. Brown, A. Brohan, M. G. Arenas, and K. Han (2023) RT-2: vision-language-action models transfer web knowledge to robotic control. In CoRL,
2. [2] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn (2024) OpenVLA: an open-source vision-language-action model. In CoRL,
3. [3] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky (2025) \pi_{0}: A vision-language-action flow model for general robot control. In RSS,
4. [4] W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, F. Wang, Q. Zhu, H. Sun, Y. Wang, S. Ma, Y. Ren, K. Zhang, H. Yu, J. Zhao, S. Zhou, Z. Qiu, H. Xiong, Z. Wang, Z. Wang, R. Cheng, Y. Li, Y. Huang, X. Zhu, Y. Shen, and K. Zheng (2026) A pragmatic VLA foundation model. arXiv preprint arXiv:2601.18692.
5. [5] T. Lin, Y. Zhong, Y. Du, J. Zhang, J. Liu, Y. Chen, E. Gu, Z. Liu, H. Cai, Y. Zou, L. Zou, Z. Zhou, G. Li, and B. Zhao (2026) Evo-1: lightweight vision-language-action model with preserved semantic alignment. In CVPR,
6. [6] S. Ross, G. Gordon, and D. Bagnell (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS,
7. [7] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer (2019) HG-DAgger: interactive imitation learning with human experts. In ICRA,
8. [8] A. Mandlekar, D. Xu, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese (2020) Human-in-the-loop imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733.
9. [9] Physical Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Hancock, K. Hausman, G. Hussein, B. Ichter, S. Jakubczak, R. Jen, T. Jones, B. Katz, L. Ke, C. Kuchi, M. Lamb, D. LeBlanc, S. Levine, A. Li-Bell, Y. Lu, V. Mano, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, C. Sharma, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, A. Swerdlow, J. Tanner, M. Torne, Q. Vuong, A. Walling, H. Wang, B. Williams, S. Yoo, L. Yu, U. Zhilinsky, and Z. Zhou (2025) \pi^{*}_{0.6}: a VLA that learns from experience. arXiv preprint arXiv:2511.14759.
10. [10] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, A. Lee, K. Fang, C. Finn, and S. Levine (2023) BridgeData V2: a dataset for robot learning at scale. In CoRL,
11. [11] A. Khazatsky et al. (2024) DROID: a large-scale in-the-wild robot manipulation dataset. In RSS,
12. [12] H. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu (2024) RH20T: a comprehensive robotic dataset for learning diverse skills in one-shot. In ICRA,
13. [13] K. Wu, C. Hou, J. Liu, Z. Che, X. Ju, Z. Yang, M. Li, Y. Zhao, Z. Xu, G. Yang, et al. (2025) RoboMIND: benchmark on multi-embodiment intelligence normative data for robot manipulation. In RSS,
14. [14] Open X-Embodiment Collaboration et al. (2024) Open X-Embodiment: robotic learning datasets and RT-X models. In ICRA,
15. [15] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg (2021) ThriftyDAgger: budget-aware novelty and risk gating for interactive imitation learning. In CoRL,
16. [16] H. Zhang, Z. Zhuang, H. Zhao, P. Ding, H. Lu, and D. Wang (2025) ReinboT: amplifying robot visual-language manipulation with reinforcement learning. In ICML,
17. [17] Z. Zhang, K. Zheng, Z. Chen, J. Jang, Y. Li, S. Han, C. Wang, M. Ding, D. Fox, and H. Yao (2025) GRAPE: generalizing robot policy via preference alignment. In ICRA FMNS Workshop,
18. [18] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. In NeurIPS,
19. [19] C. Xu, J. T. Springenberg, M. Equi, A. Amin, A. Esmail, S. Levine, and L. Ke (2026) RL Token: bootstrapping online RL with Vision-Language-Action models. arXiv preprint arXiv:2604.23073.
20. [20] H. Chen, J. Liu, Z. Yan, N. Han, R. Zhang, C. Gu, J. Gao, Z. Guo, S. Qian, Y. Wang, P. Jia, S. Zhang, and P. Heng (2026) LaST-R1: reinforcing robotic manipulation via adaptive physical latent reasoning. arXiv preprint arXiv:2604.28192.
21. [21] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky (2025) \pi_{0.5}: A vision-language-action model with open-world generalization. In CoRL,
22. [22] J. Luo, P. Dong, Y. Zhai, Y. Ma, and S. Levine (2024) RLIF: interactive imitation learning as reinforcement learning. In ICLR, External Links: Link
23. [23] K. Black, M. Galliker, and S. Levine (2025) Real-time execution of action chunking flow policies. In NeurIPS,