AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation

Abstract

Vision–language–action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.

2

1 Introduction

Recent advances in Vision-Language Models (VLMs) (Beyer et al., 2024; Deitke et al., 2024; Yang et al., 2025) have established Vision–Language–Action (VLA) (Intelligence et al., 2025b; Kim et al., 2025; Liu et al., 2025b) models as a prominent paradigm for robotic manipulation. Although imitation learning (IL) from large-scale robot demonstrations enables VLA models to acquire diverse manipulation skills, its performance is inherently constrained by the coverage of the demonstration data. Reinforcement learning (RL) overcomes this limitation by directly optimizing policies with task feedback obtained through environment interaction, and recent work (Li et al., 2026; Xu et al., 2026) has successfully applied RL to VLA models. However, existing VLA-RL methods still face substantial challenges in long-horizon manipulation tasks.

First, most methods rely on terminal-only success rewards. In an episode that completes the first k subtasks but fails at subtask k+1, the absence of a terminal success reward causes generalized advantage estimation (GAE) to propagate failure-related temporal-difference (TD) residuals backward to t_{0}, suppressing the advantages of correctly executed actions. We refer to this as credit contamination, where useful behavior in failed trajectories is systematically devalued, with the effect worsening as the horizon grows. Prior work (Chen et al., 2025; Yang et al., 2026a; Tan et al., 2025; Zhai et al., 2025) mitigates reward sparsity through stage-level preference optimization or subtask-level intermediate rewards, but these approaches alter reward density without resolving credit contamination. Second, subtask mastery is uneven, with failures increasingly concentrated in a few bottleneck subtasks during later training. Uniform task sampling and fixed exploration then waste interaction budget on saturated subtasks while providing insufficient exploration for the bottlenecks.

To address these challenges, we propose AtomicRL, as illustrated in Fig. 1, an RL post-training framework built around atomic task structure. Our key insight is that semantic subtasks provide a unified basis for both the propagation of learning signals and the allocation of exploration resources. AtomicRL couples Semantic-Aware Policy Optimization (SAPO) with Bottleneck-Adaptive Target Exploration (BATE), supported by a lightweight Progress–Completion Critic (PCC) that estimates atomic subtask progress and completion boundaries. PCC also provides progress estimates for constructing dense progress-based shaping rewards (PBSR). SAPO modulates TD-residual propagation in advantage estimation with a soft gate conditioned on semantic boundaries. Within each atomic subtask, it preserves standard GAE trace decay; across subtask boundaries, it maintains propagation between successful subtasks while attenuating residuals from failed subtasks. By leaving one-step TD residuals and value bootstrapping unchanged, SAPO reduces failure interference while preserving intra-subtask credit assignment and successful subtask dependencies. BATE tracks the mastery of each subtask via a moving average of its completion rate, jointly adjusts the sampling weight of full tasks and the strength of subtask-level entropy regularization, directing interaction and exploration resources toward bottleneck subtasks. Finally, to support physical deployment, we develop an automated Real2Sim2Real pipeline. Given a single real-world multimodal demonstration and the required auxiliary information, the pipeline reconstructs an interactive simulation environment. It connects simulation-based RL with real-robot deployment, shifting costly physical trial-and-error to parallel simulation with reproducible resets.

We conduct extensive experiments in both simulation and the real world. On LIBERO-LONG (Liu et al., 2023) AtomicRL achieves success rates of 99.0% with \pi_{0.5} (Intelligence et al., 2025b) and 98.8% with OpenVLA-OFT (Kim et al., 2025), improving over PPO by 3.6% and 4.0%. On CALVIN ABC-D (Mees et al., 2022), the average completed sequence length improves from 3.56 to 4.56, while the success rate of completing full five-subtask chain improves by 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further yields an average improvement of 21.3% on real-robot long-horizon tasks. These results demonstrate the effectiveness of AtomicRL for long-horizon VLA reinforcement learning. Overall, our contributions are as follows:

• We introduce AtomicRL, an RL post-training framework built around atomic task structure for long-horizon robotic manipulation.
• We propose SAPO to mitigate cross-subtask credit interference and BATE to adaptively direct task sampling and exploration toward bottleneck subtasks.
• We develop a Real2Sim2Real pipeline that streamlines simulation construction and RL training for efficient real-robot deployment.
• We validate the effectiveness of AtomicRL through extensive experiments conducted in both simulated environments and real-world robots.

2 Related Work

2.1 Reinforcement Learning for VLA Models

Reinforcement learning (RL) optimizes policies through environment interaction and task-level feedback. Recent work has explored both offline and online RL paradigms for VLA policies. Offline RL approaches leverage task feedback to enable continued policy improvement (Huang et al., 2025; Zhang et al., 2025b; Zhang et al., 2025a). \pi_{0.6}^{*} (Intelligence et al., 2025a) integrates expert demonstrations, autonomous interaction data, and human corrections through advantage-conditioned learning. For scalable online training, VLA-RL (Lu et al., 2025) and SimpleVLA-RL (Li et al., 2026) explore frameworks for continually improving VLA policies through environment interaction, while RL4VLA (Liu et al., 2026) systematically investigates how different RL algorithms affect VLA generalization. Furthermore, Flow Q-Learning (Park et al., 2025), Flow-GRPO (Liu et al., 2025a) and \pi_{RL} (Chen et al., 2026a) investigate online RL fine-tuning for flow-matching policies, addressing challenges in action likelihood computation and exploration.

RL for long-horizon manipulation introduces additional challenges due to sparse and delayed task feedback (Chen et al., 2026b; Li et al., 2025c; Dalal et al., 2024). STARE-VLA (Xu et al., 2025) exploits the semantic stage structure of trajectories to perform stage-level preference optimization and reward shaping, while IG-RFT (Su et al., 2026) combines trajectory-level and subtask-level rewards to provide intermediate guidance. However, dense reward shaping provides useful process guidance over long-horizon trajectories, but cannot eliminate the spurious interference caused by later-stage failures. In contrast, AtomicRL gates the propagation of advantage signals at the semantic boundaries of atomic tasks to prevent cross-task interference from unreliable signals, while allocating the exploration budget to bottleneck atomic tasks to improve the efficiency of long-horizon RL.

2.2 Online Reinforcement Learning in Real World

Online reinforcement learning for real-world manipulation is constrained by interaction costs and safety risks. To reduce reliance on real-robot trial and error, some research focuses on learning action-conditioned world models and optimizing policies through imagined interactions (Team et al., 2025; Li et al., 2025a; Zhang et al., 2026c; Jiang et al., 2026a). WoVR (Jiang et al., 2026b) addresses prediction errors and their accumulation to improve the reliability of model-generated rollouts for VLA reinforcement fine-tuning , while PlayWorld (Yin et al., 2026) and RISE (Yang et al., 2026b) learn world models from autonomous robot interactions and transfers policies trained through model-based reinforcement learning to real hardware. Although world models can effectively capture short-term dynamics, long-horizon predictions still exhibit visual distortions and action–state inconsistencies. Other studies adopt an explicit Real2Sim2Real method, combining real-scene reconstruction with physics simulation to provide interactive environments for policy optimization (Li et al., 2025b; Wang et al., 2026). GSWorld (Jiang et al., 2025) integrates Gaussian-on-Mesh assets with physics engines to support visually faithful policy training, while RialTo (Torne et al., 2024) constructs digital twins of real-world environments and uses reinforcement learning in simulation to improve policy robustness. However, constructing environments requires coordinating scene reconstruction, physical asset preparation, and control alignment, making it complex and costly to reproduce. AtomicRL introduces an automated Real2Sim2Real pipeline that uses only a single demonstration to reconstruct a real-world manipulation scene as an interactive simulation environment for long-horizon online RL. The trained policies are then transferred to physical robots for evaluation.

3 Method

3.1 Overview

As illustrated in Figure 2, AtomicRL integrates semantic-aware credit assignment and bottleneck-adaptive exploration for long-horizon robotic manipulation. Specifically, SAPO modulates credit propagation across atomic subtasks (Section 3.2), while bottleneck-adaptive targeted exploration adjusts full-task trajectory sampling and local exploration incentives (Section 3.3). Both components are supported by a Progress–Completion Critic (PCC), which estimates subtask progress and identifies completion boundaries (Section 3.4). For real-world tasks, we further introduce a Real2Sim2Real construction workflow to support simulation training and subsequent transfer evaluation (Section 3.5).

3.1.1 Problem Formulation

We consider a language-conditioned partially observable Markov decision process for long-horizon robotic manipulation. At each decision step t, the robot receives an observation o_{t}, which may include third-person images, wrist-camera images, and proprioceptive states, and outputs an action chunk a_{t}=(a_{t,1},\ldots,a_{t,H_{a}}), where H_{a} denotes the chunk length. A long-horizon instruction is decomposed into an ordered sequence of K atomic subtasks, G=(g_{1},\ldots,g_{K}). We denote the active subtask index by z_{t}\in{1,\ldots,K}. For nonterminal transitions, the semantic-boundary indicator is defined as b_{t}=\mathbb{I}[z_{t+1}\neq z_{t}], indicating whether execution of a_{t} leads to a subtask transition. Our objective is to maximize full-task success under a fixed environment-interaction budget by improving credit assignment and directing exploration toward bottleneck subtasks.

3.2 Semantic-Aware Policy Optimization

SAPO addresses temporal credit assignment in long-horizon tasks. It leverages semantic boundaries to selectively attenuate the contribution of TD residuals from failed subtasks to the advantage estimates of preceding actions, thereby regulating cross-subtask credit propagation.

Let r_{t} denote the chunk-level reward and d_{t} indicate environment termination after executing a_{t}. The one-step TD residual is

\delta_{t}=r_{t}+\gamma(1-d_{t})V_{\phi}(s_{t+1})-V_{\phi}(s_{t}),

where \gamma is the discount factor and s_{t} denotes the observation- and language-conditioned input representation available to the value function, rather than a privileged simulator state. For an episode containing T chunk-level transitions indexed by t=1,\ldots,T, the standard GAE estimate used by PPO is

\hat{A}^{\mathrm{PPO}}_{t}=\sum_{\ell=0}^{T-t}(\gamma\lambda)^{\ell}\delta_{t+\ell},

where \lambda\in[0,1] controls trace decay. This expression is evaluated within a single episode and does not span environment resets.

SAPO distinguishes semantic segmentation from environment termination and uses atomic subtask outcomes to regulate cross-boundary credit propagation. Specifically, PCC (Section 3.4) identifies semantic boundaries b_{t} between atomic subtasks. Keeping the one-step TD residual unchanged, SAPO applies an outcome-aware soft gate to the recursive GAE trace:

\hat{A}^{\mathrm{SAPO}}_{t}=\sum_{\ell=0}^{T-t}(\gamma\lambda)^{\ell}\left[\prod_{j=0}^{\ell-1}\left(1-(1-\kappa_{t+j})b_{t+j}\right)\right]\delta_{t+\ell},\qquad\kappa_{t}\in[0,1].

Within a subtask, b_{t}=0, and the trace follows the standard \gamma\lambda decay. At a semantic boundary, \kappa_{t}=1 when the subsequent subtask succeeds, preserving ordinary propagation; when it fails, we set 0<\kappa_{t}<1 to attenuate propagation across the boundary. Thus, when subtasks 1{:}k succeed but subtask k+1 fails, residuals from the failed segment propagate normally within that segment and are additionally attenuated when entering the completed prefix. Residuals within the completed prefix retain their standard trace weights. This design preserves within-task credit propagation and connections between successfully completed subtasks while reducing the direct negative influence of downstream failures on earlier successful behavior. We use the resulting advantages for updates and construct the value targets as V_{\phi}(s_{t})+\hat{A}^{\mathrm{SAPO}}_{t} before advantage normalization. By making credit propagation sensitive to semantic boundaries and local outcomes, SAPO improves credit assignment and interaction-data efficiency in long-horizon tasks.

3.3 Bottleneck-Adaptive Targeted Exploration

During the early stages of RL, broad exploration provides informative samples for most atomic subtasks. As training approaches saturation, failures may concentrate in a few bottleneck subtasks, making persistent aggressive exploration less effective and potentially disruptive to acquired skills. BATE adapts task sampling and local exploration incentives according to atomic subtask mastery.

We estimate the mastery level of each atomic-task category using an exponential moving average of its completion rate:

\mu_{k}^{(n)}=\beta\mu_{k}^{(n-1)}+(1-\beta)\hat{s}_{k}^{(n)},\qquad\beta\in[0,1),

where n indexes statistics updates and \hat{s}_{k}^{(n)} is the fraction of successful outcomes among subtask attempts whose outcomes are resolved during the update.

The competence gap, 1-\mu_{k}, guides both full-task sampling and local exploration.

w_{i}=1+\rho\left[\max_{k\in\mathcal{K}_{i}}(1-\mu_{k})\right]^{\eta},\qquad P_{\mathrm{task}}(i)=\frac{w_{i}}{\sum_{j}w_{j}},

where \mathcal{K}_{i} contains the subtasks of full task i, \eta>0 controls sensitivity to the competence gap, and \rho\geq 0 limits sampling redistribution. The baseline weight of 1 preserves coverage of mastered tasks to monitor forgetting.

Within each trajectory, mastery-dependent entropy regularization adjusts local exploration:

\mathcal{L}_{\mathrm{explore}}=-\alpha_{H}\mathbb{E}_{t}\left[f_{\mathrm{exp}}\!\left(1-\mu_{z_{t}}^{(n)}\right)\mathcal{H}_{\theta,t}\right],

where \alpha_{H} is the base entropy coefficient, \mathcal{H}_{\theta,t} denotes policy entropy, and f_{\mathrm{exp}} is a bounded exploration weight for the active subtask z_{t}. The exploration strength is thus increased for poorly mastered subtasks and reduced as subtask mastery improves.

3.4 Progress–Completion Critic

To identify atomic-task boundaries b_{t}, we introduce a lightweight Progress–Completion Critic (PCC), denoted by C_{\omega}. PCC estimates within-task progress and completion from the recent L observation–action pairs, providing semantic signals for SAPO and BATE and support the construction of progress-based shaping rewards (PBSR).

Conditioned on the current atomic task, PCC evaluates its status after executing a_{t} and outputs B-1 ordinal progress logits u_{t,j}^{\mathrm{prog}} and a completion logit u_{t}^{\mathrm{comp}}, where B is the number of progress levels. We decode progress as

\hat{p}_{t}=\frac{1}{B-1}\sum_{j=1}^{B-1}\mathbf{1}\!\left[\sigma\!\left(u_{t,j}^{\mathrm{prog}}\right)>\tau_{\mathrm{ord}}\right],\qquad\hat{q}_{t}=\sigma\!\left(u_{t}^{\mathrm{comp}}\right),

where \sigma denotes the sigmoid function and \tau_{\mathrm{ord}}=0.5 is the ordinal decoding threshold. An atomic task is considered complete when \hat{q}_{t}\geq\tau_{s} and \hat{p}_{t}\geq\tau_{p} for N consecutive decision steps. Upon confirming a non-terminal boundary, we advance the atomic-task index as z_{t+1}=z_{t}+1 and reset the tracked within-task progress while retaining the physical observation context. PCC is trained offline on demonstration data using a cumulative ordinal loss together with a class-balanced completion loss:

\mathcal{L}_{C}=\lambda_{p}\mathcal{L}_{\mathrm{ord}}+\lambda_{c}\mathcal{L}_{\mathrm{comp}},

where \mathcal{L}_{\mathrm{ord}} applies binary cross-entropy to cumulative progress targets, \mathcal{L}_{\mathrm{comp}} is a class-balanced binary cross-entropy loss for completion prediction. The coefficients \lambda_{p} and \lambda_{c} control the relative contributions of the two objectives. The progress estimates can optionally be used to construct progress-based shaping rewards (PBSR), yielding the training reward

r_{t}=r_{t}^{\mathrm{env}}+\alpha_{n}r_{t}^{\mathrm{PBSR}},

where r_{t}^{\mathrm{env}} is the sparse task reward, r_{t}^{\mathrm{PBSR}} is the progress-based shaping reward.

3.5 Real2Sim2Real Pipeline for Reinforcement Learning

We develop an automated Real2Sim2Real reinforcement learning pipeline that unifies interactive environment construction, policy optimization in simulation, and deployment and evaluation on physical robots. Given a single multimodal real-world demonstration \tau_{\mathrm{real}}, metadata m specifying the robot model and supplementary object observations \mathcal{D}_{\mathrm{aux}} where needed, the pipeline constructs a simulation environment aligned with the physical system, a shared observation–action interface specification, and a set of reproducible randomized initial layouts:

(\mathcal{E}_{\mathrm{sim}},\mathcal{C},\mathcal{S}_{0})=\mathcal{B}(\tau_{\mathrm{real}},m,\mathcal{D}_{\mathrm{aux}}),

where \mathcal{E}_{\mathrm{sim}} is the reconstructed interactive simulation environment, \mathcal{C} specifies the shared observation–action interface, \mathcal{S}_{0} is a set of reproducible randomized initial layouts, and \mathcal{B} denotes the pipeline workflow. The generated environment is then integrated into the RLinf (Yu et al., 2025) distributed training framework, where SAPO and targeted exploration optimize the initial policy to obtain \theta_{\mathrm{RL}}=\operatorname{Optimize}(\theta_{0};\mathcal{E}_{\mathrm{sim}},\mathcal{C},\mathcal{S}_{0}). Following pre-deployment validation, the optimized policy is deployed on the physical robot through the same interface specification \mathcal{C} for task evaluation.

4 Experiment

4.1 Experiments Setup

Simulation Benchmarks. We evaluate AtomicRL on two robotic manipulation benchmarks: LIBERO Liu et al. (2023) and CALVIN Mees et al. (2022). We use the LIBERO-LONG suite(10 long-horizon tasks), and for CALVIN, we follow the ABC-D protocol, training on environments A, B, and C and evaluating on the held-out environment D. All experiments are implemented within the RLinf training framework Yu et al. (2025). On LIBERO-LONG, we compare AtomicRL against PPO with two policy backbones, \pi_{0.5} Intelligence et al. (2025b) and OpenVLA-OFT Kim et al. (2025); on CALVIN, we conduct additional evaluations with \pi_{0.5}. Each training run consists of 200 optimization steps.

Real World Robot. We conduct real-world experiments using an AgiBot G1 robot across three long-horizon tabletop manipulation tasks, each consisting of 4 to 6 atomic subtasks. For each task, we collect 100 expert demonstrations for SFT. The three tasks are:

• Bread Placement(BP): Place all pieces of bread on the table onto the white plate;
• Waste Disposal(WD): Place the tissue and the car into the trash bin;
• Tabletop Clean(TC): Place the remote control into the storage box and the snacks into the box.

Evaluation metrics. For LIBERO, we report the mean success rate across all tasks, with 50 trials per task. For CALVIN, we report the success rates for completing one to five consecutive tasks and the average completed sequence length over 1,000 evaluation trials. These metrics measure cumulative sequence completion rather than independent subtask performance. For real-robot evaluation, each task is tested over 25 trials per method, and we report the mean success rate across the three tasks.

4.2 Results on Simulation

Results on LIBERO. As shown in Tab. 1, AtomicRL achieves success rates of 98.8\% with OpenVLA-OFT and 99.0\% with \pi_{0.5}, exceeding their corresponding PPO baselines by 4.0\% and 3.6\%. The consistent gains across both backbones support the applicability of AtomicRL to different VLA architectures. Both policies reach near-perfect success rates after 200 optimization iterations, demonstrating further improvements even when PPO already performs well. This efficiency can be attributed to AtomicRL’s effective use of training data: SAPO preserves useful learning signals from completed subtasks in failed trajectories, while BATE concentrates exploration on bottleneck atomic subtasks. Together, these mechanisms enable faster convergence and higher final performance.

Results on CALVIN. As shown in Tab. 2, AtomicRL improves the average completed sequence length from 3.56 with SFT to 4.56, compared with 4.36 for PPO. The success rate for completing all five tasks increases from 70.6\% with PPO to 80.1\%, a gain of 9.5\%. The larger gains on longer sequences indicate more reliable execution across the full task chain, demonstrating the effectiveness of AtomicRL for long-horizon reinforcement learning.

4.3 Results on Real-world Robot

We conduct online reinforcement learning in the constructed real-to-sim environments and evaluate both the policy improvement after RL and its sim-to-real transfer performance.

Performance on real-world robot. As shown in Tab. 3, AtomicRL improves over SFT by 21.3% on average across three long-horizon tasks and over PPO by 8.0%. As shown in Fig. 4, SAPO preserves useful learning signals from successfully completed subtasks in unsuccessful trajectories, enabling more appropriate credit assignment for suboptimal trajectories. Consequently, AtomicRL improves faster in early training, especially on the longer-horizon Bread Placement task. These results demonstrate the effectiveness of AtomicRL for long-horizon manipulation.

Simulation–reality correspondence. Fig. 3 compares the success rates between simulation and the real robot across nine combinations of three tasks and three methods. The aggregated Pearson correlation coefficient is r=0.909, indicating a strong positive correlation across the evaluated settings.

As shown in Fig. 5, the SFT policy fails after completing only part of the task, whereas the AtomicRL-optimized policy successfully completes the illustrated sequence in both simulation and the real world. The qualitative consistency in observations and execution outcomes between the two environments indicates that the reconstructed environment preserves the key visual and interaction structures relevant to the task, supporting effective policy optimization. Combined with the Real2Sim2Real deployment pipeline, AtomicRL shifts the otherwise costly and operationally complex online RL process from physical robots to simulation environments that support fast and parallel interactions. This reduces real-world interaction and reset costs while providing a practical approach for optimizing and deploying long-horizon manipulation policies.

4.4 Ablation Study

We conduct policy-learning ablations on LIBERO-LONG using a weaker SFT checkpoint trained with fewer demonstrations than those used in the main evaluations. This few-shot initialization allows us to examine the contribution of each component under a less proficient initial policy.

Reward shaping and SAPO. To investigate the effect of reward granularity, we use the PCC module to construct process-level dense rewards and subtask-level rewards, and compare them under both PPO and SAPO. As shown in Tab. 4 (a), denser rewards do not always yield better performance: under SAPO, relatively sparse subtask-level rewards outperform process-level dense rewards. When the policy success rate is low, dense rewards provide stronger exploration guidance and accelerate early learning. As the policy improves, however, their benefits diminish, while persistent auxiliary rewards may dilute the terminal success signal and shift optimization toward local gains. Motivated by this observation, we use linear annealing to gradually reduce the PBSR weight, providing stronger dense-reward guidance early in training and emphasizing the sparse terminal reward later. This balances early exploration with late-stage task alignment.

Component analysis. Table 4 (b) presents the incremental effects of successively introducing SAPO, annealed PBSR, and BATE. Starting from a PPO success rate of 78.2%, adding SAPO increases the success rate to 87.8% (+9.6 pp); introducing annealed PBSR further improves it to 89.6% (+1.8 pp); and finally incorporating BATE raises it to 91.2% (+1.6 pp), yielding a cumulative improvement of 13.0 percentage points over PPO. SAPO provides the largest performance gain, while annealed reward shaping and bottleneck-adaptive exploration further improve the policy. Combined with the results in Table 4 (a), where SAPO consistently improves performance across different reward settings, these findings suggest that its gains do not simply arise from denser reward feedback. Together, these results support the complementarity of the proposed components and their ability to further unlock the potential of RL for learning long-horizon tasks.

PCC prediction quality and computational cost. We train PCC on 150 trajectories containing both successful and failed executions and evaluate it on a test set of 500 rollout trajectories. The reference annotations for training and evaluation are automatically generated by a VLM and subsequently refined through manual correction. As shown in Tab. 5, PCC achieves 95.92% precision and 89.20% recall, with an empirical inference latency of 42.4 ms.

These results show that PCC can estimate subtask progress and completion status with high precision while maintaining low inference latency. As a core component of AtomicRL, PCC converts the semantic task structure in trajectories into progress and boundary information for policy optimization, supporting SAPO in regulating cross-subtask credit propagation and BATE in estimating subtask mastery. Its prediction accuracy and low inference overhead demonstrate that this lightweight design provides a practical perception and evaluation foundation for RL based on atomic task structure.

5 Conclusion

In this work, we introduce AtomicRL, a VLA RL post-training framework for long-horizon robotic manipulation centered on atomic task structure. To address temporal credit interference and imbalanced atomic subtask mastery, AtomicRL uses PCC to estimate task progress and completion boundaries, SAPO to regulate the cross-boundary propagation of TD residuals based on subtask execution outcomes, and BATE to shift task sampling and local exploration toward insufficiently mastered bottleneck subtasks. We further develop a Real2Sim2Real pipeline that connects real-task reconstruction, simulation-based RL, and real-robot deployment, enabling policy improvement in simulation environments with fast parallel interactions and reproducible resets. We validate AtomicRL in both simulation and real-world settings, and the results show that it converges faster to a higher performance level, demonstrating its potential for RL-based long-horizon robotic manipulation.

References

1. Beyer et al. (2024) L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Grycner, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, R. Kabra, M. Bauer, M. Bošnjak, X. Chen, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Papalampidi, O. Henaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai PaliGemma: a versatile 3b vlm for transfer. External Links: 2407.07726, Link
2. Chen et al. (2026a) K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, Q. Zhang, Z. Yu, G. Fan, T. Huang, Y. Wang, and C. Yu \pi_{\texttt{RL}}: Online rl fine-tuning for flow-based vision-language-action models. External Links: 2510.25889, Link
3. Chen et al. (2026b) Q. Chen, J. Yu, M. Schwager, P. Abbeel, Y. Shentu, and P. Wu SARM: stage-aware reward modeling for long horizon robot manipulation. External Links: 2509.25358, Link
4. Chen et al. (2025) Z. Chen, J. Huo, Y. Chen, and Y. Gao RoboHorizon: an llm-assisted multi-view world model for long-horizon robotic manipulation. External Links: 2501.06605, Link
5. Dalal et al. (2024) M. Dalal, T. Chiruvolu, D. Chaplot, and R. Salakhutdinov Plan-seq-learn: language model guided rl for solving long horizon robotics tasks. External Links: 2405.01534, Link
6. Deitke et al. (2024) M. Deitke, C. Clark, S. Lee, R. Tripathi, Y. Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, J. Lu, T. Anderson, E. Bransom, K. Ehsani, H. Ngo, Y. Chen, A. Patel, M. Yatskar, C. Callison-Burch, A. Head, R. Hendrix, F. Bastani, E. VanderBilt, N. Lambert, Y. Chou, A. Chheda, J. Sparks, S. Skjonsberg, M. Schmitz, A. Sarnat, B. Bischoff, P. Walsh, C. Newell, P. Wolters, T. Gupta, K. Zeng, J. Borchardt, D. Groeneveld, C. Nam, S. Lebrecht, C. Wittlif, C. Schoenick, O. Michel, R. Krishna, L. Weihs, N. A. Smith, H. Hajishirzi, R. Girshick, A. Farhadi, and A. Kembhavi Molmo and pixmo: open weights and open data for state-of-the-art vision-language models. External Links: 2409.17146, Link
7. Huang et al. (2025) D. Huang, Z. Fang, T. Zhang, Y. Li, L. Zhao, and C. Xia CO-rft: efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning. External Links: 2508.02219, Link
8. Intelligence et al. (2025a) P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Hancock, K. Hausman, G. Hussein, B. Ichter, S. Jakubczak, R. Jen, T. Jones, B. Katz, L. Ke, C. Kuchi, M. Lamb, D. LeBlanc, S. Levine, A. Li-Bell, Y. Lu, V. Mano, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, C. Sharma, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, A. Swerdlow, J. Tanner, M. Torne, Q. Vuong, A. Walling, H. Wang, B. Williams, S. Yoo, L. Yu, U. Zhilinsky, and Z. Zhou \pi^{*}_{0.6}: A vla that learns from experience. External Links: 2511.14759, Link
9. Intelligence et al. (2025b) P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky \pi_{0.5}: A vision-language-action model with open-world generalization. External Links: 2504.16054, Link
10. Jiang et al. (2025) G. Jiang, H. Chang, R. Qiu, Y. Liang, M. Ji, J. Zhu, Z. Dong, X. Zou, and X. Wang GSWorld: closed-loop photo-realistic simulation suite for robotic manipulation. External Links: 2510.20813, Link
11. Jiang et al. (2026a) Z. Jiang, K. Liu, Y. Qin, S. Tian, Y. Zheng, M. Zhou, C. Yu, H. Li, and D. Zhao World4RL: diffusion world models for policy refinement with reinforcement learning for robotic manipulation. External Links: 2509.19080, Link
12. Jiang et al. (2026b) Z. Jiang, S. Zhou, Y. Jiang, Z. Huang, M. Wei, Y. Chen, T. Zhou, Z. Guo, H. Lin, Q. Zhang, Y. Wang, H. Li, C. Yu, and D. Zhao WoVR: world models as reliable simulators for post-training vla policies with rl. External Links: 2602.13977, Link
13. Kim et al. (2025) M. J. Kim, C. Finn, and P. Liang Fine-tuning vision-language-action models: optimizing speed and success. External Links: 2502.19645, Link
14. Li et al. (2025a) C. Li, A. Krause, and M. Hutter Robotic world model: a neural network simulator for robust policy optimization in robotics. External Links: 2501.10100, Link
15. Li et al. (2026) H. Li, Y. Zuo, J. Yu, Y. Zhang, Y. Zhaohui, K. Zhang, X. Zhu, Y. Zhang, T. Chen, G. Cui, et al. Simplevla-rl: scaling vla training via reinforcement learning. In International Conference on Learning Representations, Vol. 2026, pp. 125846–125869.
16. Li et al. (2025b) W. Li, H. Zhao, Z. Yu, Y. Du, Q. Zou, R. Hu, and K. Xu PIN-wm: learning physics-informed world models for non-prehensile manipulation. External Links: 2504.16693, Link
17. Li et al. (2025c) Y. Li, X. Ma, J. Xu, Y. Cui, Z. Cui, Z. Han, L. Huang, T. Kong, Y. Liu, H. Niu, W. Peng, J. Qiao, Z. Ren, H. Shi, Z. Su, J. Tian, Y. Xiao, S. Zhang, L. Zheng, H. Li, and Y. Wu GR-rl: going dexterous and precise for long-horizon robotic manipulation. External Links: 2512.01801, Link
18. Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone LIBERO: benchmarking knowledge transfer for lifelong robot learning. External Links: 2306.03310, Link
19. Liu et al. (2025a) J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang Flow-grpo: training flow matching models via online rl. External Links: 2505.05470, Link
20. Liu et al. (2026) J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang What can rl bring to vla generalization? an empirical study. External Links: 2505.19789, Link
21. Liu et al. (2025b) S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu RDT-1b: a diffusion foundation model for bimanual manipulation. External Links: 2410.07864, Link
22. Lu et al. (2025) G. Lu, W. Guo, C. Zhang, Y. Zhou, H. Jiang, Z. Gao, Y. Tang, and Z. Wang VLA-rl: towards masterful and general robotic manipulation with scalable reinforcement learning. External Links: 2505.18719, Link
23. Luo et al. (2025) J. Luo, Z. Hu, C. Xu, Y. L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine SERL: a software suite for sample-efficient robotic reinforcement learning. External Links: 2401.16013, Link
24. Mees et al. (2022) O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard CALVIN: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. External Links: 2112.03227, Link
25. Pang et al. (2026) Y. Pang, B. Zhou, C. Li, X. Wang, S. Xu, D. Wang, P. Cheng, S. Di, J. Song, and M. Zhang AtomBridge: agentic vla inference plugin for long-horizon tasks in scientific experiments. External Links: 2602.09430, Link
26. Park et al. (2025) S. Park, Q. Li, and S. Levine Flow q-learning. External Links: 2502.02538, Link
27. Su et al. (2026) Z. Su, W. Kong, H. Dong, and H. Dong IG-rft: an interaction-guided rl framework for vla models in long-horizon robotic manipulation. External Links: 2602.20715, Link
28. Tan et al. (2025) H. Tan, S. Chen, Y. Xu, Z. Wang, Y. Ji, C. Chi, Y. Lyu, Z. Zhao, X. Chen, P. Co, S. Xie, G. Yao, P. Wang, Z. Wang, and S. Zhang Robo-dopamine: general process reward modeling for high-precision robotic manipulation. External Links: 2512.23703, Link
29. Team et al. (2025) G. Team, A. Ye, B. Wang, C. Ni, G. Huang, G. Zhao, H. Li, J. Zhu, K. Li, M. Xu, Q. Deng, S. Wang, W. Qin, X. Chen, X. Wang, Y. Wang, Y. Cao, Y. Chang, Y. Xu, Y. Ye, Y. Wang, Y. Zhou, Z. Zhang, Z. Dong, and Z. Zhu GigaWorld-0: world models as data engine to empower embodied ai. External Links: 2511.19861, Link
30. Torne et al. (2024) M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal Reconciling reality through simulation: a real-to-sim-to-real approach for robust manipulation. External Links: 2403.03949, Link
31. Wang et al. (2026) M. Wang, S. Tian, A. Swann, O. Shorinwa, J. Wu, and M. Schwager Phys2Real: fusing vlm priors with interactive online adaptation for uncertainty-aware sim-to-real manipulation. External Links: 2510.11689, Link
32. Xu et al. (2026) C. Xu, J. T. Springenberg, M. Equi, A. Amin, A. Esmail, S. Levine, and L. Ke RL token: bootstrapping online rl with vision-language-action models. External Links: 2604.23073, Link
33. Xu et al. (2025) F. Xu, G. Zhai, X. Kong, T. Fu, D. F. N. Gordon, X. An, and B. Busam STARE-vla: progressive stage-aware reinforcement for fine-tuning vision-language-action models. External Links: 2512.05107, Link
34. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link
35. Yang et al. (2026a) J. Yang, B. Zhu, J. Chen, and Y. Jiang Actor-critic for continuous action chunks: a reinforcement learning framework for long-horizon robotic manipulation with sparse reward. External Links: 2508.11143, Link
36. Yang et al. (2026b) J. Yang, K. Lin, J. Li, W. Zhang, T. Lin, L. Wu, Z. Su, H. Zhao, Y. Zhang, L. Chen, P. Luo, X. Yue, and H. Li RISE: self-improving robot policy with compositional world model. External Links: 2602.11075, Link
37. Yin et al. (2026) T. Yin, Z. Mei, Z. Zheng, M. Yamane, D. Wang, J. Sceats, S. M. Bateman, L. Zha, A. Badithela, O. Shorinwa, and A. Majumdar PlayWorld: learning robot world models from autonomous play. External Links: 2603.09030, Link
38. Yu et al. (2025) C. Yu, Y. Wang, Z. Guo, H. Lin, S. Xu, H. Zang, Q. Zhang, Y. Wu, C. Zhu, J. Hu, Z. Huang, M. Wei, Y. Xie, K. Yang, B. Dai, Z. Xu, J. Du, X. Wang, X. Fu, L. Shi, Z. Liu, K. Chen, W. Liu, G. Liu, B. Li, J. Yang, Z. Yang, G. Dai, and Y. Wang RLinf: flexible and efficient large-scale reinforcement learning via macro-to-micro flow transformation. External Links: 2509.15965, Link
39. Zhai et al. (2025) S. Zhai, Q. Zhang, T. Zhang, F. Huang, H. Zhang, M. Zhou, S. Zhang, L. Liu, S. Lin, and J. Pang A vision-language-action-critic model for robotic real-world reinforcement learning. External Links: 2509.15937, Link
40. Zhang et al. (2025a) H. Zhang, Z. Zhuang, H. Zhao, P. Ding, H. Lu, and D. Wang ReinboT: amplifying robot visual-language manipulation with reinforcement learning. External Links: 2505.07395, Link
41. Zhang et al. (2026a) L. Zhang, T. Tang, Z. Zhan, X. Chen, Z. Chen, J. Han, J. Zhu, P. Xu, H. Xu, H. Wu, L. Lin, and X. Liang AtomicVLA: unlocking the potential of atomic skill learning in robots. External Links: 2603.07648, Link
42. Zhang et al. (2026b) Y. Zhang, H. Zhang, F. Gao, X. Li, Z. Liu, Y. Nie, C. Zhu, J. Qiu, Y. Yan, J. Liu, W. Tang, J. Rao, Z. Fang, C. Wei, Y. Wang, W. Ding, and C. Yu Harness vla: steering frozen vlas into reliable manipulation primitives via memory-guided agents. External Links: 2607.08448, Link
43. Zhang et al. (2026c) Z. Zhang, H. Ren, Y. Sun, Y. Sheng, H. Wang, H. Lin, Z. Wu, P. Bacon, and Y. Yu Towards practical world model-based reinforcement learning for vision-language-action models. External Links: 2603.20607, Link
44. Zhang et al. (2025b) Z. Zhang, K. Zheng, Z. Chen, J. Jang, Y. Li, S. Han, C. Wang, M. Ding, D. Fox, and H. Yao GRAPE: generalizing robot policy via preference alignment. External Links: 2411.19309, Link

Appendix A Appendix/supplemental material

A.1 Future Work and Limitations

AtomicRL improves temporal credit assignment across atomic tasks through SAPO and concentrates exploration resources on insufficiently mastered bottleneck subtasks through BATE. Although the experimental results demonstrate its effectiveness in long-horizon robotic manipulation, the current approach still has several limitations.

• AtomicRL improves the utilization of interaction data through more effective credit assignment and exploration-resource allocation, but does not fundamentally address the overall efficiency of online RL training. Even with open-source RL frameworks such as RLinf (Yu et al., 2025) and SERL (Luo et al., 2025), online RL still requires substantial environment interaction and training time, particularly in real-robot settings where interaction and exploration costs remain high. Reducing the interaction required for effective policy learning, improving online RL training efficiency, and lowering the cost of real-robot training therefore remain important directions for future work.
• Our experiments primarily focus on long-horizon tabletop manipulation tasks and initialize RL from SFT policies. The effectiveness of AtomicRL on more open and dynamic tasks requiring stronger embodiment generalization remains to be further evaluated. Recent Agentic RL approaches, such as HarnessVLA (Zhang et al., 2026b) and AtomBridge (Pang et al., 2026) have shown potential for open-world tasks and embodiment generalization, providing an alternative direction for long-horizon robotic manipulation. A promising direction is to transfer AtomicRL’s use and optimization of atomic task structure to Agentic RL frameworks, enabling robots to dynamically form and adapt atomic task structures in more open environments and potentially improving the generalization and training efficiency of long-horizon manipulation.

A.2 Progress-Based Shaping Rewards

We optionally supplement the sparse task reward with progress-based shaping rewards (PBSR), using the atomic-task index and discrete progress estimates from PCC.

Progress potential.

Let K denote the number of atomic tasks and c_{t}=(B-1)\hat{p}_{t}\in\{0,\ldots,B-1\} denote the decoded progress level. After completion handling, let z_{t+1} be the active atomic-task index and \widetilde{c}_{t} the tracked progress level. We set \widetilde{c}_{t}=c_{t} unless atomic-task completion is confirmed, in which case it is reset to zero. The task index advances only if another atomic task remains. The updated progress potential is

\Phi_{t+1}=\frac{z_{t+1}-1+\widetilde{c}_{t}/B}{K}.

The potential combines the atomic-task index with within-task progress, with the latter scaled by 1/B.

Shaping reward.

For each valid non-terminal PCC update, we compute

r_{t}^{\mathrm{PBSR}}=\gamma\Phi_{t+1}-\Phi_{t},

where \gamma is the discount factor and \Phi_{t} is the previously stored potential. The stored potential is initialized to zero when the task tracker is reset and replaced by \Phi_{t+1} after each valid update. The shaping contribution is zero on terminal transitions, including environment timeouts, and whenever no valid update is available. The training reward is

r_{t}=r_{t}^{\mathrm{env}}+\alpha_{n}r_{t}^{\mathrm{PBSR}},

where r_{t}^{\mathrm{env}} is the sparse task reward and \alpha_{n} controls the shaping contribution at training iteration n. The shaping reward can be positive or negative and is not clipped.

Shaping schedule.

We linearly anneal the shaping coefficient:

\alpha_{n}=\alpha_{0}\left[1-(1-\rho)\operatorname{clip}\!\left(\frac{n-n_{s}}{n_{e}-n_{s}},\,0,\,1\right)\right],

where \alpha_{0} is the initial coefficient, n_{s}<n_{e} specify the annealing interval, \rho\in[0,1] is the final retained fraction, and \operatorname{clip}(x,0,1) clamps x to [0,1]. Setting \rho=0 removes PBSR after annealing.

PBSR modifies the reward used to compute TD residuals, whereas SAPO regulates their cross-boundary propagation during advantage estimation.

A.3 Additional Ablations

Sensitivity to the SAPO Gating Coefficient. We examine the effect of the SAPO gating coefficient \kappa on LIBERO-LONG under the same configuration as the component ablations.

Here, \kappa controls trace propagation across boundaries preceding failed subtasks, while gates preceding successful subtasks remain fixed at one. Setting \kappa=1 recovers standard GAE propagation, whereas \kappa=0 blocks cross-boundary trace propagation from failed subtasks without removing the one-step value bootstrap. As shown in Table 6, the highest observed success rate is 87.8\% at \kappa=0.3, improving over the unattenuated baseline (78.2\%) by 9.6 percentage points. Performance decreases to 85.0\% and 81.2\% as \kappa increases to 0.5 and 0.8, respectively. Hard trace blocking (\kappa=0) also performs well, achieving 87.2\%, only 0.6 percentage points below \kappa=0.3. These results support the use of soft attenuation of cross-boundary trace propagation from failed subtasks in this setting.

A.4 Training Details

Training Setup. For all LIBERO and real-world experiments, we train PCC offline on annotated demonstrations. PCC uses a frozen SigLIP visual encoder with approximately 400M parameters, initialized from the \pi_{0.5} visual backbone, together with lightweight trainable temporal and prediction modules. For data annotation, we adapt the annotation pipeline of AtomicVLA (Zhang et al., 2026a) to generate fine-grained annotations for demonstration trajectories. We annotate 150 trajectories for LIBERO and 50 trajectories per real-world task, which are subsequently used for offline PCC training. The trained PCC is kept fixed during RL post-training.

All RL post-training experiments are implemented in the RLinf framework. All training is performed on eight NVIDIA H200 GPUs, while inference is conducted on a single NVIDIA GeForce RTX 5090 GPU.

A.5 Additional Results

Detail Results on benchmarks. Fig. 6 and Fig. 7 show the training dynamics and value-function fitting of AtomicRL. On both LIBERO-LONG and CALVIN ABC-D, AtomicRL improves more rapidly during early training and consistently outperforms PPO throughout subsequent training. Meanwhile, the critic explained variance in the later stages of training is higher than PPO by approximately 12% on LIBERO and 5% on CALVIN, indicating that the value function explains a larger fraction of the variance in the training targets. These trends are consistent with the design of SAPO and BATE: SAPO attenuates the cross-boundary propagation of TD residuals from failed atomic subtasks to completed prefixes, improving temporal credit assignment and reducing interference from downstream failures with previously successful behaviors; BATE concentrates sampling and exploration resources on insufficiently mastered bottleneck subtasks. The concurrent improvement in task performance and value-function fitting further supports the effectiveness of atomic-task-structured optimization for long-horizon policy learning.

Detail Results on real-world robot. As shown in Fig. 8, when online RL is performed in the Real2Sim-reconstructed environments, AtomicRL exhibits better training performance than PPO across all three real-world tasks. The advantage is particularly pronounced during early training on Bread Placement (BP), which consists of six atomic subtasks: PPO remains around 50% success over the first 40 training steps, whereas AtomicRL reaches 60% at around step 30 and continues to improve thereafter. This gap reflects the difficulty of learning long task chains under sparse feedback and is consistent with the design of AtomicRL, which attenuates cross-boundary credit interference from downstream failures and thereby makes better use of successful prefixes in failed trajectories. AtomicRL also achieves higher late-stage training success rates on Waste Disposal (WD) and Tabletop Clean (TC), indicating that its training advantage is not limited to a single task. Combined with the simulation–real-world performance correlation analysis in Fig. 3, these training curves provide an indirect reference for the trend of real-world policy improvement.

A.6 Details of automated Real2Sim2Real pipeline

As shown in Fig. 9, we develop an automated Real2Sim2Real workflow, with GPT-6 Astra assisting tool orchestration and code implementation. The workflow connects real-task reconstruction, simulation-based online RL in RLinf (Yu et al., 2025), and real-robot deployment, with the key steps and validation requirements summarized in the figure.

A.7 Additional Visualizations

We provide additional visualizations of policy execution in simulation and the real world. As shown in Fig. 10, SFT completes the initial actions but fails at a later stage, whereas AtomicRL successfully completes the illustrated sequence in both simulation and real-world deployment.