Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance

Abstract

Online reinforcement learning (RL) enables driving policies to adapt to changing deployment conditions, but requires safe exploration throughout training. Safety filtering alone does not ensure stable policy improvement or reduced dependence on corrections. We propose Safe Online Adaptive Refinement (SOAR-Flow), a framework for safe and stable online fine-tuning of offline-initialized Flow-based driving policies. A control barrier function (CBF)–model predictive control (MPC) filter corrects potentially unsafe actions before execution. To support stable policy updates, we introduce Gaussian-return-aware value calibration based on calibrated Q-learning (Cal-QL), using recorded behavior returns and predicted return variability to smooth conservative critic updates and mitigate policy degradation. Paired corrective learning teaches feasible action adjustments, while an intervention-to-go predictor encourages decisions requiring fewer future corrections. Its predictions also regulate guidance strength, allowing recovery supervision to weaken as predicted dependence decreases. Together with gradually reduced calibration regularization, this supports continued task improvement while preserving hard execution constraints. We analyze conditions for safe execution at control steps and diminishing auxiliary learning influences. We evaluate SOAR-Flow in simulation and on an uncrewed ground vehicle (UGV). The results demonstrate improved driving performance and reduced reliance on safety corrections during protected online fine-tuning. The demonstration video and project overview are available at https://lzqw.github.io/SOAR-Flow/.

I Introduction

Learning safe and efficient driving behavior in changing environments remains a central challenge in autonomous driving [1]. Online RL enables continued policy improvement through interaction, with offline demonstrations providing useful initial skills [2, 3]. On UGVs, however, trial-and-error exploration can lead to collisions and other accidents, making safety essential throughout online learning [4].

Preventing failures before they occur is a practical approach to safe exploration. Failure-Aware Offline-to-Online Reinforcement Learning (FARL) follows this principle through failure prediction and preventive recovery in robotic manipulation [5]. Model-based safety filters check actions against explicit constraints and compute feasible corrections before execution [6]. However, successful protected execution can still depend on repeated corrections, while online updates may degrade previously learned behavior [2, 6]. Going beyond safe execution therefore requires learning effectively from protected experience to improve task performance and reduce correction demand.

Reliable value estimates are central to this improvement. Overestimation may favor actions with limited data support, whereas excessive pessimism can undervalue established behaviors as new experience is incorporated [2]. Filtering also changes the executed action, so observed rewards and transitions need to be associated with the action actually applied [6]. Cal-QL addresses value-scale mismatch by calibrating conservative estimates against a reference behavior’s return [2]. This motivates calibration that uses recorded execution outcomes while accounting for return variability during continued online learning.

Learning from corrected actions can reduce repeated interventions [6]. Human–AI Copilot Optimization (HACO) estimates cumulative human intervention costs to guide policy optimization [7], motivating consideration of future assistance alongside immediate corrections. In driving, earlier position or speed choices can determine whether later corrections are needed. Moreover, optimizing a finite-horizon MPC cost need not maximize long-horizon task return. Corrective guidance should therefore combine feasible local adjustments with anticipation of future interventions, while allowing its influence to diminish as dependence decreases. This leaves room for continued reward-driven improvement under unchanged safety constraints.

To address these challenges, we propose SOAR-Flow, a framework for safe online refinement of offline-initialized Flow-based driving policies. CBF–MPC certifies references before execution to protect online interaction. We introduce Gaussian-return-aware value calibration by extending Cal-QL with the Gaussian return model of distributional soft actor–critic (DSAC) [2, 8]. Recorded behavior returns provide calibration references, while predicted return variability smooths conservative critic updates. Paired corrective learning teaches feasible action adjustments, and an intervention-to-go predictor encourages earlier adjustments to reduce subsequent corrections. The same predictions regulate guidance strength, allowing recovery supervision to weaken as predicted dependence decreases. Together with decaying calibration regularization, this supports continued task improvement under unchanged execution constraints. The main contributions are as follows:

• A Gaussian-return-aware extension of Cal-QL combines recorded behavior returns with predicted return variability to smooth conservative critic updates and support stable online learning from protected execution.
• An adaptive guidance mechanism combines paired corrective learning with intervention-to-go prediction. The predictor guides policy actions and regulates guidance strength to reduce filter reliance while retaining task optimization.
• We establish conditions for sample-time safety of protected execution and vanishing auxiliary gradient contributions. Safety-filtered (SF) and policy-only (PO) tests in MetaDrive and on a UGV assess learning stability, safety, and autonomy with nominal MPC tracking retained.

II Related Work

Safe RL and intervention-based learning. Safe RL addresses safety through policy optimization and execution-time protection. Constrained policy optimization regulates expected costs in policy updates [9], whereas recovery policies and predictive safety filters intervene using learned risk estimates or model-based feasibility checks [10, 11]. For pretrained policies, SafeFlow enforces barrier conditions during test-time flow-based trajectory generation [12], while FARL uses an offline-trained world model and recovery policy to prevent failures during online refinement, keeping both protection components fixed online [5]. Beyond execution protection, interventions also provide information for policy learning. Safety Filtering While Training studies how filtering and correction penalties affect learned behavior [6], while CBF-RL combines training-time filtering with barrier-inspired rewards to encourage safer actions without runtime correction [13]. HACO estimates cumulative human intervention costs to guide policy optimization [7]. Building on this connection between protection and learning, SOAR-Flow combines paired corrective supervision with prediction of future filter use. The predictions guide policy actions and regulate supervision strength, while CBF–MPC certification remains active.

Value learning and online policy refinement. Online refinement seeks to improve from new experience while retaining useful pretrained behavior. Offline RL addresses unsupported actions through conservative value estimation in conservative Q-learning or in-dataset value learning in implicit Q-learning [14, 15]. However, excessive pessimism can hinder the transition to online learning, motivating Cal-QL’s calibration against a reference behavior’s return [2]. Policy regularization and objective alignment offer complementary approaches: PROTO evolves its policy regularizer during fine-tuning [16], while Uni-O4 uses a shared on-policy objective across offline and online stages [17]. Generative policies complement these optimization methods by representing diverse behaviors in prior data. Flow Q-learning combines flow-based action modeling with value-driven training of an expressive one-step actor [3]; RL-100 connects imitation, iterative offline RL, and online refinement for diffusion-based robotic policies [18]. Building on Cal-QL and DSAC’s Gaussian return model [2, 8], SOAR-Flow uses predicted return variability to smooth behavior-referenced calibration of values learned from certified execution. This value-learning objective complements adaptive intervention guidance during protected online policy refinement.

III Methodology

Building on Cal-QL [2], SOAR-Flow performs safe online refinement of a Flow policy and task critics initialized from offline data. CBF–MPC checks policy actions before execution. The resulting transitions support Gaussian-return-aware value calibration, while corrections and intervention sequences guide policy improvement. Adaptive guidance reduces auxiliary learning influences as predicted filter dependence decreases. Fig. 1 summarizes this protected interaction and update loop.

Autonomous driving is modeled as a continuous-action Markov decision process (MDP) \mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle, where \mathcal{S} and \mathcal{A} are the state and driving-reference spaces, \mathcal{P} is the transition kernel, \mathcal{R} is the task reward function, and \gamma\in[0,1) is the discount factor. The policy maximizes the entropy-augmented objective of soft actor–critic (SAC) [19]

J_{\pi}=\mathbb{E}_{(s_{i\geq t},a_{i\geq t})\sim\rho_{\pi}}\left[\sum_{i=t}^{\infty}\gamma^{i-t}\left[r_{i}+\alpha\mathcal{H}\bigl(\pi(\cdot\mid s_{i})\bigr)\right]\right],

where r_{i} is the task reward, \rho_{\pi} is the policy-induced trajectory distribution, \alpha\geq 0 is the temperature coefficient, and \mathcal{H}(\pi(\cdot\mid s))=\mathbb{E}_{a\sim\pi(\cdot\mid s)}[-\log\pi(a\mid s)]. The optimal policy satisfies \pi^{\star}\in\arg\max_{\pi}J_{\pi}.

A conditional Flow policy transforms Gaussian noise into driving references through a state-conditioned vector field v_{\phi} [3, 20]. Given state s and z\sim\mathcal{N}(0,I), the flow satisfies

\frac{\mathrm{d}x_{\tau}}{\mathrm{d}\tau}=v_{\phi}(x_{\tau},\tau,s),\qquad x_{0}=z,\quad\tau\in[0,1],

where \tau denotes flow time. The endpoint \mu_{\phi}(s,z)=x_{1} induces the policy distribution \pi_{\phi}(\cdot\mid s). Given an offline dataset \mathcal{D}_{\mathrm{off}} of state–reference pairs, we sample (s,a)\sim\mathcal{D}_{\mathrm{off}} and \tau\sim\mathcal{U}[0,1] independently of z, and let \widetilde{x}_{\tau}=(1-\tau)z+\tau a. Flow-based behavior cloning (Flow-BC) initializes the policy by minimizing

\mathcal{L}_{\mathrm{FM}}(\phi)=\mathbb{E}\!\left[\left\|v_{\phi}(\widetilde{x}_{\tau},\tau,s)-(a-z)\right\|_{2}^{2}\right].

III-A Calibrated Value Learning under Safety Filtering

Maintaining safe exploration. CBF–MPC checks policy actions using a prescribed nominal model and the same MPC tracking solution used for execution [6]. Let \xi_{t} denote the prediction state. Barrier functions h_{j} define \mathcal{C}=\{\xi\in\mathcal{X}:h_{j}(\xi)\geq 0,\ \forall j\}, where \mathcal{X} is the admissible state set. For reference a, the predicted tracking trajectory satisfies the discrete-time CBF condition [13, 21]

h_{j}(\hat{\xi}^{a}_{k+1\mid t})\geq\rho_{j}h_{j}(\hat{\xi}^{a}_{k\mid t}),\qquad\rho_{j}\in[0,1).

Here, \hat{\xi}^{a}_{0\mid t}=\xi_{t}, k=0,\ldots,N_{p}-1, and j=1,\ldots,n_{h}, with prediction horizon N_{p}. The set \mathcal{A}_{\mathrm{cert}}(\xi_{t}) contains references whose tracking solutions satisfy this condition, admissible state and input constraints, and a terminal condition in \mathcal{X}_{f}\subseteq\mathcal{C}. The terminal set admits a feasible backup controller that keeps the nominal state within it. For example, if tracking a reference would violate obstacle clearance, the filter searches for a feasible bypass reference before execution. Online interaction follows

\displaystyle a_{t}^{\mathrm{raw}} \\ \displaystyle\sim\pi_{\phi}(\cdot\mid s_{t}), \\ \displaystyle a_{t}^{\mathrm{cert}} \\ \displaystyle=\mathcal{F}_{\xi_{t}}(a_{t}^{\mathrm{raw}}),

where \mathcal{F}_{\xi_{t}} retains certifiable references or returns feasible corrections. The first input of the associated MPC tracking solution is applied as u_{t}=u^{\star}_{0\mid t}, with certification repeated at the next control step. We store \mathcal{T}_{t}=(s_{t},a_{t}^{\mathrm{raw}},a_{t}^{\mathrm{cert}},\iota_{t},r_{t},s_{t+1},d_{t}), where \iota_{t}=1 denotes a certification-triggered correction and d_{t}=1 denotes true termination. The generator input and prediction states are retained for paired updates and target certification. These transitions form \mathcal{D}_{\mathrm{on}}, with mixed replay \mathcal{D}=\mathcal{D}_{\mathrm{off}}\cup\mathcal{D}_{\mathrm{on}}. The reward r_{t} and next state s_{t+1} result from tracking a_{t}^{\mathrm{cert}} and therefore supervise the task critics at (s_{t},a_{t}^{\mathrm{cert}}) [6]. Recorded corrections provide action targets for learning references that pass certification directly.

Avoiding policy degradation. Cal-QL uses a reference behavior’s return to limit excessive pessimism during O2O learning [2]. For each transition from a completed episode ending at T, we store the discounted task return b_{t}=\sum_{k=t}^{T-1}\gamma^{k-t}r_{k} as its calibration reference. This quantity is computed from recorded offline and online trajectories; for online data, it includes rewards from both unmodified and corrected execution. Cal-QL’s candidate term \max\{Q(s_{t},a),b_{t}\} retains conservative pressure above the reference and stops it below, producing an abrupt change in its mean gradient. We extend this construction with DSAC’s Gaussian return model [8] to smooth calibration using predicted return variability. For return variable Z^{\pi}(s,a), its mean satisfies Q^{\pi}(s,a)=\mathbb{E}[Z^{\pi}(s,a)]. Each critic predicts the mean Q and variance \sigma^{2}>0 of a Gaussian approximation:

\mathcal{Z}(\cdot\mid s,a)=\mathcal{N}\!\left(Q(s,a),\sigma^{2}(s,a)\right).

For a candidate a\sim\pi_{\phi}(\cdot\mid s_{t}), let Q_{i}=Q_{\theta_{i}}(s_{t},a) and \sigma_{i}=\sigma_{\theta_{i}}(s_{t},a)>0 denote the outputs of critic i\in\{1,2\}. We define

\displaystyle\mathcal{G}_{i}(s_{t},a;b_{t}) \\ \displaystyle:={\mathbb{E}}_{\epsilon\sim\mathcal{N}(0,1)}[\max\{Q_{i}+\sigma_{i}\epsilon,b_{t}\}] \\ \displaystyle=b_{t}+(Q_{i}-b_{t})\Phi(\nu_{i})+\sigma_{i}\varphi(\nu_{i}).

The standardized offset is \nu_{i}=(Q_{i}-b_{t})/\sigma_{i}; \Phi and \varphi denote the standard normal cumulative distribution and density functions. For fixed reference and standard deviation (SD), \partial\mathcal{G}_{i}/\partial Q_{i}=\Phi(\nu_{i}), so the conservative pressure changes continuously around b_{t}. The predicted SD determines the smoothing width, and the mean-only truncation is recovered in the zero-variance limit. Gradients through \sigma_{i} are disabled in \mathcal{G}_{i} to avoid directly shrinking the return dispersion; the SD remains trainable in the Bellman term.

At the recorded execution pair, write Q_{i}^{\mathrm{cert}}=Q_{\theta_{i}}(s_{t},a_{t}^{\mathrm{cert}}) and \sigma_{i}^{\mathrm{cert}}=\sigma_{\theta_{i}}(s_{t},a_{t}^{\mathrm{cert}}). The complete critic objective is

\displaystyle\mathcal{L}_{\mathrm{task},i}={} \\ \displaystyle\beta_{\mathrm{cal},n}\mathbb{E}_{\mathcal{D},a}\!\left[\mathcal{G}_{i}(s_{t},a;b_{t})-Q_{i}^{\mathrm{cert}}\right] \\ \displaystyle+\mathbb{E}_{\mathcal{D},a^{\prime},Z^{\prime}}\!\left[\frac{(Y_{t}^{Z}-Q_{i}^{\mathrm{cert}})^{2}}{2(\sigma_{i}^{\mathrm{cert}})^{2}}+\log\sigma_{i}^{\mathrm{cert}}\right],

where \beta_{\mathrm{cal},n}\geq 0 controls calibration at update n. The first expectation samples current policy actions at replay states and uses only records with available b_{t}; its unweighted value is \mathcal{R}_{\mathrm{GCal},i}. The second uses all replay transitions to fit the Gaussian return distribution. Its mean gradient is scaled by the inverse predicted variance. Together, these terms fit recorded execution outcomes while calibrating candidate values against observed behavior.

The Bellman target follows the same protected execution rule as data collection. For (s^{\prime},\xi^{\prime})=(s_{t+1},\xi_{t+1}), we use

\displaystyle a^{\prime} \\ \displaystyle\sim\pi_{\bar{\phi}}(\cdot\mid s^{\prime}),\quad a^{\prime\mathrm{cert}}=\mathcal{F}_{\xi^{\prime}}(a^{\prime}), \\ \displaystyle Z^{\prime} \\ \displaystyle\sim\overline{\mathcal{Z}}(\cdot\mid s^{\prime},a^{\prime\mathrm{cert}}), \\ \displaystyle Y_{t}^{Z} \\ \displaystyle=r_{t}+\gamma(1-d_{t})Z^{\prime}.

Bars denote target quantities. At each (s^{\prime},a^{\prime\mathrm{cert}}), \overline{\mathcal{Z}} is the Gaussian distribution of the target critic with the smaller mean, using that same critic’s variance [22]. The sample Z^{\prime} represents the continuation return under protected execution; terminal transitions use Y_{t}^{Z}=r_{t}. The objective is used for both offline initialization and online refinement, with actor parameters, Bellman targets, and recorded behavior returns fixed during critic updates.

III-B Adaptive Guidance for Autonomous Policy Improvement

Calibrated critics support policy improvement from protected experience, but successful execution can still rely on repeated corrections. Moreover, minimizing a finite-horizon MPC cost need not maximize long-horizon task return. We therefore use corrections as local guidance while allowing the policy to discover higher-return behavior within the prescribed safety constraints.

Learning from task values and corrections. Online records provide two complementary learning signals. For \iota=0, the task critic guides improvement of the policy action; for \iota=1, the executed correction provides a feasible action target. Including the entropy term in (1), we use

\displaystyle\mathcal{L}_{Q}(\phi)={} \\ \displaystyle-\mathbb{E}_{\mathcal{D}_{\mathrm{on}}}\!\left[\bar{Q}_{\theta}(s,\mu_{\phi}(s,z))\mid\iota=0\right] \\ \displaystyle-\alpha\mathbb{E}_{s\sim\mathcal{D}_{\mathrm{on}}}\!\left[\mathcal{H}(\pi_{\phi}(\cdot\mid s))\right], \\ \displaystyle\mathcal{L}_{\mathrm{corr}}(\phi)={} \\ \displaystyle\mathbb{E}_{\mathcal{D}_{\mathrm{on}}}\!\left[\|\mu_{\phi}(s,z)-a^{\mathrm{cert}}\|_{W_{a}}^{2}\mid\iota=1\right].

The clipped value is \bar{Q}_{\theta}=\min_{i}Q_{\theta_{i}}, \alpha is the entropy coefficient, and W_{a}\succ 0 weights the corrective error. The recorded generator input z is reused for these paired updates, associating each correction with the action that triggered it. Entropy is evaluated using fresh samples from the current raw policy. The intervention labels select replay records; updated actions are checked again during execution.

Anticipating future interventions. An earlier position or speed choice can determine whether a later correction is needed. Motivated by intervention-value learning [7], we estimate discounted future filter use:

\displaystyle P^{\pi}(s_{t},a_{t}^{\mathrm{raw}}) \\ \displaystyle=(1-\gamma_{P})\mathbb{E}_{\pi,\mathcal{F}}\!\left[\sum_{\ell=0}^{\infty}\gamma_{P}^{\ell}\iota_{t+\ell}\,\middle|\,s_{t},a_{t}^{\mathrm{raw}}\right].

The discount factor satisfies \gamma_{P}\in[0,1); subsequent actions follow \pi, and transitions follow certified execution. Interventions after termination are zero, giving P^{\pi}\in[0,1]. To propagate delayed interventions, we use contiguous segments of K transitions ending at termination or a collection boundary. The observed interventions provide a partial target, and the endpoint estimate supplies the remaining continuation:

\displaystyle y_{t}^{P}={} \\ \displaystyle(1-\gamma_{P})\sum_{\ell=0}^{K-1}\gamma_{P}^{\ell}\iota_{t+\ell} \\ \displaystyle+\gamma_{P}^{K}(1-d_{t+K-1})\mathbb{E}_{a^{\prime}}[P_{\psi}(s_{t+K},a^{\prime})], \\ \displaystyle\mathcal{L}_{P}(\psi)={} \\ \displaystyle\tfrac{1}{2}\mathbb{E}_{\mathcal{D}_{\mathrm{on}}}\!\left[(P_{\psi}(s_{t},a_{t}^{\mathrm{raw}})-y_{t}^{P})^{2}\right].

The bootstrap action a^{\prime}\sim\pi_{\phi}(\cdot\mid s_{t+K}) is sampled from the raw policy. True termination removes the continuation term. Targets are fixed during regression, and P_{\psi} is bounded in [0,1]. Setting K=1 gives the one-step target; longer segments propagate observed intervention signals farther in each update. These targets approximate current-policy evaluation from collected data.

The complete actor objective is

\displaystyle\mathcal{L}_{\pi}(\phi)={} \\ \displaystyle\mathcal{L}_{Q}(\phi) \\ \displaystyle+\lambda_{F,n}\Big[\beta_{\mathrm{corr}}\mathcal{L}_{\mathrm{corr}}(\phi)+\mathbb{E}_{s,a}[P_{\psi}(s,a)]\Big],

where s\sim\mathcal{D}_{\mathrm{on}}, a\sim\pi_{\phi}(\cdot\mid s), and \beta_{\mathrm{corr}}>0. Task values favor higher returns, corrections teach feasible adjustments, and P_{\psi} discourages actions leading to later assistance. The critics, correction targets, and coefficients are fixed during actor optimization; gradients pass through the generated actions.

Relaxing corrective guidance. Corrective supervision helps the policy acquire feasible behavior, but its influence should diminish as the policy learns to operate with less assistance. We therefore reuse the intervention-to-go estimate in (12) to regulate guidance strength. At round n, we evaluate P_{\psi}(s,a) on states from the latest online batch and raw actions a\sim\pi_{\phi}(\cdot\mid s) sampled from the current policy. The average predicted dependence is smoothed and used to update the guidance weight:

\displaystyle\bar{P}_{n} \\ \displaystyle=(1-\beta_{P})\bar{P}_{n-1}+\beta_{P}\mathbb{E}_{s,a}[P_{\psi}(s,a)], \\ \displaystyle\lambda_{F,n+1} \\ \displaystyle=\Pi_{[0,\lambda_{\max}]}\!\left[\lambda_{F,n}+\eta_{\lambda}(\bar{P}_{n}-\kappa_{\lambda}\lambda_{F,n})\right].

The operator \Pi denotes interval projection, with \beta_{P}\in(0,1], \eta_{\lambda},\kappa_{\lambda}>0, and 0<\eta_{\lambda}\kappa_{\lambda}<1. The feedback statistic is computed without gradient propagation, and \lambda_{F,n+1} is held fixed during the subsequent actor updates. Predicted intervention demand counteracts weight decay; vanishing feedback gives \lambda_{F,n}\to 0. Together with the decaying calibration weight \beta_{\mathrm{cal},n}, this reduces the influence of recovery supervision and value regularization, allowing continued task improvement under unchanged execution constraints.

III-C Safety and Asymptotic Guidance Analysis

SOAR-Flow separates execution protection from learning guidance. The following result characterizes sampling-time safety and the asymptotic removal of auxiliary contributions to policy and value updates.

Theorem 1

Certified Safety and Vanishing Auxiliary Guidance. Starting from a feasible \xi_{0}\in\mathcal{C}, suppose CBF–MPC remains feasible and the realized transitions satisfy h_{j}(\xi_{t+1})\geq\rho_{j}h_{j}(\xi_{t})-\epsilon_{j,t}, with 0\leq\rho_{j}<1 and 0\leq\epsilon_{j,t}\leq\bar{\epsilon}_{j}. Then

h_{j}(\xi_{t})\geq\rho_{j}^{t}h_{j}(\xi_{0})-\frac{1-\rho_{j}^{t}}{1-\rho_{j}}\bar{\epsilon}_{j}.

In particular, zero residual bounds preserve nonnegative barriers at every sampling instant throughout online learning.

Let the guidance weight follow (15), with \bar{P}_{0}\in[0,1], \lambda_{F,0}\in[0,\lambda_{\max}], and 0<\eta_{\lambda}\kappa_{\lambda}<1. Suppose \bar{P}_{n}\to 0, \beta_{\mathrm{cal},n}\to 0, and the actor gradients of \mathcal{L}_{\mathrm{corr}} and \mathbb{E}_{s,a}[P_{\psi}(s,a)], as well as the critic-mean gradients \nabla_{\theta_{i}}Q_{\theta_{i}}(s,a) at all calibration samples, are uniformly bounded along the learning iterates. Then \lambda_{F,n}\to 0 and

\displaystyle\|\nabla_{\phi}\mathcal{L}_{\pi}-\nabla_{\phi}\mathcal{L}_{Q}\| \\ \displaystyle\to 0, \\ \displaystyle\|\beta_{\mathrm{cal},n}\nabla_{\theta_{i}}\mathcal{R}_{\mathrm{GCal},i}\| \\ \displaystyle\to 0.

All gradients follow the fixed-target, fixed-coefficient, and fixed-calibration-scale conventions specified above.

The barrier bound follows by iterating the realized CBF inequality. The guidance recursion has contraction factor 1-\eta_{\lambda}\kappa_{\lambda}\in(0,1), while \partial\mathcal{G}_{i}/\partial Q_{i}=\Phi(\nu_{i})\in[0,1] bounds the calibration gradient under the stated assumptions. Thus, the direct influence of corrective, dependence, and calibration terms vanishes as predicted dependence and calibration weights tend to zero. The task and entropy objective remains active without a persistent gradient penalty for departing from the guardian’s particular recovery choices, preserving scope for higher-return behavior within the certified region while execution constraints remain enforced.

IV Simulation Experiments

We evaluate whether SOAR-Flow improves an initialized driving policy while limiting training violations and reducing reliance on safety corrections across different simulated scenarios. Baseline comparisons and ablations examine the contributions of value calibration and adaptive guidance.

IV-A Experimental Setup

We conduct experiments in SafeMetaDriveEnv of MetaDrive [23], where scenario seeds specify different road layouts and obstacle configurations. Fig. 2 shows a scenario and the experimental workstation. An MPC controller collects offline trajectories on ten fixed scenario seeds to initialize the Flow policy and Gaussian task critics. Twenty additional seeds, excluded from offline training, are used for online fine-tuning and evaluation. The actor receives a 259-dimensional observation comprising 19 ego-state and navigation variables, including speed, heading, steering state, road-boundary distances, and upcoming checkpoints, together with 240 light detection and ranging (LiDAR)-like range measurements. Four additional safety-context variables are provided only to the task critics. The normalized actor output a_{t}=[a_{t}^{\mathrm{lon}},a_{t}^{\mathrm{lat}}]\in[-1,1]^{2} is decoded into a local target in the ego frame:

x_{t}^{\mathrm{ref}}=\tfrac{1}{2}(a_{t}^{\mathrm{lon}}+1)L,\qquad y_{t}^{\mathrm{ref}}=a_{t}^{\mathrm{lat}}W,

with L=10 m and W=3.7 m. CBF–MPC certifies or corrects the reference before a nominal nonlinear model predictive control (NMPC) tracker produces steering and longitudinal commands. Task reward and safety cost are recorded separately. The maximum commanded speed is 10.75 m/s. Online training uses 10^{6} environment steps per run and five independent training seeds.

Comparisons and metrics. Baselines include fixed CBF–MPC, offline Flow-BC, Cal-QL [2], PROTO-SAC [16], Gaussian-DSAC, and Flow-Q-only. Ablations examine corrective learning, intervention-to-go, and calibration. PO evaluation disables CBF correction but retains NMPC, whereas SF evaluation retains certification. PO reports episodic return, SS, and evaluation safety cost. SS requires arrival without collision, road departure, or nonzero episode cost. Protected training reports cumulative safety cost. In SF evaluation, certification-based intervention counts and rates are defined using \iota_{t}. Repeated trials are averaged within each training seed, followed by the mean and sample SD across seeds.

IV-B Results and Analysis

Online improvement. Fig. 3(a) and Table I present the learning curves and final comparison. The final return listed for SOAR-Flow is 352.3\pm 2.1, compared with 309.1\pm 4.8 for Flow-BC and 343.5\pm 3.0 for Gaussian-DSAC. This value is close to the fixed CBF–MPC reference of 350.0\pm 0.6, whereas Flow-Q-only falls below the offline baseline to 278.2\pm 12.1. The main contrast is improvement over offline initialization; the fixed controller provides a performance reference rather than a learning curve.

Safety and dependence. Fig. 3(b–c) and Table I summarize training cost and correction demand. SOAR-Flow is listed with a training cost of 0.18\pm 0.10 and a correction rate of 0.72\pm 0.12\%, compared with 0.85\pm 0.32 and 2.68\pm 0.28\% for Gaussian-DSAC. The table also lists 100\% SS and zero evaluation cost under PO evaluation. These metrics distinguish protected collection from execution without reference correction: low training cost alone does not establish autonomous task completion.

Ablations. The ablation comparison is presented in Table I. The variants without corrective learning and without P have correction rates of 4.98\% and 1.98\%, respectively, compared with 0.72\% for the full method. Removing calibration and using mean-only calibration give return entries of 338.0 and 348.0, respectively. Mean-only calibration narrows the return gap, but its listed correction rate remains higher at 1.20\%. The full-method entry combines the highest return with the lowest training cost among these variants. PO safe-success entries are also lower without correction or P, linking the tabulated comparison to task completion as well as correction demand.

V Real-World Experiments

We examine protected online adaptation on a UGV and the development of driving behavior requiring fewer safety corrections. A frozen-policy test on a held-out cornering task further assesses execution beyond the training layouts.

V-A Experimental Setup

Our UGV in Fig. 4 is equipped with LiDAR, cameras, radar, and an integrated navigation system (INS). Policy inputs combine ego motion, surrounding obstacle distances, and navigation checkpoints [4]. Policy inference and NMPC run onboard at 10 Hz and 20 Hz, respectively. The actor specifies a local reference; following certification, NMPC generates steering and longitudinal commands transmitted through the base adapter unit (BAU).

Offline data are collected by a human driver in the two-sided obstacle-bypass layout of Fig. 5(a). We record 24 demonstration trajectories, with 12 passing on each side, to pretrain the UGV Flow policy and Gaussian critics. Online refinement alternates between Blocked-Side Bypass in Fig. 5(b), where one demonstrated passing route is blocked, and 8-shaped Maneuver in Fig. 5(c), which requires successive turns in opposite directions around two obstacles. The first task changes which demonstrated bypass remains feasible; the second requires the exit from one bypass to accommodate the following turn. A single policy is updated across both tasks under CBF–MPC protection, with their experience stored in the same online replay buffer. Checkpoints are evaluated on both tasks without parameter updates to assess performance on each layout. The final policy is frozen and tested zero-shot on Cornering with Obstacle Avoidance in Fig. 5(d), which combines a 90^{\circ} bend with two obstacles. This layout is excluded from demonstrations, online training, tuning, and checkpoint selection. The UGV protocol uses three independent training seeds.

Comparisons and metrics. UGV comparisons use CBF–MPC, Flow-BC, and Gaussian-DSAC. We use the PO and SF evaluation modes and SS criterion described in Sec. IV-A. Each method’s reported SS is computed from 20 evaluation trials per task. Learned-policy SS and return use PO; correction rates use SF. Fixed CBF–MPC runs independently. Table II groups results by task, with continuous metrics reported as mean \pm SD.

V-B Results and Analysis

Online adaptation. Fig. 5(e–f) and Table II show blocked-side adaptation. SOAR-Flow achieves 100\% SS and a return of 350.8\pm 1.8, compared with 343.9\pm 2.6 for Gaussian-DSAC. Its correction rate is 0.68\pm 0.11\%, versus 2.46\pm 0.32\% for Gaussian-DSAC and 4.31\pm 0.41\% for Flow-BC. The 39.6 return gain over Flow-BC accompanies reduced correction demand after one demonstrated route is blocked.

Autonomous execution and efficiency. Fig. 6 shows 8-shaped refinement from corrected early and intermediate runs to late execution without CBF correction, with nominal MPC tracking retained. The first turn must prepare for the following opposite-direction turn. In Table II, SOAR-Flow reaches 100\% 8-shaped SS, compared with 95\% for CBF–MPC and 90\% for Gaussian-DSAC. Its completion time is 15.36\pm 0.24 s, versus 16.72\pm 0.31 s for CBF–MPC, a reduction of 1.36 s (approximately 8.1\%). The corresponding minimum clearances are 0.801\pm 0.019 m and 0.779\pm 0.018 m. Thus, shorter completion time accompanies slightly larger mean clearance.

Zero-shot cornering. On held-out cornering, SOAR-Flow attains 95\% SS, compared with 85\% for fixed CBF–MPC, 55\% for Flow-BC, and 90\% for Gaussian-DSAC. Fig. 5(d) contrasts SOAR-Flow clearing both obstacles with fixed CBF–MPC contacting the downstream obstacle, highlighting the need to preserve clearance for subsequent avoidance. The frozen policy transfers without further updates, although not every evaluation succeeds.

VI Conclusion

This paper presented SOAR-Flow for safe online refinement of offline-initialized Flow-based driving policies under CBF–MPC protection. Gaussian-return-aware calibration uses recorded behavioral outcomes to regulate critic updates, while paired corrections and intervention-to-go learning guide policy improvement. Intervention-responsive guidance and decaying calibration weights reduce auxiliary influences under the stated conditions. The analysis establishes conditional sample-time safety of protected execution and vanishing auxiliary gradients. Together, these components connect protected data collection with continued task optimization, aiming to reduce reliance on particular recovery behaviors while retaining the prescribed execution constraints.

References

1. [1] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez (2022) Deep reinforcement learning for autonomous driving: a survey. IEEE Transactions on Intelligent Transportation Systems 23 (6), pp. 4909–4926. External Links: Document
2. [2] M. Nakamoto, Y. Zhai, A. Singh, M. S. Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine (2023) Cal-QL: calibrated offline RL pre-training for efficient online fine-tuning. In Advances in Neural Information Processing Systems,
3. [3] S. Park, Q. Li, and S. Levine (2025) Flow Q-learning. In Proceedings of the International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, pp. 48104–48127.
4. [4] Z. Li, Y. Wang, H. Wang, Z. Li, P. Li, W. Liu, and Z. Zuo (2025) From learning to mastery: achieving safe and efficient real-world autonomous driving with human-in-the-loop reinforcement learning. arXiv:2510.06038. External Links: Document
5. [5] H. Li, K. Lei, S. Zang, K. Hu, Y. Liang, B. An, X. Li, and H. Xu (2026) Failure-aware RL: reliable offline-to-online reinforcement learning with self-recovery for real-world manipulation. arXiv:2601.07821. External Links: Document
6. [6] F. Pizarro Bejarano, L. Brunke, and A. P. Schoellig (2025) Safety filtering while training: improving the performance and sample efficiency of reinforcement learning agents. IEEE Robotics and Automation Letters. External Links: Document
7. [7] Q. Li, Z. Peng, and B. Zhou (2022) Efficient learning of safe driving policy via Human-AI copilot optimization. In International Conference on Learning Representations, External Links: 2202.10341, Link
8. [8] J. Duan, Y. Guan, S. E. Li, Y. Ren, and B. Cheng (2022) Distributional soft actor-critic: off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems 33 (11), pp. 6584–6598. External Links: Document
9. [9] J. Achiam, D. Held, A. Tamar, and P. Abbeel (2017) Constrained policy optimization. In Proceedings of the International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 70, pp. 22–31.
10. [10] B. Thananjeyan, A. Balakrishna, and S. Nair (2021) Recovery RL: safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters 6 (3), pp. 4915–4922. External Links: Document
11. [11] K. P. Wabersich and M. N. Zeilinger (2021) A predictive safety filter for learning-based control of constrained nonlinear dynamical systems. Automatica 129, pp. 109597. External Links: Document
12. [12] X. Dai, Z. Yang, D. Yu, F. Liu, H. Sadeghian, S. Haddadin, and S. Hirche (2025) SafeFlow: safe robot motion planning with flow matching via control barrier functions. arXiv:2504.08661. Note: Version 3 External Links: Document
13. [13] L. Yang, B. Werner, M. de Sa, and A. D. Ames (2026) CBF-RL: safety filtering reinforcement learning in training with control barrier functions. In Proceedings of the IEEE International Conference on Robotics and Automation, Note: arXiv:2510.14959
14. [14] A. Kumar, A. Zhou, G. Tucker, and S. Levine (2020) Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1179–1191.
15. [15] I. Kostrikov, A. Nair, and S. Levine (2022) Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations,
16. [16] J. Li, X. Hu, H. Xu, J. Liu, X. Zhan, and Y. Zhang (2023) PROTO: iterative policy regularized offline-to-online reinforcement learning. arXiv:2305.15669.
17. [17] K. Lei, Z. He, C. Lu, K. Hu, Y. Gao, and H. Xu (2024) Uni-O4: unifying online and offline deep reinforcement learning with multi-step on-policy optimization. In International Conference on Learning Representations,
18. [18] K. Lei, H. Li, D. Yu, Z. Wei, L. Guo, Z. Jiang, Z. Wang, S. Liang, and H. Xu (2025) RL-100: performant robotic manipulation with real-world reinforcement learning. arXiv:2510.14830. External Links: Document
19. [19] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, pp. 1861–1870.
20. [20] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In International Conference on Learning Representations,
21. [21] A. Agrawal and K. Sreenath (2017) Discrete control barrier functions for safety-critical control of discrete systems with application to bipedal robot navigation. In Proceedings of Robotics: Science and Systems, Cambridge, MA, USA. External Links: Document
22. [22] J. Duan, W. Wang, L. Xiao, J. Gao, S. E. Li, C. Liu, Y. Zhang, B. Cheng, and K. Li (2025) Distributional soft actor-critic with three refinements. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (5), pp. 3935–3946. External Links: Document
23. [23] Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou (2023) MetaDrive: composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (3), pp. 3461–3475. External Links: Document