RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments

Can RL-trained policy, value, and sensor models be safely embedded in a sampling-based stochastic NMPC controller to improve long-horizon perception-based navigation while retaining finite-time probabilistic safety guarantees?

Agile underactuated robots navigating unknown, cluttered environments with local vision must handle nonlinear stochastic dynamics, partial observations, and limited sensor field-of-view. Pure NMPC is often short-horizon and myopic, pure RL lacks hard safety constraints and deteriorates out-of-distribution, and many prior hybrids lack safety guarantees, ignore perception uncertainty, or are overly conservative.

Prior approaches include robust/stochastic NMPC, which enforces constraints but is computationally limited and myopic; deep RL (e.g., TD3/SAC/PPO), which offers long-range performance but no hard constraints and suffers in sim-to-real transfer; safe RL via CMDPs, CBFs, or safety layers, whose guarantees often fail outside training; and RL-MPC hybrids such as PETS, POLO, MQP, LOOP, TD-MPC, and Safe RL-MPC, none of which combined stochastic dynamics, stochastic value functions, safety guarantees, large perception inputs, and hardware validation.

The authors train a TD3 actor-critic and a non-recursive sensor-prediction network in simulation; Monte Carlo dropout approximates policy and value uncertainty. At runtime, the actor warm-starts the PAC-NMPC surrogate distribution by rolling out actions with a nominal dynamics model and predicted future depth images; the critic becomes a stochastic terminal cost; and an additional value-function improvement bound guides the optimization. PAC-NMPC then optimizes sampled feedback policies to satisfy finite-time probabilistic collision-avoidance constraints. Hardware demos use a 32-inch fixed-wing UAV with a 16×12 depth-image input and a rally-car UGV with LiDAR.

Fixed-wing simulation over 100 training-distribution environments: AC-PAC-NMPC achieved 90% success (goal reached without constraint violation) versus 85% for the best PAC-NMPC baseline (map + A* + unknown-space cost) and 81% for the raw RL actor; cost was 513.3 versus 718.14 for that baseline. Ablations removing RL warm-start or learned sensor prediction reduced success to 3% and 11%. In out-of-distribution tests with an added horizontal obstacle, AC-PAC-NMPC matched the best baseline at 76% success but with lower cost (512.1 vs 732.6), while RL actor dropped to 46%. Real-robot fixed-wing tests across 10 environments: 80% success and cost 252.2 for AC-PAC-NMPC, versus 40% success for both RL actor and PAC-NMPC (costs 403.8 and 325.4). Earlier rally-car hardware tests in 20 environments: PAC-NMPC with a learned value function had 95% success and 0% constraint violations versus 85% and 15% for the actor; with model mismatch, 80% and 0% versus 70% and 30%.

The probabilistic safety guarantees are finite-horizon and conditional on current observations, not global or infinite-horizon. Monte Carlo dropout uncertainty is not calibrated or bounded; the gap between learned and true value functions is not analyzed. Hyperparameters such as constraint penalties were hand-tuned. Hardware evaluations used only 10 fixed-wing and 20 rally-car trials in a motion-capture lab with external state estimation and offboard laptop compute, not an embedded system. Training was simulation-only, and results are limited to the tested obstacle types and dynamics; no code release is mentioned in the paper.