Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning
This preprint introduces MFPG-PPO, an algorithm that uses abundant low-fidelity simulator data as a control variate to reduce variance in scarce high-fidelity policy-gradient updates. On the hardest simulated locomotion task it matches PPO trained with 16 times more high-fidelity data, and on a physical Franka arm it reaches 91.7% success with only 4 real episodes per update, versus 55.0% for PPO on high-fidelity data alone. If validated, the approach could make on-policy robot learning much more sample-efficient in real-world deployments.
Can the multi-fidelity policy-gradient control-variate framework be extended to modern actor-critic PPO learning while preserving cross-fidelity correlation and avoiding variance inflation, for both GPU-parallel simulation and physical robots?
Standard on-policy reinforcement learning becomes unstable when only a few expensive high-fidelity rollouts are available per policy update because gradient estimates are too noisy; existing methods that pool or correct low-fidelity simulator data directly optimize biased objectives and therefore fail when low-fidelity policies do not transfer well.
Previous approaches include pooling HF and LF data (co-training), correcting LF rewards for dynamics mismatch (DARC, PAR), training only on LF data and transferring, and a multi-fidelity control variate limited to REINFORCE on small simulation tasks. These methods either depend on transfer quality or lack a PPO/actor-critic formulation that keeps the paired HF and LF statistics sufficiently correlated and prevents poorly estimated control-variate coefficients from amplifying variance.
MFPG-PPO collects three transition batches per iteration: high-fidelity rollouts, coupled low-fidelity rollouts that share action noise and periodically reset to recorded high-fidelity states, and abundant uncoupled low-fidelity rollouts. It builds separate PPO surrogate statistics and critics for the three batches, fits the control-variate coefficient on paired statistics (per step in simulation, episode sums on the robot), and applies the correction only when statistical and empirical variance-reduction checks pass; otherwise it falls back to HF-only PPO. A budget-aware variant solves a two-fidelity Monte Carlo allocation problem online to divide a fixed sampling budget between HF and LF rollouts.
Across 16 task-budget settings in Isaac Lab locomotion, MFPG-PPO-FR outperformed PPO HF-Only in 14 of 16 settings and was never distinguishably worse; MFPG-PPO-BA outperformed PPO HF-Only in all 16 settings. On the hardest task, H1 Rough, PPO HF-Only collapsed to near-zero return at the 1/128 and 1/256 HF budgets, while MFPG-PPO-FR reached a mean return around 40 at 1/256 and matched or exceeded PPO HF-Only with 16 times more HF data at the three smallest budgets. On the real Franka cube-picking task with 3 seeds, 80 iterations of 4 episodes, MFPG-PPO-FR reached 91.7% final success versus 55.0% for PPO HF-Only from a 40.0% warm start; each MFPG seed was 90-95% and each PPO seed was 30-70%. Ablations showed that removing coefficient re-fitting and monitoring produced variance ratios above 1000 in many iterations and caused training collapse.
The authors state that re-synchronizing coupled LF rollouts while uncoupled LF rollouts evolve freely can introduce bias in the estimator, and removing bias while keeping variance fully reducible remains open. Real-robot evidence is limited to one Franka cube-picking task with 3 seeds, 80 training iterations, and 60 final evaluation episodes. Simulation results cover locomotion only, with deterministic evaluation over 2048 episodes. The method assumes a reparameterizable policy, shared state/action spaces, and an LF simulator that can reset to recorded HF states; it also adds substantial LF sampling and critic-training overhead. Policies are warm-started with LF pre-training, and monitoring thresholds/cost ratios may require per-platform tuning.
Robot learning teams at humanoid, legged-robot, and manipulation companies such as Unitree, ANYbotics, and Franka-arm users could adopt this to fine-tune simulation-pretrained policies on very scarce real-robot data without demonstrations. Adoption is likely feasible in 1-3 years if teams can build coupled low-fidelity simulators that reset to real states and integrate monitoring into existing PPO pipelines; larger real-robot validation and standardized sim-to-real tooling are needed for production deployment. The budget-aware variant is also relevant to cloud robotics or simulation service providers optimizing per-sample costs.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.