FERPO: Forward Entropy-Regularized Policy Optimization
This arXiv preprint introduces FERPO, an on-policy maximum-entropy reinforcement-learning algorithm that updates a policy from critic values without taking critic action derivatives. It roughly matches REPPO on DMC and ManiSkill while reaching high scores with fewer environment interactions, and at K=16 its uncached actor-critic updates are 6.9% faster than REPPO, or 22.2% faster with cached actor updates. The method matters for massively parallel robot-policy training because it avoids unreliable learned action gradients and improves wall-clock and sample efficiency.
Can an on-policy actor be improved using critic values alone—without differentiating the critic with respect to actions—by constructing and forward-KL fitting an entropy- and KL-regularized target action distribution?
State-of-the-art on-policy continuous-control methods such as REPPO update actors by differentiating a learned soft critic with respect to actions. However, critics are trained to predict values, and accurate value predictions do not guarantee accurate action derivatives, so pathwise policy updates can be biased or unstable, especially under changing on-policy data distributions.
REPPO uses pathwise gradients of a learned soft critic with a KL fallback, while PPO uses clipped likelihood ratios and TRPO uses explicit KL constraints. Off-policy methods such as MPO, AWR, and AWAC fit policies to replay data with exponential advantage weights. Pathwise on-policy methods inherit critic-gradient error; replay-based methods can reweight states and depend on stored data rather than fresh on-policy rollouts.
FERPO first minimizes the entropy- and KL-regularized policy-improvement Lagrangian over unrestricted action densities, yielding a target proportional to exp(Q/(τ+λ)) times the rollout policy density raised to λ/(τ+λ). It then fits a Gaussian actor to that target by minimizing forward KL, estimated with K rollout-policy action samples and self-normalized importance sampling. The temperature τ and trust-region coefficient λ are updated from an entropy gap and from the log normalized effective sample size of the importance weights. The critic uses REPPO-style multi-step HL-Gauss value learning with auxiliary self-prediction. A cached variant reuses sampled actions and Q-values across actor epochs, and a state-value variant scores candidates via one-step simulation plus a learned soft value function.
On 22 DMC tasks, final IQM return was 897.7 ± 6.7 for FERPO versus 891.0 ± 6.4 for REPPO, 890.6 ± 12.8 for FastTD3, 542.8 ± 30.2 for PPO-Brax, and 309.8 ± 9.4 for PPO. On 8 ManiSkill tasks, IQM success was 95.67 ± 1.83% for FERPO versus 95.52 ± 1.85% for REPPO and 84.61 ± 9.49% for PPO. On 4 MuJoCo Playground G1/T1 locomotion tasks, REPPO reached 34.63 ± 0.17 versus 33.22 ± 0.28 for FERPO, with FERPO trailing mainly on G1 tasks and similar or slightly higher on T1 tasks. At K=16, uncached FERPO reduced combined actor-and-critic update time by 6.9% versus REPPO; caching added a further 16.4% reduction, for 22.2% total. Peak GPU memory was 22.4% higher for uncached FERPO and 25.6% higher for cached FERPO on CheetahRun. In a G1 ablation, increasing K from 8 to 16 raised mean return from 34.32 ± 0.18 to 34.81 ± 0.24; disabling entropy regularization at K=16 reduced mean return to 30.32 ± 0.27, a 4.49-point drop. Cached G1 training at K=8 reached 33.58 ± 0.40 versus 34.03 ± 0.21 uncached but reduced training time from 68.4 to 55.7 minutes, an 18.6% saving. On a bimodal cost toy task, SNIS with K=1024 rollout samples recovered the correct update direction in only 26.4% of batches, while proposal-scaled sampling reached 98.9% for κ=2 and 99.9% for κ=3. The state-value variant on AcrobotSwingupSparse used 17× more simulator transitions per rollout step and was 8.4% slower per iteration after optimization, while showing better sparse-reward returns.
The authors list critic-value errors, finite-sample importance-sampling errors, restricted Gaussian policies, and higher GPU memory use as limitations. FERPO also remained below REPPO on the G1/T1 aggregate at equal environment interactions, with the gap concentrated on G1 tasks. Reported results are simulation-only with no real-robot validation, and the state-value variant trades a 17× increase in simulator transitions for improved sparse-reward learning. The work is a preprint and not yet peer-reviewed.
Robot-learning teams that train policies in massively parallel simulators for legged locomotion or manipulation could test FERPO as a drop-in on-policy alternative to REPPO/PPO to reduce sample and wall-clock cost. Industrial robot-arm and legged-robot producers using sim-to-real pipelines would be the likely eventual users, but adoption at scale is 1–3 years away and would require real-robot transfer validation, support for more expressive policies beyond Gaussian actors, and acceptable GPU-memory overhead. Simulation-only research use with the public code is possible now.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.