Policy OptimizationExpert
FERPO: Forward Entropy-Regularized Policy Optimization
This arXiv preprint introduces FERPO, an on-policy maximum-entropy reinforcement-learning algorithm that updates a policy from critic values without taking critic action derivatives. It roughly matches REPPO on DMC and ManiSkill while reaching high scores with fewer environment interactions, and at K=16 its uncached actor-critic updates are 6.9% faster than REPPO, or 22.2% faster with cached actor updates. The method matters for massively parallel robot-policy training because it avoids unreliable learned action gradients and improves wall-clock and sample efficiency.