Papers

Beyond Reward Hacking: Proxy Divergence Across Four Layers of a Staged Humanoid Learning Pipeline

This preprint introduces a four-layer 'proxy audit' for humanoid reinforcement-learning pipelines, treating reward terms, curriculum gates, evaluation metrics, and reference motions as failure-prone proxies. The authors demonstrate measured failures and corrections in a simulated 1.91 m humanoid trained with PPO across four stages and 13,500 iterations on one laptop GPU; among the results, an averaged-error curriculum gate advanced on schedule while the target skill was missing, and a phase-aliased push test rated 0.5 m/s as more dangerous than 2.0 m/s. The low-cost reformulations matter because they can make simulated robot training results more trustworthy without retraining.