報酬ハッキングの先へ：人型ロボット学習パイプライン4層の代理指標乖離を分析

Abstract

A reinforcement-learning (RL) pipeline for a legged robot is assembled from proxies. A reward stands in for intended behaviour, a curriculum gate stands in for competence, an evaluation statistic stands in for robustness, and a reference motion stands in for an achievable skill. The traditional view treats only the first of these as optimised against, and so locates specification failure (reward hacking) in the reward alone. I argue that all four are proxies in the same formal sense, that each has a characteristic divergence mechanism, and that each admits a reformulation that closes it. For every layer I state the traditional formulation, derive the condition under which it diverges from its target, and give the alternative: first-order (L1) costs where quadratic kernels are flat, peak and outcome statistics where curriculum gates average, gate reachability and information checks, deterministic and phase-desynchronised evaluation, curriculum state treated as part of the model, feasibility-first reference design with residual feed-forward, and function-preserving input widening that lets one policy grow instead of being retrained. The arguments are illustrated by measurements from one continuous lineage of a PPO policy for a simulated 1.91 m humanoid, grown over four stages and 13,500 iterations on a single laptop GPU. Among them, a curriculum gate built on averaged error advanced at its rate limit on every check while the skill it gated was absent, and a batched push test whose synchronised resets aliased the gait phase ranked a 0.5 m/s push as more dangerous than a 2.0 m/s one.

Index Terms:

reinforcement learning, reward design, curriculum learning, evaluation methodology, motion imitation, humanoid robots, continual learning

I Introduction

GPU-parallel simulation has made it routine to train a locomotion policy in hours on one machine [1, 2]. The resulting pipeline has a standard shape: a weighted sum of shaped reward terms, a performance-gated curriculum over terrain and commands, batched evaluation in the simulator, and, for skills beyond locomotion, a tracking reward towards a reference motion [3]. Each of these components replaces something physical with something computable.

The literature on specification gaming [4, 5, 6, 7] studies the gap between a proxy and its target, but in robot learning it is applied almost exclusively to the reward. The other components are treated as plumbing: a curriculum threshold is a tuning constant, an evaluation script is assumed to measure what it reports, and a reference motion is assumed to be achievable. My thesis is that this traditional view is incomplete. A curriculum gate is optimised against just as a reward is, because the policy that passes it is the one the gate selects. An evaluation statistic is a proxy whose bias can be created by the harness itself, with no optimisation at all. A reference motion is a proxy for a skill, and it can be wrong in a way that no reward design can repair.

I-A Contributions

1. A four-layer formulation of proxy divergence (reward, curriculum, measurement, reference), with, for each layer, the traditional formulation, the condition under which it diverges, and a reformulation (Sections IV–VII, summarised in Table III).
2. Curriculum gates as estimators: a dilution bound showing that gates on error averaged over joints and time cannot reject large localised failures, a reachability condition, an information argument for counting gate refusals, and the observation that curriculum state belongs to the model, not the environment (Section V).
3. Two systematic evaluation biases that do not shrink with sample size: outcomes manufactured by exploration noise, and phase aliasing from synchronised resets, with corrected estimators (Section VI).
4. Feasibility-first reference design: why tracking an infeasible reference converges to a qualitatively different behaviour regardless of reward shaping, and a static test, a correction and a residual action that avoid it (Section VII).
5. Function-preserving policy growth through input widening and split authority, as an alternative to retraining from scratch whenever the observation changes (Section VIII).

II Related Work

RL for legged and humanoid robots. Deep RL now produces deployed locomotion and whole-body controllers [8], including zero-shot humanoid walking [9], agile whole-body skills with sim-to-real alignment [10], and general motion tracking [11]. Most of these pipelines share the structure analysed here, built on GPU-parallel simulators such as Isaac Lab [2].

Reward design and specification gaming. Practical locomotion rewards are weighted sums of heuristic terms [1], and recent work automates their design with language models [12]. Skalse et al. [4] showed that unhackable proxy rewards are very restricted; Pan et al. [5] found that more capable agents exploit misspecified rewards more, sometimes through abrupt phase transitions; Gao et al. [6] measured scaling laws for over-optimising a proxy; and Laidlaw et al. [7] defined reward hacking through the correlation between proxy and true reward under a base policy. All of this work treats the reward as the proxy. I extend the same lens to curriculum gates, evaluation statistics and reference motions.

Curricula. In locomotion, terrain and command levels are raised when agents perform well [1], and adaptive command curricula are central to fast legged locomotion [13]. The gate that decides promotion is treated as a tuning constant; I analyse it as an estimator.

Evaluation methodology. Agarwal et al. [14] and Patterson et al. [15] showed that deep-RL conclusions are fragile to seeds, aggregate statistics and experimental design. Those are largely variance effects; the biases studied here are systematic and do not shrink with sample size.

Imitation and residual control. Tracking rewards towards reference motions [3] and adversarial motion priors [16] remain the basis of humanoid skill learning [10, 11]. References retargeted from human motion capture approximately inherit the human’s dynamic feasibility; hand-authored references do not. Residual formulations learn a correction on top of a fixed controller [17], and ASAP applies a learned delta-action model on top of a pretrained policy [10].

Growing and continually training networks. Net2Net [18] introduced function-preserving expansion. Plasticity injection [19] adds a new network whose initial output is exactly zero, so predictions are unchanged at the moment of injection; Neuroplastic Expansion [20] grows the network during training. Loss of plasticity under continual training is now well documented [21], including for on-policy methods such as PPO [22], which is the regime of a policy fine-tuned across many stages.

III Setting

The robot is a simulated 1.91 m, 69.35 kg humanoid built directly from its CAD assembly. The policy controls 11 joints (hip pitch, hip roll, knee, ankle pitch and ankle roll on each leg, and a waist yaw); 10 arm and neck joints are on position drives outside the policy. It is a Gaussian MLP (three hidden layers of 128 ELU units, separate critic) trained with PPO [23] and GAE [24] (\gamma=0.99, \lambda=0.95, clip 0.2, adaptive learning rate at target KL 0.01) on 2048 environments \times 24 steps per iteration. Actions are joint offsets, q^{\ast}=q_{0}+0.5\,a, tracked by implicit PD drives at 50 Hz over 200 Hz physics. Observations include base velocities, projected gravity, commands, joint states, the previous action, and a 187-ray terrain height scan. Terrain is procedurally generated under a promotion curriculum, and random velocity pushes are applied. All training used a single RTX 5070 Ti laptop GPU.

I write r_{t}=\sum_{i}w_{i}r_{i}(s_{t},a_{t}) for the reward, k for a curriculum level that indexes a task distribution d_{k}, and J(\theta;d)=\mathbb{E}_{\xi\sim d}\,\mathbb{E}_{\pi_{\theta}}\left[\sum_{t}\gamma^{t}r_{t}\right] for the objective.

Definition 1

A proxy is a computable quantity m used to steer or judge learning in place of a physical target T. A proxy divergence is a policy, or a population of policies, for which m reports success while T is absent.

IV Reward Layer

IV-A Second-order kernels permit first-order drift

Traditional. Command tracking uses a Gaussian kernel, r_{\mathrm{trk}}=\exp(-\|\mathbf{c}-\mathbf{v}\|^{2}/\sigma^{2}), chosen because it is bounded and smooth [1].

Divergence. Consider a zero command and a scalar drift speed v\geq 0. Let g(v) collect every other effect of drifting on return, for instance that stepping slightly forward is easier to balance than standing still.

Proposition 1

If g is differentiable at 0 with g^{\prime}(0)>0, then f(v)=\exp(-v^{2}/\sigma^{2})+g(v) is not maximised at v=0. If instead the tracking term is augmented with an L1 cost -\lambda|v|, then v=0 is a local maximiser whenever g^{\prime}(0)<\lambda.

Proof:

The kernel has zero derivative at 0, so f^{\prime}(0)=g^{\prime}(0)>0 and f increases to the right of 0. With the L1 term, the right derivative at 0 on the domain v\geq 0 is g^{\prime}(0)-\lambda<0, so 0 is a local maximiser. ∎

The kernel’s loss near zero is 1-e^{-v^{2}/\sigma^{2}}\approx v^{2}/\sigma^{2}, second order, so any first-order incentive wins. With \sigma=0.5 the kernel still pays 0.85 at 0.2 m/s of drift, and a policy trained this way crept forward on every zero command.

Reformulation. Pair every saturating kernel whose target is a rest state with a first-order cost active only in that state. An L1 base-velocity cost of weight 1.5 on zero commands reduced measured drift to 0.01–0.04 m/s.

IV-B Unconstrained coordinates settle on their bounds

Traditional. Penalty terms are added for joints whose motion is undesirable and omitted elsewhere, often by inheriting a reference configuration from another robot.

Divergence. If the reward has no term in a coordinate q_{j}, its value is set by whatever residual, incidental dependence g(q_{j}) the rest of the objective has. If g is monotone on the feasible interval [q_{\min},q_{\max}], its maximiser is a bound, where the joint limit supplies the only opposing force. A quadratic regulariser -\mu(q_{j}-q_{j}^{0})^{2} makes the objective strictly concave in q_{j} and moves the maximiser into the interior whenever |g^{\prime}|<2\mu\,(q_{\max}-q_{j}^{0}).

On rough terrain the waist penalty was removed, because the configuration it came from belongs to a robot that turns with hip yaw. The waist then sat at its +45.00^{\circ} limit for 100 % of steps with zero variance, while every logged statistic improved (reward +27.4 against +25.7, falls 2.8 % against 5.3 % for the corrected policy).

Reformulation. Every actuated coordinate must be in the support of the reward, either through a regulariser or a commanded target, and inside a limit barrier. In the final configuration the waist is not penalised but commanded: a target in \pm 0.7 rad, zero half the time, is observed and rewarded by \exp(-(q_{w}-q_{w}^{\ast})^{2}/0.2^{2}), which subsumes the regulariser.

V Curriculum Layer

Traditional. A level k is raised when a statistic \bar{m} of recent episodes crosses a threshold, k\leftarrow k+1 if \bar{m}\geq\tau (or \bar{e}<\tau for an error), usually with a rate limit.

V-A Dilution

Let a reference-tracking skill be judged by tracking error averaged over |\mathcal{J}| joints and the T steps of the motion, \bar{e}=\frac{1}{|\mathcal{J}|T}\sum_{j,t}|e_{j}(t)|. Suppose one joint misses by e^{\ast} during a fraction \rho of the motion and every other error is at most \epsilon.

Proposition 2

\bar{e}\leq\rho e^{\ast}/|\mathcal{J}|+\epsilon, so the gate \bar{e}<\tau accepts every such failure with e^{\ast}<|\mathcal{J}|(\tau-\epsilon)/\rho.

The bound is direct. With ten tracked joints, \tau=0.25 rad and \rho\approx 0.3, the gate accepts misses up to several radians, i.e. it cannot reject any physically possible failure of the principal joint. In the lineage, a gate of exactly this form accepted a single-support leg-lift motion in which the lifting hip missed its target by 59^{\circ}\approx 1.03 rad; the logged mean error was 0.06–0.07 rad (Fig. 1). A no-fall criterion did not help, because the degenerate solution, keeping both feet down, never falls.

Reformulation. Gate on the peak, e_{\infty}=\max_{t}\max_{j\in\mathcal{J}_{\mathrm{act}}}|e_{j}(t)|, over the joints the motion actually moves, averaged over episodes; and conjoin it with an outcome predicate measured at the contact level (here, the lifted foot carries less than 1 N for a minimum fraction of the swing and reaches a minimum height relative to what the reference produces).

V-B Information and refusal

Let G\in\{0,1\} be the gate’s decision and C the competence it should detect. If the gate passes at every check, H(G)=0 and therefore I(G;C)=0: the gate transmits no information about competence. A gate that advances at exactly its rate limit is observationally equivalent to no gate. In Fig. 1 the seven raises came at intervals of 41 to 44 iterations against a rate limit of one episode (\approx41.7 iterations), with zero refusals.

Reformulation. Log the refusal count as a first-class diagnostic, and treat a curriculum with no refusals as unvalidated.

V-C Reachability

A threshold is useful only if some policy can meet it: \sup_{\pi}\mathbb{E}_{\pi}[m]\geq\tau. Executing the reference open-loop is itself a policy \pi_{\mathrm{ref}}, so \mathbb{E}_{\pi_{\mathrm{ref}}}[m]\geq\tau is a sufficient condition that costs one rollout. When it fails, the gate may be unreachable. In the lineage, an air-time requirement of 50 % of the swing exceeded the \approx30 % that the reference itself achieves at half amplitude (\approx59 % at full), and a starting amplitude of 0.3 produced a lift too small to clear the ground once the support leg tilted.

Reformulation. Set \tau\leq\mathbb{E}_{\pi_{\mathrm{ref}}}[m] at every curriculum level before training.

V-D Curriculum state is model state

Traditional. A checkpoint stores network weights, optimiser moments and an iteration counter. Curriculum levels live in the environment.

Divergence. Fine-tuning from a checkpoint optimises J(\theta;d_{k_{0}}) rather than J(\theta;d_{k}): the objective silently changes. Skills trained only at high levels receive no gradient to maintain them, and statistics compared across stages are expectations under different distributions. In the lineage, a speed curriculum that had reached 1.7 m/s restarted at 1.0 m/s in the next stage and was not raised again for 2000 iterations; terrain level fell from 5.6 to 3.5 at the same boundary.

Reformulation. Checkpoint the tuple (\theta,\text{optimiser},k), and compare stages only under a fixed evaluation distribution d_{\mathrm{eval}}.

VI Measurement Layer

VI-A Outcomes manufactured by exploration noise

Traditional. Outcome statistics are logged from training rollouts of the stochastic policy a=\mu_{\theta}(s)+\sigma\varepsilon.

Divergence. For a thresholded outcome m=\mathbf{1}[h(s)>h_{0}], \mathbb{E}_{\varepsilon}[m] can be positive when the deterministic rollout has h<h_{0} everywhere, because noise alone reaches the threshold. The estimate converges, with more samples, to the noise’s contribution, not the policy’s. In the lineage a foot-height metric read 61–69 % of target during training while the deterministic policy produced none: exploration noise of \approx40∘ on the ankle targets (against \approx6∘ at the hips) tipped a foot carrying \approx360 N onto its toe, raising its centre of mass.

Reformulation. Report outcomes for \mu_{\theta}, and define outcomes by predicates that noise cannot satisfy cheaply: here, height counts only while the foot’s contact force is below 1 N.

VI-B Phase aliasing in batched evaluation

Traditional. A disturbance test resets N parallel environments together, waits a fixed settling time t_{0}, applies the disturbance, and reports the failure fraction \hat{p}=\frac{1}{N}\sum_{i}f_{i}.

Divergence. A gait is approximately periodic with period P, so failure depends on the phase \phi at which the disturbance arrives, p(m\,|\,\phi). With synchronised resets, every robot has \phi_{i}\approx t_{0}\bmod P, and

\hat{p}\xrightarrow{N\to\infty}p(m\,|\,\phi_{0})\neq\bar{p}(m)=\frac{1}{P}\int_{0}^{P}p(m\,|\,\phi)\,d\phi.

The bias is independent of N: more robots make a wrong number more precise. In the lineage this produced a fall rate of 88 % for a 0.5 m/s push from one side against 50 % for 2.0 m/s from the same side, together with a large and entirely spurious left–right asymmetry.

Reformulation. Draw each robot’s disturbance time independently, t_{i}\sim\mathcal{U}[t_{0},t_{0}+W]. The induced phase density is n/W or (n+1)/W with n=\lfloor W/P\rfloor, so it differs from the uniform 1/P by at most 1/W, and is exactly uniform when W is a multiple of P. With W=1 s, longer than one step cycle, the magnitude series became monotonic (Table I) and the asymmetry disappeared; lateral and sagittal pushes then had near-equal fall rates (30 % against 26 % while walking).

VI-C Quantifier errors in aggregation

A claim of the form “the policy survives magnitude m in every direction” is a universal quantifier and must be computed with a minimum over directions. A harness that computed the largest magnitude with zero falls in any direction (an existential) reported a clean result while another direction had a 38 % fall rate. The statistic’s quantifier must match the sentence it is used to support.

VII Reference Layer

Traditional. A skill is specified by a reference trajectory q_{\mathrm{ref}}(t) and learned by maximising \exp(-\|q-q_{\mathrm{ref}}\|^{2}_{M}/\sigma^{2}) alongside a large failure penalty, with reward terms added when the result is unsatisfactory [3].

Divergence. Let \mathcal{F} be the set of trajectories the robot can execute without falling. If q_{\mathrm{ref}}\notin\mathcal{F}, a policy cannot track it, and the large failure penalty drives the optimum to a trajectory in \mathcal{F} that is close to q_{\mathrm{ref}} in the reward’s metric M. Nothing requires that projection to share the reference’s qualitative structure (its contact sequence). Additional shaping terms change M and therefore which feasible trajectory is chosen, but they cannot make the infeasible one feasible.

A static necessary condition for a single-support phase is that the centre of mass projects into the support polygon, p^{xy}_{\mathrm{com}}(q_{\mathrm{ref}}(t))\in\mathcal{S}(t). In the lineage, hand-authored single-support leg-lift references satisfied every joint limit, yet in the lifted pose the centre of mass lay 0.21 m beside the support foot. Seven reward, initialisation and exploration changes were tried against that reference; each moved its own shaped term, and none produced single support in the deterministic policy.

Reformulation, part 1: feasibility first. Before any reward design, check the static condition at every keyframe and repair violations kinematically. Here a support-leg roll \theta, with ankle roll -\theta to keep the sole flat, moves the centre of mass at a measured \kappa\approx 0.0095 m per degree, so the required shift is \theta\approx d/\kappa=0.21/0.0095\approx 22^{\circ}; at 20^{\circ}, within the 27^{\circ} limit, the residual offset is 0.02 m. Then execute the repaired reference open-loop as a dynamic check: if it reproduces the intended contact sequence for part of the motion, the reference is a sound target.

Reformulation, part 2: residual feed-forward. Rather than asking the policy to rediscover the reference from reward, feed it forward:

q^{\ast}=q_{0}+0.5\,a+r(t),

with r(t)=0 when no reference is active. Then a=0 reproduces the (feasible) reference, exploration is centred on it, and the learning problem changes from discovery to regulation, as in residual RL [17, 10]. The action space is unchanged, so existing checkpoints load without modification. Measured on the repaired leg-lift reference, open-loop feed-forward alone held the foot clear of the ground for the whole motion, whereas a policy required to discover the support-leg shift from reward held that joint at -1 to -3^{\circ} against a -20^{\circ} target.

VIII Policy Growth

VIII-A Function-preserving input widening

Traditional. When a new capability requires new observations, the input dimension changes, the checkpoint no longer loads, and training restarts from scratch.

Reformulation. Append the new inputs \mathbf{x}_{\mathrm{new}}\in\mathbb{R}^{m} after the old ones and pad the first-layer weights of actor and critic with zeros,

W^{\prime}=\begin{bmatrix}W&\mathbf{0}\end{bmatrix},\qquad W^{\prime}\begin{bmatrix}\mathbf{x}\\
\mathbf{x}_{\mathrm{new}}\end{bmatrix}=W\mathbf{x}.

Actions and values are preserved exactly, so advantage estimates are not disturbed at the handover. This is the input-side counterpart of plasticity injection [19] and Net2Net [18], which preserve a network’s function while adding capacity; here the added capacity is a new observation channel. Unlike a fully zero-initialised layer, the padded block still receives gradient, \partial L/\partial W_{\mathrm{new}}=\boldsymbol{\delta}\,\mathbf{x}_{\mathrm{new}}^{\top}, which is non-zero whenever the new inputs vary, so they are learnable from the first update. Adam’s moment estimates have the old shape and are discarded; the adaptive-KL learning rate absorbs the resulting transient. The exploration-noise parameterisation must also be read from the checkpoint, since \log\sigma and \sigma are stored under different keys. Widening 253 inputs to 274 reproduced the parent’s actions to within 10^{-5}.

VIII-B Split authority with observed exogenous motion

Traditional. Joints outside the policy are held fixed in training, so the training distribution of their state is a point mass at rest. Any later motion of those joints is a distribution shift. In the lineage, a policy trained with fixed arms fell when an arm was held out sideways: tilt grew from 0.24 to 0.97 over two seconds, because the sustained lateral moment loads the hip-roll joint, already the most heavily loaded in walking.

Reformulation. Drive the non-policy joints during training with an exogenous process (rest poses, keyframe sequences at randomised speed, random poses within soft limits, all smoothly ramped), and include their positions and velocities, but not the command generating them, in the observation. The policy is then a function of the exogenous joint state alone, so at deployment any source of motion, whether a script, a teleoperator or another controller, presents identically. Joints the policy does own are steered by commanded targets rather than external scripts, as for the waist in Section IV.

VIII-C Empirical lineage

Table II and Fig. 2 summarise one parameter lineage grown this way. S1 trained from scratch with standing, observed upper-body motion and a torso target; S2 added a forward-speed curriculum to 2 m/s; S3 added fast upper-body keyframe sequences that drive the torso target; S4 widened the input with a 21-dimensional lower-limb reference (current and 0.2 s ahead, plus an activity flag). In replay, the S1 policy drifts 0.01–0.04 m/s on a zero command and holds a +34^{\circ} torso command at 28–36∘. S3 reduced peak tilt during the fastest arm sequence from 0.34 to 0.21 and base drift from 0.37 m to 0.14 m. The four stages took \approx14.2 wall-clock hours and 6.6\times 10^{8} samples.

IX Traditional Versus Proposed: Summary

Table III collects the reformulations. Read as a checklist, it is a proxy audit: for each quantity that steers or judges learning, ask which column of the table it is in.

X Discussion

Why the reward is not special. Under the traditional view the reward is the only optimised quantity, and so the only one that can be hacked. Of the divergences above, the curriculum dilution and the noise-manufactured outcome were selected for by optimisation even though neither quantity is a reward; phase aliasing and the quantifier error needed no optimisation at all; and an infeasible reference is a specification error that no optimiser can repair. What unites them is that each reported success through a channel the engineer had already decided to trust. Validation effort should follow trust, not only optimisation pressure.

Cost. None of the reformulations requires additional training. They cost one open-loop rollout per reference and curriculum level, one deterministic replay per reported outcome, a randomised timing schedule in the evaluation harness, and a few lines in the checkpoint format.

Scope. All results are simulated, from a single training seed. The robot’s mass model derives from solid-material density, and actuator limits are those of the simulation rather than the intended motors.

XI Conclusion

The traditional RL pipeline for legged robots treats the reward as the one place where a proxy can diverge from its target. I have argued, with formal conditions and measured instances, that curriculum gates, evaluation statistics and reference motions are proxies of the same kind, and that each has a precise reformulation: first-order costs at rest states, peak and outcome-grounded gates that are reachable and informative, curriculum state saved with the model, deterministic and phase-desynchronised evaluation, and feasibility-first references with residual feed-forward. Combined with function-preserving input widening and split authority, these let a single humanoid policy grow across stages without retraining, while keeping every number that steers it honest.

Reproducibility

All figures are regenerated from the original training logs by the scripts accompanying this manuscript. Environment configurations and reward terms are available from the author on request.

Acknowledgment

This work was carried out independently, without institutional affiliation or external funding. It builds on NVIDIA Isaac Sim, IsaacLab and the rsl_rl implementation of PPO.

References

1. [1] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Proc. Conf. Robot Learning (CoRL), 2021.
2. [2] M. Mittal, P. Roth et al., “Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning,” arXiv:2511.04831, 2025.
3. [3] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne, “DeepMimic: Example-guided deep reinforcement learning of physics-based character skills,” ACM Trans. Graphics, vol. 37, no. 4, 2018.
4. [4] J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger, “Defining and characterizing reward hacking,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.
5. [5] A. Pan, K. Bhatia, and J. Steinhardt, “The effects of reward misspecification: Mapping and mitigating misaligned models,” in Proc. Int. Conf. Learning Representations (ICLR), 2022.
6. [6] L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” in Proc. Int. Conf. Machine Learning (ICML), PMLR vol. 202, pp. 10835–10866, 2023.
7. [7] C. Laidlaw, S. Singhal, and A. D. Dragan, “Correlated proxies: A new definition and improved mitigation for reward hacking,” in Proc. Int. Conf. Learning Representations (ICLR), 2025.
8. [8] C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Martín-Martín, and P. Stone, “Deep reinforcement learning for robotics: A survey of real-world successes,” Annu. Rev. Control, Robot., Auton. Syst., vol. 8, pp. 153–188, 2025.
9. [9] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath, “Real-world humanoid locomotion with reinforcement learning,” Science Robotics, vol. 9, no. 89, 2024.
10. [10] T. He et al., “ASAP: Aligning simulation and real-world physics for learning agile humanoid whole-body skills,” in Proc. Robotics: Science and Systems (RSS), 2025.
11. [11] Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu, “BeyondMimic: From motion tracking to versatile humanoid control via guided diffusion,” arXiv:2508.08241, 2025.
12. [12] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” in Proc. Int. Conf. Learning Representations (ICLR), 2024.
13. [13] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, “Rapid locomotion via reinforcement learning,” Int. J. Robotics Research, vol. 43, no. 4, pp. 572–587, 2024.
14. [14] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” in Advances in Neural Information Processing Systems (NeurIPS), 2021.
15. [15] A. Patterson, S. Neumann, M. White, and A. White, “Empirical design in reinforcement learning,” J. Machine Learning Research, vol. 25, pp. 1–63, 2024.
16. [16] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa, “AMP: Adversarial motion priors for stylized physics-based character control,” ACM Trans. Graphics, vol. 40, no. 4, 2021.
17. [17] T. Johannink et al., “Residual reinforcement learning for robot control,” in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2019.
18. [18] T. Chen, I. Goodfellow, and J. Shlens, “Net2Net: Accelerating learning via knowledge transfer,” in Proc. Int. Conf. Learning Representations (ICLR), 2016.
19. [19] E. Nikishin, J. Oh, G. Ostrovski, C. Lyle, R. Pascanu, W. Dabney, and A. Barreto, “Deep reinforcement learning with plasticity injection,” in Advances in Neural Information Processing Systems (NeurIPS), 2023.
20. [20] J. Liu, J. Obando-Ceron, A. Courville, and L. Pan, “Neuroplastic expansion in deep reinforcement learning,” in Proc. Int. Conf. Learning Representations (ICLR), 2025.
21. [21] S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton, “Loss of plasticity in deep continual learning,” Nature, vol. 632, pp. 768–774, 2024.
22. [22] A. Juliani and J. T. Ash, “A study of plasticity loss in on-policy deep reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2024.
23. [23] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017.
24. [24] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv:1506.02438, 2015.