Game-Guided Skill Discovery through Self-Play for Playable Agent Control

Can self-play in simple 1v1 competitive games discover a compact repertoire of motor skills that is semantically distinct, directly playable by humans, and reusable for unseen downstream tasks?

Building compact motor-skill interfaces for high-degree-of-freedom agents is hard because the skills must be semantically distinct, interpretable, and expressive enough for direct human control. Existing unsupervised discovery methods optimize skill distinctiveness, but in complex agents there are many ways skills can be distinguishable without being behaviorally meaningful or useful.

Previous unsupervised skill discovery relied on mutual-information objectives such as DIAYN and DADS, or Wasserstein dependency measures such as METRA/WDM. Some later methods added language, demonstration, or reference-motion guidance. Self-play has also been used for motor behavior, but usually as a monolithic game-playing policy or with predefined reference-based skills. The core shortcoming is that distinctiveness alone does not guarantee interpretable or reusable skills, and language or motion guidance can be costly or difficult to scale.

GGSD trains a two-level policy through self-play against a pool of historical checkpoints. A high-level policy selects one of five or six discrete skill codes every k=10 steps using the full game state; a low-level policy maps that code to motor actions using only the controlled agent's proprioception, and receives a mix of game reward and a mutual-information reward from a proprioceptive discriminator. High-level entropy discourages skill collapse. Four 1v1 games—AntSumo, AntFencing, FrankaAirHockey, and G1Boxing—provide lightweight game-reward guidance. After training, humans replace the high-level policy and compose the fixed low-level skills; skill transitions create combo behaviors beyond the individual primitives.

In self-play, the final policy's combined win-and-draw rate stayed above 50% against every evaluated checkpoint; for G1Boxing it exceeded 80% win rate against every past checkpoint except the 70k checkpoint itself. Four human players won only 10% of games against the trained AntSumo policy—2 wins in 20 games. GGSD achieved the highest semantic-diversity scores across all embodiments, about 2x higher than the strongest baseline on Ant and G1, and 28% higher on Franka, compared with DIAYN, DADS, and METRA. In human-play tests, AntMaze achieved 100% success with 82.7s average time, AntCubePush achieved 100% success with 62.2s, and G1Maze achieved 84% success with 22.0s. The evaluation used five participants, including two authors, each performing five trials per task after at most five minutes of practice; training was done in Isaac Lab and evaluation in MuJoCo.

The authors note the skill repertoire is bounded by the game: simple games may demand only a narrow set of behaviors, so a boxing-trained G1 is unlikely to acquire dexterous manipulation. Evaluation is simulation-only—trained in Isaac Lab and evaluated in MuJoCo—with no real-robot or hardware transfer reported. The human study is small, includes authors, and may be subject to familiarity bias. Only four games and three embodiments are tested, and no code release is mentioned. Training uses 4,096 parallel environments and 70k updates with a pool of frozen opponents, implying nontrivial compute, though the paper does not quantify cost.