ATHLETE: Learning Reactive Humanoid Tennis via Trajectory-Guided Motion Matching

Abstract

Humanoid tennis requires reactive whole-body coordination under rapidly changing ball dynamics, yet collecting task-specific motion-capture demonstrations that cover diverse approach states, contact locations, and timing conditions is costly. Inspired by how humans acquire new skills by combining reusable motor experience with task-specific demonstrations, we present ATHLETE, a framework for learning reactive humanoid tennis from generic locomotion data and ordinary tennis videos. ATHLETE combines LAFAN1 locomotion with video-derived striking motions and uses Trajectory-Guided Motion Matching (TGMM) to construct diverse task-conditioned whole-body references. These references are transformed into a reactive policy through physics-based teacher–student learning with online DAgger distillation, followed by PPO fine-tuning for the tennis task. The resulting policy responds directly to robot and ball observations without requiring reference-motion inputs at deployment. On a real humanoid robot, ATHLETE achieves forehand and backhand hit rates of 90.00\% and 86.67\%, respectively, and enables multi-shot rallies with human players. These results demonstrate that ATHLETE can learn dynamic humanoid tennis skills from reusable locomotion data and video demonstrations. Result videos and code can be found at https://anonymous.4open.science/w/ATHLETE2027/.

†

I INTRODUCTION

Humanoid tennis requires rapid locomotion, precisely timed whole-body strikes, and continuous adaptation to incoming balls. A central challenge is data coverage: a successful policy must handle diverse approach states, contact locations, and arrival times, while collecting task-specific motion-capture demonstrations for all such variations is costly.

Existing works such as LATENT [1] and AdaPT [2] demonstrate that human motion data can provide strong priors for humanoid tennis. However, collecting or reconstructing task-specific tennis motions that sufficiently cover the combinations of locomotion, contact position, and timing remains difficult. At the same time, public motion datasets such as LAFAN1 [3] and AMASS [4] already contain abundant reusable human locomotion, while ordinary tennis videos provide task-specific striking motions. This raises a natural question: can dynamic humanoid tennis be learned by composing reusable locomotion experience with only sparse task-specific striking demonstrations?

We present ATHLETE to answer this question. Our key idea is to treat locomotion and striking as complementary motion sources rather than requiring complete tennis demonstrations. ATHLETE uses Trajectory-Guided Motion Matching (TGMM) to compose generic locomotion with video-derived strikes under task, contact, and transition constraints, producing diverse whole-body approach-and-strike references. These references are then converted through physics-based teacher–student learning into a reactive policy that no longer requires reference-motion input, online motion retrieval, or explicit motion planning at deployment. The resulting controller directly responds to the robot state, ball observations, and task target.

Our contributions are threefold:

• Task-conditioned motion composition from heterogeneous motion sources. We introduce TGMM to combine abundant generic locomotion with sparse video-derived striking motions. By explicitly enforcing approach, contact, and transition constraints, the method generates diverse whole-body tennis references without collecting dedicated tennis motion-capture demonstrations.
• From composed references to reference-free reactive control. We develop a deadline-adaptive teacher–student learning framework that transfers reference-motion and contact-timing supervision into a deployable reactive policy. The final controller operates without reference motions, online motion matching, or an explicit motion planner.
• Real-world humanoid tennis from low-cost motion sources. We validate ATHLETE on a humanoid robot, achieving reactive forehand and backhand returns and multi-shot human–robot rallies, demonstrating that reusable locomotion data and sparse visual demonstrations are sufficient to support dynamic whole-body tennis skills (Fig. ).

II RELATED WORKS

II-A Data Generation for Humanoid Robots

Motion matching retrieves and blends database motions using pose and trajectory features [5], while Learned Motion Matching approximates this pipeline with neural networks to reduce memory usage [6]. Perceptive Humanoid Parkour [7] combines sparse atomic skill demonstrations with locomotion to generate long-horizon references with diverse approach states.

Generative models synthesize motion from history, text, or kinematic constraints [8, 9, 10, 11]; humanoid extensions also condition references on velocity commands, terrain, or multimodal inputs [12, 13, 14, 15]. ATHLETE uses optimized approach trajectories to compose locomotion and strikes under contact and transition constraints, followed by physics simulated tracking.

II-B Humanoid Whole-Body Control

Reference-tracking methods learn whole-body skills from human motions through reference adaptation, dynamics alignment, and large-scale training [16, 17, 18, 19]. Perception-conditioned controllers adapt these skills to terrain and obstacles [7, 20]. Interaction-aware retargeting and imitation incorporate object and scene constraints to support whole-body interaction [21, 22, 23].

For tennis, LATENT [1] learns a correctable latent action space from imperfect motion primitives and composes skills through a high-level policy. AdaPT [2] combines video-derived motions with hierarchical planning and speed-adaptive tracking. TaskNPoint [24] organizes demonstration-guided learning around critical interaction windows and randomized task targets. ATHLETE adopts its motion-tracking and goal-conditioned striking rewards, adds deadline-conditioned phase adaptation, and distills the resulting skills into a single reactive policy.

III Model-Based Reference Motion Generation

We combine locomotion sequences from the LAFAN1 dataset with video-derived striking motions through motion matching to generate task-conditioned tennis striking references; the complete pipeline is illustrated in Fig. 4. We then use simulated rollouts of a Mimic policy to produce dynamically corrected references.

III-A Source Skills and Contact Matching

Our motion library contains 16 locomotion sequences from LAFAN1 [3] and 67 backhand and 97 forehand clips segmented from a 10-minute video of a tennis player’s practice session. We reconstruct human motions from these video clips using GVHMR [25] and retarget them to the humanoid robot using GMR [26], forming the tennis striking skill library (Fig. 2).

We randomly sample a stationary target point \mathbf{b} around the robot as the task target for selecting a striking reference motion. Based on its lateral position, we first restrict the candidates to forehand or backhand clips. We then compute the contact cost between the target and the contact point of each candidate clip:

E_{c}(i)=w_{z}\lvert c_{i,z}-b_{z}\rvert+w_{xy}\lVert\mathbf{c}_{i,xy}-\mathbf{b}_{xy}\rVert_{2},

where \mathbf{c}_{i} denotes the contact point of candidate motion clip i. All positions in Eq. (1) are expressed in the robot’s initial root-facing coordinate frame. We assign a larger weight to the height difference and select the clip with the lowest contact cost.

III-B Trajectory-Guided Motion Matching

To enable natural humanoid locomotion that transitions into the striking clip, we use trajectory optimization to plan a locomotion trajectory. We then use the optimized trajectory to retrieve and match consistent whole-body locomotion from the generic motion library.

Let d be the horizontal distance from the target point to the initial position, and set d_{0}=1.5 m. For a nearby entry position (d\leq d_{0}), the approach primarily translates while retaining its initial heading, followed by heading alignment. For a distant entry position, the robot turns toward the travel direction, runs forward, and recovers the strike-entry heading. Both modes optimize their phase durations. We use planar state \mathbf{z}=(\mathbf{p},\psi,v_{f},v_{l},\omega) and control \mathbf{u}=(a_{f},a_{l},a_{\omega}), with \dot{\mathbf{p}}=R(\psi)(v_{f},v_{l})^{\top}, \dot{\psi}=\omega, \dot{v}_{f}=a_{f}, \dot{v}_{l}=a_{l}, and \dot{\omega}=a_{\omega}.

Given the current robot state \mathbf{z}_{0}=\mathbf{z}_{\mathrm{init}}, the trajectory optimizer solves:

\displaystyle\min_{\{\mathbf{z}_{k},\mathbf{u}_{k}\},\{T_{m}\}}\quad J={} \\ \displaystyle w_{T}T+w_{L}L+w_{u}\langle\|\bar{\mathbf{u}}_{k}\|^{2}\rangle \\ \displaystyle+w_{\Delta u}\Bigl\langle\Bigl\|\frac{\bar{\mathbf{u}}_{k+1}-\bar{\mathbf{u}}_{k}}{\Delta t_{k}}\Bigr\|^{2}\Bigr\rangle \\ \displaystyle+w_{e}E_{e}+E_{\mathrm{mode}} \\ \displaystyle\text{s.t.} \\ \displaystyle\mathbf{z}_{k+1}=f_{d}(\mathbf{z}_{k},\mathbf{u}_{k},\Delta t_{k}), \\ \displaystyle\|\mathbf{p}_{K}-\mathbf{p}_{e}\|\leq\epsilon_{p},\quad|\psi_{K}-\psi_{e}|\leq\epsilon_{\psi}, \\ \displaystyle\|\mathbf{v}_{K}-\mathbf{v}_{e}\|\leq\epsilon_{v},\quad|\omega_{K}-\omega_{e}|\leq\epsilon_{\omega}, \\ \displaystyle(\mathbf{z}_{k},\mathbf{u}_{k})\in\mathcal{C}_{\mathrm{mode}}, \\ \displaystyle\sum_{m}T_{m}\leq T_{\max}.

Equation (2), in which k indexes the discrete trajectory samples and K denotes the final sample, uses subscript e to denote the strike-entry state. The function f_{d} is the discrete state-transition function obtained from the planar dynamics; m indexes the motion phases, and T=\sum_{m}T_{m} is the total motion duration.

The path length L is normalized by the straight-line distance from the initial position to the strike-entry position, with the normalization distance lower-bounded by 1 m. The variable \bar{\mathbf{u}} denotes componentwise normalized controls, and \langle\cdot\rangle denotes averaging over trajectory samples. The terminal constraints bound position, heading, linear-velocity, and angular-velocity errors by \epsilon_{p}, \epsilon_{\psi}, \epsilon_{v}, and \epsilon_{\omega}, respectively. The set \mathcal{C}_{\mathrm{mode}} collects mode-dependent velocity, angular-velocity, control, and heading constraints, while the timing constraint bounds the total duration. The costs favor short, direct approaches with smooth controls; E_{e} penalizes terminal error, while E_{\mathrm{mode}} penalizes motion inconsistent with the selected approach mode. We transcribe the problem using trapezoidal collocation in CasADi [27] and solve the resulting nonlinear program with IPOPT [28], then interpolate the optimized trajectory to construct motion-matching queries.

From the optimized state sequence \{\mathbf{z}_{k}\}, we extract future planar positions, headings, and root velocities and combine them with the current motion state to construct the motion-matching query feature \hat{\mathbf{f}}_{\mathrm{query}}. Following motion matching [5], we use this query to retrieve a matching motion frame from the LAFAN1 locomotion library:

j^{*}=\arg\min_{j\in\mathcal{V}_{L}}\|\hat{\mathbf{f}}_{j}-\hat{\mathbf{f}}_{\mathrm{query}}\|_{W}^{2}.

Here, j indexes candidate motion frames, \mathcal{V}_{L} is the set of eligible locomotion frames, and j^{*} is the match with the lowest total cost. The normalized feature \hat{\mathbf{f}}_{j} contains foot positions and velocities, root velocity, racket-hand position and relative velocity, and future planar positions and headings at 0.4, 0.8, and 1.2 s. The diagonal matrix W weights the feature components. By globally searching the locomotion-frame set and connecting the best-matching frames, the planned planar trajectory is converted into a whole-body locomotion reference and connected to the striking motion selected by contact matching (Fig. 3).

III-C Composition and Dynamic Correction

At the locomotion-to-strike transition, we select the frame in the chosen strike clip whose feature vector best matches the current pose as the entry frame, and interpolate toward it. Horizontal root alignment and offset-decay blending reduce discontinuities. For joint positions with initial mismatch \Delta\mathbf{q}_{0}, we apply

\mathbf{q}(t)=\mathbf{q}_{\mathrm{in}}(t)+\left(1-\frac{t}{T_{b}}\right)^{5}\Delta\mathbf{q}_{0},\qquad 0\leq t\leq T_{b}.

Here, \mathbf{q}_{\mathrm{in}}(t) is the strike-joint trajectory starting from the entry frame, \Delta\mathbf{q}_{0}=\mathbf{q}_{\mathrm{cur}}-\mathbf{q}_{\mathrm{in}}(0) is the joint offset at the transition, and T_{b} is the blending duration. Root translations are blended similarly, while orientations are interpolated using quaternions. The selected strike motion is then played until completion.

Residual discontinuities and kinematic inconsistencies can remain after blending. Following the motion-tracking formulation of BeyondMimic [16], we train a mimic policy to track the composed kinematic references and roll it out in physics simulation. We re-collect the policy-executed trajectories as dynamics-corrected tennis motion references for training the teacher policy in Section IV (Fig. 4 ①).

IV Three-Stage Tennis Policy Learning

In this section, we use the dynamics-corrected tennis reference-motion dataset to train a goal-conditioned and time-adaptive teacher policy. Next, we match physically simulated ball trajectories with the reference motions used by the teacher policy and distill a student policy that requires no reference-motion input. Finally, in Stage III, we fine-tune the student policy using task rewards and physics domain randomization.

IV-A Stage I: Time-Adaptive Teacher Policy

Fixed-rate reference playback allows the tracking policy to reach the striking state only at a predetermined time, limiting its ability to handle balls with different speeds and flight times. To adapt the same striking reference to different contact times, we train a time-adaptive teacher policy (Fig. 4 ②) .

For reference motion i, let \phi_{t}\in[0,1] denote the current normalized phase, \phi_{i}^{\mathrm{hit}} its strike phase, and \bar{T}_{i} the time required to reach the strike phase at the nominal playback rate. We sample a target strike deadline T_{i} at 0.1 s intervals from [\bar{T}_{i}/\kappa,\bar{T}_{i}], where \kappa=2. At time step t, the teacher observes the current robot state \mathbf{s}_{t}, reference pose \mathbf{q}(\phi_{t}), task goal \mathbf{G}, and remaining time T_{t}^{\mathrm{rem}}, and outputs the joint action \mathbf{a}_{t} and phase acceleration \ddot{\phi}_{t}:

(\mathbf{a}_{t},\ddot{\phi}_{t})=\pi_{T}\left(\mathbf{s}_{t},\mathbf{q}(\phi_{t}),\mathbf{G},T_{t}^{\mathrm{rem}}\right).

The remaining time is T_{t}^{\mathrm{rem}}=T_{i}-t\Delta t. We train the teacher with Proximal Policy Optimization (PPO) [29] to strike a stationary ball at the sampled deadline. The reward uses the motion-tracking and goal-conditioned striking terms from TaskNPoint [24], augmented with a phase-timing reward that encourages the reference motion to reach its strike phase at the specified time.

The reference joint positions are parameterized by phase as \mathbf{q}^{\mathrm{ref}}(\phi). The teacher outputs phase acceleration \ddot{\phi}, which is integrated to obtain phase velocity \dot{\phi} and phase \phi. Using the chain rule, \dot{\mathbf{q}}^{\mathrm{ref}}=(\partial\mathbf{q}^{\mathrm{ref}}/\partial\phi)\dot{\phi}, we constrain phase acceleration so that the updated phase rate keeps reference joint velocities within their prescribed limits.

Stage I thus yields a time-adaptive teacher that retimes the same reference motion to meet different strike deadlines.

Earlier feasible strike deadlines allow Stage II to match the same reference motion to faster incoming balls.

This expands the range of incoming-ball conditions that can be paired with the existing reference dataset for training.

IV-B Stage II: Ball Trajectory Matching and Student Policy Distillation

In Stage II, we sample ball launch positions and velocities, simulate trajectories with gravity, drag, and ground rebound, and match trajectory points between the first and second bounces to reference strike points. A valid pair satisfies \|\mathbf{b}_{i}(T_{m})-\mathbf{c}_{m}\|_{2}\leq\epsilon and T_{m}\in[\bar{T}_{m}/\kappa,\bar{T}_{m}], where \mathbf{b}_{i} is the incoming trajectory, \mathbf{c}_{m} the reference contact point, and \epsilon the matching tolerance. The matched time becomes the teacher’s deadline. We use a library of 200 generated reference motions for training.

We load the frozen phase teacher and transfer its privileged skills to the student through DAgger-style online distillation [30] within the teacher-guided PPO (TPPO) framework (Fig. 4 ③). In Stage II, PPO updates are disabled; only action and representation distillation objectives are optimized. During rollout collection, we gradually reduce the fraction of environments controlled by the teacher and increase student control. Teacher action labels are queried at every visited state, regardless of which policy executes the actions. This provides supervision on states reached after the student deviates from the teacher’s trajectories, mitigating the distribution shift associated with offline behavior cloning. The teacher’s joint actions already reflect its adaptation to the requested strike time. We therefore distill only its 29-dimensional joint-action output, while the frozen teacher continues to provide phase acceleration in the training environment to advance the reference motion used by the teacher.

Instantaneous observations provide incomplete information about ball evolution and the forthcoming striking motion. We therefore construct a privileged training latent \mathbf{z}_{t}^{T} and a deployable student latent \mathbf{z}_{t}^{S}:

\displaystyle\mathbf{z}_{t}^{T} \\ \displaystyle=E_{T}(\mathbf{h}_{t}^{r},\mathcal{R}_{t}), \\ \displaystyle\mathbf{z}_{t}^{S} \\ \displaystyle=E_{S}(\mathbf{h}_{t}^{r},\mathbf{h}_{t}^{b}), \\ \displaystyle\hat{\mathbf{a}}_{t}^{S} \\ \displaystyle=\mu_{S}(\mathbf{o}_{t},\mathbf{z}_{t}^{S}).

Here \mathbf{h}_{t}^{r} and \mathbf{h}_{t}^{b} contain robot-state and ball-position/velocity histories, \mathcal{R}_{t} is a future reference joint-position/velocity window, and \mu_{S} is the student action mean. Both latents are 128-dimensional. The student temporal encoder fuses robot and ball observations at five historical instants; the privileged encoder uses robot-history queries to cross-attend to 12 future reference joint states. The auxiliary reference encoder E_{T} is learned in this stage and is distinct from the frozen phase teacher \pi_{T}. The student action network consumes \mathbf{z}_{t}^{S} throughout, while reference motions remain confined to the training supervision branch.

Three complementary losses supervise this transfer: Euclidean action distillation from the frozen teacher, reconstruction of normalized future references from \mathbf{z}_{t}^{T}, and latent alignment of \mathbf{z}_{t}^{S} to a stop-gradient \mathbf{z}_{t}^{T}. Let \mathcal{H} be elementwise-averaged Smooth L1 loss, D the reference decoder, \operatorname{sg} stop-gradient, and \widetilde{\mathcal{R}}_{t} the normalized reference window:

\displaystyle\mathcal{L}_{\mathrm{act}}=\mathbb{E}\big[\|\hat{\mathbf{a}}_{t}^{S}-\mathbf{a}_{t}^{T}\|_{2}\big], \\ \displaystyle\mathcal{L}_{\mathrm{ref}}=\mathbb{E}\big[\mathcal{H}(D(\mathbf{z}_{t}^{T}),\widetilde{\mathcal{R}}_{t})\big], \\ \displaystyle\mathcal{L}_{\mathrm{latent}}=\mathbb{E}\big[\mathcal{H}(\mathbf{z}_{t}^{S},\operatorname{sg}(\mathbf{z}_{t}^{T}))\big], \\ \displaystyle\mathcal{L}_{\mathrm{II}}=\mathcal{L}_{\mathrm{act}}+\mathcal{L}_{\mathrm{ref}}+0.05\mathcal{L}_{\mathrm{latent}}.

Expectations are taken over continually refreshed DAgger rollout batches, where \mathbf{a}_{t}^{T} denotes the teacher joint-action labels at the visited states. Reference reconstruction gives the privileged latent explicit motion semantics, while latent alignment transfers this supervision to a student representation conditioned only on observation histories. Consequently, \mathbf{z}_{t}^{S} provides the policy with motion intent and timing information inferred from history; neither reference motions nor explicit strike deadlines are required at execution.

Table I compares the frozen teacher, student actor, and Stage III critic using the final fine-tuning configuration. The asymmetric value network V(\mathbf{o}_{t}^{V},\operatorname{sg}(\mathbf{z}_{t}^{T})) receives uncorrupted base observations, current references, and strike timing. Value gradients do not update the privileged encoder.

IV-C Stage III: Task Refinement via PPO

We initialize from the distilled student, let it control all environments, and optimize tennis performance with PPO using task (Fig. 4 ④).. To preserve the learned representations, both latent encoders and the reference decoder are frozen while the policy and value heads are updated.

Task rewards guide pre-contact swinging and post-contact return quality: f_{\mathrm{swing}} encourages racket velocity along the reference striking direction near contact; f_{\mathrm{out}} rewards horizontal ball speed toward the landing target; f_{\mathrm{net}} rewards predicted net clearance; and f_{\mathrm{land}} rewards predicted landing accuracy for valid net-crossing trajectories. The last three terms are evaluated from the post-strike ball state.

The total reward combines these task scores with auxiliary control rewards and penalties as r_{t}=\Delta t\sum_{i}w_{i}f_{i}(t), where f_{i}(t) denotes the ith term and w_{i} its weight, with a control step of \Delta t=0.02 s. Episodes terminate when both pelvis and torso tilt exceed 50^{\circ}. A retained action-distillation coefficient of 0.05 regularizes refinement toward the teacher’s learned skills:

\mathcal{L}_{\mathrm{III}}=\mathcal{L}_{\mathrm{PPO}}+0.05\mathcal{L}_{\mathrm{act}}.

Here \mathcal{L}_{\mathrm{PPO}} includes the clipped policy surrogate, value loss, and entropy regularization. The 0.05 action-distillation weight is set independently of the Stage II latent-alignment weight; reference reconstruction and latent losses are no longer optimized in Stage III. The teacher supplies action supervision and reference-phase management only during simulation training. Deployment removes the teacher, privileged reference encoder, and decoder, retaining only E_{S} and the student action network. This single policy produces 29-dimensional joint actions at 50 Hz from current observations and encoded histories, without reference motions or a phase controller.

During fine-tuning, we follow the ball-physics domain-randomization design of LATENT [1], adapting its parameterization and sampling ranges to our simulator to address the sim-to-real dynamics gap. At simulation initialization, ball mass, ball-ground restitution, ground tangential-speed retention, and aerodynamic drag coefficients are sampled uniformly for each parallel environment. We additionally randomize racket mass, center-of-mass offsets, and racket restitution, exposing the student to variations in flight, ground rebound, and racket-contact dynamics during task refinement.

V EXPERIMENTAL RESULTS

V-A Reference Motion Coverage

We compare the original striking library with TGMM-generated reference libraries on 10,000 shared groups of forward-simulated incoming-ball trajectories, varying the number of qualified candidates per group, K, from 1 to 24. Qualified trajectories clear the net and, before the second bounce, pass within 3 m horizontally of the initial robot position at a ball-center height of at least 0.5 m; filtering is independent of the motion library. A match requires a point between the first and second bounces to lie within 0.20 m of a reference strike point and satisfy the Stage I teacher’s timing window, \bar{T}_{i}/2\leq t+0.01\,\mathrm{s}\leq\bar{T}_{i}, where \bar{T}_{i} is the reference’s nominal strike time. A group succeeds if at least one of its first K candidates matches.

At the same library size of 164 motions and K=1, the generated references achieve 42.64% matching success, compared with 7.57% for the original library (Fig. 5). This improvement is consistent with the broader distribution of generated strike locations around the robot, whereas the original locations concentrate in front of it (Fig. 2(b)). TGMM thus expands spatiotemporal coverage even at a fixed library size.

Stage II uses 200 generated references to balance reference diversity and matching cost. At this library size, the randomly sampled subsets reach 99.21% mean group matching success at K=8, supporting broad coverage with a compact library and modest candidate budget. Results for randomly sampled libraries are averaged over three subsets.

V-B Simulation Experiments

We evaluate the frozen teacher on 100 randomly selected motions from its training library under nominal physics, using gravity-compensated balls at stationary strike targets. For each motion, we independently sample five deadlines T_{i}=\alpha\bar{T}_{i}, with \alpha\sim\mathcal{U}(0.5,1). During execution, the teacher-controlled phase updates progression through the reference motion.

Across 500 trials, the teacher makes physical racket contact in 493 trials (98.6%), with 362 trials (72.4%) achieving contact within \pm 50 ms of the target deadline. Figure 6 shows the absolute timing errors of the 493 contacts, with a mean absolute error of 71.23 ms. These results demonstrate that the teacher adapts whole-body motion timing to sampled deadlines, allowing the same reference to support more ball-arrival times and provide more motion–trajectory pairings for Stage II distillation.

We compare the Stage II distilled policy (w/o PPO fine-tuning) with the full method on 300 paired incoming balls and shared per-trial targets across three seeds (100 trials each). Each policy uses its stage-specific deployment interface at 50 Hz under nominal physics, resetting the robot and observation histories before each 8 s trial. Incoming balls use the deployment sampler without reference-motion matching, and landing targets follow the Stage III training distribution.

Grouping by the ball’s side at its closest horizontal approach to the initial robot between the first two bounces yields 159 forehand-side and 141 backhand-side trials, independently of policy outcomes. Table II reports hit and return rates over all incoming balls, together with mean landing error and mean 3D ball-center-to-sweet-spot distance at first contact for struck balls.

Without PPO fine-tuning, forehand- and backhand-side hit rates are 1.26% and 5.67%, with no balls clearing the net; its detected contacts occur approximately 14 cm from the racket sweet spot on average. The full method raises hit rates to 94.97%/95.04% and return rates to 91.19%/92.20%, while reducing mean sweet-spot distances to 5.61/6.54 cm and mean landing errors to 1.150/1.314 m. These results demonstrate improved contact precision and controlled returns across varying targets, supporting subsequent real-world deployment.

V-C Real World Experiments and Benchmark

We evaluate single-shot striking and continuous human–robot rallies on a real humanoid robot (Fig. 4 ⑤). An external motion-capture system provides the ball’s 3D position and the robot’s root pose, and ball velocity is estimated from consecutive position observations. Ball position and velocity are expressed in the robot’s root frame according to the training observation definitions and supplied to the policy.

The motion-capture system covers a horizontal area of approximately 10.5\,\mathrm{m}\times 5.5\,\mathrm{m}, and we adapt the simulation training conditions to this confined space. The reward for horizontal outgoing speed projected toward the landing target(5,0) saturates at 3.5 m/s, removing the incentive from this term for faster returns. The policy runs at 50 Hz on a laptop equipped with an NVIDIA GeForce RTX 5070 Ti Laptop GPU, exchanging control commands and state information with the robot over wired Ethernet through the Unitree DDS interface. The racket is attached to the robot’s right hand using a 3D-printed connector.

The first experiment evaluates the robot’s response to different incoming-ball locations using manual throws toward its forehand and backhand sides (Fig. 7). We conduct 30 trials per side, for a total of 60, varying the throw direction and location to cover different striking positions around the robot. As shown in Table III, the robot hits 27/30 forehand balls (90.00\%) and 26/30 backhand balls (86.67\%). The robot clears the net in 14/30 forehand trials (46.67\%) and 16/30 backhand trials (53.33\%). These results demonstrate reactive real-world striking across different incoming-ball locations.

Table III includes published LATENT [1] human–robot rally results and AdaPT [2] results from 25 machine-fed trials per target region, retaining their original experiment environment. Among the approaches using video-derived motions, ATHLETE achieves higher reported forehand and backhand hit and net-clearance rates than AdaPT,under their respective evaluation protocols.

The second experiment evaluates continuous rallies between the robot and human partners with different levels of tennis experience (Fig. (e)–(h)). The longest observed rallies are 5 rounds with a novice player and 8 rounds with a professional player. These results demonstrate successful transfer to real hardware and sustained coordination of perception, locomotion, and striking during human–robot interaction.

V-D Qualitative Comparison

Table IV compares the reported data sources, control architectures, and training configurations of Vid2Player3D [31], LATENT, AdaPT, and ATHLETE. LATENT reports strong in-court return performance using five hours of newly captured motion primitives from amateur players and eight training GPUs. ATHLETE avoids dedicated tennis motion-capture collection by reusing public locomotion and sparse video-derived strikes, and trains its policies on a single RTX 4090. AdaPT also uses video-derived motions for quantitative evaluation, but retains explicit motion planning, future-ball prediction, and reference tracking at deployment. ATHLETE converts composed references into a single reactive policy that supports forehand and backhand returns and continuous rallies without these online components. Together with the real-world results, this demonstrates the value of reusing motion data while simplifying deployment. All three real-robot evaluations currently use external motion capture for state estimation.

VI Conclusion

In this work, inspired by how humans acquire complex skills by combining reusable motor experience with task-specific demonstrations, we presented ATHLETE, a framework for learning reactive humanoid tennis skills from generic locomotion data and ordinary tennis videos, without dedicated tennis motion-capture demonstrations. By combining trajectory-guided motion generation with teacher–student learning, ATHLETE learns a deployable policy that responds to incoming balls without reference-motion inputs at deployment. Experiments on a real humanoid robot demonstrate forehand and backhand returns as well as continuous rallies with human players. The current system relies on external motion capture for ball-state estimation. Future work will investigate onboard ball perception and preservation of motion-style diversity.

References

1. [1] Z. Zhang et al., “Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data,” arXiv:2603.12686, 2026.
2. [2] T. Huang et al., “Towards Professional Tennis Styles for Humanoid Robots with Adaptive Motion Planning and Tracking,” arXiv:2608.20087, 2026.
3. [3] F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal, “Robust Motion In-Betweening,” ACM Trans. Graph., vol. 39, no. 4, 2020, doi: 10.1145/3386569.3392480.
4. [4] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “AMASS: Archive of Motion Capture as Surface Shapes,” in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), pp. 5442–5451, 2019.
5. [5] S. Clavet, “Motion Matching and the Road to Next-Gen Animation,” Game Developers Conference (GDC), presentation, 2016.
6. [6] D. Holden, O. Kanoun, M. Perepichka, and T. Popa, “Learned motion matching,” ACM Trans. Graph., vol. 39, no. 4, 2020, doi: 10.1145/3386569.3392440.
7. [7] Z. Wu et al., “Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching,” arXiv:2602.15827, 2026.
8. [8] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano, “Human Motion Diffusion Model,” in Proc. Int. Conf. Learning Representations (ICLR), 2023.
9. [9] D. Rempe et al., “Kimodo: Scaling Controllable Human Motion Generation,” arXiv:2603.15546, 2026.
10. [10] H. Y. Ling, F. Zinno, G. Cheng, and M. van de Panne, “Character Controllers Using Motion VAEs,” ACM Trans. Graph., vol. 39, no. 4, 2020, doi: 10.1145/3386569.3392422.
11. [11] K. Zhao et al., “ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation,” ACM Trans. Graph., vol. 45, no. 4, Art. no. 86, 2026, doi: 10.1145/3811284.
12. [12] H. Zhang et al., “Natural Humanoid Robot Locomotion with Generative Motion Prior,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2025.
13. [13] Z. Zhang et al., “Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking,” IEEE Robot. Autom. Lett., vol. 11, no. 9, pp. 10194–10201, 2026, doi: 10.1109/LRA.2026.3710365.
14. [14] W. Xie et al., “TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Control,” arXiv:2602.07439, 2026.
15. [15] S. Huang et al., “OMG: Omni-Modal Motion Generation for Generalist Humanoid Control,” arXiv:2606.10340, 2026.
16. [16] Q. Liao et al., “BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion,” arXiv:2508.08241, 2025.
17. [17] T. He et al., “ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills,” in Proc. Robotics: Science and Systems (RSS), 2025, doi: 10.15607/RSS.2025.XXI.066.
18. [18] J. Li, X. Cheng, T. Huang, S. Yang, R.-Z. Qiu, and X. Wang, “AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control,” in Proc. Robotics: Science and Systems (RSS), 2025, doi: 10.15607/RSS.2025.XXI.061.
19. [19] Z. Luo et al., “SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control,” Sci. Robot., vol. 11, no. 117, Art. no. eaed4592, 2026, doi: 10.1126/scirobotics.aed4592.
20. [20] Z. Wang et al., “Perceptive Behavior Foundation Model: Adapting Human Motion Priors to Robot-Centric Terrain,” arXiv:2606.08059, 2026.
21. [21] L. Yang et al., “OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction,” arXiv:2509.26633, 2025.
22. [22] H. Weng et al., “HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos,” arXiv:2509.16757, 2025.
23. [23] Y. Wang et al., “HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos,” arXiv:2602.02473, 2026.
24. [24] B. Werner, I. Demler, P. Perona, and A. D. Ames, “TaskNPoint: How to Teach Your Humanoid to Hit a Backhand in Minutes,” arXiv:2606.26215, 2026.
25. [25] Z. Shen et al., “World-Grounded Human Motion Recovery via Gravity-View Coordinates,” in SIGGRAPH Asia Conference Proceedings, 2024.
26. [26] J. P. Araújo, Y. Ze, P. Xu, J. Wu, and C. K. Liu, “Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking,” arXiv:2510.02252, 2025.
27. [27] J. A. E. Andersson, J. Gillis, G. Horn, J. B. Rawlings, and M. Diehl, “CasADi—A software framework for nonlinear optimization and optimal control,” Math. Program. Comput., vol. 11, no. 1, pp. 1–36, 2019, doi: 10.1007/s12532-018-0139-4.
28. [28] A. Wächter and L. T. Biegler, “On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming,” Math. Program., vol. 106, no. 1, pp. 25–57, 2006, doi: 10.1007/s10107-004-0559-y.
29. [29] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv:1707.06347, 2017.
30. [30] S. Ross, G. Gordon, and D. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,” in Proc. Int. Conf. Artificial Intelligence and Statistics (AISTATS), vol. 15, pp. 627–635, 2011.
31. [31] H. Zhang et al., “Learning Physically Simulated Tennis Skills from Broadcast Videos,” ACM Trans. Graph., vol. 42, no. 4, pp. 1–14, 2023, doi: 10.1145/3592408.