KODA: Kinematics--Observation Data Alignment for Humanoid Learning from Human Egocentric Demonstration
This preprint turns videos people film with a chest-mounted iPhone into training data a Unitree G1 humanoid can learn from, including what the robot's own camera would have seen. Adding 100 converted human clips per task to 50 robot demonstrations raised success on unseen object setups by about 45 percentage points.
Can cheap egocentric human recordings be converted into whole-body humanoid training data that works with more than one low-level controller, without collecting more robot demonstrations?
Training a whole-body vision-language-action policy for a humanoid needs robot demonstrations with matching robot-camera images and controller-specific joint commands, which are slow and expensive to collect by teleoperation. Human videos are cheap but are filmed from the wrong viewpoint and carry only sparse motion cues (head and hands).
Earlier methods either pretrained on human data and still needed target-robot data to adapt (EgoVLA, WholeBodyVLA), or converted human data into commands tied to one controller (EgoHumanoid, HALOMI), with a fixed camera offset that does not follow the generated robot motion. Each conversion therefore produced a data silo for one controller.
KODA first generates a physically checked whole-body trajectory in MuJoCo from the camera pose and retargeted hands, using cross-entropy-method search over command residuals. It then renders what the robot's head camera would see along that trajectory, by removing the human arm, reprojecting the scene with depth, compositing a rendered robot and inpainting holes. Motion, views and instruction form a reusable canonical record; for each controller (SONIC or HEFT) KODA re-optimises commands to track that record and exports matching state and action labels for co-training NVIDIA's GR00T N1.7.
On four G1 tasks with BrainCo hands (cup to basket, parcel to shelf, basket to shelf, cup to drawer), 50 robot demos plus 100 converted human demos per task raised macro-average success on object conditions seen only in human clips from 3.75% to 48.75% with SONIC (+45.0 points) and by 46.25 points with HEFT. Success kept rising across the 25, 50, 75 and 100 human-demo budgets. All 400 human clips were recorded at 50 Hz with an iPhone including depth.
The authors say fine-grained hand-object contact is not yet reproduced faithfully. The evaluation is four tabletop-and-shelf tasks on one robot, with 20 trials per condition, so the percentages move in 5-point steps; robot demonstrations are still required for co-training. The pipeline chains many large models (hand pose, segmentation, inpainting, depth), and its compute cost is not given as a headline figure.
Humanoid makers and data vendors are spending heavily on teleoperation; a method that reuses human video across controllers attacks that cost directly. It matters most to Unitree G1 users and teams building on NVIDIA GR00T, and to Chinese labs with access to large egocentric datasets. Expect such conversion pipelines in data-vendor offerings within 12 to 24 months; factory-grade reliability is further out.
Full paper
KODA: Kinematics--Observation Data Alignment for Humanoid Learning from Human Egocentric Demonstration
Published under CC BY-NC-SA 4.0. Reproduced with attribution; original at arXiv:2610.21152 (PDF).
\project
https://koda-humanoid.github.io/koda/
Abstract
Training humanoid whole-body vision–language–action (VLA) policies requires costly robot demonstrations with aligned robot-view observations and controller-compatible whole-body state–action labels. Egocentric human demonstrations offer scalable supervision but have misaligned views and sparse motion cues. Existing approaches either pretrain policies that still require robot data for adaptation or generate controller-specific commands that limit reuse. Conversion also requires robot-view observations aligned with the generated whole-body motion. We present KODA (Kinematics–Observation Data Alignment), which converts egocentric RGB demonstrations into whole-body VLA training data using a reusable canonical trajectory to guide robot-view synthesis and generate controller-specific whole-body state–action supervision. KODA first generates a simulator-checked whole-body trajectory using a sparse tracker with sampling-based command optimization. This generated trajectory directly determines the onboard camera poses used to synthesize corresponding robot-view observations. The motion, observations, and task instruction form a canonical record. For each deployment controller, KODA optimizes commands to track the stored motion and records the corresponding action labels and proprioceptive states. When switching controllers, KODA retains this canonical record and regenerates whole-body state–action supervision for co-training with matching robot demonstrations. We evaluate KODA on four Unitree G1 loco-manipulation tasks using two whole-body controllers, SONIC and HEFT. With robot data held fixed, macro-average success on conditions covered by human demonstrations but absent from robot data improves by 45.0 and 46.25 percentage points, respectively. Under both controllers, this metric shows an upward trend across the evaluated human-data budgets, suggesting potential for further scaling.
Keywords: Egocentric Human Data, Humanoid Loco-Manipulation
††