KinoGen: Customizable Humanoid Loco-Manipulation References via Kinematically Grounded Video Generation
A Cornell and Multiply Labs preprint generates training motions for a humanoid by having a video model imagine the robot itself doing the task, then fitting the exact robot and object models to that video. Policies trained on these references succeeded 57% of the time in simulation against 42% for the best human-video pipeline, and four tasks ran on a real Unitree G1 with dexterous hands.
Can a pretrained video generator produce whole-body, hand and object reference motions for a specific humanoid, in the exact task layout, without going through a human body and retargeting, and can a user edit those references to hit a precise goal pose?
Humanoids learn loco-manipulation skills by tracking reference trajectories with reinforcement learning, but each reference must synchronise body, fingers and object for one robot and one scene. Human demonstrations must be reconstructed and retargeted, which introduces embodiment errors, and teleoperation is expensive to scale across objects, layouts and goals.
Recent pipelines either generate a human-object interaction video and retarget the recovered human motion to the robot (GenMimic, GRAIL), or render the target robot but still recover motion through a human pose estimator (GenHOI). Both satisfy the robot's kinematics only after generation, which shows up as penetration, sliding feet and lost contacts, and neither lets the user demand an exact object end pose.
KinoGen renders the simulated task instance with the real robot and object models from two calibrated cameras and uses those renders to condition a pretrained video generator. It then reconstructs the motion by fitting the same robot, hand and object models to the generated two-view video, initialised with body, hand and object priors and refined against masks, point tracks and depth, so the result lies in the robot's configuration space by construction. To customise a reference, the user edits a keyframe in the simulator, the video is regenerated around it and a constrained reconstruction enforces the edited pose exactly.
On eight tasks for a Unitree G1 with two 6-DoF OYMotion hands (box, basket, chair, dumbbell and basketball lifting, sitting, kneeling, mopping), KinoGen references had 2.30 cm penetration against 3.06 to 10.55 cm for the baselines, 4.09 cm/s foot skating against 5.11 to 9.95, and kept 91.16% of annotated contacts against 69.40% for GRAIL, 43.91% for GenHOI and 13.57% for GenMimic. Tracking policies trained on them reached 57% success against 42%, 31% and 11%, with the lowest reference deviation (0.09 m) and the highest learning-curve AUC (60.74). In a bottle-placement customisation test with 30 targets, KinoGen hit every target exactly with worst table penetration of 2.7 to 21.0 mm, while direct end-effector interpolation also hit the targets but pushed the hand through the table by about 139 to 147 mm. Policies for four of the tasks were run on the physical robot; the paper reports no real-world success rates.
This is a preprint. The authors note that reconstruction fails under heavy or persistent occlusion, that reference quality and diversity are capped by the video generator, and that the hard keyframe constraint guarantees the pose but not physical executability or contact. We add that 57% simulated success is still low for deployment, the comparison uses eight tasks with ten repeats, and the real-robot runs are shown qualitatively rather than measured.
Data for whole-body manipulation is the bottleneck for humanoid makers such as Unitree, Figure, Agility and Boston Dynamics, and most current pipelines lean on motion capture or teleoperation. Generating references directly in the robot's own body, inside the customer's exact layout, could cut the cost of adding a new task from days of teleoperation to compute time. Expect this to matter first to simulation and data-tool vendors and to labs training on the Unitree G1 within 12 to 24 months; factory use depends on success rates rising well above the levels reported here.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.