MEKAWIRE
专业arXiv

KODA：将人类第一视角演示转化为人形机器人全身训练数据的运动学与观测对齐方法

Zibo Zhou, Yueru Chen, Junsong Wu, Weiji Xie, Qingyao Xu, Sheng Yin, Shiqing Wang, Jiyuan Shi, Siheng Chen, Chenjia Bai, Xuelong LiTeleAI (China Telecom), Shanghai Jiao Tong University, Gamma Robotics
论文原文 阅读论文全文（PDF）
30 秒速读

这篇预印本把人们用胸前iPhone拍摄的视频转换成宇树G1人形机器人可学习的数据，并合成机器人自身相机应看到的画面。每个任务在50条机器人演示之外加入100条转换后的人类演示，在未见过的物体条件下成功率提高约45个百分点。

研究问题

能否在不增加机器人演示的情况下，把廉价的第一视角人类录像转化为可适配多种底层控制器的人形机器人全身训练数据？

问题

训练人形机器人全身VLA策略需要带有机器人视角图像和控制器专用关节指令的演示，靠遥操作采集既慢又贵。人类视频便宜，但视角不对，只包含头部和双手等稀疏运动信息。

既有方法

已有方法要么用人类数据预训练后仍需目标机器人数据适配（EgoVLA、WholeBodyVLA），要么把人类数据转换成绑定单一控制器的指令（EgoHumanoid、HALOMI），且用固定偏移平移相机，无法跟随生成的机器人动作，最终形成按控制器割裂的数据孤岛。

新方法

KODA先根据相机位姿和重定向后的手部数据，在MuJoCo中生成经物理校验的全身轨迹，并用交叉熵方法搜索指令残差以降低跟踪误差；随后去除人类手臂、用深度重投影场景、合成渲染的机器人并修补空洞，生成机器人头部相机视角。动作、画面和指令构成可复用的规范记录，针对每个控制器（SONIC或HEFT）重新优化指令并导出状态与动作标签，与真机数据一起训练英伟达GR00T N1.7。

结果

在装有强脑科技（BrainCo）灵巧手的G1上完成四项任务（杯子放进篮子、包裹放上货架、篮子放上货架、杯子放进抽屉），每项任务50条真机演示加100条转换人类演示后，在仅人类视频覆盖的物体条件下，SONIC的宏平均成功率从3.75%升至48.75%，提高45.0个百分点，HEFT提高46.25个百分点；人类数据从25条增至100条时成功率持续上升。

局限

作者承认尚不能精细还原手与物体的接触。评估只涉及一台机器人的四项任务、每个条件20次试验，数值以5个百分点为单位变化；联合训练仍需真机演示。流程串联手部姿态、分割、修补和深度估计等多个大模型，计算成本值得关注。

产业影响

国内人形机器人企业和数据服务商正大量投入遥操作数据采集，一种能跨控制器复用人类视频的方法直接针对这一成本。它对宇树G1用户、基于英伟达GR00T开发的团队，以及拥有大规模第一视角数据的中国实验室尤其重要。预计12至24个月内此类转换流程会进入数据服务商产品，达到工厂级可靠性还需更长时间。

论文全文

KODA: Kinematics--Observation Data Alignment for Humanoid Learning from Human Egocentric Demonstration

Zibo Zhou, Yueru Chen, Junsong Wu, Weiji Xie, Qingyao Xu, Sheng Yin, Shiqing Wang, Jiyuan Shi, Siheng Chen, Chenjia Bai, Xuelong Li

本文依据 CC BY-NC-SA 4.0 许可发布，经署名转载，原文见 arXiv:2610.21152（PDF）。

\project

https://koda-humanoid.github.io/koda/

Abstract

Training humanoid whole-body vision–language–action (VLA) policies requires costly robot demonstrations with aligned robot-view observations and controller-compatible whole-body state–action labels. Egocentric human demonstrations offer scalable supervision but have misaligned views and sparse motion cues. Existing approaches either pretrain policies that still require robot data for adaptation or generate controller-specific commands that limit reuse. Conversion also requires robot-view observations aligned with the generated whole-body motion. We present KODA (Kinematics–Observation Data Alignment), which converts egocentric RGB demonstrations into whole-body VLA training data using a reusable canonical trajectory to guide robot-view synthesis and generate controller-specific whole-body state–action supervision. KODA first generates a simulator-checked whole-body trajectory using a sparse tracker with sampling-based command optimization. This generated trajectory directly determines the onboard camera poses used to synthesize corresponding robot-view observations. The motion, observations, and task instruction form a canonical record. For each deployment controller, KODA optimizes commands to track the stored motion and records the corresponding action labels and proprioceptive states. When switching controllers, KODA retains this canonical record and regenerates whole-body state–action supervision for co-training with matching robot demonstrations. We evaluate KODA on four Unitree G1 loco-manipulation tasks using two whole-body controllers, SONIC and HEFT. With robot data held fixed, macro-average success on conditions covered by human demonstrations but absent from robot data improves by 45.0 and 46.25 percentage points, respectively. Under both controllers, this metric shows an upward trend across the evaluated human-data budgets, suggesting potential for further scaling.

Keywords: Egocentric Human Data, Humanoid Loco-Manipulation

††

Figure 1: Overview of KODA. Whole-Body Motion Generation produces simulation-feasible whole-body motion from human egocentric demonstrations, and Robot Onboard View Generation synthesizes motion-aligned onboard views. The motion, views, and task instruction form a reusable canonical record. Control Command Generation uses this record to generate controller-specific states and actions. The resulting data are combined with matching real-robot demonstrations to co-train a controller-specific vision–language–action (VLA) policy for language-guided humanoid loco-manipulation.
Figure 2: KODA pipeline. Middle: Human demonstrations become aligned whole-body motion and onboard observations, then controller-specific state–action data for VLA training (Control Command Generation). Top: Whole-Body Motion Generation combines hand estimation and contact-aware retargeting, sampling-based command optimization (SCO), and arm refinement. Bottom: Robot Onboard View Generation combines arm removal, scene reprojection, robot rendering, depth compositing, and inpainting. Input depth is observed or estimated from RGB.

References