EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
This preprint teaches robot hands from 20,000 hours of first-person human video without hand-written rules that map human fingers onto each robot. On five real tasks with three different hands and grippers it beat the usual retargeting approach (74% vs 65% success on unseen objects and positions) and cut the training time for adding new robots by 47%.
Can manipulation knowledge from egocentric human video be learned once in a robot-agnostic form and then attached cheaply to different robot hands, instead of being re-mapped by hand for each one?
Robot demonstrations collected by teleoperation are slow and expensive. Human video is plentiful, but the common way to use it, retargeting human wrist and finger poses onto a robot through geometric rules, must be recalibrated for every robot body and passes hand-tracking errors straight into robot commands. The authors measured human-robot calibration alone at 52.1 hours, 25.2% of their retargeting pipeline time, and saw bottle-cap success fall from 60% to 30% when 10 mm of noise was added to human hand annotations.
Action-free methods such as LAPA learn latent actions from frame-to-frame visual change, which can capture any visual variation rather than the effect of an action. Retargeting pipelines and DexWild use explicit hand poses captured with devices such as Aria glasses and map them to one robot's action space.
EgoJEPA has three stages. First, a JEPA-style model learns latent actions by predicting the visual result of a hand-motion change, so the action code reflects what the motion did to the scene. Second, a world model built on Alibaba's Wan2.2 video model jointly predicts future frames and these latent actions, still with no robot-specific commands. Third, small two-layer adapters connect the shared model to each robot's action space. Only the third stage is repeated for a new robot.
Pre-training used a purchased dataset of 1.11 million episodes (about 20,107 hours, 2.17 billion frames). Robot tests covered toy grasping on a Sharpa hand, cup stacking, zipper pulling and bottle-cap unscrewing on a Wuji hand, and box folding on a Xense gripper, 20 trials per task. Average success was 94% in-domain and 74% out-of-domain, against 80% and 65% for retargeting and 40% out-of-domain for DexWild. Without human pre-training the same model reached 40% out-of-domain. With 10% of the robot demonstrations it scored 35% on bottle-cap unscrewing, above the robot-only model trained on all of them (30%). Cumulative cross-robot training time fell from 743.2 to 392.7 hours.
The authors say contact-heavy tasks remain weak (bottle-cap unscrewing and box folding had the lowest out-of-domain scores) and that severe misalignment or long occlusions in human video still degrade results. Each setting has only 20 trials, so differences of 5 to 10 points carry wide uncertainty. The pre-training data is commercial and not described as released, and all tests are short tabletop tasks.
Hand makers and humanoid companies face a new retargeting job every time hardware changes. A shared model with cheap per-robot adapters would let Sharpa, Wuji, Xense and X Square Robot style teams reuse one human-video investment across product generations, and it raises the value of large egocentric video datasets that data vendors now sell. Expect such methods in lab and pilot systems within one to two years; contact-rich assembly will need force or tactile signals on top.