MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos
MoCaP, a preprint from New York University, turned one egocentric video of a person going about a room into guidance for a two-armed mobile robot and reached 80% success on two real fetch-and-grasp tasks (16 of 20 box grasps, 8 of 10 can grasps), where the CaP-X coding-agent baseline succeeded in none. The video told the robot where objects live and from which side people approach them.
Can ordinary recordings of people living in a space, with no task demonstrations, teach a mobile manipulator where to look for objects and where to park before it reaches for them?
Mobile manipulation fails less often at the grasp than before it: the robot cannot find an object that is out of view, or it stops at a spot from which its arm cannot reach. Teaching each task with robot demonstrations is expensive, and coding agents built for tabletop arms pick standing positions with simple geometry that ignores furniture.
Code-as-policy agents such as CaP-X let a language model write robot programs that call perception, navigation and grasping tools, but they search by spinning in place and choose a base pose by heuristic. Object-navigation methods such as CoW and ApexNav explore frontiers with open-vocabulary detectors and have no memory of the room before they start.
MoCaP first has GPT-5.6 watch an egocentric video recorded with Aria Gen 2 glasses and build a spatial memory of the space (where objects are) and an interaction memory (where the person stood before reaching for each object). At run time a closed-loop code agent uses that memory to navigate, then docks from the side the person approached, searches locally for a base pose from which a grasp passes collision and inverse-kinematics checks, and plans whole-arm motion with NVIDIA's cuRobo against the observed scene. The same memory serves every task.
On the YOR robot (omnidirectional base, lift, two AgileX Nero arms, ZED2i camera, Jetson Orin NX) MoCaP grasped a box on a cabinet in 16 of 20 trials and a can that started out of view in 8 of 10, 80% each. CaP-X and CaP-X+ (CaP-X with collision-aware navigation) scored 0 on both; CaP-X+ reached a grasp attempt on the box in 8 of 20 trials but often from 0.74 to 1.64 m away, and neither baseline ever found the can. In navigation to a blue bin out of sight, MoCaP succeeded 10 of 10 times against 7 of 10 without the video prior, 2 of 10 for ApexNav and 0 of 10 for CoW, which stopped at a white bin with a blue notice. The human approach prior raised the share of trials reaching a grasp-ready base pose from 18 of 45 to 38 of 45 (40.0% to 84.4%), from 0 of 15 to 14 of 15 at the start opposite the recorded human.
This is a preprint tested in one indoor environment with one video recording, three target objects and five to twenty trials per condition; the authors say it shows feasibility, not generality, and call the pooled base-placement gain not statistically significant at this sample size. The memory is a fixed summary built at startup and can miss objects or go stale, skills stop at grasp and lift, and the system leans on a proprietary language model (GPT-5.6) whose cost and latency per task are not reported. The base-placement section describes five trials per start while its table counts fifteen; we use the table.
The result matters to makers of home and service mobile manipulators, where setting up each customer's space is a large cost: a walk-through video by the resident could replace mapping and per-task teaching. It also argues for wearable capture (Meta's Aria glasses here) as a data source for robot companies building household robots. Time scale: research stage; a product would need many homes, more skills than grasping and a cheaper planner than a frontier model on every decision.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.