COWBOY: A Scalable Sim-to-Real Framework for Learning Contextual Whole-Body Manipulation
A single vision policy trained only in simulation lets a wheeled robot with an arm, a four-finger hand and a moving camera grasp everyday objects in 24 real scenes, from fridges to staircases, with 91% average success, Carnegie Mellon University researchers report in a preprint. In 8,000 held-out simulated scenes it scored 92.4%, against 63.1% for the ManipGen baseline and 0% for end-to-end reinforcement learning.
Can one policy learned in simulation decide where a mobile manipulator should park, look and reach, and how its hand should grasp, from the scene around an object rather than the object alone?
Sim-to-real grasping systems generalise across many objects but are usually trained in one fixed setting. A mobile robot in a home must also choose base position, camera view and arm path around shelves, drawers and clutter, which makes rewards and observations in simulation hard to design.
ManipGen chains point-cloud local policies with AIT* motion planning and a vision-language model that picks an intermediate target, which often leaves the arm near singularities. End-to-end reinforcement learning from point clouds fails once obstacles are present.
COWBOY splits the job. A vectorised whole-body controller built on geometric fabrics runs across thousands of simulated scenes on one GPU and brings the hand near the object while avoiding collisions and keeping the camera on target. Local reinforcement-learning experts, trained in Isaac Gym on small local scenes across hundreds of objects, handle the grasp. Both are distilled into one student policy that reads depth and lidar point clouds and controls a TidyBot++ base, a Franka arm, a LEAP Hand and an ARX X5 arm that carries the camera, 32 degrees of freedom in total.
In 24 real scenes never modelled in simulation, the policy achieved 91% average grasp success on a broad set of unseen everyday objects, including irregular and deformable ones, and chose top or side grasps and base positions from context. On 8,000 held-out simulated environments with 150 unseen objects, the student scored 94.7% on tabletops, 92.9% on drawers, 91.3% on boxes and 90.5% on shelves (92.4% average) against 95.6% for its teacher, with a 6.1% collision rate versus 4.3%. ManipGen averaged 63.1% (21.2% on shelves) and end-to-end RL 0%, or 96% when obstacles were removed. The controller brought the hand into the experts' working range in 98.4% of trials.
The authors note that the grasp is chosen from scene and object shape alone, while real tasks often need a specific grasp for the next use, which requires more conditioning. Shelf scenes needing side grasps were weaker because objects can be knocked over. Real-world results are reported as an average across scenes without a per-scene count in the text, and the task is grasping only, not full multi-step chores. This is a preprint and has not been peer reviewed.
Home and service robot makers need mobile manipulators that work in rooms they have never seen, and real-world data collection for every layout is too slow. Showing that a mostly simulated pipeline transfers to fridges, sofas and staircases supports the sim-heavy route taken by NVIDIA Isaac users and humanoid start-ups. Grasp-and-fetch in homes and stores is the near-term use, within two to three years, once task-specific grasps and recovery are added.