TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping
A grasp planner that asks a vision-language model where a tool should be held for a given job picked the right 6DoF grasp in 94.5% of 310 TaskGrasp benchmark cases, against 73.7% for the best prior method, and succeeded in 93 of 100 real trials on a Franka Research 3 arm, researchers from Purdue University and the US Army's DEVCOM Army Research Laboratory report in a preprint. It needs no task-specific training data.
Can a robot choose a grasp that suits the intended use of an object, such as holding a knife by the handle to cut but leaving the handle free to hand it over, from one RGB-D image and a text instruction, without training on task-labelled grasps?
A stable grasp is only half the job. Task-oriented grasping needs the grasp to fit the task, but learned methods are capped by finite hand-labelled datasets, scoring methods depend on a task-agnostic generator that may never propose the right region, and planar methods ignore grasps that approach from outside the image plane.
GCNGrasp propagates a knowledge graph over the TaskGrasp dataset, GraspGPT and FoundationGrasp add language-model knowledge to score candidates from a task-agnostic generator, GraspMolmo fine-tunes a vision-language model on a large synthetic corpus, and GraspCoT adds physical-property reasoning.
TGrasp6D masks and crops the object, asks a vision-language model which part of the object the task calls for and where it is in the image, then asks for the gripper's closing axis as a line through that point. Camera geometry lifts the point and line into 3D, forming a Gaussian prior around the task region. Candidate grasps sampled from that prior are refined by optimisation on the SE(3) Lie algebra, penalising collision with the object point cloud through a learned gripper distance field and rewarding grasps whose closing volume covers the task point. The main runs use GPT-5.5 as the vision-language model.
On 310 TaskGrasp object-task pairs (success means at least 0.1 IoU with an annotated grasp), TGrasp6D scored 94.5% with a 0% generation failure rate, against FoundationGrasp 73.7%, GCNGrasp 72.1%, GraspMolmo 65.0%, GraspGPT 61.7% and GraspCoT 7.5%, a 20.8-point lead. Pose refinement added 5.1 points over the vision-language prior alone (89.4%), cutting position failures from 3.5% to 1.6% and orientation failures from 3.9% to 0.6% while wrong-part failures stayed at 3.2%. Swapping the model before refinement gave 89.68% for GPT-5.5, 84.19% for Gemini 3.7 and 75.16% for Qwen3.5-35B, which failed to answer on 19.7% of inputs. On a real Franka Research 3 with five objects and ten object-task pairs, 10 trials each, it succeeded 93/100 times against 41/100 for GraspGPT and 52/100 for FoundationGrasp, with at least 8 of 10 on every pair.
One RGB-D view makes the method sensitive to depth noise and self-occlusion, which caused most real failures on the jug and the spray bottle. It depends on large hosted models, adding latency and API cost, it targets one task region at a time, and the refinement does not model contact mechanics or closed-loop correction. Failed grasps show larger orientation and position errors than trained baselines. Real tests cover five objects on one arm. The paper is a preprint.
Tool handover and grasp-for-use are everyday requirements for service, logistics and assembly robots, and today they are mostly hand-programmed per part. A zero-shot planner that more than doubles a learned baseline's real success (93 against 41 of 100) lets integrators handle new parts with a text instruction instead of new labels, at the cost of cloud-model latency. The Army Research Laboratory co-author points to defence interest in manipulation of unfamiliar objects. Expect pilots in pick-and-use cells within one to two years, once on-premise models close the gap with GPT-5.5.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.