HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
HumanoidToolBench is a preprint benchmark and dataset for humanoid tool use, containing 18 tasks across three scenarios, three execution levels, and standard/decoy tool sets, along with 3,094 teleoperated demonstrations on simulated and real Unitree G1 robots. Seven policies were evaluated in simulation and three on the real robot, revealing large gaps between selecting the right tool and actually completing the task. Focused GR00T N1.7 probes further show reduced selection accuracy on unseen tools and continued execution under unrelated instructions.
How reliably do humanoid policies select a functionally suitable tool and then coordinate grasping, manipulation, and locomotion to complete a task, and how do these behaviors change with tool appearance and instruction wording?
Humanoids need tools to overcome their physical limits, but existing benchmarks either ignore functional tool choice, evaluate only visual question answering or fixed-base arms, or do not test humanoid tool use from selection through mobile execution. This leaves no standard way to distinguish whether a failure comes from choosing the wrong tool or from failing to execute the chosen tool correctly.
Earlier humanoid benchmarks such as HumanoidBench include a spoon and window-wiping task but do not ask the robot to choose among tools with contrasting functional properties. Tool-specific benchmarks are limited: PhysToolBench is visual question answering without physical execution, DexCraft is single-arm with the tool provided, and RoboWits is dual-arm but does not divide tasks into progressive execution levels. Other humanoid suites like SIMPLE and WOLF-VLA add VLA evaluation but do not combine functional selection, stationary and mobile use, and controlled decoy variations.
The benchmark defines 18 tasks as combinations of three scenarios (BallMove, BallRetrieve, IceBreak), three execution levels (L0 selection/pickup, L1 stationary tool use, L2 mobile tool use), and two tool-set modes (standard vs decoy). Scenes are built in MuJoCo with a Unitree G1 and 55 curated tool assets; policies receive head-camera images plus robot state and output whole-body targets through a decoupled control stack adapted from GR00T-WholeBodyControl. ToolBook provides 3,094 teleoperated demonstrations: 3,003 simulation trajectories and 91 real-robot demonstrations for L1 BallMove and BallRetrieve. Evaluation records interaction events such as candidate contact, correct first contact, correct-tool lift, and tool-target contact.
Simulation covered all 18 tasks with seven policies, and the real robot covered four L1 tasks (BallMove and BallRetrieve in standard and decoy modes) with three policies. The authors find substantial gaps between acquiring the suitable tool and completing the task, with further declines when locomotion and manipulation are combined. GR00T N1.7 probes show lower correct selection when suitable tools are unseen and continued execution when instructions are unrelated. ToolBook contains 3,094 trajectories total: 1,803 simulation L0 prefixes, 1,200 successful simulation L1/L2 demonstrations, and 91 real-robot demonstrations. Exact success-rate values from Table III are not included in the supplied text.
The paper reports that selection accuracy drops on unseen tools and that policies continue under unrelated instructions. Real-robot evaluation is limited to stationary L1 BallMove and BallRetrieve, excluding L2 mobile tasks and IceBreak. Only one robot platform (Unitree G1) is used, real-robot demonstrations are modest at 91, and the benchmark covers three tabletop scenarios, so generality to other humanoids, varied terrains, or broader tool-use tasks remains open.
Humanoid manufacturers and VLA developers—especially teams building on Unitree G1 or NVIDIA GR00T-style policies—could use HumanoidToolBench and ToolBook to benchmark selection-vs-execution failures, diagnose failure modes, and fine-tune policies. It could be adopted now for internal evaluation, and within 1–3 years help improve commercial humanoid tool-use if models are trained or fine-tuned on the data and extended to more diverse real-world tools and locomotion. Wider deployment depends on robust real-world perception, safety, and broader task coverage.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.