AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning?

Existing 3D assembly systems are usually trained for one domain or category, require ground-truth part geometry or specialized pose predictors, and are not designed for interactive visual agents. General-purpose agents need a common environment that gives them rendered observations and executable pose edits, but no benchmark measured their cross-domain assembly precision, interaction behavior, and reliability at once.

Prior work relies on category-specific learned pose estimators such as DGL, SPAFormer, and Assembler, instruction-guided methods such as Manual-PA and Manual2Skill, and geometric fragment matchers such as GARF and RPF. They typically depend on mesh or point-cloud access, require assembly-specific pose prediction or fine-tuning, and are evaluated in isolation rather than through a shared interactive agent interface.

AssemblyWorld connects pretrained agents from GPT, Qwen, DeepSeek, and Claude families to a 3D environment through Model Context Protocol tools. Agents can request rendered 2D views, list object records, translate or rotate parts and groups, and inspect the result iteratively, without access to mesh vertices or faces. AssemblyWorldBench standardizes 100 tasks from PartNet, IKEA-Manual, AssemblyBench, and Fantastic Breaks with fixed initial layouts, reference conditions, and geometric evaluation. The authors also analyze interaction trajectories and test combining the agent with the geometric model GARF for refinement, all without assembly-specific fine-tuning.

Across the 100-task benchmark, Astra has 59.4% overall success rate, versus 50.0% for Fable and 44.4% for Opus, with 80.9% overall part accuracy for Astra. Open-weight systems lag: Qwen reaches 11.9% SR and DeepSeek 0%, with Qwen hitting 65 timeouts and 14 execution failures and DeepSeek 17 timeouts and 34 execution failures out of 100 runs each. Astra averages 5.8 minutes and 94 tool calls per evaluation, while Sol uses 8.7 minutes and 208 calls. On two-part fracture reassembly, Astra and Fable both reach 90% SR, but on image-conditioned PartNet tasks they drop to 40% and 25%. Against specialized methods, Astra achieves 85.35% PA / 63.60% SR on PartNet tables with image guidance, 93.27% PA / 75.44% SR on IKEA Chair/Table, and 78.32% PA / 55.91% SR on 279 AssemblyBench industrial objects. On 150 Fantastic Breaks fragments, Astra reaches 91.67% PA, compared with 98.53% for SARe-Gen and 97.28% for TORA-CKA. Reference images improve Astra, Fable, and Opus by 25, 20, and 15 SR points on PartNet benchmark objects; larger PartNet tests show +14.6 and +7.4 SR points for tables and storage. Combining GARF with Astra improves fracture PA from 83.67% for GARF alone to 93.00% when Astra refines GARF outputs, while GARF refining Astra drops PA to 87.00%.

The evaluation is purely in free-space simulation and does not establish collision-free or physically stable assembly. Tests use sampled objects, robustness checks are small and single-run, and the GARF refinement uses only two-fragment objects with a single seed. Unknown pretraining exposure to the benchmark datasets prevents contamination-free generalization claims. No physical robot validation is reported, and code availability is not stated. Precision remains a key gap: high part accuracy often masks one or a few residual positioning errors, and open systems suffer frequent timeouts and execution failures.