HandsOn-Benchmark: beste Roboterpolicy löst 29,1% von 50 Aufgaben mit Roboterhänden

\role

[*]Equal contribution. \role[†]Project lead. \role[‡]Corresponding author. \setheadertextHandsOn \reportnumber

Abstract

General-purpose manipulation requires reliable behavior across embodiments, objects, and task demands. We introduce HandsOn, a composable simulation platform and benchmark for cross-embodiment dexterous manipulation. It supports five robotic arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable assets in randomized scenes. The benchmark pairs 18,000 curated human demonstrations with 66 human-verified tasks: Core-50 supports demonstration-based learning and execution tests, while Transfer-16 evaluates asset and relation transfer, skill composition, and deformable manipulation without target-task training demonstrations. Qualitative real-robot rollouts illustrate sim-to-real transfer. Across diffusion, vision-language-action, world-action, and prompted VLM-based methods, the best trained multi-task policy achieves 29.1% Core-50 success. Without benchmark finetuning, GPT-6 using move_eef for palm-and-finger control achieves 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials. Trained-policy stage diagnostics expose failures in contact establishment, action coordination, and maintaining achieved goals, with additional sequencing errors in composed tasks. HandsOn provides a shared basis for investigating manipulation generalization and the limits of reliable physical interaction.

1 Introduction

Dexterous manipulation requires controlling how contacts change during an operation. Moving a mouse must not trigger a premature click; setting two knobs requires both knobs to remain at their target settings simultaneously; and placing a lid requires preserving alignment while releasing it. Evaluating these capabilities requires demonstrations of varied interactions and diagnostics that distinguish initial progress from reliable functional completion. Recent dexterous benchmarks broaden task and embodiment coverage [1, 2, 3, 4], but their human-demonstration sets are small and their tasks rarely include ones reserved for evaluation.

HandsOn provides 18,000 curated simulation demonstrations, approximately 56 hours of interaction with three synchronized camera views. This is 5.7–16.4 times the human trajectory count of representative dexterous benchmarks under the convention in Table 1. High-fidelity re-rendering produces varied observations of recorded behavior while preserving state/action supervision. Visual domain randomization and meaning-preserving prompt randomization broaden appearance and language coverage around task-specific configurations. Figure 1 summarizes the asset collection, environment and interaction diversity, demonstration scale, and Core-50 results. Figure 2 connects this data pipeline to evaluation and illustrates correspondence with the physical system.

The benchmark comprises two complementary suites: Core-50, 50 tasks with demonstrations for policy learning and fresh-rollout evaluation, and Transfer-16, 16 tasks reserved for generalization evaluation. Core-50 tasks combine functional and numerical understanding, articulation, spatial relations, precise contacts, and bimanual execution (Figure 3). Transfer-16 tests new asset relations and skill combinations, including three deformable interactions without deformable-task demonstrations in this dataset. The simulated embodiment follows a physical platform, and real-robot rollouts of simulation-trained checkpoints are shown in the accompanying videos.

Our Core-50 study compares ten methods: five VLA models, three WAM models, single-task Diffusion Policy, and the prompted VLM controller GPT-6. The best trained multi-task score is 29.1%. Across the trained checkpoints, tool use and precision fits are nearly unsolved, and skill-group gaps persist after adjusting for demonstration count and expert duration. Two VLA checkpoints barely condition on which object the instruction names. Failures concentrate where closing the grasp fires a button, where both hands must hold their targets together, and where reached goals must be kept. These findings motivate evaluating what happens after acquisition, alongside aggregate success.

We additionally study GPT-6 as a prompted multimodal controller, using RGB observations, measured robot state, and the move_eef palm-and-finger control interface without HandsOn finetuning. This supplementary experiment achieves 5.2% success on Core-50 and 1.9% on Transfer-16 over 660 trials. Success appears on only 12 of the 66 tasks. Qualitative action traces suggest task-level planning potential and motivate richer interfaces between that planning and high-dimensional dexterous execution. Its observation interface, trial count, and model-call budgets are disclosed alongside the trained-policy comparison.

Our contributions are:

1. A large human-demonstration dataset, re-rendered with visual and prompt randomization.
2. Core-50 and Transfer-16: 50 demonstration-backed tasks and 16 evaluation-only asset, skill-composition, and deformable tasks, scored by native stage checkers.
3. A comparison of DP, five VLA, and three WAM policies with a covariate-adjusted skill analysis and a language-conditioning probe, locating failures at three hand transitions and, in compositions, at step order and completion, together with a 660-trial GPT-6 move_eef evaluation without benchmark finetuning.

2 Related Work

DexJoCo, DexVerse, Bench2Dex, and LabDex cover functional and bimanual manipulation with diverse embodiments [1, 2, 3, 4]. DexMimicGen expands human seeds through automated trajectory generation, whereas SPD studies simulation pretraining and real-data adaptation [5, 6]. Table 1 compares data scale, randomization, policy coverage, and evaluation protocols. HandsOn evaluates DP, five VLA models, three WAM models, and a prompted VLM controller, GPT-6 with move_eef, across a demonstrated task suite and a separate dexterous task-transfer suite.

LIBERO, CALVIN, DexArt, RoboDojo, and RoboTwin motivate transfer, capability evaluation, and randomization [7, 8, 9, 10, 11]. Earlier dexterous suites learn mainly by reinforcement or from small demonstration sets [12, 13, 14]; large simulated and real manipulation corpora are dominated by grippers [15, 16, 17, 18, 19]. Dexterous teleoperation and motion capture [20, 21, 22] collect human data outside a benchmark; HandsOn records it in the evaluation simulator. Dexterous sim-to-real learning precedes HandsOn [23, 24]. Section 4.2 describes the dataset counting conventions. Appendix F presents qualitative real-robot rollouts, and Appendix B.5 specifies the proposed matched evaluation protocol.

3 The HandsOn Benchmark

HandsOn has two suites. Core-50 provides demonstrations for policy learning and fresh-rollout evaluation; Transfer-16 is reserved for generalization evaluation and contributes no target-task demonstration to the training set. Both suites are scored by native, task-specific success predicates rather than by a shared proxy, and every task was verified in person by the data-collection operators before release (Section 3.2).

3.1 Core-50 and Transfer-16

Core-50 tasks combine functional and numerical understanding, articulation, spatial relations, precise contacts, and bimanual execution (Figure 3). Transfer-16 tests new asset relations and skill compositions, including three deformable interactions for which this dataset contains no deformable-task demonstrations. Because the two suites answer different questions, we report them separately throughout: a Core-50 score measures execution on tasks with demonstrations, whereas Transfer-16 tests held-out tasks. The learned multi-task methods train on Core-50 before both evaluations; GPT-6 is evaluated on both suites without HandsOn demonstrations or finetuning.

3.2 Task Design and Feasibility Verification

Tasks and demonstrations pass a written collection-and-review procedure with separated roles: data-collection operators record and self-check episodes, independent reviewers judge them, and a third person arbitrates disagreements; nobody reviews their own data. Each task has a task card that fixes the goal, the standard initial and final states, completion evidence independent of the program’s success flag (object poses, joint states, or images), allowed and forbidden actions with known shortcuts, the time limit, and the environment, task, and checker versions with their code hashes.

A new task or major revision is first piloted with at least 20 operator episodes, all reviewed by hand and covering success, failure, time-out, reset, and boundary states; it enters collection only with no severe defect, at least 98% agreement between labels and the objective evidence, and unambiguous rules. During collection the environment and checker are frozen by hash. A checker change passes reproduction, impact analysis, two-person review, regression tests on normal, failing, boundary, and known-defect cases, and a new fully reviewed pilot; affected episodes are replayed, relabeled, or quarantined. Every recorded episode then passes automatic checks of completeness, version hashes, action plausibility, agreement between the success flag and the final physical state, and duplication. Stratified random blind review covers about 20% of each batch (all of the first batch of a new task or version), and a further 5–10% of passed episodes are blindly re-reviewed. A success flag without objective completion quarantines the task as a suspected checker defect until it is repaired and piloted again.

This procedure matters for how the results should be read. Every Core-50 and Transfer-16 task was verified in person by the data-collection operators in the released environment before evaluation. Each Core-50 task is also completed by its curated demonstrations. For 47 of the 50 Core-50 tasks, at least one evaluated multi-task checkpoint succeeds in at least one episode. Human verification establishes task feasibility under the verified configurations. The low policy success rates indicate substantial performance gaps under the evaluated protocol (Section 5.3.3). Sources adapted from prior benchmarks, and the modifications made to them, are recorded in Table 11.

3.3 Evaluation Protocol

Simulation uses the fixed native end-effector execution protocol. Learned policies retain checkpoint-owned representations, preprocessing, and normalization; GPT-6 uses the adapter in Appendix C.2. Actions are executed at 30 Hz, with each action advancing the simulation by exactly 1/30 s; integration steps are recorded separately. The small source cohort collected with another controller is identified rather than assumed equivalent.

Time budgets.

Physical-time budgets are fixed before evaluation and shared across methods; GPT-6 additionally has the recorded model-call budgets described below. For a Core-50 task i, let F_{i,(k)} denote its demonstration lengths in frames, sorted in ascending order with zero-based index k. The frozen duration table uses the non-interpolated quantile

q_{i}=F_{i,(\min(n_{i}-1,\lfloor 0.75n_{i}\rfloor))},\qquad H_{i}=3q_{i}/30,\qquad B_{i}=\lceil 30H_{i}\rceil.

Here H_{i} is the budget in simulated physical seconds and B_{i} the maximum number of executed actions. Source demonstrations are recorded at 30 fps and evaluation advances exactly 1/30 s per action, verified from the physics timestep and substeps. Inference latency, rendering time, and action-chunk length do not redefine this physical budget. The released table is computed from the curated package’s source cohorts, including its trajectory validation partition, without using evaluated policy outcomes.

For the 13 non-Flex Transfer-16 tasks no target-task demonstration distribution exists. We instead use H_{i}=3\sum_{j\in\mathcal{S}_{i}}q_{j}/30, where \mathcal{S}_{i} is a predeclared mapping to supporting atomic tasks. This is a duration proxy, not the measured P75 of composed trajectories. For example, Open Laptop and Click Mouse uses (196+125) source frames, giving 32.1 s or 963 actions. The three Flex tasks retain explicit native budgets: 240 s for sponge compression and 180 s each for towel folding and pillow packing. Table 12 lists all Transfer-16 budgets and their mappings.

Stopping and success.

After every executed action the native environment checks success, task-defined failure, and timeout. The rollout stops at the first terminal outcome, including within an action chunk; unused predicted actions are discarded. Success requires the complete native predicate, including any release, stability, or dwell requirement. Native failure terminates with zero success; exhausting the budget without success also scores zero. Success reached on the final permitted action is accepted. There is no generic early stop for apparent stagnation or for completing only an intermediate stage. Task-specific irreversible violations retain their documented native rules.

The GPT-6 move_eef experiment also uses a model-call budget, which can end an unsuccessful trial before its physical-time limit. Its recorded context and budget settings, outcome acceptance, and comparability are described in Appendix C. This prompted controller uses no HandsOn finetuning and is reported separately from the trained-checkpoint aggregates.

Validity and attribution.

An invalid policy output, such as a nonfinite action, is recorded as a policy failure under the declared adapter. Simulator defects, rejected resets, process interruption, and infrastructure failures are recorded separately and do not become numerical policy failures. Wall-clock watchdogs bound infrastructure execution only. Results retain executed steps, physical time, terminal reason, validity, and protocol revision; historical longer-budget successes are never relabeled as successes within a later budget. A repaired environment requires fresh target-task trials, and missing tasks and model/task coverage remain explicit in aggregates.

3.4 Diagnostics and Statistical Reporting

Success and failure alone do not say where a policy stops working. We record native-stage diagnostics where supported by the task checker and report their coverage explicitly.

Task success and ordered progress.

Primary scores are per-task success and macro-averages over declared task sets. Native executable success predicates and physical-duration holds determine completion. For a task with K_{i} stages, ordered progress is P_{e}=L_{e}/K_{i}, where L_{e} is the longest valid ordered prefix completed in episode e. Stages are defined by semantic operations; merely approaching or touching an object does not complete a functional stage. Progress never overrides the final success predicate. Ordered progress is reported for the trained multi-task methods; GPT-6 has no graded-progress measurement.

Capability-specific diagnostics.

Functional completion separates acquiring a tool from operating it. Stability measures include drops, slip, and displacement of an object supported by the other hand when these quantities are observable. Deformable tasks additionally report task-specific geometric progress. Contact-force measures require validated sensors or simulator measurements; command magnitude is never treated as measured force. We distinguish tool acquisition from functional activation, requested numeric setting from any actuator movement, and articulated travel from mere object contact. Capability labels overlap and use disclosed membership; their averages are not added into a new overall score.

Trials and uncertainty.

Trials preserve task-local resets, assets, source/goal weights, state/action semantics, and checkpoint-owned preprocessing and normalization. Independent evaluation seeds are shared across compatible model comparisons. Binary success receives Wilson intervals; aggregate uncertainty is reported with the resampling unit specified. Missing or environment-invalid trials are not numerical policy failures. Every aggregate identifies its task and trial coverage, and checkpoint selection, benchmark-development exposure, and historical data exposure are disclosed in the run record.

Reading the diagnostics.

Native event-arrival rates and ordered completion scores are different quantities. A conditional transition rate requires joint, order-consistent events, not division of unrelated stage marginals. Retention diagnostics identify both the achieved condition and the subsequent observation interval; success-triggered termination can give episodes unequal exposure. Recorded action commands are not measured wrist motion or contact force.

4 The HandsOn Data Platform

HandsOn pairs a teleoperated demonstration corpus with a re-rendering pipeline. Collection produces exact state and action supervision at the cost of a fixed set of recorded appearances; re-rendering then varies the observations around that recorded behaviour while the interaction and its labels stay fixed. The two halves are described in turn.

4.1 Collection System and Operators

Demonstrations are collected in NVIDIA Isaac Sim 5.1 [25]. MANUS Prime I gloves capture finger articulation, and wrist-mounted HTC VIVE Trackers provide global wrist poses [26, 27]. After frame registration and temporal alignment, wrist motion and a palm-centred 21-keypoint hand representation are converted to commands. The arm branch produces feasible end-effector targets through robot kinematics; Wuji retargeting maps hand keypoints to joint targets, with 20 outputs for the first-generation Wuji Hand [28]. Simulation records synchronized observations, measured state, commanded actions, language, and source metadata (Figure 4).

At least ten operators contributed over 1,050 collection sessions, which is why the feasibility procedure of Section 3.2 separates recording from review. All demonstrations are simulated: no physical robot recordings enter the training set, and the physical platform of Section 4.5 is used only for the qualitative rollouts of Appendix F.

4.2 Dataset Composition and Identity

The released dataset contains 18,000 curated trajectories: 15,178 training and 2,822 validation. Selection uses task-group quotas of 500, 300, or 100 trajectories, favouring shorter eligible demonstrations within each quota. This is a curated distribution, not an unbiased sample of all collected behaviour. Three stages of one multi-stage source task share a quota, giving 52 quota groups. Three simultaneous camera views do not multiply the interaction duration. Across the package the median trajectory has 262 steps (10th–90th percentile 96–705); per Core-50 task, training trajectories range from 46 to 463 (median 268) and expert P75 durations from 2.5 to 49.2 s (median 12.7 s). Split membership is defined at trajectory or session level and re-rendered derivatives inherit it; normalization fitted for this package uses training data only, and a checkpoint that inherits another normalization or has historical exposure retains that provenance.

Table 3 lists the counting units that recur through the paper. Keeping them separate matters: the 54,000 video streams are three views of the same 18,000 interactions, and the raw collection before deduplication and curation is not the training size of any checkpoint.

The evaluation catalog contains the Core-50 tasks and excludes one source task, Chinese Chess, for every model. This is a scope exclusion: it does not remove demonstrations from the 18,000-trajectory package and does not rewrite any checkpoint’s training provenance. A mapping record links each paper task to its source task or cohort, goal, asset identities, episodes, and executable evaluation task.

4.3 High-Fidelity Re-rendering

Re-rendering varies appearance while replaying the recorded interaction. Final-package capture records use 32 samples per pixel; higher-sampling presentation renders are identified separately. The implemented bank includes four table materials, three background configurations, and three ceiling styles, with texture scale and orientation, surface roughness and tint, and illumination direction, intensity, and colour-temperature variations. One verified source camera configuration samples each camera once per trajectory within \pm 2 mm translation and \pm 0.3^{\circ} rotation per axis, retaining intrinsics. Episode records identify source trajectory and runtime, render revision, appearance seed and configuration, and camera configuration. Appearance is sampled per trajectory and remains fixed during replay; independently resampling appearance every frame is outside this static-domain protocol, and any temporal sensor noise would require a separate model.

Not every factor can be varied by replay alone. Table 4 states what each factor must preserve. Background and appearance changes can be applied during replay because the recorded actions remain valid; geometry and dynamics changes alter the reachable interaction and require valid re-execution rather than frozen-state replay. We use the taxonomy of Table 4 as the rule for adding variation: a factor is admitted only with the evidence its row requires.

Camera perturbations are composed with task-specific nominal calibration. Within the trained-checkpoint Core-50 evaluation, which of the four table materials, three colour palettes, or two roughness levels is drawn does not measurably change success (Section 5.3.4). Those factors lie inside the training distribution, so this is an invariance check rather than a visual-generalization result.

4.4 Language Bank and Prompt Sampling

The frozen language bank contains 108 task–instruction entries, of which 91 occur in the final 18,000-trajectory manifest. For these 91 active entries, there are 32 training and eight held-out paraphrases per entry, yielding 2,912 training and 728 held-out paraphrases. For example, “Pick the apple and place it into the bucket” and “Grab the apple and put it in the bucket” preserve the same goal. Replacing apple with banana, changing a knob angle, or changing operation order instead requires an episode with the corresponding executable goal, which is why those variations are task variations rather than paraphrases. Held-out strings have no exact overlap with any active canonical or training string; this is a lexical disjointness check, and semantic equivalence additionally requires preserving the episode’s target and constraints. Inactive bank entries are not counted as demonstrated goals.

The packaged sampler hashes the language seed, epoch, episode identity, frame index, draw index, and split to choose an instruction, and offers training, canonical, and unseen modes. The reference training mixture is 20% canonical and 80% training paraphrases; fixed validation uses the run’s declared language mode. Language sampling does not resample the scene, cameras, or physics. A model record must identify the language bank revision, the actual sampling policy, prompt preprocessing, and whether conditioning is text, a structured goal, or a fixed task identifier. Section 5.3.2 examines whether policies condition their actions on the object specified in the instruction.

4.5 Physical Platform and Simulated Correspondence

The platform uses ROKAE AR5-08 arms (seven degrees of freedom, 5 kg rated payload, 683 mm reach) and first-generation Wuji Hands with 20 actively controlled joints per hand [29, 30]. The simulated kinematics, asset scale, cameras, state and action frames, and timing follow this physical system, and the correspondence record is given in Appendix B.1.

4.6 Portability Across Arms, Hands, and Physics Backends

HandsOn is organized around a task and asset layer that is independent of the particular hardware that executes it, so that a task definition, its assets, its success predicate, and its demonstration set survive a change of arm, hand, or physics back-end. The released code supports the substitution of several commercial arms, including the ROKAE arms used in the reported experiments and additional supported arms from KUKA, UR, ABB, and FANUC; several dexterous hands, including the first- and second-generation Wuji Hands and the Sharpa Hand; and two physics back-ends, NVIDIA PhysX and Newton. The benchmark therefore measures a manipulation capability rather than a specific piece of hardware: the same Core-50 and Transfer-16 definitions can be instantiated against any supported arm–hand pair, and the demonstrations can be re-collected on that pair without editing the task itself.

All experiments reported in this paper use one configuration: the ROKAE AR5-08 arms and first-generation Wuji Hands of Section 4.5, with NVIDIA PhysX as the physics back-end. This is the configuration of the 18,000-trajectory corpus, of every collection session, and of every Core-50 and Transfer-16 result in Section 5. The other arms, hands, and back-ends are supported by the platform and are the natural basis for studying transfer across embodiments, but they do not contribute episodes to the results reported here, and the numbers in this paper should not be read as performance on those configurations.

5 Experiments and Analysis

We evaluate five VLA models, three WAM models, single-task DP, and GPT-6 with move_eef on Core-50. Section 5.1 describes the experimental setup. Section 5.2 presents the main results on Core-50 and Transfer-16, and Section 5.3 examines demonstration scale and task duration, instruction conditioning, failure modes, task grouping, and evaluation reliability. GPT-6 has ten trials per task on both suites. Tables 5 and 6 include this prompted reference, with its full results in Section 5.2.3. Skill-group and per-task comparisons and outcome summaries also include GPT-6 where measured results are available. Trained-policy averages and analyses of learning, stage transitions, and checkpoint reliability retain their original populations.

5.1 Models and Experimental Setup

We evaluate ten methods: single-task DP, five VLA models, three WAM models, and GPT-6 as a prompted VLM controller. The eight VLA/WAM checkpoints are trained on the Core-50 demonstrations. The five vision-language-action (VLA) models are \pi_{0.5} [31], DM0.5 of the DM0 family [32], LingBot-VLA [33], HY-Embodied-0.5-VLA [34], and Spatial Forcing [35], which aligns the visual features of \pi_{0.5} with a pretrained 3D foundation model during training. The three world-action models (WAMs), OpenWAM [36], FastWAM [37], and Motus [38], combine action prediction with modeling of future observations. Each checkpoint executes its integration’s default action chunk per query (Table 5, Steps/plan). The selected checkpoints are \pi_{0.5} at 120,000 updates, DM0.5 at 86,000, OpenWAM at 70,000, Motus and FastWAM at 100,000, Spatial Forcing at 90,000, HY-Embodied at 125,000, and LingBot-VLA at 151,000; they are fixed, not chosen by benchmark success. The single-task reference, Diffusion Policy (DP) [39], trains one 30,000-update model per Core-50 task. Without a predeclared routing protocol it is not run on Transfer-16; this is not a measured zero-shot failure. Pretraining data and compute differ across models, so the comparison does not isolate architecture.

GPT-6 with move_eef.

We additionally evaluate GPT-6 Astra (API identifier gpt-6-astra, reasoning effort medium) without benchmark finetuning or demonstrations in its prompt. At each turn it receives three RGB views, measured palm poses and finger joints, the task instruction, and execution feedback. Its only callable function, move_eef, specifies absolute palm poses and 20 independent finger-joint targets per hand. A kinematic adapter converts these targets to coordinated 30 Hz trajectories for the shared native executor, with no learned grasp primitive; simulation pauses during API inference. The batch assigns ten fixed trials to each of the 66 tasks, 500 on Core-50 and 160 on Transfer-16. We use ten trials per task because of API-cost and wall-clock evaluation-time constraints, allocating the available budget across the full task catalog. These trial counts, the prompted interface, and the additional call budget differ from the trained-policy experiments. Appendix C specifies the prompts, controller, recorded budgets, and interruption handling; Section 5.2.3 reports the completed 660-trial experiment.

5.2 Benchmark Results

5.2.1 Performance on Core-50

Table 5 reports all 50 Core-50 tasks for the eight multi-task checkpoints and the DP and GPT-6 references (per task in Appendix D). For analysis we group the tasks by the hand skill they demand (Section 5.3.4; Figure 5a).

Policies move objects and mechanisms; tools and precision fits remain unsolved.

Among the eight trained multi-task checkpoints, OpenWAM has the highest success, 29.1%, and picking the best checkpoint per task would give 35.4%. Success is ordered by what the task asks of the fingers beyond closing them. Pushing or flicking without a grasp (two tasks, 51% pooled over checkpoints), moving a hinge, drawer, lever, or lid (43%), and grasp–transport–release including handovers and object selection (35%) are the relatively better-performing groups in the suite; all nine tasks with mean success above 40% are of these kinds. Operating a trigger, button, knob, slider, or shutter with the fingers drops to 14%, pouring to 13%, and regrasping or reorienting an object in or between the hands to 11%. Two groups are essentially unsolved: acting on a third object through a held tool (137 of 4,800 episodes, 2.9%; hammer, spoon, set square, and spatula together 9 of 3,200) and precision fits such as peg insertion, stacking, or face alignment (107 of 4,800, 2.2%). The best checkpoint on any of these twelve tasks reaches 21% (OpenWAM on Stir Pot); on ten of them none exceeds 14%. Skill-group gaps persist after adjusting for demonstration count and expert duration: precision fits have more demonstrations per task (median 413) than grasp–transport–release (392), and with log demonstration count and log expert duration as covariates the tool, precision, in-hand, and finger-actuation groups stay 20–25 points below grasp–transport–release, with task-bootstrap intervals excluding zero (Section 5.3.1).

Checkpoints differ in using a grasped object, not in grasping it.

On grasp–transport–release tasks four checkpoints are within two points (OpenWAM 44, \pi_{0.5} 43, FastWAM 42, DM0.5 42), and the OpenWAM–\pi_{0.5} gap on primitive tasks is +0.5 points (paired task bootstrap 95% interval -6.9 to +7.6). The same gap on functional tasks is +9.2 (+4.7 to +13.9): Pour Seasonings 60 versus 34%, Move and Click Mouse 44 versus 16, Press Soap Pump 33 versus 9, Stir Pot 21 versus 4. Section 5.3.3 locates this gap at specific transitions: OpenWAM clicks before the mouse is ready in 16 of 100 episodes and \pi_{0.5} in 67, OpenWAM keeps the laptop display at the requested angle in 67 and \pi_{0.5} in 42, and OpenWAM completes the handover to the receiving hand in 41 and \pi_{0.5} in 20. Two VLA checkpoints place fixed objects as well as the leaders but collapse when the instruction selects the object: HY-Embodied and LingBot-VLA score 55 and 61% on Place Cup on Plate (\pi_{0.5} 62) but at most 22% on Place Ball in Bowl, Place Fruit in Bucket, and Place Largest Fitting Ball, where \pi_{0.5} scores 35–75%. They barely read the object name: in an offline probe (Section 5.3.2), naming the other ball moves the 1-s palm endpoint of \pi_{0.5} and DM0.5 by twice the resampling change (15 and 17 versus 7 and 8 cm) but that of HY-Embodied and LingBot-VLA by no more (7–9 cm), and with the swapped name they still head for the original ball in 57 and 71% of first frames (DM0.5 10%). Motus is a strong picker (76% on Place Ball in Bowl) but near zero wherever a mechanism must be held open or a load carried (Open Micropipette Box 8%, Open Drawers 4%, Lift Carton 8%). Spatial Forcing is weakest in every group and, unlike all other checkpoints, worse on single-arm (3.8%) than on bimanual tasks (14.5%).

World-action models finish later but finish more.

The three WAM checkpoints complete their successes at a median of 1.13–1.27 p75 durations versus 0.88–0.99 for the four 30-step VLA checkpoints, and more of their time-outs are still making progress after 1\times (14.5–18.7% of episodes versus 10.1–11.8%). Counting only completions within 1\times (Table 5, @1\times), \pi_{0.5} leads with 14.6% and OpenWAM ranks sixth with 8.3%; 12 of the 28 checkpoint pairs change order between the two limits (Figure 6). The WAMs execute 16 control steps per query, but Spatial Forcing, a VLA that also executes 16, is not slow (median 1.09), so the query cadence alone does not explain the difference; @1\times is therefore the secondary score (Section 3.3).

Reliability and feasibility.

The 95% interval over tasks and episodes (22.8–36.0% for OpenWAM, Table 15) mainly reflects the task mix (episodes alone: 28.0–30.1%). Paired task resampling orders \pi_{0.5} above DM0.5 in 99% of draws but HY-Embodied above Motus in only 61% (Section 5.3.4). Leaving out any one task swaps at most HY-Embodied and Motus and moves no score by more than 1.4 points. Every task is completed by verified human demonstrations (Section 3.2). For 47 of the 50 Core-50 tasks, at least one evaluated multi-task checkpoint succeeds in at least one episode.

Single-task reference.

DP averages 9.7% over 50 tasks: 38 use an earlier goal sampler, 11 use the current protocol, and Lift Carton is evaluated separately (Appendix D). On the 11 matched tasks it reaches 9.9%, against 22.7–34.5% for seven multi-task checkpoints and 9.3% for Spatial Forcing. This does not separate multi-task data sharing from pretraining or model scale.

5.2.2 Generalization to Transfer-16

Table 6 groups Transfer-16 results by asset/relation transfer, skill composition, and deformable manipulation, with ordered progress crediting the longest completed prefix of stages. Transfer-16 is much harder than Core-50, and we rank trained checkpoints by ordered progress, which also credits partial completion. OpenWAM’s 12.9% exceeds HY-Embodied, LingBot-VLA, and Spatial Forcing by 6.0, 3.7, and 2.6 points (paired intervals exclude zero); the other four are within 1.7 points.

Human verification establishes the feasibility of Transfer-16 tasks under the verified configurations (Section 3.2). The low success rates indicate substantial policy performance gaps under the evaluated protocol. For the trained checkpoints, success is close to zero on all three families, which makes aggregate success a weak discriminator: ordered progress separates the checkpoints where success cannot. The three deformable tasks are unsolved by every checkpoint, and the compositions, which require completed skills to be chained, are the hardest family after them (Section 5.3.3).

5.2.3 GPT-6 with move_eef

GPT-6 achieves 26 successes in 500 Core-50 trials (5.2%) and three in 160 Transfer-16 trials (1.9%), for 29/660 overall (4.4%; Table 7). All 66 tasks have ten scored outcomes. No demonstrations or HandsOn policy training are provided. The model chooses palm poses and individual finger targets from RGB observations, measured robot state, the task instruction, and execution feedback.

Success is concentrated in a small part of the catalog.

Only ten Core-50 tasks and two Transfer-16 tasks have any successful trial (Appendix C.6). Rotate Propeller has the highest observed success, 7/10; Orient Cup Upright and Lift Carton each have 4/10, followed by Open Micropipette Box and Place Cup on Plate at 3/10. The remaining five solved Core-50 tasks each have 1/10. Across the primitive, functional, and contact-rich categories, success is 7.6%, 1.1%, and 7.3%, respectively (Figure 7a). The contact-rich score comes entirely from Rotate Propeller and Lift Carton; it does not indicate broad competence in precision fitting. Transfer-16 successes occur only on Open Trash Bin (2/10) and Orient Bottle Upright (1/10), giving 4.3% on asset/relation transfer and no observed success on skill composition or deformable manipulation.

Call budgets and physical failures both limit completion.

Of the 660 trials, 260 (39.4%) end at their applicable model-call budget, 150 (22.7%) reach the physical-time limit, and 221 (33.5%) end through a native termination or task-rule violation without success (Figure 7b). These are distinct from the 29 successes. Budget stops are particularly common in the composition and deformable families (31/60 and 24/30). They show that the prompted controller often cannot complete within its available interaction budget; they do not establish what an unlimited-call controller would achieve.

Zero-shot task-level planning signals.

Saved action notes suggest task-aware strategies despite low completion. For example, in Open Laptop and Click Mouse, GPT-6 cautions that “neither object has yet visibly moved, so do not lift or click yet.” In Fold Towel Twice, it proposes a small test lift to “verify that the hem follows before carrying it forward for the first fold.” Both trials fail, but their notes articulate prerequisites and attempts to monitor execution. These illustrative traces motivate studying GPT-6 as a high-level planner. They do not measure planning accuracy across the catalog or independently verify the physical states described by the model. Here zero-shot refers to the absence of HandsOn demonstrations and finetuning; the task instruction, robot contract, and static guidance remain available.

High-dimensional dexterity calls for a richer agent harness.

The current harness exposes only move_eef. Alongside choosing subgoals, GPT-6 must specify each palm pose and up to 40 finger-joint targets, infer object geometry and contact from images and robot/execution feedback, and recover from rejected or ineffective motions. It has no grasp primitive, obstacle-aware planner, explicit object-pose estimate, or tactile feedback. The inspected traces include repeated kinematic rejections and attempts to establish or recover grasps. This suggests a plausible gap between task-level strategy and its grounding in reliable high-dimensional control. A promising design is to pair the multimodal planner with estimated object/contact state and reusable, feedback-driven manipulation skills. The present experiment motivates that design; it does not establish how much of the failure rate is caused by the harness or how much a richer interface would improve success.

Scope of the comparison.

The Core-50 score is below every evaluated learned-policy reference, but measures GPT-6 together with this minimal harness. The RGB-plus-state interface, static task guidance, variable execution length, and call budgets differ from the trained-policy conditions. Isolating planner capability and the benefits of richer tools requires matched evaluations that vary the interface and feedback while controlling the task population and budgets. Appendix C records the implemented interface, settings, scoring qualifications, and uncertainty.

5.3 Detailed Analysis

5.3.1 Does Demonstration Scale Explain the Remaining Gap?

We finetune \pi_{0.5} separately on Place Ball in Bowl (Task 24) and Squeeze Scissors (Task 43) on nested subsets of 10, 50, 100, 200, and all 438 and 439 training trajectories, each run from the same pretrained initialization for 20,000 updates at batch size 32, with one training seed and the same 100 evaluation seeds, goals, and prompts per task (Appendix E).

Success rises at every tested size (Figure 8a): Task 24 from 7% to 83% and Task 43 from 1% to 44%, and going from 200 trajectories to the full pool still adds 18 and 26 points. The functional scissors task remains much harder. These task-local curves do not establish a cross-task scaling law or isolate an architectural cause, and normalization uses the full training pool at every size.

We also examine whether the Core-50 skill-group gaps persist after adjusting for demonstration count and expert duration. The hand-skill groups differ in how many training trajectories their tasks have and how long the expert takes (Table 8), so group differences could reflect data volume or episode length rather than the skill. Across the 50 tasks, mean success over the eight checkpoints has Spearman \rho=0.27 with the number of training trajectories and \rho=-0.71 with the expert P75 duration. Regressing task success on the group indicators alone explains R^{2}=0.62 of its variance; the two covariates alone explain 0.37. With both covariates added (R^{2}=0.73), duration remains negatively associated with success, whereas the interval for demonstration count includes zero. The finger-actuation, in-hand, tool, and precision groups remain 20–25 points below grasp–transport–release. The in-hand and tool groups have few demonstrations, but precision fits have a similar median demonstration count to the higher-performing grasp–transport–release group. By quota, the seven 100-trajectory tasks average 6.1%, the 24 300-trajectory tasks 23.4%, and the 19 500-trajectory tasks 24.4%; within the 500 quota, the four precision fits average 2.8% and the eight grasp–transport–release tasks 37.8%.

5.3.2 Do Policies Read the Instruction?

To test whether a checkpoint reads the object name, we query it offline on 500 demonstration episodes of Place Ball in Bowl (training and validation trajectories) at frames 0 and 15, 1,000 inputs in total. For each input we sample twice with the true instruction and once with the instruction naming the other ball, and compare the palm position commanded 30 steps (1 s) ahead. The probe uses checkpoints of the evaluated training runs, within 6,000 updates of the evaluated ones. For \pi_{0.5}, DM0.5, HY-Embodied, and LingBot-VLA, respectively, swapping the object name shifts the predicted palm endpoint by 14.8, 17.0, 7.3, and 8.5 cm, versus 7.0, 8.2, 8.8, and 8.1 cm between two correct-instruction samples. In the same model order, the correct-instruction endpoint is closer to the demonstration than the swapped one in 92%, 88%, 61%, and 71% of inputs.

For a target-level test we use the \pi_{0.5} true- and swapped-instruction endpoints as references for the two balls and keep the 314 first frames where they are at least 8 cm apart: DM0.5 heads for the named ball in 93% of them and, with the swapped instruction, still for the originally named ball in 10%; HY-Embodied does so in 58 and 57%, and LingBot-VLA in 74 and 71% (an earlier LingBot-VLA checkpoint: 55 and 55%). A logistic classifier on the frame-0 layout reaches 53% cross-validated accuracy, near the 52% majority baseline. Together with the weak response to instruction swaps, these results suggest limited instruction conditioning in HY-Embodied and LingBot-VLA; scene-dependent memorization is one possible explanation. The serving and offline paths produce bitwise-identical actions, ruling out discrepancies between these two execution paths.

5.3.3 Where Do Episodes Fail?

Averaged over the eight trained multi-task checkpoints, 21.4% of Core-50 episodes succeed, 13.9% end early, and 64.8% time out, mostly as stalls (Figures 8b and 9): 65–83% of instrumented time-outs record no new checker state after 1\times p75. The hands keep moving in a stall: in the last third of time-outs the commanded mean finger angle still reverses 3–4 times per 10 s while its range shrinks from 0.35–0.39 to 0.20–0.28 rad. Acquisition is not the main loss on Core-50. From native checker state we locate most remaining losses at three transitions of multi-finger bimanual control, pooling 800 episodes per task over the eight checkpoints (Figure 5b–d).

Forming a grasp can fire the function.

The fingers that hold a tool often rest on its button, trigger, or handles, so closing the grasp can itself trigger an irreversible event. Move and Click Mouse ends as soon as a button is pressed before the mouse has moved 8 cm onto its pad. This ends 443 of 800 episodes, 404 of them before the mouse was ever ready and 297 before it had moved at all (Figure 5b); of the 362 episodes that reach the ready state without an early click, 196 succeed; the checkpoints differ mainly in the early clicks (16 for OpenWAM, 78 for Spatial Forcing). In Photograph Logo the first shutter press precedes aiming in 774 of 778 episodes, and 163 of the 185 episodes that ever aim with the right hand on the shutter succeed. In Water Plant the trigger is a separate finger joint and the grasp is clean, so the loss moves to that finger: of 514 episodes that aim the held sprayer, 238 never spray, 178 of them after moving the trigger at least halfway. In Squeeze Scissors the same fingers must hold and cut: failing episodes close the blades 1,368 times but score 182 cuts, because at 57% of the closings the scissors are not lifted and at 26% a handle contact has been lost. Where the functional part lies under the holding contacts, grasping fires it early; where it is a separate finger, that finger is not driven to threshold at the right moment; where holding and actuating share fingers, the policy actuates only when it stops holding.

Each side reaches its target, but not at the same time.

Set Dual Knobs requires both knobs within 5^{\circ} of their targets at once, nearly still, held, and then released. Of 800 episodes, the left and right knob reach their target in 411 and 416, both at some point in 219, both at once in 38, and 5 succeed (Figure 5c). When the first knob arrives, the other is still a median 64^{\circ} away; the first knob then leaves its 7^{\circ} exit band in 570 of 608 episodes after a median 0.33 s, in 505 cases by overshooting and in 566 with the hand still in contact. The hands turn each knob through its target instead of holding it there while the other hand finishes. Timing alone is not the obstacle: in Pour Seasonings the two streams overlap in 306 of the 398 episodes where both hit their targets.

Reaching a goal and keeping it are different skills.

In Open Laptop, Spatial Forcing and OpenWAM bring the display to the requested angle about equally often (73 and 71 of 100 episodes), but it is still there at the end in 23 and 67 (Figure 5d). Of the 154 pooled episodes that lose the angle, 148 end over-opened (median 31.5^{\circ} past), all with the hand on the display: it is pushed through the target, not dropped. The Cover Pitcher lid stays inserted in 47 of the 86 episodes that insert it, and Stack Blocks keeps four layers in 30 of 164, with the hand already released at 70 of the 134 losses. In Pass Block, the receiver takes over alone in 188 of 367 episodes where both hands hold the block; without takeover the block drops in 175 of 179, a median 0.30 s after the giving hand lets go. Policies must stop pushing at the target and release only when the other hand holds.

Tools are bypassed, and failures are not shared across checkpoints.

Line Up Blocks, Scoop Marbles, and Flip Sponge end on direct hand contact with the manipulated object in 81, 44, and 28% of episodes (2 successes in 2,400): policies pursue the object with the hand rather than through the tool. On the 41 fixed-goal tasks at least one checkpoint solves 62.4% of scenes, close to the 64.0% expected under independence (median pairwise \kappa=0.03): the failed scenes differ by checkpoint, so no shared set of hard scenes explains the failures, although each checkpoint may have its own.

Transfer-16 fails at the same transitions, and new objects fail earlier.

Every matched transition fails more often than in its Core-50 source (Figure 8c). In Spray Shoe grasps form in 232 of 344 episodes, but at most 3 of 43 per checkpoint complete a trigger cycle. On a solved source, a new object moves failure to acquisition: the left hand picks up the glue stick in 5 of 400 Pass Glue Stick episodes, against 40–88% per checkpoint in Pass Block.

Compositions add a fourth failure: steps are repeated or started early.

After the laptop step (146 episodes), no episode brings the mouse onto its pad, and 111 push the display out of range again. The shutter is pressed before the fruit is in the bucket in 311 of 399 Place Fruit and Photograph Logo episodes, water runs in 107 of 400 Scrub Bottle Then Fill Cup episodes although none finishes scrubbing, and the box lid moves before the pitcher lid in 340 of 400 Cover Pitcher and Box episodes. Nothing marks a finished step, so completion must be inferred from the scene.

5.3.4 Task Grouping and Reliability

Table 9 lists the grouping of Core-50 tasks by the hand skill that decides success, used in Figure 5a and throughout Section 5. Each task is assigned to exactly one group by its success condition; tasks that also require reaching and grasping are assigned to the later skill. The grouping is an analysis lens and is independent of the interaction and structure labels of the catalog. Figure 10 shows per-task success for the eight trained checkpoints and GPT-6 in this grouping.

For the eight trained multi-task checkpoints, per-task signal-to-noise, the spread of success across checkpoints divided by its binomial standard error, is below 1 on 6 tasks and at least 3 on 14. Twelve tasks selected by this ratio on even-indexed episodes reproduce 96% of the 28 pairwise orders measured on odd-indexed episodes (88% for random twelve-task subsets), and 24 tasks reproduce all of them. Episode-only 95% half-widths of the 50-task macro score are about 1.9, 1.3, and 0.9 points for 25, 50, and 100 episodes per task. Selecting a checkpoint per task on half of the episodes and scoring it on the other half gives 32.9% versus 29.1% for the best single checkpoint.

An earlier evaluation campaign of the same \pi_{0.5} and OpenWAM checkpoints used the fixed canonical prompt and a single goal per task; on the 48 shared tasks it gives 24.8 versus 23.4% (\pi_{0.5}) and 30.5 versus 28.5% (OpenWAM), with per-task Spearman \rho of 0.98 and 0.97 and per-task differences whose standard deviation (4.3 and 6.3 points) is close to the binomial expectation (5.1 and 5.5). Within the Core-50 snapshot, success is not measurably affected by which of the four table materials, three colour palettes, or two roughness levels is drawn (largest within-task effect 0.8 points, permutation p\geq 0.17), and training paraphrases score within 2.2 points of the canonical sentence for seven checkpoints (OpenWAM: canonical +4.0 points, z=2.9). All appearance factors and paraphrases are inside the training distribution, so these are invariance checks, not generalization results.

6 Discussion and Conclusion

HandsOn provides a common experimental basis for studying the scope and limits of manipulation competence. By treating embodiment, environment, and task structure as configurable experimental variables, it enables systematic investigation of how policy behavior changes across conditions and where generalization breaks down.

Our trained-policy evaluation spans more than 40,000 Core-50 episodes and includes eight multi-task checkpoints, revealing substantial differences in performance across task categories. Pooled across those eight checkpoints, success rates reach 35–51% for pushing, articulation, and pick-and-place, but remain at 2–3% for tool use and precision fitting. These disparities persist after accounting for demonstration count and expert demonstration duration. All tasks in both suites have been successfully demonstrated by human operators, making even the hardest cases concrete, achievable targets for policy improvement.

A prompted GPT-6 controller using move_eef succeeds in 29 of 660 trials: 5.2% on Core-50 and 1.9% on Transfer-16. Its saved action notes provide qualitative signs of task-aware sequencing and execution monitoring without benchmark demonstrations or finetuning, suggesting potential as a high-level planner. Reliable execution remains difficult: the minimal harness requires the same model to translate subgoals into palm poses and up to 40 finger-joint targets with limited geometric and contact feedback. This motivates richer agent interfaces that connect high-level planning to state estimation and feedback-driven manipulation skills. The high dimensionality and contact sensitivity of dexterous tasks make this interface a substantive design problem. Matched harness ablations are needed to quantify the contribution of these components.

The diagnostic results point to a common requirement: policies must regulate the state of an interaction throughout execution. They need to distinguish contacts that stabilize an object from those that actuate it, condition release on whether the partner’s hand has secured the object, and maintain achieved goals as the task progresses. Failures on composed tasks further highlight the importance of tracking completed steps and satisfying prerequisites before advancing. Together, these findings motivate learning methods that explicitly account for contact roles, inter-hand dependencies, and task progress.

Future evaluations should measure both task completion and maintenance of success conditions, testing whether control improvements transfer across embodiments and to physical robots. Promising directions include evaluating VLM planners with richer agent harnesses, expanding embodiment and compositional-task coverage, strengthening paired simulation and real-world evaluation, and modeling contact state, inter-hand coordination, and task progress.

7 Future Work

Tactile sensing.

HandsOn policies receive vision and proprioception without tactile feedback, but the wrist cameras are mounted on the hand and the held object occludes them: once a grasp forms, the very region the policy must control leaves the view. The failures we report sit exactly there. A grasp that closes over a button hides the button as it presses it; a cut that loses a handle contact happens on the far side of the object; a goal pushed past its target is obscured by the hand doing the pushing. Tactile sensing would observe these contact events directly instead of inferring them from a partially hidden view. Adding it requires taxels placed identically on the simulated and physical hand, a contact model verified rather than assumed, and tasks that still admit a visual solution, so that visual and tactile policies remain comparable on the same predicates.

Quantified physical evaluation.

The real-robot rollouts in Appendix F show that simulation-trained checkpoints act sensibly on the physical platform, but they are recorded evidence, not a measurement. A physical success rate needs the discipline the simulated suite already has: paired simulation and real instances, an executable predicate on both, a fixed trial count, and reported failure modes rather than selected successes. The protocol is specified in Appendix B.5; what remains is the trial budget to run it alongside simulation-plus-real and real-only training under an explicit data budget.

Long-horizon household and production tasks.

HandsOn scores individual skills, and even the compositional Transfer-16 tasks chain at most a few operations. Household and production work is longer and less forgiving: tens of operations over a changing world state, objects found rather than placed, and success measured in throughput and rework rather than a binary predicate. Our failure analysis indicates what such a suite must measure beyond success, namely whether a policy tracks which steps are complete, notices that an earlier step went wrong, and can recover without restarting the episode. That requires long-horizon demonstrations, a stage model that admits recovery rather than enforcing an ordered prefix, and evaluation that reports progress and interventions.

Evaluating VLMs across richer agent harnesses.

The GPT-6 experiment motivates evaluating VLMs with harnesses that add estimated object poses, contact feedback, reusable manipulation skills, and hierarchical control to direct move_eef commands. Comparing harnesses for a fixed model, and models within a shared harness, would help distinguish planning capability from interface design. Evaluations should match tasks, resets, physical-time and query budgets, disclose training of lower-level skills, and measure success, subgoal completion, recovery, and inference cost. This would establish when VLMs can serve as effective high-level planners for dexterous manipulation.

References

1. [1] Hanwen Wang, Weizhi Zhao, Xiangyu Wang, Siyuan Huang, He Lin, Boyuan Zheng, Rongtao Xu, Gang Wang, Yao Mu, He Wang, Lue Fan, Hongsheng Li, Zhaoxiang Zhang, and Tieniu Tan. DexJoCo: A benchmark and toolkit for task-oriented dexterous manipulation on MuJoCo. arXiv preprint arXiv:2605.16257, 2026.
2. [2] Yunchao Yao, Zhuxiu Xu, Tianqi Zhang, Zixian Liu, Sikai Li, Zhenyu Wei, Feng Chen, Dihong Huang, Kechang Wan, Chenyang Ma, Shuqi Zhao, Shenghua Gao, Masayoshi Tomizuka, Yi Ma, and Mingyu Ding. DexVerse: A modular benchmark for multi-task, multi-embodiment dexterous manipulation. arXiv preprint arXiv:2607.08751, 2026.
3. [3] Zhenjie Yang, Yideng Zhang, Dongjie Zhang, Chenyu Jiang, Xianshuai Liu, Yufeng Li, Zuhao Ge, Xingyu Jiao, Zheng Zhang, Kaiyu He, He Wang, Yuwen Zhong, Yi Deng, Muyun Jiang, Xianliang Huang, Haisheng Su, Donghang Zhang, Jian Zhang, Xue Yang, Hongyang Li, Zuxuan Wu, Yu-Gang Jiang, Xiaosong Jia, and Junchi Yan. Bench2Dex: Benchmarking visuo-tactile bimanual dexterous manipulation across dexterous hands. arXiv preprint arXiv:2609.15726, 2026.
4. [4] Zhipeng Tang, Sihang Chen, Sha Zhang, Peihao Yang, Yan Liu, Wentao Zhao, Xinrui Liu, Rui Huang, Wensheng Du, Yuting Huang, Jiajun Deng, Lidian Wang, Yuan Zhang, and Yanyong Zhang. LabDex: A hierarchical benchmark for dexterous manipulation in laboratories. arXiv preprint arXiv:2608.18618, 2026.
5. [5] Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Jim Fan, and Yuke Zhu. DexMimicGen: Automated data generation for bimanual dexterous manipulation via imitation learning. In IEEE International Conference on Robotics and Automation, 2025.
6. [6] Sarthak Kamat, Adam Rashid, Satvik Sharma, Aseem Doriwala, Chelsea Finn, Phillip Isola, and C. Karen Liu. Pre-training visual dexterity in simulation. arXiv preprint arXiv:2608.15917, 2026.
7. [7] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.
8. [8] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. arXiv preprint arXiv:2112.03227, 2022.
9. [9] Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. DexArt: Benchmarking generalizable dexterous manipulation with articulated objects. arXiv preprint arXiv:2305.05706, 2023.
10. [10] Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu, Ruihai Wu, Jingquan Zhou, Kai-Chong Lei, Haibao Yu, Yuanfeng Ji, Weiyang Jin, Guanyu Lin, Xiaofan Li, Qi Xiong, Renjing Xu, Zhongyu Li, Wenhao Chai, Enze Xie, Ziwei Wang, Yao Mu, Hao Dong, Wojciech Matusik, Mingyu Ding, Wenbo Ding, Ping Luo, and Masayoshi Tomizuka. RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. arXiv preprint arXiv:2607.04434, 2026.
11. [11] Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan-ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, and Yao Mu. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025.
12. [12] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017.
13. [13] Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Zongqing Lu, Stephen Marcus McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2022.
14. [14] Qingtao Liu, Yu Cui, Zhengnan Sun, Gaofeng Li, Jiming Chen, and Qi Ye. VTDexManip: A dataset and benchmark for visual-tactile pretraining and dexterous manipulation with reinforcement learning. In International Conference on Learning Representations, 2025.
15. [15] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. ManiSkill2: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659, 2023.
16. [16] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems, 2024.
17. [17] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, et al. BEHAVIOR-1K: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, 2022.
18. [18] Open X-Embodiment Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation, 2024.
19. [19] Alexander Khazatsky, Karl Pertsch, Suraj Nair, et al. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems, 2024.
20. [20] Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. AnyTeleop: A general vision-based dexterous robot arm-hand teleoperation system. arXiv preprint arXiv:2307.04577, 2023.
21. [21] Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey, Siddhant Haldar, Soumith Chintala, and Lerrel Pinto. OPEN TEACH: A versatile teleoperation system for robotic manipulation. arXiv preprint arXiv:2403.07870, 2024.
22. [22] Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C. Karen Liu. DexCap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788, 2024.
23. [23] OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177, 2018.
24. [24] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Jean-Francois Lafleche, Dieter Fox, and Gavriel State. DeXtreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5977–5984, 2023.
25. [25] NVIDIA Corporation. NVIDIA Isaac Sim 5.1 documentation. https://docs.isaacsim.omniverse.nvidia.com/5.1.0/, 2026. Accessed: 2026-08-26.
26. [26] MANUS. MANUS Prime Series Infokit 2019. Manufacturer infokit: https://a.storyblok.com/f/77127/x/9e90554ca1/infokit-v5.pdf, 2019. Accessed: 2026-08-26.
27. [27] HTC Corporation. VIVE Tracker (3.0). Manufacturer product page: https://www.vive.com/us/accessory/tracker3/, 2021. Accessed: 2026-08-26.
28. [28] Guanqi He and Wentao Zhang. WujiHand Retargeting. Software repository: https://github.com/wuji-technology/wuji-retargeting, 2026.
29. [29] ROKAE Robotics. Humanoid Force-Controlled Arm AR Series. Manufacturer product page: https://www.rokae.com/en/product/show/595/Humanoid-Force-Controlled-Arm-AR.html, 2026. Accessed: 2026-08-26.
30. [30] Wuji Technology. Wuji Hand: Product Introduction. Manufacturer documentation: https://docs.wuji.tech/docs/en/wuji-hand/v1/overview/, 2026. Accessed: 2026-08-26.
31. [31] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, et al. \pi_{0.5}: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.
32. [32] En Yu, Haoran Lv, Jianjian Sun, Kangheng Lin, Ruitao Zhang, et al. DM0: An embodied-native vision-language-action model towards physical AI. arXiv preprint arXiv:2602.14974, 2026.
33. [33] Wei Wu, Fan Lu, Yunnan Wang, Shuai Yang, Shi Liu, Fangjing Wang, Qian Zhu, He Sun, Yong Wang, Shuailei Ma, et al. A pragmatic VLA foundation model. arXiv preprint arXiv:2601.18692, 2026.
34. [34] He Zhang, Lingzhu Xiang, Haitao Lin, Zeyu Huang, Minghui Wang, Dingyan Zhong, et al. Hy-Embodied-0.5-VLA: From vision-language-action models to a real-world robot learning stack. arXiv preprint arXiv:2606.14409, 2026.
35. [35] Fuhao Li, Wenxuan Song, Han Zhao, Jingbo Wang, Pengxiang Ding, Donglin Wang, Long Zeng, and Haoang Li. Spatial forcing: Implicit spatial representation alignment for vision-language-action model. arXiv preprint arXiv:2510.12276, 2025.
36. [36] Yuran Wang, Siqiao Huang, Mingleyang Li, Chenhao Zhang, Jiaqi Liang, Weiyang Jin, et al. OpenWAM: An open, modular exploration towards systematic world-action model pretraining. arXiv preprint arXiv:2609.07398, 2026.
37. [37] Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-WAM: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026.
38. [38] Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, and Jun Zhu. Motus: A unified latent action world model. arXiv preprint arXiv:2512.13030, 2025.
39. [39] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.

Appendix A Task Inventory and Taxonomy

Table 10 lists all 66 tasks in HandsOn: Core-50 provides 50 tasks for learning and evaluation; Transfer-16 provides 16 additional tasks exclusively for generalization evaluation. Transfer-16 contains 6 compositional tasks (EVAL-Comb), 7 generalization tasks (EVAL-Gen), and 3 tasks in the EVAL-Flex family. Evaluation-only tasks are excluded from the dataset for policy training. Within each split, task identifiers follow alphabetical order of the English task names.

Source superscripts distinguish task-design adaptation (J: DexJoCo; V: DexVerse) from asset reuse (JA and VA, respectively). The sources and modifications are detailed in Appendix A.1. Unmarked entries do not imply that every asset or component is original.

The three label columns are the axes of the taxonomy in Figure 3. Interaction Type records the dominant manipulation requirement. Asset Type includes all applicable types of task-relevant assets, including designated targets and receptacles, according to their current simulation models and active degrees of freedom. The robot, generic support surfaces, and background assets are excluded. Task Structure describes the organization of semantic goals and operations; approaching, grasping, and releasing do not by themselves create separate stages.

Abbreviations. Interaction Type: P = Primitive; F = Functional; CR = Contact-rich. Task Structure: SS = Single-stage; RP = Repetitive; LH = Long-horizon (multi-stage). Asset Type: RB = Rigid body; AP = Articulated (prismatic); AR = Articulated (revolute); DF = Deformable.

Classification notes. Primitive covers object transport or reorientation whose success is decided by reaching a target configuration, including pushing, pulling, or rolling an object and moving an articulated part along its free degree of freedom (e.g., opening a drawer or a hinged lid). Functional requires the purposeful use of a tool, appliance, or control interface (buttons, knobs, levers, faucets, triggers) to produce an effect. Contact-rich denotes tasks dominated by fine contact coordination, such as regrasping, precise mating, sustained contact-force regulation, coordinated bimanual articulation, or controlled deformation. RB describes each non-articulated rigid asset and does not limit a task to one object. Rigid links of articulated assets are not counted separately. Deformable labels follow the current simulation model; visual liquid effects do not add a DF label. In Task 08, the nail has a prismatic joint relative to the wood block, and the hammer is a separate rigid object. EVAL-Flex03 combines a deformable foam pillow with a rigid carton. SS denotes a single semantic goal. RP denotes repeated applications of the same kind of manipulation operation, whereas LH requires distinct, dependent subgoals. Routine setup and terminal activation, release, or waiting do not by themselves turn a repetitive task into LH. Task 39 (Set Timer) is classified as RP because its main manipulation consists of setting the minute and second values through button operations; starting the timer, withdrawing the hands, and observing a countdown tick are terminal completion conditions.

A.1 Task Provenance and Adaptation

HandsOn incorporates selected task designs and assets from DexJoCo [1] and DexVerse [2]. Table 11 separates task-design lineage from the reuse of particular assets. The J and V markers identify documented adaptations; JA and VA identify asset provenance without asserting that the whole task was transferred. Shared infrastructure and generic similarities, such as grasping or stacking, are insufficient to establish task-specific lineage.

Compositions and Transfer-16.

Source credit also follows reused components into new evaluations. EVAL-Comb02 combines the adapted laptop and mouse skills, EVAL-Comb04 reuses the photograph skill, and EVAL-Gen06 transfers the spray interaction from Water Plant. These are compositions or task variants of the credited components, rather than entire tasks taken from an upstream benchmark. The evaluation-only designation specifies exclusion from policy-training demonstrations; it does not imply that all constituent skills or assets are new.

Attribution boundaries.

Task names alone do not establish reuse. For example, Prepare Coffee and Squeeze Scissors have related counterparts in DexVerse but different documented workflows or asset lineages; we do not label them as direct ports on that basis. The source markers identify documented task-design adaptations and retained asset reuse.

Appendix B Additional Data, Platform, and Protocol Records

B.1 Physical System and Alignment Record

Figure 2a compares simulated and physical views of the toy-aircraft setup from the setup, wrist, and main cameras.

The ROKAE AR5-08 has seven degrees of freedom, a rated payload of 5 kg, a 683 mm reach, and manufacturer-specified repeatability of \pm 0.1 mm [29]. The Wuji Hand has 20 actively actuated joints, four per finger, with position control over USB 2.0 [30]. Its specified 1 kHz low-level frequency is distinct from policy inference and host command rates.

The correspondence record covers hand/robot geometry and limits, asset scale, camera calibration, state/action coordinate frames, joint/side ordering, timing, initial conditions, and success. Calibration data are disclosed independently of target-task demonstrations.

B.2 Transfer-16 Duration Map

Table 12 reports the 30 Hz protocol. The source-frame column sums each mapped atomic task’s P75 before multiplying by three. Task validity is established by the verification in Section 3.2; a repaired task requires fresh trials under its qualified implementation.

Table 10 lists the paper task IDs and names.

B.3 Task Partitions and Generalization Design

A task is defined by its environment, initial-state distribution, observations, actions, success predicate, and physical time budget:

\mathcal{T}_{i}=(\mathcal{E}_{i},p_{i}(s_{0}),\mathcal{O}_{i},\mathcal{A}_{i},G_{i},H_{i}).

We distinguish training trajectories, held-out trajectories from training tasks, fresh closed-loop trials on those tasks, and evaluation-only tasks. Trajectory validation measures action prediction; fresh closed-loop trials measure execution under the task’s native training-source distribution. Transfer-16 tasks contribute no demonstrations to benchmark training or task-specific finetuning. Their absence from this dataset does not establish their absence from a foundation model’s external pretraining.

The evaluation catalog contains six EVAL-Comb, seven EVAL-Gen, and three EVAL-Flex tasks. These implementation groups support complementary generalization questions rather than a set of perfectly orthogonal factors. Each task must specify the supporting training skills and assets, the held-out change, and its executable goal. Multiple changes are identified when they co-occur.

Asset and relation transfer.

This dimension tests the use of skills with new objects, asset pairings, or goal relations. An unseen combination of familiar assets is distinguished from a new instance or category. For example, covering a new container and orienting a different object test transfer of related functional behavior, but whether an asset is actually familiar is determined by source and asset identities, not by its task name.

Skill transfer and composition.

These tasks reuse skills in new sequences or contexts. Open Laptop and Click Mouse combines opening an articulated object with interface operation; Scrub Bottle Then Fill Cup connects tool interaction to a subsequent goal. Evaluation records where an execution stops and whether a successful stage leaves the state needed by the next stage. A previously unseen atomic operation is identified separately from a new composition of trained skills.

Transfer to deformable manipulation.

Compress Sponge into Mug, Fold Towel Twice, and Pack Pillow require handling deformation without deformable-task demonstrations in the benchmark training set. These are transfer challenges across object, goal, and physical behavior. Isolating compliance as a causal factor would additionally require matched geometry and goals; the present suite does not assume that all differences are due to material properties.

B.4 Visual and Language Robustness Protocols

Training randomization and evaluation under domain shift serve different roles. Fresh samples from the training distribution measure in-distribution performance; held-out combinations or expanded ranges of visual factors measure robustness under specified shifts. These tests retain the task goal and identify the factors changed. They are reported separately from the evaluation-only tasks, which change asset relations, skills, or physical interactions. Combining both changes is an additional challenge condition, not a replacement for the matched-distribution baseline.

The eight held-out paraphrases per active goal define a language-only test: the environment, target values, reset seeds, and success predicate stay fixed while the instruction changes. Paired trials compare canonical, training-paraphrase, and held-out-paraphrase success. We report the success gap and paired outcome agreement, so consistently failing all wordings cannot masquerade as successful language robustness. This test is separate from Transfer-16 physical tasks and from visual domain shifts.

A complementary goal-sensitivity test changes the requested object, relation, or numerical setting only within a task’s supported configurations, with its executable goal updated accordingly. For example, the timer must match the requested minutes and seconds; moving a knob must reach the requested angle and direction. Such tests assess whether the policy uses the instruction, rather than repeating a familiar motion. Unseen numerical values require an explicitly held-out goal split. Text-invariant policies are not scored for language robustness; their structured-goal performance is reported separately. These additional tests require dedicated trials and are not inferred from the canonical-prompt evaluations.

B.5 Matched Physical-Transfer Protocol

A paired task comprises simulation and real instances with a correspondence record:

\mathcal{P}_{i}=(\mathcal{T}^{\mathrm{sim}}_{i},\mathcal{T}^{\mathrm{real}}_{i},\mathcal{C}_{i}).

The record specifies robot and camera configuration, state/action frames, control timing, initialization conditions, and success definitions. Direct transfer deploys the same checkpoint without target-task real demonstration training or real-data finetuning. Calibration measurements and any real images used to configure the system are disclosed separately from policy training. External pretraining of a VLA or WAM retains its own data identity.

Only tasks with physical counterparts enter the paired analysis, which reports their coverage and per-task simulation and real trial counts. The signed success-rate difference for model m and task i is defined as

\Delta_{i,m}=S^{\mathrm{sim}}_{i,m}-S^{\mathrm{real}}_{i,m}.

Same-task real transfer and Transfer-16 generalization are independent claims. The recorded real-robot rollouts of this platform are reported qualitatively (Appendix F) and do not constitute a scored physical evaluation.

B.6 Additional Task-Capability Examples

Articulated collection assets occur in 21 tasks: 12 involve revolute joints and 11 prismatic joints, with two containing both. Goals specify motion direction, travel, angle, or unlocking prerequisites. Set Timer uses requested minutes and seconds; Set Dual Knobs uses left/right angle and direction targets; and Unfold Fan targets 60, 90, or 120 degrees. Fixed three-cycle scissors and propeller requirements do not by themselves establish unseen-count generalization. Place Tableware combines identity, position, and orientation; Place Largest Fitting Ball and Sort Nested Cups add size comparisons. Prepare Coffee and Load and Start Microwave require intermediate states that enable later operations. These capability labels overlap the catalog’s 17 primitive, 18 functional, and 15 contact-rich tasks and its 31 single-stage, seven repetitive, and 12 long-horizon tasks.

Appendix C GPT-6 move_eef Evaluation Protocol

C.1 Model, Task Population, and Observations

We request gpt-6-astra with reasoning effort medium through the Responses API, a maximum output of 4,096 tokens per request, required function calling, and parallel calls disabled. The only exposed function is move_eef; the controller returns one call per turn. No task demonstrations, benchmark finetuning, or learned low-level policy are used. This describes benchmark-provided exposure; it makes no claim about the model’s external pretraining. Requested model names, returned model identifiers, provider provenance, and token usage are retained in the request records rather than inferred from a saved client label.

The fixed batch contains ten slots for each Core-50 and Transfer-16 task, 660 in total. The choice of ten episodes per task reflects the available API-cost and wall-clock evaluation-time budgets while retaining coverage of all 66 tasks. This sample-count allocation is separate from the per-trial model-call cap and simulated physical-time limit. It yields coarser per-task estimates than the larger trained-policy evaluations; the uncertainty analysis below accounts for the smaller sample count. A slot retains its task, goal, seed/accepted-reset record, prompt, and source identity across attempts. Chinese Chess is excluded. The declared reset-admission rules and any repaired reset identities remain in the record; this batch is not assumed to be a seed-paired subsample of every trained policy. Pilot trajectories do not enter this population.

Each observation supplies the main, left-wrist, and right-wrist RGB views as native-resolution PNGs with high-detail image input, plus measured world-frame palm poses, all 20 finger joints per hand, elapsed physical time, and the remaining physical budget. The robot contract supplies active arms, joint order, and joint-position bounds. The prompt contains the official task instruction verbatim and a static task card: a general completion reminder, simple goal clarification for the three basic pick-and-place tasks, or the declared carton scoring exception. This additional text is part of the GPT-6 condition. Object poses, hidden rewards, and live task-checker state are not supplied to the model.

C.2 The move_eef Palm-and-Finger Interface

For either active hand, move_eef accepts world position xyz in metres, absolute extrinsic-xyz Euler orientation rpy_deg in degrees, and 20 absolute joints_rad targets. Joints are ordered thumb, index, middle, ring, and little finger, four per digit. An omitted palm target holds the measured pose; omitted finger targets retain their last commanded setpoints, preserving a commanded grip even when contact prevents arrival. Finger targets are clipped to native joint-position bounds. Inactive arms cannot be commanded. There is no scalar gripper command or task-specific grasp primitive.

A read-only kinematic planner constructs arm waypoints, paced at at most 0.03 rad per planning step and any stricter native limit. Finger targets follow quintic interpolation from their previous commanded values, paced by the native bounds and at most 0.18 of each joint’s range per step. Arm and finger paths share a schedule. These are command-generation conventions, not guarantees of measured velocity, collision clearance, or target arrival. The planner has no obstacle model. The adapter passes each 52-dimensional two-hand action (palm position, rotation vector, and 20 joint targets per side) through the benchmark decoder and task-native executor, retaining native controller and physical actuation behavior.

Every executed action advances physical time by 1/30 s; the recorded physics timestep times the substep count must equal this duration. A move_eef schedule is limited to 300 actions (10 physical seconds), with longer plans rejected before execution. A no-op consumes one action. Malformed or infeasible move_eef calls receive feedback without advancing physics and still consume their model calls. Native termination can interrupt a schedule. A completed schedule means that commands were executed, not that the target was reached or a grasp succeeded; the next observation supplies measured state and palm-target error. There is no give_up tool and no extension of the task’s physical-time budget. API latency does not advance simulation or imply real-time execution.

C.3 Context and Interaction Budgets

The controller retains images from the latest two observation turns, including all three views at each turn. Its compact-context configuration summarizes executed steps, physical duration, and execution status, removes duplicate execution feedback, and rounds model-facing numerical state to four decimal places. Full-precision simulator state, actions, responses, and execution diagnostics are retained for verification. Earlier image turns are removed; the recorded text history is retained.

The final controller configuration allows 100 counted model calls per trial. Invalid or infeasible move_eef calls consume this budget without advancing physics; an additional upstream request also consumes a call. Cached successful replies and replay copies are not counted again. The reported batch retains the recorded settings of earlier completed runs, some of which use full text feedback or exceed 100 calls. Thus the 660-trial result describes one fixed evaluation population under its recorded budgets, not a controlled compact-context or uniform-100-call ablation. These settings are included in the machine-readable record.

A trial succeeds only when the native task predicate is satisfied. An unsuccessful call-budget stop counts in the denominator even when the native environment has not terminated. Such outcomes require an independent check of task/seed identity, loaded source, 30 Hz execution, finite recorded state and actions, saved calls, and completed video files; their acceptance does not certify native terminal completion. Physical limits follow Section 3.3 and Appendix B.2, including the three Flex exceptions.

C.4 Scoring, Verification, and Comparability

The scored population has exactly one outcome for each of the 660 fixed task/goal/seed slots. Infrastructure interruptions alone are not policy failures, and no unresolved slot remains in the final report. Result and video hashes, source and seed records, controller invariance, and physical clocks were checked on the server for all scored outcomes and 1,980 three-view videos. This check concerns the implemented protocol and file integrity; it is not a certificate of full training-source alignment.

Two declared qualifications apply to the corresponding results. Open Latched Door retains the known mismatch between its frozen instruction (left lever above 45^{\circ}, right door) and admitted native source (right lever at least 25^{\circ}, left door). Lift Carton observes contact force but waives the 12 N magnitude threshold while retaining bilateral support, lift, hold, and drop conditions; its four successes certify the lift/hold task, not a 12 N force limit. Line Up Blocks records forbidden direct hand contact as a task-rule failure, including when the native return flags have not yet asserted termination.

The controller receives measured robot state and static task guidance in addition to images. Its trial count, accepted resets, task information, query timing, and call budget differ from those of the trained policies; its ten trials per task are not assumed to be seed-paired with every checkpoint. The comparison is consequently descriptive. It does not identify an architectural effect or equate external pretraining, task information, or inference compute across model families.

C.5 Aggregation and Uncertainty

Success is the unweighted mean of per-task success rates within each suite or task family. Every task has ten trials, so the macro rate equals pooled successes divided by trials. The overall 29/660 is provided for coverage accounting; the Core-50 and Transfer-16 scores remain separate because their task populations answer different questions.

Table 7 gives 95% percentile intervals from 20,000 within-task bootstrap resamples: for each task, redraw ten binary outcomes from its observed success rate, retain the task catalog, and recompute the macro rate. The intervals quantify conditional finite-rollout variation; they exclude task-selection, provider/model-version, and protocol-setting uncertainty. Table 15 additionally reports a 2,000-resample task-and-episode interval for Core-50, redrawing tasks and then ten outcomes per selected task from its observed rate. Ten trials give only coarse per-task resolution, and an observed 0/10 does not imply zero underlying success probability.

C.6 Per-Task GPT-6 Results

Each task contributes ten scored trials. The identifiers and names below follow Table 10; runtime directory aliases are resolved through the machine-readable task map. Budget stops contribute to the denominator and are unsuccessful outcomes. A measured 0/10 is not evidence that the true success probability is zero. The Lift Carton and Open Latched Door qualifications in Appendix C.4 apply.

Appendix D Core-50 Results and Coverage

The trained-policy rows in Table 5 are computed from 45,000 episodes on all 50 Core-50 tasks. The eight multi-task checkpoints contribute 100 episodes per task (39,999 valid; HY-Embodied has 99 valid episodes on Place Wine Glass), and single-task DP contributes 5,000 episodes from task-specific checkpoints. Overall success is the unweighted mean of the 50 task success rates; interaction-type and hand-skill means use the same rule within each group. Chinese Chess remains outside the catalog. GPT-6 (move_eef) has ten trials per task and appears separately in the uncertainty and per-task tables below; its outcomes are not pooled into trained-policy aggregates. Table 14 also lists its call-budget stops and Transfer-16 results.

Scope and validity.

All multi-task evaluations passed the recorded runtime validation and match the selected checkpoints; the DP checks are described below. The evaluator samples each reset from the admitted source templates of its task, so the scores describe this implemented reset distribution. Native reset rejection is handled before policy scoring by selecting the first accepted candidate while retaining the assigned goal and language; rejected resets and infrastructure interruptions are not policy failures. On 41 tasks, the eight multi-task checkpoints share the reset seed, variant, goal, appearance, and prompt at nearly every episode index; on the nine goal-conditioned tasks, the goal and prompt are sampled per checkpoint.

Replacement and scoring records.

Exact original-slot associations include the recovered minority-goal Fit Device Cover episodes and the recovered DM0.5 Place Wine Glass episode; no slot is counted twice. Each checkpoint contributes 100 Lift Carton episodes evaluated using the following criteria: both hands support opposite lower sides of the carton, the entire carton clears the table, and it remains continuously airborne for three seconds.

Single-task DP reference.

DP uses a distinct checkpoint at 30,000 updates for each task. Ten tasks use completed retraining with corrected near-degenerate normalization; the other tasks retain their previously selected task checkpoints. Lift Carton is evaluated in a separate batch using the lift/hold criteria described above; its result is included in the 50-task score but excluded from the 11-task matched comparison. Original and retrained model identities are preserved per task. The 38 earlier evaluations use the earlier native goal sampler, whereas the newer runs use frozen training-frame-weighted goal quotas, so DP is a descriptive reference rather than a goal-distribution-matched comparison. Three rejected knife-task resets are omitted under the original first-100-accepted-seeds rule. The DP audit checks checkpoint hashes, native completion, native success, and recorded three-view video completion.

The accompanying machine-readable record stores the task map, checkpoint identities, source plans, supplemental-slot associations, and the hash of the episode-level snapshot.

Uncertainty.

Table 15 reports two 95% percentile bootstrap intervals. The episode-only interval (20,000 resamples) redraws each task’s outcomes from its observed success rate with the 50 tasks fixed; it quantifies finite rollout sampling under an independent-episode assumption. The task-and-episode interval (2,000 resamples) also resamples the 50 tasks with replacement, so it describes a new draw of tasks from the same catalog. Neither interval includes training-run or checkpoint-selection uncertainty.