不用相机和力传感器，仅凭关节感知完成精密装配：MIT盲插策略实机成功率73%

Abstract

Every robot already carries high-resolution contact sensors: its own joints. Legged robots and humanoids exploit them to walk, run, and track whole-body motions without vision, but there the contacts are large and forceful. Is joint sensing also enough for precise, contact-rich assembly, where the robot must find a millimeter-scale opening, mesh gear teeth, or catch a thread through light contact? We study this question by training blind assembly policies that observe only a short history of joint positions, velocities, and their own actions, without visual, force, torque, or tactile sensing. The poses of both the fixture and the part are unknown to the policy, and the grasp is randomized substantially. The policy must therefore assemble the parts through contact, sensed exclusively through its own proprioception. Trained with reinforcement learning entirely in simulation over randomized parts, grasps, and contacts, such policies learn peg insertion, gear meshing, and nut threading. The peg insertion policy transfers zero-shot to the real world and inserts pegs of different shapes and sizes in 73% of trials. Controlled comparisons with matched visual policies in simulation show that blind policies generalize far better to unseen part shapes and sizes, though they can be slower. Training such policies needs thousands of different assemblies, each simulated with its clearance intact, so we also release an open-source generator that builds them procedurally from exact convex pieces for MuJoCo; it is more accurate and faster than convex decomposition or MuJoCo’s signed-distance collision. Videos and interactive simulations are at younghyopark.me/blind-proprioceptive-assembly/.

1 Introduction

A force applied anywhere on a robot’s body travels through its links to the joints, which then move differently from what their commands alone would produce. The encoders record that difference, so the joints of every arm and leg double as contact sensors. Legged robots exploit this proprioceptive signal to walk, run, and climb stairs blind (Lee et al., 2020; Siekmann et al., 2021a; Siekmann et al., 2021b; Kumar et al., 2021), and humanoids to walk and track whole-body motions (Radosavovic et al., 2024; Cheng et al., 2024; He et al., 2024). In these settings, however, contacts are large and forceful, and joints often comply meaningfully while recording measurable deviations. Can robots exploit the same signal for precise, contact-rich assembly?

Precise assembly with a robotic arm occupies the opposite regime: the robot must find an opening with a millimeter or less of clearance, bring gear teeth into mesh, or catch a thread, through light contact whose effect on the joints is easy to miss. We study this question by training blind assembly policies that have no information outside of their own joint sensing, and verify that proprioception contains enough information for three such tasks: peg insertion, gear meshing, and nut threading (figure 2). In the real world, our peg insertion policy inserts eight pegs of different cross-sections, three of them outside its training distribution, with 0.5–1.5 mm of clearance per side, in 73% of 237 trials, without being given the hole’s location, the peg’s pose in its grasp, the part’s size, or its clearance (section 3.4). To our knowledge, this is the first such demonstration.

We train our policies entirely in simulation with reinforcement learning and deploy the peg insertion policy in the real world without fine-tuning (figure 1). Training randomizes part geometry, contact properties, fixture poses, and grasps, which can hold the part off-center on the finger pads and tilted in the grasp. The policy observes a 0.6 s history of joint positions and velocities, the gripper pose and twist computed from them, and its own previous actions. It receives no image and no force signal, so it can perceive contact and find the hole, shaft, or bolt only through its own motion (section 2).

On all three tasks, probing the blind policies’ learned representations and comparing them with matched visual policies revealed two interesting properties of visually blind behaviors:

• Visually blind yet intelligent. The learned behavior is not an open-loop pattern: its learned representation encoding its own proprioceptive inputs contains meaningful task-relevant information that the policy itself heavily relies on to adjust its behavior (section 3.2).
• Proprioception generalizes better. Because a blind policy depends on the part only through its contacts, it degrades far less than vision on part shapes and sizes it has not seen during training (section 3.3).

Training for this robustness requires diverse parts: thousands of different assemblies, generated procedurally and simulated in parallel, each with its millimeter clearance intact. In MuJoCo (Todorov et al., 2012), building holes from convex primitives by hand does not scale to random shapes, and the automatic alternatives, convex decomposition and signed-distance-field (SDF) collision, are less accurate and slower at these clearances. We therefore built mjfit, which procedurally generates pegs and holes from any planar outline, as well as threads and gears, out of exact convex prisms. To summarize, our contributions are:

1. Blind policies, trained purely in simulation, that learn peg insertion, gear meshing, and nut threading from proprioception alone, given no fixture pose, grasp, or part model (section 3.1), with analyses showing that on all three tasks they search reactively, acting on the contact state their joint histories encode rather than following a fixed pattern (section 3.2), and that they generalize to unseen part shapes and sizes better than matched visual policies (section 3.3);
2. The first sim-to-real policy for precise insertion from proprioception alone: trained only in simulation, the peg insertion policy transfers zero-shot to a real robot and inserts pegs of different cross-sections, including pegs outside its training distribution (section 3.4);
3. mjfit, an open-source procedural generator of assemblies for MuJoCo (pegs and holes, threads, and gears) with exact convex collision geometry, which supplies the thousands of different training assemblies, and a benchmark against analytic ground truth in which it is more accurate and faster than convex decomposition and MuJoCo’s SDF collision (section 2.2).

2 Learning blind assembly

We train blind policies for three assemblies that mjfit generates: peg insertion, gear meshing, and nut threading. We describe the tasks (section 2.1); how mjfit, our open-source procedural generator for MuJoCo, supplies thousands of different assemblies with exact convex collision geometry that preserves their clearances (section 2.2); and the policy, whose auxiliary contact supervision makes learning faster without giving the actor any privileged information (section 2.3; figure 14).

2.1 Tasks

In all three tasks (figure 2), a 7-DOF robot arm equipped with a parallel jaw gripper holds a part and must assemble it with a fixture that can sit almost anywhere in its workspace, and tilted (figure 3a). The part’s geometry, its pose in the grasp (figure 3b), and the contact properties are randomized and not observed by policy (Table 1). Episodes start with the part touching the fixture but misaligned, and succeed once it is fully assembled. This requires the policy to both search for alignment through contact, and then perform the assembly, based on its joint position feedback alone.

• Peg insertion: the pegs are prisms with varying cross-section, size, aspect ratio, mass, and length, with 1.0–1.5 mm of clearance per side (Appendix A). Pegs start typically around 2 cm from the hole, tilted far more than the clearance admits, and succeed when the peg is seated in the hole.
• Gear meshing: spur gears of varying size and tooth count must slide down a shaft into mesh with one or two anchor gears that turn on their own axles, with 0.2–0.6 mm of backlash. Episodes start with the gear in contact with the shaft tip, but misaligned and with its teeth out of phase with the anchors’, and succeed when the gear is seated on the shaft with teeth meshed.
• Nut threading: metric nuts from M24 to M42 must thread onto a bolt with 0.05–0.2 mm of thread clearance. Episodes start with the nut resting on the bolt’s tip, off its axis and tilted, and succeed a quarter turn after the thread catches.

2.2 Simulating tight insertion at scale

Training on this distribution means simulating thousands of parallel worlds that each hold a different, randomly generated assembly, with per-side clearances down to 0.5 mm. MuJoCo collides meshes through their convex hulls, so a concave hole must be given an SDF or built from convex pieces, which simulation benchmarks assemble from boxes by hand (Zhu et al., 2020; Tao et al., 2024; Zakka et al., 2025).

Exact convex decomposition.

mjfit automates the hand-built hole and generalizes it to arbitrary outlines. Given the planar outline of a hole, optionally with islands such as connector pins, it emits a ready-to-simulate MJCF model in which every contact surface is a MuJoCo primitive or a convex prism (figure 4). The plate around the hole is split into convex polygons and extruded; the peg is the hole outline offset inward by the clearance, so every peg edge sits exactly the clearance from its hole edge, and islands become pockets in a hollow peg. Curves are tessellated by a circumscribed polygon, so the wall never intrudes into the true hole, as it does in rings of boxes that inscribe the circle (Zhu et al., 2020); the only error is extra play at the corners, c\,(1/\cos(\pi/n)-1) for a circle of n sides at radial clearance c. Because the construction knows which region must stay empty, the user specifies the clearance directly rather than tuning a concavity tolerance and checking the result. Companion generators apply the same idea to bolt–nut pairs and spur gears, and a sampler draws random outlines, islands, and dimensions, so one training run can hold thousands of different assemblies.

Benchmark.

We compare against the three automatic ways to give MuJoCo a concave hole: CoACD (Wei et al., 2022), V-HACD (Mamou & Ghorbel, 2009), and MuJoCo’s native mesh SDF, each swept over its granularity setting. The test is a 16 mm round or square hole with clearance down to 20 µm; we measure each method’s error against its time per step, on CPU and on GPU with MuJoCo Warp (Google DeepMind and NVIDIA, 2025) (figure 6; Appendix B). Approximate decomposition tolerates concavity far larger than an assembly’s clearance, so it leaves wall material inside the hole (figure 5); refining it narrows the error but never closes it (figure 6a). Mesh SDF misses contacts and rounds the rim. These errors change outcomes: a centered peg stops on the rim or jams on every baseline’s default plate, but falls cleanly through ours (figure 6c). mjfit is also an order of magnitude faster than mesh SDF on CPU at equal or better accuracy (Table 4), and has the best speed–accuracy trade-off on GPU (figure 6b). Each assembly is only a few dozen convex pieces, so every one of our 12,288 parallel training worlds (Zakka et al., 2026) can hold a different one.

2.3 Policy and training

Policy.

At each step, the actor (figure 7) observes its joint positions and velocities, the gripper pose and twist computed from them by forward kinematics, and its previous action. It never receives the hole pose, the part geometry, a wrench estimate, or an image. A temporal convolutional encoder compresses the last 0.6 s of observations into a latent, and an MLP maps the latent and the current observation to joint-position increments. A joint impedance controller tracks them on an arm model whose armature, damping, and friction were identified from real-world measurements. Additional training details in Appendix A.

Training.

We use PPO (Schulman et al., 2017) with a sparse reward: a bonus on success and one-time bonuses for intermediate milestones, such as the peg’s first capture by the opening (Appendix A.1 gives the gear’s and nut’s). Two training-only signals speed up learning from the sparse reward without giving the actor privileged information: an asymmetric critic (Pinto et al., 2018) that also sees the part’s pose relative to the fixture and the true contact labels, and the contact supervision described next. Randomizing dynamics and geometry (Tobin et al., 2017; Peng et al., 2018) covers the ranges of Table 1.

Contact supervision.

A blind policy has to infer contact from how the arm moves. To give that inference a dense training signal, we supervise the actor’s latent with the contact state it cannot observe. At every step, the simulator labels points sampled on the mating features of the part and fixture, as free, touching but misaligned, or aligned (figure 7, top; Appendix A details each task’s labels). A small contact decoder predicts these labels from the latent z_{t}, and its class-weighted cross-entropy is added to the PPO loss (Jaderberg et al., 2017). The gradient of this loss trains the encoder, but the prediction never enters the policy, so nothing privileged reaches the actor and the decoder is dropped at deployment.

Visual policies.

To isolate the sensing modality, we train visual policies with the same task, resets, randomization, reward, actions, and success check. A camera rigidly mounted on the wrist renders a 64\times64 RGB image of the held part and the fixture. The visual policy sees only images, over the same 0.6 s history, through a compact CNN.

3 Results

We design experiments to answer the following questions:

1. (section 3.1) Can a policy that senses only its own joints learn precise assembly tasks including peg insertion, gear meshing, and nut threading?
2. (section 3.2) Do blind policies learn intelligent, contact-reactive search, or is it learning scripted patterns?
3. (section 3.3) Do blind policies generalize better than visual policies to part geometries outside the training distribution?
4. (section 3.4) Can blind policies that rely on contact-rich search be transferred zero shot with high fidelity to the real world?

3.1 Proprioception is sufficient

All three tasks achieve near-perfect success on in-distribution geometries (figure 11, left). The joint history alone therefore carries enough information about contact to find an opening, bring teeth into mesh, and engage a thread.

Compared with visual policies, which can typically observe the poses of the grasped and mating parts directly, blind policies may be slower. On peg insertion, the blind policy has to search for the opening through contact, so it takes a less direct path and seats later (figure 9a, b) than a visual policy. The gap is smaller on the other two tasks (Appendix C), plausibly because the grasped part occludes more of the mating part from the camera’s view, so the visual policy, too, has to search to some extent.

3.2 Visually blind but intelligent

If blind policies search reactively rather than follow a script, their history latent z_{t}, through which everything the actor knows about the past passes (figure 7), must encode the state of contact. To test this, we roll out each policy in simulation and label every state with one of seven modes of its task, computed from the privileged part pose. For the peg, these are lifted, beside the plate, searching far from the opening or near it, partly engaged, and fully inside, either moving or stuck; the gear’s and nut’s modes follow similar stages of their tasks (Appendix C). We then fit a linear probe that predicts the mode from z_{t}.

On all three tasks, the probe separates modes that call for different actions, such as lifted, engaged, and stuck (figure 10), and does so more accurately from z_{t} than from the current observation alone (accuracy 61–73%, against 41–64%; chance 14%). Furthermore, the policy heavily relies on these inferred contact states to choose actions. Swapping in the latent of a state in another mode changes the action 1.3–3.0\times as much as a swap within the same mode (figure 15; Appendix C). Fitted only in simulation, the peg probe decodes the same sequence of modes on the real robot (figure 10a).

Reacting to contacts also makes a blind assembly efficient. A fixed pattern sweeps the same path regardless of what the arm feels, whereas a reactive policy can use the information gathered through contact, such as the peg catching the rim of the opening, to steer its search. We test this on peg insertion, where a scripted blind baseline is standard: spiral search, which moves the peg outward along a spiral until it drops into the hole, is a common strategy for insertion under position uncertainty in industrial assembly (Chhatpar & Branicky, 2001; Park et al., 2017; Marvel et al., 2018). From the same starts, the blind policy travels a median 37 mm before seating, against 95 mm for an optimistic spiral that starts flush with the hole opening, and is given the peg’s orientation and the best starting direction (figure 9c).

3.3 Generalization to unseen parts

On all three tasks, both blind policies and visual policies are reliable on the parts they were trained on. However, the blind policy is more reliable on assemblies sampled from outside the training distribution (figure 11). The visual policy is particularly vulnerable to geometry changes that significantly change how the parts look; it consistently underperforms the blind policy across tasks when evaluated on parts that are of a different shape, or extremely different size, than the training distribution. The visual policy learns to align visual features which may not appear in held-out assemblies. The blind policy, on the other hand, learns a searching strategy that is much more transferable.

3.4 Zero-shot transfer to the real world

Setup.

We deploy the frozen blind policy on a Flexiv Rizon 4S with a Robotiq 2F-85 at 50 Hz, with no real-world data or fine-tuning. It receives only encoder-derived joint and gripper state; no camera, force/torque sensor, or hole pose enters the policy. The deployed policy is a policy trained on random convex, smooth, and mixed cross-sections under a handle (Table 2). We test eight 3D-printed pegs (figure 12): five in-distribution objects, and three that are drawn from outside the training distribution, a tighter clearance (0.5 mm per side), a larger cross-section, and a taller handle. Each trial starts from a start state drawn from the simulated reset distribution, with the peg grasped at a randomized position and tilt and touching the fixture near the hole. An operator marks a trial as successful once the peg is seated.

Results.

One policy, with no parameter changes, succeeds in 173 of 237 trials (73%; figure 12), pressing onto the surface, sweeping, and dropping in once the opening is under the peg, as in simulation (figure 1). It succeeds in 73–80% of trials on each in-distribution peg, and in 66%, 66%, and 72% on the tighter, larger, and taller-handled pegs, respectively.

4 Related work

Robotic insertion.

Classical peg-in-hole assembly combines compliance with search patterns such as spirals, rasters, and tilting (Whitney, 1982; Chhatpar & Branicky, 2001; Marvel et al., 2018), in some cases without force feedback (Park et al., 2017); these strategies assume a known grasp and a hole within the pattern and are tuned for each part. Learned insertion policies usually add force/torque sensing (Inoue et al., 2017; Beltran-Hernandez et al., 2020), wrist vision (Schoettler et al., 2020; Bogunowicz et al., 2020; Spector & Di Castro, 2021; Haugaard et al., 2021; Seo et al., 2025), or both (Lee et al., 2019; Luo et al., 2021; Watson et al., 2020). Wrist force/torque sensing combined with a soft wrist has allowed policies to tolerate 10 mm of hole-position error and an unseen peg shape (Fuchioka et al., 2024), while a blind dexterous policy relying on fingertip tactile sensing (Pan et al., ) was demonstrated for precise insertion. Others (Tang et al., 2023; Tang et al., 2024; Noseworthy et al., 2025; Narang et al., 2022) observe joint state and a direct goal pose from perception or calibration with a few millimeters of error. Our policy instead uses proprioception alone: it is given no hole pose, part model, or force signal, and must locate the hole through contact while the peg’s pose in its grasp is unknown.

Proprioceptive policies.

Legged robots traverse rough terrain with policies built on histories of onboard state (Hwangbo et al., 2019; Lee et al., 2020; Siekmann et al., 2021b; Kumar et al., 2021; Radosavovic et al., 2024). In manipulation, proprioceptive policies reorient unseen objects in hand (Chen et al., 2021; Qi et al., 2022), and Blind Dexterity treats joint and action histories as a whole-body tactile channel (Bhatt et al., 2026). These tasks involve forceful contact or centimeter tolerances; insertion needs neither. In imitation learning, robot state can hurt (Team et al., 2024) by suppressing visual learning (Lu et al., 2026).

Simulating tight contact.

Approximate convex decomposition (Mamou & Ghorbel, 2009; Wei et al., 2022) tolerates concavity relative to the mesh size, which can exceed an assembly’s clearance. Exact polygon partitions are classical (Keil, 1985), and simulation benchmarks build holes from primitives by hand (Zhu et al., 2020; Tao et al., 2024; Zakka et al., 2025); mjfit automates this for arbitrary outlines and generates assemblies in bulk. SDFs handle concavity directly and underlie GPU assembly simulation in Factory and its successors (Narang et al., 2022; Macklin et al., 2020; Tang et al., 2023), whose assets obtain their clearance approximately, by displacing mesh vertices or eroding voxels (Tang et al., 2024; Wang et al., 2025); SDF accuracy depends on field resolution and contact search. Newton combines SDFs with hydroelastic contact (The Newton developers, 2025), and barrier methods simulate threads and gear trains without interpenetration (Ferguson et al., 2021; Lan et al., 2022). Our benchmark (section 2.2) follows the simulator comparisons of Erez et al. (2015) and Le Lidec et al. (2024). CRISP (Lee et al., 2026) recently compared engines on tight insertion and threading; we instead keep MuJoCo’s solver fixed and vary only the collision geometry, using passive probes with analytic answers and speed–accuracy Pareto fronts.

5 Limitations and conclusion

We trained blind policies that insert pegs, mesh gears, and thread nuts from proprioception alone, with no fixture pose, grasp, or part model, on thousands of assemblies that mjfit generates with exact collision geometry. The peg insertion policy transfers zero-shot to the real world, and in simulation, blind policies generalize to unseen parts far better than matched visual policies. The peg insertion policy is, however, a last-mile manipulation skill, and its limits are those of one. First, it assumes it starts in contact: every training episode begins with the peg already touching the fixture a few centimeters from the hole, so another motion, scripted or visual, must bring it there. Second, its behavior is local. It searches the surface around where it starts and corrects the misalignments it met in training, but it cannot handle events that call for more than a local correction, such as a peg that slides out from the grasp and lands far from the hole.

Acknowledgments

This research was financially supported by the Ministry of Trade, Industry, and Energy (MOTIE), Korea, under the “Global Industrial Technology Cooperation Center program” supervised by the Korea Institute for Advancement of Technology (KIAT). (Grant No. P0028435)

References

1. Beltran-Hernandez et al. (2020) Cristian C Beltran-Hernandez, Damien Petit, Ixchel G Ramirez-Alpizar, and Kensuke Harada. Variable compliance control for robotic peg-in-hole assembly: A deep-reinforcement-learning approach. Applied Sciences, 10(19):6923, 2020.
2. Bhatt et al. (2026) Aditya Bhatt, Oleg Kaidanov, Puze Liu, and Jan Peters. Blind dexterity: Whole-body humanoid manipulation via pure proprioception, 2026.
3. Bogunowicz et al. (2020) Damian Bogunowicz, Aleksandr Rybnikov, Komal Vendidandi, and Fedor Chervinskii. Sim2real for peg-hole insertion with eye-in-hand camera, 2020. URL https://arxiv.org/abs/2005.14401.
4. Chen et al. (2021) Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning, 2021.
5. Cheng et al. (2024) Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. 2024.
6. Chhatpar & Branicky (2001) Siddharth R Chhatpar and Michael S Branicky. Search strategies for peg-in-hole assemblies with position uncertainty. In IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 3, pp. 1465–1470. IEEE, 2001.
7. Erez et al. (2015) Tom Erez, Yuval Tassa, and Emanuel Todorov. Simulation tools for model-based robotics: Comparison of Bullet, Havok, MuJoCo, ODE and PhysX. In IEEE International Conference on Robotics and Automation, pp. 4397–4404, 2015. doi: 10.1109/ICRA.2015.7139807.
8. Ferguson et al. (2021) Zachary Ferguson, Minchen Li, Teseo Schneider, Francisca Gil-Ureta, Timothy Langlois, Chenfanfu Jiang, Denis Zorin, Danny M. Kaufman, and Daniele Panozzo. Intersection-free rigid body dynamics. ACM Transactions on Graphics, 40(4), 2021. doi: 10.1145/3450626.3459802.
9. Fuchioka et al. (2024) Yuni Fuchioka, Cristian C. Beltran-Hernandez, Hai Nguyen, and Masashi Hamaya. Robotic object insertion with a soft wrist through sim-to-real privileged training. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 9159–9166, 2024. doi: 10.1109/IROS58592.2024.10801575.
10. Google DeepMind and NVIDIA (2025) Google DeepMind and NVIDIA. MuJoCo Warp: GPU-accelerated MuJoCo. https://github.com/google-deepmind/mujoco_warp, 2025.
11. Haugaard et al. (2021) Rasmus Haugaard, Jeppe Langaa, Christoffer Sloth, and Anders Buch. Fast robust peg-in-hole insertion with continuous visual servoing. In Conference on Robot Learning, pp. 1696–1705. PMLR, 2021.
12. He et al. (2024) Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. 2024.
13. Hwangbo et al. (2019) Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science robotics, 4(26):eaau5872, 2019.
14. Inoue et al. (2017) Tadanobu Inoue, Giovanni De Magistris, Asim Munawar, Tsuyoshi Yokoya, and Ryuki Tachibana. Deep reinforcement learning for high precision assembly tasks. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 819–825, 2017. doi: 10.1109/IROS.2017.8202244.
15. Jaderberg et al. (2017) Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2017.
16. Keil (1985) J. Mark Keil. Decomposing a polygon into simpler components. SIAM Journal on Computing, 14(4):799–817, 1985. doi: 10.1137/0214056.
17. Kumar et al. (2021) Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. 2021.
18. Lan et al. (2022) Lei Lan, Danny M. Kaufman, Minchen Li, Chenfanfu Jiang, and Yin Yang. Affine body dynamics: Fast, stable and intersection-free simulation of stiff materials. ACM Transactions on Graphics, 41(4), 2022. doi: 10.1145/3528223.3530064.
19. Le Lidec et al. (2024) Quentin Le Lidec, Wilson Jallet, Louis Montaut, Ivan Laptev, Cordelia Schmid, and Justin Carpentier. Contact models in robotics: a comparative analysis. IEEE Transactions on Robotics, 40:3716–3733, 2024.
20. Lee et al. (2020) Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47):eabc5986, 2020. doi: 10.1126/scirobotics.abc5986.
21. Lee et al. (2019) Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. arXiv preprint arXiv:1907.13098, 2019.
22. Lee et al. (2026) Somang Lee, Sunkyung Park, Jinhee Yun, Seoki An, and Dongjun Lee. CRISP: Contact-rich robotic simulation platform with extensive geometries and contact solvers, 2026. URL https://arxiv.org/abs/2609.21761.
23. Lu et al. (2026) Jingxian Lu, Wenke Xia, Yuxuan Wu, Zhiwu Lu, and Di Hu. When would vision-proprioception policies fail in robotic manipulation? 2026.
24. Luo et al. (2021) Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Wenzhao Lian, Chang Su, Mel Vecerik, Ning Ye, Stefan Schaal, and Jon Scholz. Robust multi-modal policies for industrial assembly via reinforcement learning and demonstrations: A large-scale study. In Robotics: Science and Systems, 2021.
25. Macklin et al. (2020) Miles Macklin, Kenny Erleben, Matthias Müller, Nuttapong Chentanez, Stefan Jeschke, and Zach Corse. Local optimization for robust signed distance field collision. In Proceedings of the ACM on Computer Graphics and Interactive Techniques, volume 3, 2020. doi: 10.1145/3384538.
26. Mamou & Ghorbel (2009) Khaled Mamou and Faouzi Ghorbel. A simple and efficient approach for 3D mesh approximate convex decomposition. In IEEE International Conference on Image Processing, pp. 3501–3504, 2009. doi: 10.1109/ICIP.2009.5414068.
27. Marvel et al. (2018) Jeremy A Marvel, Roger Bostelman, and Joe Falco. Multi-robot assembly strategies and metrics. ACM Computing Surveys (CSUR), 51(1):1–32, 2018.
28. Narang et al. (2022) Yashraj Narang, Kier Storey, Iretiayo Akinola, Miles Macklin, Philipp Reist, Lukasz Wawrzyniak, Yunrong Guo, Adam Moravanszky, Gavriel State, Michelle Lu, et al. Factory: Fast contact for robotic assembly. 2022.
29. Noseworthy et al. (2025) Michael Noseworthy, Bingjie Tang, Bowen Wen, Ankur Handa, Chad Kessens, Nicholas Roy, Dieter Fox, Fabio Ramos, Yashraj Narang, and Iretiayo Akinola. Forge: Force-guided exploration for robust contact-rich manipulation under uncertainty, 2025.
30. (30) Jiahe Pan, Stelian Coros, Jitendra Malik, and Toru Lin. Beyond binary: Sim-to-real dexterous manipulation with physics-aware contact representation.
31. Park et al. (2017) Hyeonjun Park, Jaeheung Park, Dong-Hyuk Lee, Jae-Han Park, Moon-Hong Baeg, and Ji-Hun Bae. Compliance-based robotic peg-in-hole assembly strategy without force feedback. IEEE Transactions on Industrial Electronics, 64(8):6299–6309, 2017. doi: 10.1109/TIE.2017.2682002.
32. Peng et al. (2018) Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation, 2018.
33. Pinto et al. (2018) Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. In Robotics: Science and Systems, 2018.
34. Qi et al. (2022) Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, 2022.
35. Radosavovic et al. (2024) Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(89):eadi9579, 2024.
36. Schoettler et al. (2020) Gerrit Schoettler, Ashvin Nair, Jianlan Luo, Shikhar Bahl, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020.
37. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
38. Seo et al. (2025) Joohwan Seo, Arvind Kruthiventy, Soomi Lee, Megan Teng, Seoyeon Choi, Xiang Zhang, Jongeun Choi, and Roberto Horowitz. Equicontact: A hierarchical se (3) vision-to-force equivariant policy for spatially generalizable contact-rich tasks, 2025.
39. Siekmann et al. (2021a) Jonah Siekmann, Yesh Godse, Alan Fern, and Jonathan Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In 2021 IEEE international conference on robotics and automation (ICRA), pp. 7309–7315. IEEE, 2021a.
40. Siekmann et al. (2021b) Jonah Siekmann, Kevin Green, John Warila, Alan Fern, and Jonathan Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning. 2021b.
41. Spector & Di Castro (2021) Oren Spector and Dotan Di Castro. InsertionNet: A scalable solution for insertion. IEEE Robotics and Automation Letters, 6(3):5509–5516, 2021.
42. Tang et al. (2023) Bingjie Tang, Michael A Lin, Iretiayo Akinola, Ankur Handa, Gaurav S Sukhatme, Fabio Ramos, Dieter Fox, and Yashraj Narang. Industreal: Transferring contact-rich assembly tasks from simulation to reality. 2023.
43. Tang et al. (2024) Bingjie Tang, Iretiayo Akinola, Jie Xu, Bowen Wen, Ankur Handa, Karl Van Wyk, Dieter Fox, Gaurav S. Sukhatme, Fabio Ramos, and Yashraj Narang. AutoMate: Specialist and generalist assembly policies over diverse geometries. In Robotics: Science and Systems, 2024.
44. Tao et al. (2024) Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI, 2024. URL https://arxiv.org/abs/2410.00425.
45. Team et al. (2024) Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. 2024.
46. The Newton developers (2025) The Newton developers. Newton: GPU-accelerated physics simulation for robotics. https://github.com/newton-physics/newton, 2025.
47. Tobin et al. (2017) Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017.
48. Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026–5033. IEEE, 2012.
49. Wang et al. (2025) Yian Wang, Bingjie Tang, Chuang Gan, Dieter Fox, Kaichun Mo, Yashraj Narang, and Iretiayo Akinola. MatchMaker: Automated asset generation for robotic assembly. In IEEE International Conference on Robotics and Automation, 2025.
50. Watson et al. (2020) James Watson, Austin Miller, and Nikolaus Correll. Autonomous industrial assembly using force, torque, and rgb-d sensing. Advanced Robotics, 34(7-8):546–559, 2020.
51. Wei et al. (2022) Xinyue Wei, Minghua Liu, Zhan Ling, and Hao Su. Approximate convex decomposition for 3D meshes with collision-aware concavity and tree search. ACM Transactions on Graphics, 41(4), 2022. doi: 10.1145/3528223.3530103.
52. Whitney (1982) Daniel E Whitney. Quasi-static assembly of compliantly supported rigid parts. 1982.
53. Zakka et al. (2025) Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground, 2025.
54. Zakka et al. (2026) Kevin Zakka, Qiayuan Liao, Brent Yi, Louis Le Lay, Koushil Sreenath, and Pieter Abbeel. mjlab: A lightweight framework for gpu-accelerated robot learning, 2026.
55. Zhu et al. (2020) Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddukuri, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning, 2020. URL https://arxiv.org/abs/2009.12293.

Appendix A Training details

Study and deployed policies.

To cover as much real-world variation as possible, the deployed peg policy (section 3.4) is trained on the full ranges of part geometry, physics parameters, and object pose that we can (limited by physics stability, geometric constraints like the gripper width, etc.). We also train similarly diverse gear and nut policies for the latent probe (section 3.2; Appendix A.2). In order to test the policies’ capacity for generalizing out of the training distribution, we design study training ranges for all three tasks (section 3) that are more narrow (Table 1), so that we can systematically widen each property beyond training (section 3.3).

A.1 Study policies

Tasks.

Physics runs at 1 kHz under a 50 Hz policy, and each world holds one part and fixture drawn from mjfit’s generators. Peg insertion: each episode starts with the peg touching the fixture’s top face, a median 1 cm from the hole center. It succeeds when the entire peg bottom face is seated in the hole. Bonuses of +1 reward the peg’s first capture by the opening and success. Contact labels mark each of 64 points on the peg’s bottom edge as free (more than 1 mm above the fixture’s top face), in contact (on the rim or top face, within 1 mm), or inside the opening. An episode lasts at most 5 s. Gear meshing: each world holds a spur-gear set from mjfit’s gear generator, and the robot holds the gear by a printed boss. Each episode starts with the gear resting on the shaft tip, off the shaft axis, with its teeth out of phase with the anchors’. It succeeds when the bore surrounds the shaft and the gear sits within 1 mm of the shaft’s collar, which the anchors’ teeth allow only in mesh. Bonuses reward the first capture of the shaft, the first mesh with every anchor, and success; contact labels mark 64 points on the shaft’s rim (free, on the gear, or inside the bore) and 64 on the gear’s tooth tips (free, on an anchor, or in mesh); an episode lasts at most 10 s. Nut threading: each world holds an ISO metric nut and bolt pair from mjfit’s thread generator. Each episode starts with the nut resting on the bolt tip, tilted and off the bolt axis, short of the turn at which the thread catches it. It succeeds a quarter turn after the thread catches, with the nut within 1 mm of the bolt axis; bonuses reward seating, capture, each new extent of the turn, and success; contact labels mark 64 points on the rim of the nut’s bore (free, on the bolt’s tip, around the bolt, or caught by the thread, within a band of 0.29 pitch, 1 mm at M30\times3.5); an episode lasts at most 15 s. Every episode also ends when the part moves far from the fixture, and a nut episode when the nut leaves the grasp.

Architecture.

The actor observes 33 values per step: seven joint positions, seven joint velocities, the gripper pose and twist (twelve), and its previous action (seven). The nut policy’s actor encodes the gripper orientation by two columns of its rotation matrix rather than by Euler angles (36 values), since the wrist turns through half a revolution while threading. A temporal convolutional encoder maps the last 30 observations (0.6 s) to a 32-dimensional latent; an MLP (256, 256, 128 units) maps the latent and the current observation to seven joint-position increments, scaled by the action scale of Table 1 and added to the measured joint positions. The critic sees the part’s pose relative to the fixture, the contact labels, and, for the nut, the thread’s geometry (240, 434, and 317 inputs for the peg, gear, and nut).

PPO.

12,288 worlds, 32-step rollouts, 5 epochs of 20 minibatches (24 for the gear and nut), fixed learning rate 3\times 10^{-4}, clip 0.2, \gamma=0.98, \lambda=0.95, entropy coefficient 0.003, value-loss coefficient 1 with clipping, gradient norm clipped at 1, and initial action standard deviation 0.5. The contact decoder has two hidden layers of 128 units and class weights 0.25 (free), 3.0 (contact), and 2.0 (inside, in mesh, around the bolt, or caught). The visual policies see 64\times64 wrist RGB images over the same 30 steps.

A.2 Deployed and full-range policies

Deployed peg policy.

The policy we deploy (section 3.4) trains as the study peg policies do, on the ranges of Table 2: random convex cross-sections under a handle, and wider contact and pose ranges. Its starts lie a median 2 cm from the hole center. It uses the gear’s and nut’s controller and a staged safety penalty, which starts once the task has been learned without safety limits and penalizes joint speeds and commanded torques above one sixth of the hardware limits. PPO uses 24 minibatches and an exploration anneal, during which the learning rate adapts to a target KL of 0.01.

Full-range gear and nut policies.

The gear and nut probes of section 3.2 use policies trained on the full ranges. Gears have a module of 2–3.5 mm and 12–28 teeth, pressure angles of 20–25∘, tooth widths of 0.44–0.48 of the pitch, face widths of 8–15 mm, 0.2–0.6 mm of backlash per anchor, and 0.5–1 mm of bore clearance, on 4–6 mm shafts 3–10 mm above the gear; with the boss they weigh 128–276 g, set by their geometry. Nuts are 870 ISO metric nut and bolt pairs, M24–M42 at coarse and fine pitch, with 0.05–0.2 mm of flank clearance, across flats and heights within 10% of ISO 4032, and steel masses of 73–855 g. Their contact and pose ranges are wider too: anchor-tooth friction 0.2–1.0, thread friction 0.2–0.4, and the deployed peg policy’s plate, finger, gripper, fixture, and gripper-rotation ranges. Gears start a median 12 mm off the shaft axis; nuts start a median 19 mm off the bolt axis and 60–170∘ of turning short of the thread catching. With deterministic actions, the gear policy succeeds in 98% of 8,192 episodes on 4,096 unseen gear sets drawn from its training distribution, and the nut policy in 93% of 8,192 episodes on 512 unseen nut and bolt pairs drawn likewise.

Appendix B Collision benchmark details

All scenes use a 1 ms time step, the implicit-fast integrator, the Newton solver, \mathrm{solref}=(2\,\mathrm{ms},1), \mathrm{solimp}=(0.99,0.999,0.001), \mu=1, and zero margin. Baselines receive an exact, watertight plate mesh (the circle tessellated at 1,024 segments, 0.04 µm sagitta), and ground truth uses the analytic circle, so mjfit is charged for its own chord error. The peg is always an exact MuJoCo primitive, so every difference comes from the hole. Timings come from one idle machine (Intel i9-13900KS, NVIDIA RTX 5090). Warp contact and constraint buffers are calibrated per representation, and every GPU run is checked for buffer overflow, which would silently drop contacts. At a 2 ms contact time constant, frictionless drops let pegs up to 20 µm oversize slip in for every method, including exact geometry, so S1 cannot resolve geometric errors below about 20 µm; the play probes can.

Setup.

The round and square holes are 16 mm wide in a 15 mm plate, with radial clearance c from 20 to 200 µm; at c=20 µm, 2c/D=0.0025. The M10\times1.5 bolt and nut have flank clearance c of 20 and 100 µm and are probed for axial, radial, and tilt play and for screwing under load. CPU times use one environment on one thread; GPU times use 512 MuJoCo Warp worlds, except in figure 6b, which uses 256 and measures the error on Warp as well.

Appendix C Additional results

Behaviour.

The blind policy travels farther than the visual policy from 99% of matched starts (Table 5). On unseen random convex pegs, the blind policy succeeds in 36.2% of episodes and the visual policy in 3.9% (both seeds); these pegs have a median 53% of the matching stadium’s area.

Speed on gear meshing and nut threading.

On the in-distribution episodes of the study policies of figure 11 (8,192 per policy, one seed, deterministic actions), counting successful episodes only, the blind gear policy succeeds in a median 0.58 s against the visual policy’s 0.64 s, and takes longer on 43% of the 8,059 starts from which both succeed. Plausibly, the held gear hides the shaft from the wrist camera while it aligns, so visual policies also have to search. The blind nut policy succeeds in a median 0.76 s against 0.64 s, and takes longer on 98% of the 8,139 starts from which both succeed (by a median 0.10 s).

Contact supervision.

Without the contact loss (section 2.3), the blind policies still learn all three tasks, but need 2.7 (peg), 3.8 (gear), and 5.1 (nut) times as many environment steps to reach 90% training success (figure 14). The unsupervised peg policy reaches 97.8% after 486M steps; the supervised policies pass 99% by 242M.

Probe details.

The search modes are split at 5 mm from the opening, and an inside peg is stuck when it progresses less than 0.5 mm in 0.2 s. The probe uses the deployed policy (section 3.4). We roll it out in 1,536 worlds of its training distribution (1.15M states), train the probes on even-numbered worlds, and test on odd ones. Balanced accuracy over the seven modes is 70% (chance 14%) from the history latent, against 56% from the current observation alone and 33% from a randomly initialized encoder. The history latent recognizes lifted, moving, and stuck pegs with 84–90% recall; it decodes far search as near 19% of the time, and near as far 26%. Swapping latents between a searching and an inside state changes the action 1.4–1.9\times as much as swapping between two states of the same mode; swapping between near and far changes it 0.99–1.11\times as much (figure 15a). On the robot, we run the policy in 30 further trials on peg I3 (figure 12); its actions recomputed from the logged observations match the logged ones to within 2\times 10^{-5}. At the end of successful trials, the peg pose estimated from the joint encoders and planned grasp is a median 6.6 mm (10th–90th percentile 2.6–17.4 mm) from the seated pose, as the peg slips and turns in the fingers, and places only 3% of their states fully inside the hole. Scored states have every point of the peg’s bottom edge at least 3 mm above the plate (lifted) or at least 3 mm inside the opening (inside). The estimate never places the peg clearly above the plate; the probe labels 62% of the inside states correctly, against 100% in simulation from the same start states, and labels the final state of 17 of the 24 successful trials as inside. In figure 10a, true and decoded modes of the start state shown agree in simulation on 74% of steps (median over the 30 start states: 65%).

The gear and nut probes use the full-range policies of Appendix A.2 (PPO updates 3,085 and 2,000) with the same recipe: one episode with deterministic actions in each of 1,536 worlds of held-out geometries from their training distributions (133k and 85k states), probes trained on even-numbered worlds and tested on odd ones. The modes come from the environments’ contact labels and the part’s pose: lifted; off the shaft (gear only; the nut leaves its bolt too rarely to probe); resting on the shaft or bolt tip and searching, far or near (split at the median start distance, 7.2 and 19.7 mm); entering, with the tip inside the bore but not yet captured; for the gear, descending the shaft before its teeth reach the anchors, then meshing, with its teeth on the anchors’ tops or in mesh; for the nut, seated on the bolt before the thread catches, then threading, or stuck when it advances less than 0.05 pitch in 0.2 s. Balanced accuracy over the seven modes is 61% for the gear and 73% for the nut (chance 14%), against 41% and 64% from the current observation and 29% and 46% from a randomly initialized encoder; near search is decoded as far 42% and 31% of the time. Swapping latents between a searching and an engaged state changes the action 1.3–1.6\times (gear) and 1.4–3.0\times (nut) as much as swapping between two states of the same mode; swapping between near and far changes it 0.97–1.04\times and 0.94–1.18\times as much (figure 15b, c). The start states shown come from a re-run of each rollout, decoded with the same probes and rendered from the same states. Among the episodes that end engaged and visit at least five modes, the nut’s agrees with the true modes on 74% of steps (the median), and the gear’s on 79% (the 75th percentile; median 69%).

Unseen parts.

Each unseen part of figure 11 widens one property of the part beyond its study training range (Table 1) while every other range stays at its study value. Table 6 lists the widened ranges for each task, grouped as in the figure: a tighter fit; a mild and a strong change of the part’s proportions; a mild and an extreme change of size, on both sides of the training range; and a new shape. Table 7 lists every bar.

Real-world pegs.

The pegs of figure 12 are 3D printed under an 80\times44 mm stadium handle 40 mm tall, except H, whose handle is 70 mm tall; they weigh 147–164 g, and every hole is 20 mm deep. I1–I5 are cross-sections drawn from the deployed policy’s training distribution (polygonal, smooth, and mixed outlines) with 1.11–1.49 mm of clearance per side. C is a polygonal peg with 0.5 mm of clearance; L enlarges a training outline by 1.28\times1.09 to a 101\times37 mm hole; H puts a mixed outline under the taller handle. Each peg has 30 planned resets, and each reset counts once, by its first attempt that reached a verdict: the operator’s success or failure, or the 20 s horizon, which counts as a failure. Each simulated rate is the same policy over 2,048 worlds of that peg’s geometry at the same 20 s horizon, with deterministic actions.