Kontinuierliches Lernen für 6-DoF-Greifsynthese durch Erfahrung und Demonstrationen

Abstract

Most current grasp synthesis systems are trained offline and remain fixed during deployment. While this works well when deployment conditions resemble the training data, performance can degrade when robots encounter conditions they have not seen before, such as unfamiliar objects. In this work, we present a continual-learning framework for single-view 6-DoF grasp synthesis for a parallel-jaw gripper in cluttered scenes. Rather than finetuning a large parametric model, our method adapts through memory in a learned embedding space: grasp outcomes update future grasp scores, while optional user demonstrations are recalled and transferred to new scenes as additional candidate grasps. We evaluate our method in simulation and in extensive real-world experiments comprising over 1500 grasp trials. We show that our method matches the performance of existing 6-DoF grasping baselines even before adaptation, improves online on unseen objects from categories absent or underrepresented during training, and supports long-horizon continual learning with limited forgetting. In real-world experiments, our method reaches over 90% success rates on several challenging object categories after only 50 online grasp attempts. Videos and code at https://giuschio.github.io/cl_grasping/.

Keywords: Grasping, Continual Learning, Robot Manipulation

1 INTRODUCTION

Current 6-DoF grasp synthesis systems can generate grasps in cluttered scenes with impressive reliability. Most do so by training parametric models on large offline datasets of synthetic or real-world grasp examples [6, 1, 10, 3, 29]. Once trained, however, these models are typically frozen during deployment, and can still fail when the robot encounters conditions that are poorly covered by the training distribution, such as unfamiliar object geometries that were absent or underrepresented during training. One promising direction is to let grasping systems continue learning on site from the feedback available to them, such as grasp outcomes and occasional user demonstrations.

In this work, we present a continual-learning framework for 6-DoF grasp synthesis. As in many prior works [10, 30], our system (Figure 1) has two main components: a proposal module that generates candidate grasps and a scoring module that ranks them for execution. Unlike standard learned grasping systems, however, both modules can change during deployment through memory. Instead of using a parametric grasp scorer that remains fixed after training, we use a memory-based scorer in a learned embedding space, so new grasp outcomes immediately influence future scores. We also augment heuristic grasp proposals with lightweight recall from user demonstrations, transferring previously shown grasps to geometrically similar regions in new scenes. Together, these mechanisms let the system combine offline experience, deployment trials, and demonstrations into a single adaptive system. In addition, because adaptation only requires adding entries to memory, each update is simple and practical to apply on site, without requiring gradient-based retraining.

We evaluate our method in simulation and on a real robot and compare it with common 6-DoF grasping baselines. Our method achieves strong offline performance, improves online through deployment experience on unfamiliar objects, and supports long-horizon continual learning with limited forgetting. In real-world experiments comprising over 1500 grasp trials, online adaptation improves performance across challenging object categories and reaches over 90% success on several of them.

2 RELATED WORK

2.1 Deep Grasp Synthesis

6-DoF grasp synthesis aims to predict feasible SE(3) grasp poses from scene observations. A common strategy is a proposal-and-score pipeline: generate a set of candidate grasps, then rank them according to predicted grasp success. Some methods sample candidates with geometric heuristics, such as GPD [30] and EdgeGraspNet [10]. Others evaluate dense anchors over points or voxels, as in VGN [3], ICGNet [36], ContactGraspNet [29], GraspNet [6], and AnyGrasp [5]. Recent progress has come from larger datasets and stronger priors, such as adversarial sim-to-real alignment [35], antipodal constraints [17], and center-of-gravity priors [5]. Other methods learn the proposal distribution with VAEs [19] or diffusion models [20, 28, 2]. Despite this progress, current systems remain tied to the coverage of their training data, and can still fail when deployment scenes differ from the examples seen offline.

2.2 Adaptation and Continual Learning for Grasping

Prior work has addressed the problem of how grasping models should be adapted during deployment. Existing approaches use self-training signals [8, 15], continual-learning regularizers [23], reinforcement learning [27, 14], or retraining with deployment data [12]. Other works exploit structural invariances in 2D top-down grasping to adapt from few examples [33]. These methods show that deployment data is valuable, but often consider settings with simplifying assumptions, such as top-down grasping or access to test-time CAD models. We instead consider single-view 6-DoF grasping in scenes with multiple objects. Our approach also differs in how adaptation is performed: rather than updating the parameters of the grasping model, we incorporate deployment experience non-parametrically by adding entries to memory. In our experiments, we compare our approach against parametric finetuning using the protocol of Julian et al. [12].

2.3 Registration- and Retrieval-Based Grasping

Retrieval-based methods store grasps and transfer them to new scenes through registration or correspondences. Early work transfers grasps geometrically [13]. Later methods integrate category templates [32], semantic correspondences [11], or continual retrieval from successful grasps [21]. These frameworks can often incorporate new demonstrations without retraining, but they depend on database coverage and can be sensitive to registration errors. Our method differs in three main ways. First, recall adds to geometric sampling instead of replacing it; the system can still propose grasps for unfamiliar objects. Second, whereas retrieval methods learn from successful grasps or demonstrations only, our method can learn from both successful and failed grasp attempts. Third, registration is used only to generate candidates, not to determine their scores. This allows us to reject a well-registered recall when the accumulated outcome evidence indicates that the transferred grasp is unlikely to succeed. In our experiments, we isolate these differences by comparing against a recall-only pipeline that disables geometric sampling and adaptive scoring.

2.4 Exploratory and Trial-Based Grasp Learning

Exploratory and trial-based grasp learning methods are closely related to our work. Dex-Net [18], BORGES [4], and LEGS [7] maintain probabilistic grasp success estimates and update them from heuristic evaluations or real-world trials. These works primarily target efficient grasp dataset generation [18] or exploratory grasping for a single isolated object [7]. They also tend to treat grasps independently, maintaining separate success estimates for each candidate grasp or object pose. We build on the same trial-based idea, but use it for 6-DoF grasping from partial observations in cluttered scenes. Rather than maintaining independent beliefs over individual grasps, we operate in a learned embedding space, so evidence from one grasp attempt can generalize to similar grasps.

3 METHOD

We use a 7-DOF robotic arm with a depth camera and a parallel-jaw gripper operating in a 30 × 30 \mathrm{c}\mathrm{m} tabletop workspace. From a single depth observation, captured from a randomly sampled viewpoint, the robot must generate feasible 6-DoF grasp candidates. Our pipeline is organized into two main modules: a proposal module which generates grasp candidates from the current observation, and a scoring module which ranks them for execution according to their estimated success likelihood. The full pipeline is shown in Figure 1.

3.1 Grasp Proposal

At each inference step, we construct the proposal set (i.e. the set of candidate grasps that will be scored later) from two sources: geometric sampling and demonstration recall. This lets us generate a broad set of grasp candidates for any scene, while leveraging demonstrations to add object-specific grasp modes that might be missed by the fixed heuristics. Our geometric sampler follows the contact-based 6-DoF proposal strategy which is common in prior work [10, 29, 36]: it samples approach and contact points from the current pointcloud and uses the local surface normal to define the gripper orientation. Alongside these contact-based grasps, we also include top-down proposals with a fixed vertical approach direction, which we empirically find useful on smaller objects for which depth observations give unreliable side normals. For demonstration recall, we query a recall memory in which each entry consists of a demonstrated grasp and the local spherical pointcloud patch observed around it. We match these stored patches to regions in the current scene using FPFH features [25] followed by point-to-plane ICP registration [24]. When registration succeeds, the estimated transform maps the demonstrated grasp into the current scene, adding it as a new proposal. Finally, we filter all sampled and recalled proposals for kinematic feasibility and collision.

3.2 Grasp Scoring

After sampling, we score every candidate (both sampled and recalled). We do this in two steps. First, an encoder maps each grasp to a low-dimensional embedding space. Then, a memory-based scorer estimates the success probability of each grasp by aggregating nearby labeled data points, including data from offline training, real deployment grasp outcomes and demonstrations. The advantage of this approach over an end-to-end parametric model is that it can be updated instantly by adding new data to the memory.

3.2.1 Encoder Training

We represent each grasp by a local pointcloud patch expressed in the grasp frame. We serialize the patch with a Basis Point Set [22] and pass it through an MLP encoder. Let x_{i} denote the serialized pointcloud for grasp i, let y_{i}\in\{0,1\} denote its binary success label, and let z_{i}=E_{\phi}(x_{i}) denote its embedding, where E_{\phi} is the encoder. The encoder outputs a 32-dimensional embedding, which we normalize to lie on the unit sphere.

Following prior work on training embeddings for non-parametric classification [26], we train the encoder on a mixed soft nearest-neighbor and reconstruction loss. The soft nearest-neighbor loss [26], computed over a batch of size B, encourages grasps with the same outcome to cluster together:

\mathcal{L}_{\mathrm{SNN}}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\sum_{j\neq i}\exp\!\left(-\frac{\|z_{i}-z_{j}\|^{2}}{\tau_{\mathrm{train}}}\right)\mathbf{1}[y_{j}=y_{i}]}{\sum_{j\neq i}\exp\!\left(-\frac{\|z_{i}-z_{j}\|^{2}}{\tau_{\mathrm{train}}}\right)}.

Here \tau_{\mathrm{train}} is a temperature parameter. The reconstruction loss, parametrized through an auxiliary decoder D_{\psi}, keeps the embedding tied to local geometry:

\mathcal{L}_{\mathrm{rec}}=\frac{1}{B}\sum_{i=1}^{B}\|x_{i}-D_{\psi}(z_{i})\|_{2}.

We first pretrain the encoder with \mathcal{L}_{\mathrm{rec}}, and then optimize the mixed loss \mathcal{L}_{\mathrm{embed}}=\alpha\,\mathcal{L}_{\mathrm{SNN}}+(1-\alpha)\,\mathcal{L}_{\mathrm{rec}} (we use \alpha=0.8). Additional embedding and scorer implementation details are provided in Appendix D, while Appendix H compares this soft nearest-neighbor-trained embedding against one trained using a binary cross-entropy loss.

3.2.2 Memory-based scorer

To score each grasp, we aggregate local evidence in the learned embedding space produced by the encoder. We maintain a scoring memory \mathcal{M}=\{(z_{i},y_{i},\rho_{i})\}_{i=1}^{N}, where each entry contains a grasp embedding z_{i}, a binary outcome label y_{i}\in\{0,1\}, and a base evidence weight \rho_{i}. We initialize the scoring memory with offline training data and then expand it during deployment with grasp outcomes and demonstrations. At inference, given a query grasp g_{q}, we compute its embedding z_{q} and retrieve its K nearest labeled neighbors from the scoring memory. We model our belief over the success probability s_{q} with a Beta posterior p(s_{q}\mid\mathcal{M})=\mathrm{Beta}(s_{q};\alpha_{q},\beta_{q}), where each neighbor contributes evidence in the form of pseudo-counts. Each neighbor contributes a pseudo-count weighted by its base weight and by an RBF kernel of its distance to the query,

w_{j}=\rho_{j}\exp\!\left(-\frac{\|z_{q}-z_{j}\|^{2}}{\tau_{\mathrm{eval}}}\right),

where \tau_{\mathrm{eval}} is a bandwidth parameter. For each query, we cap the total contribution of offline neighbors at W_{\max}^{\mathrm{off}} pseudo-counts. This prevents the offline part of the scoring memory from dominating the posterior and slowing adaptation, while preserving the local offline success-to-failure ratio. This gives the posterior parameters

\displaystyle\alpha_{q}=\alpha_{0}+\kappa^{\mathrm{off}}\sum_{j\in\mathcal{N}_{\mathrm{off}}}w_{j}y_{j}+\sum_{j\in\mathcal{N}_{\mathrm{on}}}w_{j}y_{j},

\displaystyle\beta_{q}=\beta_{0}+\kappa^{\mathrm{off}}\sum_{j\in\mathcal{N}_{\mathrm{off}}}w_{j}(1-y_{j})+\sum_{j\in\mathcal{N}_{\mathrm{on}}}w_{j}(1-y_{j}).

where \mathcal{N}_{\mathrm{off}} and \mathcal{N}_{\mathrm{on}} denote the offline and online neighbors of g_{q}, \kappa^{\mathrm{off}} is the offline rescaling factor, and \alpha_{0} and \beta_{0} are priors that define the default score for grasps with little nearby evidence. The grasp score is the posterior mean,

\mathbb{E}[s_{q}]=\frac{\alpha_{q}}{\alpha_{q}+\beta_{q}}

which increases when nearby evidence is mostly successful and decreases when it is mostly failed. We use different base weights \rho_{i} for offline and online evidence, with higher weight on online outcomes to improve the speed of adaptation. Demonstrations are weighted adaptively from the current posterior odds, choosing enough positive evidence for the demonstrated grasp to exceed the current best grasp by a margin, while enforcing a minimum weight so every demonstration affects the scorer. Appendix E further discusses how these weights are tuned.

3.3 Online Operation

We use simulation data, collected using geometric heuristic grasp sampling, to train the encoder and initialize the memory-based scorer, which gives the robot a strong prior before deployment-time adaptation. During deployment, we add each binary grasp outcome to the scoring memory, so trial-and-error immediately informs the scoring of future candidates. Labels are automatic: on the real robot, we read the gripper finger positions and label a grasp as successful if the resulting finger width is greater than zero. User demonstrations provide an optional second source of adaptation. When a demonstration is available, we add it to the scoring memory with an adaptive weight chosen from the current posterior, so that the demonstrated grasp exceeds the current best candidate by a fixed margin. This encodes the intuition that user-demonstrated grasps should be preferred in sufficiently similar conditions, while still letting the required evidence depend on the current local memory. We also add the demonstration to the recall memory, allowing it to be transferred to future scenes with similar local geometry. The system can therefore adapt autonomously from grasp outcomes alone, while demonstrations can additionally introduce grasp modes missing from the geometric sampler.

4 Experiments

We evaluate our method in simulation and on a real robot. The experiments first test the base model trained from simulation data, then measure how deployment experience changes performance on unseen objects from absent or underrepresented categories, both with category-specific adaptation and in a continual-learning setting.

4.1 Simulation Experiments

We use the PyBullet simulation benchmark introduced by VGN and later adopted by EdgeGraspNet and ICGNet [3, 10, 36]. Following the packed setting from VGN, each cluttered scene contains one to five upright objects placed close together on a table. We train on the VGN object set, which is also used by EdgeGraspNet and ICGNet. For evaluation, we replace the training objects with 443 unseen objects from DexGraspNet [31], grouped into ten semantic categories: Airplane, Animal, Bottle, Bowl, Drill, Fasteners, Hammer, Mug, Pliers, and Screwdriver. We chose categories underrepresented in training but representative of objects a robot might encounter. All instances are unseen. We give a more detailed analysis of the test object distribution in Appendix A.

Baseline Performance and Single-Category Adaptation

We first compare the offline version of our model against existing 6-DoF grasping baselines (namely ICGNet [36], ContactGraspNet [29], EdgeGraspNet [10]), before any deployment-time adaptation. As shown in Table 2, our method matches the strongest baseline, EdgeGraspNet, across the tested categories and slightly improves the average success rate. The non-parametric scorer therefore does not trade away offline performance.

We then adapt to each object category independently. For each category, the robot collects 100 grasp attempts and receives at most 5 user demonstrations. First, we remove demonstration recall and adapt the scorer using automatically observed grasp outcomes only. This isolates outcome-based scoring adaptation and measures how much the system can improve autonomously through trial and error. Second, we remove geometric sampling and the memory-based scorer, so grasps are generated only by recalling demonstrations. This recall-only ablation serves as a proxy for methods that transfer grasps to new scenes through geometric retrieval and registration [21, 13, 32]. Third, we finetune EdgeGraspNet using the same 100 per-category grasp labels and a balanced loss that weights category-specific data and the original EdgeGraspNet training data equally. This provides a parametric adaptation baseline under the same 6-DoF grasping setting and deployment-data budget. The 50/50 mixture follows the finetuning protocol of Julian et al. [12].

As shown in Table 2, online adaptation improves performance on all evaluated categories. Our full method reaches 98.1% average success, up from 94.6% for the base model, and achieves at least 98% success on 7 of 10 categories. Adapting the scorer alone already performs well, reaching 97.1% success rate on average. Recall-only performs worse because each category contains many objects and configurations, while only a small number of demonstrations is available. This supports our use of recall as a supplement to geometric sampling rather than as the sole proposal mechanism. Finetuning yields only a small gain, from 92.9% to 93.5%, suggesting that 100 category-specific labels are not enough to effectively finetune a large parametric model, even with access to the original training data. The contrast is also computational: our method adapts by adding memory entries at negligible cost, while finetuning takes one hour per category on an NVIDIA RTX 4070 GPU.

Continual Learning

We next test the same adaptation mechanism in a longer continual-learning sequence. The robot adapts to all ten evaluation categories in the order shown in Table 3 (the order is chosen arbitrarily). Each stage follows the same protocol as the previous experiment: 100 adaptation attempts on the current category, with a user demonstration after every failed attempt and a maximum of 5 demonstrations. Table 3 reports the results. Our approach successfully adapts to each category sequentially, while previous categories remain strong rather than being forgotten. At the end of the sequence, our model has adapted to ten evaluation categories covering 443 unseen objects. The worst performance drop due to forgetting is 2.4 percentage points, and the final model outperforms the base model on every object category. Memory and runtime costs remain small: after adaptation, the memory footprint is 1.35 MB and the full planning cycle remains below 1.5 s. Appendix J analyzes how the runtime and memory costs of our method scale with accumulated experience.

4.2 Real-World Experiments

Next, we move to the real robot. Our setup uses a Franka Panda arm [9], a parallel-jaw gripper, and a RealSense D435i depth camera. We use 54 objects: one control set of 13 objects and five challenge sets, with 11 mugs and bowls, 9 kitchen tools, 5 pliers, 7 screwdrivers, and 9 toys (Figure 2). The control objects are modeled on the evaluation set used by ICGNet [36], and roughly match the training distribution. The challenge sets stress real-world effects such as thin geometry, material variation, and uneven mass distributions. We construct cluttered scenes using the same protocol as in simulation. We evaluate non-targeted clearing without semantic segmentation. At each step, the robot executes the highest-scoring collision-free grasp, removes the object if successful, and repeats until the scene is cleared or terminated. We report grasp success rate (i.e. the fraction of attempts that succeed) and scene-clear rate (i.e. the fraction of scenes fully cleared). If no feasible grasp is found, the robot replans from the next viewpoint in a fixed fallback sequence, up to three times, reducing sensitivity to unfavorable initial views while preserving deterministic comparisons across methods. If all three viewpoints fail, we terminate the scene and count one failed attempt. In total, our evaluation comprises more than 1500 real-world grasp attempts.

Baseline Performance and Single-Category Adaptation

We first adapt to each real-world category independently. For each category, we allow up to 50 grasp attempts for adaptation and provide a user demonstration after each failed attempt. We then evaluate our model before and after adaptation, and compare it against EdgeGraspNet, AnyGrasp [5], and M2T2 [34]. For each category, we construct 20 evaluation scenes and use the same set of scenes for all methods. Because each scene contains multiple objects, each category-level evaluation corresponds to several dozen executed grasps per method1. The first six columns of Table 4 summarize the category-level results. On the control set, our base model is comparable to the strongest baselines, reaching 80.0% success compared with 80.2% for EdgeGraspNet and 78.8% for AnyGrasp. Across the challenge sets, AnyGrasp is generally the strongest external baseline, but our base model outperforms all baselines on four of the five sets, with AnyGrasp slightly ahead on toys. The gap is largest on pliers and screwdrivers, where AnyGrasp, the strongest baseline on both, reaches 28.6% and 54.9% success, compared with 62.5% and 85.7% for our base model. EdgeGraspNet and M2T2 perform substantially worse on these two sets. EdgeGraspNet exhibits a particularly clear proposal-generation failure on pliers and screwdrivers, where it achieves 0.0% success and 0.0% scene-clear rate. On these thin objects, it rarely proposes any collision-free grasps, and the few valid proposals fail. We attribute this failure mode to its contact-normal-based grasp-candidate sampling, a common strategy in current methods [36, 10, 29]. This heuristic works well for bulky objects with clear side surfaces, but struggles on thin geometries, where side normals and sharp edges are difficult to recover from depth. Our model avoids this failure mode by complementing contact-based proposals with top-down grasps, which do not require estimated normals. 2 Nevertheless, our base model still fails in some cases. Some geometrically plausible grasps slip on low-friction surfaces or become unstable because the object has a non-uniform mass distribution. In other cases, the geometric grasp sampler misses grasp modes when depth observations are incomplete.

Online adaptation increases success rates across all categories, often substantially. After adaptation, our method achieves the highest success rate among all evaluated methods in every category and exceeds 90% success in five of six categories. Qualitatively, the selected grasps change in ways consistent with the observed failures: repeated outcomes shift grasps away from low-friction areas and toward the center of mass, while demonstrations add grasp modes missing under partial observations. We show some examples of this in Figure 3. However, there remain cases where adaptation helps only partially. On the pliers category, success improves only from 62.5% to 68.3%. This is because one plier instance requires a grasp strategy disjoint from those that work on the others: grasps that work on the other pliers tend to fail on this one, and vice versa. Since these instances are difficult to distinguish from geometry alone, our system selects the strategy that works most reliably on average, which in turn causes repeated failures on this specific pliers instance.

Continual Learning

We construct 20 additional mixed scenes containing objects from all categories. We do not build this model by adapting to each category sequentially. Instead, we start from the category-specific models learned in the previous experiment, and merge their scoring and recall memories. We then evaluate this merged-memory model directly, without additional finetuning, consolidation, or corrective interaction. The final column of Table 4 reports this mixed-scene evaluation: the merged-memory model reaches 89.6% success and 100.0% scene-clear rate, compared with 72.4% and 85.0% for the base model, 69.2% and 80.0% for AnyGrasp, 50.0% and 45.0% for M2T2, and 60.0% and 30.0% for EdgeGraspNet. This shows that independently collected scoring and recall memories can be combined without destructive interference in our real-world setting. It also points to a possible form of parallel adaptation: robots deployed in different environments, such as different households, could collect grasp outcomes and demonstrations independently, then pool the resulting memories.

5 Limitations

First, like other non-parametric methods, our approach must manage the memory and runtime cost of accumulated experience. In our system, the main cost is the runtime associated with demonstration recall. This was not a limiting factor in our experiments, even after adapting to hundreds of objects. For larger deployments (i.e. thousands of objects, hundreds of demonstrations), semantic filtering before recall could reduce this cost by recalling only relevant demonstrations. Appendix J analyzes how the runtime and memory costs of our method scale with accumulated experience. Second, our pipeline relies on geometric observations alone. This can fail when geometry is not enough to distinguish similar objects that require different grasp strategies, as in the pliers failure case in our real-world experiments. Incorporating instance-level cues could help resolve these ambiguities. Third, our adaptation strategy cannot recover geometry that is missing or distorted in the input observation. This limitation is not specific to our camera: depth sensors generally have failure modes, and reflective, transparent, or metallic objects remain challenging for many perception pipelines. In preliminary experiments, we tested learned depth enhancement models [16]. Although they helped in some cases, they could also distort surfaces or absolute distances, introducing new errors for grasp planning. Finally, our experiments focus primarily on challenging objects and on object categories that are absent or underrepresented in the training data. We did not systematically evaluate other sources of deployment variation, such as sensor noise or changes in contact dynamics.

6 Conclusion

In this work, we presented a continual-learning framework for single-view 6-DoF grasp synthesis in cluttered scenes. Our method augments a standard sample-and-score pipeline with memory: grasp outcomes update future scores, while user demonstrations add grasp modes that fixed heuristics may miss. Across simulation and real-world experiments, we show our system preserves strong offline performance while giving the robot a practical way to improve during deployment. In the real world, adaptation improves performance across every evaluated category and exceeds 90% success rate in five of the six categories tested. Furthermore, we show that our system can keep accumulating experience over hundreds of new objects without forgetting previous adaptations. These results show that grasping systems need not stop learning once offline training is complete. By storing deployment outcomes and demonstrations in structured memories, a robot can turn failures, successes, and occasional user input into immediate changes in behavior, enabling grasping policies that continue to improve during deployment.

Acknowledgments

This work was partially supported by the HILTI Group.

We acknowledge the use of artificial-intelligence-based tools as writing aids (editing and grammar enhancement) in the preparation of this manuscript. All intellectual content and conclusions remain the sole responsibility of the authors.

References

1. [1] S. Back, J. Lee, K. Kim, H. Rho, G. Lee, R. Kang, S. Lee, S. Noh, Y. Lee, T. Lee, and K. Lee (2025) GraspClutter6D: a large-scale real-world dataset for robust perception and grasping in cluttered scenes. IEEE Robotics and Automation Letters 10 (10), pp. 10498–10505. External Links: Document
2. [2] K. R. Barad, A. Orsula, A. Richard, J. Dentler, M. Olivares-Mendez, and C. Martinez (2023) GraspLDM: generative 6-dof grasp synthesis using latent diffusion models. arXiv preprint arXiv:2312.11243.
3. [3] M. Breyer, J. J. Chung, L. Ott, S. Roland, and N. Juan (2020) Volumetric grasping network: real-time 6 dof grasp detection in clutter. In Conference on Robot Learning,
4. [4] M. Danielczuk, A. Balakrishna, D. Brown, and K. Goldberg (2021) Exploratory grasping: asymptotically optimal algorithms for grasping challenging polyhedral objects. In Proceedings of the 2020 Conference on Robot Learning, J. Kober, F. Ramos, and C. Tomlin (Eds.), Proceedings of Machine Learning Research, Vol. 155, pp. 377–393. External Links: Link
5. [5] H. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu (2023) AnyGrasp: robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics (T-RO).
6. [6] H. Fang, C. Wang, M. Gou, and C. Lu (2020) GraspNet-1billion: a large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11444–11453.
7. [7] L. Fu, M. Danielczuk, A. Balakrishna, D. S. Brown, J. Ichnowski, E. Solowjow, and K. Goldberg (2022) LEGS: learning efficient grasp sets for exploratory grasping. In 2022 International Conference on Robotics and Automation (ICRA), Vol. , pp. 8259–8265. External Links: Document
8. [8] M. Gilles and V. Rau (2022) Continual learning of vacuum grasps from grasp outcome for unsupervised domain adaption. In 2022 2nd International Conference on Robotics, Automation and Artificial Intelligence (RAAI), Vol. , pp. 164–171. External Links: Document
9. [9] S. Haddadin, S. Parusel, L. Johannsmeier, S. Golz, S. Gabl, F. Walch, M. Sabaghian, C. Jähne, L. Hausperger, and S. Haddadin (2022) The franka emika robot: a reference platform for robotics research and education. IEEE Robotics & Automation Magazine 29 (2), pp. 46–64. External Links: Document
10. [10] H. Huang, D. Wang, X. Zhu, R. Walters, and R. Platt (2022) Edge grasp network: a graph-based se (3)-invariant approach to grasp detection. arXiv preprint arXiv:2211.00191.
11. [11] Y. Ju, K. Hu, G. Zhang, G. Zhang, M. Jiang, and H. Xu (2025) Robo-abc: affordance generalization beyond categories via semantic correspondence for robot manipulation. In European Conference on Computer Vision, pp. 222–239.
12. [12] R. Julian, B. Swanson, G. Sukhatme, S. Levine, C. Finn, and K. Hausman (2021) Never stop learning: the effectiveness of fine-tuning in robotic reinforcement learning. In Proceedings of the 2020 Conference on Robot Learning, J. Kober, F. Ramos, and C. Tomlin (Eds.), Proceedings of Machine Learning Research, Vol. 155, pp. 2120–2136. External Links: Link
13. [13] M. Kopicki, R. Detry, M. Adjigble, R. Stolkin, A. Leonardis, and J. L. Wyatt (2016) One-shot learning and generation of dexterous grasps for novel objects. Int. J. Rob. Res. 35 (8), pp. 959–976. External Links: ISSN 0278-3649, Link, Document
14. [14] H. Le, P. Schillinger, M. Gabriel, A. Qualmann, and N. A. Vien (2024) Pseudo labeling and contextual curriculum learning for online grasp learning in robotic bin picking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 788–794. External Links: Document
15. [15] J. Liu, J. Xie, S. Huang, C. Wang, and F. Zhou (2024) Continual learning for robotic grasping detection with knowledge transferring. IEEE Transactions on Industrial Electronics 71 (9), pp. 11019–11027. External Links: Document
16. [16] M. Liu, Z. Zhu, X. Han, P. Hu, H. Lin, X. Li, J. Chen, J. Xu, Y. Yang, Y. Lin, X. Li, Y. Yu, W. Zhang, T. Kong, and B. Kang (2025) Manipulation as in simulation: enabling accurate geometry perception in robots. arXiv preprint.
17. [17] H. Ma, M. Shi, B. Gao, and D. Huang (2024) Generalizing 6-dof grasp detection via domain prior knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18102–18111.
18. [18] J. Mahler, F. T. Pokorny, B. Hou, M. Roderick, M. Laskey, M. Aubry, K. Kohlhoff, T. Kröger, J. Kuffner, and K. Goldberg (2016) Dex-net 1.0: a cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards. In 2016 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 1957–1964. External Links: Document
19. [19] A. Mousavian, C. Eppner, and D. Fox (2019) 6-DOF GraspNet: Variational Grasp Generation for Object Manipulation . In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , Los Alamitos, CA, USA, pp. 2901–2910. External Links: ISSN , Document, Link
20. [20] A. Murali, B. Sundaralingam, Y. Chao, J. Yamada, W. Yuan, M. Carlson, F. Ramos, S. Birchfield, D. Fox, and C. Eppner (2026) GraspGen: a diffusion-based framework for 6-dof grasping with on-generator training. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), External Links: Link
21. [21] T. Patten, K. Park, and M. Vincze (2020) DGCM-net: dense geometrical correspondence matching network for incremental experience-based robotic grasping. Frontiers in Robotics and AI Volume 7 - 2020. External Links: Link, Document, ISSN 2296-9144
22. [22] S. Prokudin, C. Lassner, and J. Romero (2019) Efficient learning on point clouds with basis point sets. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4332–4341.
23. [23] L. Ren, J. Dong, D. Huang, and J. Lü (2024) Digital twin robotic system with continuous learning for grasp detection in variable scenes. IEEE Transactions on Industrial Electronics 71 (7), pp. 7650–7660. External Links: Document
24. [24] S. Rusinkiewicz and M. Levoy (2001) Efficient variants of the icp algorithm. In Proceedings Third International Conference on 3-D Digital Imaging and Modeling, Vol. , pp. 145–152. External Links: Document
25. [25] R. B. Rusu, N. Blodow, and M. Beetz (2009) Fast point feature histograms (fpfh) for 3d registration. In 2009 IEEE International Conference on Robotics and Automation, Vol. , pp. 3212–3217. External Links: Document
26. [26] R. Salakhutdinov and G. Hinton (2007) Learning a nonlinear embedding by preserving class neighbourhood structure. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, M. Meila and X. Shen (Eds.), Proceedings of Machine Learning Research, Vol. 2, San Juan, Puerto Rico, pp. 412–419. External Links: Link
27. [27] Y. Shi, P. Schillinger, M. Gabriel, A. Qualmann, Z. Feldman, H. Ziesche, and N. A. Vien (2024) Uncertainty-driven exploration strategies for online grasp learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 781–787. External Links: Document
28. [28] P. Song, P. Li, and R. Detry (2025) Implicit grasp diffusion: bridging the gap between dense prediction and sampling-based grasping. In Proceedings of The 8th Conference on Robot Learning, P. Agrawal, O. Kroemer, and W. Burgard (Eds.), Proceedings of Machine Learning Research, Vol. 270, pp. 2948–2964. External Links: Link
29. [29] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox (2021) Contact-graspnet: efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 13438–13444. External Links: Document
30. [30] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt (2017) Grasp pose detection in point clouds. The International Journal of Robotics Research 36 (13-14), pp. 1455–1473. External Links: Document, Link, https://doi.org/10.1177/0278364917735594
31. [31] R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang (2022) DexGraspNet: a large-scale robotic dexterous grasp dataset for general objects based on simulation. arXiv preprint arXiv:2210.02697.
32. [32] H. Wen, J. Yan, W. Peng, and Y. Sun (2022) TransGrasp: grasp pose estimation of a category of objects by transferring grasps from only one labeled instance. In European Conference on Computer Vision, pp. 445–461.
33. [33] Y. Yang, H. Yu, X. Lou, Y. Liu, and C. Choi (2024) Attribute-based robotic grasping with data-efficient adaptation. IEEE Transactions on Robotics 40 (), pp. 1566–1579. External Links: Document
34. [34] W. Yuan, A. Murali, A. Mousavian, and D. Fox (2023) M2T2: multi-task masked transformer for object-centric pick and place. In 7th Annual Conference on Robot Learning,
35. [35] L. Zheng, W. Ma, Y. Cai, T. Lu, and S. Wang (2023) GPDAN: grasp pose domain adaptation network for sim-to-real 6-dof object grasping. IEEE Robotics and Automation Letters 8 (8), pp. 4585–4592. External Links: Document
36. [36] R. Zurbrügg, Y. Liu, F. Engelmann, S. Kumar, M. Hutter, V. Patil, and F. Yu (2024) ICGNet: a unified approach for instance-centric grasping. In 2024 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 4140–4146. External Links: Document

Supplementary Material

This supplementary material provides additional details and analyses that support the main paper. Appendix A analyzes the relationship between the simulation training and evaluation distributions. Appendix B describes the grasp-frame convention and the geometric grasp proposal samplers. Appendix C discusses failure cases of the contact-normal-based heuristic on thin objects. Appendix D gives implementation details for the memory-based scorer, including local patch encoding, encoder architecture, and scoring hyperparameters. Appendix E details how evidence from different sources is weighted. Appendix F describes how demonstrations are recalled as additional grasp proposals. Appendix G shows the simple Open3D interface used to provide demonstrations. Appendix H compares the SNN-trained embedding against a BCE-trained embedding. Appendix I reports the full forward-adaptation results in simulation. Appendix J reports runtime and memory costs before and after adaptation. Appendix K contains an analysis of the sources of executed grasps. Finally, Appendix L compares real-world adaptation with and without demonstrations.

Appendix A Training–Evaluation Distribution Analysis

We train in simulation using the object set shared by VGN, EdgeGraspNet, and ICGNet. For evaluation, we use 443 object instances from DexGraspNet that are not present in the training set. We attempted to choose categories that were underrepresented in training while still resembling objects a robot might encounter. Bowls, hammers, screwdrivers, and pliers occur in the training set only rarely, jointly accounting for less than 5% of its objects, while airplanes and fasteners are absent. The remaining evaluation categories are present in the training set.

To assess how far the evaluation distribution lies from the training distribution, we compare their grasp embeddings. We encode grasp proposals from the evaluation scenes with our learned encoder, compute the distance of each embedding to its fifth-nearest neighbor in the offline training memory, and report the median within each category. As an in-distribution reference, we apply the same procedure to held-out IID grasps on training objects.

Most evaluation categories have a larger median distance than the IID reference, with airplanes, animals, bowls, hammers, and mugs showing the clearest shift. Bottles lie close to the IID reference, while fasteners and pliers have smaller distances. This indicates that several evaluation categories induce grasp geometries outside the training distribution, although the shift is not uniform across categories.

Appendix B Grasp Parametrization and Grasp Proposal Sampling

We represent each grasp as a pose (R,c)\in\mathrm{SE}(3), where c\in\mathbb{R}^{3} is the gripper center and R=[x\;y\;z]\in\mathrm{SO}(3) is the grasp frame. The axis y is the finger closing direction, z is the approach direction, and x=y\times z completes the right-handed frame. We generate grasp proposals with two geometric heuristics.

The first heuristic is inspired by EdgeGraspNet, ContactGraspNet, and ICGNet [10, 29, 36]. We sample an approach point p_{a} and a nearby contact point p_{c}, estimate the surface normal n_{c} at p_{c}, set the closing direction to y=n_{c}, and define the approach direction as the normalized vector z\propto n_{c}\times(n_{c}\times(p_{a}-p_{c})). The center is then placed as c=p_{a}-\delta z, where \delta accounts for the gripper depth and the offset between p_{a} and p_{c} along z. This heuristic produces diverse side grasps, but depends on estimating n_{c} reliably. Figure 5 illustrates typical failure cases when thin side surfaces are not observed accurately.

We therefore also use a top-down sampler that does not rely on contact normals. We sample two nearby points p_{i} and p_{j} from the upper layer of the pointcloud, set the center to their midpoint c=(p_{i}+p_{j})/2, define the closing direction as the normalized horizontal vector y\propto(p_{j}-p_{i})_{xy}, and fix the approach direction to z=(0,0,-1). We then set x=y\times z and shift the center slightly along z to vary insertion depth. This sampler is less expressive, but does not require reliable side geometry or local surface normals.

Appendix C Failure cases of heuristic grasp proposal

As detailed in the previous section, the contact-based proposal sampler parameterizes each candidate around an observed contact point and its estimated local surface normal. This is effective when the depth observation contains clean side surfaces, but it can fail on thin tabletop objects. These objects have only a small visible side area from the camera viewpoint, so the side geometry is often either smoothed into a sloped surface or missing from the pointcloud entirely.

Figure 5 summarizes the resulting failure modes. If the reconstructed side is smoothed into a slope, the estimated normal points diagonally. A grasp constructed normal to this slope can place the opposing finger below the object, causing a collision with the table and making the proposal invalid. If the side surface is missing, there are no side contact points from which the sampler can generate a grasp.

These examples arise from naturally occurring artifacts in the real depth observations. We did not inject controlled levels of sensor noise, so they illustrate observed perception failure modes rather than a quantitative noise-robustness evaluation.

Appendix D Implementation details: memory-based scorer

This section gives the implementation details for the memory-based scorer. We first describe how local grasp patches are extracted and encoded, then give the encoder architecture and scoring hyperparameters. Finally, we describe how user demonstrations are weighted when they are added to the scoring memory.

Patch extraction and basis point set encoding

We crop a spherical patch with diameter 11 \mathrm{c}\mathrm{m} centered at the grasp pose and transform it in the grasp reference frame. Normals are estimated from the depth observation and transformed together with the points. We encode the aligned local patch with a spherical Basis Point Set representation. We use 512 fixed basis points sampled inside a sphere with the same dimension as the extracted patch. For each basis point, we find the nearest observed scene point and store the displacement vector to that point and the normal of that point. Finally, we serialize the representation to obtain a 6 \times 512 descriptor, which is then used as input to the encoder MLP. Figure 6 shows the process.

Encoder and decoder.

The serialized BPS features are passed to an MLP encoder with hidden dimensions 512, 256, and 128, which outputs a 32-dimensional embedding. The final embedding is then L2-normalized. A mirror MLP decoder reconstructs the BPS geometry from the embedding. At inference time, only the normalized encoder output is used by the continual-learning module.

Scorer hyperparameters.

We retrieve K=100 nearest neighbors from memory for each query grasp. The SNN training temperature is \tau_{\mathrm{train}}=0.2. At inference time, the RBF kernel temperature \tau_{\mathrm{eval}} is tuned to maximize average precision on validation data, which gives \tau_{\mathrm{eval}}=0.3. We use a Beta prior with \alpha_{0}=0.1 and \beta_{0}=0.9, and cap the total offline evidence at W_{\max}^{\mathrm{off}}=10.

Appendix E Tuning of Evidence Weights

We normalize the base evidence weights relative to offline data, setting \rho_{\mathrm{off}}=1. Online grasp outcomes use \rho_{\mathrm{on}}=3, so a nearby online observation contributes three times as much evidence as a nearby offline observation before distance-based attenuation. This ratio controls how quickly consistent deployment experience can revise the offline estimate.

We cap the aggregate offline contribution to each query at W_{\max}^{\mathrm{off}}=10. Consequently, three mutually consistent online observations that are close to the query contribute up to nine pseudo-counts, which is comparable to the maximum evidence contributed by the entire local offline neighborhood. They can therefore substantially shift the posterior when they contradict the offline data, while a single online observation is not sufficient to dominate a strong offline estimate.

Demonstrations are added as positive evidence with an adaptive base weight. To compute this weight, we first score the demonstrated grasp, obtaining posterior parameters \alpha_{d},\beta_{d}, and the current best planner candidate, obtaining \alpha_{q}^{*},\beta_{q}^{*}. We choose the demonstration weight so that adding it would raise the demonstrated grasp’s odds above the current best candidate’s odds by a margin m_{\mathrm{demo}}:

\tilde{\rho}_{d}=\left(\frac{\alpha_{q}^{*}}{\beta_{q}^{*}}+m_{\mathrm{demo}}\right)\beta_{d}-\alpha_{d},\qquad\rho_{d}=\max(m_{\mathrm{demo}},\tilde{\rho}_{d}).

We use m_{\mathrm{demo}}=1.0 in all experiments. Intuitively, this means that if the same observation and candidate set were encountered again, the planner would select the demonstrated grasp, provided that it remains feasible.

Appendix F Implementation details: demonstration recall

As described in the main text, demonstration recall adds grasp proposals that may be missing from the geometric sampler. When we receive a demonstration, we store:

• a local pointcloud patch around the demonstrated grasp, using a radius of 4\,\mathrm{cm}. We compute FPFH descriptors and store them alongside the patch.
• the gripper pose (relative to the stored patch)
• the grasp embedding, obtained using the same encoder we use for the memory-based scorer

During inference, we use the FPFH features to match each stored patch against local neighborhoods in the current pointcloud. We sample up to 100 candidate neighborhood centers per demonstration, and rank them by the cosine similarity between their mean FPFH descriptor and the stored patch descriptor. Candidates with similarity below 0.7 are discarded.

For each remaining candidate, we align the stored patch to the current neighborhood with point-to-plane ICP, initialized at the neighborhood centroid. The resulting transform maps the demonstrated grasp pose into the current scene, producing a recalled grasp proposal. We then encode each recalled grasp and keep only recalls whose feature is close to the original demonstration feature. Specifically, for each demonstration we keep recalls with

\lVert f_{\mathrm{rec}}-f_{\mathrm{demo}}\rVert_{2}\leq d_{\min}+0.1,

where d_{\min} is the smallest feature distance among recalls from that demonstration in the current scene.

This recall mechanism is deliberately simple, but we find that it works well in practice. We use it only to add proposals to the geometric sampler, not to choose the final grasp directly. Recalled grasps therefore still pass through the same collision filtering and memory-based scoring stages as all other proposals, which removes many poor matches. The use of a small local patch also helps recall generalize across similar objects, as reflected in the category-adaptation experiments. Nevertheless, recall is the part of our pipeline that would most benefit from future work. Local geometry alone is sometimes ambiguous, so more complex recall mechanisms could use semantics, instance recognition, or class-level templates in the future.

Appendix G Demonstration Interface

We use a simple demonstration interface implemented in Open3D. The user is shown the robot’s current observation together with a model of the parallel-jaw gripper, as shown in Figure 7. The user moves the gripper pose with keyboard controls and presses enter to confirm the demonstration.

The resulting gripper pose is then added to the scoring memory and recall memory as described above. Because the interface operates on the stored robot observation rather than requiring direct robot teleoperation, demonstrations can also be provided a posteriori. In principle, this means that a user could annotate failed or uncertain grasps after the robot has already collected the corresponding observations, without needing physical access to the robot at demonstration time.

Appendix H Comparison between SNN embedding and BCE embedding

In our method, we train the low-dimensional grasp embedding with a soft nearest-neighbor loss and a reconstruction loss, following prior work on embeddings for non-parametric classifiers [26]. As an ablation, we also train an encoder with a standard binary-cross-entropy objective and use its normalized penultimate features as the embedding space. The scoring and online update rules are otherwise unchanged. This comparison tests whether the continual-learning behavior comes mainly from the non-parametric update rule, or whether the metric structure induced by the SNN objective is important.

Table 6 shows that the BCE embedding gives similar base performance to the SNN embedding, suggesting that both encoders learn features that are useful for offline grasp scoring. The difference becomes clearer during sequential adaptation. The BCE embedding still supports some continual improvement, but it exhibits more interference between related categories. For example, after adapting to mugs, performance on bowls drops strongly. The last few categories also show weaker improvement after adaptation, suggesting that the embedding space becomes less useful as more online evidence is added.

To investigate this difference, we plot the cumulative explained variance of the PCA components of each embedding space in Figure 8 below.

The BCE objective concentrates variance along one dominant direction. This likely causes two effects. First, geometric details that are not directly useful for offline classification are discarded, which can later lead to interference between categories. Second, datapoints become concentrated in a small region of the embedding space, causing a saturation effect where new online evidence has limited influence. This could explain why the final adaptation categories show only limited improvement. By contrast, the SNN+reconstruction embedding spreads variance across more dimensions, preserving richer geometric structure and separating multiple success and failure modes. This can keep updates local to genuinely similar grasps and improve continual learning.

Nevertheless, we note that the pipeline still works reasonably reliably even with the BCE embedding. This is likely because we are not learning a sequence of strongly conflicting tasks, but refining graspability estimates in regions that were underrepresented during offline training. Therefore, even if the embedding is less well-structured, the online evidence can still improve the decision boundary without having to overwrite previously learned knowledge.

Appendix I Forward Adaptation in Simulation

Table 7 reports the full category-by-category performance matrix after adapting to each simulation category in the sequential continual-learning simulation experiment. Unlike the table in the main paper, this includes performance both on categories the model has seen already, and those it has not currently seen.

In general, we observe that adding new categories to the training data does not decrease performance on other unseen categories, indicating that the improvement on some categories does not come at the cost of lower performance on the others. Instead, there are some signs of forward adaptation: for example, adding the Animal category causes performance to increase for the Airplane and Bowl categories. Average performance on all categories climbs as more categories are added.

Appendix J Runtime and Memory Cost of Continual Learning

Because continual learning adds data throughout deployment, it is important to verify that adaptation does not make inference progressively slower or require impractical storage. We therefore measure the memory footprint and per-stage runtime before and after the sequential adaptation experiment. Timings are reported per planning cycle, from proposal generation through scoring. Runtime was measured on a machine equipped with an NVIDIA RTX 4090 GPU with 24 GB VRAM, a 12th Gen Intel Core i9-12900K CPU, and 32 GB RAM.

As shown in Table 8, the memory footprint remains small after adaptation. Adding new points to the memory-based scorer has little effect on either memory or runtime, since each new memory entry is only a low-dimensional embedding vector and a label. Most of the added cost comes from demonstration recall. We keep this bounded with a fixed 1000 ms recall budget, divided uniformly across demonstrations. For each demonstration, we stop once the time budget is exhausted or 30 grasp proposals are produced.

The fixed recall budget is a pragmatic choice. By default, we attempt to register each stored demonstration into the current scene, so without a budget the recall time would grow roughly linearly with the number of demonstrations. That is, uncapped recall has complexity O(N_{\mathrm{demo}}) in the number of stored demonstrations. With the budget, runtime stays bounded, but each demonstration receives less registration time as the recall memory grows. In principle this could reduce recall quality after very long deployments. We do not observe such a drop in our experiments, even after adapting to hundreds of new objects. Adaptation required fewer than 50 demonstrations in total, so recall was not a bottleneck at the scale evaluated here. It may become important at larger scales. A natural extension would be to filter demonstrations before registration, for example using semantic cues, so that recall spends its budget only on demonstrations likely to be relevant to the current scene. If object-class information were available, matching only demonstrations for the relevant class would reduce the per-scene work to O(N_{c}), where N_{c} is the number of demonstrations for that class.

Appendix K Source of Executed Grasps

For the adapted real-world model, we report the fraction of executed grasps originating from demonstration recall and from geometric sampling.

The source distribution varies substantially across object sets. Mugs and bowls require no recalled grasps, whereas recall accounts for most executed grasps on the control objects, kitchen tools, and screwdrivers. Pliers and toys rely more heavily on geometric sampling. This variation supports using demonstration recall to supplement geometric sampling rather than replace it.

Appendix L Real-World Adaptation With and Without Demonstrations

While in simulation we carry out ablations that isolate the contributions of grasp outcomes and demonstrations, in our real-world experiments we use our full method: grasp outcomes are added to the scoring memory, and user demonstrations are added to the scoring and recall memories. Here we add a smaller real-world comparison to illustrate how demonstrations affect adaptation on individual difficult objects. We select three objects that were difficult to grasp for our base policy in the real-world trials: a toilet-cleaner bottle, a shampoo bottle, and a rubber duck (Figure 9). For each object, the robot starts from the base model and receives 20 adaptation attempts. We compare two variants: outcome-only adaptation, which updates the scoring memory from the binary grasp outcomes, and full adaptation, which is additionally aided by user demonstrations. For the full method, we initialize the demonstration memory with two demonstrations and then provide a new demonstration after each failed grasp.

Figure 10 shows the resulting traces. The outcome-only variant needs more failed attempts before stabilizing on the toilet cleaner and shampoo bottle. Demonstrations reduce the number of failures from 7 to 3 on the toilet cleaner and from 5 to 2 on the shampoo bottle. On the rubber duck, demonstrations reduce failures from 6 to 4, although they do not remove all late failures. In both variants, failures generally become sparser over the 20 attempts, reflecting adaptation from accumulated experience. These traces are not intended as a comprehensive real-world benchmark, but they are consistent with the mechanism observed elsewhere in the paper: outcome feedback can improve scoring over repeated trials, while demonstrations can accelerate adaptation when a useful grasp mode is missing or underweighted. In this sense, demonstrations can trade a small amount of user input for fewer robot failures, which may be important in settings where failed grasps are costly.