흡착 그리퍼와 집게 그리퍼를 단 두 팔이 물체를 주고받게 하자 단일 팔로 불가능한 작업 72.5% 성공: 인도공대 등
흡착 그리퍼 팔과 평행 집게 그리퍼 팔을 짝지어 매 순간 어느 팔이 물체를 잡을지 계획했더니 팔 하나로는 전혀 못 하던 시뮬레이션 운반 작업 네 가지의 72.5%를 해냈다고 인도공대 만디와 인도과학원, 브라운대 연구진이 프리프린트에서 밝혔다. 탐욕적 계획기는 20%였다. 빔 탐색 버전은 계획 시간을 82.9% 줄이고도 65%를 성공했다.
산업용 로봇은 대부분 흡착컵이나 평행 집게처럼 싸고 튼튼한 그리퍼를 쓰지만 각각 잘하는 잡기와 못하는 잡기가 뚜렷하다. 서로 다른 단순 그리퍼를 단 두 팔이 필요할 때 물체를 주고받으며, 사람이 한 번 보여준 물체 이동 경로만으로 어느 한쪽도 혼자 못 하는 일을 해낼 수 있는지가 질문이다.
흡착컵은 평평한 접시를 들 수 있지만 좁은 틈에는 못 들어가고, 두 손가락 집게는 머그컵은 단단히 잡지만 바닥에 누운 접시는 집지 못한다. 실제 운반 작업은 요구가 다른 여러 단계를 거치므로 그리퍼 하나는 어딘가에서 실패한다. 언제 어떤 잡기로 넘겨줄지는 조합 폭발이 일어나는 계획 문제다.
서로 다른 그리퍼를 쓰는 로봇 연구는 주로 잡기 자세 생성과 집기에 머물렀다. 다른 길은 사람 손을 닮은 다관절 손이지만 산업용 그리퍼보다 비싸고 약하다.
계획기는 시연된 물체 경로를 구간으로 나누고 구간마다 흡착 팔, 집게 팔, 두 팔 간 전달 중 무엇이 물체를 옮길지 정한다. 후보 잡기 자세를 도달성, 동작 실현 가능성, 조작성, 두 그리퍼 간 호환성으로 평가하고, 행렬식 점과정(DPP) 샘플링으로 다양한 잡기를 고른 뒤 팔 교체에 벌점을 주는 동적계획법으로 전체적으로 일관된 배정을 찾는다. 빔 폭 10, 보폭 3의 빔 탐색 버전은 최적성을 일부 포기하고 속도를 얻는다.
프랑카 FR3 두 대를 모사한 시뮬레이션의 네 작업(접시를 탁자에서 선반으로, YCB 접시를 선반에서 탁자로, 머그컵 뒤집기, 접시를 상자에서 상자로)에서 로봇 한 대의 성공률은 0%였다. 정확한 계획기는 72.5%를 완수했고 전달 성공률 100%, 평균 팔 교체 1.3회였으며 탐욕적 계획기는 20.0%였다. 빔 버전은 계획 시간을 532.96초에서 91.05초로 줄이면서 65.0%를 기록했고, 작업별로는 탁자에서 선반 접시 50%부터 머그컵 뒤집기 80%까지였다. 교체 벌점을 빼자 평균 교체가 1.3회에서 10.7회로 늘고 성공률은 25.0%로 떨어졌으며 실행 시간은 211.4% 늘었다. 조작성 항을 빼면 52.5%, 잡기 다양성을 빼면 계획 시간이 1015.87초로 약 두 배가 됐다. 연구진은 두 작업을 ROS 2와 MuJoCo 기반 디지털 트윈을 거쳐 실제 FR3 두 대로 실행해 계획된 전달을 재현했다고 밝혔지만 실기 성공률 수치는 내지 않았다.
모든 성공률은 시뮬레이션 결과이고 실기는 수치 없는 시연이다. 물체별 CAD 메시와 사람이 준 고정 경로가 필요해 새 물체에는 바로 쓸 수 없다고 저자들도 인정했다. 작업당 계획에 약 1.5~9분이 걸려 대부분의 생산 라인에는 느리다. 정확한 계획기로도 세 번에 한 번은 실패한다. 동료 심사를 거치지 않은 프리프린트다.
값싼 그리퍼 두 개로 비싼 다관절 손을 대신한다는 발상은 흡착과 집게 그리퍼가 이미 표준인 물류센터, 포장, 이커머스 출고 셀에 맞는다. 실험에 쓴 FR3를 만드는 프랑카 로보틱스와 양팔 작업대를 꾸리는 시스템통합 업체에 관련이 있다. 물류 자동화 투자가 이어지는 국내 풀필먼트 현장도 다관절 손 도입 전에 검토할 수 있는 저비용 대안이다. 실제 배치에는 계획 시간을 분에서 초 단위로 줄이고 CAD 모델 없는 물체에도 통해야 해 2~4년이 걸릴 것으로 보인다.
논문 전문
Embodiment-Aware Planning for Heterogeneous Bimanual Manipulation
CC BY 4.0 라이선스로 공개된 논문입니다. 출처를 밝혀 전재하며, 원문은 arXiv:2610.23551(PDF)에서 볼 수 있습니다.
Abstract
Many manipulation tasks that are effortless for dexterous human hands remain challenging for robotic systems equipped with simple, task-specialized grippers. Industrial platforms commonly rely on end-effectors such as suction cups or parallel-jaw grippers, yet no single mode can reliably satisfy the diverse constraints encountered across different phases of complex manipulation tasks, including pickup, reorientation, insertion, and constrained placement. In this work, we investigate whether complementary simple grippers within a heterogeneous bimanual setup can collectively realize tasks that exceed the capability of any individual gripper alone. Given a single human demonstration specifying the desired object motion, we formulate embodiment-aware execution as a finite-horizon optimization over arm–trajectory assignments. At each segment, the planner determines whether single-arm manipulation or a coordinated dual-arm handover is required, jointly reasoning over heterogeneous grasp feasibility, motion constraints, manipulability, and cross-gripper compatibility. Exploiting this formulation, we compute globally consistent assignments with explicit switching regularization. Experiments on a dual Franka FR3 platform equipped with suction and parallel-jaw grippers demonstrate improved feasibility and reliability in tasks that are infeasible under single-gripper execution. WEBSITE
I Introduction
Humans perform complex manipulation tasks using highly dexterous hands capable of continuously varying contact modes, regrasping strategies, and in-hand adjustments. In contrast, most robotic systems rely on simple, task-specialized end-effectors such as suction cups or parallel-jaw grippers. While these grippers are robust, affordable, and widely deployed in industrial environments, each offers limited affordances. A suction gripper excels at lifting flat surfaces but struggles in confined insertions; a parallel-jaw gripper provides stable force-closure grasps yet often fails on thin or flush objects. Consequently, many tasks that are effortless for human hands cannot be reliably executed using a single simple robotic gripper.
In modern industrial settings such as warehouse automation, packaging lines, and retail fulfillment, objects vary widely in geometry, thickness, surface properties, and accessibility. During a single transport task, different phases may demand different grasp modalities. Rather than relying on complex anthropomorphic hands, a practical alternative is to combine complementary simple grippers within a heterogeneous bimanual system. Such a configuration has the potential to approximate the functional diversity of human manipulation while retaining the robustness and simplicity of industrial hardware. However, enabling two specialized grippers to collectively realize tasks that exceed the capability of either alone requires systematic coordination and planning.
Figure illustrates this principle. A thin plate must be transferred from a narrow shelf to a table. The suction gripper cannot safely execute the pickup due to geometric interference during approach, whereas the parallel jaw gripper successfully performs the initial pick and transport. However, it cannot complete the placement due to collision constraints with table. The task becomes feasible only when the system performs a coordinated dual-arm handover, allowing the suction gripper to execute the final placement. Importantly, neither gripper alone can solve the task, yet their coordinated use enables seamless execution. This example captures the central premise of our work: heterogeneous coordination can elevate simple industrial grippers’ performance, provided that switching and compatibility are planned systematically.
In this work, we study embodiment-aware realization of demonstrated manipulation tasks using a heterogeneous dual-arm robotic system. A single human demonstration provides the desired object motion as a task-level specification. Rather than learning a policy, our objective is to determine how the demonstrated task can be executed under the constraints of heterogeneous robotic hardware. Given a discretized object trajectory, we determine at each segment which arm should manipulate the object and when coordinated handovers are necessary to maintain feasibility under grasp, kinematic, and collision constraints.
Heterogeneous execution introduces unique planning challenges. At different phases of a task, only specific gripper modalities may satisfy feasibility requirements. Coordinated handovers must respect cross-gripper grasp compatibility, and excessive switching increases execution time and failure risk creating the need for a global reasoning. We formulate this problem as a finite-horizon optimization over arm–trajectory assignments. Each state encodes both the active robot and its associated grasp-induced motion. Segment costs capture feasibility and manipulability quality, while transition costs encode heterogeneous grasp-pair compatibility and switching penalties. Exploiting the optimal substructure of this formulation, we compute a globally consistent execution plan via dynamic programming, equivalent to a structured shortest-path computation over a time-expanded graph.
Our contributions are threefold:
- Heterogeneous gripper-level coordination: We formalize how a complementary simple gripper system can collectively realize manipulation tasks that are infeasible for any single modality alone, extending beyond simple pick-only tasks.
- Globally consistent embodiment-aware assignment: We introduce a structured planning formulation that integrates heterogeneous grasp affordances and handover constraints into a globally consistent assignment.
- Sim-to-real transfer: We demonstrate the framework on a dual Franka FR3 platform equipped with suction and parallel-jaw grippers on a task requiring modality transitions and coordinated handovers.
II Related Work
II-A Dexterous Manipulation vs. Simple Grippers
Robotic dexterous hands aim to replicate the versatility of human manipulation through multi-fingered contact, in-hand regrasping, and rich contact reasoning [1, 2]. While highly expressive, they introduce substantial mechanical and control complexity. Classical multi-fingered manipulation established many of the analytical foundations for contact kinematics and grasp planning [3, 4, 5]. Industrial systems instead rely on simple, task-specific end-effectors such as suction cups or parallel-jaw grippers for their robustness and cost efficiency. However, each supports only limited contact modalities. This leaves the challenge of expanding manipulation capability without complex anthropomorphic hardware largely open. Recent reinforcement learning has enabled impressive dexterous manipulation on anthropomorphic hands, including large-scale sim-to-real policies and reorientation tasks [6, 7, 8]. We instead explore extending capability by coordinating complementary simple grippers in a heterogeneous bimanual system. Rather than increasing single-hand dexterity, we study whether structured coordination between specialized modalities can approximate the functional diversity.
II-B Bimanual Manipulation and Grasp Planning
Bimanual manipulation has been widely studied for coordinated transport, reorientation, and in-hand manipulation [9]. Early work largely focused on symmetric dual-arm systems with identical end-effectors, emphasizing synchronized motion planning and closed-chain control [10, 11]. Several works also address grasp generation and object handover between two arms. GraspGen [12] generates grasps for parallel-jaw grippers, while sampling-based and graph-based methods search over candidate grasp pairs and intermediate transfer poses [13] [14].
More recently, learning-based approaches have demonstrated coordinated dual-arm behaviors, including end-to-end Vision–Language–Action models [15], learned bimanual coordination [16], and reactive self-handover in confined environments [17]. Manipulability-aware dual-arm grasp planning has also been explored in redundant systems [18], along with physics-aware saliency learning for bimanual grasp point selection [19]. However, most existing methods assume identical end-effectors or treat grasp feasibility separately for each arm. Coordination with heterogeneous grippers and explicit reasoning over grasp compatibility and switching across task phases still remains a challenge. Our approach instead performs global planning that jointly models grasp compatibility, motion feasibility, and modality transitions.
II-C Heterogeneous Gripper Systems
There is limited prior work on planning with heterogeneous grippers to extend the manipulation capability of bimanual robots. HeterBot [20] introduces a mobile platform with two different grippers but focuses mainly on teleoperation rather than autonomous coordination. VIKI-R [21] assigns task primitives across heterogeneous platforms on high-level, but does not explicitly reason about cross-gripper low-level manipulation. [22],[23] and [24] employs heterogeneous grippers; however, its applicability is restricted to object picking tasks. Overall, existing work focuses on gripper selection for individual pick actions, while holistic task planning and feasibility across heterogeneous embodiments remain understudied. Positioning: Most existing methods assume identical end-effectors, treat grasp selection separately from task planning, or rely on heuristic or manual switching between arms. In contrast, our method jointly reasons about grasp feasibility, grasp-pair compatibility, and arm switching in a single global planning framework.
III Problem Formulation
We consider a heterogeneous bimanual manipulation system with two robot arms \mathcal{R}:=\{R_{L},R_{R}\} operating in a shared workspace. Each arm R_{j} is equipped with a possibly different end-effector G_{j}\in\{\text{parallel-jaw},\text{suction},\dots\}, leading to asymmetric grasp affordances.
Task Setup: An object o must be moved from a start pose X_{start}\in SE(3) to a goal pose X_{goal}\in SE(3). A single human demonstration provides an object-centric trajectory \xi:[0,T]\rightarrow SE(3) describing the desired object motion. The trajectory is discretized into N ordered segments \{\xi_{1},\dots,\xi_{N}\}. Due to heterogeneous gripper affordances and kinematic limits, different segments may only be executable by specific arms.
For each segment \xi_{i}, we construct a finite set of feasible arm–trajectory pairs s_{i}=(r_{i},\tau_{i}), where r_{i}\in\mathcal{R} is the active robot and \tau_{i} is a grasp-conditioned, collision-free trajectory that executes \xi_{i}. Let \mathcal{S} denote the set of all feasible states. A plan is a sequence \mathbf{s}_{1:N}=(s_{1},\dots,s_{N}) such that exactly one arm manipulates the object at each segment. The objective is to select a sequence that remains feasible while minimizing motion and switching effort over the full task.
IV Embodiment-Aware Planning Framework
Given the feasible state set \mathcal{S} for each segment, task execution is formulated as a finite-horizon discrete optimization problem. Candidate grasps on the object generate multiple trajectory instances for each arm. These candidates are evaluated for grasp validity, kinematic reachability, and collision-free motion. Each feasible state is assigned a segment cost, while transitions between states encode switching penalties and grasp compatibility. A dynamic programming procedure then computes the minimum-cost sequence of states over the task horizon, producing a feasible and smooth bimanual execution plan such as \{(r_{1},\tau_{1}),(r_{1},\tau_{1}),(r_{2},\tau_{8}),(r_{2},\tau_{8}),\dots\}.
The individual components of the framework are described in the following subsections.
IV-A Trajectory Extraction from Human Demonstration
We extract the object trajectory via model-based 6D pose tracking using FoundationPose [25], leveraging the known CAD mesh to enforce geometric consistency during manipulation (See Figure 2). While the mesh is required for planning, recent work enables reliable mesh reconstruction of novel objects from multiple RGB-D images [26]. The pipeline initializes an SE(3) pose from RGB-D observations and then tracks the object by conditioning each estimate on the previous pose to maintain temporal coherence. At each timestep, pose alignment is optimized between observed depth and the rendered mesh. The resulting poses are temporally filtered to suppress high-frequency jitter while preserving rigid-body consistency, yielding smooth trajectories robust to partial occlusion. From these trajectory points, we define motion segments \xi_{i}=(p_{i},p_{i+1}) between consecutive poses.
While Foundation Pose and trajectory extraction from Human Demonstration serve as a general approach, our method works with any method that can provide an object trajectory; thus, we also tested our approach with the RRT* algorithm for object trajectory generation.
IV-B Grasp Generation, Ranking and End-Effector Trajectory Formation
A feasible plan requires at least one robot to have a valid grasp pose g for gripper G_{j} at object pose p_{i}. Let \mathcal{G}_{j}=\texttt{SampleGrasps}(o,G_{j}) denote the candidate grasp set; feasibility requires \mathcal{G}_{j}\neq\emptyset. This captures gripper-dependent affordances (e.g., suction on flat surfaces and parallel-jaw grasping on accessible edges), modeled over the coupled state of both grippers and the object rather than any single gripper in isolation. We use a geometry-based grasp generator [12]. Scene-conditioned generation is avoided because a grasp infeasible in one segment may become feasible later due to changes in arm assignment or handover configuration, complicating consistent reasoning across segments.
To encourage grasp diversity, we bypass the generator’s internal ranking and instead perform entropy-based subset selection. Given candidate grasps \{g_{1},\dots,g_{M}\} with positive quality scores w_{i}, we construct a Determinantal Point Process (DPP) using an L-ensemble. The L-ensemble matrix is defined as
L_{ij}=w_{i}w_{j}K_{ij},\qquad K_{ij}=\exp\!\left(-\tfrac{1}{2}d(g_{i},g_{j})^{2}\right).The grasp distance metric is given by
d(g_{i},g_{j})^{2}=\frac{\|p_{g,i}-p_{g,j}\|^{2}}{\sigma_{p}^{2}}+\frac{\bigl(1-|\langle o_{g,i},o_{g,j}\rangle|\bigr)^{2}}{\sigma_{r}^{2}}.where p_{g,i} and o_{g,i} denote the position and orientation (quaternion) of grasp g_{i}, respectively. We select a subset S of size k by greedily maximizing \log\det(L_{S}), which jointly promotes grasp quality (via w_{i}) and spatial–rotational diversity (via K_{ij}). Each selected grasp is used to generate the corresponding end-effector trajectory. These trajectories share the same time parameterization as the original object trajectory.
IV-C Segment-wise Feasibility and Cost Formulation
For each transport segment \xi_{i}, we determine which arm R_{j}\in\mathcal{R}=\{R_{L},R_{R}\} can execute the motion. An action is defined as A_{i,j}=(o,p_{i},p_{i+1},R_{j}), representing transport of object o from p_{i} to p_{i+1} using arm R_{j}. Its feasibility F(A_{i,j})\in\{\text{True},\text{False}\} depends on grasp and motion feasibility.
IV-C1 Grasp Feasibility
A valid grasp must exist for gripper G_{j} at pose p_{i}. Let \mathcal{G}_{i,j}=\texttt{SampleGrasps}(o,p_{i},G_{j}) denote the candidate grasp set; feasibility requires \mathcal{G}_{i,j}\neq\emptyset (already enforced by the filtering step in Section IV-B). We additionally compute mutually feasible grasp pairs across robots, i.e., collision-free grasp pairs that can simultaneously hold the object, which are required for handovers.
IV-C2 Motion Planning Feasibility
A collision-free trajectory must exist for arm R_{j} to transport the object from p_{i} to p_{i+1}. Feasible motions must also exist for pick at p_{0}, release at p_{T}, and handover execution. If any motion primitive is infeasible, the action is rejected.
An analytical Inverse Kinematics (EAIK) [27] solver evaluates exact endpoint reachability in microseconds, acting as a rapid feasibility filter. If analytical reachability is confirmed, we then evaluate the required motion primitives.
IV-C3 Segment Cost
In order to avoid awkward or near-singular configurations, we define a task cost based on the Yoshikawa manipulability measure and smoothness of the transitions between points. For a configuration q_{i,j} corresponding to action A_{i,j}, we define two cost components.
Smoothness Cost.
The smoothness term penalizes large joint variations between consecutive configurations:
C_{1}(A_{i,j})=\ \left\lVert q_{i+1,j}-q_{i,j}\right\rVert^{2}.Manipulability Cost.
Given the Jacobian J(q), the Yoshikawa manipulability index [28] for a joint configuration q is
w(q)=\sqrt{\det\!\left(J(q)J(q)^{\top}\right)}.To penalize near-singular configurations along the transport segment, we evaluate the index at both the start (q_{i,j}) and end (q_{i+1,j}) configurations of the action. We define the cost as:
C_{2}(A_{i,j})=\frac{1}{w(q_{i,j})+\epsilon}+\frac{1}{w(q_{i+1,j})+\epsilon},where \epsilon>0 is a small constant for numerical stability.
Combined Task Cost.
Let F(A_{i,j})\in\{\mathrm{True},\mathrm{False}\} denote the feasibility of the action. The overall task cost is
C(A_{i,j})=\begin{cases}\alpha\,C_{1}(A_{i,j})+\beta\,C_{2}(A_{i,j}),&\text{if }F(A_{i,j})=\mathrm{True},\\[8.0pt]
\infty,&\text{otherwise},\end{cases}where \alpha,\beta>0 balance smoothness and manipulability. Through a grid search for \alpha\in\{2.5,\allowbreak 5.0,\allowbreak 10.0,\allowbreak 20.0,\allowbreak 40.0,\allowbreak 50.0\} and \beta\in\{0.0,\allowbreak 0.25,\allowbreak 0.5,\allowbreak 1.0,\allowbreak 2.0,\allowbreak 4.0\}, these weights were set to \alpha=40 and \beta=0.5 to yield the best performance. For kinematically redundant manipulators, we include the manipulability term directly in the inverse kinematics (IK) optimization too. By doing so, the solver naturally prefers well-conditioned configurations, resulting in smooth and highly manipulable robot motions.
IV-D Transition Design for Heterogeneous Handover
We define the handover primitive at a point p_{i} as a dual-arm exchange between a transfer and a receiver arm using predefined grasp transforms. Each end-effector pose is computed by combining the object pose with the inverse of its grasp frame. The primitive consists of three stages: Pre-alignment, where the object is slightly offset while the receiver aligns to this intermediate pose; Receiver approach, where the receiver interpolates smoothly to the final grasp along the grasp normal; and Transfer retreat, where the original arm retreats along a fixed axis after handoff. This sequence ensures smooth, coordinated, and geometrically consistent dual-arm handovers.
Transitions between identical robot states are disallowed by assigning infinite cost to such edges. Although counterintuitive, this is necessary for feasibility: switching planners mid-trajectory requires an instantaneous release and regrasp, which may cause the object to fall. If a grasp change is required, it can be achieved through two handovers, but handover costs discourage unnecessary frequent switching unless required for feasibility.
Following Section IV-C, we also enforce paired heterogeneous grasp feasibility by assigning infinite cost to infeasible simultaneous grasp configurations, thereby eliminating transitions corresponding to infeasible dual-arm trajectories. For feasible handover primitives, the cost is defined analogously to the segment cost in Section IV-C. Specifically, the total cost of a handover primitive \mathcal{H} is given by
C_{\text{switch}}(\mathcal{H})=\sum_{(i,j)\in\mathcal{H}}C(A_{i,j}),where C(A_{i,j}) is the combined smoothness–manipulability cost defined previously.
The transition cost between states s and s^{\prime} is defined as
\Phi(s^{\prime},s)=\begin{cases}C_{\text{switch}},&r(s^{\prime})\neq r(s);\ \text{grasp pair feasible},\\[6.0pt]
\infty,&r(s^{\prime})\neq r(s);\ \text{grasp pair infeasible},\\[6.0pt]
\infty,&r(s^{\prime})=r(s);\ \tau(s^{\prime})\neq\tau(s).\end{cases}Here, r(s) denotes the active robot, \tau(s) the associated trajectory index, and C_{\text{switch}}>0 is a switching penalty that regularizes inter-robot handovers.
IV-E Dynamic Programming and Optimal Sequence Recovery
The problem admits optimal substructure and is solved via dynamic programming. The value function is defined as,
V_{i}(s)=\min_{s_{1:i-1}}\left(\sum_{k=1}^{i}C_{k}(s_{k})+\sum_{k=2}^{i}\Phi(s_{k-1},s_{k})\right),which represents the minimum cumulative cost of reaching segment i and terminating in state s.
Initialization.
V_{1}(s)=\begin{cases}C_{1}(s),&s\in\mathcal{S}_{0},\\
\infty,&\text{otherwise}.\end{cases}Bellman Recursion.
For i=2,\dots,N, recursion is
V_{i}(s)=C_{i}(s)+\min_{s^{\prime}\in\mathcal{S}}\left(V_{i-1}(s^{\prime})+\Phi(s^{\prime},s)\right).Infeasible transitions are excluded by setting \Phi(s^{\prime},s)=\infty. The recursion is equivalent to a shortest-path computation on a layered directed acyclic graph. For policy recovery, we store the predecessor mapping
\pi_{i}(s)=\arg\min_{s^{\prime}\in\mathcal{S}}\left(V_{i-1}(s^{\prime})+\Phi(s^{\prime},s)\right).Each optimal state encodes both the selected robot and its associated trajectory, i.e., s_{i}^{\star}=(r_{i}^{\star},\tau_{i}^{\star}). The resulting assignment sequence is \mathbf{s}_{1:N}^{\star}=((r_{1}^{\star},\tau_{1}^{\star}),\dots,(r_{N}^{\star},\tau_{N}^{\star})), from which the robot and trajectory sequences can be extracted as \mathbf{r}_{1:N}^{\star}=(r_{1}^{\star},\dots,r_{N}^{\star}) and \boldsymbol{\tau}_{1:N}^{\star}=(\tau_{1}^{\star},\dots,\tau_{N}^{\star}). Switching events are defined as \mathcal{H}=\{\,i\in\{2,\dots,N\}\mid r(s_{i}^{\star})\neq r(s_{i-1}^{\star})\,\}.
IV-F Complexity Reduction via Beam Search and Stride
To mitigate the \mathcal{O}(N|\mathcal{S}|^{2}) worst-case time complexity of the dynamic programming pass and ensure scalability to longer horizons, we introduce a beam search with a stride parameter. We evaluate potential dual-arm handovers only every k steps (stride) rather than at every discretized trajectory waypoint. Furthermore, during the forward pass, we retain only the top B lowest-cost states (beam width) for evaluation at the next step. This significantly prunes the state space and branching factor, preserving performance while drastically reducing planning times.
V Experiments and Results
Our experiments are designed to systematically evaluate the proposed system framework. Specifically, we aim to answer the following questions:
- To what extent does the proposed dynamic programming formulation improve task completion rates in scenarios involving heterogeneous grippers?
- What is the impact of incorporating diversity-aware grasp selection and manipulability-based costs on overall system performance?
- Which components are most crucial to the method’s performance, and what are the dominant failure modes observed in practice?
V-A Experimental Setup
All experiments are conducted on a heterogeneous dual-arm platform with two 7-DoF Franka FR3 manipulators: one equipped with a parallel-jaw gripper and the other with a suction gripper. The arms are arranged with overlapping workspaces to enable coordinated manipulation and object handovers, and their relative configuration is varied to induce different reachability and feasibility constraints. This setup allows evaluation of grasp-dependent affordances and asymmetric reachability. All experiments are conducted on a workstation equipped with an AMD Ryzen Threadripper PRO 7975WX processor and an NVIDIA RTX 6000 Ada Generation GPU.
Grasp Generation, Motion Planning and Execution:
- Inverse Kinematics: An EAIK [27] based IK solver with multi-joint enumeration (using the redundancy of 7 DOF) to check for the most continuous and manipulable joint pose.
- Grasp Generation: 6-DoF grasp candidates are sampled via GraspGen[12] for both parallel-jaw and suction grippers, followed by collision-based feasibility filtering. 40 grasps are sampled per gripper type, except in Task A: Plate from Table to Shelf (as described below) due to its restrictive environment, thus requiring more diverse sampling (60 grasps).
- Collision modeling: Full robot self-collision, object and environment meshes, and dual-arm interactions checked using MuJoCo’s default collision engine.
Simultaneous dual-arm motions use an asymmetric leader-follower setup: Arm A follows the planned, while Arm B tracks a reference via a reactive mink QP with collision-avoidance constraints. We have used mink here due to its natural incoporation with mujoco and being a QP solver it can incorporate posture tasks (for warm starts), collision handling and velocity/joint limits in its optimization while finding optimal control steps to control the arm (during planning we wanted speed in order to iterate fast upon a lot of possible configuration and solutions, thus used EAIK). Arm A’s joints are pinned, and transport segments are executed as constrained joint tracking to faithfully follow the object trajectory.
V-B Task Design
The tasks are designed with two objectives: (1) to enforce heterogeneous gripper use by making at least one trajectory segment infeasible for a single arm, requiring a handover; and (2) to introduce reachability-based infeasibility, demonstrating that the framework generalizes beyond collision constraints. We design both single-switch and two-switch tasks, described below (see Figure 3 for task examples).
- Task A: Plate from Table to Shelf A flat plate is positioned flush on the table to mandate a suction pick, while the destination shelf’s narrow clearance prohibits suction entry, deliberately forcing a handover for final placement.
- Task B: YCB Plate from Shelf to Table The recessed YCB plate blocks suction access, requiring a parallel-jaw grasp; however, tight boundaries at the target cause gripper collisions, necessitating a handover to suction for top-down placement on the table.
- Task C: Mug Inversion An inverted mug prevents parallel-jaw rim grasping, requiring suction pickup. A handover to the parallel jaw then enables flipping and upright placement in a container.
- Task D: Plate from box to box A flat plate is transferred between two containers. Only the suction gripper can pick and place at the containers due to collision constraints for the parallel jaw. However, intermediate regions are unreachable by suction due to a wall, along the constraint that the plate can’t go over it, thereby requiring two handovers.
- Single-Arm Baseline
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- Single Robot
- 0.0
- 0.0
- -
- 0
- -
- -
- Bimanual Method
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- 자료 없음
- Greedy (beam=1)
- 30.0
- 20.0
- 75.0
- 1.5
- 24.03
- 95.8
- No-Diversity
- 75.0
- 67.5
- 100.0
- 1.3
- 1015.87
- 84.8
- No-Manipulability (\beta=0)
- 80.0
- 52.5
- 89.1
- 2.7
- 502.43
- 138.7
- No-Switching Penalty (C_{\text{switch}}=0)
- 80.0
- 25.0
- 81.6
- 10.7
- 154.60
- 286.5
- Proposed (Exact DP)
- 80.0
- 72.5
- 100.0
- 1.3
- 532.96
- 92.0
- Proposed (Beam 10, Stride 3)
- 72.5
- 65.0
- 100.0
- 1.3
- 91.05
- 87.9
V-C Baselines
Exisiting object manipulation research with heterogeneous grippers has mainly focused on grasp generation and picking. We therefore compare our method against the following configurations:
V-C1 Greedy Reachability
At each segment, the robot with a feasible grasp and higher instantaneous manipulability is greedily selected (equivalent to a beam width of 1).
V-C2 No-Diversity Grasp Selection
The top-k grasps are selected purely based on grasp quality scores, without applying DPP-based diversity filtering.
We also evaluated single-robot execution, which achieved 0% success across all scenarios, confirming that the tasks require heterogeneous bimanual coordination.
V-D Quantitative Results
Table II summarizes performance across 520 simulated runs to address Q1, evaluating success rates, switches, and timing. Our exact DP method achieves 72.5% task success compared to the greedy baseline’s 20.0%, demonstrating the critical need for global look-ahead over myopic reachability decisions in complex transport tasks. Furthermore, excluding DPP-based grasp diversity (67.5% success) heavily penalizes planning time (1015.87 s), as concentrating grasps in similar poses redundantly expands the dynamic programming search space without improving workspace reachability.
While global optimization is vital, the Beam 10, Stride 3 configuration offers a practical computational trade-off. It drastically cuts average planning time from 532.96 s (Exact DP) to 91.05 s, while still maintaining a 65.0% task success rate and a 100.0% handover success rate.
V-E Per-Experiment Details and Evaluation
Table II details performance across the four tasks using the practical Beam 10, Stride 3 configuration, maintaining a robust 100.0% handover success globally. Task C: Mug Inversion achieved the highest success (80.0%) and rapid execution (63.8 s).
Conversely, Task A: Plate from Table to Shelf proved most challenging (50.0% success, 129.87 s planning time) due to environmental complexity and increased grasp sampling (60 vs. 40). Failures primarily stemmed from the inability to find valid robot assignments or confined insertion tolerances that forced suboptimal grasps, leading to transport slippage. Tasks B and D maintained balanced rates of 60.0% and 70.0%.
V-F Ablation Studies
We evaluate the contribution of key components via ablation studies, maintaining the rest of the pipeline unchanged (Table II).
V-F1 Switching Penalty and Frequency (C_{\text{switch}}=0)
Removing the switching penalty leads to excessive, unsystematic handovers, increasing the average switches from 1.3 to 10.7. Consequently, task success plummets to 25.0%, handover success drops to 81.6%, and execution time increases by 211.4%, demonstrating that unregularized handovers compound failure points. Conversely, systematic switching remains highly effective: valid executed plans requiring two switches maintain a robust 87.0% task success rate, compared to 90.3% for one-switch plans. This confirms that deliberate handovers safely navigate restricted workspaces without the severe performance drops of unregularized switching.
V-F2 Manipulability Cost (\beta=0)
Disabling the manipulability term drops overall task success to 52.5% and handover success to 89.1%. Without this objective, the solver frequently selects poorly conditioned kinematic states, resulting in jerky trajectories and unstable grasps. Prioritizing manipulability is therefore vital to smoothly and safely execute the complex, contact-rich motions required during physical dual-arm handovers.
V-G Real-World Validation via Sim and Real Digital Twin
To validate our simulated results, we deployed the framework on a physical dual Franka FR3 setup using Task B: YCB Plate from Shelf to Table and Task C: Mug Inversion. We bridged the sim-to-real gap via a ROS 2 and MuJoCo-based digital twin, computing dynamically verified trajectories with the Beam=10, Stride=3 configuration (which demonstrated a 60% and 80.0% task success rate). During physical deployment, impedance control safely absorbed minor calibration offsets during contact-rich handovers, successfully replicating the digital twin’s precise and safe bimanual behavior.
VI Limitations and Conclusion
Limitations We rely on pre-computed CAD meshes for tracking and DPP-based grasp generation, which limits zero-shot deployment on novel objects. Recent advances such as FoundationPose [25] have enabled on-the-fly mesh reconstruction from RGB-D observations and reduce this dependency. The framework also assumes a fixed, human-provided object trajectory. Future work could allow embodiment-aware trajectory refinement, where the planner slightly adjusts the demonstrated trajectory to achieve more stable handovers.
In conclusion, we present an embodiment-aware planning framework for heterogeneous bimanual manipulation. By jointly optimizing segment reachability, asymmetric grasp compatibility, and transition penalties, our method translates a single human demonstration into a globally optimal execution plan. Supported by extensive simulation sweeps and validated via a real-world digital twin, this approach systematically elevates simple industrial grippers toward the versatile execution of complex manipulation tasks.