DuoMindが分散マルチロボット協調を実現、意味通信でVLMとVLAを階層統合

Abstract

Vision–language models (VLMs) and vision–language–action models (VLAs) have recently driven rapid progress in general-purpose robots, yet most progress has focused on single-robot settings. Extending these capabilities to multi-robot systems remains challenging because robots must coordinate long-horizon behaviors while maintaining reliable, fine-grained execution. We introduce DuoMind, a distributed hierarchical framework for multi-robot coordination through semantic communication. Each robot uses a VLA-based action model for low-level execution and a VLM-based orchestrator for high-level reasoning and inter-agent coordination. At each planning step, the orchestrator at each robot reasons over the task instruction, local observations, and messages received from other robots. It then generates low-level instructions for the action model and semantic messages for peer robots. This architecture exploits the complementary strengths of pretrained models by combining the semantic reasoning capabilities of VLMs with the precise action-generation capabilities of VLAs. To address the scarcity of benchmarks for multi-robot coordination, we further develop RoboPoly, a benchmark comprising long-horizon manipulation tasks that require coordinated, closed-loop execution under distributed control. Experiments on RoboPoly and RoboTwin demonstrate that DuoMind improves multi-robot task performance, while ablation studies confirm the contributions of hierarchical orchestration and semantic communication. More details are available on our project page.

1 Introduction

Recent advances in vision-language models (VLMs) and vision-language-action models (VLAs) have substantially expanded the capabilities of general-purpose robots. VLMs (Beyer et al., 2024; Yang et al., 2025) provide strong visual understanding and high-level reasoning, while VLAs translate visual observations and natural-language instructions into robot actions (Zitkovich et al., 2023; Kim et al., 2024; Black et al., 2024; Intelligence et al., 2025b; Team et al., 2025). Building on these complementary capabilities, recent design of robot agents (Song et al., 2025a; Intelligence et al., 2025b; Team et al., 2025; Intelligence et al., 2025a) combines a high-level embodied reasoning model with a low-level VLA policy. In this design, the reasoning model aims to understand the environment and then decompose complex tasks into more specific instructions, while the VLA executes these instructions through fine-grained robot actions. Such a separation of functionalities allows the high-level model to focus on semantic understanding and long-horizon planning, while the VLA specializes in executing detailed instructions with precise low-level control.

Notably, most of the above progress has focused on a single robot acting in isolation, but many practical tasks are infeasible for a single agent and inherently require multi-robot collaboration, and others can benefit from multiple robots working together to improve efficiency. Effective multi-robot coordination requires each robot to construct appropriate subgoals and continuously adapt its behavior as the shared workspace evolves. For many practical settings, distributed execution (Amirkhani and Barshooi, 2022; Lo et al., 2024; Chen et al., 2024a) is particularly desirable, where each robot makes decisions primarily from its own observations rather than relying on a centralized controller, making the system more scalable as the number of robots increases (Nayak et al., 2023). Yet combining distributed multi-agent decision making with complex robotic manipulation creates a particularly challenging coordination problem, requiring robots to reason about both team-level cooperation and precise low-level task execution.

Conventional multi-agent learning and planning methods are often studied in relatively simple environments (Feng et al., 2026; Li et al., 2024; Zou et al., 2025; Wong et al., 2023; Liang et al., 2025a), with limited physical interaction and shorter task horizons. More recently, language-model-based robotic systems have leveraged the high-level reasoning capabilities of LLMs to decompose complex objectives and schedule robot behaviors (Chen et al., 2024b; He et al., 2026; Song et al., 2025b; Liu et al., 2025; Mandi et al., 2024). However, most of these systems focus primarily on high-level planning, while low-level control is executed through simply predefined action primitives or rule-based skills. An important gap remains between high-level multi-agent reasoning and end-to-end robot control. In addition, existing benchmarks provide limited support for systematic evaluation of multi-robot coordination, leaving the field without a comprehensive testbed for measuring such capabilities.

To address the above challenges, we introduce DuoMind, a distributed hierarchical framework for multi-agent robotic coordination, together with RoboPoly, a benchmark designed specifically for long-horizon multi-robot coordination. Each robot in DuoMind operates with a VLM as its orchestrator for high-level reasoning and a VLA as its action model for low-level control. At each planning step, each orchestrator reasons over the high-level instruction, its local observation, and the shared message from other robots. It then generates both a low-level instruction for the action model and a structured message for other robots. These two outputs serve complementary roles. The low-level instruction guides the action model to produce fine-grained robot actions, while the message communicates the robot’s intention, subgoal, and task belief, allowing each agent to remain aware of its peers despite partial observability (Liang et al., 2025b) in the distributed setting. In this way, DuoMind provides an agentic harness around the action model: the orchestrator dynamically reasons and invokes the action model to realize long-horizon coordinated behavior at a relatively low frequency, while the action model focuses on high-frequency execution. Importantly, this decoupled design allows DuoMind to capitalize on the strong pretrained capabilities of existing VLM and VLA backbones. Rather than heavily fine-tuning models specifically for multi-robot coordination, high-level coordination and low-level manipulation can be largely handled by off-the-shelf VLM models, requiring only limited adaptation to the multi-robot setting. To systematically evaluate the coordination capabilities, we introduce RoboPoly, which provides challenging long-horizon manipulation tasks in which multiple robots must coordinate and complete the task through closed-loop physical execution rather than high-level planning alone. We further evaluate DuoMind across diverse cooperative scenarios and systematically study the contributions of hierarchical design and inter-agent communication.

The main contributions of this paper are summarized as follows:

• We introduce DuoMind, a distributed hierarchical framework for end-to-end multi-agent robotic coordination. DuoMind utilizes a VLM as the orchestrator at each robot for high-level reasoning and a VLA as the action model for low-level control, enabling inter-agent semantic communication with natural language. This design harnesses the strong reasoning and control capabilities of existing foundation models to enable effective coordination in complex multi-robot tasks.
• We develop RoboPoly, a benchmark for long-horizon multi-robot coordination under distributed observations and control. The benchmark emphasizes closed-loop physical execution and includes tasks that are challenging for a single agent, requiring multiple agents to coordinate and schedule their behaviors throughout task execution.
• We conduct extensive experiments and ablation studies to evaluate DuoMind across challenging cooperative manipulation tasks on both RoboPoly and RoboTwin. Our results demonstrate the effectiveness of the proposed hierarchical design in multi-robot coordination and further quantify the contributions of integrating orchestrator and inter-agent communication.

2 DuoMind: Distributed Hierarchical Multi-Agent Framework

We introduce DuoMind, a distributed hierarchical framework for multi-agent robotic coordination. We first present the VLM-based orchestrator for high-level planning (2.1). We then describe how we bridges high-level planning and low-level control via embodied reasoning and agentic harness (2.2). Finally, we highlight the structured inter-agent communication (2.3).

2.1 Multi-Agent Coordination via Embodied Reasoning

Existing studies on hierarchical robotic architectures have shown that action models are effective at low-level control and instruction following when provided with specific commands (Team et al., 2025). To extend this capability toward more general and long-horizon tasks, a promising approach is to introduce a reasoning model as an orchestrator, leveraging its strong visual understanding, semantic grounding, and planning capabilities to generate appropriate instructions for the action model. In distributed multi-agent settings, where task coordination are substantially more complex, we extend this hierarchical paradigm to distributed multi-robot coordination in DuoMind.

Specifically, DuoMind decomposes multi-agent coordination into two complementary levels: high-level multi-agent reasoning and low-level execution. Each robot employs a VLM as an orchestrator that reasons over the task-level instruction, its local visual observation, and message shared by other agents. The orchestrator determines the robot’s current subtask and translates it into a single-agent, short-horizon instruction for low-level control. The VLA, serving as the action model, then follows the low-level instruction using local observations to generate fine-grained robot actions. This division of responsibility allows each module to focus on the capability for which it is best suited: the orchestrator handles high-level understanding and inter-agent coordination in multi-agent setting, while the action model performs reliable low-level control.

Importantly, this decomposition avoids requiring the action model to directly interpret complex multi-agent state that includes the joint behavior of multiple robots and environment, which can be extremely challenging since action models are primarily trained with single-agent demonstrations. It therefore decouples multi-agent reasoning from low-level control, reducing the need for the action model itself to possess strong reasoning capabilities for multi-agent coordination. Moreover, DuoMind does not require the orchestrator, the action model, or the interface between them to be extensively redesigned for multi-agent settings. These foundation models developed for single-agent systems can be reused largely off the shelf, with only lightweight adaptation when necessary. This greatly extends the possibility of deploying DuoMind to diverse multi-robot tasks and offers strong compatibility to future upgrades on components.

The design of DuoMind naturally supports coordination among multiple robots. However, the complexity of task design, data collection, and evaluation increases substantially as the number of agents grows. Considering the high computational cost when the number of robots scales up, we focus on two-agent settings in this work and leave coordination at larger team scales for future study.

2.2 Bridging High-Level Reasoning and Low-Level Control via Natural Language Based Semantic Communication

Hierarchical robotic architectures bridge high-level reasoning and low-level control through different intermediate representations, including natural-language subgoals or motion descriptions (Shi et al., 2025; Belkhale et al., 2024), spatial representations such as trajectories or visual subgoals (Li et al., 2025; Zhao et al., 2025), and learned latent representations in tightly coupled dual-system models (Chen et al., 2025a; Bjorck et al., 2025). Among these approaches, natural language offers interpretability and broad compatibility with language-conditioned models, whereas other representations can convey compact information but typically require stronger coupling between the two modules, making them harder to replace or adapt across architectures and embodiments.

Our framework adopts natural language as a unified interface between the high-level orchestrator and the low-level action model. The orchestrator harnesses the action model by generating low-level instructions in natural language, thereby decoupling high-level reasoning from low-level control. This design allows existing VLMs and VLAs to be incorporated as off-the-shelf components with minimal adaptation, without requiring them to be jointly trained through a specialized interface. It also enables either component to be independently replaced as stronger reasoning or action models become available.

Natural language further provides a common interface for inter-agent semantic communication. Each orchestrator can interpret messages from other agents even when they use different reasoning models, facilitating coordination across heterogeneous agents. The same interface also supports human participation by allowing users to inspect or revise instructions using ordinary language. More broadly, this shared linguistic interface provides a scalable foundation for future multi-robot systems, where heterogeneous robots and multiple human users can coordinate through a common communication protocol.

2.3 Inter-Agent Information Sharing

Distributed multi-agent systems face the challenges of partial observability and non-stationarity: each agent has only partial knowledge of the global task state, while the environment continues to evolve as other agents act. Insufficient awareness of such information can lead to conflicting actions or ineffective collaboration. To address these challenges, DuoMind enables agents to share and integrate structured natural-language messages during high-level reasoning. These messages convey key information about other agents’ decisions and task-relevant changes in the environment, providing additional context for more precise high-level reasoning. To support effective decision-making, we structure each shared message using four fields:

Intention. It is identical to the low-level instruction sent to the action model and describes the next subtask that the ego agent intends to execute. Sharing this intention makes other agents aware of the ego agent’s next action, helping them avoid behaviors that may interfere with its execution.

Subgoals. This field specifies the expected task state after the ego agent completes its current subtask. It explicitly communicates the future state that the ego agent aims to achieve, allowing other agents to act proactively to facilitate task completion or avoid undesirable outcomes.

Belief of task. This part describes task-relevant information observed or inferred by the ego agent, such as the location of a key object or whether a particular subtask has been completed. For example, in a kitchen task, a robot operating near the cabinets may inform another robot preparing to cook that “the pot is in the third cabinet.” Sharing such information reduces the impact of partial observability by propagating locally available knowledge across agents.

Uncertainty of belief. This optional field communicates the uncertainty associated with the ego agent’s task belief. It is particularly useful in complex tasks, where information inferred through reasoning may not always be reliable. This uncertainty provides an additional confidence signal that allows the orchestrator to appropriately weigh different beliefs during high-level reasoning, especially when agents hold inconsistent beliefs or when the available information is ambiguous.

Through this structured information sharing, DuoMind provides each agent with compact contextual information about the intentions, subgoals, beliefs, and associated uncertainty of other agents. This additional context enables the orchestrator to generate more precise low-level instructions, thereby improving the safety and efficiency of multi-robot coordination.

3 RoboPoly

RoboPoly is designed to facilitate research on multi-robot manipulation by providing challenging long-horizon tasks with explicit cooperation requirements. Unlike RoboTwin Chen et al. (2025c), which primarily targets dual-arm manipulation with two arms on the same embodiment and does not necessarily require explicit cooperation, RoboPoly considers two independent robots and tasks specifically designed around multi-robot coordination. These tasks are either infeasible or inefficient for a single robot to complete independently, requiring each robot to reason and determine when cooperation is needed, and act at the appropriate time.

RoboPoly highlights distributed execution, where each robot acts based on its own observations and policy while cooperating with another robot. The benchmark is built upon the ManiSkill3 framework and contains seven multi-robot manipulation tasks. In this section, we describe the task suite, basic settings, and dataset generation.

3.1 Tasks for Multi-Robot Coordination

All tasks share a common workspace configuration with two robot arms positioned on opposite sides of a table. The workspace and object placement are designed such that a single robot cannot always complete the task independently or efficiently, making cooperation necessary. RoboPoly contains the following seven tasks: Food Serve, Prepare Snack, Clean Table, Cook Pot, Put Object Cabinet, Hang Bag, Exchange Bread. We further detail the task description in A.

3.2 Observations, Actions, and Robots

Each environment contains three RGB cameras: one global camera with a resolution of 512\times 384 and two wrist cameras with a resolution of 128\times 128. The global camera is positioned beside the table to provide an overview of the workspace, while each robot is equipped with a wrist camera that provides an egocentric view of its gripper. We adopt a distributed observation setting in which each robot receives observations from the global camera and its own wrist camera, but has no access to the wrist camera of the other robot. This design follows the distributed system setting and requires the robots to coordinate using only their locally available observations.

Both robots use 7-DoF Franka Emika Panda manipulators equipped with grippers. Under the distributed control setting, each policy controls only one robot rather than directly generating the joint actions of both robots. To improve task diversity, key objects are randomized within predefined task-specific regions at the beginning of each episode.

3.3 Datasets

We construct the datasets using the motion-planning-based trajectory generation pipeline provided by ManiSkill3. For each task, we first decompose the full manipulation procedure into a sequence of key states, such as grasping, releasing, or moving toward a target location. Given these key states, the generation script invokes the motion planner to construct smooth robot trajectories between consecutive states. The spatial and temporal constraints for each trajectories are carefully designed with the planner to generate the desired multi-robot coordination demonstrations.

In addition to robot trajectories, we associate each manipulation stage with a low-level instruction for each robot describing the current behavior. We also provide a high-level instruction that specifies the overall task objective. During data generation, the pipeline first plans a successful trajectory and records the corresponding robot actions and states. It then replays the trajectory in simulator to render the RGB camera observations. The resulting dataset therefore contains visual observations, states, robot actions, high-level, and low-level language instructions for both robots. We further open-source the dataset to foster research on multi-robot coordination.

4 Experiments

We evaluate DuoMind on 7 tasks from our RoboPoly and 8 dual-arm tasks from RoboTwin 2.0 (Chen et al., 2025c), both under distributed control. Since RoboTwin originally adopts centralized control over the dual-arm robot, we adapt its setting by treating the two arms as independent robots and control them separately. Task overview are shown in fig. 5 and fig. 6. All the finetuning and evaluation tasks are conducted on 4 NVIDIA A100 GPU, each taking 10–24 hours.

In the RoboPoly experiments, each Franka Emika Panda robot has 7 DoF and is equipped with a wrist camera, while both robots share a global camera positioned beside the workspace. We adopt the same distributed setting in which each robot has its own observations and policy. For each task, we collect 50 dual-robot demonstrations using planning-based trajectory generation and then decompose them into a unified single-robot dataset containing 100 trajectories for finetuning.

On RoboTwin, we select the tasks that require the coordination of two arms for multi-robot experiment. We treat the dual-arm Aloha-AgileX as two individual robots, each operating with its own observations and independently controlled by a separate policy. Each arm has 6 DoF with a wrist camera and a shared head camera. At finetuning stage, we use 50 official dual-arm demonstrations on each task. Each demonstration is decomposed into separate left- and right-arm trajectories, where each trajectory is paired with observations from the corresponding wrist camera and the shared head camera. We then merge the trajectories from both arms into a unified single-arm dataset, resulting in 100 demonstrations per task.

4.1 Experiment Settings

Low-Level Control. We adopt \pi_{0.5} (Intelligence et al., 2025b) as the low-level VLA controller in DuoMind due to its strong grounding in single-arm manipulation and generalization capability across diverse tasks and environments. We fine-tune \pi_{0.5} using LoRA with batch sizes ranging from 256 to 1024 for 2k–10k training steps on RoboPoly and with batch sizes 32 for 20k training steps on RoboTwin, depending on the task. Detailed fine-tuning configurations are provided in D.

In RoboPoly experiments, the dataset includes low-level instructions, which are directly used to fine-tune the VLA. In RoboTwin experiments, however, the official dataset does not include low-level instructions and instead provides a high-level instruction, which we use during fine-tuning. At evaluation time, we retain the same hierarchical setting as in RoboPoly and provide decomposed low-level instructions to the VLA. Although these instructions does not fully match the fine-tuning dataset, the decomposition still benefits execution through the inherent generalization capability of the pretrained VLA. We further elaborate on this effect in section 4.2.

High-Level Reasoning. We integrate Qwen3-VL-4B-Instruct (Bai et al., 2025) as the high-level reasoning module in DuoMind, as it provides strong visual grounding while remaining compact enough for practical inference. To prevent the model from generating low-level instructions that exceed the instruction-following capability of the VLA policy or fall outside its training distribution, we restrict each generated instruction to a predefined set of admissible examples. This constraint improves the reliability and consistency of low-level execution. In addition, to prevent the system from remaining indefinitely at an intermediate stage, we introduce a simple progress mechanism that forces the high-level instruction to advance when reasoning stagnates.

Baseline. To compare against other control models, we fine-tune \pi_{0.5} and \pi_{0} (Black et al., 2024) on each task using the corresponding high-level instruction as baseline methods. Except for the instruction setting, all fine-tuning configurations are kept identical to those used for DuoMind.

Evaluation. During evaluation, DuoMind and all baseline methods receive the same high-level instruction and observations as input from the simulation environment. We evaluate DuoMind and the baselines over 400 rollouts per task and report the corresponding success rates in table 2 and table 1. Detailed evaluation settings are provided in D.

4.2 Main Results

We evaluate DuoMind on seven tasks in RoboPoly and eight tasks in RoboTwin against strong baselines, with success rates reported in table 1 and table 2, respectively. Detailed task rollout examples are provided in appendix E. Overall, DuoMind matches or outperforms the baseline methods across most tasks. We attribute these gains to the high-level reasoning and inter-agent communication enabled by the hierarchical design.

For long-horizon multi-robot tasks, a global high-level instruction provides only coarse guidance throughout the entire task execution and offers limited information about what each robot should do at the current stage or how it should adapt to the behavior of its peers. With the hierarchical design, in contrast, the orchestrator reasons over the current task state together with messages from other agents, decomposes the task into appropriate subtasks, and provides specific low-level instructions to the action model. This allows each robot to adapt its behavior as the task evolves while avoiding conflicts with other agents.

Inter-agent communication is particularly important under partial observability, where local observations may not reveal the intentions or future actions of other robots. Explicit message sharing provides complementary information for high-level reasoning, enabling the orchestrator to make better-informed coordination decisions. With more precise low-level instructions, the action model can focus on executing the assigned subtask rather than implicitly reasoning about the overall task progression and multi-agent coordination. As shown in fig. 3, the robots coordinate effectively and complete the task in the proper sequence without conflicts.

This distinction becomes especially important when execution deviates from the planned behavior. Policies such as \pi_{0} and \pi_{0.5} that continuously follow a fixed high-level instruction may become uncertain about the appropriate next action after unexpected interactions or execution errors. Such situations occur frequently in multi-robot manipulation, where the behavior of one robot can alter another robot’s workspace. The orchestrator can recognize these changes and adaptively issue new instructions, allowing the action model to recover and continue the task.

Additionally, tasks in RoboPoly generally have longer horizons and require more stable coordination than those in RoboTwin, resulting in lower overall success rates. Nevertheless, the performance improvement of DuoMind over the baselines is more significant on RoboPoly, further demonstrating the effectiveness of high-level reasoning and inter-agent communication and suggesting that their benefits become increasingly significant as the task horizon and coordination complexity increase.

4.3 Ablation of Inter-Agent Communication

Inter-agent communication provides complementary information for decision making and helps mitigate partial observability. To evaluate its effect, we conduct an ablation, following the evaluation setting in section 4.2 except that messages from peer agents are disabled. As shown in fig. 4(a), removing inter-agent communication leads to more frequent conflicts and asynchronous behaviors, resulting in degraded coordination performance. These results demonstrate that explicitly sharing peer information is important for effective multi-robot coordination under distributed observations. We further elaborate the ablation study in B.

4.4 Action Model Compatibility

DuoMind decouples high-level orchestration from low-level control, allowing different action models to be integrated into the framework. To study the influence of the action model, we evaluate DuoMind with \pi_{0} in place of \pi_{0.5}. As shown in fig. 4(b), since \pi_{0} is a weaker action model than \pi_{0.5}, DuoMind with \pi_{0} achieves lower success rates than DuoMind with \pi_{0.5}, as expected. Nevertheless, it substantially outperforms the \pi_{0}-only baseline, demonstrating that the hierarchical orchestration of DuoMind consistently improves multi-robot coordination across different action models. At the same time, the stronger action model provides more reliable low-level execution, which further improves the overall task success rate. We detail this analysis in C.

5 Related Work

5.1 Learning-Based Multi-Agent Coordination

Classical multi-agent learning studies decentralized decision-making under partial observability and non-stationarity through centralized training, value decomposition, and learned communication (Lowe et al., 2017; Rashid et al., 2020; Yu et al., 2022; Foerster et al., 2016). These methods are effective in cooperative games and simulation, but often require task-specific joint training and encode coordination in latent representations that are hard to interpret or integrate with pretrained robot foundation models. Recent work extends coordination to physical robots through decentralized imitation learning, shared representations, and VLA policies (Dong et al., 2025; He et al., 2025; Qin et al., 2025; Doshi et al., 2026). In contrast, DuoMind uses decentralized reasoning with explicit natural-language communication for interpretable exchange of task-relevant information.

5.2 Robot Learning and Multi-Robot Planning

Generalist robot policies leverage large-scale robot data and pretrained vision/language representations to map observations or language instructions to actions (Zitkovich et al., 2023; Kim et al., 2024; Black et al., 2024; Zhou et al., 2026; Team et al., 2025; Gao et al., 2026). Hierarchical approaches further separate high-level semantic reasoning from low-level continuous control (Shi et al., 2025; Intelligence et al., 2025b; Song et al., 2025a; Team et al., 2025), but mainly focus on single-robot execution. LLM- and VLM-based multi-robot systems explore task decomposition, allocation, communication, feedback, and replanning (Kannan et al., 2024; Chen et al., 2024b; Mandi et al., 2024; Liu et al., 2025; Chen et al., 2025b; Wang et al., 2024; He et al., 2026), while typically relying on predefined skills, symbolic actions, waypoints, or motion planners. DuoMind instead combines distributed high-level reasoning and communication with VLA-based physical execution.

5.3 Robot Learning and Multi-Agent Benchmarks

Established robot-learning benchmarks provide diverse manipulation tasks, demonstrations, language instructions, and long-horizon evaluation, but largely focus on single robots or centrally controlled embodiments (Yu et al., 2020; James et al., 2020; Gu et al., 2023; Mees et al., 2022; Liu et al., 2023). Recent multi-agent benchmarks study collaborative planning, heterogeneous capabilities, and manipulation (Mandi et al., 2024; Liu et al., 2025; Chen et al., 2025b; Chang et al., 2025; Qin et al., 2025). However, they often emphasize either abstract high-level planning or low-level manipulation. RoboPoly instead evaluates long-horizon coordination and closed-loop execution under distributed observation and control.

6 Conclusion

We presented DuoMind, a distributed hierarchical framework for multi-robot coordination that combines high-level reasoning, low-level control, and semantic inter-agent communication. We further introduced RoboPoly, a benchmark for evaluating long-horizon multi-robot manipulation under distributed observations and control. Experiments on RoboPoly and RoboTwin demonstrate that hierarchical orchestration improves multi-robot task execution, while ablation studies highlight the importance of explicit information sharing for coordination under partial observability. The decoupled design also allows different reasoning and action models to be integrated without redesigning the overall framework. As stronger VLMs and VLAs continue to emerge, DuoMind can naturally incorporate these advances to further improve its coordination capabilities. Leveraging such advances toward more capable and efficient real-world multi-robot coordination is a promising direction for future work.

Acknowledgments

We thank Keyu Zhu and Vivian Xie for their assistance in developing RoboPoly.

References

1. Amirkhani and Barshooi (2022) A. Amirkhani and A. H. Barshooi Consensus in multi-agent systems: a review. Artificial Intelligence Review 55 (5), pp. 3897–3935.
2. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.
3. Belkhale et al. (2024) S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh Rt-h: action hierarchies using language. arXiv preprint arXiv:2403.01823.
4. Beyer et al. (2024) L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: a versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726.
5. Bjorck et al. (2025) J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: an open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734.
6. Black et al. (2024) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. \pi_{0}: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164.
7. Chang et al. (2025) M. Chang, G. Chhablani, A. Clegg, M. Dallaire Cote, R. Desai, M. Hlavac, V. Karashchuk, J. Krantz, R. Mottaghi, P. Parashar, et al. Partnr: a benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations, Vol. 2025, pp. 65205–65268.
8. Chen et al. (2025a) H. Chen, J. Liu, C. Gu, Z. Liu, R. Zhang, X. Li, X. He, Y. Guo, C. Fu, S. Zhang, et al. Fast-in-slow: a dual-system foundation model unifying fast manipulation within slow reasoning. arXiv preprint arXiv:2506.01953.
9. Chen et al. (2024a) J. Chen, J. Feng, W. Gao, and K. Wei Decentralized natural policy gradient with variance reduction for collaborative multi-agent reinforcement learning. Journal of Machine Learning Research 25 (172), pp. 1–49.
10. Chen et al. (2025b) J. Chen, C. Yu, X. Zhou, T. Xu, Y. Mu, M. Hu, W. Shao, Y. Wang, G. Li, and L. Shao Emos: embodiment-aware heterogeneous multi-robot operating system with llm agents. In International Conference on Learning Representations, Vol. 2025, pp. 56670–56690.
11. Chen et al. (2025c) T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, et al. Robotwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088.
12. Chen et al. (2024b) Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan Scalable multi-robot collaboration with large language models: centralized or decentralized systems?. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4311–4317.
13. Dong et al. (2025) D. Dong, M. Bhatt, S. Choi, and N. Mehr Mimic-d: multi-modal imitation for multi-agent coordination with decentralized diffusion policies. arXiv preprint arXiv:2509.14159.
14. Doshi et al. (2026) R. Doshi, T. Gao, A. Chen, C. Finn, and J. Bohg CHORUS: decentralized multi-embodiment collaboration with one vla policy. arXiv preprint arXiv:2606.12352.
15. Feng et al. (2026) Z. Feng, R. Xue, L. Yuan, Y. Yu, N. Ding, M. Liu, B. Gao, J. Sun, X. Zheng, and G. Wang Multi-agent embodied ai: advances and future directions. Science China Information Sciences 69 (5), pp. 151202.
16. Foerster et al. (2016) J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems 29.
17. Gao et al. (2026) D. Gao, B. Zhao, A. Lee, I. Chuang, H. Zhou, H. Wang, Z. Zhao, J. Zhang, and I. Soltani Vita: vision-to-action flow matching policy. In International Conference on Learning Representations, Vol. 2026, pp. 100903–100933.
18. Gu et al. (2023) J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, et al. Maniskill2: a unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659.
19. He et al. (2025) C. He, G. S. Camps, X. Liu, M. Schwager, and G. Sartoretti Latent theory of mind: a decentralized diffusion architecture for cooperative manipulation. arXiv preprint arXiv:2505.09144.
20. He et al. (2026) Y. He, L. Wei, H. Cen, J. Jiang, Z. Li, G. Lu, Y. Yang, D. Zhang, and W. Zheng A closed-loop multi-agent framework for robust multi-robot manipulation. arXiv preprint arXiv:2607.06990.
21. Intelligence et al. (2025a) P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, et al. \pi_{0.6}^{*}: A VLA That Learns from Experience. arXiv preprint arXiv:2511.14759.
22. Intelligence et al. (2025b) P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky \pi_{0.5}: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054.
23. James et al. (2020) S. James, Z. Ma, D. R. Arrojo, and A. J. Davison Rlbench: the robot learning benchmark & learning environment. IEEE Robotics and Automation Letters 5 (2), pp. 3019–3026.
24. Kannan et al. (2024) S. S. Kannan, V. L. Venkatesh, and B. Min Smart-llm: smart multi-agent robot task planning using large language models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 12140–12147.
25. Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: an open-source vision-language-action model, 2024. URL https://arxiv. org/abs/2406.09246 1 (2), pp. 4.
26. Li et al. (2024) H. Li, H. N. Mahjoub, B. Chalaki, V. Tadiparthi, K. Lee, E. Moradi-Pari, M. Lewis, and K. Sycara Language grounded multi-agent reinforcement learning with human-interpretable communication. Advances in Neural Information Processing Systems 37, pp. 87908–87933.
27. Li et al. (2025) Y. Li, Y. Deng, J. Zhang, J. Jang, M. Memmel, C. Garrett, F. Ramos, D. Fox, A. Li, A. Gupta, et al. Hamster: hierarchical action models for open-world robot manipulation. In International Conference on Learning Representations, Vol. 2025, pp. 24040–24068.
28. Liang et al. (2025a) F. Liang, T. Zheng, C. Chan, Y. Yim, and Y. Song Llm-hanabi: evaluating multi-agent gameplays with theory-of-mind and rationale inference in imperfect information collaboration game. arXiv preprint arXiv:2510.04980.
29. Liang et al. (2025b) J. Liang, H. Miao, K. Li, J. Tan, X. Wang, R. Luo, and Y. Jiang A review of multi-agent reinforcement learning algorithms. Electronics 14 (4), pp. 820.
30. Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, pp. 44776–44791.
31. Liu et al. (2025) K. Liu, Z. Tang, D. Wang, Z. Wang, X. Li, and B. Zhao Coherent: collaboration of heterogeneous multi-robot system with large language models. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 10208–10214.
32. Lo et al. (2024) Y. L. R. Lo, B. Sengupta, J. Foerster, and M. Noukhovitch Learning multi-agent communication with contrastive learning. In International Conference on Learning Representations, Vol. 2024, pp. 51029–51046.
33. Lowe et al. (2017) R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30.
34. Mandi et al. (2024) Z. Mandi, S. Jain, and S. Song Roco: dialectic multi-robot collaboration with large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 286–299.
35. Mees et al. (2022) O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard Calvin: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters 7 (3), pp. 7327–7334.
36. Nayak et al. (2023) S. Nayak, K. Choi, W. Ding, S. Dolan, K. Gopalakrishnan, and H. Balakrishnan Scalable multi-agent reinforcement learning through intelligent information aggregation. In International conference on machine learning, pp. 25817–25833.
37. Qin et al. (2025) Y. Qin, L. Kang, X. Song, Z. Yin, X. Liu, X. Liu, R. Zhang, and L. Bai Robofactory: exploring embodied agent collaboration with compositional constraints. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10075–10085.
38. Rashid et al. (2020) T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21 (178), pp. 1–51.
39. Shi et al. (2025) L. X. Shi, B. Ichter, M. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, et al. Hi robot: open-ended instruction following with hierarchical vision-language-action models. arXiv preprint arXiv:2502.19417.
40. Song et al. (2025a) H. Song, D. Qu, Y. Yao, Q. Chen, Q. Lv, Y. Tang, M. Shi, G. Ren, M. Yao, B. Zhao, et al. Hume: introducing system-2 thinking in visual-language-action model. arXiv preprint arXiv:2505.21432.
41. Song et al. (2025b) S. Song, X. Xie, Z. Li, Y. Li, S. Wang, and B. Qi Leveraging adaptive group negotiation for heterogeneous multi-robot collaboration with large language models. arXiv preprint arXiv:2602.06967.
42. Team et al. (2025) G. R. Team, A. Abdolmaleki, S. Abeyruwan, J. Ainslie, J. Alayrac, M. G. Arenas, A. Balakrishna, N. Batchelor, A. Bewley, J. Bingham, et al. Gemini robotics 1.5: pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer. arXiv preprint arXiv:2510.03342.
43. Wang et al. (2024) Y. Wang, R. Xiao, J. Y. L. Kasahara, R. Yajima, K. Nagatani, A. Yamashita, and H. Asama Dart-llm: dependency-aware multi-robot task decomposition and execution using large language models. arXiv preprint arXiv:2411.09022.
44. Wong et al. (2023) A. Wong, T. Bäck, A. V. Kononova, and A. Plaat Deep multiagent reinforcement learning: challenges and directions: a. wong et al.. Artificial Intelligence Review 56 (6), pp. 5023–5056.
45. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
46. Yu et al. (2022) C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems 35, pp. 24611–24624.
47. Yu et al. (2020) T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine Meta-world: a benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094–1100.
48. Zhao et al. (2025) Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. Cot-vla: visual chain-of-thought reasoning for vision-language-action models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1702–1713.
49. Zhou et al. (2026) H. Zhou, B. Lynch, R. Goyal, D. Gao, B. Kasap, B. Zhao, and J. Zhang Agile-wam: an agile tactile world action model for contact-rich robot control. arXiv preprint arXiv:2609.20761.
50. Zitkovich et al. (2023) B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: vision-language-action models transfer web knowledge to robotic control. In Conference on robot learning, pp. 2165–2183.
51. Zou et al. (2025) J. Zou, R. Qiu, G. Li, X. Yang, K. Tieu, P. Lu, K. Shen, H. Tong, Y. Choi, J. He, et al. Latent collaboration in multi-agent systems. arXiv preprint arXiv:2511.20639.

Appendix A Task Settings of RoboPoly

RoboPoly contains the following seven tasks:

Food Serve. Two pieces of bread and a serving tray are placed across the workspace such that one robot cannot reach both the bread and the tray. The robots must transfer the bread between each other when necessary and place both pieces onto the tray.

Prepare Snack. Two plates are placed near the center of the table, while four pieces of bread are distributed across the two sides of the workspace. The robots must place two pieces on each plate. Both robots can operate simultaneously and must coordinate their motions to efficiently complete the task while avoiding collisions.

Clean Table. Two cubes and a container are placed on the table such that neither robot can independently reach all cubes and container. Completing the task requires a cube to be passed between the robots when the container lies outside the workspace of a robot.

Cook Pot. A carrot, a pot with a lid, and an oven are placed around the center of the workspace. The robots must first remove the lid, place the carrot into the pot, and replace the lid. They should then jointly grasp the two handles of the pot and transport it to the oven.

Put Object Cabinet. A cube and a cabinet are placed in the workspace. The robots must open the bottom drawer of the cabinet, place the cube inside, and close the drawer. The task requires coordination between two robot arms for higher efficiency.

Hang Bag. A bag and a hook are placed on opposite sides of the workspace such that neither robot can reach both. One robot must first pass the bag to the other robot, which then hangs the bag onto the hook.

Exchange Bread. A piece of bread and a target box are placed beside each robot arm. Each robot must place its own bread into the box on the opposite side of the table. Since the target box lies outside its reachable workspace, the robot must first hand the bread over to the other robot, which then places it into the corresponding box.

Appendix B Effect of Inter-Agent Communication

Inter-agent communication shares complementary information for decision making and helps mitigate the partial observability inherent in distributed multi-robot settings. To further study the effect of the shared messages, we conduct an ablation using DuoMind with \pi_{0.5} as the action model, following the evaluation setting in section 4.2 except that messages from peer agents are disabled. Under this setting, each orchestrator receives no information from the other agent during reasoning and generates low-level instructions based only on its local observation and the task instruction.

The results are reported in fig. 7. Without inter-agent communication, we observe more frequent conflicts and asynchronous behaviors between the two robots. For example, in Cook Pot, the two robots are required to lift the pot handles in a coordinated manner. Without communication, one robot often starts lifting while the other has only begun grasping the handle, causing the coordinated manipulation to fail. In Prepare Snack, the two robots may independently select the same plate as their target and attempt to place bread simultaneously, which can lead to collisions between the robot arms above the plate.

These failure cases illustrate a limitation of relying solely on local observations: an individual agent may not have sufficient information to infer the peer agent’s current intention or upcoming behavior. Explicit inter-agent communication provides a direct and efficient mechanism for sharing such information, thereby reducing conflicts and improving synchronized and safe multi-robot coordination.

Appendix C Action Model Compatibility

DuoMind decouples the high-level orchestrator from the low-level action model, enabling different action models to be integrated without changing the overall coordination framework. To analyze the influence of the action model, we conduct an additional experiment using \pi_{0} in place of \pi_{0.5}, while following the same fine-tuning procedure described in section 4.2.

As shown in fig. 8, because \pi_{0} provides weaker low-level control capability than \pi_{0.5}, DuoMind with \pi_{0} achieves lower success rates than DuoMind with \pi_{0.5}, as expected. However, DuoMind with \pi_{0} still substantially outperforms the \pi_{0}-only baseline. This comparison shows that the performance gain is not solely determined by the underlying action model. By introducing high-level reasoning and subtask decomposition through the orchestrator, DuoMind provides more appropriate low-level instructions for the evolving multi-agent state and improves coordination over the action-model-only setting.

The comparison also highlights the complementary roles of the two components. The orchestrator improves high-level reasoning and multi-agent coordination, while a stronger action model provides more reliable and stable low-level execution. Consequently, improving the capability of the action model can further increase the overall success rate. These results demonstrate that DuoMind is compatible with different action models while continuing to benefit from advances in low-level VLA policies.

Appendix D Training and Evaluation Configurations

The fine-tuning and evaluation settings of RoboPoly and RoboTwin are summarized in tables 3 and 4. All methods use LoRA fine-tuning and 400 evaluation rollouts per experiment. The batch sizes reported in the table refer to global batch size. For RoboTwin, the policy takes the head camera and each robot’s own wrist image as input, while on RoboPoly it takes the global camera and each robot’s own wrist image. The fine-tuning and evaluation of \pi_{0.5} and \pi_{0} adopt the same settings.

In the tables, Execute / horizon denotes the number of actions executed from each predicted action chunk and the total action-chunk horizon, respectively. VLM interval denotes the number of policy chunks executed between two consecutive calls to the high-level VLM orchestrator. We use a linear warmup over the first 1,000 training steps to reach the peak learning rate, followed by cosine decay to a final learning rate of 2.5\times 10^{-6}.

Appendix E Evaluation Visualization

Appendix F VLM System Prompt

We guide the orchestrator with a general system prompt shared across agents and tasks, which specifies its role and the type of low-level instruction to generate for the action model. The orchestrator serves two roles. First, it performs high-level planning by reasoning over the ego agent’s reachable workspace, the peer agent’s intention, and the current task state. It selects a complementary subgoal that avoids conflicts with other agents and contributes effectively to the overall task. Second, it generates a structured message for inter-agent communication, providing concise information about the ego agent’s intention, belief, and uncertainty to support the peer agent’s reasoning and coordination.