Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination
Preprint. The authors introduce Watch, Infer, Coordinate, a benchmark and inference method for a helper robot to watch a constrained partner collaborate with a demonstrator, infer the partner's low-level physical constraints, then coordinate zero-shot on a new joint-carrying task. In simulation across three manipulation settings, the approach reaches 85–93% success on obstacle tasks, close to an oracle that knows the true constraints, and far above baselines. This matters for multi-robot fleets where one unit may degrade or fault and partners must adapt without explicit communication.
Can a robot infer a partner's low-level physical action constraints by observing the partner coordinate with another robot, and then use that inferred capability for zero-shot coordination on a new manipulation task?
In physically coupled multi-robot manipulation such as carrying a shared object, one robot may have unknown hardware limitations—restricted joint positions, velocities, or force directions—due to degradation or faults. Existing systems either treat capabilities as known, adapt reactively without inferring persistent constraints, or reason only about task-level capabilities, so a helper may plan motions that the partner physically cannot execute when the task changes.
Earlier work includes partner-aware assistance methods such as Watch-and-Help, TICC-POMDP, Smart Help, CHAIC, CE-CM-Div, CAMO, and GRACE, which infer goals, beliefs, or coarse task-level capabilities; and physically coupled collaboration methods such as CHORUS, model-based planners, reinforcement learning, and diffusion policies, which learn decentralized control without explicitly inferring persistent low-level constraints. Constraint-inference-from-demonstration methods mostly recovered environmental constraints or single-agent limits, not joint constraints from multi-agent coupled behavior.
The benchmark has three simulated joint-carrying settings—2D rod carrying, fixed-base dual-UR5 carrying, and mobile dual-UR5 carrying—with 2,665, 172, and 475 candidate capabilities respectively. After observing obstacle-free demonstrations, the helper scores each candidate constraint by how well it predicts both robots' actions under a boundedly rational shared-objective model; in higher-dimensional settings, learned models estimate these likelihoods. The inferred constraint is then used in a CEM-based MPC planner to restrict the partner's feasible actions during rollouts on a new obstacle task.
Inference error (normalized Hamming distance, lower better) with 8 demonstrations: 2D rod 0.002 for Ours vs 0.302 for CE-CM-Div; fixed-base dual-UR5 0.019 vs 0.074; mobile dual-UR5 0.009 vs 0.250. With 1 demonstration, Ours still outperforms: 0.058 vs 0.304, 0.031 vs 0.074, 0.190 vs 0.306. An ablation using only the constrained partner's behavior rather than both agents roughly doubles inference error in fixed-base dual-UR5 and is worse in mobile dual-UR5; in 2D rod, including the demonstrator helps most at 1 and 2 demonstrations (0.104 vs 0.058 at N=1). On new obstacle tasks, success: Ours 92.6%, 85.2%, 63.0% for 2D rod, fixed-base, mobile; Oracle 96.3%, 88.9%, 70.4%; CE-CM-Div 55.6%, 44.4%, 37.0%; capacity-blind CEM 51.9%, 0.0%, 25.9%; behavioral cloning 0% all. SNA on obstacle tasks: Ours 72.0, 31.1, 34.5 vs Oracle 76.0, 31.3, 38.4. On obstacle-free control, Ours and Oracle both achieve 100% success in all three settings; capacity-blind fixed-base gets 0% success. Results are means over three random seeds.
Authors state assumptions: known behavioral model, known goals and dynamics, full observations, and a finite candidate constraint set; directly comparable baselines are limited. The evaluation is simulation-only with no physical-robot trials, over three seeds; finite candidate constraints may miss novel or continuous faults, and the framework assumes the helper can observe both agents' states and actions. Code is planned but not yet released.
This could inform multi-robot and mobile-manipulator products where one robot must compensate for a degraded partner—e.g., warehouse automation, factory robot cells, construction or logistics robots, and eventually human-robot teams. The current work is a preprint simulation benchmark, so initial value is likely in research and internal fleet modeling. In 1–3 years, if validated on physical robots with learned or online-updated constraints, robot fleet providers could use this to reduce downtime and manual reconfiguration after actuator faults. In 3+ years, similar techniques might enable safe ad hoc coordination across heterogeneous robots or with humans, but that requires solving partial observability, unknown/continuous constraints, and real-world dynamics drift.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.