Permutation Robustness Is Not Enough: Action Collapse in Multi-Agent Transformer Policies

Do transformer-based multi-agent robot policies learn coordination or agent ordering, and how should their robustness to input permutations be measured without confusing it with action collapse?

Multi-robot teams are unordered, but transformer policies process agents as ordered token sequences and often use learned identity or positional embeddings; this can bind behavior to token slots and make policies fail when agent order changes. Existing permutation-error metrics can be gamed by degenerate policies that emit identical actions for all agents, so low robustness error may hide a lack of differentiated team behavior.

Prior cooperative MARL methods (e.g., MADDPG, COMA, VDN, QMIX, PPO/MAPPO) and attention-based interaction models typically report task return, not order sensitivity or whether teams remain behaviorally diverse. Set-based approaches such as Deep Sets or Set Transformer enforce permutation symmetry, but symmetric outputs can still be achieved by trivial homogeneous actions rather than meaningful coordination.

The authors formalize permutation metrics—logit equivariance error, policy KL, and greedy action disagreement—and introduce collapse diagnostics: normalized action diversity, same-action fraction, and maximum action frequency. They train a centralized transformer actor-critic with PPO/MAPPO-style objectives on the PettingZoo/MPE cooperative navigation task, using 128-dimensional observation tokens, two self-attention blocks with four heads, learned agent-ID and position embeddings, and optional equivariance regularization (and in one case pairwise policy-diversity loss). Evaluations run under normal, random, and reversed agent ordering for N=3 across three seeds, plus a seed-0 N=4 stress test over a range of regularization coefficients.

For N=3, PPO-ID achieved return -34.67±4.45 and the most diverse behavior (D_div=0.649±0.047, F_same=0.187±0.108) but remained order-sensitive (random action disagreement 0.498±0.052, equivariance error 0.293±0.355). Adding weak equivariance regularization λ_eq=0.01 reduced equivariance error to 0.009±0.006 and action disagreement to 0.196±0.103, but dropped diversity to 0.425±0.053 and raised same-action fraction to 0.739±0.139. Stronger λ_eq=0.05 produced near-perfect permutation metrics (eq error 0.009±0.003, action disagreement 0.112±0.102) but lower diversity 0.378±0.043 and high same-action fraction 0.878±0.117. In the N=4 seed-0 stress test, λ_eq=0.01 gave return -28.04 but collapsed completely (D_div=0.25, F_same=1, F_max=1); λ_eq=0.0005 gave the best return among regularized runs (-43.12 vs -58.43 for PPO-ID) with eq error 0.025 and action disagreement 0.121 but same-action fraction 0.811; λ_eq=0.00025 preserved the most diversity (D_div=0.427, F_same=0.293) but had worse return (-64.99).

The experiments are limited to simulated cooperative navigation with a centralized actor that observes the joint observation; the N=4 results are a seed-0 stress test rather than a multi-seed benchmark. The anti-collapse regularizer is a simple pairwise action-distribution term and may not suit tasks where identical actions are appropriate. No real-robot evaluation, code release, or compute cost is reported.