SCOUT: Testzeit-Koordination für Offline-MARL durch zerlegte Wertegradienten

Abstract

Offline multi-agent reinforcement learning (MARL) faces a persistent trade-off. Expressive generative policies can represent multi-modal coordination in the data, but cannot distinguish high-value regions, while value-optimized policies exploit the learned Q-function but collapse the multi-modal into a single dominant mode. A single agent’s mode collapse can break joint coordination, and simultaneous drift across agents can push the joint policy into unseen regions of the action space. We propose scalable coordination via optimal unified transport (SCOUT), the first offline MARL framework to combine a generative foundation model with a learned value function through test-time action refinement. SCOUT trains two decoupled components: a flow-matching behavioral prior and a decomposed value function. At test-time, it transports behavioral samples toward high-value regions via Stein variational gradient descent. The number of transport steps controls adaptive test-time scaling, replacing a fixed regularization coefficient. Under the individual-global-max (IGM) principle, we prove a single-term KL bound on the joint soft-value gap that vanishes as transport converges, with an irreducible additive residual proportional to the IGM violation. Empirically, SCOUT achieves the best average performance across discrete and continuous offline MARL benchmarks and yields performance improvements in all offline-to-online configurations.

1 Introduction

Offline MARL is motivated by the need to learn coordinated multi-agent behaviors in safety-critical or cost-constrained domains where active data collection is infeasible or expensive, e.g., autonomous driving [1], logistics [2], and robotics [3]. In this setting, the learning objective is twofold. Agents must learn to coordinate their behavior from heterogeneous and suboptimal demonstrations, while avoiding the distributional shift that arises when the learned policy visits state-action regions unsupported by the data. Early offline MARL methods [4, 5, 6] rely on simple policy classes (i.e., Gaussian), which struggle to represent multi-modal and coordinated joint action distributions. When offline data contains a mixture of diverse coordination strategies, a unimodal policy often collapses onto a single mode and produces out-of-distribution joint actions that no agent in the data ever executed together.

To address this expressiveness gap, recent works adopt powerful generative models, including diffusion models [7, 8] and flow matching models [9, 10], as policy representations in offline MARL. Although these models improve the representational power of policy, their iterative sampling process incurs a high inference cost. MAC-Flow [10] and OM2P [9] mitigate this via one-step distillation and mean-flow generation, respectively. However, improved expressiveness and faster inference do not resolve a more fundamental limitation. Since the outcome policy is optimized against a learned Q-function, it inherently tends to collapse diverse coordination modes into a single or blurred one.

Methods that parameterize a policy and optimize it to maximize a learned Q-function must balance value improvement against staying close to the behavioral distribution. A fixed regularization coefficient controls this trade-off. If it is too large, the policy stays too close to suboptimal data; if it is too small, the policy exploits errors in the learned Q-function and produces OOD actions. Although this tension is well-known in single-agent offline RL [11, 12], the multi-agent setting introduces the following compounding difficulty. Training-time value maximization deforms each agent’s policy away from the behavioral prior, and the regularization coefficient bounds how far each marginal may drift, but cannot bound the compound effect on the joint. Even when every marginal remains within its individual support, the joint policy can place mass on action combinations that no trajectory in the data ever visited (i.e., regions where the learned Q-function is least reliable). In fact, the data may support multiple distinct joint modes of comparably high estimated value, but committing each marginal to a single greedy mode at training time discards the others before the agent has any way to tell them apart. Once new interaction data arrives, the policy can no longer use it to discriminate which mode is truly higher-value, because the alternatives have already been suppressed.

A natural alternative is to decouple two objectives, motivated by test-time training [13, 14, 15]. Instead of learning a policy that directly maximizes value, we keep the behavioral policy as a foundational prior and defer value-based refinement to test time. At deployment, we sample action candidates from this prior and then move them toward high-value regions using value gradients. The extent of this adjustment acts as adaptive control, replacing the need for a fixed regularization coefficient.

In this work, we ask a question not yet studied by prior works:

How do we avoid explicit policy parameterization for value maximization, and instead recover coordination at test time by directly transporting behavioral samples toward high-value regions?

We answer this question with the proposed solution, SCOUT (scalable coordination via optimal unified transport), an offline MARL framework that combines flow-matching behavioral priors with decomposed value gradient for test-time action refinement. Our key contributions are:

• We propose a policy-free value maximization for offline MARL that separates behavioral modeling from value improvement, replacing training-time regularization tuning with a test-time refinement budget.
• We derive decomposed value gradient transport under the IGM principle, enabling fully decentralized test-time refinement that recovers inter-agent coordination with a bounded approximation gap.
• Empirically, test-time refinement strengthens coordination without retraining, achieving the best average performance in continuous and discrete domains. Notably, unlike existing flow-based MARL, our method yields improvements in all offline-to-online RL experiments.

2 Related Works

Offline MARL. Several works have explored the data-driven sequential decision-making problem, including offline RL and MARL, which aims to extract a return-maximizing policy within the support of an offline dataset. Specifically, offline MARL requires learning coordinated policies for multiple agents. This is not trivial because it requires modeling dependencies among agents in a combinatorially large joint action space. The simplest baselines are multi-agent extensions of single-agent offline RL methods, either by introducing conservatism in the value function [11, 12] or the actor [16], or by performing Bellman updates without querying out-of-distribution actions [17]. To further promote coordination, more advanced works are often combined with additional multi-agent design choices, such as value decomposition [5, 18], policy optimization with a centralized critic [4, 6], or a centralized world model [19]. However, these methods often struggle to capture the multi-modal distributions present in multi-agent offline datasets because of their limited policy expressiveness. To address this issue, recent work has adopted more expressive generative policies, including an autoregressive model [20], diffusion models [7], and flow matching [10]. However, these methods couple value learning with the generative process itself: they either anchor the policy via BC-like losses [21, 22, 10, 23], which produces overly conservative behavior, or backpropagate value gradients through iterative sampling [24, 25], which is unstable. In contrast, the proposed solution bypasses explicit policy parameterization altogether. Instead, Stein variational gradient [26, 27, 14] fields refine flow-generated action candidates toward high-value regions at test time. This enables adaptive test-time scaling while retaining generative expressiveness.

Test-time action refinement. Our main idea in this work is based on test-time refinement (i.e., a\leftarrow a+\nabla_{a}Q(s,a)) [13], which refines actions from a base generative policy at inference time rather than during training. This paradigm has been explored along several directions in single-agent RL. Rejection sampling methods generate multiple action candidates and select the highest-valued one [28, 29]. Gradient-based approaches directly optimize sampled actions via Q-function gradients [25, 14, 30]. Residual policies instead train a separate network to modify a base policy in noise or action space [31, 32]. Complementary to these, optimal transport (OT) offers a principled framework for moving probability mass toward high-value regions [33, 34]. More recently, particle-based methods adopt an OT viewpoint to transport samples toward value-preferred regions [35]. However, these approaches remain limited to single-agent settings. The proposed solution is motivated by this direction but specifically designed to solve the multi-agent coordination problem. This avoids explicit policy parameterization and naturally handles the combinatorial joint action space.

3 Preliminaries

Problem setting. We consider cooperative MARL problems modeled as a decentralized partially observable Markov decision process (Dec-POMDP) [36], defined by the tuple \mathcal{M}=\langle\mathcal{I},\mathcal{S},\{\mathcal{O}_{i}\},\{\mathcal{A}_{i}\},\mathcal{T},\{\Omega_{i}\},\{r_{i}\},\gamma\rangle, where \mathcal{I}=\{1,2,\ldots,I\} is a finite set of agents, \mathcal{S} denotes the state space, \mathcal{O}_{i} and \mathcal{A}_{i} are the observation and action spaces of agent i, \mathcal{T}(s^{\prime}\mid s,\mathbf{a}):\mathcal{S}\times\mathcal{A}_{1}\times\cdots\times\mathcal{A}_{I}\to\Delta(\mathcal{S}) is state transition function over state s and joint action \mathbf{a}=[a_{1},\ldots,a_{I}], \Omega_{i}(o_{i}\mid s):\mathcal{S}\to\Delta(\mathcal{O}_{i}) is the observation function, r_{i}(s,\mathbf{a}):\mathcal{S}\times\prod_{i}\mathcal{A}_{i}\to\mathbb{R} is an individual reward, and \gamma\in[0,1) is the discount factor. In the offline setting, agents learn from a fixed dataset \mathcal{D}=\{\tau^{(n)}\}_{n=1}^{N}, where each trajectory \tau^{(n)} is collected by unknown behavioral policies. The behavioral policy can be empirically defined as \mu(\mathbf{a}\mid\mathbf{o})\approx p_{\mathcal{D}}(\mathbf{a}\mid\mathbf{o}) and its per-agent marginals as \mu_{i}(a_{i}\mid o_{i}). We consider the centralized training and decentralized execution (CTDE) [4, 6, 37, 10].

Offline RL methods typically aim for Q-value maximization because optimizing cumulative return at a given MDP is infeasible without additional interaction [38, 12]. To make this tractable in MARL, we decompose the global action-value function Q_{\mathrm{tot}}(\mathbf{o},\mathbf{a}) into per-agent utilities \{Q_{i}(o_{i},a_{i})\}_{i=1}^{I} under the individual-global-max (IGM) principle [39, 40] as follows:

\arg\max_{\mathbf{a}}Q_{\mathrm{tot}}(\mathbf{o},\mathbf{a})=\bigl(\arg\max_{a_{1}}Q_{1}(o_{1},a_{1}),\ldots,\arg\max_{a_{I}}Q_{I}(o_{I},a_{I})\bigr).

We adopt the additive decomposition throughout. Under this, each agent independently maximizes its own value, and the resulting joint action is optimal, i.e., \max_{\Pi}\mathbb{E}_{\mathbf{o}\sim\mathcal{D}}[Q_{\mathrm{tot}}(\mathbf{o},\Pi(\mathbf{o}))]=\sum_{i=1}^{I}\max_{\pi_{i}}\mathbb{E}_{o_{i}\sim\mathcal{D}}[Q_{i}(o_{i},\pi_{i}(o_{i}))]. However, direct maximization of a learned Q-function leads to OOD errors. Therefore, in practice, the per-agent policy extraction problem can be defined as follows:

\pi_{i}^{*}=\arg\max_{\pi_{i}}\mathbb{E}_{o_{i}\sim\mathcal{D},a_{i}\sim\pi_{i}(\cdot|o_{i})}\bigl[Q_{i}(o_{i},a_{i})\bigr],\quad\text{s.t.}\quad\mathbb{E}_{o_{i}\sim\mathcal{D}}\Bigl[D\bigl(\pi_{i}(\cdot|o_{i}),\mu_{i}(\cdot|o_{i})\bigr)\Bigr]\leq\epsilon,

where D is a divergence measure, e.g., KL-divergence or \ell_{2}-norm, and \epsilon is a threshold.

Policy extraction. The behavior-regularized objective function in (2) can be optimized by two families of solutions in general [41], differentiated by what they focus mode-seeking and mode-covering.

For mode-seeking, a representative example is a parameterized gradient [16, 42], which extracts a Gaussian policy by converting the constraint into a penalty, following DDPG+BC objective [5, 41]:

\max_{\pi_{i}}\mathbb{E}_{o_{i}\sim\mathcal{D}}\Bigl[\mathbb{E}_{a_{i}\sim\pi_{i}}\bigl[Q_{i}(o_{i},a_{i})\bigr]-\beta D(\pi_{i}(\cdot|o_{i}),\mu_{i}(\cdot|o_{i}))\Bigr].

An alternative for a mode-seeking family is to use distillation to compress the multi-step generative model into a one-step policy [43, 22, 10]. However, these methods output Gaussian policies, which may be insufficient because they collapse into a single mode and lack expressivity. For expressive generative policies such as diffusion [44] or flow models [45], the parameterized gradient requires backpropagation through multi-step iterations, which is expensive and numerically fragile [21].

A second family samples actions from \mu and reweights them by values. A closed-form solution is \pi^{*}(a|s)\propto\mu(a|s)\exp(Q(s,a)/\beta), approximated by weighted BC [46, 6, 47]:

\max_{\pi}\mathbb{E}_{o_{i}\sim\mathcal{D},a_{i}\sim\mu_{i}}\!\bigl[\exp(Q_{i}(o_{i},a_{i})/\beta)\cdot\log\pi_{i}(a_{i}\mid o_{i})\bigr].

While this has been viewed as overly conservative [48], it carries a distinct advantage for multi-agent coordination [41]. A key implication is that preserving the coordinated strategies in the dataset is often more important than aggressive mode-seeking, as independent mode selection can break inter-agent compatibility. As a theoretically near-optimal solution for this problem, best-of-N draws N candidate actions from \mu and returns the highest-valued one [49, 50, 51, 14] as follows:

a^{*}_{i}=\arg\max_{a^{k}_{i}\sim\mu_{i}(\cdot|o_{i}),k\in[N]}Q_{i}(o_{i},a^{k}_{i}).

This requires no policy training in terms of value maximization beyond behavioral policy modeling.

Flow-matching and behavioral policy modeling. To model behavioral policy in a multi-agent system, generative modeling offers a flexible framework that can capture multi-modal distributions [7, 8, 10, 9, 52]. Specifically, flow-matching can relax the tradeoff between coordination performance and computational cost in MARL. Flow-matching learns a time-dependent velocity field v_{\phi}(t,x):[0,1]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} that transports noise p_{0}=\mathcal{N}(0,I_{d}) to the data distribution p_{1} [45]. Given the conditional linear path x_{t}=(1-t)x_{0}+tx_{1}, the training objective is as follows:

\mathcal{L}_{\mathrm{FlowMatching}}(\phi)=\mathbb{E}_{x_{0}\sim p_{0},x_{1}\sim p_{1},t\sim\mathrm{Unif}([0,1])}\!\bigl[\lVert v_{\phi}(t,x_{t})-(x_{1}-x_{0})\rVert_{2}^{2}\bigr].

At inference, a sample is produced by integrating the ODE from initial noise [53, 54, 55].

Under multi-agent learning, different structural assumptions on the velocity field yield a hierarchy of policy representations [9, 10]. A joint flow policy \mu_{\phi}(\mathbf{o},\mathbf{z}) conditions the velocity field on the full joint observation \mathbf{o} and joint noise \mathbf{z}\sim\prod_{i}\mathcal{N}(0,I_{d_{i}}), capturing inter-agent dependencies, as follows:

\mathcal{L}_{\mathrm{JointFlow}}(\phi)=\mathbb{E}_{\mathbf{z}\sim p_{0},~(\mathbf{o},\mathbf{a})\sim\mathcal{D},~t\sim\text{Unif}([0,1])}\bigl[\lVert v_{\phi}(t,\mathbf{o},x_{t})-(\mathbf{a}-\mathbf{z})\rVert_{2}^{2}\bigr].

While expressive, the joint policy violates CTDE at execution time. Instead, a factorized model trains individual policy \hat{p}(\mathbf{a}\mid\mathbf{o})=\prod_{i}\hat{p}_{i}(a_{i}\mid o_{i}) [8]. This is fully decentralized but discards coordination patterns. A projected model [10, 52] bridges the two by learning a joint flow and then distilling it into the factorized one, \hat{p}_{\mathrm{proj}}=\arg\min_{\hat{p}\in\mathcal{P}}\mathcal{L}(\hat{p}\|p), retaining as much coordination as the factorized structure permits. In practice, this can be written as follows:

\mathcal{L}_{\text{ProjectedFlow}}(\mathbf{w})=\mathbb{E}_{\mathbf{o}\sim\mathcal{D},~\mathbf{z}\sim\mathbf{p}_{0}}\Bigg[\sum_{i=1}^{I}\left|\left|\mu_{w_{i}}(o_{i},z_{i})-[\mu_{\phi}(\mathbf{o},\mathbf{z})]_{i}\right|\right|^{2}_{2}\Bigg].

4 Scalable Coordination via Optimal Unified Transport (SCOUT)

As motivated in Section 1, our goal is to develop an offline MARL algorithm that can extract Q-maximizing multi-agent policies without explicit policy parameterization for value maximization. Following prior works [14, 56, 57], we consider the MaxEnt surrogate of (2), \mathbb{E}_{a\sim\pi}[Q(s,a)]+\alpha\mathcal{H}(\pi(\cdot|s)), whose optimal policy takes the Boltzmann form \pi^{*}(a|s)\propto\exp(Q(s,a)/\alpha).

Rather than fitting an explicit actor to approximate \pi^{*}, we construct a transport map that moves samples from \mu toward \pi^{*} via variational gradient descent, using \nabla_{a}Q as the driving force. Specifically, given a set of action candidates sampled from a flow-matching behavioral policy, we iteratively refine them at test time via the simplest possible update as a\leftarrow a+\eta\nabla_{a}Q(o,a). The resulting refined actions constitute the implicit policy output, with the number of gradient steps L serving as a tunable knob that controls coordination strength at deployment without retraining.

Why do we need an implicit policy? This direction offers several appealing properties. First, because the initial actions are drawn from the behavioral policy, the implicit policy refines the behavioral prior rather than replacing it. Then, the transport budget provides an adaptive test-time scaling. When L=0, it recovers the best sampling from the behavioral policy, while increasing L progressively pushes actions toward Q-maximizing regions. Furthermore, the gradient-based transport can discover actions beyond the reference distribution when the Q-function generalizes well.

4.1 Why is this hard in multi-agent settings?

The above idea and problem hold straightforwardly in the single-agent setting [14, 48], where the transport operates on a single action space \mathcal{A} guided by a single Q-function. However, extending this to multi-agent settings is non-trivial. Under the CTDE paradigm, each agent i can only access its local observation o_{i} at test time. This means the action refinement must run independently for each agent, using only its local Q-function Q_{i}(o_{i},a_{i}) and its own action particles. The refined joint action (a_{1}^{*},\ldots,a_{I}^{*}) must be globally coherent. Namely, each agent’s refined action should be coordinated with those of other agents. This tension between decentralized execution and coordinated behavior has been a long-standing challenge in MARL [39, 40].

The most common solution is the IGM principle [39] in (1), which ensures independent per-agent maximization to align with joint maximization. However, several practical questions remain. First, IGM guarantees alignment at the greedy argmax, but the gradient-based transport involves iterative updates that modify the particle distribution across multiple steps. It is unclear whether the per-agent transport dynamics remain aligned with the joint optimum throughout this process, especially when the critic must evaluate actions that are progressively shifted away from the behavioral support during training. Second, the individual behavioral prior might discard the inter-agent correlations present in the offline data. Even if the transport dynamics are correctly factorized, the initial particles may lack the coordination patterns necessary for effective joint behavior.

4.2 Proposed solution

In this subsection, we now present the full SCOUT framework for multi-agent test-time coordination.

Flow-matching behavioral prior. We learn a per-agent velocity field via flow matching [45].

\mathcal{L}_{\text{Flow-BC}}(w_{i})=\mathbb{E}_{x_{i}^{0}\sim p_{0},~(o_{i},a_{i})\sim\mathcal{D},~t\sim\text{Unif}([0,1])}\left[\|v_{w_{i}}(t,o_{i},x_{i}^{t})-(a_{i}-x^{0}_{i})\|_{2}^{2}\right].

This is a generative model of the behavioral policy \mu_{i}(\cdot|o_{i}) that serves as the source distribution for the transport. At inference time, action samples are produced by Euler integration over K steps.

Decomposed value function. The value function has to satisfy that it must decompose under IGM so that per-agent gradient transport is globally consistent. We achieve this by training \{Q_{\theta_{i}}\}_{i=1}^{I} whose TD targets are computed through an additive decomposition as Q_{\mathrm{tot}}(\mathbf{o},\mathbf{a})=\sum_{i=1}^{I}Q_{\theta_{i}}(o_{i},a_{i}). This is the simplest decomposition satisfying IGM. The critic is trained via TD learning with targets y=r+\gamma Q_{\mathrm{tot}}(\mathbf{o}^{\prime},\mathbf{a}^{\prime}), where the next-state actions a_{i}^{\prime} are obtained by policy. This global credit signal is what enables per-agent gradient transport to produce coordinated behavior at test time.

Action refinement via decomposed gradient transport. Given the pre-trained behavioral prior and the decomposed value function, we now introduce the test-time refinement as the core of SCOUT. Each agent i begins by drawing P action particles from its behavioral flow policy, a_{i,j}^{(0)}=\mu_{w_{i}}(o_{i},z_{i,j}) with z_{i,j}\sim\mathcal{N}(0,I) for j=\{1,\ldots,P\}. These particles are then iteratively refined for L steps, motivated by Stein variational gradient descent (SVGD) as transport in Stein geometry [26, 27] as follows:

\hat{\phi}_{i}^{*}(a)=\frac{1}{P}\sum_{j=1}^{P}\left[\frac{1}{\alpha}k(a_{i,j}^{(\ell)},a)\nabla_{a_{i,j}^{(\ell)}}Q_{i}(o_{i},a_{i,j}^{(\ell)})+\nabla_{a_{i,j}^{(\ell)}}k(a_{i,j}^{(\ell)},a)\right]~~\text{s.t.}~~a_{i,j}^{(\ell+1)}=a_{i,j}^{(\ell)}+\epsilon\hat{\phi}_{i}^{\mathrm{clip}}(a_{i,j}^{(\ell)}),

where k(x,y)=\exp(-\gamma\|x-y\|^{2}) is an RBF kernel with adaptive median bandwidth. The first term in (8) is a kernel-smoothed value gradient that drives particles toward high-Q_{i} regions; the second term is a repulsive force that prevents particle collapse and preserves multi-modality.

For discrete action spaces, the refinement operates in the logit space. The flow policy produces logit vectors \ell_{i,j}\in\mathbb{R}^{|\mathcal{A}_{i}|}, and the value gradient is computed as \nabla_{\ell}\sum_{a}\mathrm{softmax}(\ell)_{a}\cdot Q_{i}(o_{i},a) via automatic differentiation through the softmax-weighted Q-value.

4.3 Theoretical analysis

We now provide theoretical guarantees for the SCOUT framework.

Soft value gap under approximate IGM. The learned decomposition is at best an approximation of the true joint value, so we adopt the following residual assumption.

Assumption 4.1.

The global Q-function decomposes as Q_{\mathrm{tot}}(\mathbf{o},\mathbf{a})=\sum_{i=1}^{I}Q_{i}(o_{i},a_{i})+\Delta(\mathbf{o},\mathbf{a}), where the residual interaction term satisfies \|\Delta\|_{\infty}\leq\delta.

When \delta=0, exact additive IGM holds and decentralized transport recovers the global Boltzmann optimum. The following result quantifies the soft-value gap when \delta>0.

Theorem 4.2 (Soft value gap under approximate IGM).

Let \tilde{\pi}^{L}=\prod_{i}\tilde{\pi}_{i}^{L} be the joint policy after L steps of independent per-agent transport, each targeting \pi_{i}^{*}\propto\exp(Q_{i}/\alpha). Let \pi_{\mathrm{joint}}^{*}\propto\exp(Q_{\mathrm{tot}}/\alpha) be the true joint Boltzmann target and \mathcal{J}_{\mathrm{soft}}(\pi):=\mathbb{E}_{\pi}[Q_{\mathrm{tot}}]+\alpha\mathcal{H}(\pi). Then

0\leq\mathcal{J}_{\mathrm{soft}}(\pi_{\mathrm{joint}}^{*})-\mathcal{J}_{\mathrm{soft}}(\tilde{\pi}^{L})\leq\alpha\underbrace{\mathrm{KL}\!\left(\tilde{\pi}^{L}\Big\|\textstyle\prod_{i}\pi_{i}^{*}\right)}_{\mathrm{transport~error}}+\underbrace{4\delta}_{\mathrm{IGM~violation}}.

The transport error is a single \mathrm{KL} term that vanishes as L\to\infty under idealized population SVGD with exact Q-functions and suitable step sizes. The SVGD performs gradient descent on \mathrm{KL}(\cdot\|\prod_{i}\pi_{i}^{*}) in the kernel-induced Stein geometry [26, 27]. The 4\delta term decomposes as 2\delta from the per-agent transport target \pi_{i}^{*} and 2\delta from the gap between the factorized and joint Boltzmann optima; both arise from the IGM decomposition itself and apply equally to all decomposed-Q methods, including MAC-Flow’s distillation [10]. What distinguishes SCOUT is that the transport error can be driven down by increasing L at test time without retraining, whereas distillation-based methods fix their approximation quality at training time.

Decomposed transport approximates joint best-of-P. Even though each agent’s gradient update is entirely local, we show that the combination of decomposed transport and per-agent selection tracks the joint best-of-P outcome up to the irreducible IGM gap.

Proposition 4.3 (Decomposed transport under IGM).

Consider P joint candidates with independently initialized particles (a_{1,j}^{(0)},\ldots,a_{I,j}^{(0)})\sim\prod_{i}\mu_{i}. After L steps of per-agent gradient transport: (i) when \Delta=0, per-agent selection a_{i}^{*}=\arg\max_{j}Q_{i}(o_{i},a_{i,j}^{(L)}) yields \mathrm{Cov}(a_{i}^{*},a_{i^{\prime}}^{*})=0 and is jointly optimal; (ii) when \|\Delta\|_{\infty}\leq\delta, per-agent selection approximates joint best-of-P within the irreducible IGM gap as Q_{\mathrm{tot}}(a_{1}^{*},\ldots,a_{I}^{*})\geq\max_{j\in[P]}Q_{\mathrm{tot}}\bigl(a_{1,j}^{(L)},\ldots,a_{I,j}^{(L)}\bigr)-2\delta.

Herein, (i) follows since each agent’s particles and selection depend only on its own Q_{i} and initial noise. Then, (ii) follows from |Q_{\mathrm{tot}}-\sum_{i}Q_{i}|\leq\delta uniformly; full proofs are in Appendix A.

Implicit coordination via TD targets. Proposition 4.3 shows that decomposed transport is safe, i.e., it does not lose more than the irreducible 2\delta relative to a hypothetical centralized refiner. The complementary question is whether the per-agent gradient \nabla_{a_{i}}Q_{i} itself encodes inter-agent dependencies. Recall the TD target y=r+\gamma\sum_{j}Q_{j}(o^{\prime}_{j},a^{\prime}_{j}). At the TD fixed point, Q_{i} inherits sensitivity to other agents’ actions through the joint distribution of a^{\prime}_{j} used to form the target. Concretely, fitted Q_{i} can be interpreted as approximating \nabla_{a_{i}}Q_{i}(o_{i},a_{i})\approx\mathbb{E}_{a_{-i}\sim\rho}[\nabla_{a_{i}}Q_{\mathrm{tot}}(\mathbf{o},\mathbf{a})] along the training distribution \rho. Under additive decomposition, \partial Q_{\mathrm{tot}}/\partial Q_{i}=1, so \nabla_{a_{i}}Q_{i} preserves the sign of the joint gradient \nabla_{a_{i}}Q_{\mathrm{tot}}, and per-agent transport along \nabla_{a_{i}}Q_{i} is consistent with joint Q-improvement. Coordination is thus inherited from training, not synthesized at test time, and decomposed transport amplifies it through repeated kernel-smoothed updates over the particle set.

4.4 Algorithm summary

We summarize the SCOUT framework in Pseudocode 1. Training alternates between two decoupled objectives: (i) the flow-matching loss \mathcal{L}_{\mathrm{BC}} fits the individual behavioral prior with ODE in Pseudocode 2, and the TD loss \mathcal{L}_{\mathrm{critic}} learns decomposed Q functions whose targets are computed through the additive decomposition Q_{\mathrm{tot}}=\sum_{i}Q_{i} using actions refined by L_{\mathrm{train}} transport steps in Pseudocode 3.

At test time, each agent independently generates P behavioral candidates via ODE integration, refines them for L_{\mathrm{test}} steps of decomposed value gradient flow, and selects the highest-Q_{i} particle. The transport budget (L_{\mathrm{test}},\epsilon) need not match training. In particular, L_{\mathrm{test}}=0 recovers best-of-P sampling, L_{\mathrm{test}}=L_{\mathrm{train}} matches training conditions, and L_{\mathrm{test}}>L_{\mathrm{train}} enables additional test-time scaling when the Q-function generalizes beyond the data support.

4.5 Didactic example: Why do we need test-time value guidance, not training time?

To illustrate the trade-off between training-time and test-time value incorporation, we design a two-agent coordination game (Figure 2 (a)). Each agent selects a scalar action a_{i}\in[-2,2], and the joint reward has two anti-correlated Gaussian modes: an optimal mode at (-1,1) with reward 4 and a sub-optimal mode at (1,-1) with reward 2. The offline dataset consists of 400 transitions, with only 10\% near each peak. Both methods share the same VDN critic Q_{\mathrm{tot}}=Q_{1}(a_{1})+Q_{2}(a_{2}).

The key difference is how each method incorporates the Q-function. MAC-Flow jointly optimizes distillation and Q-maximization, forcing the projected one-step policy to balance behavioral cloning and value improvement during training. In this toy example, this training-time compromise anchors each agent’s output to the broad data distribution, while the decomposed Q-gradient alone cannot represent the anti-correlated structure, causing the policy to collapse near the origin (Figure 2 (c)). In contrast, SCOUT keeps the behavioral flow unchanged and applies value guidance only at test time. This preserves the multi-modal prior, and five refinement steps are sufficient to move candidates toward the optimal mode (Figure 2 (b)). In this setting, the selected per-agent actions align with the same anti-correlated mode because the additive utilities are trained from coordinated demonstrations.

5 Experiments

The subsection presents the experimental setup and answers the following research questions.

RQ1. How good is SCOUT over continuous and discrete control domains of MARL testbeds?

RQ2. How does SCOUT improve with additional interaction data?

RQ3. How sensitive is SCOUT to the refinement step parameter L_{\text{train}} and L_{\text{test}}?

5.1 Experimental setup

We evaluate SCOUT on two cooperative MARL testbeds with discrete and continuous action spaces, spanning 7 scenarios and 24 offline datasets. We first introduce the selected testbeds and baselines.

SMACv1. Agents control a team of units in StarCraft II micromanagement tasks against a built-in AI opponent. We test on four maps, including 3m, 8m, 2s3z, and 5m_vs_6m. Each map is paired with three offline dataset quality levels (Good, Medium, Poor) from the off-the-grid benchmark [58], resulting in 12 configurations. The quality levels differ in the proportion of near-optimal trajectories, with Poor containing mostly suboptimal behavior.

MA-MuJoCo. In this testbed, a single robot is factorized into multiple agents, with each agent responsible for a specific subset of the robot’s joints. This setup requires agents to learn coordinated behaviors to accomplish shared tasks. We source the corresponding offline datasets from [6].

Baseline. We organize baselines into three families by their policy type. For the Gaussian policy, BC, MATD3BC [16] , MABCQ [11], MACQL [12], and ICQ [4] extend single-agent methods to the multi-agent setting. For the diffusion policy, MADiff [7] models the joint policy as a diffusion process over the full action space, but we follow a distributed version (MADiff-D) that masks other agents’ information. DoF [8] factorizes the diffusion into per-agent components to satisfy decentralized execution. For the flow policy, Flow BC trains a per-agent flow-matching behavioral prior, and MAC-Flow [10] learns a joint flow and distills it into one-step projected policies with Q-maximization.

5.2 Experimental results

RQ1 (Offline RL). Tables 1 and 2 report offline RL performance on SMACv1 and MA-MuJoCo. SCOUT achieves the highest average score on both benchmarks, outperforming all baselines across policy classes. The gain is consistent, i.e., SCOUT matches or exceeds the best baseline on the majority of individual configurations. Notably, the advantage over MAC-Flow, which shares the same flow-matching backbone, isolates the benefit of test-time value transport over training-time distillation.

RQ2 (Offline-to-online RL). Figure 3 evaluates how well each method transitions from offline pre-training to online fine-tuning. SCOUT outperforms MAC-Flow across all four scenarios. After the online transition, SCOUT continues to improve steadily. Because the behavioral prior is never deformed by a training-time value loss, it provides a stable foundation that online fine-tuning can build upon rather than correct. The advantage is most pronounced on Poor-quality datasets. Here, MAC-Flow’s offline policy is anchored to low-quality demonstrations by its distillation objective, and online fine-tuning struggles to overcome this initialization. SCOUT, by contrast, treats the offline prior as a sampling distribution rather than a fixed policy, so the online phase simply adjusts the transport budget to exploit newly available data without inheriting the limitations of the offline dataset.

RQ3 (Hyperparameter sensitivity). Figure 4 varies the refinement steps for training and test across four SMACv1-Medium scenarios. First, applying even a small number of refinement steps at test time improves over the no-refinement baseline (L_{\text{test}}=0), confirming that decomposed value gradient transport adds meaningful coordination beyond best-of-P selection alone. Second, the optimal L_{\text{test}} varies across scenarios. This is expected because each scenario and dataset quality presents a different Q-function landscape; the number of steps needed to reach high-value regions depends on how far the behavioral prior is from those regions and how smooth the Q-surface is. In practice, L_{\text{test}} can be selected via a small validation sweep at deployment time without retraining, which is precisely the adaptivity that training-time methods cannot offer.

6 Conclusion

In this paper, we propose SCOUT, the first test-time policy refinement framework for offline MARL that decouples behavioral modeling from value maximization. While prior generative policies often compromise expressiveness by integrating value guidance during training, SCOUT preserves the multi-modal behavioral prior and defers value-driven refinement strictly to test-time inference via decomposed Stein variational gradient transport. We provide a single-term KL bound on the joint soft-value gap under approximate IGM and constructively explain how additive TD targets transfer inter-agent dependencies into per-agent gradients.

Limitation. While decoupling the behavioral prior from value maximization is effective across a range of tasks, the refinement hyperparameters exhibit some task- and dataset-dependent sensitivity. Therefore, a principled adaptive stopping criterion remains an open direction for future work.

Acknowledgements

This work was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. NSF 2340651, NSF 2402650, and ARO W911NF-24-1-0193. This research used the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing computational resources.

References

1. [1] S. Shalev-Shwartz, S. Shammah, and A. Shashua (2016) Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295.
2. [2] D. Tamagawa, E. Taniguchi, and T. Yamada (2010) Evaluating city logistics measures using a multi-agent model. Procedia-Social and Behavioral Sciences 2 (3), pp. 6002–6012.
3. [3] J. Ota (2006) Multi-agent robot systems as distributed autonomous systems. Advanced engineering informatics 20 (1), pp. 59–70.
4. [4] Y. Yang, X. Ma, C. Li, Z. Zheng, Q. Zhang, G. Huang, J. Yang, and Q. Zhao (2021) Believe what you see: implicit constraint approach for offline multi-agent reinforcement learning. In NeurIPS,
5. [5] L. Pan, L. Huang, T. Ma, and H. Xu (2022) Plan better amid conservatism: offline multi-agent reinforcement learning with actor rectification. In ICML,
6. [6] X. Wang, H. Xu, Y. Zheng, and X. Zhan (2023) Offline multi-agent reinforcement learning with implicit global-to-local value regularization. In NeurIPS,
7. [7] Z. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang (2024) MADiff: offline multi-agent learning with diffusion models. In NeurIPS,
8. [8] C. Li, Z. Deng, C. Lin, W. Chen, Y. Fu, W. Liu, C. Wen, C. Wang, and S. Shen (2025) DoF: a diffusion factorization framework for offline multi-agent reinforcement learning. In ICLR,
9. [9] Z. Li, X. Wang, H. Zhong, and L. Huang (2026) OM2P: offline multi-agent mean-flow policy. In AAMAS,
10. [10] D. Lee, D. Lee, and A. Zhang (2026) Multi-agent coordination via flow matching. In ICLR,
11. [11] S. Fujimoto, D. Meger, and D. Precup (2019) Off-policy deep reinforcement learning without exploration. In ICML,
12. [12] A. Kumar, A. Zhou, G. Tucker, and S. Levine (2020) Conservative Q-learning for offline reinforcement learning. In NeurIPS,
13. [13] S. Park, K. Frans, S. Levine, and A. Kumar (2024) Is value learning really the main bottleneck in offline RL?. In NeurIPS,
14. [14] H. Xu, K. Hu, S. Sojoudi, and A. Zhang (2026) Reinforcement learning via value gradient flow. In ICLR,
15. [15] D. Lee, D. Lee, T. Kwack, W. Choi, and H. Woo (2025) Policy compatible skill incremental learning via lazy learning interface. In NeurIPS,
16. [16] S. Fujimoto and S. S. Gu (2021) A minimalist approach to offline reinforcement learning. In NeurIPS,
17. [17] I. Kostrikov, A. Nair, and S. Levine (2022) Offline reinforcement learning with implicit Q-learning. In ICLR,
18. [18] Z. Liu, Q. Lin, C. Yu, X. Wu, Y. Liang, D. Li, and X. Ding (2025) Offline multi-agent reinforcement learning via in-sample sequential policy optimization. In AAAI,
19. [19] P. Barde, J. Foerster, D. Nowrouzezahrai, and A. Zhang (2024) A model-based solution to the offline multi-agent reinforcement learning coordination problem. In AAMAS,
20. [20] L. Meng, M. Wen, C. Le, X. Li, D. Xing, W. Zhang, Y. Wen, H. Zhang, J. Wang, Y. Yang, et al. (2023) Offline pre-trained multi-agent decision transformer. Machine Intelligence Research 20 (2), pp. 233–248.
21. [21] Z. Wang, J. J. Hunt, and M. Zhou (2023) Diffusion policies as an expressive policy class for offline reinforcement learning. In ICLR,
22. [22] S. Park, Q. Li, and S. Levine (2025) Flow Q-learning. In ICML,
23. [23] S. Lee, D. Kim, E. Balachandar, Z. S. Mustafaoglu, and K. Pingali (2026) Towards efficient and expressive offline RL via flow-anchored noise-conditioned Q-learning. In ICML,
24. [24] A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz (2025) Diffusion policy policy optimization. In ICLR,
25. [25] M. S. Mark, T. Gao, G. G. Sampaio, M. K. Srirama, A. Sharma, C. Finn, and A. Kumar (2024) Policy agnostic RL: offline RL and online RL fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685.
26. [26] Q. Liu and D. Wang (2016) Stein variational gradient descent: a general purpose Bayesian inference algorithm. In NeurIPS,
27. [27] Q. Liu (2017) Stein variational gradient descent as gradient flow. In NeurIPS,
28. [28] P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine (2023) IDQL: implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573.
29. [29] Q. Li, Z. Zhou, and S. Levine (2025) Reinforcement learning with action chunking. In NeurIPS,
30. [30] Q. Li and S. Levine (2026) Q-learning with adjoint matching. In ICLR,
31. [31] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine (2021) Parrot: data-driven behavioral priors for reinforcement learning. In ICLR,
32. [32] A. Wagenmaker, M. Nakamoto, Y. Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine (2025) Steering your diffusion policy with latent space reinforcement learning. In CoRL,
33. [33] W. Dabney, M. Rowland, M. Bellemare, and R. Munos (2018) Distributional reinforcement learning with quantile regression. In AAAI,
34. [34] R. Dadashi, L. Hussenot, M. Geist, and O. Pietquin (2021) Primal Wasserstein imitation learning. In ICLR,
35. [35] A. Asadulaev, R. Korst, A. Korotin, V. Egiazarian, A. Filchenkov, and E. Burnaev (2024) Rethinking optimal transport in offline reinforcement learning. In NeurIPS,
36. [36] D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein (2002) The complexity of decentralized control of Markov decision processes. Mathematics of operations research 27 (4), pp. 819–840.
37. [37] Y. Li, G. Xie, and Z. Lu (2025) Revisiting cooperative off-policy multi-agent reinforcement learning. In ICML,
38. [38] S. Levine, A. Kumar, G. Tucker, and J. Fu (2020) Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643.
39. [39] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi (2019) QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML,
40. [40] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson (2020) Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21 (178), pp. 1–51.
41. [41] D. Lee, D. Lee, and A. Zhang (2026) A recipe for stable offline multi-agent reinforcement learning. arXiv preprint arXiv:2603.08399.
42. [42] D. Tarasov, V. Kurenkov, A. Nikulin, and S. Kolesnikov (2023) Revisiting the minimalist approach to offline reinforcement learning. In NeurIPS,
43. [43] Z. Ding and C. Jin (2024) Consistency models as a rich and efficient policy class for reinforcement learning. In ICLR,
44. [44] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In NeurIPS,
45. [45] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In ICLR,
46. [46] X. B. Peng, A. Kumar, G. Zhang, and S. Levine (2019) Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177.
47. [47] H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan (2023) Offline RL with no OOD actions: in-sample learning via implicit value regularization. In ICLR,
48. [48] F. Wu, W. Xuan, X. Lu, M. Liu, Y. Dong, Z. Harchaoui, and Y. Choi (2025) The invisible leash: why RLVR may or may not escape its origin. arXiv preprint arXiv:2507.14843.
49. [49] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu (2021) EMaQ: expected-max Q-learning operator for simple yet effective offline and online RL. In ICML,
50. [50] A. Beirami, A. Agarwal, J. Berant, A. D’Amour, J. Eisenstein, C. Nagpal, and A. T. Suresh (2025) Theoretical guarantees on the best-of-n alignment policy. In ICML,
51. [51] J. Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami (2024) Asymptotics of language model alignment. In ISIT,
52. [52] D. Huk, O. Hamelijnck, D. Demiris, and T. Damoulas (2026) MAGIC: multi-agent generative intention coordination. In ICLR Workshop,
53. [53] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan (2021) Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research 22 (57), pp. 1–64.
54. [54] J. Wildberger, M. Dax, S. Buchholz, S. Green, J. H. Macke, and B. Schölkopf (2023) Flow matching for scalable simulation-based inference. In NeurIPS,
55. [55] Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Chen, D. Lopez-Paz, H. Ben-Hamu, and I. Gat (2024) Flow matching guide and code. arXiv preprint arXiv:2412.06264.
56. [56] B. Eysenbach and S. Levine (2022) Maximum entropy RL (provably) solves some robust RL problems. In ICLR,
57. [57] M. Bloem and N. Bambos (2014) Infinite time horizon maximum causal entropy inverse reinforcement learning. In CDC,
58. [58] C. Formanek, A. Jeewa, J. Shock, and A. Pretorius (2023) Off-the-grid MARL: datasets and baselines for offline multi-agent reinforcement learning. In AAMAS,

Appendix A Proofs

In this section, we provide complete proofs for the theoretical results in Section 4.3. Throughout, we use the following standing notation. Let \mathcal{O}=\prod_{i=1}^{I}\mathcal{O}_{i} and \mathcal{A}=\prod_{i=1}^{I}\mathcal{A}_{i} denote the joint observation and action spaces. Let \mu_{i}(\cdot\mid o_{i}) denote the per-agent behavioral prior, and let \pi_{i}^{*}(a_{i}\mid o_{i})\propto\exp(Q_{i}(o_{i},a_{i})/\alpha) denote the per-agent factorized Boltzmann target. The joint Boltzmann target is defined as follows:

\pi_{\text{joint}}^{*}(a\mid o)\propto\exp\left(\frac{Q_{\text{tot}}(o,a)}{\alpha}\right).

We denote by \tilde{\pi}^{L}=\prod_{i=1}^{I}\tilde{\pi}_{i}^{L} the joint policy obtained after L steps of independent per-agent transport in (8).

A.1 Proof of Theorem 4.2

Proof.

Let \pi_{\text{fact}}^{*}:=\prod_{i}\pi_{i}^{*} and \mathcal{J}_{\text{add}}(\pi):=\mathbb{E}_{\pi}[\sum_{i}Q_{i}]+\alpha\mathcal{H}(\pi). The product Boltzmann distribution \pi_{\text{fact}}^{*} is the unique maximizer of \mathcal{J}_{\text{add}} on the joint simplex (it is the Gibbs distribution for the additive utility), and the standard variational identity yields the gap-as-KL relation:

\mathcal{J}_{\text{add}}(\pi_{\text{fact}}^{*})-\mathcal{J}_{\text{add}}(\pi)\;=\;\alpha\,\mathrm{KL}\!\bigl(\pi\,\|\,\pi_{\text{fact}}^{*}\bigr)\quad\text{for any joint }\pi.

Under Assumption 4.1, |\mathcal{J}_{\text{soft}}(\pi)-\mathcal{J}_{\text{add}}(\pi)|=|\mathbb{E}_{\pi}[\Delta]|\leq\delta for any \pi.

Combining (10) with the \delta-shift between \mathcal{J}_{\text{soft}} and \mathcal{J}_{\text{add}}:

\displaystyle\mathcal{J}_{\text{soft}}(\pi_{\text{fact}}^{*})-\mathcal{J}_{\text{soft}}(\tilde{\pi}^{L}) \\ \displaystyle\leq\bigl[\mathcal{J}_{\text{add}}(\pi_{\text{fact}}^{*})+\delta\bigr]-\bigl[\mathcal{J}_{\text{add}}(\tilde{\pi}^{L})-\delta\bigr] \\ \displaystyle=\alpha\,\mathrm{KL}\!\bigl(\tilde{\pi}^{L}\,\|\,\pi_{\text{fact}}^{*}\bigr)+2\delta.

Since \pi_{\text{joint}}^{*} maximizes \mathcal{J}_{\text{soft}}, \mathcal{J}_{\text{soft}}(\pi_{\text{joint}}^{*})-\mathcal{J}_{\text{soft}}(\pi_{\text{fact}}^{*})\geq 0. For the upper bound, since \pi_{\text{fact}}^{*} maximizes \mathcal{J}_{\text{add}}, \mathcal{J}_{\text{add}}(\pi_{\text{fact}}^{*})\geq\mathcal{J}_{\text{add}}(\pi_{\text{joint}}^{*}), so

\mathcal{J}_{\text{soft}}(\pi_{\text{joint}}^{*})-\mathcal{J}_{\text{soft}}(\pi_{\text{fact}}^{*})\leq\bigl[\mathcal{J}_{\text{add}}(\pi_{\text{joint}}^{*})+\delta\bigr]-\bigl[\mathcal{J}_{\text{add}}(\pi_{\text{fact}}^{*})-\delta\bigr]\leq 2\delta.

Adding (11) and (12):

\mathcal{J}_{\text{soft}}(\pi_{\text{joint}}^{*})-\mathcal{J}_{\text{soft}}(\tilde{\pi}^{L})\;\leq\;\alpha\,\mathrm{KL}\!\bigl(\tilde{\pi}^{L}\,\|\,\pi_{\text{fact}}^{*}\bigr)+4\delta.

The lower bound 0 follows directly from the \mathcal{J}_{\text{soft}}-optimality of \pi_{\text{joint}}^{*}. ∎

A.2 Proof of Proposition 4.3

Proof.

Mathematical derivations are divided into two parts of Proposition 4.3. For (i), the per-agent transport for agent i as follows:

a_{i,j}^{(\ell+1)}=a_{i,j}^{(\ell)}+\epsilon\,\mathrm{clip}_{c}\bigl(\hat{\phi}_{i}^{*}\bigl(a_{i,j}^{(\ell)};\{a_{i,k}^{(\ell)}\}_{k=1}^{P},Q_{i}(o_{i},\cdot)\bigr)\bigr),

depends only on agent-i particles and Q_{i}. Initialization is independent across agents and no cross-agent information enters any update, so independence is preserved:

\bigl(a_{i,j}^{(L)}\bigr)_{j=1}^{P}\perp\!\!\!\perp\bigl(a_{i^{\prime},j}^{(L)}\bigr)_{j=1}^{P}\quad\text{for }i\neq i^{\prime}.

Per-agent selection j_{i}^{*}=\arg\max_{j}Q_{i}(o_{i},a_{i,j}^{(L)}) depends only on agent-i particles, so a_{i}^{*}=a_{i,j_{i}^{*}}^{(L)} are functions of independent sets, giving \mathrm{Cov}(a_{i}^{*},a_{i^{\prime}}^{*})=0. When \Delta=0, Q_{\mathrm{tot}}=\sum_{i}Q_{i}, and per-agent argmax is jointly optimal by the IGM identity (1).

Next, for (ii), let j^{*}\in[P] be any joint candidate index. By per-agent optimality, \sum_{i}Q_{i}(o_{i},a_{i,j_{i}^{*}}^{(L)})\geq\sum_{i}Q_{i}(o_{i},a_{i,j^{*}}^{(L)}). Using Q_{\mathrm{tot}}=\sum_{i}Q_{i}+\Delta and \|\Delta\|_{\infty}\leq\delta:

\displaystyle Q_{\mathrm{tot}}\bigl(a_{1,j_{1}^{*}}^{(L)},\ldots,a_{I,j_{I}^{*}}^{(L)}\bigr) \\ \displaystyle=\textstyle\sum_{i}Q_{i}\bigl(a_{i,j_{i}^{*}}^{(L)}\bigr)+\Delta\bigl(a_{1,j_{1}^{*}}^{(L)},\ldots,a_{I,j_{I}^{*}}^{(L)}\bigr) \\ \displaystyle\geq\textstyle\sum_{i}Q_{i}\bigl(a_{i,j^{*}}^{(L)}\bigr)-\delta \\ \displaystyle\geq Q_{\mathrm{tot}}\bigl(a_{1,j^{*}}^{(L)},\ldots,a_{I,j^{*}}^{(L)}\bigr)-2\delta.

Maximizing over j^{*}\in[P] on the right-hand side yields the claim. ∎

Appendix B Experimental Details

B.1 Computational resources

All experiments were conducted on a single server equipped with an Intel Xeon E5-2650 v3 CPU and an NVIDIA A40 GPU. The software stack consists of CUDA 12.2, cuDNN 8.9.7, Python 3.9, and JAX 0.4.30.

B.2 Implementation details

Flow BC prior. We instantiate the foundation as a conditional flow-matching model that learns the behavior distribution of the offline dataset. Given an observation and a Gaussian noise sample, the actor parameterizes a velocity field that transports x_{0} to a data action x_{1} along the linear interpolation, trained by minimizing the flow-matching loss. At inference time, candidate actions are produced by a fixed-step Euler integration of the learned ODE, so the prior remains an explicit, multimodal generative model of the dataset rather than a unimodal collapse. Because the resulting samples are biased toward the modeled behavioral support, they encourage the agent to remain near in-distribution behaviors, while still preserving the diverse coordination patterns present in the offline corpus.

Sinusoidal time embedding. The flow conditioning variable t\in[0,1] carries information about how close the current sample x_{t} is to data; feeding it as a raw scalar concatenated to the network input yields poor expressivity, since a single-channel signal interacts weakly with high-dimensional weights and forces the actor to learn very different behaviors at neighboring t values from a one-dimensional cue. We therefore lift t into a Fourier feature space \varphi(t)=\bigl[\sin(2^{k}\pi t),\cos(2^{k}\pi t)\bigr]_{k=0}^{K-1}\in\mathbb{R}^{2K} with K=8, echoing the positional encodings widely used in diffusion and neural-field models. The exponentially spaced frequencies provide a multi-resolution decomposition of the time axis: low-frequency components encode the coarse phase of the integration trajectory while high-frequency components let the velocity field model sharp transitions near t=0 and t=1.

Online fine-tuning. After offline gradient steps, we roll out the policy and store the resulting trajectories in an online replay buffer. After an additional warm-up step with no online updates, each gradient step samples a 50:50 mixture of offline data and newly collected online data.

B.3 Hyperparameters

We select L_{\text{train}} and L_{\text{test}} from a small grid using a held-out validation split of the offline dataset; no environment interaction is used for selection. Baselines use their officially reported hyperparameters with comparable tuning budgets.

B.4 Baselines

MADiff-D [7]. We utilize the official open-source implementation, specifically adopting the decentralized variant, MADiff-D, to align with our Dec-POMDP formulation. The model trains a conditional diffusion policy on centralized joint demonstrations but executes decentrally; each agent samples actions via reverse diffusion based solely on local observations while predicting teammates’ trajectories. We retain the original architecture and hyperparameters, applying a Gaussian DDPM scheduler with 200 denoising steps across all environments.

DoF [8]. We use the official implementation of the DoF-Trajectory agent equipped with the W-concat factorization. All training and deployment configurations strictly follow the original paper. For action sampling, we similarly employ a Gaussian DDPM scheduler with 200 denoising steps for all environments.

MAC-Flow [10]. We utilize the official open-source implementation. MAC-Flow models multi-modal data distributions via flow matching, utilizes a learned Q-function for guidance, and distills the guided flow into one-step decentralized policies. We maintain the original architecture and hyperparameters for all experiments.

Appendix C Additional Results

Offline-to-online RL. Figure 5 extends the offline-to-online evaluation to the continuous control domain. Across all six configurations, SCOUT demonstrates consistent improvement after the online transition at normalized step 0.5. These empirical results suggest that the undeformed flow prior allows the transport budget to flexibly exploit newly collected interaction data. Together with the SMACv1 results in Figure 3, SCOUT achieves positive online improvement in all offline-to-online RL configurations across both benchmarks, confirming that the test-time transport paradigm generalizes across discrete and continuous action space types.

Hyperparameter sweep. Figure 6 reports the sensitivity of SCOUT to the refinement steps L_{\text{train}} and L_{\text{test}} across all four dataset qualities in MA-MuJoCo. In the Expert and Medium-Expert datasets, performance is largely insensitive to the choice of L_{\text{test}}, which is expected because the behavioral prior already closely matches high-value regions, leaving little room for gradient transport to improve upon. In contrast, the Medium and Medium-Replay datasets exhibit a clearer benefit from moderate refinement steps, where increasing L_{\text{test}} from 0 to 3 yields substantial gains. However, excessive refinement L_{\text{test}}=10 can degrade performance in some cases, suggesting that the critic’s accuracy degrades when actions are transported too far from the behavioral support.

NeurIPS Paper Checklist

1. Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer: [Yes]
4. Justification: The abstract and introduction clearly state three contributions.
5. Guidelines: The answer NA means that the abstract and introduction do not include the claims made in the paper. The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
6. Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer: [Yes]
9. Justification: The paper includes a dedicated limitation discussion in the Conclusion section.
10. Guidelines: The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. The authors are encouraged to create a separate "Limitations" section in their paper. The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
11. Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
13. Answer: [Yes]
14. Justification: The paper states Assumption 4.1 explicitly and provides three theoretical results (Theorem 4.2, Proposition 4.3) with complete proofs in Appendix A.
15. Guidelines: The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
16. Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
18. Answer: [Yes]
19. Justification: The paper provides detailed algorithm pseudocode (Algorithm 1), complete hyperparameter tables, implementation details, and baseline descriptions in Appendix B.
20. Guidelines: The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
21. Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer: [Yes]
24. Justification: The paper provides an anonymous code repository link in the Abstract. The offline datasets used from the off-the-grid dataset [58] are publicly available benchmarks.
25. Guidelines: The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details. While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
26. Experimental setting/details
27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
28. Answer: [Yes]
29. Justification: Section 5 describes the experimental setup, including testbeds, datasets, and baselines. Appendix B provides comprehensive implementation details, default and per-configuration hyperparameters, and baseline configurations.
30. Guidelines: The answer NA means that the paper does not include experiments. The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, or as supplemental material.
31. Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer: [Yes]
34. Justification: All results in Tables 1 and 2 report mean \pm two standard deviations over 8 random seeds.
35. Guidelines: The answer NA means that the paper does not include experiments. The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean. It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
36. Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
38. Answer: [Yes]
39. Justification: The paper reports the type of GPU/CPU hardware used for the experiments.
40. Guidelines: The answer NA means that the paper does not include experiments. The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
41. Code of ethics
42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
43. Answer: [Yes]
44. Justification: The work conforms with the NeurIPS Code of Ethics.
45. Guidelines: The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
46. Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer: [N/A]
49. Justification: This work is foundational research on offline MARL. It does not have a direct path to negative societal applications.
50. Guidelines: The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
51. Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
53. Answer: [N/A]
54. Justification: The paper does not pose risks for misuse comparable to pretrained language models or image generators.
55. Guidelines: The answer NA means that the paper poses no such risks. Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
56. Licenses for existing assets
57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer: [Yes]
59. Justification: The paper cites the original papers for all benchmarks and datasets used.
60. Guidelines: The answer NA means that the paper does not use existing assets. The authors should cite the original paper that produced the code package or dataset. The authors should state which version of the asset is used and, if possible, include a URL. The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
61. New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer: [Yes]
64. Justification: The paper releases code via an anonymous repository linked in the abstract.
65. Guidelines: The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. The paper should discuss whether and how consent was obtained from people whose asset is used. At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
66. Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
68. Answer: [N/A]
69. Justification: This paper does not involve crowdsourcing or research with human subjects.
70. Guidelines: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
71. Institutional review board (IRB) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
73. Answer: [N/A]
74. Justification: This paper does not involve crowdsourcing or research with human subjects.
75. Guidelines: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
76. Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
78. Answer: [N/A]
79. Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
80. Guidelines: The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.