How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning

Preprint

HOW TO FIND AND REUSE POLICIES FOR CONTIN UOUS ADAPTATION IN LIFELONG REINFORCEMENT LEARNING

arXiv:2610.03119v1 [cs.LG] 2 Oct 2026

Saptarshi Nath, Inish D’Souza & Andrea Soltoggio Department of Computer Science Loughborough University {s.nath,i.m.d-souza,a.soltoggio}@lboro.ac.uk Antonio Carta Department of Computer Science University of Pisa [email protected]

Soheil Kolouri Department of Computer Science [email protected]

ABSTRACT

In lifelong reinforcement learning, retaining previously learned policies is not sufficient for effective transfer to a new task. Useful knowledge may be distributed across several prior policies, and its relevance may change as the learner acquires experience. One hypothesis is that task similarity can be effectively used in a continual learning setting to find and combine previously learned policies. To test it, Adaptive Mask Selection and Composition (AMSC) is designed to estimate similarity from online experience via non-parametric Wasserstein task embeddings from state-action-reward samples. The z-score-normalized sparsemax of the similarity scores are used to derive a variable-size support to periodically choose and weight policies to form a prior when learning a new task. On CT-graph and MiniGrid, AMSC achieves higher mean performance and forward transfer than the evaluated modular composition baselines while exhibiting no forgetting. Results on Continual World suggest that identifying relevant prior knowledge and determining its layer-specific composition may require additional layer-specific tuning. Ablations show that selecting relevant sources and determining how strongly to reuse them are central to these gains. Independently measured pairwise transfer is also positively associated with task-embedding similarity. These results indicate that task similarity can be an effective criterion to select and weight specific knowledge for reuse in lifelong reinforcement learning.1

INTRODUCTION

Lifelong learning requires both retention and reuse of prior knowledge. Although forgetting remains a central challenge, retention alone does not ensure positive transfer to later tasks. As experience accumulates, a model must identify relevant knowledge without allowing irrelevant or conflicting knowledge to cause interference. Prior work has extensively studied retention and increasingly evaluates forward transfer (Rusu et al., 2016; Kirkpatrick et al., 2017; Zenke et al., 2017; Mallya & Lazebnik, 2018; Rolnick et al., 2019; von Oswald et al., 2020), but commonly reuses shared or aggregated knowledge. Selective reuse remains comparatively underexplored. This distinction is especially consequential in reinforcement learning (RL). Transferred knowledge affects the actions taken and the state-action distribution from which the policy learns. An unsuitable prior can bias exploration and induce negative transfer (Taylor & Stone, 2009; Ahn et al., 2025; Uchendu et al., 2023). Sparse rewards provide little early evidence that a policy is harmful (Gimelfarb et al., 2021). Moreover, useful structure may be distributed unequally across several priors. 1

https://github.com/Chocological45/amsc

Preprint

Existing lifelong RL methods transfer through replay, modular routing, parameter isolation, and learned composition (Isele & Cosgun, 2018; Kessler et al., 2022; Powers et al., 2022; Mendez et al., 2022; Ben-Iwhiwhu et al., 2023; Zhang et al., 2024; Malagón et al., 2024; Hu et al., 2025; Huang et al., 2025). Modular approaches can preserve completed policies, but reusing the full library may admit interfering priors. Learning how strongly to reuse each policy from the target objective can identify useful policies but may converge only after their opportunity to improve early learning has passed. Measured task similarity can instead act as a method of estimating the relevance of prior policies without optimization (Carroll & Seppi, 2005; Taylor & Stone, 2009; Li et al., 2019; Gimelfarb et al., 2021; Wu et al., 2025; Nath et al., 2026). In task-incremental lifelong RL, the availability of prior policies is determined by curriculum history, completed policies are frozen, and the number of useful sources is unknown. A suitable selector must therefore adapt to the current library. As experience is collected on a new task, both the selected knowledge and its contribution to learning may also change. We ask whether online task similarity can determine both which policies to reuse and how strongly they should contribute under these constraints. 0.8 0.7

Method

Fine-tuning No-reuse SER-s SER-r SDW OWL SANE

0.5 Mask-BLC Mask-LC CKA-RL CompoNet MOSAIC MOSAIC-noperf AMSC

0.4

0.6

0.3

0.5 0.2

0.4

Stability (FGT)

0.9

Performance (AUC)

Adaptive Mask Selection and Composition (AMSC) learns task-specific policy masks over a fixed backbone network. It uses online task similarity to select a variable-size support of prior masks and determine their composition weights. Selected and current mask scores are normalized before composition with an ℓ2 straight-through estimator, and only the current policy-mask scores are optimized. AMSC selects without needing task descriptions, task labels, or transfer probes.

0.3 0.1 0.2 0.1 −0.4

−0.2

0.0

0.2

0.4

0.6

0.8

0.0

Plasticity (FWT) AMSC is evaluated on CT-graph (CT28), MiniGrid (MG16), and Continual World (CW10). It obtains AUC/FWT/FGT of 0.94/0.94/0.00 on CT28 and 0.73/0.54/0.00 on MG16 (Fig- Figure 1: Transfer-retention trade-off on CT28 ure 1). In CT28 and MG16, similarity-derived and MG16. Each marker reports mean FWT and weights perform comparably to learned com- AUC across seven seeds; colour denotes FGT. Erposition weights. In CW10, similarity-derived ror bars are 95% confidence intervals. weights underperform with respect to learned mask composition. Learning composition over a sparse selection of prior policies may provide a performance benefit. Ablations show that selecting relevant policies from the agent’s library and determining how to reuse each through similarity or learned weights are critical to more effective forward transfer and successful learning in lifelong RL. Results also show that task similarity is positively associated with independently measured pairwise transfer. Lastly, results suggest that masks can be quantized with minimal forgetting and loss in AUC and FWT.

RELATED WORK

Selective reuse in lifelong reinforcement learning. Forward transfer and forgetting are distinct objectives in lifelong RL. Preserving earlier tasks does not imply faster learning on later tasks (Wolczyk et al., 2021; 2022). Prior work promotes selective reuse through sparse or compositional subnetworks (Sokar et al., 2022; Ostapenko et al., 2021), related and unrelated task sequences (Ke et al., 2020), selective replay (Isele & Cosgun, 2018; Huang et al., 2025), and adaptive structure allocation (Lee et al., 2021; Dekhovich et al., 2023; Savadikar et al., 2026). These methods select data, capacity, or structures, often to determine what knowledge is worth retaining. Modular routing and composition. Modular lifelong RL isolates task-specific knowledge and transfers through routing or composition. Prior methods activate modules, select policy heads, infer contexts, or route adapters (Powers et al., 2022; Kessler et al., 2020; Zhang et al., 2024; Julian et al., 2025); others compose masks, modules, policies, or task vectors (Ben-Iwhiwhu et al., 2023; Kessler et al., 2022; Hu et al., 2025; Malagón et al., 2024). 2

Preprint

Task similarity and optimal transport. Task similarity predicts transfer and supports policy selection in RL (Carroll & Seppi, 2005; Taylor & Stone, 2009; Li et al., 2019; Gimelfarb et al., 2021), while transferability methods use representations, probe networks, likelihoods, or parameter subspaces (Zamir et al., 2018; Achille et al., 2019; Nguyen et al., 2020; Lin et al., 2022; Wu et al., 2025). Optimal transport compares task distributions through OTDD and Wasserstein embeddings (Courty et al., 2018; Alvarez-Melis & Fusi, 2020; Liu et al., 2025), and supports memory, selection, distillation, context detection, and expert diversity (Wang et al., 2022; 2024; Xu et al., 2023; Dick et al., 2024; Hihn & Braun, 2022).

PRELIMINARIES

3.1

PROBLEM SETTING

Consider a task-incremental lifelong RL setting with a sequence of K tasks {Tk }K k=1 . Each task is an MDP Tk = (S, A, pk, rk, γrl ), where S and A are the state and action spaces, pk is the transition kernel, rk is the reward function, and γrl is the discount factor. Tasks arrive sequentially with known boundaries, and only the current task environment is available for interaction. The learner trains on each task for a fixed budget before proceeding to the next task. A lifelong learning agent aims to maintain performance on previously learned tasks, or improve it through backward transfer, while continuing to learn new tasks through forward transfer. Forward transfer measures whether knowledge acquired from earlier tasks accelerates learning of a new task (Parisi et al., 2019; Khetarpal et al., 2022; Wolczyk et al., 2021). 3.2

MODULAR POLICY MASKS

Score-based masking learns task-specific subnetworks over a shared, fixed backbone. Supermasks in Superposition stores multiple tasks as binary masks over the same weights (Wortsman et al., 2020). Modulating Masks adapts this mechanism to lifelong RL and introduces learned linear combinations of the current and completed task masks (Ben-Iwhiwhu et al., 2023). CKA-RL is a related modular composition method that combines and merges dense task-specific residuals rather than binary masks (Hu et al., 2025). ℓ Let {W ℓ }L ℓ=1 be the fixed policy-backbone matrices and Sk the real-valued score matrix for task k at masked layer ℓ. The mask in the forward pass is Mkℓ = g(Skℓ ), g(x) = 1{x≥0}, (1)

where g is applied elementwise. The effective parameters are W ℓ ⊙ Mkℓ . A straight-through estimator propagates gradients through g. Let Siℓ,∗ be the frozen score matrix stored after task i, and let P(k) ⊆ {1, . . ., k − 1} contain the priors admitted for task k. A generic convex score composition is X X ℓ ℓ ℓ Skℓ,comb = αk,i Siℓ,∗ + αk,k Skℓ, = 1, (2) αk,i i∈P(k)

i∈P(k)∪{k}

with non-negative weights. Ben-Iwhiwhu et al. (2023); Hu et al. (2025) use the complete prior library and learn layer-specific weights. For m > 0, Ben-Iwhiwhu et al. (2023) initializes the coefficients uniformly in Mask-LC or balances prior and current-mask mass in Mask-BLC. For m = |P(k)| > 0, 1 ℓ,(0) ℓ,(0) Mask-LC: αk,k = αk,i = , m+1 (3) 1 1 ℓ,(0) ℓ,(0) Mask-BLC: αk,k =, αk,i = , i ∈ P(k). 2 2m When m = 0, the current mask has an initial weight of 1. Prior scores remain frozen, and the final composition is consolidated, Skℓ,∗ ← Skℓ,comb, for later tasks.

ADAPTIVE MASK SELECTION AND COMPOSITION

AMSC consists of three operations: normalized sparsemax retrieval, scale-normalized mask scores, and composition from the retrieval weights. 3

Preprint

4.1

MASK LIBRARY AND ONLINE TASK REPRESENTATIONS

At task k, AMSC initializes a new current score matrix Skℓ and the policy library contains the frozen score matrices {Siℓ,∗ }L i<k,ℓ=1 following the formulation defined in Section 3.2. Let Dk be (t)

the current-task experience buffer and Xk a sampled batch matrix. AMSC uses an embedding (t) (t) map ψ(Xk ) = ek ∈ Rde that places online experience from different tasks in a common vector space. The main experiments use a non-parametric linear Wasserstein embedding (WTE) of state– action–reward samples (Liu et al., 2025) which has been demonstrated in RL tasks (Nath et al., 2026). Discrete actions are one-hot encoded, states are standardized within each sampled batch, and rewards are retained as observed. Given a fixed reference matrix X0 with R samples and an optimal transport matrix Γ⋆, the WTE batch embedding is ! (t) (RΓ⋆ )⊤ Xk − X0 (t) √ ek = vec . (4) R The current representation is smoothed online as   (t) (t−1) (t) ēk = normalize ρēk + (1 − ρ)ek,

(5)

where the first update initializes the normalized representation and ρ = 0.5 thereafter. Completed task representations remain fixed. Appendix B.1 formalizes the optimal transport plan and the em(t) pirical distributions represented by Xk and X0 . 4.2

SPARSE RETRIEVAL

Cosine similarity has been used to evaluate task relationships through gradient alignment, and task embeddings (Lopez-Paz & Ranzato, 2017; Achille et al., 2019; Yu et al., 2020a; Nath et al., 2026). Following this, AMSC compares the current representation of task k with every completed task using cosine similarity, sk,i = cos(ēk, ēi ), i < k. (6) ⊤ Let sk = [sk,1, . . ., sk,k−1 ] . AMSC retrieves wk = sparsemax(zscore(sk )/τ ), P(k) = {i < k : wk,i > 0}. (7) The z-score has zero mean and unit variance within the available library, and τ = 1 in all main experiments. Sparsemax projects these relative scores onto the probability simplex and assigns P exact zeros, so i∈P(k) wk,i = 1 (Martins & Astudillo, 2016). 4.3

SCALE-NORMALIZED FIXED COMPOSITION

Positive rescaling of a task’s scores changes its influence when scores from several tasks are combined before thresholding. Large magnitude scores become more difficult to alter. Consequently, raw score magnitude can override the linear combination weights. We therefore normalize each selected prior and current score to unit ℓ2 norm in the forward pass, using a straight-through estimator to preserve the current score’s gradient,   Siℓ ℓ bℓ = S ℓ + sg − S , (8) S i i i max(∥Siℓ ∥F, εm ) where sg denotes stop-gradient. Learning composition weights from the RL objective requires the agent to discover useful sources through task interaction. AMSC instead can use the same online similarity signal for both retrieval and weighting, making source contributions available at the first retrieval event without optimization. Let m = |P(k)|. Instead of learning the composition, AMSC applies weights computed from task similarity which,  bℓ,  S m = 0,   k ℓ X m 1 bℓ Ck = Mkℓ = g(Ckℓ ). (9) bℓ, m > 0, Sk + wk,i S  i  m + 1 m+1 i∈P(k)

Preprint

For m > 0, the current mask receives mass 1/(m + 1), while the selected priors share m/(m + 1) according to their retrieval weights. These weights are shared across layers. Each task begins without retrieved priors. Once sufficient experience is available, AMSC updates the current embedding and recomputes equation 7–equation 9 between training iterations to reflect new data. At the task boundary, the final composed scores Ckℓ are consolidated as Skℓ,∗ ← Ckℓ, frozen, and added to the reusable mask library. Appendix B.2 Algorithm 1 gives the complete training pseudo-code.

RESULTS

5.1

EXPERIMENTAL SETUP

Benchmarks. We evaluate AMSC on CT28, a sparse-reward decision-tree curriculum with hierarchical tasks (Soltoggio et al., 2023); MG16, sparse-reward MiniGrid navigation tasks with shuffled observations (Chevalier-Boisvert et al., 2018); and CW10, Continual World continuous robotic control tasks (Wolczyk et al., 2021). In CT28 and MG16, tasks within each family share a common observation space and increase in complexity. Families differ in their representations while sharing input dimensions and actions. These curricula test reuse when prior knowledge varies in relevance in relation to the next task. Details and pairwise-transfer matrices are reported in Appendix C. Baselines. We compare against replay methods (SER (Isele & Cosgun, 2018) and SDW (Huang et al., 2025)), modular methods (Mask-LC/BLC (Ben-Iwhiwhu et al., 2023), CKA-RL (Hu et al., 2025), CompoNet (Malagón et al., 2024), OWL (Kessler et al., 2022), SANE (Powers et al., 2022), MOSAIC (Nath et al., 2026), and ProgNet (Rusu et al., 2016)), fine-tuning, and learning from scratch. We also evaluate retrieval guided by independently measured pairwise transfer. Tables 1 and 2 report comparisons to baselines on CT28, MG16 and CW10. Appendix E provides details of the hyperparameters for AMSC and baselines. Protocol and Metrics. All empirical results are evaluated with seven shared seeds. CT28 and MG16 use PPO with 51, 200 and 512, 000 steps per task, respectively. CW10 uses SAC with 1M steps per task. We report mean task area under the evaluation curve (AUC), forward transfer (FWT), and forgetting (FGT) from normalized performance between [0, 1], following (Wolczyk et al., 2021). We report these metrics together because zero forgetting can reflect strong retention but not necessarily strong policies. Appendix E specifies training and evaluation hyperparameters, Appendix F.3 reports hyperparameter sensitivity, and Appendix G reports the details of how the metrics are computed. 5.2

COMPARISON TO BASELINES

Selective reuse improves learning. AMSC obtains strong AUC and FWT with zero measured forgetting on CT28 and MG16 (Table 1), occupying a favorable region of the transfer-retention tradeoff (Figure 1) with respect to baselines. AMSC achieves AUC/FWT of 0.94/0.94 and 0.73/0.54 on CT28 and MG16, respectively. From-scratch learning also avoids forgetting but obtains only 0.22 and 0.23 AUC. Surprise-prioritized SER slightly exceeds AMSC on CT28 (0.96/0.95) but incurs 0.54 FGT, while reservoir SER obtains 0.63/0.20 on MG16 with 0.07 FGT. Comparison with modular reuse. AMSC achieves the highest mean AUC and FWT among modular methods on CT28 and MG16 (Table 1). These methods differ in how they reuse prior knowledge: Mask-LC/BLC learn weights over all prior masks, CompoNet learns to combine policy-unit outputs, and MOSAIC selects masks through absolute similarity and performance gates before optimizing composition. AMSC instead periodically updates both its selected sources and their weights using library-relative similarity. On CT28, AMSC achieves AUC/FWT of 0.94/0.94, compared with 0.81/0.79 for MOSAIC and 0.69/0.64 for CompoNet. On MG16, both baselines have negative FWT, whereas AMSC achieves 0.54. AMSC’s gains over Mask-LC, CompoNet, and MOSAIC are significant on both metrics and curricula (p ≤ 0.023). Its MG16 advantage over Mask-BLC (0.73/0.54 versus 0.55/0.23, p = 0.016 and p = 0.016, respectively). These results associate 5

Preprint

Table 1: Evaluation on CT28 and MG16. Each cell reports the mean AUC/FWT/FGT with an 95% confidence interval (Colas et al., 2018). AVG is computed per seed by averaging CT28 and MG16, then bootstrapping the seven paired averages. Bold and underlining denote the best and second-best non-oracle result for each metric. Significance tests are reported in Appendix H. CT28

MG16

AVG

Method

AUC ↑

FWT ↑

FGT ↓

AUC ↑

FWT ↑

FGT ↓

AUC ↑

FWT ↑

FGT ↓

Fine-tuning From-scratch

.36+.02 −.02 .22+.01 −.01

.22+.03 −.03 .00+.00 −.00

.38+.02 −.02 .00+.00 −.00

.30+.05 −.05 .23+.03 −.02

.10+.08 −.09 .00+.00 −.00

.34+.08 −.07 .00+.00 −.00

.33+.02 −.02 .22+.01 −.01

.16+.05 −.06 .00+.00 −.00

.36+.03 −.03 .00+.00 −.00

SER-s SER-r SDW

.96+.02 −.03 .88+.05 −.05 +.11 .76−.10

.95+.03 −.03 .87+.06 −.06 +.13 .73−.13

.54+.17 −.18 .09+.08 −.07 -.01+.01 −.01

.42+.06 −.07 .63+.04 −.03 .53+.05 −.06

-.13+.17 −.17 .20+.14 −.20 +.12 .02−.18

.51+.10 −.09 .07+.11 −.09 .15+.13 −.12

.69+.03 −.04 .76+.02 −.02 .65+.06 −.07

.41+.08 −.08 .53+.08 −.10 .37+.10 −.14

.52+.09 −.11 .08+.09 −.06 +.06 .07−.06

OWL SANE Mask-BLC Mask-LC CKA-RL CompoNet MOSAIC MOSAIC ¬Iperf

.15+.01 −.01 .46+.02 −.03 .27+.02 −.02 .48+.03 −.03 .36+.05 −.04 +.08 .69−.10 .81+.03 −.03 .81+.03 −.03

-.31+.17 −.20 .37+.03 −.03 +.06 .10−.06 .35+.06 −.07 .25+.05 −.05 +.09 .64−.12 .79+.03 −.03 .78+.03 −.03

.18+.03 −.04 .00+.00 −.00 .00+.00 −.00 .00+.00 −.00 .36+.05 −.05 .00+.00 −.00 .00+.00 −.00 .00+.00 −.00

.21+.03 −.02 .36+.09 −.08 .55+.10 −.10 .40+.14 −.13 .34+.11 −.09 .31+.07 −.08 .29+.11 −.11 .29+.13 −.11

-.29+.18 −.19 .18+.11 −.09 +.21 .23−.19 -.24+.41 −.37 -.21+.20 −.24 -.85+.24 −.25 -.50+.40 −.45 -.60+.47 −.50

.22+.03 −.02 .00+.00 −.00 .00+.00 −.00 +.00 .00−.00 .40+.14 −.12 .00+.00 −.00 .00+.00 −.00 .00+.00 −.00

.18+.02 −.02 .41+.04 −.04 .41+.06 −.06 .44+.06 −.06 .35+.07 −.06 .50+.05 −.06 .55+.05 −.06 .55+.06 −.05

-.30+.09 −.09 .28+.05 −.04 +.13 .17−.12 .06+.21 −.20 .02+.11 −.13 -.10+.15 −.15 .14+.20 −.22 +.22 .09−.24

.20+.02 −.02 .00+.00 −.00 .00+.00 −.00 .00+.00 −.00 .38+.08 −.06 .00+.00 −.00 .00+.00 −.00 .00+.00 −.00

Transfer oracle AMSC

.82+.00 −.01 .94+.02 −.03

.72+.07 −.13 .94+.02 −.03

.00+.00 −.00 .00+.00 −.00

.66+.08 −.09 .73+.06 −.06

.54+.10 −.10 .54+.12 −.14

.00+.00 −.00 .00+.00 −.00

.74+.04 −.04 .84+.03 −.03

.63+.08 −.09 .74+.06 −.07

.00+.00 −.00 .00+.00 −.00

explicit, relative selection and composition with stronger reuse than dense composition or fixed absolute selection. Table 2: CW10 performance across seven seeds. Entries report means Early learning across reuse mechanisms. Over the first with 95% confidence intervals. 20% of CT28 tasks 1–27, AMSC obtains AUC20 = 0.86, compared with 0.35 for MOSAIC (p = 0.016). The first posMethod AUC FWT FGT itive training reward appears after 2.34k versus 5.83k steps +7.20 +.17 -7.12−13.38 .39+.19 −.21 (p = 0.016). SER-surprise reaches 0.91 early AUC and SER- Fine-tuning .34−.18 From-scratch .41+.04 .00+.00 .00+.00 −.04 −.00 −.00 reservoir 0.86; neither differs significantly from AMSC on -.31+.14 .00+.00 PNN .34+.02 −.03 −.14 −.00 this metric. Thus AMSC’s early advantage is clear relative to Mask-LC +.34 +.00 +.06 .47−.06 -.22−.40 .00−.00 MOSAIC, while replay can match or exceed its early learn- Mask-BLC .53+.04 .12+.19 .00+.00 −.04 −.20 −.00 +1.76 +.00 ing. Appendix Tables 8 and 9 report aggregate and per-task CKA-RL -2.12 .39+.10 −.14 −2.24 .00−.00 +3.74 +.00 results. MOSAIC .26+.04 -4.76 .00 −.03 −6.41 −.00 AMSC

5.3

.48+.05 −.05

-1.21+.79 −.98

.00+.00 −.00

LIMITS OF SIMILARITY WEIGHTING

Does fixed similarity weighting extend to continuous control? On CW10, AMSC obtains 0.48 AUC with zero measured forgetting, but its FWT is −1.21; Mask-BLC obtains 0.53 AUC and 0.12 FWT (Table 2). Their composition weights overlap on some tasks, yet AMSC often allocates more mass to prior masks later in the curriculum. At task 9, mean actor prior mass is 0.78 for AMSC versus 0.46 for Mask-BLC (Appendix F.2). AMSC applies the same similarity-derived proportions across masked layers, whereas Mask-BLC learns layer-specific weights. This difference suggests that identifying relevant prior policies and determining how to compose their knowledge are distinct challenges, even when learned composition addresses both through a single set of weights. 5.4

TASK REPRESENTATION SENSITIVITY

One question is whether the performance of AMSC is dependent on the specific representation approach, or whether alternatives to WTE can be used with similar performance. Table 3 compares WTE (Liu et al., 2025), sliced-Wasserstein embeddings (Naderializadeh et al., 2021), an empiricalFisher adaptation of Task2Vec (Achille et al., 2019), recurrent encoders motivated by PEARL and VariBAD (Rakelly et al., 2019; Zintgraf et al., 2021), and FID-inspired Fréchet moments (Heusel et al., 2017). All comparisons receive complete (s, a, r, s′, d) as Task2Vec-style and learned con6

R3

0.27

0.06

0.72

0.28 0.65

R4 R2

R3

R4

Target depth

R5

0.50 0.75 1.00

0.6 0.4 0.2 0.0

0.61

R2

0.47

0.45

0.18 0.42

0.20 0.04

0.34

0.30

R3

R4

R5

Target depth

Component Current R2 R3 R4 Cross

(C) Similarity and transfer ( AUC)

1.00 Pearson r=0.67

1.0

0.50

0.8

Spearman =0.52 0.75 n=350

0.25 0.00

0.6

0.4 0.6 1.0 Pearson r=0.48 0.5

Selection frequency

0.91

R2

0.52

Pairwise transfer ( AUC) Pairwise transfer ( AUC)

Mean composition mass

transfer (B) Similarity-weighted composition (A) Retrieval and causal transferCausal ( AUC) 1.00 1.0 Cross 0.75 1.000.33 0.12 0.03 0.01<0.01 D2 0.8 0.38 0.50 0.45 0.42 0.39 0.33 0.30 Component Current 0.75 D2 0.670.30 0.13 0.05 0.01 D3 0.6 D3 0.23 0.25 D4 0.580.31 0.21 0.07 D4 0.22 0.23 0.22 0.50 D5 0.4 0.530.30 0.20 D6 D5 0.09 0.10 0.16 0.16 0.62 D7 0.06 0.50 0.440.32 0.25 Cross D6 0.2 0.33 0.27 0.25 0.23 0.22 0.39 D7 0.0 D2 D3 D4 D5 D6 D7 D8 0.00 D2 D3 D4 D5 D6 D7 D8 1.0 0.06 Cross 0.75 0.09 0.01 <0.01 0.25 0.39 0.8

Mean composition mass

MG16

CT28

Preprint

0.8

1.0

0.4

Spearman =0.42 n=120

0.2

0.0 0.5 0.4

0.6

0.8

Online task similarity

1.0

0.0

Figure 2: AMSC on CT28 (top) and MG16 (bottom). Same-family source policies share the target task’s observation space. (A) Mean pairwise ∆AUC by source category. Annotations give the mean total sparsemax retrieval mass assigned to that category. (B) Mean composition mass after applying Equation 9 to selected sources. (C) Online similarity versus pairwise transfer; color denotes selection frequency and the line shows bin means with target-cluster 95% confidence intervals. CT28 includes depths D4–D8, whereas MG16 includes all causal non-self pairs. Retrieval uses seven seeds; pairwise transfer uses one source-policy seed. text encoder variants require the reward and next state as part of the objective. The main AMSC experiments use only (s, a, r) (also reported in Table 3 for reference). Representation choice is task-dependent but AMSC remains agnostic. Among the variants, WTE–transition has the highest mean AUC/FWT on CT28 (0.93/0.92), while Fréchet moments lead on MG16 (0.70/0.47). SWE and random recurrent features also yield positive transfer on both curricula. AMSC therefore accommodates several representations, although performance depends on the specific distributions being measured and specific configurations used and no single approach is universally best.

5.5 RETRIEVAL,

COMPOSITION, AND CAUSAL TRANSFER

Table 3: Task-representation sensitivity of AMSC. Reports AUC/FWT with 95% confidence intervals across seven seeds. T denotes complete transitions; R and L denote random and online-learned. Bold and underlining indicate the best and second-best results.

Method

CT28 AUC↑/FWT↑

MG16 AUC↑/FWT↑

WTE (T) SWE (T) Fisher (T2V) Context (R) Context (L) Fréchet

+.04 .93+.04 −.04 /.92−.04 +.05 .92+.04 /.91 −.03 −.05 +.08 .87+.06 /.85 −.07 −.10 +.03 .92−.03 /.91+.03 −.04 +.03 .90−.03 /.87+.06 −.06 +.06 .90+.03 −.03 /.87−.07

+.12 .61+.07 −.08 /.38−.14 +.15 .67+.04 /.39 −.04 −.16 +.17 .45+.12 /.21 −.10 −.18 +.06 .64−.07 /.41+.10 −.12 +.09 .44−.10 /.14+.21 −.28 +.13 .70+.07 −.07 /.47−.16

WTE (SAR)

+.02 .94+.02 −.03 /.94−.03

+.12 .73+.06 −.06 /.54−.14

We test whether similarity identifies useful prior knowledge using independently measured pairwise transfer. For each source–target pair, we measure ∆AUC (target task AUC with a source minus AUC without it). Does task similarity rank useful priors? Figure 2C presents the correlation between pairwise transfer and similarity. Online similarity correlates with pairwise transfer on CT28 depths 4–8 (Pearson r = 0.67, 95% CI [0.57, 0.77] and MG16 (r = 0.48, [0.31, 0.62]) with Spearman ρ = 0.52 and 0.42. These correlations suggest that similarity is informative of which prior masks are likely to transfer. Does retrieval exploit this ranking? We define a pair as regularly selected if it is retrieved in at least half of its available selection events. On CT28, regularly selected pairs have mean ∆AUC = 0.66, versus 0.02 for other pairs. These pairs cover 81% of positive-transfer pairs and include no negative-transfer pairs. On MG16, corresponding means are 0.49 and 0.15, but retrieval covers only 19 of 47 positive-transfer pairs as a result of the sparse selection. 7

Preprint

How is reuse allocated within priors? On CT28, support grows from one prior at depth 3 to approximately four at depth 8, reducing current-mask mass from 0.50 to 0.22. At depth 8, predecessors at depths 7 and 6 receive 0.30 and 0.25 of the mass. MG16 shows a similar pattern: at depth-5 (R5), the current mask and depth-4 (R4) and depth-3 (R3) masks receive 0.30, 0.45, and 0.20 and below 0.01 to selected cross-family masks. Reuse therefore concentrates on related hierarchical sub-task masks obtained earlier in learning, while retaining a current-task component (Figure 2B). 5.6

ABLATION STUDIES

Table 4: Ablation of AMSC’s selection and composition mechanism. All runs use Controls. Table 4 tests whether the performance WTE embeddings of state–action–reward tugains of AMSC are attributed to the combination ples. Values are means with 95% confidence of components. Random permutes selected pri- intervals. All configurations obtain zero FGT. ors while preserving how many are selected and their specific similarity-derived weights. Learned- Configuration CT28 AUC↑/FWT↑ MG16 AUC↑/FWT↑ +.04 +.15 .90+.03 .61+.10 α optimizes the retrieved weights, whereas No No Sim.-α −.03 /.88−.04 −.10 /.41−.15 +.03 +.26 .89+.03 .59+.09 Sim.-α tests a uniform composition over selected No L2-STE −.03 /.86−.03 −.10 /.21−.29 +.04 +.05 +.11 +.14 No Periodicity .87 /.85 .63 /.42 −.05 −.06 −.13 −.15 prior masks. No Periodicity retains the first re+.05 +.16 trieval for the entire learning duration. Top-1 re- Top-1 .88+.04 .63+.09 −.04 /.86−.04 −.08 /.39−.16 +.15 +.04 .60+.11 .92+.03 tains the highest-similarity prior after sparsemax Learned-α −.03 /.91−.04 −.11 /.33−.14 +.07 +.05 +.02 .36−.06 /.08+.13 .35−.02 /.13−.06 selection. Transf. Oracle uses independent pair- Random −.13 +.00 +.07 +.10 .66+.08 Transf. Oracle .82 /.72 −.01 −.13 −.09 /.54−.10 wise ∆AUC measurements to guide retrieval of +.02 +.12 multiple masks but is not an upper bound on joint AMSC .73+.06 .94+.02 −.03 /.94−.03 −.06 /.54−.14 composition. Does source relevance matter beyond sparsity? Randomizing which prior masks are selected significantly reduces AUC/FWT to 0.35/0.13 on CT28 and 0.36/0.08 on MG16 (p = 0.016). Because support size and weights are preserved, these reductions show that sparse composition alone is insufficient and selecting relevant priors is central to the gains. Do retrieval updates and multiple sources improve transfer? Freezing the first retrieval lowers AUC/FWT to 0.87/0.85 on CT28 and 0.63/0.42 on MG16. These decreases suggest that revising the support and weights as experience accumulates contributes beyond the initial selection. In AMSC, selection remains stable throughout learning. Consecutive-selection Jaccard similarity is 0.94–1.00 on CT28 and 0.97–1.00 on MG16. AMSC successfully identifies relevant prior masks early on in learning then gradually refines its selection. Restricting retrieval to one prior mask results in 0.88/0.86 and 0.63/0.39, respectively. Since total prior mass is unchanged, this comparison supports composing multiple sources rather than assigning the entire reuse allocation to the highestranked prior. Transfer-informed retrieval. Transfer oracle also obtains positive FWT, obtaining AUC/FWT of 0.82/0.72 on CT28 and 0.66/0.54 on MG16. It nevertheless falls below AMSC on CT28 and matches only its FWT on MG16. Selection is guided by pair-wise transfer, therefore it does not provide an upper bound on joint composition. Do composition weights need to be learned? Learning weights within the retrieved support yields AUC/FWT of 0.92/0.91 on CT28 and 0.60/0.33 on MG16, compared with 0.94/0.94 and 0.73/0.54 for AMSC. Optimization thus provides no improvement in mean performance in these experiments. Replacing similarity-derived source proportions with uniform weights also lowers performance, to 0.90/0.88 and 0.61/0.41. These comparisons suggest that similarity provides useful information about how strongly to reuse selected masks, beyond determining which masks enter the composition; however, task similarity in AMSC cannot determine how knowledge should be uniquely composed at every layer of the network as indicated by the earlier CW10 results. Does mask-score normalization matter? Composition weights determine source contributions only relative to the magnitudes of the scores being combined. Removing L2-STE lowers AUC/FWT to 0.89/0.86 on CT28 and 0.59/0.21 on MG16. These results support normalizing scores before composition so that differences in score magnitude do not override the intended similarity-derived allocation. 8

Preprint

5.7

ADDRESSING PER-TASK MASK MEMORY COSTS

Continuous-value masks are memory intensive. We test whether masks can be quantized with minimal performance loss. Ternary quantization maps scores to {−1, 0, 1}, preserving their sign while suppressing near-zero scores. On CT28, ternary quantization achieves AUC/FWT/FGT of 0.94/0.91/0.04 compared to 0.941/0.935/0.000 for Float32 masks. Ternary scores also reduce per-task storage by 16× relative to Float32, while retaining better AUC/FWT (Appendix A).

DISCUSSION AND FUTURE WORK

The results suggest that task similarity can guide effective policy reuse without directly optimizing retrieval or composition weights when relevant prior knowledge is available. On CT28 and MG16, AMSC achieves competitive AUC and FWT relative to replay methods while avoiding forgetting through per-task masks. The random-selection ablation indicates that sparsity alone is insufficient. Periodic retrieval further improves reuse as more experience is collected and task representations are updated. The differing performance of transition embeddings and Fréchet moments also suggests that representation choice depends on the distributions. Future work could evaluate combinations of priors beyond pairwise transfer to determine how reliably embedding similarity predicts joint transfer. The CW10 results highlight a distinction between identifying relevant knowledge and determining how to reuse it. Similarity-derived weights shared across layers may be too restrictive for effective composition. Future work could test whether task embeddings can guide layer-specific reuse or whether similarity-based selection could be combined with learned composition, including an adaptive balance between prior and current knowledge. Another direction is determining when an agent should abstain from reuse when no relevant prior knowledge exists. This work examines how sparsity and selectivity can enable effective forward transfer as the policy library grows, while managing that growth remains a complementary challenge. Since available knowledge influences what can be effectively learned, consolidation and compression should be evaluated for their effects on future transfer as well as retained performance. Finally, extending AMSC beyond known task boundaries requires detecting distribution changes and deciding when to retrieve, adapt, or create a new mask.

CONCLUSION

This work introduced Adaptive Mask Selection and Composition (AMSC) to address how lifelong agents can selectively reuse prior knowledge when learning new tasks. AMSC uses task similarity to periodically select and compose a sparse set of policy masks, without directly optimizing selection or composition. On CT28 and MG16, it outperforms modular baselines in mean AUC and forward transfer and remains competitive with replay methods while retaining prior policies without forgetting. These findings show that task similarity can provide an effective alternative to learning compositions, with ablations supporting the importance of relevant selection and updating reuse during learning. The advantage is not universal, however, as the CW10 results suggest that identifying relevant prior knowledge and adapting its composition across layers may require different mechanisms. The effectiveness of similarity-guided reuse depends on how prior knowledge is composed per-layer. Together, the results position task similarity as a practical approach for selective reuse.

ACKNOWLEDGMENTS

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under contract No. HR001121901 (Shared Experience Lifelong Learning).

REFERENCES

1. Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6429–6438, 2019. doi: 10.1109/ICCV.2019.00653. 9 Preprint Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, and Taesup Moon. Prevalence of negative transfer in continual reinforcement learning: Analyses and a simple baseline. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=KAIqwkB3dT. David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21428–21439. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/f52a7b2610fb4d3f74b4106fb80b233d-Paper.pdf. Eseoghene Ben-Iwhiwhu, Saptarshi Nath, Praveen Kumar Pilly, Soheil Kolouri, and Andrea Soltoggio. Lifelong Reinforcement Learning with Modulating Masks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id= V7tahqGrOq. J.L. Carroll and K. Seppi. Task similarity measures for transfer in reinforcement learning task libraries. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 803–808 vol. 2, 2005. doi: 10.1109/IJCNN.2005.1555955. Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic Gridworld Environment for OpenAI Gym. https://github.com/maximecb/gym-minigrid, 2018. Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. arXiv preprint arXiv:1806.08295, 2018. Nicolas Courty, Rémi Flamary, and Mélanie Ducoffe. Learning wasserstein embeddings. In International Conference on Learning Representations, 2018. URL https://openreview.net/ forum?id=SJyEH91A-. Aleksandr Dekhovich, David M. J. Tax, Marcel H. F Sluiter, and Miguel A. Bessa. Continual pruneand-select: class-incremental learning with specialized subnetworks. Applied Intelligence, 53 (14):17849–17864, 2023. doi: 10.1007/s10489-022-04441-z. URL https://doi.org/10. 1007/s10489-022-04441-z. Jeffery Dick, Saptarshi Nath, Christos Peridis, Eseoghene Benjamin, Soheil Kolouri, and Andrea Soltoggio. Statistical Context Detection for Deep Lifelong Reinforcement Learning, 2024. Michael Gimelfarb, Scott Sanner, and Chi-Guhn Lee. Contextual policy transfer in reinforcement learning domains via deep mixtures-of-experts. In Cassio de Campos and Marloes H. Maathuis (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pp. 1787–1797. PMLR, 27–30 Jul 2021. URL https://proceedings.mlr.press/v161/gimelfarb21a.html. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning That Matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11694. URL https://ojs.aaai. org/index.php/AAAI/article/view/11694. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf. Heinke Hihn and Daniel Alexander Braun. Mixture-of-variational-experts for continual learning. In ICLR Workshop on Agent Learning in Open-Endedness, 2022. URL https://openreview. net/forum?id=HhzzNQPZLb9. 10 Preprint Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=nZeVKeeFYf9. Jinwu Hu, ZiHao Lian, Zhiquan Wen, ChenghaoLi, Guohao Chen, Xutao Wen, Bin Xiao, and Mingkui Tan. Continual knowledge adaptation for reinforcement learning. In The Thirtyninth Annual Conference on Neural Information Processing Systems, 2025. URL https: //openreview.net/forum?id=QRlVickNdN. Zhiyi Huang, Xiaohan Shan, and Jianmin Li. Lifelong reinforcement learning with similarity-driven weighting by large models, 2025. URL https://arxiv.org/abs/2503.12923. David Isele and Akansel Cosgun. Selective Experience Replay for Lifelong Learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018. ISBN 9781-57735-800-8. Jack Julian, Yun Sing Koh, and Albert Bifet. Building adaptive knowledge bases for evolving continual learning models. npj Artificial Intelligence, 1(1):26, 2025. doi: 10.1038/s44387-025-00028-4. URL https://doi.org/10.1038/s44387-025-00028-4. Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of a mixed sequence of similar and dissimilar tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J Roberts. Unclear: A straightforward method for continual reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, 2020. Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J. Roberts. Same state, different task: Continual reinforcement learning without interference, 2022. Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401– 1476, 12 2022. doi: 10.1613/jair.1.13673. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. Nils Koster, Oliver Grothe, and Achim Rettinger. Signing the Supermask: Keep, Hide, Invert. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=e0jtGTfPihs. Seungwon Lee, Sima Behpour, and Eric Eaton. Sharing less is more: Lifelong learning in deep networks with selective layer transfer. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6065–6075. PMLR, 18–24 Jul 2021. URL https://proceedings. mlr.press/v139/lee21a.html. Siyuan Li, Fangda Gu, Guangxiang Zhu, and Chongjie Zhang. Context-aware policy reuse. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, pp. 989–997, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450363099. Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. TRGP: Trust region gradient projection for continual learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=iEvAf8i6JjO. 11 Preprint Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, and Soheil Kolouri. Wasserstein Task Embedding for Measuring Task Similarities. Neural Networks, 181:106796, 2025. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2024.106796. URL https://www.sciencedirect. com/science/article/pii/S0893608024007202. David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017. Mikel Malagón, Josu Ceberio, and Jose A. Lozano. Self-composing policies for scalable continual reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. Arun Mallya and Svetlana Lazebnik. Packnet: Adding Multiple Tasks to a Single Network by Iterative Pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765–7773, 2018. Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1614–1623, New York, New York, USA, 20–22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/martins16.html. Jorge A Mendez, Harm van Seijen, and Eric Eaton. Modular Lifelong Reinforcement Learning via Neural Composition. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=5XmLzdslFNN. Navid Naderializadeh, Joseph F Comer, Reed Andrews, Heiko Hoffmann, and Soheil Kolouri. Pooling by sliced-wasserstein embedding. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 3389–3400. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/ file/1bc2029a8851ad344a8d503930dfd7f7-Paper.pdf. Saptarshi Nath, Christos Peridis, Eseoghene Benjamin, Xinran Liu, Soheil Kolouri, Peter Kinnell, Zexin Li, Cong Liu, Shirin Dora, and Andrea Soltoggio. Policy search, retrieval, and composition via task similarity in collaborative agentic systems. Proceedings of the AAAI Conference on Artificial Intelligence, 40(29):24504–24512, Mar. 2026. doi: 10.1609/aaai.v40i29.39633. URL https://ojs.aaai.org/index.php/AAAI/article/view/39633. Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. LEEP: A new measure to evaluate transferability of learned representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7294–7305. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/nguyen20b.html. Oleksiy Ostapenko, Pau Rodríguez, Massimo Caccia, and Laurent Charlin. Continual Learning via Local Module Composition. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393. German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2019.01.012. URL https://www. sciencedirect.com/science/article/pii/S0893608019300231. Sam Powers, Eliot Xing, and Abhinav Gupta. Self-activating neural ensembles for continual reinforcement learning. In Sarath Chandar, Razvan Pascanu, and Doina Precup (eds.), Proceedings of The 1st Conference on Lifelong Learning Agents, volume 199 of Proceedings of Machine Learning Research, pp. 683–704. PMLR, 22–24 Aug 2022. URL https://proceedings.mlr. press/v199/powers22a.html. 12 Preprint Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5331–5340. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/rakelly19a.html. David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv preprint arXiv:1606.04671, 2016. Chinmay Savadikar, Michelle Dai, and Tianfu Wu. Cheem: Continual learning by reuse, new, adapt and skip - a hierarchical exploration-exploitation approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 25066–25076, June 2026. Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Avoiding forgetting and allowing forward transfer in continual learning via sparse networks. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part III, pp. 85–101, Berlin, Heidelberg, 2022. Springer-Verlag. ISBN 978-3-031-26408-5. doi: 10.1007/978-3-031-26409-2_6. URL https://doi.org/10.1007/978-3-031-26409-2_6. Andrea Soltoggio, Eseoghene Ben-Iwhiwhu, Christos Peridis, Pawel Ladosz, Jeffery Dick, Praveen K. Pilly, and Soheil Kolouri. The configurable tree graph (ct-graph): measurable problems in partially observable and distal reward environments for lifelong reinforcement learning, 2023. URL https://arxiv.org/abs/2302.10887. Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(56):1633–1685, 2009. URL http://jmlr.org/ papers/v10/taylor09a.html. Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. JumpStart Reinforcement Learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 34556–34583. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/ v202/uchendu23a.html. Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJgwNerKvB. Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. Inscl: A data-efficient continual learning paradigm for fine-tuning large language models with instructions, 2024. URL https://arxiv.org/abs/2403.11435. Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Tiehang Duan, and Mingchen Gao. Improving task-free continual learning by distributionally robust memory evolution. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22985–22998. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/wang22v.html. Anderson M. Winkler, Gerard R. Ridgway, Matthew A. Webster, Stephen M. Smith, and Thomas E. Nichols. Permutation inference for the general linear model. NeuroImage, 92:381–397, 2014. ISSN 1053-8119. doi: https://doi.org/10.1016/j.neuroimage.2014.01.060. URL https:// www.sciencedirect.com/science/article/pii/S1053811914000913. 13 Preprint Maciej Wolczyk, Michał Zajac, ˛ Razvan Pascanu, Łukasz Kuciński, and Piotr Miłoś. Continual world: A robotic benchmark for continual reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=5qsptDcsdEj. Maciej Wolczyk, Michał Zajac, ˛ Razvan Pascanu, Łukasz Kuciński, and Piotr Miłoś. Disentangling transfer in continual reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 6304–6317. Curran Associates, Inc., 2022. doi: 10.52202/ 068431-0456. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/2938ad0434a6506b125d8adaff084a4a-Paper-Conference.pdf. Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in Superposition. Advances in Neural Information Processing Systems, 33:15173–15184, 2020. Yanru Wu, Jianning Wang, Xiangyu Chen, Enming Zhang, Yang Tan, Hanbing Liu, and Yang Li. Exploiting task relationships in continual learning via transferability-aware task embeddings. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=V8FnYzDX35. Zihao Xu, Xuan Tang, Yufei Shi, Jianfeng Zhang, Jian Yang, Mingsong Chen, and Xian Wei. Continual learning via manifold expansion replay. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 733–738, 2023. doi: 10.1109/SMC53992.2023.10393967. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5824–5836. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 3fe78a8acf5fda99de95303940a2420c-Paper.pdf. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094–1100. PMLR, 2020b. Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning, 2018. URL https://arxiv.org/abs/ 1804.08328. Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987–3995. PMLR, 2017. Tiantian Zhang, Zichuan Lin, Yuxing Wang, Deheng Ye, Qiang Fu, Wei Yang, Xueqian Wang, Bin Liang, Bo Yuan, and Xiu Li. Dynamics-adaptive continual reinforcement learning via progressive contextualization. IEEE Transactions on Neural Networks and Learning Systems, 35(10):14588– 14602, 2024. doi: 10.1109/TNNLS.2023.3280085. Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: Variational Bayes-Adaptive Deep RL via Meta-Learning. J. Mach. Learn. Res., 22(1), January 2021. ISSN 1532-4435. 14 Preprint A MASK-SCORE SENSITIVITY AND QUANTIZED STORAGE Motivation and relation to binary supermasks. AMSC represents task i in layer ℓ by a realvalued score tensor Siℓ over a frozen weight tensor W ℓ . The forward pass uses a binary gate
   Miℓ = I[Siℓ ≥ 0], hℓ+1 = σ (W ℓ ⊙ Miℓ )hℓ, (10) and propagates gradients to Siℓ with a straight-through estimator (STE). This follows Wortsman et al. (2020); Ben-Iwhiwhu et al. (2023), which optimize a subnetwork over a frozen backbone. Related signed-supermask work allows connections to be kept, hidden, or inverted (Koster et al., 2022). AMSC instead keeps the deployed gate in Equation 10 binary. It retains continuous scores for the linear combination before the gate is applied in the forward pass on the resulting mask. Score magnitude can therefore affect composition even though only the score sign determines an individual task’s subnetwork. Wortsman et al. (2020) also superimposes learned binary masks, but optimizes their mixture to infer a task at evaluation time. AMSC composes completed policies during target-task learning, then consolidates the composition as the next reusable policy. The following experiments test whether continuous score information is required for this transfer mechanism and whether completed masks can instead be stored in quantized form. Magnitude-controlled perturbations. We first test whether score magnitude identifies stable gates. For a score tensor with Dℓ entries, define its root-mean-square (RMS) scale as v u Dℓ u 1 X riℓ = t (S ℓ )2 . (11) Dℓ j=1 i,j For a selected index set I, each score receives independent zero-mean Gaussian noise,
   ℓ ℓ Sei,j = Si,j + ϵℓi,j, ϵℓi,j ∼ N 0, (αriℓ )2, j ∈ I. (12) Zero-mean noise preserves each score in expectation. RMS scaling makes α dimensionless and comparable across layers with different score scales. Thus, α = 0.5 sets the noise standard deviation to one half of the corresponding layer RMS. For a nonzero score s, the probability that Equation 10 changes state is   |s| (13) Pr[I[s + ϵ > 0] ̸= I[s > 0] | s] = Φ − ℓ, αri where Φ is the standard-normal cumulative distribution function. Equation 13 predicts frequent gate changes near zero and rapidly fewer changes as normalized magnitude increases. The structural sweep perturbs the lowest-magnitude, random, or highest-magnitude fraction of each tensor. We evaluate fractions f ∈ {0.10, 0.25, 0.50} and scales α ∈ {0.01, 0.05, 0.10, 0.25, 0.50, 1.00} using final checkpoints from the seven CT28 and MG16 seeds. Matched-state policy analysis perturbs actor-relevant trunk and head scores and measures deterministic-action disagreement. Fresh-rollout return estimates use three episodes per task and serve as a diagnostic rather than an equivalence test. Perturbation results. Across actor heads and shared trunks, 3.7–4.8% of scores lie below 0.05 RMS and 7.1–9.4% lie below 0.10 RMS. At α = 0.5, perturbing the lowest-magnitude quartile changes 37.3% of the selected CT28 gates and 39.3% of the selected MG16 gates. The corresponding rates are 14.2% and 16.4% for random scores, but only 0.2% and 0.5% for the highestmagnitude quartile. Low-magnitude perturbations also change the selected action on 6.5% of CT28 states and 6.2% of MG16 states, compared with 0.8% for high-magnitude perturbations on both benchmarks. One-sided Wilcoxon signed-rank tests support the paired low-versus-high differences in CT28 (p = 0.0078) and MG16 (p = 0.017). Mean task-return changes at α = 0.5 are −0.005 on CT28 and −0.007 on MG16 for low-magnitude perturbations. The low-versus-high return differences are not significant (p = 0.60 and p = 0.61, respectively). Score magnitude therefore predicts gate and action stability, but the diagnostic rollout data do not establish that low-magnitude scores can be removed without a performance cost. The difference between action changes and mean-return changes also indicates behavioral redundancy in the learned subnetworks. 15 100% 80% 60% 40% 20% 0% CT28 MG16 shared trunk actor head critic head 10 4 10 6 10 2 100 Normalized magnitude |S|/RMS(S) 10 5 10 1 101 10 3 Normalized magnitude |S|/RMS(S) Perturbed scores changing gate (%) Scores with magnitude x (\%) Preprint (a) Score-magnitude distributions. 10.0% 0.0% 0.2 0.4 0.6 0.8 Noise scale (layer RMS) 20.0% 10.0% 0.0% 1.0 0.2 0.4 0.6 0.8 Noise scale (layer RMS) 0.0 0.2 0.4 0.6 0.8 Noise scale (layer RMS) 1.0 0.0 CT28 low random high 20.0% low random high 30.0% MG16 Magnitude region 30.0% MG16 Magnitude region 0.2 0.4 0.6 0.8 Noise scale (layer RMS) 1.0 (b) Gate changes under perturbation. Mean task-return change Action disagreement (%) CT28 CT28 40.0% 1.0 MG16 0.04 0.02 0.00 0.02 0.04 0.02 0.00 0.02 low random high Magnitude region ( = 0.5 × RMS) low random high Magnitude region ( = 0.5 × RMS) (d) Diagnostic return change at α = 0.5. (c) Action disagreement on matched states. Figure 3: Mechanistic analysis of continuous mask scores. Panel (a) gives the empirical cumulative distribution of up to 2,048 sampled scores per task–layer tensor, normalized by layer RMS. Panels (b)–(d) perturb 25% of scores from each magnitude region. Lines report seed means; bands in (b) and (c) span seed minima and maxima. Return estimates in (d) use three episodes per task. Quantized mask libraries. The second experiment tests whether continuous completed-task scores are required for subsequent composition. Binary storage retains only score sign,  +1, s > 0, Qbin (s) = (14) −1, s ≤ 0, whereas ternary storage introduces an RMS-scaled dead zone,  +1, s > δr, Qδ (s; r) = 0, |s| ≤ δr,  −1, s < −δr. (15) Increasing δ widens the dead zone and maps a larger fraction of the library to zero. The value 0 removes a near-boundary connection. The value −1 remains inactive when a stored task is evaluated alone. These values differ during composition because AMSC combines score tensors before thresholding. For a new task k, the composed score is X Sbkℓ = ak,k N (Skℓ ) + ak,i N (Q(Siℓ )), Mkℓ = I[Sbkℓ > 0], (16) i∈P(k) where N is the L2-STE transformation and Q is either Equation 14 or Equation 15. After task k, its consolidated score is quantized before entering the reusable library. The experiment therefore evaluates online learning with quantized sources rather than post-hoc compression of an already completed run. Quantization results. Float32 AMSC obtains AUC/FWT/FGT of 0.941/0.935/0.000. Binary storage reduces AUC/FWT to 0.750/0.709 despite preserving each stored score’s sign. Score sign alone therefore does not preserve the information required by later composition. Ternary storage reaches its best trade-off at δ = 0.50, obtaining 0.935/0.912/0.041 while setting 29.9% of stored scores to zero. Paired tests against Float32 do not reject zero AUC (p = 0.43) or FWT difference (p = 0.22), although this does not establish equivalence. FGT increases by 0.041 (p = 0.031). At δ = 0.625, AUC/FWT/FGT is 0.915/0.908/0.056 with 41.4% zeros; the AUC reduction is significant (p = 0.023) while the FWT reduction is not (p = 0.086). Increasing δ further monotonically increases sparsity but reduces acquisition and retention, reaching 0.624/0.568/0.265 at δ = 1. Thus, the dead zone can remove near-boundary connections, but aggressive thresholding deletes gates required by completed policies. 16 Preprint Ternary threshold sweep 1.0 Retention cost 0.3 0.8 FGT Performance 0.9 0.7 0.1 0.6 0.5 AUC 0 0.1 FWT 0.25 0.5 1.0 0 Normalization at = 0.5 AUC FWT Performance 0.8 Performance 0.0 0.625 0.75 0.875 Ternary threshold (RMS); 0 is binary 1.0 0.6 0.4 0.2 0.0 0.2 Float32 Ternary full L2 Ternary prior L2 1.00 0.75 0.50 0.25 0.00 0.25 0.50 Ternary no L2 0.1 0.25 0.5 0.625 0.75 0.875 1.0 Ternary threshold (RMS); 0 is binary Matched fixed-backbone LoRA controls AUC Float32 mask Ternary mask LoRA-4 local LoRA-8 local FWT LoRA-4 composed 428.1 Fixed-width storage Empirical sparsity 102 48.4 26.8 24.2 Mask 2-bit LoRA-4 LoRA-8 LoRA-8 float32 float32 float16 24.2 13.4 Mask float32 Mask 1-bit Stored zero scores (%) Theoretical storage (KiB/task) (a) Quantization, normalization, and matched LoRA controls. 60 40 20 0 0.1 0.25 0.5 0.625 0.75 0.875 Ternary threshold (RMS) 1 (b) Theoretical storage and ternary sparsity. Figure 4: Quantized storage of completed task masks on CT28. In (a), dashed lines in the threshold sweep give Float32-mask performance and error bars denote asymmetric 95% bootstrap confidence intervals across seven seeds. The LoRA controls use the same frozen signed-constant backbone. Storage in (b) counts packed values only and excludes container and index overhead. p Why normalization is necessary. For a ternary tensor, ∥Qδ (Siℓ )∥F = nℓi, where nℓi is its number of nonzero entries. Without normalization, the effective influence of source i is therefore prop portional to ak,i nℓi, rather than the intended similarity coefficient ak,i . The trainable current score introduces an additional arbitrary continuous scale. L2-STE gives every component unit forward norm, making the composition coefficients comparable while retaining the identity gradient for the current score. This calibration is particularly important after ternary quantization because all retained entries have magnitude one, leaving density as the only source-dependent tensor scale. At δ = 0.50, removing normalization yields AUC/FWT/FGT of 0.283/0.070/0.291. Normalizing only the priors also fails (0.269/0.049/0.005) because unit-norm sources remain incommensurate with the raw current score. Joint normalization is therefore required for sparsemax weights to control actual mask contributions. Storage comparison. CT28 contains Dm = 109,600 mask scores per task. Float32 scores require 428.1 KiB per task. Packed binary and ternary scores require 13.4 and 26.8 KiB, respectively. For comparison, a rank-r LoRA adapter (Hu et al., 2022) over the same layers stores DLoRA (r) = r X out (din ℓ + dℓ ) = 1,548r ℓ 17 (17) Preprint values per task. Rank-4 and rank-8 Float32 adapters therefore require 24.2 and 48.4 KiB; rank-8 Float16 also requires 24.2 KiB. Ternary masks are similar in theoretical size to rank-4 LoRA and smaller than rank-8 Float32 LoRA, whereas binary masks are smaller than both. A task-local rank-4 adapter obtains AUC/FWT/FGT of 0.198/−0.231/0.000, and rank 8 obtains 0.201/−0.150/0.000. Both fall below the 0.22 AUC of full from-scratch learning. They combine a low-rank capacity constraint with no mechanism for reusing earlier solutions, which is especially restrictive in the hierarchical CT28 curriculum. In contrast, normalized-sparsemax retrieval and composition of rank-4 deltas raises performance to 0.885/0.870/0.000. At 24.2 KiB per task, this adapter is slightly smaller than a packed ternary mask, but remains below the latter by 0.050 AUC (paired 95% CI [0.020, 0.087]) and 0.042 FWT ([−0.016, 0.101]). Composed LoRA must express and recompress the transferred update within a rank-4 subspace, whereas mask composition can independently combine all frozen backbone connections. The LoRA controls retain zero measured forgetting, while ternary quantization incurs a small boundary loss of 0.041 ([0.020, 0.061]) when a completed continuous score is replaced by its quantized representation. These results suggest that quantized masks can approach low-rank-adapter storage while preserving stronger composition in this setting. 18 Preprint B IMPLEMENTATION DETAILS B.1 WASSERSTEIN TASK EMBEDDING CONSTRUCTION AMSC constructs WTEs from experience generated by the current policy (Liu et al., 2025). The current-task buffer Dk stores transitions (s, a, r, s′, d). At an embedding update, AMSC samples N (s, a, r) entries uniformly with replacement from Dk . The stored state has already received the benchmark-specific policy-input scaling. Let ds denote the flattened state dimension and let da denote the action-feature dimension. Discrete actions are one-hot encoded; continuous actions retain their vector representation:  onehot(an ) ∈ Rda, discrete actions, (18) ϕ(an ) = an ∈ Rda, continuous actions. The state block is standardized independently for each sampled batch. The implementation computes one scalar mean and standard deviation over all N ds state entries, rather than separate statistics for each state coordinate. For batch update t, (t) µ(t) s = meann,j sn,j, σs(t) = stdn,j sn,j, sbn = sn − µs (t) . (19) max(σs, εx ) The implementation uses εx = 10−8 . The input row and embedding batch are then (t) ⊤ ⊤ xn = [b s⊤ n, ϕ(an ), rn ], Xk = [x1, . . ., xN ]⊤ ∈ RN ×dx, (20) where dx = ds + da + 1. The reward is used at the scale stored by the RL replay buffer; WTE applies no additional reward normalization. At initialization, AMSC samples a reference matrix X0 = [x01, . . ., x0R ]⊤ ∈ RR×dx, with each coordinate sampled independently from U[0, 1]. A seeded random generator draws X0 once, and the same ordered reference is retained for all tasks in a run. The reference is not estimated from a task and is not updated during training. Reusing X0 fixes the coordinates and dimension of every task embedding. The empirical batch and reference distributions have uniform masses a= 1 1N, N b= 1 1R . R (21) Let Cn,r = ∥xn − x0r ∥22 be entry (n, r) of the cost matrix C. AMSC uses the unregularized Earth Mover’s Distance solver in the Python Optimal Transport library to compute  ×R Γ⋆ = argmin ⟨Γ, C⟩F, Π(a, b) = Γ ∈ RN : Γ1R = a, Γ⊤ 1N = b . (22) + Γ∈Π(a,b) Since each reference point receives mass 1/R, its barycentric projection into the current empirical distribution is N X yr = R Γ⋆n,r xn . (23) n=1 The WTE records the displacement of every ordered reference point under this projection:
   1 (t) ek = √ vec [y1 − x01, . . ., yR − x0R ]⊤ ∈ RRdx . R (24) Equation equation 24 is the expanded form of Equation equation 4. The construction contains no learned encoder or embedding loss. (1) (1) The first update initializes ēk = normalize(ek ). Subsequent updates combine the stored normalized representation with the raw batch embedding, then normalize the result,   (t) (t−1) (t) ēk = normalize ρēk + (1 − ρ)ek . (25) 19 Preprint Here, normalize(e) = e/ max(∥e∥2, εe ), with εe = 10−8 and ρ = 0.5. This is the update in Equation equation 5 used for the reported CT28 and MG16 experiments. Completed embeddings are frozen at task boundaries, and the current-task buffer is cleared before the next task. Every retrieval event therefore compares an evolving current-task estimate with fixed representations of the completed tasks. The reported CT28 and MG16 experiments use R = 50. CT28 samples N = 128 transitions and updates both the embedding and retrieval support every PPO iteration. MG16 samples N = 1024, updates the embedding every iteration, and updates the retrieval support every five iterations. The CW10 implementation uses R = 50, N = 1000, and an update period of 1000 environment steps. B.2 PSEUDO-CODE Algorithm 1 gives the AMSC training procedure. Task boundaries determine when a completed mask and embedding are frozen and added to their respective libraries. Embedding and retrieval updates use periods femb and fsel, respectively. Retrieval weights are not optimized by the RL loss; gradients update only the current task’s mask scores. Algorithm 1 AMSC training with adaptive similarity-guided selection Require: Task sequence {Tk }K k=1, per-task training budget B, rollout length z Require: WTE batch size N, reference size R, EMA factor ρ, sparsemax temperature τ Require: Update periods femb and fsel 1: Initialize frozen backbone {W ℓ }L ℓ=1, mask library M, and embedding table E 2: Sample and fix reference matrix X0 ∈ RR×dx 3: for k = 1 to K do 4: Initialize current mask-score matrices {Skℓ }L ℓ=1 5: Set Dk ← ∅, P(k) ← ∅, steps ← 0, and t ← 0 6: while steps < B do 7: Form {Ckℓ }L ℓ=1 and the binary policy masks using Equation equation 9 8: Collect a rollout of z transitions using the resulting policy 9: Append transitions to Dk ; update steps ← steps + z and t ← t + 1 10: Compute the RL loss and update only {Skℓ }L ℓ=1 through Equation equation 8 11: if |Dk | ≥ N and t mod femb = 0 then (t) 12: Sample N tuples from Dk and construct Xk (t) 13: Compute ek using Equations equation 22–equation 24 14: if this is the first embedding update for task k then (t) 15: Set ēk ← normalize(ek ) 16: else 17: Update ēk using Equation equation 5 18: end if 19: end if 20: if k > 1, ēk exists, and t mod fsel = 0 then 21: Compute sk, wk, and P(k) using Equations equation 6–equation 7 22: end if 23: end while 24: Recompute {Ckℓ }L ℓ=1 using Equation equation 9 25: Store Skℓ,∗ ← Ckℓ for every ℓ in M, and store ēk in E 26: end for 20 Preprint C BENCHMARKS For CT28 and MG16, we use family to denote tasks that share an observation encoding and a nested task construction across depths. All tasks within each benchmark have the same formal observation dimensions and action interface. Cross-family tasks differ in their observation dataset or symbolic remapping, rather than in tensor shape, and can still share behavioral structure that supports policy transfer. CT-Graph CT28. CT-Graph is a configurable decision-tree environment for continual RL (Soltoggio et al., 2023). At each decision state, the agent selects a branch and receives a sparse reward of 1 only after reaching the target terminal node. Each state is represented by an image generated from a family-specific image dataset. Datasets do not overlap across families, whereas tasks within one family share observations for their common graph prefix. CT28 is based on the evaluation setup used in (Nath et al., 2026), and contains four families with depths 2 through 8, giving 28 tasks. Within each family, the solution at depth d forms a prefix of the solution at depth d + 1. A depth-8 target is encountered after 24–27 previous tasks. Six of the available policies are lower-depth members of the target family, and the remaining policies belong to other families. CT28 therefore tests retrieval from a library in which relevant predecessors form a minority that decreases as the curriculum progresses. MiniGrid MG16. MiniGrid provides partially observable grid-world navigation tasks with symbolic object, color, and state observations (Chevalier-Boisvert et al., 2018). MG16 extends MultiRoom into four room-chain families. Each family contains tasks R2 through R5. Under a fixed layout seed, Rd + 1 preserves the complete Rd layout and adds one room and connecting transition. The agent receives sparse reward for reaching the goal in the final room. The action space and physical task remain fixed across families. MG16 assigns each family a distinct symbolic remapping of wall, door, and floor cells. The remapping changes the observed object type and colour but preserves collision, traversal, and door-toggle affordances. Each remapping remains fixed across depth, so lower-depth policies share the target family’s observation code. Figure 5 shows the four profiles. Tasks are ordered by room depth, giving four R2 tasks, followed by four R3, R4, and R5 tasks. No family identifier is included in the policy observation. K K K K W W W W W K W ▲ W W W W D W W W K W W W W D W W W B W W W W W W W W W W B W W D B D W L K L K K K W W G W W W W W K K K K B K L L L L O L O K L K L K L L G L O O L O L O L O L O K F K W W W B K F W W W L K B W ▲F K L K K K B W K K K K K B K K L K L K K K L K B K K K G K K K L L K L K L K L ▲ O K L K L L F K L L L L L L L L L O L L L K O L K L L L L L L L L L L L L L O O O O O B O O O O G O O O O L O O O O O O O L O O O O O B O O O O B L O O O ▲ O B O B O L O O O O O O Figure 5: Depth-5 layouts for the four MG16 task families. Symbols denote the observed representation of each functional cell; the underlying wall, door, and floor affordances remain unchanged. Identity uses grey walls (W), blue ball floors (B), and yellow doors (D). KeyBoxSwap uses yellow key walls (K), grey floors (F), and green box doors (B). LavaBall uses green lava walls (L), yellow key floors (K), and blue ball doors (O). BallLava uses purple ball walls (O), purple box floors (B), and red lava doors (L). Continual World CW10. Continual World evaluates continual RL on sequences of MetaWorld robotic manipulation tasks (Wolczyk et al., 2021). We use one pass through the tentask CW10 curriculum: hammer, push-wall, faucet-close, push-back, stick-pull, handle-press-side, push, shelf-place, window-close, and peg-unplug-side. Each task is trained for 1M environment steps with SAC, giving 10M steps in total. CW10 pro21 Preprint vides shaped rewards and four-dimensional continuous actions. It therefore tests AMSC beyond the sparse-reward, discrete-action conditions of CT28 and MG16. Our experiments use the current MetaWorld-v2 goal-observable task definitions (Yu et al., 2020b). Each observation is a 39-dimensional state vector, and no one-hot task label is appended. Initial object and goal configurations remain randomized across episodes. Evaluation uses binary task success rather than shaped return. An episode is successful if the environment reports success at any step before termination. Every 50,000 training steps, the deterministic policy is evaluated for ten episodes on every task encountered so far. Each evaluation uses a fresh environment and a task-specific seed, and does not alter the training environment or replay buffer. The resulting success-rate matrix supplies the CW10 AUC, FWT, and FGT measurements. Figure 6: The ten MetaWorld-v2 tasks in the CW10 curriculum, shown in execution order from left to right and top to bottom Wolczyk et al. (2021). 22 Preprint D BASELINES Comparison protocol. All methods use seven seeds, identical task orders, and the same interaction budget. CT28 and MG16 use the common PPO architecture and optimizer settings reported in Section 5. Task labels are used at task switches and evaluation but are not included in policy observations or policy selection. CW10 uses the SAC schedule and evaluation protocol in Appendix C. Architectures are matched where the algorithms permit direct substitution. CompoNet and ProgNet retain their released two-layer, 256-unit ReLU models and a matching no-reuse run is used to calculate their FWT. Other CW10 methods use the four-layer, 256-unit CW architecture with first-layer LayerNorm. Learning references and regularization. Fine-tuning updates one policy throughout the curriculum without a retention mechanism. No-reuse trains an independent policy on each task and provides the zero-transfer learning curves used to calculate FWT. Replay. SER (Isele & Cosgun, 2018) maintains short-term FIFO memory and bounded episodic memory. We report surprise-prioritized and reservoir-sampled SER variants. SDW (Huang et al., 2025) estimates task similarity and uses it to control replay and policy/value consistency weights. These methods select or weight stored experience rather than stored policy modules. Parameter isolation and routing. ProgNet (Rusu et al., 2016) adds a frozen policy column for each task and learns lateral connections from earlier columns. OWL (Kessler et al., 2022) combines task-specific heads with oEWC on shared features and selects a head using an exponential-weights bandit. SANE (Powers et al., 2022) maintains an ensemble of actor–critic modules and activates a module using optimistic value uncertainty. Our task-incremental SANE configuration clones the selected module at each task boundary and freezes earlier modules. Policy composition. Mask-LC and Mask-BLC (Ben-Iwhiwhu et al., 2023) learn softmax coefficients over the current mask and every earlier mask. Mask-LC initializes the active coefficients uniformly. Mask-BLC initially allocates half of the mass to the current mask and divides the remainder among the earlier masks. CKA-RL (Hu et al., 2025) learns task-specific parameter residuals around a shared policy and fuses residuals into a bounded knowledge pool. CompoNet (Malagón et al., 2024) adds one policy unit per task and uses learned attention to combine the outputs of frozen earlier units with the current unit. These baselines learn routing or composition without first constructing a library-relative sparse support. Similarity-guided composition. MOSAIC (Nath et al., 2026) was introduced for knowledge exchange among concurrently learning agents. We adapt MOSAIC as a baseline, limiting candidates to frozen masks from earlier tasks but retaining MOSAIC’s absolute cosine threshold, performance gate, learned composition, and acquisition protocol. A successful acquisition consolidates the active composition and reinitializes its coefficients from source and target performance. We also evaluate MOSAIC without the performance gate, denoted MOSAIC ¬Iperf . 23 Preprint E HYPERPARAMETERS Table 5 reports the principal AMSC configuration and shared optimization settings. Table 6 reports method-specific settings for the CT28 and MG16 baselines. Table 7 reports the corresponding CW10 settings. Methods share the optimization and evaluation protocol for each benchmark unless stated otherwise. Table 5: Hyperparameters, seeds, and training budgets for the principal AMSC experiments. One CT28 or MG16 PPO iteration collects 4×128 = 512 environment samples. The CW10 configuration uses the 39-dimensional MetaWorld-v2 observations. Component Hyperparameter CT28 MG16 CW10 Optimization Learner Network width Activation PPO 3 × 200 ReLU PPO 3 × 200 ReLU Optimizer Actor/critic learning rate Number of workers Discount factor GAE parameter Entropy coefficient Rollout length per worker Optimization epochs Number of mini-batches PPO ratio clip Gradient clipping Replay-buffer size Replay batch size Learning starts Initial random actions Update interval Gradient steps per update Target smoothing coefficient RMSprop 1.5 × 10−4 4 0.99 0.99 0.1 128 8 64 0.1 5 — — — — — — — RMSprop 1.5 × 10−4 4 0.99 0.99 0.01 128 8 64 0.1 5 — — — — — — — SAC 4 × 256 Tanh (first), LeakyReLU (remaining) Adam 10−3 /10−3 1 0.99 — Automatic — — — — — 106 128 1,000 steps 10,000 steps 50 steps 50 0.005 Task representation Reference size R Embedding sample count N Embedding-update frequency Selection frequency EMA coefficient ρ Similarity normalization Sparsemax temperature τ Mask-score normalization Mask type and scope Critic WTE–SAR 50 128 1 iteration 1 iteration 0.5 z-score 1.0 ℓ2-STE Discrete actor– critic Masked WTE–SAR 50 1024 1 iteration 5 iterations 0.5 z-score 1.0 ℓ2-STE Discrete actor– critic Masked LayerNorm None None Budget per task Total budget 51,200 steps 1,433,600 steps 512,000 steps 8,192,000 steps Evaluation interval Evaluation episodes 10 iterations 10 20 iterations 10 Experiment seeds CT-Graph configuration seed CT-Graph image seeds 3 1, 2, 3, 4 AMSC Training Seeds 24 WTE–SAR 50 1,000 1,000 steps 1,000 steps 0.5 z-score 1.0 ℓ2-STE Discrete full actor Dense, tasklocal Task-specific affine 1,000,000 steps 10,000,000 steps 50,000 steps 10 per seen task 86, 87, 88, 89, 90, 91, 92 — — — — Preprint Table 6: Method-specific CT28 and MG16 hyperparameters for parameter isolation, routing, and policy-composition methods. Algorithm Hyperparameter Value (CT28/MG16) SER Storage strategy Episodic-memory size FIFO-memory size Batch size Replay epochs FIFO ratio Coverage distance Surprise/reservoir 20,000 2,000 256 1 0.5 1.0 SDW Buffer size Batch size Probe batch size Base replay ratio Similarity smoothing Maximum policy/value weights Minimum consistency weight Similarity-feature weights 105 256 128 0.8 2 0.01/0.005 −10−4 0.4, 0.4, 0.2 OWL oEWC coefficient Bandit learning rate Bandit decay Exploration ϵ 500 0.88 0.9 0 SANE Maximum modules Uncertainty scale Task-boundary cloning CT28: 28; MG16: 16 1.0 Enabled Mask-LC Mask type Prior support Coefficient initialization Discrete Full library Uniform Mask-BLC Mask type and support Initial composition mass Same as Mask-LC Current: 0.5; all priors: 0.5 CKA-RL Knowledge-pool capacity α scale α learning rate α initialization Major mass Fixed/global/scaled α Fused components ∆Θ mode Reset critic 5 10−2 2.5 × 10−4 Random normal 0.6 Fixed: no; global: yes; scaled: yes Actor: yes; encoder: no T Enabled CompoNet Attention dimension Composed output 200 Policy logits MOSAIC-LL Task representation Reference size R Embedding sample count N Cosine-similarity threshold Selection cap Selection frequency Performance gate Acquisition update Coefficient initialization WTE–SAR 50 128 0.5 None CT28: 10; MG16: 25 iterations Source return > current target return Consolidate composition and reset coefficients Reward-guided Performance gate Disabled; otherwise as MOSAIC-LL MOSAIC ¬Iperf 25 Preprint Table 7: Method-specific CW10 baseline hyperparameters. Compatible methods use the shared SAC configuration in Table 5. CompoNet and ProgNet retain their released architectures and use the matched native no-reuse run as their FWT reference. Algorithm Hyperparameter CW10 value No-reuse Network 4×256, first-layer LayerNorm No-reuse (for CompoNet and ProgNet) Network 2×256, ReLU, no LayerNorm Mask-LC Mask type Mask scope LayerNorm Prior support Composition Coefficient initialization Continuous Actor and critics, trunks and heads Task-specific affine Full library Learned Uniform Mask-BLC Mask type, scope, and support Initial composition mass Same as Mask-LC Current: 0.5; all priors: 0.5 CKA-RL Knowledge-pool capacity α initialization α scale Major mass Fixed/global/scaled α Fused components ∆Θ mode 8 Random normal 10−3 0.6 Fixed: no; global: yes; scaled: yes Heads: yes; shared trunk: no T CompoNet Network Composition 2×256, ReLU, no LayerNorm Learned attention ProgNet Network Prior columns 2×256, ReLU, no LayerNorm Frozen MOSAIC-LL Mask type and scope LayerNorm Task representation Reference size R Embedding sample count N Selection frequency Cosine-similarity threshold Performance estimate Continuous full network Task-specific affine WTE–SAR 50 1,000 1,000 steps 0.5 Mean success over 10 recent online episodes Source success > current target success Consolidate composition and reset coefficients Success-guided Performance gate Acquisition update Coefficient initialization 26 Preprint F ADDITIONAL RESULTS F.1 EARLY ACQUISITION We compare early acquisition against MOSAIC, which also retrieves masks for composition, and the surprise- and reservoir-based SER replay methods. AUC20 averages the first two of ten target-task evaluations, covering the first 20% of each CT28 task budget. Time to first reward is the end of the first 512-step training iteration with positive mean reward; time to first positive evaluation return is measured at evaluation checkpoints. Both times are right-censored at the 51,200-step task budget. Aggregate statistics exclude task 0 because no source policy is then available. Across tasks 1–27, AMSC obtains AUC20 = 0.86, compared with 0.35 for MOSAIC (two-sided paired p = 0.016). Its first positive training reward appears after 2.34k rather than 5.83k steps (p = 0.016). SER-surprise achieves AUC20 = 0.91 and reaches positive reward after 1.59k steps; SER-reservoir obtains 0.86 and 3.27k. Neither SER variant differs significantly from AMSC on early AUC or time to first reward. The MOSAIC comparison supports an early advantage for AMSC within mask-based composition; it does not imply an early advantage over replay. Tables 8 and 9 provide the aggregate and per-task results. Table 8: CT28 early acquisition over tasks 1–27 across seven matched seeds. Entries are means with conservative 95% bootstrap CI. T+ and Teval+ are time to first positive training reward and evaluation return, in thousands of environment steps, right-censored at 51.20k. Exact two-sided paired sign-flip p-values compare AMSC with each method; no multiple-comparison adjustment is applied.

Method

AMSC MOSAIC SER-s SER-r

F.2

AUC20

T+ (k)

Teval+ (k)

0.86 ± 0.02 2.34 ± 0.91 8.05 ± 1.19 0.35 ± 0.03 5.83 ± 0.84 14.44 ± 1.35 0.91 ± 0.04 1.59 ± 0.72 7.12 ± 1.38 0.86 ± 0.05 3.27 ± 1.98 10.10 ± 2.65

pAUC20

p T+

pTeval+

— 0.016 0.125 1.000

— 0.016 0.312 0.516

— 0.016 0.422 0.281

SIMILARITY- DERIVED COMPOSITION VS . LEARNED COMPOSITION IN CW10

We compare the composition weights at the end of training of AMSC and Mask-BLC. Figure 7 shows the coefficient assigned to the current-task mask at each layer, alongside within-task evaluation success. AMSC uses one similarity-derived allocation across its masked actor layers. MaskBLC learns separate weights for each actor and critic layer. Averaged over tasks 1–9, Mask-BLC assigns approximately 0.87 to the current mask in its middle actor layers, but only 0.01 in the action-mean head. Thus, a single task-level similarity-derived weighting cannot recover the layerdependent composition learned by Mask-BLC in CW10. The largest mean Mask-BLC–AMSC gaps occur on task 9 (peg-unplug-side, 0.92 versus 0.63 mean evaluation success) and task 3 (push-back, 0.41 versus 0.23). The task-9 gap occurs in all seven paired seeds, whereas the task-3 gap is less consistent. AMSC also exceeds Mask-BLC on task 6 (0.51 versus 0.22), so the difference is not uniform across the curriculum. Task 3 remains above its from-scratch reference; it explains part of the between-method performance gap, but not AMSC’s negative forward transfer. Normalized FWT can additionally amplify small deficits on tasks whose from-scratch performance is close to one. Figure 8 identifies the prior tasks receiving actor coefficient weight. AMSC’s sparse support excludes some sources entirely, while Mask-BLC distributes learned coefficients over the available library. F.3

HYPERPARAMETER SENSITIVITY

Each sweep changes one AMSC hyperparameter while retaining a benchmark-specific sweep reference for all others. Table 10 reports every value considered. Frequencies are measured in PPO iterations, and bold values denote the sweep-reference settings. Each setting contains seven seeds. The reported performance intervals are the minimum and maximum of the seven-seed means across the swept values, rather than confidence intervals. 27

Preprint

Table 9: Per-task CT28 early acquisition across seven matched seeds. The first four numeric columns are AUC20 ; the remaining four give T+ in thousands of environment steps. Task A–D2 (†) has no previous source and is excluded from the aggregate comparison. AUC20

T+ (k steps)

Task

AMSC MOSAIC SER-s SER-r

AMSC MOSAIC SER-s SER-r

A–D2† B–D2 C–D2 D–D2

1.00 0.79 0.93 0.79

1.00 1.00 1.00 0.86

1.00 0.64 1.00 0.71

0.93 0.79 1.00 0.86

0.95 0.66 0.80 1.10

0.95 0.73 0.66 1.46

0.66 2.56 0.51 1.68

0.73 1.76 1.02 1.17

A–D3 B–D3 C–D3 D–D3

1.00 1.00 1.00 0.93

0.64 0.29 0.50 1.00

1.00 1.00 0.93 1.00

1.00 1.00 0.93 1.00

0.80 0.95 0.66 0.88

2.56 1.39 3.00 1.46

0.51 0.51 0.59 0.51

0.51 0.51 0.51 0.51

A–D4 B–D4 C–D4 D–D4

1.00 1.00 0.71 1.00

0.43 0.50 0.07 0.36

1.00 1.00 1.00 0.86

1.00 1.00 1.00 1.00

0.95 0.80 0.95 1.10

3.51 4.17 3.07 4.90

0.51 0.51 0.51 0.51

0.51 0.51 0.51 0.51

A–D5 B–D5 C–D5 D–D5

0.93 1.00 1.00 1.00

0.00 0.21 0.14 0.14

1.00 1.00 1.00 0.93

1.00 1.00 1.00 1.00

1.10 1.02 1.02 0.80

2.19 6.88 4.75 5.19

0.51 0.51 0.51 1.24

0.51 0.51 0.51 0.51

A–D6 B–D6 C–D6 D–D6

1.00 1.00 0.14 1.00

0.07 0.14 0.00 0.43

1.00 1.00 1.00 0.57

1.00 1.00 1.00 0.93

0.88 1.02 6.07 0.88

7.68 7.17 6.80 6.44

0.51 0.51 0.80 5.92

0.59 0.51 0.66 0.51

A–D7 B–D7 C–D7 D–D7

1.00 0.93 0.50 1.00

0.64 0.00 0.00 0.43

1.00 0.79 0.86 0.79

1.00 0.29 0.29 0.86

1.10 1.02 11.34 1.02

5.19 8.27 10.83 10.02

0.66 0.80 2.12 7.97

0.51 8.70 5.63 8.05

A–D8 B–D8 C–D8 D–D8

1.00 0.71 0.50 0.43

0.50 0.00 0.00 0.07

1.00 0.93 0.93 0.64

0.64 0.43 0.43 0.79

0.88 1.24 17.85 6.29

7.83 10.90 17.04 13.39

0.51 0.66 0.73 10.02

0.66 15.07 22.24 15.07

Table 10: Sensitivity ranges for AMSC. Each AUC/FWT interval is the range of mean performance across the values in the adjacent sweep. All settings obtain zero measured FGT. N is the number of task samples, R is the reference-distribution size, and u is the reference seed. MG16 uses N = 512 outside the sample-count sweep. Bold marks sweep references. CT28 Parameter

Swept values

Temperature τ Samples N

0.25, 0.5, 1, 2, 4 32, 64, 128, 256, 512

Reference size R Embedding period femb Selection period fsel EMA coefficient ρ Reference seed u

10, 25, 50, 100, 200 1, 2, 5, 10 1, 2, 5, 10 0, 0.25, 0.5, 0.75, 0.9 11, 37, 71, 98, 131

MG16 AUC

FWT

Swept values

0.89–0.96 0.88–0.95 0.25, 0.5, 1, 2, 4 0.94–0.95 0.93–0.95 128, 256, 512, 1024, 2048 0.89–0.94 0.88–0.94 10, 25, 50, 100, 200 0.84–0.94 0.83–0.94 1, 2, 5, 10 0.83–0.94 0.81–0.94 1, 2, 5, 10 0.93–0.95 0.92–0.95 0, 0.25, 0.5, 0.75, 0.9 0.92–0.96 0.91–0.96 11, 37, 71, 98, 131

AUC

FWT

0.50–0.67 0.16–0.50 0.54–0.73 0.23–0.54 0.55–0.70 0.57–0.62 0.57–0.65 0.62–0.68 0.62–0.68

0.35–0.52 0.34–0.43 0.35–0.43 0.43–0.51 0.39–0.50

Tables 11– 17 report the complete setting-level results, including bootstrap confidence intervals. Representation construction and retrieval. The CT28 reference records AUC/FWT of 0.94/0.94 and remains stable, with both metrics remaining between 0.93 and 0.95. Moderate temperature changes are also tolerated. τ = 0.5 records 0.96/0.95, whereas flattening the retrieval distribution with τ = 4 reduces the result to 0.89/0.88. Reference sizes of 10, 25, and 50 remain near 0.94/0.94, while sizes of 100 and 200 obtain 0.89/0.88 and 0.91/0.90, respectively. MG16 is more sensitive to representation quality. The independently repeated reference configuration records 0.62/0.43; its difference from the earlier N = 512 reference run (0.62/0.37) is not significant (two-sided paired p = 0.92/0.41). Lowering τ to 0.5 raises the sweep mean to 28

Preprint

B Mask-BLC actor

C Mask-BLC critics

0.50

0.50

0.50

0.50

0.50

0.50

0.52

0.93

1.00

0.56

0.00

0.57

0.49

0.99

0.90

0.90

0.88

Faucet-close

0.50

0.50

0.50

0.50

0.50

0.50

0.20

0.89

0.75

0.23

0.00

0.52

0.64

0.83

0.57

0.17

0.18

Push-back

0.50

0.50

0.50

0.50

0.50

0.50

0.07

0.81

0.79

0.80

0.00

0.28

0.40

0.84

0.81

0.62

0.48

Stick-pull

0.36

0.36

0.36

0.36

0.36

0.36

0.12

0.77

0.95

0.78

0.00

0.33

0.20

0.64

0.70

0.59

0.40

Handle-press

0.45

0.45

0.45

0.45

0.45

0.45

0.16

0.68

0.70

0.42

0.00

0.26

0.59

0.85

0.59

0.13

0.12

Push

0.42

0.42

0.42

0.42

0.42

0.42

0.16

0.96

0.96

0.78

0.04

0.34

0.52

1.00

0.90

0.56

0.11

Shelf-place

0.29

0.29

0.29

0.29

0.29

0.29

0.20

1.00

1.00

0.92

0.00

0.34

0.21

0.86

0.97

0.51

0.08

Window-close

0.33

0.33

0.33

0.33

0.33

0.33

0.12

0.80

0.73

0.28

0.00

0.32

0.52

0.86

0.62

0.21

0.13

Peg-unplug

0.22

0.22

0.22

0.22

H1

H2

H3

H4

D Task outcomes

1.0 Mean within-task success

0.22

an me

0.22

0.18

1.00

1.00

0.71

H1

H2

H3

H4

st log

0.02

an me

E Task 9: Peg-unplug

1.0

0.35

0.62

1.00

0.89

0.16

0.00

H1

H2

H3

H4

out

st log

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2 AMSC Mask-BLC

0.0

4 5 6 Task index

0.0

0.2

0.4 0.6 0.8 Steps on task (million)

1.0

0.8 0.6 0.4 0.2 0.0

F Task 3: Push-back

1.0

0.8

1.0

Current-mask coefficient

AAMSC actor Push-wall

0.0

0.2

0.4 0.6 0.8 Steps on task (million)

1.0

Figure 7: CW10 task-end composition and acquisition across seven seeds. Top: mean current-task mask coefficients by task and layer. Bottom: mean evaluation success by task and learning curves for the two tasks with the largest Mask-BLC–AMSC success gaps. Curves use 20 evaluation points per task. Coefficients describe the saved training-time allocation, not the effective contribution of each mask after task-boundary consolidation.

0.43

Mask-BLC 1

0.40

0.07

0.27

0.30

0.50

0.13

0.33

0.09

0.29

0.06

0.10

0.07

0.28

0.05

0.07

0.23

0.09

0.19

0.04

0.11

0.14

0.10

0.04

0.05

0.06

0.07

0.03

0.03

0.05

0.06

0.03

0.35 0.52

0.39

0.09

0.05

0.18

0.31

0.21

8 9

0.24 0.11

0.03

0.35

0.05

0.42

0.07

0.05

Prior source task

0.15

0.8

0.08

0.6

0.07

1.0 Mean actor coefficient across layers and seeds

0.50

Target task

Target task

AMSC 1

0.4

0.05 0.16

0.2

0.23

0.06

0.08

0.07

0.11

0.06

0.06

Prior source task

0.0

Figure 8: Mean actor coefficient assigned to each prior source task, averaged over layers and seven seeds. Rows are target tasks and columns are earlier source tasks; gray cells denote sources not yet available. Only coefficients of at least 0.03 are annotated.

0.67/0.50, while τ = 4 obtains 0.50/0.16. Increasing N from 128 to 1024 changes AUC/FWT from 0.54/0.23 to 0.73/0.54, but 2048 samples reduce FWT to 0.30. Reference-size results are 29

Preprint

Table 11: Sparsemax-temperature sensitivity. Values are means with asymmetric 95% bootstrap confidence intervals across seven seeds. Bold entries denote the sweep-reference setting; All runs obtain zero measured FGT. CT28

MG16

AUC

FWT

FGT

AUC

FWT

FGT

0.25 0.5 1 2 4

0.93+0.04 −0.04 0.96+0.03 −0.03 0.94+0.02 −0.02 0.93+0.03 −0.03 +0.03 0.89−0.02

0.92+0.05 −0.05 0.95+0.04 −0.04 0.94+0.02 −0.03 0.92+0.03 −0.04 +0.04 0.88−0.03

0.00 0.00 0.00 0.00 0.00

0.25 0.5 1 2 4

0.63+0.07 −0.08 0.67+0.10 −0.09 0.62+0.08 −0.09 0.54+0.10 −0.12 +0.07 0.50−0.07

0.45+0.09 −0.08 0.50+0.12 −0.11 0.43+0.13 −0.12 0.31+0.12 −0.14 +0.18 0.16−0.22

0.00 0.00 0.00 0.00 0.00

Table 12: Task-sample-count sensitivity. Values are means with asymmetric 95% bootstrap confidence intervals across seven seeds. Bold entries denote the principal sample counts (CT28: N = 128; MG16: N = 1024). All runs obtain zero measured FGT. CT28

MG16

AUC

FWT

32 64 128 256 512

0.94+0.03 −0.04 0.95+0.02 −0.03 0.94+0.02 −0.02 0.95+0.03 −0.03 0.95+0.03 −0.04

0.93+0.04 −0.04 0.95+0.03 −0.03 0.94+0.02 −0.03 0.95+0.03 −0.04 0.95+0.03 −0.03

FGT 0.00 0.00 0.00 0.00 0.00

N 128 256 512 1024 2048

AUC

FWT

FGT

0.54+0.11 −0.12 0.57+0.12 −0.13 0.62+0.08 −0.09 0.73+0.06 −0.06 0.63+0.10 −0.13

0.23+0.22 −0.28 0.33+0.17 −0.18 0.43+0.13 −0.12 0.54+0.12 −0.14 0.30+0.29 −0.44

0.00 0.00 0.00 0.00 0.00

likewise non-monotonic: R = 10 obtains 0.70/0.52, while R = 25 obtains 0.55/0.35. These point estimates show that more samples do not monotonically improve acquisition. Update schedule and smoothing. Frequent updates are most important on CT28. Increasing either femb or fsel from 1 to 10 reduces AUC/FWT from 0.94/0.94 to 0.84/0.83 and 0.83/0.81, respectively. On MG16, increasing femb from 1 to 10 changes 0.62/0.43 to 0.57/0.34. Selection frequency is less ordered: the sweep-reference period of 5 has the highest FWT (0.43), whereas a period of 10 has the highest AUC (0.65). EMA smoothing is not required for the best point estimate in either sweep. CT28 peaks at ρ = 0.25 with 0.95/0.95, and MG16 peaks without EMA at 0.68/0.51. Changing the random reference seed produces AUC ranges of 0.92–0.96 and 0.62–0.68, and FWT ranges of 0.91–0.96 and 0.39–0.50, on CT28 and MG16, respectively. These sweeps describe one-factor sensitivity around the stated references; they do not establish robustness to joint hyperparameter changes or document how the principal configuration was selected. Retention is unaffected across all settings. The observed sensitivity concerns acquisition. F.4

PER-TASK ACQUISITION AND RETENTION

Tables 18 and 19 report task-level AUC, FWT, and FGT for AMSC and full-library learned composition. Table 20 additionally reports CW10 comparisons with Mask-LC and Mask-BLC. These results expose variation hidden by curriculum averages. The supplied MG16 per-task table summarizes an earlier result set: averaging its displayed task AUCs gives approximately 0.62, rather than the principal 0.73. It is retained as a separately identified analysis and must not be used as a decomposition of the current main-text aggregate. CT28 gains are broad but diminish on several late, high-depth tasks. On MG16, both methods exhibit negative normalized FWT on some R2 cold-start tasks, while AMSC improves acquisition across the later R3–R5 tasks. Both parameter-isolation methods retain every completed task, yielding zero measured FGT throughout.

30

Preprint

Table 13: Reference-distribution-size sensitivity. Values are means with asymmetric 95% bootstrap confidence intervals across seven seeds. Bold entries denote the sweep-reference setting. All runs obtain zero measured FGT. CT28

MG16

AUC

FWT

10 25 50 100 200

0.94+0.03 −0.04 0.94+0.03 −0.03 0.94+0.02 −0.02 0.89+0.03 −0.02 0.91+0.03 −0.03

0.94+0.04 −0.05 0.93+0.03 −0.04 0.94+0.02 −0.03 0.88+0.04 −0.03 0.90+0.03 −0.03

FGT 0.00 0.00 0.00 0.00 0.00

AUC

FWT

FGT

10 25 50 100 200

0.70+0.08 −0.09 0.55+0.09 −0.09 0.62+0.08 −0.09 0.69+0.10 −0.10 0.66+0.10 −0.11

0.52+0.14 −0.14 0.35+0.16 −0.16 0.43+0.13 −0.12 0.50+0.14 −0.12 0.48+0.14 −0.14

0.00 0.00 0.00 0.00 0.00

Table 14: Embedding-update-period sensitivity. Values are means with asymmetric 95% bootstrap confidence intervals across seven seeds. Bold entries denote the sweep-reference setting. All runs obtain zero measured FGT. CT28

MG16

femb

AUC

FWT

1 2 5 10

0.94+0.02 −0.02 0.91+0.03 −0.03 0.90+0.03 −0.03 0.84+0.03 −0.03

0.94+0.02 −0.03 0.91+0.04 −0.03 0.90+0.04 −0.05 0.83+0.05 −0.05

FGT 0.00 0.00 0.00 0.00

femb

AUC

FWT

FGT

1 2 5 10

0.62+0.08 −0.09 0.59+0.14 −0.15 0.58+0.13 −0.15 0.57+0.13 −0.13

0.43+0.13 −0.12 0.41+0.18 −0.19 0.36+0.21 −0.25 0.34+0.14 −0.13

0.00 0.00 0.00 0.00

Table 15: Selection-update-period sensitivity. Values are means with asymmetric 95% bootstrap confidence intervals across seven seeds. Bold entries denote the sweep-reference setting; MG16 uses N = 512 outside the sample-count sweep. All runs obtain zero measured FGT. CT28

MG16

fsel

AUC

FWT

1 2 5 10

0.94+0.02 −0.02 0.91+0.03 −0.02 0.86+0.02 −0.02 0.83+0.03 −0.02

0.94+0.02 −0.03 0.90+0.03 −0.03 0.85+0.02 −0.02 0.81+0.04 −0.04

FGT 0.00 0.00 0.00 0.00

fsel

AUC

FWT

FGT

1 2 5 10

0.64+0.08 −0.08 0.57+0.14 −0.15 0.62+0.08 −0.09 0.65+0.08 −0.09

0.40+0.18 −0.20 0.35+0.14 −0.16 0.43+0.13 −0.12 0.42+0.12 −0.13

0.00 0.00 0.00 0.00

Table 16: Embedding-EMA sensitivity. Values are means with asymmetric 95% bootstrap confidence intervals across seven seeds. Bold entries denote the sweep-reference setting; MG16 uses N = 512 outside the sample-count sweep. All runs obtain zero measured FGT. CT28

MG16

AUC

FWT

0 0.25 0.5 0.75 0.9

0.95+0.03 −0.03 0.95+0.02 −0.03 0.94+0.02 −0.02 0.93+0.03 −0.03 0.93+0.03 −0.03

0.94+0.03 −0.04 0.95+0.02 −0.03 0.94+0.02 −0.03 0.92+0.03 −0.03 0.92+0.04 −0.04

FGT 0.00 0.00 0.00 0.00 0.00

AUC

FWT

FGT

0 0.25 0.5 0.75 0.9

0.68+0.10 −0.12 0.66+0.10 −0.10 0.62+0.08 −0.09 0.67+0.09 −0.08 0.66+0.12 −0.13

0.51+0.14 −0.14 0.48+0.14 −0.14 0.43+0.13 −0.12 0.48+0.12 −0.10 0.47+0.17 −0.17

0.00 0.00 0.00 0.00 0.00

31

Preprint

Table 17: Random-reference-seed sensitivity. Values are means with asymmetric 95% bootstrap confidence intervals across seven seeds. Bold entries denote the sweep-reference setting; MG16 uses N = 512 outside the sample-count sweep. All runs obtain zero measured FGT. CT28

MG16

AUC

FWT

FGT

AUC

FWT

FGT

11 37 71 98 131

0.96+0.02 −0.03 0.93+0.03 −0.03 0.92+0.03 −0.03 0.94+0.02 −0.02 0.92+0.03 −0.03

0.96+0.02 −0.03 0.92+0.04 −0.04 0.91+0.04 −0.03 0.94+0.02 −0.03 0.91+0.04 −0.04

0.00 0.00 0.00 0.00 0.00

11 37 71 98 131

0.65+0.08 −0.08 0.65+0.09 −0.09 0.62+0.08 −0.09 0.62+0.08 −0.09 0.68+0.10 −0.12

0.47+0.12 −0.13 0.46+0.14 −0.15 0.39+0.14 −0.14 0.43+0.13 −0.12 0.50+0.16 −0.17

0.00 0.00 0.00 0.00 0.00

Table 18: Per-task CT28 acquisition and retention. Values are means across seven seeds. FWT is undefined when the single-task expert saturates the normalization denominator and can fall below −1 when the expert’s residual headroom is small. AMSC

Mask-LC

Task

AUC

FWT

FGT

AUC

FWT

FGT

A–D2 B–D2 C–D2 D–D2 A–D3 B–D3 C–D3 D–D3 A–D4 B–D4 C–D4 D–D4 A–D5 B–D5 C–D5 D–D5 A–D6 B–D6 C–D6 D–D6 A–D7 B–D7 C–D7 D–D7 A–D8 B–D8 C–D8 D–D8

1.00 0.96 0.99 0.96 1.00 1.00 1.00 0.99 1.00 1.00 0.94 1.00 0.99 1.00 1.00 1.00 1.00 1.00 0.60 1.00 1.00 0.99 0.56 1.00 1.00 0.94 0.67 0.77

— — — 0.00 1.00 1.00 1.00 0.98 1.00 1.00 0.97 1.00 0.99 1.00 1.00 1.00 1.00 1.00 0.63 1.00 1.00 0.99 0.56 1.00 1.00 0.97 0.68 0.81

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.00 0.97 1.00 0.89 1.00 0.80 0.99 0.59 0.97 0.66 0.39 0.37 0.76 0.34 0.41 0.00 0.84 0.00 0.00 0.00 0.69 0.00 0.00 0.00 0.66 0.00 0.00 0.00

— — — 0.00 1.00 0.63 0.95 -0.28 0.98 0.67 0.40 0.37 0.79 0.34 0.42 0.00 0.88 0.00 0.00 0.00 0.71 0.00 0.00 0.00 0.68 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

32

Preprint

Table 19: Per-task MG16 acquisition and retention for the earlier result set (mean displayed AUC approximately 0.62). This table does not decompose the principal 0.73 AUC result. Values are means across seven seeds. FWT is undefined when the single-task expert saturates the normalization denominator and can fall below −1 when the expert’s residual headroom is small. AMSC

Mask-LC

Task

AUC

FWT

FGT

AUC

FWT

FGT

BallLava–R2 LavaBall–R2 KeyBoxSwap–R2 Identity–R2 BallLava–R3 LavaBall–R3 KeyBoxSwap–R3 Identity–R3 BallLava–R4 LavaBall–R4 KeyBoxSwap–R4 Identity–R4 BallLava–R5 LavaBall–R5 KeyBoxSwap–R5 Identity–R5

0.83 0.81 0.91 0.79 0.72 0.56 0.74 0.50 0.62 0.69 0.45 0.24 0.61 0.57 0.49 0.38

0.24 -1.27 0.21 0.21 0.72 0.56 0.72 0.46 0.62 0.69 0.45 0.24 0.61 0.57 0.49 0.39

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.81 0.78 0.81 0.40 0.63 0.44 0.47 0.13 0.36 0.33 0.42 0.18 0.37 0.13 0.13 0.06

0.23 -2.41 -2.14 -2.57 0.63 0.44 0.47 -0.42 0.36 0.33 0.42 0.18 0.38 0.13 0.13 0.06

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Table 20: Per-task CW10 acquisition and retention across seven shared seeds. AUC averages the 20 active-task evaluations; FWT compares trapezoidal areas with the seed-matched from-scratch run. FWT is undefined when the reference headroom is below 10−3 and can be strongly negative when that headroom is small. FGT compares task-end and final success. Window-close FWT uses only one non-saturated seed; its AUC and FGT and all other rows use seven seeds. AMSC

Mask-LC

Mask-BLC

Task

AUC FWT FGT

AUC FWT FGT

AUC FWT FGT

0: Hammer 1: Push-wall 2: Faucet-close 3: Push-back 4: Stick-pull 5: Handle-press-side 6: Push 7: Shelf-place 8: Window-close 9: Peg-unplug-side

0.24 0.31 0.95 0.23 0.01 0.95 0.51 0.01 0.99 0.62

0.36 0.37 0.99 0.44 0.00 0.99 0.09 0.00 0.98 0.53

0.36 0.37 0.99 0.41 0.00 0.99 0.22 0.02 0.98 0.92

-1.62 0.00 -0.05 -0.01 -3.27 0.00 0.23 0.00 -0.07 0.00 -4.67 0.00 0.43 0.00 0.01 -0.01 1.00 0.00 -2.09 0.00

33

-0.67 0.00 0.08 0.00 0.22 0.00 0.44 -0.01 -0.09 0.00 0.93 0.00 -0.11 -0.01 0.00 0.00 0.80 0.00 -2.92 0.00

-0.67 0.08 0.13 0.42 -0.09 0.43 0.06 0.02 0.50 0.60

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Preprint

EVALUATION METRICS

Consider a curriculum of N tasks with an equal interaction budget ∆ per task and total budget T = N ∆. Task i is trained during [ti, ei ], where ti = (i − 1)∆ and ei = i∆. Let pi (t) ∈ [0, 1] be the normalized evaluation performance on task i at checkpoint t. Each checkpoint value averages the evaluation episodes conducted without training. CT28 returns and CW10 success rates already lie in [0, 1]. For MG16, we divide each return by the corresponding task’s theoretical shortest-path return and clip the result to [0, 1]. The same taskspecific normalization is applied to lifelong and single-task runs. Evaluations made before a task is first encountered are excluded from acquisition metrics. Acquisition AUC. Suppose task i’s active training block contains Mi regularly spaced evaluation checkpoints ti,1, . . ., ti,Mi . Following the discrete aggregation used in the Continual World result code (Wolczyk et al., 2021), the task-local acquisition score is M

AUCi =

1 Xi pi (ti,j ). Mi j=1

(26)

This is the mean height of the normalized learning curve while task i is being trained. The reported curriculum score is the task macro-average N

AUC =

1 X AUCi . N i=1

(27)

Thus, AUC measures acquisition performance rather than retention after a task block and lies in [0, 1]. Forward Transfer. Forward transfer compares each lifelong learning curve with a seed-matched single-task reference trained from scratch (Wolczyk et al., 2021). In the FWT calculation, task-local areas are estimated by the normalized trapezoidal rule. For a sequence q1, . . ., qM, define PM −1 1 1 j=2 qj + 2 qM 2 q1 + A(q1:M ) = . (28) M −1 Let ACL and AST denote this area for the lifelong and single-task learning curves, respectively. i i Then X 1 ACL − AST i FWTi = i , FWT = FWTi . (29) ST |IFWT | 1 − Ai i∈IFWT

The set IFWT contains every task for which 1 − AST i

−3

≥ 10 . This exclusion avoids unstable ratios when the single-task reference is effectively saturated. No task is otherwise removed from the reported FWT; in particular, CT28’s four initial depth-2 tasks are retained. They are excluded only from selection analyses that require an earlier hierarchy-relevant source. Zero FWT matches single-task acquisition, positive values indicate faster acquisition, and negative values indicate slower acquisition. The normalized score is at most one but can be substantially negative when a strong single-task reference appears in the denominator. AUC and FWT therefore need not rank methods identically: AUC measures absolute acquisition, whereas FWT first normalizes each task’s improvement by the remaining headroom over its single-task reference and then macro-averages the task scores. The reported AUC and the trapezoidal area used by FWT also differ slightly in endpoint weighting. Forgetting. Continual World forgetting compares each task’s performance at the end of its training block with its performance at the final curriculum checkpoint (Wolczyk et al., 2021): N

FGTi = pi (ei ) − pi (T ),

FGT =

1 X FGTi . N i=1

(30)

Positive values indicate forgetting and negative values indicate subsequent improvement. A value of zero indicates no net average change; complete measured retention additionally requires every per-task difference to be zero. The final task contributes zero because eN = T . 34

Preprint

SIGNIFICANCE TESTING

Statistical protocol. Deep reinforcement learning results vary across random seeds and environment stochasticity (Henderson et al., 2018; Colas et al., 2018). Each comparison uses the seven shared seeds 86–92. We compute one metric value per seed, then test the mean paired difference ∆ = AMSC − comparison using an exact two-sided sign-flip test over all 27 = 128 sign assignments (Winkler et al., 2014). The tables also report 95% percentile-bootstrap confidence intervals for the mean paired difference using 10,000 paired resamples with seed zero. We report unadjusted p-values with 95% confidence intervals. Green cells denote p < 0.05; red cells denote p ≥ 0.05. Color indicates evidence for a difference, not which method is better. Positive ∆ favors AMSC for AUC and FWT, whereas negative ∆ favors AMSC for FGT. Baseline comparisons. Tables 21, 22, and 23 use the principal configurations reported in the main text. CT28 uses N = 128; MG16 uses the seven completed N = 1024, R = 50, ρ = 0.5 runs from the sample-count sweep. With seven pairs, the smallest attainable two-sided raw p-value is 2/128 = 0.015625. Table 21: CT28 baseline comparisons across seven matched seeds. Each metric reports the exact two-sided raw p and a 95% paired percentile-bootstrap CI for ∆ = AMSC − comparison. Green denotes p < 0.05; red denotes p ≥ 0.05. Tests are not adjusted for multiple comparisons. Color indicates significance, not which method is better.

Fine-tuning From-scratch SER-s SER-r SDW OWL SANE Mask-BLC Mask-LC Mask-LC (L2) CKA-RL CompoNet MOSAIC MOSAIC ¬Iperf

AUC 95% CI

Method

0.016 [0.56, 0.60] 0.016 [0.69, 0.75] 0.422 [−0.06, 0.02] 0.188 [0.00, 0.12] 0.047 [0.06, 0.28] 0.016 [0.76, 0.82] 0.016 [0.44, 0.53] 0.016 [0.63, 0.69] 0.016 [0.44, 0.49] 0.016 [0.37, 0.47] 0.016 [0.51, 0.64] 0.016 [0.15, 0.37] 0.016 [0.08, 0.17] 0.016 [0.09, 0.17]

FWT 95% CI

0.016 [0.70, 0.74] 0.016 [0.91, 0.96] 0.578 [−0.06, 0.02] 0.172 [0.00, 0.14] 0.047 [0.07, 0.34] 0.016 [1.09, 1.43] 0.016 [0.52, 0.61] 0.016 [0.78, 0.89] 0.016 [0.53, 0.65] 0.016 [0.44, 0.58] 0.016 [0.60, 0.75] 0.016 [0.18, 0.43] 0.016 [0.11, 0.20] 0.016 [0.11, 0.20]

FGT 95% CI

0.016 1.000 0.016 0.125 0.500 0.016 1.000 1.000 1.000 1.000 0.016 1.000 1.000 1.000

[−0.40, −0.36] [0.00, 0.00] [−0.70, −0.36] [−0.18, −0.02] [0.00, 0.02] [−0.21, −0.14] [0.00, 0.00] [0.00, 0.00] [0.00, 0.00] [0.00, 0.00] [−0.41, −0.31] [0.00, 0.00] [0.00, 0.00] [0.00, 0.00]

Table 22: MG16 baseline comparisons across seven matched seeds. Each metric reports the exact two-sided raw p and a 95% paired percentile-bootstrap CI for ∆ = AMSC − comparison. Green denotes p < 0.05; red denotes p ≥ 0.05. Tests are not adjusted for multiple comparisons. Color indicates significance, not which method is better.

Method

Fine-tuning From-scratch SER-s SER-r SDW OWL SANE Mask-BLC Mask-LC Mask-LC (L2) CKA-RL CompoNet MOSAIC MOSAIC ¬Iperf

AUC 95% CI

FWT 95% CI

FGT 95% CI

0.016 0.016 0.016 0.031 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016

[0.38, 0.50] [0.45, 0.57] [0.25, 0.36] [0.05, 0.16] [0.14, 0.26] [0.46, 0.59] [0.31, 0.46] [0.09, 0.27] [0.20, 0.45] [0.21, 0.35] [0.34, 0.47] [0.34, 0.51] [0.35, 0.53] [0.34, 0.56]

0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016 0.016

[0.33, 0.56] [0.40, 0.66] [0.50, 0.85] [0.23, 0.47] [0.43, 0.62] [0.67, 1.00] [0.28, 0.43] [0.17, 0.45] [0.44, 1.10] [0.16, 0.35] [0.64, 0.87] [1.18, 1.63] [0.72, 1.38] [0.74, 1.54]

0.016 1.000 0.016 0.375 0.094 0.016 1.000 1.000 1.000 1.000 0.016 1.000 1.000 1.000

[−0.41, −0.27] [0.00, 0.00] [−0.60, −0.41] [−0.18, 0.02] [−0.28, −0.03] [−0.24, −0.20] [0.00, 0.00] [0.00, 0.00] [0.00, 0.00] [0.00, 0.00] [−0.54, −0.28] [0.00, 0.00] [0.00, 0.00] [0.00, 0.00]

CW10 comparisons. CW10 uses seed-matched online success evaluations. AMSC’s FWT is lower than Mask-BLC (p = 0.031) and from-scratch learning (p = 0.016) at the nominal 0.05 35

Preprint

threshold. The AUC difference from Mask-BLC is not significant (p = 0.328). FWT excludes task– seed reference headroom below 10−3 ; near-saturated reference curves can still amplify negative values. Table 23: CW10 baseline comparisons across seven matched seeds. ProgNet uses its native twolayer scratch FWT reference; other methods use the four-layer reference. Each metric reports the exact two-sided raw p and a 95% paired percentile-bootstrap CI for ∆ = AMSC − comparison. Green denotes p < 0.05; red denotes p ≥ 0.05. Tests are not adjusted for multiple comparisons. Color indicates significance, not which method is better. p

Method

Fine-tuning From-scratch ProgNet Mask-LC Mask-BLC CKA-RL MOSAIC

AUC 95% CI

FWT 95% CI

0.234 [−0.04, 0.35] 0.703 0.062 [0.02, 0.12] 0.016 0.016 [0.08, 0.21] 0.141 0.781 [−0.08, 0.10] 0.062 0.328 [−0.12, 0.03] 0.031 0.141 [0.00, 0.20] 0.453 0.016 [0.16, 0.29] 0.016

[−1.15, 18.58] [−2.19, −0.43] [−1.90, −0.06] [−2.01, −0.22] [−2.21, −0.58] [−0.59, 2.77] [0.49, 9.03]

FGT 95% CI

0.062 [−0.58, −0.18] 0.500 [−0.01, 0.00] 0.500 [−0.01, 0.00] 1.000 [−0.01, 0.01] 0.500 [−0.01, 0.00] 0.500 [−0.01, 0.00] 0.500 [−0.01, 0.00]

Composition and retrieval controls. Table 24 compares each control with configuration-matched AMSC runs across seven shared seeds. CT28 uses N = 128, while all MG16 comparisons use N = 1024, R = 50, and ρ = 0.5. Each control changes only the specified retrieval or composition component. FGT is omitted because all controls and their matched AMSC references have zero measured forgetting. Table 24: CT28 and MG16 paired ablation comparisons across seven matched seeds. Each metric reports the exact two-sided unadjusted p-value and a 95% paired bootstrap CI for ∆ = AMSC − control. Green denotes p < 0.05; red denotes p ≥ 0.05. Positive differences favor AMSC. CT28 uses N = 128. MG16 uses N = 1024, with R = 50 and ρ = 0.5. CT28

MG16

AUC Configuration Learned β Uniform No ℓ2-STE Frozen retrieval Top-one Matched random Transfer-informed

95% CI

FWT p

95% CI

AUC p

95% CI

0.484 [−0.02, 0.06] 0.453 [−0.02, 0.08] 0.031 [0.04, 0.23] 0.172 [−0.01, 0.09] 0.172 [−0.01, 0.11] 0.047 [0.04, 0.20] 0.047 [0.01, 0.08] 0.016 [0.04, 0.11] 0.734 [−0.10, 0.16] 0.047 [0.03, 0.12] 0.062 [0.03, 0.14] 0.031 [0.02, 0.22] 0.078 [0.02, 0.11] 0.078 [0.02, 0.12] 0.844 [−0.12, 0.09] 0.016 [0.56, 0.62] 0.016 [0.74, 0.85] 0.016 [0.17, 0.34] 0.016 [0.10, 0.15] 0.016 [0.15, 0.33] 0.531 [−0.14, 0.05]

36

FWT p

95% CI

0.047 [0.06, 0.38] 0.016 [0.05, 0.21] 0.172 [−0.05, 0.36] 0.016 [0.05, 0.22] 0.844 [−0.16, 0.10] 0.016 [0.18, 0.39] 0.031 [−0.27, −0.08]

Preprint

Table 25: Alignment of initial and final AMSC retrieval with aggregate pairwise transfer. Weighted ∆AUC normalizes weights over retrieved sources and therefore excludes current-mask mass. Parentheses give support-size-matched random expectations. Statistics aggregate seven lifelong runs; the transfer value assigned to each pair comes from one source-policy training seed. Benchmark

Phase

Support

Mean ∆AUC

Weighted ∆AUC

Positive precision

Positive-mass recall

Best source

CT28

Initial Final Initial Final

3.11 2.57 2.42 1.94

0.52 (0.10) 0.57 (0.10) 0.47 (0.20) 0.53 (0.20)

0.49 0.62 0.51 0.55

0.61 (0.19) 0.67 (0.19) 0.74 (0.46) 0.80 (0.46)

0.87 (0.23) 0.86 (0.20) 0.63 (0.33) 0.58 (0.27)

0.98 (0.47) 0.96 (0.43) 0.84 (0.41) 0.84 (0.36)

MG16

H.1

PAIRWISE TRANSFER MATRICES

Figures 9 and 10 report the unaggregated pairwise transfer measurements used in the retrieval analysis. Each cell is ∆AUC, the target-task AUC obtained with the row source minus the AUC of the corresponding no-source run. Positive values indicate improved learning over the target budget and negative values indicate interference. Rows are source policies and columns are target tasks; tasks are grouped by family and ordered by increasing depth or radius. The complete matrices include self-pairs and sources that occur after the target in the lifelong curriculum. The curriculum-restricted analysis in Section 5.5 retains only sources available when the target is encountered. H.2

PER-TASK POLICY-MASK UTILIZATION

Figures 11 and 12 report how AMSC distributes composition mass across the current and retrieved policy masks. Rows are target tasks and columns are policy-mask components. Diagonal cells therefore correspond to the current mask, while colored cells below the diagonal correspond to prior masks available at that point in the curriculum. Values are the mean fixed composition coefficients across retrieval events and seven seeds. Grey cells identify available priors that are never selected; blank cells correspond to future masks that are not yet available. These coefficients quantify mask reuse, rather than the fraction of backbone parameters activated by a mask. H.3

INITIAL RETRIEVAL AND TEMPORAL REFINEMENT

We compare AMSC’s first eligible retrieval and its final retrieval with the pairwise-transfer measurements above. The first event occurs after the current-task experience buffer contains enough samples to construct an embedding. We exclude task runs with fewer than two measured candidate sources because they contain no meaningful source-selection decision. Table 25 reports alignment with measured transfer. Positive-transfer precision is the fraction of selected sources with ∆AUC > 0. Positive-mass recall is given by P i∈P(k) max(∆AUCi→k, 0) P , (31) i<k max(∆AUCi→k, 0) which weights recovered sources by transfer magnitude rather than counting all positive pairs equally. Best-source inclusion indicates whether the support contains a source attaining the largest available ∆AUC. Parenthesized values are analytic expectations under uniform sampling without replacement of a support with the same size. The initial support is already strongly transfer-aligned. Its mean ∆AUC exceeds the random expectation by 0.42 on CT28 and 0.27 on MG16; seed-bootstrap 95% confidence intervals are [0.40, 0.43] and [0.25, 0.29]. Initial retrieval also contains the best available source in 98% and 84% of task runs. Refinement trades some positive-transfer mass recall for greater precision, consistent with pruning an initially over-inclusive support. Although only 10% of consecutive CT28 retrieval events and 7% of MG16 events change support, these changes accumulate over training. Across seeds, CT28 support changes add 23 and remove 122 source occurrences; their mean ∆AUC values are 0.39 and 0.16, respectively. MG16 changes add 8 and remove 55 occurrences, with corresponding means of 0.28 and 0.22. The source37

.3

A4

.3

-.1

.9

-.1

-.1

-.1

.2

.4

-.2

-.2

-.1

A5

.3

.9

.4

-.1

-.6

-.3

-.1

A6

.3

.9

-.1

.2

-.2

-.1

-.1

A7

.3

.9

A8

.3

.8

-.4

-.1

.1

.3

-.2

-.5

-.1

-.2

-.4

.1

-.1

.2

-.2

-.1

B2

-.1

-.3

.5

.9

-.2

-.3

-.1

-.3

-.1

B3

-.1

B4

-.1

-.1

.5

-1

-.1

.5

-.3

-.1

-.3

-.1

B5

.1

.9

-.1

.5

-.2

-.5

-.1

B6

-.3

.2

.5

-.1

-.1

-.3

-.1

B7

-.2

B8

.1

.2

.5

-.2

-.6

-.2

-.1

-.2

.5

-.1

-.7

-.4

-.5

-.1

C2

.3

-.1

.3

.1

-.1

-.1

-.1

C3 C4

-.2

.5

-.1

.2

.1

-.1

-.4

-.1

C5

.2

.8

-.3

.1

-.1

-.1

C6

.2

.2

-.1

-.1

C7

.3

.5

.5

.9

-.1

C8

.2

.9

-.1

-.5

-.1

D2

.1

-.3

-.2

-.9

.1

.9

D3

.3

-.2

-.1

-1

.1

.9

.9

D4

-.1

-.2

-.1

.2

.1

.1

.1

.9

D5

-.2

.1

-.2

.1

-.1

-.3

.1

.9

D6

.1

.1

.3

-.1

-.5

.1

.9

D7

.1

-.2

-.4

-.2

-1

.1

.9

.9

D8

.2

-.2

.1

-.1

-1

.1

.9

Target task

1.00

0.75

0.50

0.25

0.00

Pairwise transfer, AUC

D8

D7

D6

-.1

D5

-.4

D4

-.1

D3

C8

D2

C7

C6

C5

C4

C3

B8

C2

B7

B6

B5

A3

-.2

B4

-.1

B3

B2

A8

A7

A6

A5

.3

A4

A3

A2

A2

Source policy

Preprint

0.25

0.50

0.75

1.00

Figure 9: Raw pairwise transfer on CT28. Target training is initialized from the source policy and then fine-tuned. Cell annotations report ∆AUC relative to learning the target without a source; red denotes positive transfer and blue denotes negative transfer. Source policies are taken from the family-isolated seed-92 runs. Family labels A–D and depths 2–8 define the row and column order.

38

.94

.60

.08

.30

.34

.71

.81

-.24 -.19

.55

BallLava R4

.06

.39

.96

.06

.36

.34

.92

.78

.18

.24

.73

BallLava R5

.06

.94

.89

.98

.10

.28

.36

.36

.92

.87

.88

.06

.21

.58

LavaBall R2 -.51

.16

-.54

.80

-.58

-.46 -.10

.81

.14

.46

.90

.82

-.58

.65

-.39 -.71

LavaBall R4 -.39

.16

.42

.19

.31

-.07 -.23

LavaBall R5 -.02

.46

.16

.46

.98

.98

.19

-.12 -.46

KeyBoxSwap R2

.06

.54

.82

.08

.26

.36

.85

.06

-.13

KeyBoxSwap R3

.02

.80

.80

.12

.34

.18

.36

.96

.97

.93

.12

.21

KeyBoxSwap R4

.74

.80

.12

.34

.28

.96

.97

.91

.12

.15

.74

KeyBoxSwap R5

.06

.88

.18

.98

.16

.16

.60

.38

.94

.93

.94

.10

.19

Identity R2 -.40

.12

.44

.34

.96

.93

.82

.12

.23

.91

Identity R3 -.02

.46

.14

.42

.04

.34

.90

.81

.43

.12

.23

.97

Identity R4 -.08

.84

.12

.40

.30

.90

.91

.83

.12

.23

.95

.06

.76

.58

.12

.44

.34

.92

.87

.12

.21

.91

R2

R3

R4

R5

va

ll R 5

va

ll R 4

va

Target task

0.25

0.00

0.25

0.50

0.75

1.00

ntit Ide

ntit Ide

ntit Ide

ntit

Ide

oxS w

0.50

Key B

oxS w

Key B

oxS w

Key B

oxS w

Key B

Lav

aBa

Lav

aBa

Lav

aBa

Bal lLa

Bal lLa

Identity R5

Lav

Source policy

LavaBall R3

0.75

Pairwise transfer, AUC

.06

1.00

yR

BallLava R3

yR

yR

yR

.10

R5

.06

R4

ap

.48

R3

.73

ap

.34

ap

.42

R2

.50

ap

-.06 -.54

aBa

.98

ll R 3

.98

ll R 2

.93

Bal lLa

.06

Bal lLa

BallLava R2

va

Preprint

Figure 10: Raw pairwise transfer on MG16. Each source mask is composed with a newly initialized target mask using an initially equal mixture, and both the target mask and mixture coefficients are optimized on the target task. Cell annotations report ∆AUC relative to the matched no-source run; red denotes positive transfer and blue denotes negative transfer. Source masks are taken from the family-isolated seed-86 runs. Tasks are grouped by observation-remapping family and ordered by radius 2–5.

39

Preprint

1.00

A D3

0.50

0.50

A D4

0.27

0.39

0.33

A D5

0.12

0.21

0.41

0.25

A D6

0.02

0.03

0.28

0.40

0.27

A D7

0.01

0.02

0.18

0.23

0.34

0.22

A D8

0.01

<.01

0.05

0.15

0.23

0.35

B D2

0.50

B D3

--

--

B D4

--

--

--

B D5

--

--

--

--

B D6

--

--

--

--

--

B D7

--

--

--

--

--

--

--

--

--

--

--

---

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

1.0

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

0.8 --

C D3

--

--

C D4

--

--

--

C D5

--

--

--

--

C D6

--

--

--

--

--

C D7

--

--

--

--

--

--

C D8

--

--

--

--

--

--

D D2

--

D D3

--

--

D D4

--

--

--

D D5

--

--

--

--

D D6

--

--

--

--

--

D D7

--

--

--

--

--

D D8

--

--

0.50

0.50

0.21

0.46

0.33

--

0.07

0.21

0.46

0.26

0.03

0.16

0.21

0.38

0.22

0.01

0.03

0.13

0.26

0.35

0.22

<.01

<.01

0.07

0.17

0.28

0.27

0.20

0.48

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

------selected --Available prior, never

--

--

--

--

--

2 BD

AD

AD

AD

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

0.6 --

--

0.50

0.50

0.16

0.50

0.33

--

0.01

0.19

0.47

0.33

0.02

0.04

0.24

0.41

0.29

0.02

0.03

0.11

0.24

0.33

0.27

<.01

<.01

0.05

0.12

0.21

0.34

0.27

--

0.4

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

0.50

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

0.2

0.50

0.50

0.23

0.43

0.33

0.15

0.28

0.32

0.25

0.03

0.18

0.19

0.38

0.22

0.01

0.07

0.22

0.21

0.29

0.20

0.01

0.02

0.07

0.20

0.28

0.25

0.18

0.0

8 CD 2 CD 3 CD 4 CD 5 CD 6 CD 7 CD 8 DD 2 DD 3 DD 4 DD 5 DD 6 DD 7 DD 8

--

AD

--

BD

--

BD

--

BD

--

--

BD

--

--

BD

--

--

BD

--

--

--

--

--

--

--

--

AD

--

0.50

0.50

--

--

--

Mean composition weight

-0.03

B D8 C D2

AD

0.21

--

0.50

AD

Target task

Mean current and prior weights: fixed sparsemax composition A D2

Policy-mask component

Figure 11: Per-task policy-mask utilization on CT28. Cell annotations report the mean composition coefficient assigned to each current or prior mask. The block structure shows that AMSC places most prior mass on masks from the same task family, while retaining limited cross-family reuse where the online representation retrieves it.

40

Preprint

1.00

BallLava R3

0.52

0.47

BallLava R4

0.20

0.47

0.33

BallLava R5

0.06

0.18

0.49

LavaBall R2

0.50

LavaBall R3

0.04

0.12

LavaBall R4

<.01

<.01

<.01

LavaBall R5

0.01

0.01

0.01

KeyBoxSwap R2

0.50

KeyBoxSwap R3

<.01

<.01

KeyBoxSwap R4

<.01

<.01

<.01

KeyBoxSwap R5

--

--

--

Identity R2

0.03

Identity R3

--

--

Identity R4

--

--

Identity R5

--

<.01 0.01

--

--

<.01

1.0

--

--

<.01

--

<.01

<.01

--

--

--

--

0.8

--

0.31

--

0.19

0.49

0.32

0.05

0.27

0.38

0.27

--

0.37

0.02

0.16

0.48

0.35

<.01

--

--

--

0.39

0.17

0.48

0.33

0.05

0.20

0.45

ntit y Ide

0.30

0.0

Key B

oxS w

0.2

0.52

R2

<.01

R5

<.01

R4

--

ap

--

R3

0.01

ap

<.01

ap

0.01

oxS w

0.07

R2

0.02

Key B

--

0.43

ap

ll R Key B

oxS w

aBa

--

Lav

ll R aBa

Lav

ll R aBa

Lav

ll R

Lav

aBa

aR lLa v

0.44

0.54

--

Bal

0.18

0.4

<.01

R5

--

0.49

R4

--

0.51

ntit y

--

--- prior,--never--selected -Available

--

Ide

--

--

aR

--

R3

--

--

lLa v

--

ntit y

--

--

Bal

<.01

0.6

Ide

--

--

aR

<.01

--

ntit y

--

--

aR

--

--

0.50

--

--

--

oxS w

<.01

--

Ide

--

--

Key B

<.01

0.53

Mean composition weight

0.50

lLa v Bal

Bal

<.01

<.01 0.27

lLa v

Target task

Mean current and prior weights: fixed sparsemax composition BallLava R2

Policy-mask component Figure 12: Per-task policy-mask utilization on MG16. Cell annotations report the mean composition coefficient assigned to each current or prior mask. In contrast to CT28, AMSC assigns nonnegligible mass to several cross-family components, consistent with the less family-separable transfer structure of MG16.

41

Preprint

Table 26: Temporal retrieval diagnostics. J denotes support Jaccard similarity. Improved and worsened are fractions among task runs whose first and final supports differ, measured by sourcenormalized, similarity-weighted ∆AUC. Benchmark CT28 MG16

Consecutive J

First–final J

Changed

Support

Weighted ∆AUC

Improved

Worsened

0.97 0.97

0.82 0.82

0.43 0.39

3.11 → 2.57 2.42 → 1.94

0.49 → 0.62 0.51 → 0.55

0.70 0.74

0.23 0.16

normalized weighted improvement from first to final retrieval is 0.13 on CT28 (95% CI [0.11, 0.15]) and 0.03 on MG16 ([0.01, 0.05]). These diagnostics associate online refinement with greater aggregate pairwise transfer; they do not provide time-local transfer labels. A No Periodicity control tests repeated retrieval by retaining the first valid support and weights throughout each task. On CT28, AUC/FWT falls from 0.94/0.94 to 0.87/0.85, with paired p = 0.047 and p = 0.062. On MG16, retaining the first retrieval lowers AUC/FWT from 0.73/0.54 to 0.63/0.42, with paired (p = 0.031) and (p = 0.016), respectively, using matched (N = 1024) configurations. These results support updating retrieval as target-task experience accumulates. The pairwise transfer measurements summarize complete target learning curves from one source-policy seed, so their alignment with retrieval should be interpreted as a transfer proxy rather than direct evidence of instantaneous benefit.

42