FedCKA, 표현 유사성 기반 층 개인화로 연합 3D 인지 성능 개선

Abstract

Robust perception in intelligent vehicles demands 3D object detectors that remain dependable under domain shifts, such as changes in time of day, location, or weather. However, due to costly annotation and rare shifts, some environments lack sufficient data to train a standalone detector. Federated learning offers a privacy-preserving framework for collaborative model training, enabling clients to benefit from shared learning across diverse environments. Yet, this framework traditionally relies on a single global consensus model, which struggles to perform across heterogeneous local data distributions. Local conditions are better captured by adapting a subset of the model, but many personalization approaches rely on predefined layer partitions or fixed personalization ratios, thereby limiting adaptation to client-specific divergence. To reduce this rigidity, we propose FedCKA, a Centered Kernel Alignment (CKA)-based strategy that dynamically handles the personalization-globalization trade-off. Specifically, FedCKA computes layer-wise feature similarities between local client models and the global consensus model during training. By converting layer-wise similarity scores into client-specific aggregation masks, FedCKA selectively shares representation-consistent layers. Evaluation on a unified multi-domain benchmark based on nuScenes shows that FedCKA outperforms established federated baselines, including FedBN, FedRep, and FedSelect, improving average NDS by 7 percentage points over the strongest baseline. The findings offer both a comparative benchmark and a promising direction for robust federated 3D perception across shifts in location, weather, and illumination. Code is available at https://github.com/j-verhoog/FedCKA.

I Introduction

3D Object Detection (3D-OD) is central to robotic perception in Intelligent Vehicles (IV), providing object poses and dimensions needed for planning and safe navigation. However, domain shifts across vehicles, times of day, weather, and locations substantially degrade 3D detection performance [1] (see Fig. 1). Because labeled data in niche or edge-case domains can be too scarce to support robust standalone training, a collaborative framework is desirable. Yet, for IV applications, centrally collecting multimodal sensor data is constrained by privacy barriers and massive data volumes [2], while naive localized retraining remains computationally expensive [3].

These constraints motivate Federated Learning (FL), which enables privacy-preserving collaborative training for multi-domain adaptation without data centralization [4, 5, 6], by sharing only model updates. However, real-world deployment across heterogeneous data distributions introduces a central personalization-globalization trade-off: which model components should be globally shared versus locally personalized? Many personalization methods resolve this trade-off through static layer partitions or fixed personalization ratios, which may fail to capture client- and layer-wise representation divergence. For multi-modal 3D-OD, this rigidity is particularly problematic because domain shifts can affect sensor backbones and fusion modules asymmetrically [7].

Despite this need, FL for multimodal 3D-OD remains underexplored. While prior federated detection and, more recently, federated 3D-OD studies exist, they do not systematically isolate location-, weather-, and illumination-induced client heterogeneity. This limits insight into whether personalization helps under these structured client-level domain shifts. While recent federated 3D detection approaches, including FedM2Former [9] and FedPillarNet [10], address client heterogeneity through personalized learning, they do not systematically isolate location-, weather-, and illumination-induced domains or compare representative FL personalization strategies within a unified 3D-OD benchmark.

To address this gap, we study how federated 3D-OD methods handle location-, weather-, and illumination-induced client heterogeneity, and whether representation consistency can determine which layers should be globally aggregated or locally personalized. We isolate client-data heterogeneity by focusing on labeled, client-level domain adaptation across location, weather, and illumination, leaving system-level constraints such as client reliability and sensor-configuration variability to future work. We make the following contributions:

1. Establish a unified multi-domain federated 3D object detection benchmark based on explicit location-, weather-, and illumination-induced client heterogeneity, enabling controlled evaluation under realistic domain shifts.
2. Introduce FedCKA, a dynamic CKA-based layer masking strategy that uses layer-wise representation similarity to determine which model components should be globally aggregated versus locally personalized.
3. Demonstrate that FedCKA outperforms established federated baselines, including FedBN, FedRep, and FedSelect, improving average NDS by 7 percentage points over the strongest competitor by identifying domain-sensitive layers.

II Related Work

3D Object Detection methods are commonly categorized by sensor modality: camera, LiDAR, or multimodal [11]. While multimodal methods typically fuse LiDAR and camera data via Bird’s-Eye-View (BEV) representations [12], they struggle across varying domains like day, night, and adverse weather [13]. This limitation motivates methods that can exploit multi-domain data, particularly when such data are distributed across clients and cannot be centrally pooled.

Federated Learning (FL) enables such collaborative training on decentralized data. It commonly operates under client data that is not independent and identically distributed (IID), not centrally shared, or not of equal size across clients [14]. This statistical and system heterogeneity introduces challenges for convergence [5] and cross-domain generalization [15].

The foundational FedAvg [2] algorithm aggregates independent local updates via weighted averaging. While effective under IID conditions, its performance degrades significantly under heterogeneous data distributions [16].

Data heterogeneity in FL fundamentally shapes what client models should share and how the server should aggregate their updates. Approaches to managing this heterogeneity span data preprocessing, training objectives, and aggregation schemes [14]. For all of these, the main distinction is the objective: aligning all clients toward one domain-invariant model, or producing multiple domain-adapted models.

To learn a more domain-invariant global model, methods such as FedProx [5], FedDyn [17], and SCAFFOLD [6] reduce client drift during training. Other approaches use task arithmetic [18] to align client update gradients and estimate a less conflicting global update direction [19, 20, 21]. However, a single global model may be suboptimal under strong client heterogeneity, and these methods require problem-specific manual parameter tuning. Personalized FL addresses this by adapting models to client-specific domains while retaining shared knowledge. Common approaches keep batch normalization or other targeted layers client-specific, as in FedBN [4] and FedRep [22]. Because these methods require manually predefined global and domain-specific layer partitions, this static approach can be suboptimal when domain specificity varies across layers or clients.

Recent work addresses this rigidity by using model updates to infer which parameters to mask for personalization. FedSelect [23] assumes the largest parameter updates are the most domain-significant and personalizes them. FedMC [24] uses the Fisher information derived from gradient magnitudes to identify the most sensitive parameters for personalization. Although these methods avoid manual partitioning, they still rely on parameter-space importance by implicitly assuming that same-index parameters across clients correspond to comparable functional components, which may not hold [25]. To overcome the limitations of parameter-space comparisons and manual layer selection, this work selects personalized layers based on representational consistency across clients.

III Proposed Method

III-A Motivation

FedCKA is motivated by the idea that layers should be shared across clients only when their representations remain consistent after local adaptation. While existing federated methods rely on parameter-space distances [23] or gradient magnitudes [24], this work argues that representational consistency is a more reliable proxy for transferability and uses it to determine which layers to personalize. This strategy incorporates Centered Kernel Alignment (CKA) [26] to measure how differently two layers map the same set of incoming features to their outgoing features. This metric produces a normalized score bounded in [0,1], with 0 indicating strong dissimilarity and 1 indicating high similarity in the representation space transformation for each layer.

By comparing layers at identical depths across the global and local models, FedCKA estimates whether each layer remains transferable. The most dissimilar layers can then be personalized by masking them from the global model aggregation. This lets certain layers drift to domain-specific optima, while sharing representationally similar layers to transfer global knowledge. Because only unmasked parameters are transmitted, FedCKA reduces shared parameter information by the fraction of masked parameters, improving bandwidth efficiency and potentially reducing information exposure.

III-B FedCKA System Architecture

As illustrated in Fig. 2, FedCKA entails coordinated exchanges between the clients and the central server at every communication round. Let i\in{1,\ldots,N} denote the client index and t the communication round, with G_{t} denoting the global model after round t and C_{i,t} the corresponding client model. During round t, client i locally trains C_{i,t-1} for E local training epochs, yielding the locally updated model \widetilde{C}_{i,t}.

The first step is to estimate local divergence by computing the CKA score between the saved state of the previous global model, G_{t-1}, and the newly updated local model \widetilde{C}_{i,t}. Using G_{t-1} as a common reference ensures that each client’s layer-wise divergence is measured against the same pre-aggregation model state. This computation uses a small random subset of the client’s local training data passed through both models without requiring ground-truth labels. Based on the layers’ representational similarity, the client determines which layers to personalize and thus mask from global aggregation. Consequently, each client sends its unmasked layers, alongside a lightweight binary mask identifying them, to the central server.

The server aggregates these shared parameters layer-by-layer to form the updated global model G_{t}. Finally, each client updates \widetilde{C}_{i,t} by injecting the unmasked parameters from G_{t}, yielding the updated client model C_{i,t}. This process allows each client to personalize its most dissimilar layers while benefiting from the knowledge learned by other clients.

III-C Centered Kernel Alignment

Centered Kernel Alignment [26] is a similarity index between two layers, based on the Hilbert-Schmidt Independence Criterion (HSIC) [27] for comparing the representations induced by two layers. Different layer parameterizations may yield similar representations, making direct comparison in parameter space misleading. CKA addresses this limitation by comparing layer activations rather than parameters. CKA has similarly been used in representation analysis of vision architectures for transformers and convolutional neural networks [28]. In this work, this principle is used in FL, where CKA measures whether a locally trained client layer remains representationally aligned with the previous global model.

Given layer outputs X\in\mathbb{R}^{n\times p_{1}} and Y\in\mathbb{R}^{n\times p_{2}}, where n denotes the number of aligned activation samples and p_{1} and p_{2} the respective feature dimensions, their representational similarity is computed using CKA, which normalizes the empirical HSIC between their respective Gram matrices K=XX^{T} and L=YY^{T}:

\operatorname{CKA}(X,Y)=\frac{\operatorname{HSIC}(K,L)}{\sqrt{\operatorname{HSIC}(K,K)\operatorname{HSIC}(L,L)}}.

CKA, therefore, indicates whether the same samples produce a similar relational structure at a given layer. Mean-centering and normalization make it invariant to isotropic scaling and orthogonal transformations. Because CKA does not rely on one-to-one alignment between individual neurons or feature channels, the exact ordering of neurons does not affect the comparison. It therefore captures whether a layer similarly organizes the same input samples across representations, rather than whether individual activations match directly.

III-D Personalization Strategies

At each communication round, the personalization masks are constructed with the layer-wise CKA scores using one of two strategies: selecting a target percentage of parameters, analogous to FedSelect [23] and FedMC [24], or selecting all layers below a CKA similarity threshold.

To formally define the personalization strategies, let N be the total number of clients, each with n_{i} local data samples. For client i at round t, let w_{i,l}^{t} denote the weights of layer l and s_{i,l}^{t}\in[0,1] its computed CKA similarity score. This score is used to define the binary mask M_{i,l}^{t}\in\{0,1\}, where M_{i,l}^{t}=1 denotes personalization. For each layer, the server aggregates the weight matrices only across the clients that keep that layer shared:

\bar{w}_{l}^{t}=\frac{\sum_{i=1}^{N}n_{i}(1-M_{i,l}^{t})w_{i,l}^{t}}{\sum_{i=1}^{N}n_{i}(1-M_{i,l}^{t})}.

If all clients personalize layer l, the denominator is zero, and the server keeps the previous global layer, i.e., \bar{w}_{l}^{t}=\bar{w}_{l}^{t-1}. Subsequently, each client i\in\{1,\dots,N\} overwrites its local model by computing the following update for every layer:

\hat{w}_{i,l}^{t}=M_{i,l}^{t}w_{i,l}^{t}+(1-M_{i,l}^{t})\bar{w}_{l}^{t},\quad\forall l

The persistence of the personalization mask across communication rounds differs between the two strategies. For readability, client and round indices are omitted in the mask construction below. Depending on the chosen strategy, the mask M_{l} is constructed in one of two ways:

Percentage-based masking: For percentage-based masking, personalization is cumulative across communication rounds: once selected, a layer remains personalized for that client. We retain this permanent decoupling to align the masking protocol with FedSelect [23] and FedMC [24], enabling a more direct comparison in which the primary difference is the criterion used to identify parameters for personalization.

Let the layers be sorted by their CKA scores, denoted as s_{(1)}\leq s_{(2)}\leq\dots\leq s_{(L)}, where the subscript (k) represents the layer index with the k-th lowest similarity score, p_{l} be the parameter count of layer l, and P be the total parameters in the network. Given a target personalization ratio \rho\in[0,1] at each communication round, we iteratively select the most dissimilar layers until their combined parameter count reaches the threshold \rho P. Specifically, we select the first k sorted layers such that:

\sum_{j=1}^{k-1}p_{(j)}<\rho P\leq\sum_{j=1}^{k}p_{(j)}

The layers selected in the current round are then defined as V=\{(1),\dots,(k)\}, and the cumulative mask is applied as:

M_{l}=\begin{cases}1&\text{if }M_{l}^{\mathrm{prev}}=1\text{ or }l\in V\\
0&\text{otherwise}\end{cases}

Here, M_{l}^{\mathrm{prev}} denotes whether layer l was personalized in the previous communication round. Since masking is performed at the layer granularity, the selected personalized ratio can exceed \rho when the final selected layer, (k), crosses the target.

Threshold-based masking: Instead of enforcing a target proportion of the network to be personalized, this strategy evaluates the divergence of each layer independently. For a predefined similarity threshold \tau\in[0,1], a layer is personalized only if its CKA score indicates sufficient divergence from the global model:

M_{l}=\begin{cases}1&\text{if }s_{l}<\tau\\
0&\text{otherwise}\end{cases}

Unlike the percentage-based approach, which dictates a fixed personalization footprint for each client regardless of actual layer alignment, threshold-based masking dynamically adjusts the volume of personalized parameters. Moreover, the threshold-based mask is not cumulative. Consequently, a layer personalized in round t-1 can become globally shared again in round t if its CKA similarity no longer falls below \tau. Consequently, the fraction of parameters withheld can vary across clients and across rounds according to their layer-wise divergence, allowing the system to adapt.

IV Experiment Design

IV-A Dataset

The experimental framework assigns each client to a distinct environmental domain. Table I defines these five domains (A-E), spanning Boston and Singapore across varying weather and lighting conditions. This benchmark uses the nuScenes dataset, one of the most impactful datasets for onboard perception [29], which distinguishes between time of day and weather. The scenes, each consisting of roughly 40 annotated samples in a 20-second time window, are split such that each client acts as a distinct domain and possesses all data for that domain, as shown in Table I. These clients exhibit heterogeneity not only in location, time of day, and weather, but also in scene count and class composition. To illustrate, the Singapore training set contains, on average, three times as many annotated motorcycles, while Boston features more than twice as many cars. Although domain E’s three validation scenes may yield noisy standalone metrics, their compounded domain shifts and the small training set provide a stress test for knowledge transfer at the edge of the federated system.

IV-B Detector

To isolate federated data heterogeneity from baseline detector instability, we employ the Cross-Modal Transformer (CMT). CMT provides strong robustness via attention-based multimodal fusion [30] and outperforms state-of-the-art methods on synthetic corruption benchmarks [13]. Across all experiments, detector configuration and hyperparameters remain constant to isolate the impact of federated strategies.

IV-C Federated Strategies

The benchmark compares strategies that either improve a shared global model or preserve client-specific parameters for personalization. Globalization approaches include FedAvg [2], FedDyn [17] to limit client drift, and PCGrad [19] to align client gradients. Personalization baselines are divided into two categories. Layer-wise methods include FedBN [4] for personalized batch normalization and FedRep [22] for head personalization. Meanwhile, parameter-selection methods include FedSelect [23] to personalize the highest-updating parameters and FedMC [24] for the most influential parameters. All methods are run for 40 rounds with one local epoch.

IV-D Evaluation

To contextualize the benchmark, we establish two reference points: isolated own-domain training (a client’s baseline without federated collaboration) and centralized training (the expected upper bound of the federated system). These points contextualize the federated results, which are reported in mean Average Precision (mAP), nuScenes Detection Score (NDS), and Car Average Precision (CAP). Given that domains C, D, and E lack 1, 3, and 6 of the 10 target classes, respectively, mAP and NDS may be skewed. The mAP and NDS are computed only over classes or attributes that are available in the validation domains. Consequently, to provide a stable, high-volume single-class comparison, we report CAP, which is the mean over the four nuScenes distance thresholds. Finally, to account for the imbalance in validation scene counts across domains, all reported averages are weighted by the number of validation scenes.

V Experiments

Table II compares FedCKA with the federated and non-federated references defined above. Table II demonstrates that our approach outperforms all federated baselines and most closely approaches the centralized reference, achieving a validation-scene-weighted NDS of 0.63, compared to 0.55 for FedSelect and 0.56 for FedMC. Qualitative results for the most shifted domain, Singapore night rain, are also shown in Fig. 3.

V-A Globalization versus Personalization

Traditional globalization methods do not produce a consensus model that rivals centralized performance. Static-layer personalization strategies, FedBN and FedRep, which personalize the batch normalization (BN) layers or the head of each client, perform comparably to globalization methods. However, even with this targeted adaptation, these models fall short. These BN and head layers may not have sufficient power to move the client models towards their local optima, as they have fewer personal parameters than the parameter-level personalization strategies and do not influence the crucial intermediate cross-modal merging stages.

Among the established baselines, the parameter-level personalization strategies of FedSelect and FedMC achieve the highest performance. However, the proposed FedCKA method achieves the highest performance across the three metrics (mAP, NDS, CAP), suggesting that adaptive layer-wise personalization better accommodates domain discrepancies than fixed or parameter-magnitude-based personalization.

Ultimately, the difficulty of achieving a global consensus model across these clients highlights the severity of the domain gap. Because globalization forces different domains toward a single shared optimum, the resulting model remains suboptimal. Conversely, the success of the parameter-level baselines and FedCKA demonstrates that, by allowing a sufficiently large or important group of parameters to adapt to a domain, client models can move closer to their local optima.

V-B Cross-Domain Generalization

Cross-domain generalization is a key test of robustness in a domain-heterogeneous setting, revealing the limitations of both isolated training and standard aggregation, as detailed in Table III. Table III(a) shows that although isolated training yields rich local models for domains A and C, their cross-domain transfer remains poor. For instance, Model A drops to an NDS of 0.15 on domain E, indicating that strong own-domain performance does not necessarily translate into general cross-domain knowledge. This underscores that the benchmark captures meaningful domain heterogeneity rather than only differences in local dataset size.

Conversely, global aggregation with FedAvg, FedDyn, or PCGrad, as shown in Table II, sacrifices local performance on the richer domains A, B, and C, while maintaining stable but limited cross-domain performance. Because globalization forces all clients to share a single global model, a cross-domain evaluation matrix for these methods yields identical rows for every client, indicating stable yet consistently suboptimal generalization across all domains.

Personalized federated methods aim to mitigate this trade-off by retaining client-specific components while still benefiting from collaborative training. As shown in Table III(b), FedSelect improves cross-domain transfer on average over isolated training, confirming that personalized models can still benefit from shared knowledge across clients.

Compared with FedSelect, Table III(c) shows that FedCKA, which aggregates only layers with high representational similarity, boosts not only cross-domain performance but also own-domain performance (shown on the diagonal), surpassing all federated methods previously evaluated in Table II. It is the only federated method where all five clients improve over training only on their own local datasets, and the cross-domain transfer achieves the highest average off-diagonal NDS. This suggests that FedCKA enables beneficial knowledge transfer across clients while preserving domain-specific representations through personalization.

V-C Masking Strategy Ablation

Both proposed personalization strategies, detailed in Section III-D, depend on either a personalization ratio (\rho) or a similarity threshold (\tau). At the upper bound (\rho=1 or \tau=1), the models are entirely personalized (except those with mathematically perfect representational alignment), reducing to the “Own Domain Only” baseline in Table II; at the lower bound (\rho=0 or \tau=0), the system defaults to FedAvg aggregation. Table IV summarizes the impact of varying these parameters. Although thresholding is monotonic for a fixed set of CKA scores, each \tau induces a different training trajectory and permits later layer re-integration, so the personalization ratio need not vary monotonically with \tau. The \rho=0.40 final personalization target is additionally used as an ablation reference to compare the fixed personalization ratio used by FedSelect and FedMC. For other detectors or applications, the masking hyperparameter can be selected according to the desired trade-off between global alignment and client-specific adaptation. A small set of values spanning this range can first be evaluated, after which the search can be refined toward the better-performing region. The resulting masks remain stable across repeated runs and consistently isolate the same structural components, suggesting that FedCKA reliably targets domain-sensitive layers rather than collapsing into arbitrary or fully local personalization.

V-D Communication and Computation Footprint

To analyze the trade-off between parameter sharing and local specialization, we estimate client-to-server upload payloads based on the average fraction of shared parameters. We model mask metadata as bit-packed binary masks: layer-level for FedBN, FedRep, and FedCKA, and parameter-level for FedMC and FedSelect. This first-order estimate excludes protocol overhead, latency, compression, optimizer states, and server-to-client broadcasts.

As shown in Table V, FedCKA retains an upload payload comparable to full-model aggregation, reinforcing that its primary objective is dynamic personalization rather than communication compression. By using representation similarity, FedCKA flexibly decides which layers to share and which to retain locally, allowing clients to personalize different structural subsets with negligible metadata overhead. In contrast, parameter-level masking methods reduce the transmitted model payload more substantially in our experiments because they use higher personalized parameter fractions; at matched personalization ratios, FedCKA incurs lower masking metadata because it operates at the layer level.

The additional computational cost of FedCKA comes from the client-side forward passes required to compute layer-wise CKA similarities. In our setup, this adds approximately 14{,}000 seconds of computation over the full training procedure, corresponding to 2.7\% of the total training time, with a peak VRAM usage of 6 GB during the CKA computation.

V-E Robustness Analysis

To assess run-to-run variability, we repeat the main non-federated and personalized federated experiments three times. Table VI reports the mean and standard deviation across these runs. FedCKA achieves an average NDS of 0.635\pm 0.008 and CAP of 0.810\pm 0.014, remaining the strongest personalized federated method across the repeated experiments. Although some variation is present between runs, the observed performance margin over the competing personalized methods is substantially larger than the corresponding run-to-run standard deviations. These results indicate that FedCKA’s performance gains are not attributable to a single favorable training run.

We additionally evaluate the robustness of the CKA-based masking decisions to the number of local samples used to estimate representation similarity. Across three independent runs, using 10 randomly sampled inputs produced the same layer-selection decisions as using 100 samples. We therefore use 10 samples throughout the experiments. Moreover, because masking decisions are re-evaluated every round, layers erroneously personalized by noisy CKA estimates can revert to global aggregation for the main method. This choice is specific to the evaluated dataset and detector and should not be interpreted as a universally optimal sample count.

VI Discussion

The results on this problem-specific benchmark indicate that, for federated, data-heterogeneous 3D object detection, the global aggregation methods underperform relative to certain personalization methods. Although manually chosen static layer personalization provides a baseline improvement over FedAvg, directly inferring personalization masks from the client models further boosts performance. Compared with magnitude- or gradient-based personalization, the results suggest that representation similarity is a stronger criterion for selecting personalized layer or parameter subsets, improving both out-of-domain and smaller-domain performance.

FedCKA reduces the need to hand-pick specific layer types or impose fixed personalization ratios, since the masking decision is instead based on measured representational similarity. Furthermore, FedCKA does not assume that same-index parameters serve the same function across clients, since it compares layer activations rather than parameter-space differences.

Inspection of the learned masks suggests the architectural sources of domain sensitivity. Normalization layers are frequently personalized, reflecting their sensitivity to domain-specific feature scaling driven by weather, illumination, and geography. Similarly, personalization in deeper backbone, neck, and attention layers indicates that intermediate cross-modal representations, particularly the fusion weights of camera and LiDAR inputs, can be domain-sensitive. The task head remains largely shared, which may help preserve knowledge of classes absent from specific local domains and suggests that object-level prediction semantics are transferable, while the internal representations may adjust due to various shifts. This indicates that FedCKA does not create independent local models but instead preserves a common detection-task structure.

VII Limitations

The primary limitation of this study is that the evaluation is restricted to a single detector, dataset, and domain split. The domain split is also highly imbalanced, meaning that the observed gains for smaller domains may depend on the specific scene distribution and validation composition.

A second limitation is that the framework assumes supervised local training, which requires the availability of reliable client-side labels or a labeling pipeline. This assumption is restrictive because newly collected IV data does not naturally include high-quality 3D bounding boxes. Although offline autolabelling on idle vehicles could provide pseudo-labels, this work does not evaluate how pseudo-label noise would affect federated training.

Another limitation is that although the CKA computation is relatively lightweight compared to model training, it still incurs additional computational costs for the client. It also requires flattening intermediate representations into two-dimensional matrices, which may simplify the original positional information. Although this simplification potentially limits the captured positional dependencies, our results indicate that the flattened representations remain effective for layer-wise similarity assessment in the evaluated setting.

VIII Conclusion and Future Work

This work studied federated learning for data-heterogeneous 3D object detection under shifts in weather, time, and location, showing that adaptive partial personalization can outperform a single consensus model on this benchmark. The findings suggest that representation similarity is a useful criterion for balancing this globalization and local adaptation. For real-world robotic perception in IV deployment, cross-domain robustness and local domain adaptation are both vital, and these findings may help clarify which model layers are best suited for local adaptation.

Future work should evaluate whether the observed benefits hold across different detectors, datasets, domain definitions, and types of heterogeneity. Methodologically, FedCKA could be extended with adaptive thresholds, temporally smoothed masks, unsupervised domain clustering based on CKA scores, and further analysis of mask stability across domain batches and batch sizes. Furthermore, a separate direction is to evaluate whether masking reduces global drift, and whether FedCKA remains effective under pseudo-labeled local training from offline autolabelling pipelines.

Finally, FedCKA does not, by itself, provide formal privacy guarantees. Combining CKA-guided masking with differential privacy is therefore an important direction for privacy-preserving deployment. Overall, this work shows that representation similarity can provide a principled and empirically effective basis for deciding what should remain shared and what should become client-specific in federated 3D-OD.

IX Acknowledgements

The Cynergy4MIE project is supported by the Chips Joint Undertaking and its members, including the top-up funding by National Authorities under Grant Agreement No 101140226.

Research reported in this work was partially facilitated by computational resources and support of the Delft AI Cluster (DAIC) at TU Delft (RRID: SCR_025091), but remains the sole responsibility of the authors, not the DAIC team.

The authors acknowledge the use of Google Gemini, ChatGPT, and Grammarly to assist in the preparation of this manuscript. These were utilized throughout the work for structural editing, rephrasing, checking internal consistency, and visually enhancing Figure 2. The authors iteratively guided, carefully reviewed, and refined all AI-assisted edits and take full responsibility for the final content and scientific integrity of the work.

References

1. [1] Y. Dong, C. Kang, J. Zhang, Z. Zhu, Y. Wang, X. Yang, H. Su, X. Wei, and J. Zhu (2023) Benchmarking robustness of 3D object detection to common corruptions. In Proc. CVPR, pp. 1022–1032.
2. [2] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas (2017) Communication efficient learning of deep networks from decentralized data. In Proc. AISTATS, pp. 1273–1282. External Links: Link
3. [3] E. Strubell, A. Ganesh, and A. McCallum (2019) Energy and policy considerations for deep learning in NLP. In Proc. ACL, pp. 3645–3650. External Links: Link, Document
4. [4] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou (2021) FedBN: federated learning on non-IID features via local batch normalization. In Proc. ICLR,
5. [5] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith (2020) Federated optimization in heterogeneous networks. In Proc. MLSys, pp. 429–450.
6. [6] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. U. Stich, and A. T. Suresh (2020) SCAFFOLD: stochastic controlled averaging for federated learning. In Proc. ICML, pp. 5132–5143.
7. [7] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson (2014) How transferable are features in deep neural networks?. In Proc. NeurIPS, pp. 3320–3328.
8. [8] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom (2020) nuScenes: a multimodal dataset for autonomous driving. In Proc. CVPR, pp. 11621–11631.
9. [9] L. Zhao, X. Li, X. Jia, and L. Fu (2025) Personalized FedM2former: an innovative approach towards federated multi-modal 3D object detection for autonomous driving. Processes 13 (2), pp. 449. External Links: Link, ISSN 2227-9717, Document
10. [10] B. Li, S. Ren, S. Zhao, M. Cui, and K. Huang (2025) FedPillarNet: unifying personalized and global features for federated 3D LiDAR object detection. Journal of Systems Architecture 168, pp. 103584. External Links: ISSN 1383-7621, Document, Link
11. [11] H. Wang, J. Liu, H. Dong, and Z. Shao (2025) A survey of the multi-sensor fusion object detection task in autonomous driving. Sensors 25 (9), pp. 2794.
12. [12] J. Cai and Y. Wu (2025) Three-dimensional object detection for autonomous driving via deep learning: a review. Engineering Applications of Artificial Intelligence 161, pp. 112238. External Links: ISSN 0952-1976, Document, Link
13. [13] T. Beemelmanns, Q. Zhang, C. Geller, and L. Eckstein (2024) MultiCorrupt: a multi-modal robustness dataset and benchmark of LiDAR-camera fusion for 3D object detection. In Proc. IV, Vol. , pp. 3255–3261. External Links: Document
14. [14] J. Pei, W. Liu, J. Li, L. Wang, and C. Liu (2024) A review of federated learning methods in heterogeneous scenarios. IEEE Transactions on Consumer Electronics 70 (3), pp. 5983–5999. External Links: Document
15. [15] Y. Li, X. Wang, R. Zeng, P. Kumar Donta, I. Murturi, M. Huang, and S. Dustdar (2025) Federated domain generalization: a survey. Proceedings of the IEEE 113 (4), pp. 370–410. External Links: Document
16. [16] T. H. Hsu, H. Qi, and M. Brown (2020) Federated visual classification with real-world data distribution. In Proc. ECCV, pp. 76–92. External Links: Document
17. [17] D. A. E. Acar, Y. Zhao, R. Matas Navarro, M. Mattina, P. N. Whatmough, and V. Saligrama (2021) Federated learning based on dynamic regularization. In Proc. ICLR,
18. [18] G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi (2023) Editing models with task arithmetic. In Proc. ICLR,
19. [19] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn (2020) Gradient surgery for multi-task learning. In Proc. NeurIPS, pp. 5824–5836.
20. [20] X. Zhang, W. Sun, and Y. Chen (2024) Tackling the non-IID issue in heterogeneous federated learning by gradient harmonization. IEEE Signal Processing Letters 31, pp. 2595–2599. External Links: Document
21. [21] T. B. Nguyen, D. M. Nguyen, J. Park, V. Q. Pham, and W. Hwang (2025) Federated domain generalization with data-free on-server matching gradient. In Proc. ICLR,
22. [22] L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai (2021) Exploiting shared representations for personalized federated learning. In Proc. ICML, pp. 2089–2099.
23. [23] R. Tamirisa, C. Xie, W. Bao, A. Zhou, R. Arel, and A. Shamsian (2024) FedSelect: personalized federated learning with customized selection of parameters for fine-tuning. In Proc. CVPR, pp. 29485–29494.
24. [24] Y. Gao, X. He, and Y. Chen (2025) Personalized federated learning algorithm based on information content model customization. In Proc. CAMMIC, pp. 834–838. External Links: ISBN 9798400713873, Link, Document
25. [25] S. P. Singh and M. Jaggi (2020) Model fusion via optimal transport. In Proc. NeurIPS, pp. 1849. External Links: ISBN 9781713829546
26. [26] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton (2019) Similarity of neural network representations revisited. In Proc. ICML, pp. 3519–3529.
27. [27] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf (2005) Measuring statistical dependence with Hilbert-Schmidt norms. In Proc. ALT, pp. 63–78.
28. [28] M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy (2021) Do vision transformers see like convolutional neural networks?. In Proc. NeurIPS, pp. 12116–12128.
29. [29] M. Liu, E. Yurtsever, J. Fossaert, X. Zhou, W. Zimmer, Y. Cui, B. L. Zagar, and A. C. Knoll (2024) A survey on autonomous driving datasets: statistics, annotation quality, and a future outlook. IEEE Transactions on Intelligent Vehicles 9 (11), pp. 7138–7164. External Links: Document
30. [30] J. Yan, Y. Liu, J. Sun, F. Jia, S. Li, T. Wang, and X. Zhang (2023) Cross modal transformer: towards fast and robust 3D object detection. In Proc. ICCV, pp. 18268–18278.