Physical AI for Intralogistics: Benchmarking the NVIDIA Sim-to-Real Ecosystem for Cosmos-Enhanced COMPASS Navigation on Industrial Mobile Robots

Abstract

Classical robot navigation stacks are reliable but often require re-engineering for each new robot type and site. Learning-based navigation shifts this effort from hand-tuned parameters to data generation and training, where a policy adapts to each new embodiment or facility by fine-tuning in simulation. Whether policies adapted this way stay reliable on real industrial robots has seen little systematic testing. We study this for NVIDIA’s X-Mobility and COMPASS workflow for training learned navigation policies on Idealworks robots. In simulation, we extend the COMPASS benchmark with two embodiments, the differential-drive iw.hub and an omnidirectional Humanoid Mobile Platform (HMP), and with a digital twin of the Idealworks R&D center. For the real world, we fine-tune X-Mobility on data from this digital twin, with and without sim-to-real variants generated by NVIDIA Cosmos, train COMPASS on top of it from scratch in the digital twin, deploy the resulting policies on the iw.hub’s onboard Jetson AGX Xavier, and benchmark them in the Idealworks R&D center against COMPASS trained only on NVIDIA’s environments and the existing Idealworks navigation stack on task success, efficiency, safety, smoothness and onboard compute. Running every static scenario both in a digital-twin replica and in the real facility measures the sim-to-real gap and how far Cosmos-augmented data closes it. Training in the digital twin with Cosmos sim-to-real augmentation raises the real-world success rate of learned navigation from 65.3% to 91.7% across four scenarios and nearly halves its weighted travel time compared with simulation-only training, bringing it within 8.3 points of the production navigation stack, fully onboard; with an integrated global planner, it also reaches far, occluded goals in every trial.

\abscontent

1 Introduction

Autonomous mobile robots (AMRs) have become a core component of modern intralogistics, moving material between storage, production and shipping areas without fixed infrastructure. Their navigation is typically built as a modular pipeline: the robot localizes itself in a prior map, a global planner computes a route to the goal, and a local planner follows that route while avoiding obstacles, as in the widely used ROS 2 navigation stack Nav2 (Macenski et al., 2020). The local planners of such stacks are hand-designed, for example the dynamic window approach (Fox et al., 1997) or timed elastic bands (Rösmann et al., 2017), and operate on costmaps built from lidar and depth sensors.

These stacks are mature and dependable in the conditions they were engineered for, but they are expensive to extend. Their behavior emerges from many interacting parameters, such as cost weights, inflation radii, velocity and acceleration limits and recovery behaviors, which must be re-tuned for every new robot geometry, drive configuration and facility layout. Their perception is largely geometric: an obstacle is an occupied cell, with no notion of what it is or how it will move, which limits their behavior around people, other vehicles and semi-structured clutter. As fleets grow to more robot types and more sites, this per-robot and per-site engineering becomes a dominant cost of deployment.

Physical AI, the application of learned models to systems that perceive and act in the physical world, offers a different path: rather than being engineered, a navigation policy is learned from data and improves as more data becomes available. Several developments now make this practical. GPU-accelerated simulation such as Isaac Lab (Mittal et al., 2023) trains policies across thousands of parallel environments, which makes interaction data cheap. World models learn compact, predictive representations of the environment from which behavior can be learned (Ha and Schmidhuber, 2018). Navigation foundation models trained on data from many robots, such as GNM, ViNT and NoMaD (Shah et al., 2023a; Shah et al., 2023b; Sridhar et al., 2024), show that a single policy can drive different embodiments, in line with the cross-embodiment results of large-scale robot learning (Open X-Embodiment Collaboration, 2024). Training in simulation, however, raises its own challenge: a policy that performs well in simulation can fail on a real robot, because rendered images, physics and sensor noise never match reality exactly. This sim-to-real gap is the main obstacle to deploying simulation-trained policies (Zhao et al., 2020). A classical answer is domain randomization, which varies textures, lighting and other simulation parameters during training so that the real world appears to the policy as just another variation (Tobin et al., 2017). Generative world foundation models such as NVIDIA Cosmos (NVIDIA, 2025a) take this a step further: they turn simulated footage into photorealistic variations that preserve its geometry and motion, bringing the training data closer to what a robot’s cameras see in a real facility.

Together, these advances make learned navigation policies the natural next step for industrial navigation. A learned policy adapts to a new robot or a new site through data and fine-tuning rather than manual re-engineering; it can use camera images and the semantics they carry, not only geometry; and a single training pipeline can serve a whole fleet of embodiments. Two questions stand between this promise and industrial deployment: whether learned policies remain reliable outside simulation, on real robots in real facilities, and whether they can match the production navigation stacks they would replace on the metrics that matter in operation, namely task success, efficiency, safety and onboard compute.

NVIDIA’s X-Mobility (Liu et al., 2024) and COMPASS (Liu et al., 2025) are a concrete instance of this approach. X-Mobility learns an end-to-end navigation policy: a world model, trained on random-exploration data, encodes camera images, route information and robot state into a latent representation, and a policy head, trained by imitation on Nav2 expert demonstrations, decodes this representation into velocity commands. COMPASS freezes the X-Mobility policy and trains a residual policy on top of it with reinforcement learning in Isaac Lab, learning only the corrections a given embodiment needs, either as a specialist for one robot or as a generalist distilled across several.

Idealworks develops autonomous mobile robots for intralogistics together with the software that operates them. Its iw.hub is a differential-drive AMR that navigates with the Idealworks navigation stack, its production navigation system, and its Humanoid Mobile Platform (HMP) is a humanoid robot on an omnidirectional mobile base. New navigation capabilities at Idealworks are developed and validated first in simulation and then in the Idealworks R&D center, a facility used to test robots under warehouse conditions before customer deployment. This makes the Idealworks robots and the R&D center a natural testbed for the two questions above.

Scope of This Work

This work evaluates whether the X-Mobility (Liu et al., 2024) and COMPASS (Liu et al., 2025) navigation stack, combined with sim-to-real data augmentation using NVIDIA Cosmos (NVIDIA, 2025a), can be adapted to Idealworks robots and deployed in a real facility. We address this question in two benchmarks, one in simulation and one on a real robot.

Simulation benchmark. We extend the simulation benchmark of COMPASS with two Idealworks embodiments, the differential-drive iw.hub and the omnidirectional HMP, and with a new evaluation environment, the digital twin of the Idealworks R&D center (Section 2). Because the released stack moves the robot with a kinematic controller that bypasses actuation dynamics, we adapt physics-based controllers to both robots: a differential drive controller for the iw.hub and an omnidirectional controller for the HMP. The HMP is evaluated in simulation only.

Real-world benchmark. On a physical iw.hub in the Idealworks R&D center, we compare three COMPASS policies with the Idealworks navigation stack, the production navigation system of the iw.hub (Section 3). The three add training on the R&D center step by step: NVIDIA COMPASS, the iw.hub Specialist of the simulation benchmark, is trained only on NVIDIA’s environments; IW.COMPASS fine-tunes X-Mobility on data generated in the digital twin of the R&D center and trains its COMPASS residual there; and IW.CE-COMPASS does the same on a hybrid version of that data extended with Cosmos-augmented sim-to-real variants. Each static scenario is run both in a digital-twin replica and in the R&D center. For NVIDIA COMPASS, this measures how it generalizes to a new facility, in simulation and in the real world; for the policies trained in the digital twin, it exposes their sim-to-real gap and how far Cosmos-enhanced data closes it. The systems are compared on task success, efficiency, safety, motion smoothness and onboard compute. All learned policies run on the iw.hub’s onboard NVIDIA Jetson AGX Xavier. For far and occluded goals, a global planner integrated with the COMPASS ROS 2 node supplies the route that the COMPASS policy cannot plan on its own; it is used with IW.CE-COMPASS on the real robot.

Organization. Section 2 introduces the two embodiments and the digital twin of the R&D center, and reports the simulation benchmark across the original and new embodiments and environments. Section 3 presents the real-world benchmark on the iw.hub: the compared navigation systems, their training data, the scenarios, the metrics, the onboard deployment and the results in the digital-twin replica and in the R&D center. Section 4 concludes and outlines future work. The appendices follow the order of the main text and detail the controller type study (Appendix A), the omnidirectional controller of the HMP (Appendix B), the global planner integration (Appendix C), the training datasets (Appendix D) and training pipeline (Appendix E) of the real-world models, samples of the Cosmos sim-to-real augmentation (Appendix F), the deployment on the Jetson AGX Xavier (Appendix G) and the results of all metrics of the COMPASS policies in the digital-twin replica (Appendix H).

2 Extending the COMPASS Benchmark to New Embodiments and Environments

In this section, we build on the simulation benchmark of COMPASS (Liu et al., 2025) and extend it with two new embodiments, the iw.hub and the Humanoid Mobile Platform (HMP), and one new evaluation environment, the digital twin of the Idealworks R&D center.

iw.hub. The iw.hub is a differential-drive autonomous mobile robot (Figure 1(a)) designed for intralogistics tasks. It features a compact form factor suitable for navigating warehouse aisles, a payload capacity for transporting goods, and a sensor package that includes one LIPSEdge AE400 camera, two 2D LiDARs, and IMU sensors. The robot is equipped with an onboard Jetson AGX Xavier for on Edge computations.

HMP. The HMP is an omnidirectional Humanoid Mobile Platform (Figure 1(b)), a universal mobile base designed for autonomous mobile manipulation robots (AMMRs). It can carry a humanoid upper body with multiple degrees of freedom, an omnidirectional drive system for enhanced maneuverability, and a comprehensive sensor suite including multiple cameras, 2D LiDARs and IMU sensors. For computation, the platform carries two onboard NVIDIA modules, a Jetson AGX Orin and a Jetson AGX Thor.

Idealworks R&D center. The Idealworks R&D center is a physical research and development facility. The simulation environment in Figure 2 is its digital twin, replicating the layout, shelving, and other key features of the physical environment, providing a realistic simulation setting for evaluating navigation policies. In this benchmark, the policies are evaluated in the digital twin on a single, standalone evaluation layout of the R&D center (Figures 2(a) and 2(b)). Its floor is populated with 22 objects (7 pallets, one box stack, 5 dollies, 7 iw.hubs and 2 forklifts), placed in groups of one to three at random positions and orientations in the open zone south of the racks. Groups keep enough space between them for the robot to pass, so the layout stays fully navigable: 241 m2 remain drivable for the iw.hub, about two thirds of the drivable area of the empty R&D center. The lighting is randomized as well, slightly brighter than nominal, with 10 of the 46 ceiling lights switched off. Start and goal poses are sampled on the occupancy grid of this layout.

Integration into Isaac Lab and COMPASS. The released COMPASS stack moves the robot with a kinematic controller that updates its pose directly, bypassing actuation dynamics. We adapted physics-based controllers to our embodiments: a differential drive controller (DDC) for the iw.hub and an omnidirectional controller for the HMP (Appendices A and B detail their design and testing).

Training. All policies in this benchmark build on the pre-trained X-Mobility policy released by NVIDIA, which is used as is, without fine-tuning, as the frozen base policy of COMPASS (Figure 3). Each new embodiment is integrated with its own controller, and a COMPASS residual policy is trained for it in NVIDIA’s environments of the original benchmark, which yields the iw.hub and HMP Specialists. The digital twin of the R&D center is used only for evaluation; no policy in this benchmark is trained on it. The Generalist is distilled from the two Specialists, following COMPASS (Liu et al., 2025).

We evaluate trained policies across five environments: the four environments of the original benchmark (Warehouse Multi-Rack, Warehouse Single-Rack, Simple Office and Combined Multi-Rack, abbreviated in the tables as Warehouse M-R, Warehouse S-R, Office and Combined M-R) and the digital twin of the R&D center. Each policy is trained for 1,000 iterations using 32 parallel simulation environments. Each policy is evaluated over 20 episodes in each of 32 parallel environments, that is 640 trials per environment, as in COMPASS. An episode terminates upon successful goal completion, collision, or timeout.

We report the success rate (SR), the percentage of episodes in which the agent reaches the goal without collision or timeout, and the weighted travel time (WTT), the average travel time to the goal conditioned on success and normalized by the success rate. We compute both metrics exactly as the COMPASS benchmark does, so that the results are directly comparable with those reported for the original embodiments; as in COMPASS, the WTT values are of the order of thousands, and they are not comparable in absolute terms with the per-trial travel times of the real-world benchmark of Section 3. We report the findings in Table 1, which summarizes the performance of the COMPASS Specialist and Generalist policies across all evaluation environments.

For the iw.hub, the Specialist reaches the goal in 58% (Office) to 85% (Warehouse Multi-Rack) of the episodes and outperforms the Generalist in every environment, by 5 to 22 points, the largest gap being in the Office. For the HMP, the Generalist matches the Specialist: their success rates differ by at most 3 points in every environment, so distilling across embodiments costs this platform almost nothing. The HMP succeeds less often than the iw.hub overall (29% to 74% for its Specialist) and takes about twice as long to reach the goal. The Office is the hardest environment for every policy, with success rates of 29% to 58%. The digital twin of the R&D center, on which no policy is trained gives the iw.hub Specialist its second-best success rate (82%) and every other policy its best one (72% to 77%).

Overall, both Idealworks embodiments navigate across all five environments with COMPASS on top of NVIDIA’s pre-trained X-Mobility policy, and the policies trained in NVIDIA’s environments transfer to the digital twin of the R&D center without having seen it. This extends the COMPASS benchmark with a differential-drive and an omnidirectional platform, each of which required a dedicated physics-based controller, and with a new environment modeled on a real intralogistics facility.

3 Real-World Navigation Benchmark

This section presents a real-world navigation benchmark that assesses the deployability and performance of COMPASS navigation policies on a physical iw.hub in the Idealworks R&D center. We evaluate the policies along several dimensions: generalization to out-of-distribution scenarios and environments, deployability on a real robot, onboard compute efficiency, safety and robustness.

COMPASS policies are trained entirely in simulation, so running them in the real scenarios of the R&D center measures their intrinsic sim-to-real transfer. To make this transfer as robust as production use demands, leaving as little as possible to chance when moving from simulation to the real facility, we also strengthen the X-Mobility backbone of COMPASS: we generate photorealistic variants of the training episodes with NVIDIA Cosmos and train the backbone on a hybrid dataset of original and Cosmos-augmented episodes.

The benchmark compares four navigation systems (described under Models): the Idealworks navigation stack as the robotics baseline and three COMPASS policies that add training on the R&D center step by step: the COMPASS policy trained only on NVIDIA’s environments (the iw.hub Specialist of Section 2), the COMPASS policy trained on simulated data from the digital twin of the R&D center, and the Cosmos-enhanced COMPASS policy trained on a hybrid of that data and its sim-to-real variants. Each system is evaluated across the scenarios and with the metrics described under Scenarios and Metrics, and we report the findings in Table 4 for the R&D center, and in Table 5 for onboard compute. The COMPASS policies are also evaluated on a digital-twin replica: a one-to-one copy of the static scenarios in the digital twin of the R&D center (Table 6). For NVIDIA COMPASS, which has never seen the R&D center, the digital-twin replica measures how it generalizes to a new facility in simulation, and the real runs how it transfers to the real world; for IW.COMPASS and IW.CE-COMPASS, comparing the two gives their sim-to-real gap.

Models.

Three COMPASS models are compared; Figure 4 shows how each is obtained. They form a lineage that adds training on the target facility step by step: NVIDIA COMPASS has seen no data from the R&D center, IW.COMPASS is trained on simulated data from its digital twin, and IW.CE-COMPASS additionally makes that data photorealistic with Cosmos. Comparing them shows what fine-tuning in simulation on the target facility brings, and what sim-to-real data adds on top of it.

NVIDIA COMPASS. Exactly the iw.hub Specialist of Section 2, evaluated there in simulation and deployed here unchanged on the real robot: NVIDIA’s pre-trained X-Mobility policy, kept frozen, with a COMPASS residual trained for the iw.hub and its differential drive controller in NVIDIA’s environments (Figure 3). Neither its X-Mobility base nor its residual has seen any data from the R&D center.

IW.COMPASS. Starting from the NVIDIA checkpoints, we fine-tune the X-Mobility world model on the random-exploration episodes of the digital twin (described under Training data below), which yields IW.WM, and then the policy on the path-following episodes, which yields IW.IL (Appendix E). The COMPASS residual policy is then trained from scratch on top of the frozen IW.IL in layout 04, a dense layout of the digital twin described under Training data below.

IW.CE-COMPASS. The same fine-tuning on the hybrid dataset, in which every simulated episode is followed by its Cosmos sim-to-real copy (50% simulated, 50% sim-to-real; Appendix F), yields IW.CE-WM and IW.CE-IL, and the COMPASS residual policy is again trained from scratch on top of the frozen IW.CE-IL, in layout 04 and with the same setup as for IW.COMPASS.

IW Nav. The robotics baseline is the Idealworks navigation stack, the proprietary navigation system deployed on the iw.hub in production. It is a LiDAR-based stack that plans routes with a global planner and follows them with a local planner. It is evaluated in the real world on the same scenarios, start poses and goals as the learned policies, but not in the digital twin.

The prefix IW. marks a model whose X-Mobility base is fine-tuned from the NVIDIA checkpoints on data from the digital twin of the Idealworks R&D center, and IW.CE one fine-tuned on the Cosmos-enhanced (CE) hybrid version of that data; in both, the COMPASS residual is trained from scratch in the digital twin. NVIDIA COMPASS, without prefix, keeps NVIDIA’s pre-trained X-Mobility base and has its residual trained in NVIDIA’s environments. Appendix E details the training stages, hyperparameters and evaluations of all models mentioned.

Training data.

The IW models are trained on data generated in the digital twin of the R&D center. Rather than its single original arrangement, we build 16 layouts of the R&D center, each with its own floor objects (pallets, dollies, iw.hubs and forklifts) at one of four density tiers, from near-empty to dense, and its own lighting (Figure 12 in Appendix D). Layouts 14 and 15, one sparse and one dense, are held out for testing. In these layouts we record with MobilityGen (NVIDIA, 2025b), using the iw.hub and its own front camera, 605 random-exploration episodes (76,284 frames) for the world model and 240 path-following episodes (56,535 frames) for the IL policy. Their semantic labels include the iw.hub and the dolly, two classes that the released X-Mobility model does not have and that we added to it (Appendix D.6). For the IW.CE models, Cosmos Transfer 2.5 turns every simulated episode into a photorealistic, frame-aligned copy, and the hybrid dataset appends these copies to the simulated episodes (Figure 5(b)).

The COMPASS residual of the IW models is trained online in one of these layouts, layout 04 (Figure 5(a)), chosen as a hard training environment. It belongs to the dense tier, with 36 objects (9 pallets, 5 box stacks, 8 dollies, 11 iw.hubs and 3 forklifts), so that only 185 m2, about half of the drivable area of the empty R&D center, remain drivable, and about 28% of its training routes are blocked by an obstacle across the width of the robot. Its lighting is dim, at 0.65 of the nominal brightness, with 9 of the 46 ceiling lights switched off. Appendix D details the layouts and datasets, Appendix E.3 the COMPASS training in layout 04, and Appendix F the Cosmos augmentation.

Scenarios

Table 2 lists the benchmark scenarios and Figure 6 shows their layouts. Each scenario is run over about 30 trials. The static scenarios S1, S2, S3 and S5 are also recreated in the digital-twin replica, with the same layout, obstacles, start poses and goals, and the COMPASS policies are evaluated there as well. The dynamic scenario S4 is not replicated: moving obstacles in the digital twin are not explored in this work. The Idealworks navigation stack is benchmarked only in the real world.

COMPASS is a local policy: it navigates towards a goal it can reach from its current surroundings, but on its own it cannot plan a route to a goal that is far away and hidden behind racks, walls or other occluders. For scenario S5 we therefore integrated a global planner with the COMPASS stack and modified the COMPASS ROS 2 node to accept its output.

Appendix C describes the integration and the node modifications. The global planner is implemented in the COMPASS ROS 2 node and is therefore used only on the real robot, where S5 is run with IW.CE-COMPASS, the best-performing learned model on S1–S4, and with the Idealworks navigation stack (Table 4). In the digital-twin replica, all three COMPASS policies are run on S5 without the global planner. None of them reaches the goal, although they cover most of the distance to it (Table 6 and Appendix H): the COMPASS policy alone, whatever its training, cannot plan this far ahead around an occluding obstacle. This observation from simulation is what motivated the global planner integration.

Metrics

Metrics are grouped into categories (Table 3). Arrows give the preferred direction. Unless stated otherwise, each metric is averaged over all trials of a scenario. Interaction metrics are reported for scenarios with dynamic obstacles only (S4); protective stops are reported for all scenarios. A trial succeeds when the robot reaches the goal within 0.5 m, with a linear speed below 0.1 m/s and an angular speed below 0.1 rad/s. A collision ends the trial as a failure, and so does a timeout when the robot is stuck: 2 min in general, longer in S3, where the robot may need more time to work its way through the obstacles.

Onboard Deployment

All learned policies run on the iw.hub’s onboard computer, an NVIDIA Jetson AGX Xavier, with no offboard compute. To run in real time on this hardware, the COMPASS model is exported to ONNX and compiled into a TensorRT engine for the Xavier, and inference is served by the COMPASS ROS 2 node on the robot. The Xavier runs CUDA 11.4 and TensorRT 8.5.2, so the model is exported with ONNX opset 16 for compatibility with this TensorRT version, and the engine is built in FP16 precision. The resulting engine takes 177 MB and runs at 5–8 Hz on the robot. Appendix G details the target platform and the conversion; Table 5 reports the resulting onboard compute.

Results

Table 4 reports all navigation systems on the iw.hub in the R&D center; scenario S5, which requires the global planner, is run only with IW.CE-COMPASS and the Idealworks navigation stack. Table 6 compares the success rate and weighted travel time of each COMPASS policy in the digital-twin replica of S1, S2, S3 and S5 with its real-world runs; this table is a reduced view, and Appendix H reports the results of all metrics in the digital-twin replica. In S5, the replica runs are without the global planner and the real runs with it. Table 5 reports onboard compute on the iw.hub.

Task success and efficiency. The scenarios separate the models more and more as they get harder. In the open space of S1, all three COMPASS policies reach the goal in every trial and differ only in how fast they get there: IW.CE-COMPASS needs 27.0 s, against 38.2 s for IW.COMPASS and 41.3 s for NVIDIA COMPASS. In the aisles of S2, NVIDIA COMPASS starts to fail (83.3%), while IW.COMPASS (96.8%) and IW.CE-COMPASS (100%) remain reliable, and IW.CE-COMPASS is the fastest system of the benchmark there, IW Nav included (13.5 s against 16.2 s). The static obstacles of S3 are the decisive scenario. NVIDIA COMPASS reaches the goal in only 6.7% of the trials and covers less than half of the distance to it on average (progress ratio, PR, of 46.2%); IW.COMPASS reaches 54.8%, and IW.CE-COMPASS 83.3%, with a time to goal of its successful trials (TTG) of 35.0 s, the same as IW Nav (34.9 s). Because the weighted travel time (WTT) divides the travel time by the success rate, it penalizes failed trials as well as slow ones, which is why the WTT of NVIDIA COMPASS in S3 reaches 4,998 s. In the dynamic scenario S4, IW.CE-COMPASS again has the highest success rate (83.3%) and the lowest WTT (52.8 s) of the learned policies. It recovers fastest once an obstacle has left its path (recovery time, RcT, of 5.02 s, close to the 4.19 s of IW Nav), although it reacts later than the other systems (reaction time, RT, of 6.21 s against about 3 s). With the global planner, IW.CE-COMPASS also reaches the far, occluded goal of S5 in every trial, within 6% of the travel time of IW Nav (37.9 s against 35.7 s).

Safety and smoothness. No COMPASS policy collides in S1 and S2. In S3 and S4, collisions (collision count, CC) and protective stops of the safety laser scanner (PS) appear for all of them. IW.CE-COMPASS triggers far fewer protective stops than NVIDIA COMPASS (1.5 against 5.7 per trial on average over S1–S4) but more than IW.COMPASS (0.7), which also collides least: IW.COMPASS is the most cautious learned policy, while IW.CE-COMPASS is the most successful and the fastest. IW.CE-COMPASS oscillates least: its commanded angular velocity changes sign 10.3 times per trial on average (oscillation count, OC), against 15.1 for IW.COMPASS and 48.0 for NVIDIA COMPASS, whose path in S3 is almost three times longer than the shortest one (path length ratio, PLR, of 2.83). The jerk of IW.CE-COMPASS is slightly higher than that of the other learned policies, but lower than that of IW Nav in every scenario except S2.

Comparison with the Idealworks navigation stack. IW Nav is the production navigation stack of the iw.hub, a mature system refined over many deployments, and the reference the learned policies aim to approach. It reaches the goal in every trial of every scenario, along almost straight paths (PLR of 1.00 to 1.04) and without collisions. It also starts from a stronger position: it localizes in a prior map of the facility and perceives obstacles with LiDAR, whereas COMPASS has no map and sees the scene through a single RGB camera. IW.CE-COMPASS comes closest to it: on average over S1–S4, it reaches 91.7% of the goals against 100%, and its WTT is 40% higher (33.9 s against 24.2 s), most of the gap coming from S3 and S4. That a camera-only learned policy, trained in simulation, comes this close to a LiDAR-based production stack is the strongest result of this benchmark.

Onboard compute. IW.CE-COMPASS runs entirely on the Xavier with a latency comparable to that of IW Nav (54 ms against 50 ms), a lower peak memory (0.89 GB against 1.16 GB) and a similar CPU load (Table 5). Unlike IW Nav, it uses the GPU (38.6%), and it issues commands at 5–8 Hz, against 15–20 Hz.

Sim-to-real gap. In the digital-twin replica (Table 6), IW.COMPASS and IW.CE-COMPASS perform almost the same in every scenario, for example 72.0% and 71.0% in S3, as expected from two policies trained on the same simulated layouts. On the real robot, they separate in S3, and NVIDIA COMPASS, already the weakest in simulation (51.3%), falls to 6.7%. Equal results in simulation and different results in the real world isolate the effect of the Cosmos sim-to-real data: it does not change how the policy behaves in simulation, but it makes it robust to the gap between simulation and reality. The digital-twin replica is a close, not exact, copy of the real scenarios: object poses and start headings could only be reproduced approximately in the real facility, while the replica was randomized more widely. Absolute simulated and real success rates are therefore not directly comparable, and a higher real than simulated rate (83.3% against 71.0% in S3) is not a negative sim-to-real gap. What is comparable is the ordering of the models, since all of them ran on the same scenarios with the same start and goal poses. Under these conditions, the simulation-only model drops in the real facility (72.0% to 54.8%), while the Cosmos-enhanced one does not. In S5, no policy reaches the goal in the replica, without the global planner, while IW.CE-COMPASS reaches it in every real trial with it.

Overall. Training in the digital twin of the R&D center, and making that data photorealistic with Cosmos, turns COMPASS from a policy that rarely clears obstacles in the real facility into one that approaches the production stack. On average over S1–S4, IW.CE-COMPASS raises the real-world success rate of learned navigation from 65.3% (NVIDIA COMPASS) to 91.7%, a gain of 26.3 points. The Cosmos sim-to-real augmentation alone accounts for 8.8 of these points over IW.COMPASS, while it reduces the weighted travel time by 47% (from 63.8 s to 33.9 s) and the oscillation count by 32%. In S3, the scenario that separates the policies most, it raises the success rate from 54.8% to 83.3% (+28.5 points).

4 Conclusion and Future Work

This work extended the COMPASS benchmark for learned navigation to two Idealworks embodiments and the digital twin of the Idealworks R&D center, and evaluated COMPASS policies on a physical iw.hub against the Idealworks navigation stack. In simulation, both embodiments navigated across all five environments, and policies trained only in NVIDIA’s environments transferred to the digital twin of the R&D center. On the real robot, training in the digital twin and augmenting its data with Cosmos sim-to-real episodes raised the success rate of learned navigation from 65.3% to 91.7% over S1–S4 and nearly halved the weighted travel time compared with simulation-only training. The resulting IW.CE-COMPASS came within 8.3 points of the Idealworks navigation stack while running entirely on the iw.hub’s Jetson AGX Xavier, and with the global planner integration it reached the far, occluded goal of S5 in every trial. The digital-twin replica showed where this gain comes from: IW.COMPASS and IW.CE-COMPASS performed almost the same in simulation and separated only on the real robot, so the Cosmos sim-to-real data made the policy robust to the gap between simulation and reality rather than changing its behavior in simulation.

Overall, this work shows that the NVIDIA sim-to-real ecosystem, combining digital twins, X-Mobility, Cosmos and COMPASS, can take a learned navigation policy from simulation to an industrial mobile robot and close most of the gap to a production navigation stack engineered for that robot. With learned navigation established as feasible, the next question is its ROI justification. A tuned traditional stack such as the Idealworks navigation stack remains a strong incumbent on the robots it was engineered for, so the clearest value of learned navigation lies where no such stack exists yet, as when a fleet adopts a new embodiment. A natural next step is therefore a cost and effort study on the HMP, comparing the engineering effort of adapting the Idealworks navigation stack with that of transferring a learned policy across embodiments, to quantify the return on investment at fleet scale. Such a study would build on the HMP Specialist and Generalist of Section 2, which first need to be validated on the physical HMP to confirm that cross-embodiment transfer holds in the real world.

Beyond this study, several steps remain before production use: extending the benchmark to more facilities and robots; meeting industrial safety requirements such as ISO 3691-4 for a camera-based local planner, together with the safety laser scanner; and a hybrid integration alongside the production global planner, started in Appendix C, which would enable long-term deployment and fine-tuning on logged data.

Such an integration would also benefit from a policy that makes its understanding of the scene explicit. COMPASS currently maps camera images directly to velocity commands, without exposing where the obstacles it reacts to are. Extending it with an obstacle-sensing capability, which detects obstacles and localizes them relative to the robot, would allow the global planner to replan around obstacles the policy has perceived, and would make the policy’s decisions more explainable, which supports both debugging and the safety assessment.

Finally, the pipeline of this work, from data generation to deployment, is well suited to agentic workflows. COMPASS already provides agent skills1 to train and evaluate residual policies, onboard a new embodiment and diagnose a broken setup. The steps of this work could be contributed as further skills: generating randomized layouts of a new facility, recording X-Mobility data with the target robot and camera, building the Cosmos-enhanced dataset, fine-tuning X-Mobility with new semantic classes, adding a physics-based controller, and deploying the policy on a Jetson. Chained by an agent, they would bring a new robot or site from its digital twin to an onboard policy with little manual effort, which would also lower the engineering effort that the cost and effort study weighs.

References

1. Fox et al. (1997) Dieter Fox, Wolfram Burgard, and Sebastian Thrun. The dynamic window approach to collision avoidance. IEEE Robotics & Automation Magazine, 4(1):23–33, 1997.
2. Ha and Schmidhuber (2018) David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018.
3. Liu et al. (2024) Wei Liu, Huihua Zhao, Chenran Li, Joydeep Biswas, Billy Okal, Pulkit Goyal, Yan Chang, and Soha Pouya. X-Mobility: End-to-end generalizable navigation via world modeling. arXiv preprint arXiv:2410.17491, 2024.
4. Liu et al. (2025) Wei Liu, Huihua Zhao, Chenran Li, Yuchen Deng, Joydeep Biswas, Soha Pouya, and Yan Chang. COMPASS: Cross-embodiment mobility policy via residual rl and skill synthesis. arXiv preprint arXiv:2502.16372, 2025.
5. Macenski et al. (2020) Steve Macenski, Francisco Martín, Ruffin White, and Jonatan Ginés Clavero. The marathon 2: A navigation system. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020. arXiv:2003.00368.
6. Mittal et al. (2023) Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8(6):3740–3747, 2023.
7. NVIDIA (2025a) NVIDIA. Cosmos world foundation model platform for physical AI. arXiv preprint arXiv:2501.03575, 2025a.
8. NVIDIA (2025b) NVIDIA. MobilityGen: Synthetic data generation for mobile robots in Isaac Sim. https://github.com/NVlabs/MobilityGen, 2025b.
9. Open X-Embodiment Collaboration (2024) Open X-Embodiment Collaboration. Open X-embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024. arXiv:2310.08864.
10. Rösmann et al. (2017) Christoph Rösmann, Frank Hoffmann, and Torsten Bertram. Integrated online trajectory planning and optimization in distinctive topologies. Robotics and Autonomous Systems, 88:142–153, 2017.
11. Shah et al. (2023a) Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hirose, and Sergey Levine. GNM: A general navigation model to drive any robot. In IEEE International Conference on Robotics and Automation (ICRA), 2023a. arXiv:2210.03370.
12. Shah et al. (2023b) Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. ViNT: A foundation model for visual navigation. In Conference on Robot Learning (CoRL), 2023b. arXiv:2306.14846.
13. Sridhar et al. (2024) Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. NoMaD: Goal masked diffusion policies for navigation and exploration. In IEEE International Conference on Robotics and Automation (ICRA), 2024. arXiv:2310.07896.
14. Tobin et al. (2017) Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017. arXiv:1703.06907.
15. Zhao et al. (2020) Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In IEEE Symposium Series on Computational Intelligence (SSCI), 2020. arXiv:2009.13303.

Appendix A Controller Type Study

The released COMPASS stack uses a kinematic controller. To deploy on our embodiments we adapted a physics-based differential drive controller for the iw.hub. This appendix compares the two formulations:

Non-Holonomic Perfect Controller (NHPC).

A kinematic, teleportation-based controller that directly updates the robot pose using the commanded linear and angular velocities, bypassing physical actuation and joint-level dynamics.

Differential Drive Controller (DDC).

A physics-based controller that converts linear and angular velocity commands into wheel-joint velocity targets. Motion is generated through simulated actuator dynamics subject to acceleration and deceleration constraints.

A.1 Evaluation Results

We compare the Differential Drive Controller (DDC) against the Non-Holonomic Perfect Controller (NHPC). Both variants use the same policy architecture, observations, reward formulation, and general training configuration, but differ in how linear and angular velocity commands are executed.

The DDC clamps and ramps the commanded velocities according to acceleration and deceleration limits, then converts them into left and right wheel-velocity targets using differential-drive kinematics. Motion is generated through physics-based wheel actuation. In contrast, the NHPC integrates the commanded velocities and directly teleports the robot root state to the resulting pose, bypassing wheel actuation and joint-level dynamics.

Table 7 compares the evaluation performance of the two controller formulations across all environments. The corresponding training curves and an analysis of their learning behavior are given in Appendix A.2.

The NHPC achieves slightly higher success rates in four environments and matches the DDC in the digital twin of the R&D center. The largest success-rate difference is observed in the Office environment, where the NHPC achieves 67% compared with 58% for the DDC.

However, the DDC produces a lower WTT in four of the five environments, indicating more efficient successful trajectories under the weighted travel time metric. The only exception is the Warehouse Multi-Rack environment, where the NHPC obtains a slightly lower WTT.

Despite the success-rate differences, the DDC is selected for the final COMPASS Specialist because it models wheel actuation, acceleration limits, and robot dynamics more faithfully. This provides a more realistic control interface and reduces the discrepancy between simulated policy commands and the behavior of a physically actuated differential-drive robot.

A.2 Training Results

This section compares the training behavior of the DDC and NHPC variants over 1,000 iterations. Both use the same policy architecture, environment configuration, observation space, and reward formulation. Dotted lines show the signals smoothed with a moving average, and the shaded bands show \pm 1 standard deviation of the raw signal over the same window.

A.2.1 Analysis

NHPC.

The NHPC learns faster across all metrics. Its return starts rising after approximately 150 iterations and reaches about 115, while base-contact terminations drop sharply within the first 150 iterations. Heading and distance rewards converge early, since direct pose updates make corrections take effect immediately. However, its explained variance peaks near 0.76 and then declines to approximately 0.6, suggesting the critic struggles to track the rapidly shifting returns.

DDC.

The DDC converges more gradually, as expected under physics-based actuation and acceleration limits. Its return becomes positive around iteration 500 and plateaus near 76, and its progress rewards dip temporarily before recovering as the policy learns more assertive motion. Despite the slower start, base-contact terminations match the NHPC after iteration 400, and the explained variance stabilizes around 0.8, indicating a more consistent value-function fit.

Overall, the NHPC is easier to optimize, while the DDC learns a stable policy under a realistic control interface. Note that each DDC iteration uses 350 rollout steps versus 256 for the NHPC, so the curves compare iterations rather than equal numbers of environment steps.

Appendix B Omnidirectional Controller Implementation

COMPASS was originally demonstrated on differential-drive and legged embodiments, all of which follow non-holonomic or gait-based motion models. To extend its usability to a broader class of mobile platforms, we deploy the framework on the HMP, an omnidirectional robot capable of independent translation along both planar axes and simultaneous rotation. Supporting this embodiment required a dedicated low-level controller, since none of the existing COMPASS action interfaces can express holonomic motion or actuate steerable drive modules. We therefore developed a physics-based Omnidirectional Swerve Controller (OSC) from scratch. It converts the policy’s body-twist commands into steering-angle and wheel-velocity targets while respecting the platform’s kinematic and dynamic limits. This appendix describes the platform model, the controller formulation, and the design iterations that led to the final version.

B.1 Platform Model

Unlike the iw.hub, the HMP is holonomic. Its chassis has two independently steerable drive modules mounted at diagonally opposite corners, at (0.205,-0.205) m and (-0.205,0.205) m in the base frame, together with four fully passive casters. Each module has a drive joint under velocity control and a steering joint under position control. Table 8 lists the actuator models.

B.2 Command Interface

The OSC operates on a planar body-frame twist \mathbf{u}^{\star}=(v_{x}^{\star},v_{y}^{\star},\omega_{z}^{\star}), comprising longitudinal velocity, lateral velocity, and yaw rate, which together span the full holonomic motion space of the platform. The policy action is retained as a 6-dimensional twist to preserve compatibility with the COMPASS policy head, of which only the three planar components are actuated. To ensure dynamically feasible commands, each component is first saturated to the platform velocity bounds and subsequently slew-rate limited at every physics step \Delta t:

\tilde{v}_{k}=\tilde{v}_{k-1}+\mathrm{clip}\!\left(v^{\star}-\tilde{v}_{k-1},\,-a_{\max}\Delta t,\,a_{\max}\Delta t\right).

B.3 Swerve Inverse Kinematics

For module i at position (r_{x,i},r_{y,i}), the desired ground velocity is

v_{x,i}=\tilde{v}_{x}-\tilde{\omega}_{z}\,r_{y,i},\qquad v_{y,i}=\tilde{v}_{y}+\tilde{\omega}_{z}\,r_{x,i},

which gives a module speed s_{i}=\lVert(v_{x,i},v_{y,i})\rVert and a desired steering angle \theta_{i}^{\star}=\operatorname{atan2}(v_{y,i},v_{x,i})+\theta_{0}, where \theta_{0} is a calibration offset. All angular differences below are wrapped to (-\pi,\pi] by \mathrm{wrap}(\cdot). Let \theta_{i} denote the measured steering angle and r_{w} the wheel radius.

B.4 Design Iterations

The final OSC is the result of three design iterations. Each one addressed failure modes observed in the previous version.

Naive Swerve Control.

The first version applies the IK outputs directly, with \theta_{i}^{\mathrm{cmd}}=\theta_{i}^{\star} and \dot{q}_{i}^{\mathrm{cmd}}=s_{i}/r_{w}. This exposed three problems. First, the module heading is undefined as s_{i}\to 0, and \operatorname{atan2} snaps the wheel back to zero at every stop. Second, velocity reversals require up to 180^{\circ} of steering travel. Third, the wheel is driven at full speed while still misaligned, which produces lateral scrubbing against the ground.

Fully Constrained Control.

The second version adds three safeguards. The last valid heading is held whenever s_{i}<s_{\min}. Each module takes the shortest steering path: with e_{i}=\mathrm{wrap}(\theta_{i}^{\star}-\theta_{i}), if |e_{i}|>\pi/2 the module targets \theta_{i}^{\star}+\pi and reverses its drive sign. Finally, the drive command is gated by steering alignment:

\alpha_{i}=\max\!\left(0,\,\cos\,\mathrm{wrap}(\theta_{i}^{\mathrm{cmd}}-\theta_{i})\right),\qquad\dot{q}_{i}^{\mathrm{cmd}}=\alpha_{i}\,\hat{s}_{i}/r_{w},

where \hat{s}_{i} is the signed module speed. This removes scrubbing but over-constrains the motion. During large steering swings the gated wheels brake against the momentum of the body, and the shortest-path flip chatters when |e_{i}|\approx\pi/2.

Smoothly Constrained Control (Final OSC).

The final controller keeps the safeguards of the constrained version and introduces two relaxations. The first is flip hysteresis: each module keeps a flip state f_{i} that is updated as

f_{i}\leftarrow\begin{cases}|e_{i}|>\pi/2-h,&\text{if }f_{i},\\
|e_{i}|>\pi/2+h,&\text{otherwise},\end{cases}

with h=0.26 rad. This prevents chattering at the flip boundary. The second is roll feed-forward: instead of braking while the module re-orients, each wheel rolls with the body. The measured body twist is propagated through the IK to obtain the measured module velocity (\bar{v}_{x,i},\bar{v}_{y,i}), which is projected onto the current wheel heading:

u_{i}=\bar{v}_{x,i}\cos\theta_{i}+\bar{v}_{y,i}\sin\theta_{i}.

The drive command blends the aligned command with this feed-forward term:

\dot{q}_{i}^{\mathrm{cmd}}=\begin{cases}\hat{s}_{i}/r_{w},&\text{if }s_{i}<s_{\min},\\
\left[\alpha_{i}\,\hat{s}_{i}+(1-\alpha_{i})\,u_{i}\right]/r_{w},&\text{otherwise}.\end{cases}

Commanded stops bypass the blend, so the robot always brakes fully when asked to. In addition, the measured state is read at the start of each control step, so alignment gating does not act on a stale estimate.

B.5 Effect on Navigation Performance

With the OSC, the HMP COMPASS Specialist achieves robust navigation performance across the warehouse environments, reaching success rates of up to 74% in the digital twin of the R&D center. These results are obtained under fully physics-based actuation, with all velocity and acceleration limits of the platform enforced throughout training and evaluation. This demonstrates that COMPASS can be effectively extended to holonomic, steer-actuated embodiments without relying on idealized kinematic control.

Appendix C Global Planner Integration with COMPASS

This appendix describes how we extended the COMPASS stack with a global planner so that it can reach far and occluded goals (scenario S5, Section 3), and the modifications this required in the COMPASS ROS 2 node.

C.1 Motivation

COMPASS is a learned local policy. Its inputs are the camera image, a short route window and the ego speed, and it has no map. It was trained to reach a goal along a route that runs straight from the robot to the goal. The released ROS 2 node follows the same logic: without an external route, its only route is a straight line from the robot to the goal. When the goal is beyond the range of the policy, or hidden behind racks, walls or other occluders, that straight line crosses the obstacle, and the policy has no information about the way around it. Scenario S5 (Section 3) tests exactly this case. In its digital-twin replica, run without a global planner, none of the three COMPASS policies reaches the goal (Appendix H), whatever its training. The global planner supplies the missing route, and the COMPASS ROS 2 node was modified to accept it; since this integration lives in the ROS 2 node, it is used on the real robot only.

C.2 Architecture

A goal pose is requested from the robot behavior tree (BT) through a navigation goal action. The navigation adaptor, a ROS 2 component created for this integration, requests a plan from the global planner, simplifies it, and passes the route and goal to COMPASS. Figure 10 shows the data flow.

Navigation adaptor and global planner.

The adaptor implements the goal action that the BT calls. For each goal it requests a plan from the global planner of the Idealworks navigation stack in the map frame. The plan is simplified (Section C.2.1) and published to COMPASS as the global route, followed by the goal pose. COMPASS returns the velocity command, which the adaptor forwards to the base. Table 10 lists the configuration of the integration.

C.2.1 Path simplification

COMPASS was trained on straight routes from the robot to a goal. A curved polyline in the route window is therefore out of distribution, and the policy does not hold such a path reliably, even in an empty field. The adaptor reduces the plan to a few straight legs instead. The plan is first resampled to 1.0 m spacing, then simplified with the Ramer–Douglas–Peucker (RDP) algorithm at tolerance \varepsilon=0.5 m.

RDP starts with the segment joining the first and last pose and finds the interior pose farthest from it. If that distance exceeds \varepsilon, the pose is kept and the procedure is applied to both halves. Otherwise all interior poses are dropped and the segment becomes one leg. Every dropped pose lies within \varepsilon of the kept polyline, so the legs stay close to the original plan. A larger \varepsilon gives fewer legs but cuts further across the plan. Coincident waypoints are merged. The resulting waypoints are the ones the COMPASS node tracks (Section C.4).

C.3 Integration Configuration

C.4 COMPASS ROS 2 Node Modifications

The released node builds its route only in mapless mode, as a straight line from the robot to the goal. The modified node keeps that path and adds a sparse subgoal mode that follows the global route one waypoint at a time. The global route is received on /compass_model/global_route with reliable, volatile QoS of depth 1. Each message replaces the cached path and resets the subgoal index to 1.

Waypoint tracking.

The node keeps the latest waypoint as the current subgoal. The route window is a straight line from the robot to that waypoint, sampled at 1.0 m, which is the route shape the policy was trained on. The goal heading is the orientation of the waypoint, which points to the next waypoint. The subgoal advances when the robot is within 1.5 m of the current waypoint, or when it has passed the leg end along the leg direction. Advancing before the waypoint is reached keeps the robot from stopping at every waypoint, which gives a smooth trajectory.

Interfaces.

Table 11 lists the interfaces between the node and the adaptor. The navigation type tells the adaptor whether the node drives in local mode (straight to the goal, as in the released node) or in global mode (along the global route).

C.5 Logic Visualization

Figure 11 illustrates the logic on a goal behind a rack. The straight route of the released node crosses the rack. The global plan goes around it but is curved. RDP reduces it to three straight legs. At R_{a} the robot drives straight to W_{1}, with the heading pointing to W_{2}. At R_{b} the robot has entered the switch radius of W_{1}, so the subgoal advances and the route now runs straight to W_{2} before W_{1} is reached.

C.6 Other Modalities and Future Enhancements

Full-trajectory mode.

Before the sparse waypoint mode, the node was set up to follow the curved global plan directly. The plan was resampled into 1.0 m segments, and the global planner was invoked every 2 s (0.5 Hz) to compensate for the drift of the local planner. Following the curved plan naively gave suboptimal paths. The policy did not hold the curved route, so the robot left the global plan and entered restricted zones that the plan was routed around.

Restricted-zone-aware simplification.

The RDP-simplified route resolves this issue, and the results in Section 3 confirm the effectiveness of this implementation. However, the simplification does not guarantee that the straight legs respect restricted zones. Each leg may deviate from the plan by up to \varepsilon, so a leg can cut through a restricted zone at a corner. A future implementation should check every leg against the restricted zones and keep additional waypoints where a leg would cross one.

Replanning on drift.

When the local planner avoids an obstacle, the robot can drift significantly from the originally planned path. In that case the remaining waypoints no longer give the best route from the robot’s current pose. A future implementation should invoke a replan when the robot deviates from the planned path beyond a threshold, so that the route stays optimal and collision free.

Appendix D Training Layouts and Datasets of the Real-World Benchmark Models

This appendix describes the environments and data the models of the real-world benchmark (Section 3) are trained on; Appendix E describes how each training stage uses them. Rather than the single arrangement of the original digital twin, we build 16 layouts of the R&D center, each with its own set of floor objects at a different density and its own lighting (Figure 12). All data and training use the iw.hub embodiment with its own camera setup, not the NVIDIA Carter robot or its camera used for the released models. The same 16 layouts serve both training settings:

• X-Mobility data generation. The random-exploration and path-following datasets used to fine-tune the world model (stage 1) and the IL policy (stage 2) are recorded in these layouts with MobilityGen (NVIDIA, 2025b) (Appendix D.5). The hybrid versions, used for the IW.CE models, append a Cosmos-augmented sim-to-real copy of every episode (Appendix D.5).
• COMPASS training. COMPASS (stage 3) learns online, directly in layout 04, one of the dense layouts. We implemented a custom start and goal sampling: each episode starts at a random collision-free pose, and the goal is sampled around it at a distance between 2 and 8 m (Appendix E.3).

D.1 The Area

All layouts share the building of the R&D center (Table 12). The north side of the building is a racking zone that the robot does not drive in; the open zone south of the racks is where the floor objects are placed and where the robot navigates.

The drivable area is the free floor on which the robot center keeps a clearance of 1.1 m from every obstacle, taken as the largest connected region. The 1.1 m clearance is the planning radius of the iw.hub: the drive axle, which is the robot origin, is 0.40 m behind the front and 1.03 m ahead of the rear, so 1.1 m keeps the rear clear when the robot turns in place. All drivable areas in this appendix are measured in the same way, on a 5 cm occupancy map of the obstacles between 0.05 and 0.4 m high.

D.2 Layout Generation

The layouts are generated procedurally from the digital twin in the following steps:

1. Empty scene. The original floor objects of the R&D center are removed; walls, racks, fixtures, offices, signs and lights are kept. The occupancy map of this empty scene is the base on which every layout is planned.
2. Density tier. Each layout is assigned a density factor equal to a random value in [0.9,1.1] times a tier scale. The tier scale cycles through dense (1.0), near-empty (0.1), medium (0.6) and sparse (0.3), so all densities appear across the 16 layouts and the two test layouts cover both ends (layout 14 sparse, layout 15 dense).
3. Object counts. Base counts are drawn per layout and multiplied by the density factor: 8–12 pallets, 8–12 dollies, 8–12 iw.hubs and 2–4 forklifts (stackers and other forklift models).
4. Placement in groups. Objects are placed group by group at random free positions: a group holds 1, 2 or 3 objects side by side (with probabilities of 45, 35 and 20%), and 60% of the time a neighbor is of another category (e.g. an iw.hub next to a pallet, or a dolly crosswise next to an iw.hub); 70% of the groups are aligned with the building (\pm 4^{\circ}), like real staging areas, and the rest are placed at random angles; pallets are stacked 1–3 high (60, 30 and 10%), and 30% of the pallet stacks carry a box stack on top; groups keep at least 2.3 m from each other, about twice the robot clearance, so the robot can pass between any two groups and obstacles form weaving paths rather than walls; they also keep at least 1.0 m from walls and racks, so objects stand in the open floor; a group is rejected if it cuts more than 1% of the drivable area off from the rest (no blocked aisles or closed corners), and the drivable area never drops below 30% of that of the empty layout; groups are added until every category reaches its count, or until 400 placements in a row fail (the layout is full).
5. Lighting. All lights are scaled by a brightness factor in [0.5,1.3], the color temperature is set between 3500 and 6500 K (warm to cool white, \pm 150 K per light), and 0–30% of the 46 ceiling area lights are switched off, creating dark patches and uneven light.
6. Dolly blockers. An invisible collider is placed over each dolly, so that planners treat a dolly as a solid obstacle and a robot that would fit under it collides instead of driving under it. The collider matches the dolly footprint in the data-generation scenes and is enlarged by 0.3 m in the COMPASS training scenes.
7. Labels. Every placed object is a copy of an original scene object, with the same materials, colliders and semantic class, and each object is mapped to its class (iw.hub, dolly, pallet, forklift, …) for the segmentation labels.

D.3 The 16 Layouts

Table 13 lists the objects, the drivable area and the lighting of each layout. Placed counts are the objects in the scene: each level of a pallet stack counts as one pallet, and box stacks are counted separately. They can exceed the requested counts because groups of up to three objects are added at once. The drivable area is what remains of the 363 m2 of the empty layout once the objects and the robot clearance around them are removed: the dense layouts keep about half of it, the near-empty ones almost all of it.

D.4 Roles of the Density Tiers

Each density tier exposes the models to a different kind of navigation (Table 14).

The mix of tiers is chosen for the following reasons:

• A spread of difficulty rather than one average density: the policy sees both easy open-floor driving, which makes up most of a real warehouse, and the hard cluttered cases, and the world model sees each class both alone and in clutter.
• Test layouts at both ends. Layouts 14 (sparse) and 15 (dense) are never trained on, so generalization is measured on new object arrangements and new lighting, in both an easy and a hard layout.
• Mixed groups and random angles. Objects stand next to other classes and at varied angles, so the model cannot learn that iw.hubs always come in rows or rely on fixed positions.
• Gaps of at least 2.3 m and no cut-off areas. Every layout stays fully navigable, and obstacles force weaving paths rather than dead ends.
• Lighting independent of density. Dim, bright, warm, cool and partly switched-off lighting occurs in every tier, so appearance is not tied to clutter.

D.5 The Dataset

The X-Mobility datasets are recorded in the 16 layouts with MobilityGen and consist of two sets (Table 15):

• Path following (stage 2, IL policy): in each episode, MobilityGen’s path-following expert drives the robot to one random goal after another until a time limit of 60 s or a collision, with 15 episodes per layout recorded at 4 Hz. The expert plans a shortest path to each goal on the occupancy map of the scene, with obstacles inflated by the 1.1 m robot clearance, and follows it with a look-ahead heading controller at a constant 1.0 m/s, turning in place for heading errors above 45^{\circ}. This set is named nav2 after the X-Mobility dataset convention, but no ROS 2 Nav2 stack is used. Each goal segment is stored as a separate file, about 3.9 per episode.
• Random exploration (stage 1, world model): in each episode the robot drives randomly until a collision, and episodes are kept if they last at least 5 s. They are recorded at 5 Hz and stored in chunks of 50 frames.

The test split consists of layouts 14 and 15, which are never trained on; the train and validation splits are a random split by episode over layouts 00–13. All episodes are driven by the iw.hub embodiment, and all images are rendered from the front camera of the real iw.hub, with its calibrated mounting pose and intrinsics, at a resolution of 640\times 360; no data comes from the Carter robot or the Carter camera of the NVIDIA datasets. In the semantic labels, the floor under a dolly or within 0.3 m of its footprint is labeled as not drivable. The random-exploration data of layouts 03 and 11 is smaller than that of the other layouts because two of their recorders hung and were stopped; overall, 96% of the planned random-exploration data was recorded.

Sim-to-real version.

For the IW.CE models we also create a sim-to-real version of the dataset. Cosmos Transfer 2.5 augments every simulated episode into a photorealistic episode, frame-aligned with the original so that its actions, states and labels remain valid; the augmentation parameters and samples are given in Appendix F. The hybrid dataset is built by appending all the sim-to-real episodes to the simulated ones, which doubles the size of the dataset and gives a 50%/50% split between simulated and sim-to-real episodes. Each sim-to-real episode keeps the train, validation or test split of its source episode, so layouts 14 and 15 remain unseen in both halves.

Table 16 summarizes the datasets used at each training stage, including the hybrid versions used for the IW.CE models.

D.6 Semantic Labels for the iw.hub and Dolly Classes

The iw.hub and dolly classes are added without changing the architecture of X-Mobility, by giving two of its seven existing semantic class slots a new meaning. The NVIDIA model predicts background, navigable, forklift, pallet, cone, sign and fence. Since the R&D center contains no traffic cones or fences, slot 4 (cone) is reassigned to the iw.hub and slot 6 (fence) to the dolly, both in the label definitions of the model and in the training labels, where iw.hub and dolly pixels are mapped to classes 4 and 6. The evaluations of Appendix E.1 therefore report iw.hub and dolly in place of cone and fence.

Because the pretrained weights of these two slots had learned to recognize cones and fences, only the parts of the network specific to them are re-initialized. In each of the three semantic segmentation heads (full, half and quarter resolution), the final 1\times 1 output convolution has one row of weights and one bias per class; rows 4 and 6 are reset to the default PyTorch initialization of a new convolution layer. Every other parameter keeps the values of the NVIDIA checkpoint: the DINOv2 image encoder, the recurrent state model of the world model, the shared decoder layers, the output rows of the other five classes and the action policy. In total, 352 of the 358 weight tensors of the model are identical to the NVIDIA checkpoint.

Starting from this checkpoint, the whole model is fine-tuned with a low learning rate (1\times 10^{-5}, one-cycle schedule) in two steps (Appendix E):

1. the world model is fine-tuned on the random-exploration data with the new labels, so that the re-initialized outputs learn the iw.hub and dolly classes from features the encoder already provides;
2. the NVIDIA action policy is merged back in, since world-model checkpoints contain no policy, and the world model and policy are fine-tuned together on the path-following data.

Starting from the NVIDIA weights, changing only the two reassigned outputs and fine-tuning with a low learning rate preserves the existing classes and the driving behavior on our data. Fine-tuning still specializes the model to the R&D center: on NVIDIA’s own test set, the accuracy on the other classes, such as pallet and forklift, drops noticeably. This is a limitation of the current models, and mixing part of NVIDIA’s data back into training is a way to mitigate it.

Appendix E Training Pipeline of the Real-World Benchmark Models

This appendix documents how the IW.COMPASS and IW.CE-COMPASS models evaluated in Section 3 were obtained. A COMPASS model is the result of three training stages, each producing the model the next stage builds on:

1. World model (WM). The X-Mobility world model is trained on random-exploration data.
2. Imitation-learning (IL) policy. With the world model, the X-Mobility policy is trained by imitation learning on path-following expert demonstrations.
3. COMPASS. The IL policy is frozen and used as the base policy for residual reinforcement learning on the iw.hub embodiment.

For both variants, the world model and the IL policy start from the NVIDIA base checkpoints and are fine-tuned exclusively on data generated in the digital twin of the Idealworks R&D center, while the COMPASS residual policy is always trained from scratch in the digital twin on top of the resulting IL policy (Appendix E.3). Following the naming convention of Section 3, every model trained this way carries the prefix IW., and IW.CE marks the Cosmos-enhanced ones (e.g. IW.WM and IW.CE-WM for the world models). The datasets used at stages 1 and 2 are described in Appendix D. Each variant is a chain of its own models:

• IW.COMPASS: IW.WM \rightarrow IW.IL \rightarrow residual RL;
• IW.CE-COMPASS: IW.CE-WM \rightarrow IW.CE-IL \rightarrow residual RL.

IW.CE-COMPASS differs from IW.COMPASS only in its fine-tuning data: its world model and IL policy are fine-tuned on datasets that combine the original episodes with Cosmos-augmented sim-to-real variants, and its residual policy is trained on top of the Cosmos-enhanced IL policy. At the world model and IL stages we also evaluate the NVIDIA base model, without fine-tuning, on the same evaluation data as a baseline.

E.1 Stage 1: World Model

The world model is fine-tuned by continuing the training of the released NVIDIA world model on the random-exploration data of the digital twin (Table 18). The training recipe is the released one; the only change is the setup of the iw.hub and dolly labels described below. The checkpoint used in the evaluations is the one with the lowest validation loss.

The released X-Mobility world model has no iw.hub or dolly class, two objects common in the R&D center; the evaluations below therefore report them in place of two reassigned classes of the released model, as described in Appendix D.6.

Imagination evaluation.

The world model predicts future scenes from its latent state, without new observations, and the predicted semantic segmentation is compared with the ground truth at increasing time horizons (Table 20). The world model first observes five frames and then imagines the following frames from the recorded actions alone; the table reports the semantic IoU at the end of each horizon.

E.2 Stage 2: Imitation-Learning Policy

The IL policy is fine-tuned from the NVIDIA checkpoint on the path-following data, on top of its own stage-1 world model (IW.WM for IW.IL, IW.CE-WM for IW.CE-IL). The world model is not frozen: it continues to train together with the policy (Table 21).

Open-loop evaluation.

The policy predicts actions and paths for the recorded episodes of the validation split of the path-following dataset (23 episodes, 5,501 frames; Appendix D.5), and its predictions are compared with the expert’s (Table 22). We report the mean absolute errors of the linear speed, the angular speed and the predicted path (linear, angular and path MAE), as defined in X-Mobility (Liu et al., 2024), together with the weighted action and path losses of the X-Mobility training objective.

Closed-loop evaluation.

The policy drives the iw.hub in the digital twin of the R&D center (Table 23). The policies are evaluated closed-loop in layouts 14 and 15, the test layouts of the dataset (Appendix D.5), which are never seen in training and cover a sparse and a dense layout. An episode counts as a success when the robot reaches the goal within 0.5 m with a linear velocity below 1.0 m/s and an angular velocity below 1.0 rad/s. These velocity thresholds are deliberately loose: the path-following expert drives at a constant 1.0 m/s and never brakes at the goal (Appendix D.5), so the IL policy, which imitates it, also reaches the goal at full speed. A stopping criterion such as the one used for COMPASS (Appendix E.3) would count these arrivals as failures; stopping at the goal is learned only by the COMPASS residual policy. Each policy is evaluated over 300 rollouts per layout, with start and goal poses sampled uniformly across the drivable area of the layout.

Both fine-tuned policies far outperform the NVIDIA base policy, which reaches the goal in only 2.3% of the rollouts, against 69.3% for IW.IL and 66.7% for IW.CE-IL over both layouts (Table 23). IW.IL and IW.CE-IL perform almost the same on every metric, and this is expected: the test layouts are rendered in simulation, where only the simulated half of the IW.CE-IL training data matches the test images. That the two stay this close is a good sign: adding the Cosmos sim-to-real episodes does not cost any performance in simulation, and their benefit is expected on the real robot (Section 3).

E.3 Stage 3: COMPASS Residual Reinforcement Learning

The residual policy is trained from scratch for each variant, on top of its frozen IL policy (IW.IL for IW.COMPASS, IW.CE-IL for IW.CE-COMPASS). A COMPASS residual learns corrections to the actions of the specific base policy it was trained with, so a residual trained on one X-Mobility base, such as the NVIDIA COMPASS residual on the NVIDIA base policy, does not transfer to a fine-tuned base, and fine-tuning it on top of IW.IL or IW.CE-IL is not a viable starting point.

Compared with the released COMPASS training setup, we made the following additions:

• iw.hub embodiment asset with its sensor configuration, including its real front camera.
• Differential drive controller (Appendix A).
• Layout 04 of the digital twin of the R&D center (Appendix D) as training environment.
• Custom start and goal sampling (below).

Training scenario.

Both residual policies are trained online in a single layout, layout 04 of Appendix D: a dense layout with 36 objects, in which about 28% of the training routes are blocked by an obstacle across the width of the robot. We implemented a custom logic that sets the start and goal of each training episode:

1. Start pose. At each reset, the robot is placed at a uniform random pose inside the sampling box of the scene (x 3–46 m, y 3.5–10.5 m) with any yaw. The pose is accepted when the iw.hub body, grown by 0.25 m, is free on the occupancy map, with up to 10 tries.
2. Goal pose. The goal is then sampled around the start position in polar coordinates: a distance d\sim\mathcal{U}(2\,\text{m},8\,\text{m}) and a direction \theta\sim\mathcal{U}(-\pi,\pi), giving the goal position \mathbf{p}_{\text{start}}+d\,(\cos\theta,\sin\theta), with a goal heading drawn uniformly from its range.

Because the distance is uniform, short and long goals are equally frequent; per unit of area, however, goals are denser near the robot than they would be with uniform-area sampling. The base policy is given a straight-line route to the goal. Each training iteration runs 450 steps (45 s), after which every robot is reset.

Rewards and success criterion.

The rewards are listed in Table 24. A goal is reached when the robot is within 0.5 m of it, below 0.1 m/s and 0.1 rad/s, with its heading within 0.1 rad of the goal heading. PPO uses an exploration noise of 0.1 on all residual outputs, an entropy bonus of 0.01 and an adaptive learning rate. The X-Mobility base policy is frozen; only the residual policy and the critic are trained.

Training curves.

The two runs use the same configuration, and the only difference between them is the X-Mobility base policy the residual is trained on top of. Since both use 32 robots per update, their curves are directly comparable (Figure 13); both are shown up to iteration 440. The figure reports the share of finished episodes that reach the goal and that end in a collision, the mean duration of successful episodes, the number of episodes finished per 45 s iteration over all 32 robots (higher means faster episodes), the mean training reward per step, and the mean value the residual adds to the forward-speed command of the base (positive means faster than the base). A finished episode ends in a goal or a collision; episodes cut off at the end of an iteration count as neither.

Table 26 compares the two runs at the same checkpoints; each value is the mean of the 10 iterations before the checkpoint.

The two trainings differ in four ways:

1. IW.CE-COMPASS starts much slower. For the first {\sim}100 iterations its residual reaches almost no goals (6% against 34% at iteration 100): the IW.CE-IL base needs more correction on the iw.hub at first. It catches up between iterations 150 and 250.
2. IW.CE-COMPASS ends slightly ahead on success and safety. At iteration 440 it reaches the goal in 79% of the finished episodes with 21% collisions, against 76% and 24% for IW.COMPASS. The difference of 2–3 points is about the size of the iteration-to-iteration noise, so it needs to be confirmed in an evaluation.
3. IW.COMPASS is faster. At iteration 440 it reaches the goal in 10.2 s against 10.6 s and finishes more episodes per iteration (137 against 125).
4. Both have plateaued in success since roughly iteration 300–400. Late in training both residuals add more and more forward speed, from about 0 to +0.15 m/s: the gains go into arriving sooner rather than into fewer collisions.

These are training metrics, measured with exploration noise; the deployed, noise-free policy usually scores somewhat higher. The fair comparison is a noise-free evaluation on the same start and goal pairs, ideally on the test layouts 14 and 15, which are not used in training. Moreover, both residuals are trained and measured on a single layout, so these curves do not show how well either generalizes to other scenes.

Appendix F Sim-to-Real Augmentation Samples

This appendix shows samples of the sim-to-real augmentation used for the IW.CE models (Section 3, Appendix E). The Cosmos-enhanced datasets are generated with Cosmos Transfer 2.5, part of the NVIDIA Cosmos world foundation models (NVIDIA, 2025a), which re-renders a simulated episode as a photorealistic video while preserving its scene geometry and motion. Each episode’s camera frames, recorded at 4 or 5 Hz, are encoded into a 10 fps video with one video frame per recorded frame,passed through Cosmos Transfer 2.5, and the generated frames are written back into the episode, frame-aligned with the original, so the actions, states and labels of the episode remain valid.

All episodes are generated with the same configuration (Table 27). The generation is conditioned on three controls (Figure 14). The semantic segmentation rendered by the simulator, with weight 1.0, fixes the layout and the objects of the scene. An edge map and a blurred copy of the simulated frames, both computed from the input video, with weights 0.4 and 0.1, keep the geometry and the coarse appearance close to the original while leaving Cosmos free to re-render textures and lighting. No depth control is used. Each episode is output at its input resolution and frame rate, 640\times 360 at 10 fps, so that every generated frame replaces exactly one simulated frame.

Figures 15 and 16 show 28 samples of simulated episodes and their Cosmos-generated counterparts.

Appendix G Onboard Deployment on the Jetson AGX Xavier

This appendix describes how the COMPASS models of Section 3 were deployed on the iw.hub’s onboard computer (Section 3): the target hardware and the conversion from the trained model to a TensorRT engine.

G.1 Target Hardware and Software

G.2 Model Conversion

Appendix H Digital-Twin Replica of the Real-World Scenarios

This appendix reports the full results of the COMPASS policies in the digital-twin replica of the static scenarios S1, S2, S3 and S5 (Section 3): a one-to-one copy in the digital twin of the R&D center, with the same layouts, obstacles, start poses and goals as in the real world and the metrics of Section 3. The dynamic scenario S4 is not replicated, as moving obstacles in the digital twin are not explored in this work, so no interaction metrics are reported, and protective stops are not measured in simulation. The Idealworks navigation stack is evaluated only in the real world. Unlike the simulation benchmark of Section 2, which evaluates policies trained in NVIDIA’s environments across those environments and the digital twin, these runs reproduce the real-world benchmark of Section 3. Table 6 compares the success rate and weighted travel time of each policy with its real-world runs. S5 is run without the global planner, which is implemented only in the COMPASS ROS 2 node of the real robot: no policy reaches the goal, which shows that the COMPASS policy alone cannot reach a far, occluded goal (Section 3).