LOCUS, 공간 그래프로 동일 클래스 수납함을 구분해 숨은 물건 탐색

Abstract

As robots are increasingly deployed in unstructured, real-world environments, the ability to reason about complex spatial and semantic relationships among objects remains a fundamental challenge in enabling robust and generalizable manipulation and navigation. For example, deciding where to search for an object that is not in plain sight depends on where it is physically plausible as well as semantically likely. Popular methods such as cosine similarity with CLIP struggle to disambiguate between spatially and semantically similar objects that could contain a target object. We propose Landmark-Oriented Container Discrimination Using Spatial Graphs (LOCUS). To jointly reason about spatial information and semantics, we train a GNN to update CLIP embeddings on a full environment scene graph by passing embedding information between nodes based on proximity. CLIP embeddings, augmented by semantic knowledge from a fusion of household ontologies provide a robust semantic signal. We evaluate in simulation on a noiseless scene graph from simulator metadata, making the approach detection-agnostic. Our approach outperforms random, CLIP, Tidybot, and an LLM planner in the majority of room classes and configurations in simulation. To show our approach is robust to scene graph node placement and label noise, we demonstrate a physical mobile manipulator running in an exploration pipeline start-to-finish including scene graph labels from Detic.

I Introduction

To accomplish many complex tasks in the real world, robots must understand physical and semantic relationships between objects in their environment. For example, to find occluded objects in household or search-and-rescue tasks, a robot needs both general and domain-specific knowledge of object relationships to hypothesize where an object may be hidden, including inside other containers. The robot may also be provided with a general floor plan which it can use to inform its search, but have no knowledge about the items within each room. In many cases, there are multiple instances of the same container object which could contain the target object, such as multiple drawers that could contain a fork, and the robot must find a way to discriminate between the containers to generate a container search order. We posit that the correct approach requires both the generality of Large Language Models (LLMs) or Vision Language Models (VLMs) and the structured knowledge of an ontology.

We introduce a novel approach, Landmark-Oriented Container discrimination Using Spatial Graphs (LOCUS), the first reasoning pipeline for concealed objects that makes instance-level decisions between containers of the same class. Locus is Latin for place — reflecting our hypothesis that where a container is matters as much as what it looks like. LOCUS combines domain specific formal ontologies and general semantic knowledge. It accounts for cases not explicitly covered in the ontologies with the spatial relationship between objects in the environment in order to find an occluded object and is generalizable sim-to-sim and sim-to-real. We pretrain a GNN over scene graphs and use ontologies to learn typical spatial relationships and augment each scene graph node’s CLIP [1] embedding.

Our contributions include:

1. a formal ontology for sorting objects into containers, landmarks, or dual role; 2. a dataset with ontology-informed placement strategy for occluded objects in containers (iTHOR Object-Container Spatial Relationship Dataset for LOCUS); 3. a scene graph builder that uses these ontologies; and 4. a room-agnostic GNN architecture to be trained on explored environment scene graph that accurately passes semantic information from the environment to update CLIP embeddings. We will publish our codebase upon acceptance (LOCUS Codebase).

II Background and Problem Formulation

To reason over object search and retrieval solutions, many approaches [2, 3, 4] use CLIP for retrieval, comparing search locations by cosine similarity with the query; however, same-class containers receive nearly identical embeddings, whether from a bounding-box crop or a wider frame containing context (e.g., a drawer with the sink visible). Methods exist for semantic object search [5, 6, 7, 8, 9, 10, 11] and container reasoning [12, 13, 14], but none produce a principled pipeline to search for an occluded object in a realistic environment with multiple containers of the same class to choose between during exploration given a floor plan and an ontology for the domain. Scene-graph GNNs. rank navigation goals in observable space [15, 16]; HOLM [17] and VLMaps [18] resolve only viewpoint occlusion, where the target becomes visible once the agent looks in the right direction; semantic mechanical search [19] reasons over occupancy rather than which of several identical containers to open.

Semantic World Knowledge for Object Search Decisions Many approaches investigate commonsense reasoning while exploring for not completely occluded objects, with LLMs as a popular solution. The approaches that reason over room-object navigation decisions for unknown environments either combine preexisting pretrained models for image parsing and an LLM to reason over objects in the image to inform the search [5, 6], use LLM chain-of-thought prompting for commonsense context [7], or build multi-layer VLM value maps from LLM-inferred semantic priors [8, 9]. An approach to incorporating environment understanding into the reasoning pipeline is using a scene graph of the current environment to aid in decision making. For partially known environments with a floor plan, [10] and [11] use a scene graph to inform their neural network based approaches. [10] use LLMs to generate training data for their network which can lead to hallucinations and [11] manually craft the semantic categories governing where objects are placed, which makes the vocabulary sparse and unable to accurately reflect or scale to real-world distributions. These approaches lack principled semantic grounding through ontologies or human placement data, which would impact performance for occluded object search.

Container Reasoning Approaches for object-container reasoning don’t consider instances of containers but rather consider object to container class relationship (e.g. a fork goes in a drawer, and all drawers are treated as the same since they belong to the same class) [12, 13, 14]. Approaches that consider finding entirely occluded objects rely on simulation metadata at runtime and abstract away manipulation task choices, as if the agent is within range of a fully occluded object, then it is found [12, 13], while the approach that considers sensor data for labels using d-DETR [20] rather than the ground truth labels from simulation metadata at run time only considers fully visible objects [14]. TidyBot [21] likewise selects a receptacle type from LLM preferences, so it cannot order same-class instances. To our knowledge, no prior work discriminates same-class container instances under full occlusion. Notably, all of these approaches are in simulation and do not generalize cross-simulation. There is a gap for real world robotics deployment when the robot needs to choose between containers, some of which are of the same class, to search to physically retrieve an occluded object. Therefore, our goal is to answer, Given a room level floor plan of an environment and task/environment relevant ontologies, can a robot distinguish between instances of the same object class with different semantic and spatial information and generalize to unseen environment configurations and domains?

Our work focuses on bridging the gap between semantic world knowledge for object search decisions and instance container reasoning for realistic object retrieval. CLIP provides strong semantic understanding. Rather than training an entirely new network, we update CLIP with spatial context for instance-level container discrimination via a GNN that learns over an environment’s scene graph.

To showcase LOCUS’s ability to provide detailed distinctions between nodes of the same class, at run time we score nodes by cosine similarity between the updated container embedding and the query text, retaining CLIP’s comparability while adding spatial and semantic reasoning.

III Technical Approach

III-A Graph Construction and Ontology

LOCUS represents a scene as a heterogeneous scene graph \mathcal{G}=(\mathcal{V},\mathcal{E}) where each detected object in the environment is a node v\in\mathcal{V} and edges e\in\mathcal{E} encode spatial proximity between objects. To construct the graph, as shown in Fig. 1, every object must be assigned a node role that determines how it participates in reasoning. Prior scene-graph methods assign such roles through hierarchical graph layers [22, 23, 24] or task-specific heuristics [25]. These categorizations do not generalize: they provide no criterion for classifying a newly encountered object. We instead derive node roles from formal ontologies (see Sec. IV-B).

We define three node roles. Containers are receptacles that can be opened (e.g. drawers, cabinets, refrigerators) and are the objects the robot must rank when searching for a hidden target. Landmarks are large, stationary objects that cannot be picked up or opened (e.g., countertops, stove burners, sinks) and serve as spatial reference points. Dual-role nodes are containers that also function as landmarks: a refrigerator is both a searchable receptacle and a spatial anchor that contextualizes nearby drawers and cabinets. In addition to these object nodes, a single room node is added to represent the room class (e.g., kitchen, bathroom) and provide room-level semantic context.

Two types of edges connect nodes in the graph. Landmark-to-container edges are directed from each landmark to nearby containers, weighted by w=\exp(-d) where d is the Euclidean distance, so closer landmarks exert stronger influence during message passing. Room-to-container edges are bidirectional and connect the room node to every container with unit weight, allowing room-class context to flow to containers and aggregated container information to flow back to the room node.

III-B Graph Neural Network

We require a mechanism to propagate spatial and semantic context between scene graph nodes so that containers of the same class, with nearly identical CLIP embeddings, can be distinguished by their neighborhood. We use a graph neural network (GNN) to update each node’s representation by aggregating information from connected landmarks and the room node, as shown in Fig. 2.

III-B1 Semantic Information

To distinguish between nodes of the same class, a node’s representation must capture not only what the object looks like but also its location relative to the room and other object nodes. Each node’s feature vector contains visual appearance, semantic identity, spatial context, and relative position.

Each container node concatenates a CLIP visual embedding of the container’s bounding box crop, aggregated across all viewpoints by element-wise maximum to retain the most distinctive features from any angle (maxpooling); a CLIP text embedding of the container’s class label; a full-frame CLIP embedding from the viewpoint where the container occupies the largest pixel area, capturing surrounding scene context; and a height encoding that bins vertical position into floor, counter, and wall tiers (a 4-d feature: normalized height plus the three-tier one-hot). A soft landmark-distance vector is appended, encoding proximity to each of the landmark types observed during training as \exp(-d) for the nearest instance of each type. At deployment, hard class assignments are replaced by soft assignments: each detection’s CLIP embedding is compared against per-class prototype embeddings learned from training, and a softmax produces a weight vector over landmark types. Landmark nodes use the same input layout as well. Room nodes concatenate the CLIP text embedding of the room class with the mean text embeddings of member containers and landmarks. All container and landmark features are mapped to a common embedding space through a shared residual encoder. Room nodes use a separate linear projection.

III-B2 Message Passing

After encoding, each node’s representation reflects only its own features. Message passing allows containers to absorb context from their spatial neighbors: a drawer near a sink should be represented differently from a drawer near a stove, even if both look identical.

We use APPNP [26] to propagate information across the graph. Edges run from landmarks to containers with weight \exp(-d), and from rooms to containers with unit weight. After propagation, container embeddings are projected through a linear head, L2-normalized, and scored against query text embeddings by cosine similarity.

III-B3 Loss

The primary objective combines two contrastive losses. The first loss is a multi-positive InfoNCE [27] that, for each query, contrasts every container holding the target object against all containers and landmarks in the scene:

\mathcal{L}_{\text{InfoNCE}}=-\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\log\frac{\exp(s_{i}/\tau)}{\sum_{j\in\mathcal{C}\cup\mathcal{L}}\exp(s_{j}/\tau)}

where s_{k}=\cos(\mathbf{z}_{k},\mathbf{q}) is the cosine similarity between node k’s post-GNN embedding and the query text embedding, with the temperature \tau applied only inside the exponent, \mathcal{P} is the set of positive containers, \mathcal{C} all containers, and \mathcal{L} all landmarks in the scene. Landmarks are unlabeled: they appear in the denominator but never receive positive or negative labels. Each query is weighted by its inverse positive rate across the training set.

The second loss is a soft ranking loss that penalizes the expected number of containers a robot would open before finding the target:

\mathcal{L}_{\text{search}}=\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\log\!\Bigl(1+\sum_{j\in\mathcal{C}\cup\mathcal{L}}\sigma(s_{j}-s_{i})\Bigr)

where \sigma is the sigmoid function at unit temperature. This loss directly targets our primary evaluation metric Expected Containers Opened (ECO) (see Sec. IV-D.)

Two auxiliary losses regularize the encoder during training . A binary cross-entropy head, \mathcal{L}_{\text{contain}}, on stop-gradient embeddings predicts \hat{y}=\sigma(f(\mathbf{z})), the probability that node \mathbf{z} is a container. A learned scalar \beta boosts container scores by +\beta\,\hat{y} and suppresses landmarks by +\beta\,(\hat{y}-1). A second auxiliary head,\mathcal{L}_{\text{teacher}}, matches each container’s representation to a soft K-means cluster assignment derived from open-view CLIP embeddings, preventing the encoder from collapsing visual distinctions that closed-door appearance obscures.

The total loss is as follows, where \lambda_{sf},\lambda_{c}, and \lambda_{t} are constants used to weight the respective losses:

\mathcal{L}=\mathcal{L}_{\text{InfoNCE}}+\lambda_{\text{sf}}\,\mathcal{L}_{\text{search}}+\lambda_{\text{c}}\,\mathcal{L}_{\text{contain}}+\lambda_{\text{t}}\,\mathcal{L}_{\text{teacher}}

IV Experiment

IV-A Task

We task the robot to search for an item given its text label (e.g., fork). The item may be visible (found through mapping to create the scene graph) or entirely occluded within a container object, in which case the robot must decide which containers to search and in what order.

IV-B Ontology

The ontology is used only for dataset creation. We define three node roles using simulator metadata and formal knowledge bases in Sec. III-A. Containers are objects flagged as both openable and receptacle in AI2-THOR [28]. Landmarks are non-pickupable, non-openable objects classified under landmark-relevant upper classes in at least one of three ontological backends: SUMO [29] (Furniture, MajorAppliance, ElectricDevice, CoolingDevice, etc.), SOMA [30] (DesignedFurniture, Appliance, DesignedComponent), or WordNet (furnishing.n.02, appliance.n.02, fixture.n.01). Countertops and stove burners qualify as landmarks while garbage bags do not.

IV-C Metrics

We report four metrics. Instance Rank Accuracy (IRA) measures within-class discrimination: among all containers of the same class, is the top-scored instance the ground-truth positive? IRA is computed only over discriminative queries—those where at least one same-class container holds the target and at least one does not. Expected Containers Opened (ECO) is the rank of the first positive container in descending score order—the number of containers a robot would open before finding the target. ECO/N normalizes ECO by the number of containers N in each scene, giving the fraction of containers opened before finding the target (0 = first try, 1 = every container). This enables direct comparison across environments with different container counts (e.g., iTHOR {\sim}10 vs. RoboCasa {\sim}34). S-IRA (Semantic IRA) relaxes IRA so that any co-placement equivalent container of the same class counts as correct, accounting for functional interchangeability (e.g., two drawers that both hold forks). S-ECO/N (Semantic ECO/N) applies the same relaxation to ECO/N. We report both because their gap is itself the measurement: the ECO/N vs S-ECO/N gap quantifies the errors that select a functionally equivalent container. IRA{=}1.0 is unattainable under partial information, as identical drawers cannot be told apart without priors; therefore ECO is our primary metric, and it also covers more of the test set. The 42 query classes yield 201 (scene, query) episodes, of which ECO uses all 201 and IRA only the 165 discriminative ones (hence dashes in Table III).

IV-D Baselines

Random selects containers uniformly (5-seed average). Visual CLIP [1] ranks containers by cosine similarity between the query text embedding and the full-frame ViT-B/32 image embedding of the frame in which each container was most visible. Class Text CLIP ranks by cosine similarity between the query text and the CLIP text embedding of each container’s class name. Priors rank by P(q{\mid}\text{room}){\times}P(q{\mid}\text{type}), optionally scaled by a landmark-distance term f(\text{dist}){=}e^{-d/\sigma}. TidyBot [21] is ported into our evaluation pipeline. The commonsense (zero-shot) and summarization (informed with our training placements) variants select a receptacle type, per their design, and instances within type are randomly selected (chance IRA, so this upper bound covers ConceptGraphs too [31]). LLM scene graph gives an LLM (Claude Haiku 4.5) each test scene’s full serialized graph and requests a ranking over container instances zero-shot. LLM raw detections is the ontology-free and graph-free counterpart, so the same LLM model ranks instances from the raw detection list. MLP over CLIP with full spatial features, is the toggled K{=}0.

V Results

V-A iTHOR Simulation

V-A1 Setup

Training and evaluation use iTHOR to isolate the GNN from perception noise. The scene graph comes directly from simulator metadata, so there is no exploration phase in simulation or sim-to-sim transfer.

Dataset We demonstrate LOCUS in the household domain, due to the multiple household ontologies [29, 30]. We collect data from all 120 iTHOR [28] scenes spanning four room classes (30 kitchens, 30 living rooms, 30 bedrooms, 30 bathrooms), split into 80 train / 20 validation / 20 test scenes stratified by room class. For each scene an exhaustive grid sweep captures RGB frames at every reachable position, four yaw rotations, and three camera tilts, producing dense multi-view coverage.

AI2-THOR’s default scenes have mostly empty containers, so we define a semantic placement policy that distributes 42 query-object classes into openable receptacles. The class-level mappings follow the same conventions validated by HouseKeep’s [13] crowdsourced annotations (e.g. cutlery in Drawer, Cabinet in kitchens; perishables in Fridge or Cabinet). Objects that serve the same purpose, such as fork or spoon, are placed together into the same container when capacity allows, and containers near a semantically related landmark are preferred (e.g. pots near a stove burner). A soft capacity limit of five objects per container prevents unrealistic crowding. Three random seeds produce independent placements per scene; ground-truth labels are aggregated across seeds, yielding 99.6% cross-seed agreement. Our placement rules additionally create spatial variation at the instance level; two drawers in the same kitchen may hold different objects depending on their proximity to landmarks, which is the discrimination signal LOCUS is designed to learn.

Training We train on the 80-scene training split above for 500 epochs with learning rate 10^{-4}, weight decay 10^{-4}, batch size 32, and \tau\!=\!0.07. The encoder is a 3-layer residual MLP with skip connections and LayerNorm. APPNP [26] uses K\!=\!2 hops and teleport \alpha\!=\!0.3. We evaluate on a held-out 20-scene test split (5 per room class) and average all LOCUS results over 30 training seeds. For training we assume the robot has perfect sensor readings of the environment, so it has correct object class labels and poses during the training process.

We do not train with perception noise or Detic labels: the GNN consumes only the scene graph from simulator metadata, and Detic labels are noisy pseudo-labels. The zero-shot Stretch 3 run with real perception error provides that evidence instead(See Section V-C).

V-A2 Results

Table I reports all metrics on the 201-episode iTHOR test split. TidyBot, ConceptGraphs [31], and the P(q{\mid}\text{room}){\times}P(q{\mid}\text{type}) prior score containers by class so the first container opened is a random pick among all the containers sharing that class. This means the upper bound on IRA for all these methods is uniform random chance, regardless of how accurate the class prediction is. Adding f(\text{dist}) to the prior only raises IRA because its distance term matches the rule we used to place objects, so that gain is circular, and it still loses every search-cost metric. Both CLIP baselines perform at or below random on IRA, unable to distinguish same-class instances. Given raw detections alone the LLM names the correct class at nearly LOCUS’s rate, yet its IRA is close to random, which supports our claim of instance discrimination being the bottleneck. ECO on the other hand does not have that upper bound, yet we outperform the baselines. TidyBot, which names the correct class 92.0% of the time yet opens .338N containers per search, against LOCUS which names the correct class 95.2% and opens.257N per search, a 24.0% improvement. A stronger LLM model baseline would likely not raise the ECO score, since the missing information is about instances, not classes. The LLM can reason about instances as it is given the full scene graph, so it is the strongest baseline on ECO search cost but trails LOCUS on every metric.

The K\!=\!0 ablation, an MLP over the same features with no message passing, already reaches an IRA 40% higher than the chance upper bound for methods ranking containers by class alone, showing that the spatial features alone are enough to reason about instances rather than classes. APPNP propagation (K\!=\!2) increases IRA by another 7% confirming that inter-node context provides a modest but consistent benefit across all 30 training seeds (p{<}.001, paired t). S-ECO/N counts functionally equivalent containers as successes, and doing so lowers search cost by 17% (ECO 2.83 to S-ECO 2.17 containers), so much of what ECO determines as error is LOCUS opening a container that would also have been a reasonable place to look.

Tables II and III break down results by room class and query object. Kitchens are hardest, yet LOCUS achieves ECO/N 63% better than random and S-ECO/N less than ECO/N, meaning over a third of kitchen errors are semantically equivalent. Living rooms are worst (ECO/N\!=\!0.471) because fewer landmarks result in less spatial signal to exploit. Per-query, LOCUS dominates utensils and tableware where spatial context resolves which drawer to open, while fridge-stored foods are trivial for any method with container-class priors (ECO\!=\!1.0). IRA .367 means the correct instance is ranked first {\sim}37\% of the time; therefore, LOCUS reduces search cost but it does not reliably identify the target on the first opening.

Failures LOCUS errors are less costly. Errors are right-type wrong-instance or wrong-type. Low IRA does not limit usefulness as 90% of LOCUS errors are right-type wrong-instance, so adding a few openings rather than failing the search, while {\sim}50\% of baseline errors are costly wrong-type (derived from Table IV; per-room breakdown in Table V).

Compute The average iTHOR scene graph has 10 containers, 11 landmarks, and 122 edges. The fully-connected graph matches the best tuned cutoff and handles RoboCasa’s 20–40 containers.

TidyBot requires an LLM call per query at run time, with occasional unparseable output that our port handles fairly by tying remaining candidates. LOCUS is 3.18M parameters at {<}3 ms on CPU, offline and deterministic.

V-A3 Circularity and Determinism

To rule out the possibility of circularity, in which LOCUS would learn the label generator rather than spatial conventions, we tested relabeled the test split to strip out both of our generator’s contributions: the landmark-proximity rule that chose which instance, replaced by a uniform draw, and our type conventions, replaced by HouseKeep [13] crowd votes (Table VI). A model that had memorized the generator would lose accuracy in proportion, but zero-shot on these labels, LOCUS’s search cost, ECO/N, is essentially unchanged. HouseKeep’s votes place 89% of episodes in a cabinet, so the margin to our comparators narrows and a query-independent ranker that opens cabinets first is best under HouseKeep labels and near-worst under ours.

Determinism: 99.6% measures the generator’s cross-seed reproducibility. The near-equal RoboCasa performance on novel and trained queries (Table IX) suggest that LOCUS is learning neither the script’s rules nor specific object–container pairings.

V-A4 Ablations

We retrained all feature and loss ablations on iTHOR (Table VII). Removing the 4-d height-tier block from containers and landmarks (-height) raises ECO, suggesting that the height features contribute to the reduction in search cost seen in Table I. Message passing improves ECO by {\sim}0.28, with or without soft-ldist, as the graph carries landmark context, but results indicate the strongest performance comes from having both features. Uniform and e^{-d} edge weights perform similarly, so finer geometry is unlikely to improve performance.

V-B Sim-to-Sim Transfer: RoboCasa

V-B1 Setup

Few simulators support placing objects inside closed containers, which limits dataset scale; to zero-shot test cross-domain generalization without real-world perception noise, we evaluate the iTHOR-trained model on RoboCasa [32] kitchen scenes rendered in MuJoCo [33]. RoboCasa draws on a different object asset library and distribution of floor plans than iTHOR, yet accepts the same placement policy, so the environment shifts while the labeling convention is held fixed. Performance indicates that the learned spatial conventions transfer across simulators. We evaluate 10 layouts \times 3 styles \times 5 placement seeds, giving 1,188 episodes. RoboCasa scenes hold 20-40 openable containers–roughly 5 \times iTHOR–making instance discrimination much harder.

V-B2 Results

Table VIII evaluates LOCUS zero-shot: trained only on iTHOR, tested in RoboCasa kitchens with no finetuning. Despite the fivefold larger candidate set, LOCUS continues to beat the baselines, suggesting that the learned spatial priors transfer across kitchen simulators.

Table IX splits results by query novelty and shows LOCUS generalizes to unseen object categories without finetuning. LOCUS maintains consistent performance across both splits, demonstrating that the graph-based spatial reasoning LOCUS learns generalizes beyond the training vocabulary. Additional in distribution (ID) and out-of-distribution (OOD) analysis per-query determined LOCUS achieves the best IRA on 14 of 17 Robocasa queries. Strong performance on utensils (fork, tongs, ladle) and food (shrimp, mustard) confirms that spatial conventions (e.g., utensils near countertops, perishables in the fridge) transfer across kitchen simulators.

V-C Mobile Manipulator Hardware

V-C1 Setup

To show that our GNN, trained in simulation, works with a domain shift, we implemented our approach on a Hello Robot Stretch 3 mobile manipulator and tested it in a real kitchen. We used a deliberately cluttered environment, thereby adding extraneous detections as additional scene-graph nodes to test real-world robustness. We use Detic [34] for open-vocabulary object detection and extrapolate 3D poses from bounding boxes and the associated depth image. We deduplicate overlapping detections by confidence, keeping the highest-scoring box, and use the surviving poses and labels to generate nodes. Nodes whose labels match a container or handle class in our ontology are labeled openable; all other nodes serve as spatial context. The GNN ingests and ranks the full scene graph, and the robot searches the ranked openable nodes. Fig. 3 shows the end-to-end pipeline.

We average 3 runs per query–method combination from a fixed start, panning the camera to map the kitchen, for the queries fork, mug, and sponge. Fig. 4 shows the test kitchen and a completed scene graph.

Building the scene graph from real detections introduces perception noise absent from simulation (See Fig 4): estimated rather than ground-truth poses, sensor noise in depth readings, and reduced frame rates from ROS message throttling over rosbridge, which limits the number of frames available to the labeler and can degrade detection quality at oblique viewing angles.

V-C2 Results

Table X shows that on average LOCUS performs competitively. For the query sponge, Detic mislabels the cabinet that contains the sponge as a drawer. This impacts both LOCUS and Text CLIP, but our spatial reasoning yields “smarter mistakes” under real-world mislabeling.

VI Conclusion

To our knowledge, LOCUS is the first approach enabling a robot to choose among containers, including same-class instances, to retrieve an occluded object. Our results show our GNN updated CLIP embeddings transfer across simulators and to hardware. Limitations LOCUS ranks a fully built scene graph rather than a partial one, so it requires an exploration pre-step. Queries with weak spatial signal (e.g., pots and pans, whose placement varies across kitchens) stay near-random, and landmark-sparse rooms like living rooms offer little signal for message passing. No public dataset provides instance-level object-in-container ground truth across diverse room classes; prior rearrangement work [13] gives only class-level receptacle preferences. We therefore built our own from AI2-THOR using a hand-crafted placement policy that encodes commonsense conventions but not the full variability of real homes.

References

1. [1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning Transferable Visual Models From Natural Language Supervision. arXiv. Note: arXiv:2103.00020 [cs] External Links: Link, Document
2. [2] D. Maggio, Y. Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone (2024) Clio: Real-Time Task-Driven Open-Set 3D Scene Graphs. IEEE Robotics and Automation Letters 9 (10), pp. 8921–8928. External Links: ISSN 2377-3766, Link, Document
3. [3] N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher (2024) Vlfm: vision-language frontier maps for zero-shot semantic navigation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 42–48.
4. [4] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song (2023) CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23171–23181. External Links: ISSN 2575-7075, Link, Document
5. [5] K. Zhou, K. Zheng, C. Pryor, Y. Shen, H. Jin, L. Getoor, and X. E. Wang (2023) ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation. In Proceedings of the 40th International Conference on Machine Learning, pp. 42829–42842 (en). External Links: ISSN 2640-3498, Link
6. [6] D. Shah, M. R. Equi, B. Osiński, F. Xia, B. Ichter, and S. Levine (2023) Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning. In Proceedings of The 7th Conference on Robot Learning, pp. 2683–2699 (en). External Links: ISSN 2640-3498, Link
7. [7] V. S. Dorbala, J. F. Mullen, and D. Manocha (2024) Can an Embodied Agent Find Your ”Cat-shaped Mug”? LLM-Guided Exploration for Zero-Shot Object Navigation. IEEE Robotics and Automation Letters 9 (5), pp. 4083–4090 (en). Note: arXiv:2303.03480 [cs] External Links: ISSN 2377-3766, 2377-3774, Link, Document
8. [8] Q. Wang, Y. Xu, V. Kamat, and C. Menassa (2025) OVAMOS: A Framework for Open-Vocabulary Multi-Object Search in Unknown Environments. arXiv (en). Note: arXiv:2503.02106 [cs] External Links: Link, Document
9. [9] Q. Zhao, J. Han, M. Xu, and F. Zeng (2025) Integrating Semantic Priors in Vision-Language Frontier Maps for Zero-Shot Object Navigation. In 2025 40th Youth Academic Annual Conference of Chinese Association of Automation (YAC), pp. 3003–3007. External Links: ISSN 2837-8601, Link, Document
10. [10] M. F. Ginting, S. Kim, D. D. Fan, M. Palieri, M. J. Kochenderfer, and A. Agha-Mohammadi (2024) SEEK: Semantic Reasoning for Object Goal Navigation in Real World Inspection Tasks. arXiv. Note: arXiv:2405.09822 [cs] External Links: Link, Document
11. [11] A. Kurenkov, R. Martín-Martín, J. Ichnowski, K. Goldberg, and S. Savarese (2021) Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search. arXiv. Note: arXiv:2012.04060 [cs] External Links: Link, Document
12. [12] K. Mirakhor, S. Ghosh, D. Das, and B. Bhowmick (2024) Task Planning for Object Rearrangement in Multi-room Environments. arXiv. Note: arXiv:2406.00451 [cs] version: 1 External Links: Link, Document
13. [13] Y. Kant, A. Ramachandran, S. Yenamandra, I. Gilitschenski, D. Batra, A. Szot, and H. Agrawal (2022) HouseKeep: tidying virtual households using commonsense reasoning. In European Conference on Computer Vision (ECCV), pp. 355–373.
14. [14] G. Sarch, Z. Fang, A. W. Harley, P. Schydlo, M. J. Tarr, S. Gupta, and K. Fragkiadaki (2022) TIDEE: Tidying Up Novel Rooms Using Visuo-Semantic Commonsense Priors. In Computer Vision – ECCV 2022, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner (Eds.), Vol. 13699, pp. 480–496 (en). Note: Series Title: Lecture Notes in Computer Science External Links: ISBN 978-3-031-19841-0 978-3-031-19842-7, Link, Document
15. [15] D. Shah, B. Osiński, B. Ichter, and S. Levine (2022) LM-nav: robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning (CoRL), pp. 492–504.
16. [16] M. Lingelbach, C. Li, M. Hwang, A. Kurenkov, A. Li, R. Martín-Martín, L. Fei-Fei, J. Wu, and S. Savarese (2023) Task-driven graph attention for hierarchical relational object navigation. In IEEE International Conference on Robotics and Automation (ICRA), pp. 886–893.
17. [17] V. Cirik, L. Morency, and T. Berg-Kirkpatrick (2022) HOLM: hallucinating objects with language models for referring expression recognition in partially-observed scenes. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL),
18. [18] C. Huang, O. Mees, A. Zeng, and W. Burgard (2023) Visual Language Maps for Robot Navigation. arXiv (en). Note: arXiv:2210.05714 [cs] External Links: Link, Document
19. [19] S. Sharma, H. Huang, K. Shivakumar, L. Y. Chen, R. Hoque, B. Ichter, and K. Goldberg (2023) Semantic mechanical search with large vision and language models. External Links: 2302.12915, Link
20. [20] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2021) Deformable {DETR}: deformable transformers for end-to-end object detection. In International Conference on Learning Representations, External Links: Link
21. [21] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser (2023) TidyBot: personalized robot assistance with large language models. Autonomous Robots.
22. [22] A. Werby, C. Huang, M. Büchner, A. Valada, and W. Burgard (2024) Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation. In Robotics: Science and Systems XX, (en). External Links: ISBN 979-8-9902848-0-7, Link, Document
23. [23] A. Kurenkov, M. Lingelbach, T. Agarwal, E. Jin, C. Li, R. Zhang, L. Fei-Fei, J. Wu, S. Savarese, and R. Martın-Martın (2023) Modeling dynamic environments with scene graph memory. In International Conference on Machine Learning, pp. 17976–17993.
24. [24] I. Mahdi, M. Cassinelli, F. Despinoy, T. Welschehold, and A. Valada (2026) Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search. arXiv (en). Note: Version Number: 1 External Links: Link, Document
25. [25] Z. Zeng, A. Röfer, and O. C. Jenkins (2020) Semantic Linking Maps for Active Visual Object Search. arXiv (en). Note: arXiv:2006.10807 [cs] External Links: Link, Document
26. [26] J. Gasteiger, A. Bojchevski, and S. Günnemann (2019) Predict then propagate: graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR),
27. [27] A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
28. [28] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. Gupta, and A. Farhadi (2022) AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv (en). Note: arXiv:1712.05474 [cs] External Links: Link, Document
29. [29] I. Niles and A. Pease (2001) Towards a standard upper ontology. In Proceedings of the 2nd International Conference on Formal Ontology in Information Systems (FOIS), pp. 2–9. External Links: Document
30. [30] D. Beßler, R. Porzel, M. Pomarlan, A. Vyas, S. Höffner, M. Beetz, R. Malaka, and J. A. Bateman (2021) Foundations of the socio-physical model of activities (SOMA) for autonomous robotic agents. In Formal Ontology in Information Systems (FOIS), Frontiers in Artificial Intelligence and Applications, pp. 159–174.
31. [31] Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, C. Gan, C. M. De Melo, J. B. Tenenbaum, A. Torralba, F. Shkurti, and L. Paull (2024) ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, pp. 5021–5028 (en). External Links: ISBN 979-8-3503-8457-4, Link, Document
32. [32] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu (2024) RoboCasa: large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS),
33. [33] E. Todorov, T. Erez, and Y. Tassa (2012) MuJoCo: a physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026–5033.
34. [34] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra (2022) Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision (ECCV),