ObjDream: Guiding Vision-Language-Action Models with Object-Level Understanding and Future State Prediction

Abstract

Vision-language-action (VLA) models have emerged as a powerful paradigm for robotic manipulation, enabling robots to generate actions from visual observations and language instructions. However, most existing VLA models still operate on image-level visual representations inherited from pretrained vision-language backbones, while lacking fine-grained object understanding for manipulation. As a result, their robustness can degrade under out-of-distribution object layouts, clutter, and background variations. To address these limitations, we introduce ObjDream, an object-centric VLA framework centered on object-level understanding and future object state prediction. ObjDream uses a spatio-temporal object encoder to organize visual observations into temporal object slots, where persistent object identity and dynamic object state are represented through separate channels. These structured object representations are inserted into the token sequence of the vision-language model (VLM), where identity-guided routing modulates VLM attention to the corresponding object tokens. In addition, an action-conditioned object dynamics module predicts compact future object states relevant to manipulation during training, explicitly supervising object state evolution under robot actions without requiring future prediction at inference. Experiments on LIBERO, LIBERO-Plus, and RoboTwin 2.0 show that ObjDream improves generalization under diverse distribution shifts and scene perturbations. Real-world robot experiments further demonstrate that ObjDream transfers to physical manipulation settings with novel object arrangements and background changes. Project page.

1 Introduction

Robust robot policies must reliably manipulate objects in out-of-distribution environments, where their appearance, position, and surrounding context may differ from those in the training data. VLA models have made rapid progress in generalist robot control by scaling pretrained vision-language backbones, robot data, and action decoders (Brohan et al., 2023; Kim et al., 2024; Black et al., 2024; NVIDIA et al., 2025; Wang et al., 2026b). To improve generalization, recent work has sought to preserve knowledge acquired during pretraining (Guo et al., 2026; Hancock et al., 2026), use high-level plans to guide low-level control (Li et al., 2025c), or anticipate scene changes through future visual or latent-state prediction (Zhang et al., 2025; Sun et al., 2026). Although these approaches improve generalization to some extent, most VLA models still lack fine-grained object-level understanding. Their visual representations, inherited from VLMs, remain largely image-level, leaving the identity and spatial state of task-relevant objects entangled with appearance and surrounding context. This entanglement can weaken out-of-distribution robustness under shifts in object appearance, spatial layout, and background context.

Object-centric approaches offer a natural way to strengthen object-level understanding by explicitly representing individual objects and their properties. Prior work incorporates object-aware visual representations to help policies focus on task-relevant regions (Zhu et al., 2023; Li et al., 2025a; Liang et al., 2026). Other methods extend object representations to relational reasoning (Hanyu et al., 2025), object-level memory (Chung et al., 2026), and long-horizon manipulation (Yang et al., 2026). Although these approaches demonstrate the value of object-level representations, they often encode persistent object identity and time-varying object state in the same slot or token. As appearance and geometry change, this coupling can obscure persistent identity cues by mixing them with dynamic object state. Recent dual-state slot models show that separating identity from dynamic appearance can improve temporal consistency in video perception by keeping identity information distinct from time-varying visual content (Tran et al., 2026). Building on this insight, we investigate whether persistent identity references can be represented alongside dynamic object states and further used to guide VLM attention for language-conditioned manipulation.

Beyond modeling the current visual observation, predictive models support manipulation by anticipating future observations or object states (Ferraro et al., 2025; Chen et al., 2025a; Villar-Corrales and Behnke, 2025). The choice of prediction target is important: reconstructing complete images requires modeling texture and background details that may be irrelevant to control (Zhang et al., 2025), while goal-image methods such as Goal-VLA introduce an additional synthesis stage before estimating the target object pose (Chen et al., 2025a). In contrast, object-centric world models show that scene evolution can be modeled using compact object representations (Jeong et al., 2025). Motivated by this trade-off, we use action-conditioned prediction of compact future object states, including geometry and visibility, as auxiliary supervision for VLA training without requiring future object state prediction at inference.

To this end, as illustrated in Fig. 1, we introduce ObjDream, an object-centric vision-language-action framework for improving manipulation generalization. The architecture of ObjDream builds on a pretrained VLM backbone and integrates a spatio-temporal object encoder, an object-level forward dynamics module, and a flow-matching action expert. Specifically, the spatio-temporal object encoder transforms visual observations into structured object slots, where persistent object identity and dynamic object state are represented through separate channels. These factorized object representations are mapped into the VLM hidden space and inserted into the token sequence, where identity-guided routing modulates attention to the corresponding object tokens. The forward dynamics module predicts future object states during training, providing auxiliary supervision for future object state evolution under robot actions without requiring future prediction at inference. The object encoder is pretrained with automatically generated offline object supervision and frozen during VLA training. We optimize ObjDream with a joint objective combining action imitation with current and future object state supervision. Extensive evaluations on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world robot experiments demonstrate competitive manipulation performance across diverse scene perturbations.

The main contributions of this paper are listed below:

• We present ObjDream, an object-centric VLA framework centered on object-level understanding and future object state prediction to improve generalization under perturbations. ObjDream introduces a spatio-temporal object encoder that explicitly represents persistent object identity and dynamic object state through separate channels. The resulting representations are integrated into the VLM hidden space with a novel identity-guided routing mechanism for robust language-conditioned manipulation.
• We introduce an action-conditioned object-level forward dynamics module for modeling object evolution during manipulation. It predicts compact future object states during training to provide direct auxiliary object-level supervision without requiring future object state prediction at inference. We further model inter-object relations from object features and pairwise geometry to provide structured context for action generation.
• Extensive evaluations on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world robot experiments demonstrate competitive manipulation performance across diverse scene and environmental perturbations. Ablations further assess the effects of the key architectural components.

2 Related Work

Vision-language-action models.

VLA models combine pretrained language or vision-language backbones with action decoders to learn general-purpose manipulation from large-scale robot demonstrations (Brohan et al., 2022; Brohan et al., 2023; O’Neill et al., 2024; Team et al., 2024; Kim et al., 2024; Black et al., 2024; Physical Intelligence et al., 2025). Recent work improves VLA performance through stronger action decoders and fine-tuning strategies (Chi et al., 2025; Zhao et al., 2023; Kim et al., 2025; Hou et al., 2025; NVIDIA et al., 2025; Qu et al., 2025), larger-scale cross-embodiment training (Cai et al., 2026; Team et al., 2026; Wu et al., 2026a; Wu et al., 2026b; Wang et al., 2026b), and better preservation or exploitation of pretrained visual-semantic and spatial priors (Guo et al., 2026; Li et al., 2026a; Wang et al., 2026c). Predictive objectives further use future images, video tokens, subgoal images, or latent states to provide temporal supervision for manipulation (Zhao et al., 2025; Zhang et al., 2025; Cen et al., 2025b; Cen et al., 2025a; Li et al., 2026b). Despite these advances, existing VLA models still largely operate on dense image-level visual representations, where object-level structure is not explicitly organized for manipulation generalization. ObjDream addresses this limitation by enhancing VLA policies with object-level understanding and future object state supervision.

World models for robot control.

World models learn action-conditioned environment dynamics and support robot control by planning or learning policies from predicted futures in observation or latent spaces (Hafner et al., 2019; Wu et al., 2023). Recent work extends this paradigm through language-conditioned video generation (Du et al., 2023; Zhou et al., 2024b), pretrained visual latent dynamics (Zhou et al., 2024a; Assran et al., 2025), and world-action or unified video-action models that couple future prediction with executable actions (Ye et al., 2026; Li et al., 2026c). These approaches provide dynamics supervision by modeling anticipated scene evolution. However, prediction in pixel or video space requires modeling high-dimensional visual details and can introduce training or inference overhead. Recent approaches therefore explore more efficient formulations, including avoiding explicit future imagination at inference or exposing compact representations of scene changes (Yuan et al., 2026; Bai et al., 2026). ObjDream follows this direction by using compact future object state prediction as supervision during training, while directly generating actions at inference without explicit future object state decoding.

Object-centric representations for manipulation.

Object-centric representation learning decomposes visual scenes into entity-level latent variables and has been widely studied in scene decomposition, temporal tracking, and structured dynamics (Locatello et al., 2020; Kipf et al., 2021; Wu et al., 2022). In robotic manipulation, object-level representations have been used to improve sample efficiency and enhance visuomotor attention and robust policy conditioning through object proposals, grounded geometry, object-centric prompting, and physically grounded object concepts (Zhu et al., 2023; Vo et al., 2025; Li et al., 2025b; Wei et al., 2026), enable efficient VLA learning through compact object-centric tokenization (Bendikas et al., 2025), and support structured control through object-centric dynamics and relational representations (Ferraro et al., 2025; Mosbach et al., 2024; Hanyu et al., 2025; Yang et al., 2026). Complementary approaches use structured affordance representations to provide action-relevant object grounding, interaction localization, and geometric guidance for VLA policies (Yu et al., 2026b). Recent predictive approaches further incorporate object-aware future representations, masks, or addressable slot states into world-action learning (Wang et al., 2026a; Yu et al., 2026a; Liu et al., 2026). Recent dual-state slot models separate per-frame appearance from persistent object identity, highlighting the value of disentangling identity from changing visual content in object-centric representations (Tran et al., 2026). From this perspective, ObjDream combines persistent identity references with dynamic object states and uses identity-guided routing to provide object-specific context for language-conditioned manipulation.

3 Method

Following the VLA policy formulation in Sec. 3.1, Fig. 2 illustrates the architecture of ObjDream. ObjDream combines a spatio-temporal object encoder, an object dynamics module, and a flow-matching action expert. The encoder produces structured object slots that are mapped to VLM object tokens, while identity-guided routing uses cached identity references to modulate attention to these tokens (Sec. 3.2). The object dynamics module provides future object state supervision during training (Sec. 3.3), and the action expert generates action chunks.

3.1 VLA Policy Formulation

We formulate ObjDream as an object-centric VLA policy that predicts continuous action chunks from language instructions, visual observations, proprioceptive states, and object-level embeddings. At control step t, the policy receives a language instruction x, a current multi-view observation I_{t}, and proprioceptive state s_{t}. The VLM consumes all policy views in I_{t}, while the object encoder operates on a designated external or third-person view with causal history \mathbf{I}_{t}=(I_{t-T+1}^{\mathrm{enc}},\ldots,I_{t}^{\mathrm{enc}}). At episode initialization, a phrase parser extracts a task-relevant phrase set P_{0} from the instruction x and initial observation I_{0}. A spatio-temporal object encoder E_{\omega} then produces structured object slots O_{t}=E_{\omega}(\mathbf{I}_{t},P_{0}). The VLA backbone f_{\psi} encodes these inputs into a multimodal context C_{t}, and the action expert parameterizes the conditional distribution over an action chunk A_{t}=[a_{t},\ldots,a_{t+H_{a}-1}]:

C_{t}=f_{\psi}(x,I_{t},s_{t},O_{t}),\qquad p_{\theta}(A_{t}\mid x,I_{t},s_{t},O_{t})=p_{\theta}(A_{t}\mid C_{t}).

Here, H_{a} denotes the action horizon. In our implementation, p_{\theta} is instantiated by a flow-matching action expert (Lipman et al., 2022; Black et al., 2024). Given a demonstrated action chunk A_{t}^{\star}, Gaussian noise Z_{t}\sim\mathcal{N}(0,I), and flow time \tau\sim\mathcal{U}[0,1], the action loss is

A_{t}^{\tau}=(1-\tau)Z_{t}+\tau A_{t}^{\star},\qquad\mathcal{L}_{\mathrm{act}}=\mathbb{E}\!\left[\left\|v_{\theta}(A_{t}^{\tau},\tau;C_{t})-(A_{t}^{\star}-Z_{t})\right\|_{2}^{2}\right].

At inference, the learned velocity field is integrated to generate an action chunk, which is executed in a receding-horizon manner.

3.2 Object-Centric Tokenization and VLM Routing

Spatio-temporal object encoder.

ObjDream uses a spatio-temporal object encoder E_{\omega} to construct factorized object slots from the causal image window \mathbf{I}_{t} and the cached phrase set P_{0}. At episode initialization, Qwen3-VL-8B (Bai et al., 2025) processes the instruction x and initial observation I_{0} once to extract task-relevant object phrases, which are cached for all subsequent control steps. Query ordering is fixed across frames, with reserved robot queries and padding to N_{s} slots. For each phrase p_{i}, a frozen CLIP text encoder (Radford et al., 2021) produces f_{i}, and a learnable projection gives \iota_{i}=\operatorname{Normalize}(P_{\mathrm{sem}}(f_{i})). The object query is initialized as

q_{i}^{(0)}=P_{\mathrm{id}}(\iota_{i})+P_{\mathrm{text}}(f_{i})+E_{\mathrm{type}}(\chi_{i}),

where P_{\mathrm{id}},P_{\mathrm{text}} are learnable projections and E_{\mathrm{type}} embeds the query type \chi_{i}. A frozen DINOv2 encoder (Oquab et al., 2023) extracts a visual reference d_{i} from the initial object region predicted by the encoder, which is cached throughout the episode. Robot queries use learned identities and type priors without the direct text term.

A spatial query module applies self-attention and cross-attention to learnably projected features from a frozen SAM2 image backbone (Ravi et al., 2025), followed by causal temporal encoding. The encoder outputs

O_{t}=E_{\omega}(\mathbf{I}_{t},P_{0})=\left\{u_{i,t}^{\mathrm{obj}}\right\}_{i=1}^{N_{s}},\qquad u_{i,t}^{\mathrm{obj}}=(e_{i}^{\mathrm{id}},e_{i,t}^{\mathrm{state}},\nu_{i,t}).

The persistent identity e_{i}^{\mathrm{id}}=[\iota_{i};d_{i};r_{i}] combines semantic and visual references with a role embedding r_{i}, while e_{i,t}^{\mathrm{state}}=[c_{i,t}^{\mathrm{app}};g_{i,t}^{\mathrm{geo}};\tau_{t}] encodes dynamic appearance, geometry, and temporal information. Slot validity \nu_{i,t} masks inactive or padded slots and is distinct from predicted visibility. During pretraining, regional features obtained by mask pooling from the projected SAM2 features, extracted before query cross-attention, are aligned with frozen CLIP text features. The objective is described in Sec. 3.4.

Factorized object injection.

We map identity and dynamic state components separately into the VLM hidden space:

h_{i,t}^{\mathrm{obj}}=\left[P_{\mathrm{id}}^{\mathrm{inj}}(e_{i}^{\mathrm{id}});P_{\mathrm{obj}}\left([P_{\mathrm{state}}(e_{i,t}^{\mathrm{state}});\phi_{\nu}(\nu_{i,t})]\right)\right].

Here, P_{\mathrm{id}}^{\mathrm{inj}}, P_{\mathrm{state}}, and P_{\mathrm{obj}} are lightweight projections, and \phi_{\nu} embeds slot validity. The factorization is imposed at the encoder output and initial object-token construction; subsequent VLM contextualization is unrestricted, while the cached identity reference e_{i}^{\mathrm{id}} remains separate for routing. Attention keys and values are computed from the full contextualized token.

Relation and prospective object tokens.

Beyond individual object tokens, ObjDream introduces a relation token and N_{p}=2 prospective object tokens:

h_{t}^{\mathrm{rel}}=\Phi_{\mathrm{rel}}\left([\mathrm{Pool}_{i}(h_{i,t}^{\mathrm{obj}});\mathrm{Pool}_{i,j}(\Delta g_{ij,t})]\right),\qquad h_{k,t}^{\mathrm{pro}}=\Phi_{\mathrm{pro}}\left([h_{i_{k},t}^{\mathrm{obj}};h_{t}^{\mathrm{rel}}]\right).

Here, \Phi_{\mathrm{rel}} aggregates object features and pairwise geometry into a compact global relational context, while \Phi_{\mathrm{pro}} combines this context with individual object features. \mathrm{Pool} denotes aggregation over valid slots or valid slot pairs, and \Delta g_{ij,t} encodes the relative geometry between slots i and j. The indices i_{k} correspond to the top-N_{p} valid slots ranked by a learned slot importance gate. These prospective tokens are optimized through the action objective to provide object-specific context for action generation. Together, the tokens form the object block

X_{t}^{\mathrm{obj}}=[h_{1,t}^{\mathrm{obj}},\ldots,h_{N_{s},t}^{\mathrm{obj}},h_{t}^{\mathrm{rel}},h_{1,t}^{\mathrm{pro}},\ldots,h_{N_{p},t}^{\mathrm{pro}}].

The block is inserted between the multimodal prefix and the action-generation prompt. The attention mask retains causal attention and additionally permits bidirectional attention within the object block, excluding invalid and padded slots as keys.

Identity-guided object routing.

For the base object token associated with slot i at VLM position j_{i}, query position p, layer \ell, and attention head m, let \bar{h}_{p}^{\ell} denote the normalized query hidden state. Using the cached identity reference e_{i}^{\mathrm{id}}, we compute an identity matching score b_{pi}^{\ell} and a head-specific gate \eta_{p}^{\ell,m}:

b_{pi}^{\ell}=\cos\!\left(U^{\ell}\bar{h}_{p}^{\ell},A^{\ell}e_{i}^{\mathrm{id}}\right),\qquad\eta_{p}^{\ell,m}=\eta_{\max}\sigma\!\left((w_{g}^{\ell,m})^{\top}\bar{h}_{p}^{\ell}+c_{g}^{\ell,m}\right).

Here, U^{\ell} and A^{\ell} are learnable projections shared across heads, and \cos(\cdot,\cdot) denotes cosine similarity between their \ell_{2}-normalized outputs. The parameters w_{g}^{\ell,m} and c_{g}^{\ell,m} define the head-specific gate \eta_{p}^{\ell,m}\in(0,\eta_{\max}). For each valid base slot (\nu_{i,t}=1), the backbone attention logit is augmented as

s_{pj_{i}}^{\ell,m}=\frac{(q_{p}^{\ell,m})^{\top}k_{j_{i}}^{\ell,m}}{\sqrt{d_{k}}}+\eta_{p}^{\ell,m}b_{pi}^{\ell}+M_{pj_{i}},

where q_{p}^{\ell,m} and k_{j_{i}}^{\ell,m} are the standard backbone query and key, d_{k} is the key dimension, and M is the attention mask. The cached identity affects only the additive bias; standard keys and values are computed from the full contextualized tokens. All other logits remain unchanged and are normalized by the same softmax.

3.3 Object-Level Forward Dynamics Module

Action-conditioned object forward dynamics.

Let o_{i,t} denote the final VLM hidden state of the object token associated with slot i. An auxiliary head predicts an H_{a}-step action chunk \tilde{A}_{t} from mean-pooled, stop-gradient features of valid slots, without using the action expert’s output. We construct a position-aware action embedding e_{t}^{a} and action conditioning gates \beta_{i,t}:

e_{t}^{a}=\frac{1}{H_{a}}\sum_{h=1}^{H_{a}}P_{a}([\mathrm{sg}(\tilde{A}_{t,h});p_{h}]),\qquad\beta_{i,t}=\sigma(P_{\beta}(\mathrm{sg}(o_{i,t}))).

Here, p_{h} is a learned temporal embedding, P_{a} and P_{\beta} are trainable networks, \sigma is sigmoid, and \mathrm{sg} denotes stop-gradient. The action conditioning gate \beta_{i,t} is distinct from the slot importance gate. During warmup, demonstrated actions A_{t}^{\star} replace \tilde{A}_{t} in the conditioning path. For valid slots, current states are decoded directly, while future prediction is restricted to non-robot object slots:

\hat{Y}_{i,t}^{\mathrm{cur}}=D_{\mathrm{cur}}(o_{i,t}),\qquad\hat{Y}_{i,t+1:t+H_{f}}^{\mathrm{fut}}=D_{\mathrm{fut}}(o_{i,t}+\gamma\beta_{i,t}e_{t}^{a}).

Here, D_{\mathrm{cur}} decodes current states, \gamma is a fixed residual scale, and D_{\mathrm{fut}} predicts all H_{f} future horizons in parallel, producing horizon-indexed geometry, visibility, and mask logits; robot slots are excluded since their motion is represented by the action chunk. The auxiliary branch is skipped at inference, when the policy directly generates action chunks.

3.4 Training Objectives

Object encoder pretraining.

We pretrain E_{\omega} using structured object-level supervision:

\mathcal{L}_{\mathrm{enc}}=\lambda_{\mathrm{mask}}\mathcal{L}_{\mathrm{mask}}+\lambda_{\mathrm{geo}}\mathcal{L}_{\mathrm{geo}}+\lambda_{\mathrm{vis}}\mathcal{L}_{\mathrm{vis}}+\lambda_{\mathrm{rt}}\mathcal{L}_{\mathrm{rt}}.

Here, \mathcal{L}_{\mathrm{mask}} combines binary cross-entropy and Dice losses for object segmentation. \mathcal{L}_{\mathrm{geo}} supervises object geometry, including boxes, centers, areas, mask–geometry consistency, and auxiliary slot attributes. \mathcal{L}_{\mathrm{vis}} supervises visibility labels derived from offline annotations; visibility is distinct from slot validity. \mathcal{L}_{\mathrm{rt}} is a bidirectional, multi-positive region–text contrastive loss between projected regional visual features and frozen CLIP text features.

VLA training.

During VLA training, we keep the pretrained object encoder E_{\omega} frozen and optimize the remaining trainable components with action imitation and auxiliary losses:

\mathcal{L}=w_{\mathrm{act}}\mathcal{L}_{\mathrm{act}}+\mathcal{L}_{\mathrm{aux}},\qquad\mathcal{L}_{\mathrm{aux}}=\mathcal{L}_{\mathrm{cur}}+\mathcal{L}_{\mathrm{fut}}+\mathcal{L}_{\mathrm{prop}}+\mathcal{L}_{\mathrm{gate}}.

Here, \mathcal{L}_{\mathrm{act}} is the flow-matching action loss, which also optimizes the prospective object tokens through the action objective. \mathcal{L}_{\mathrm{cur}} supervises current object geometry, visibility, and mask logits, while \mathcal{L}_{\mathrm{fut}} supervises future object states predicted by the forward dynamics module. \mathcal{L}_{\mathrm{prop}} trains the auxiliary coarse action proposal against the demonstrated action chunk, while \mathcal{L}_{\mathrm{gate}} regularizes the learned slot-importance gates used for object-slot prioritization. The routing projections and gates are optimized through the existing objectives without an additional routing loss. \mathcal{L}_{\mathrm{fut}} is computed only for valid non-robot object slots, excluding robot, invalid, and padded slots.

4 Experiments

Benchmarks.

We evaluate ObjDream on LIBERO, LIBERO-Plus, and RoboTwin 2.0, covering both single-arm and bimanual manipulation under diverse distribution shifts. For LIBERO, we follow the standard protocol over the Spatial, Object, Goal, and Long suites (Liu et al., 2023). We further evaluate zero-shot robustness on LIBERO-Plus under seven controlled shifts in camera viewpoint, robot initialization, object layout, language, lighting, background, and sensor noise (Fei et al., 2026). For RoboTwin 2.0, we evaluate on a fixed 8-task subset: adjust bottle, click bell, grab roller, move can pot, open microwave, place can basket, scan object, and turn switch. These tasks cover diverse manipulation behaviors and object-state transitions. We report results under both Easy and Hard settings (Chen et al., 2025b).

Implementation, training, and evaluation protocol.

We use a Qwen3-VL-4B (Bai et al., 2025) backbone and a layerwise flow-matching action expert, following the StarVLA training recipe (Community, 2026). Training proceeds in three stages: (1) we pretrain the spatio-temporal object encoder using offline object-level supervision; (2) with the encoder frozen, we jointly optimize the VLM, action expert, and policy-side object modules using Eq. 13; and (3) we freeze the VLM and object-related modules and refine only the action expert with \mathcal{L}_{\mathrm{act}}. Training is conducted on 8 NVIDIA H100 GPUs, while policy inference during evaluation is performed on NVIDIA RTX 4090 GPUs. Each LIBERO task is evaluated over 50 episodes, with the same checkpoint evaluated on LIBERO-Plus without additional fine-tuning. For RoboTwin 2.0, the policy is trained on 50 clean demonstrations per task and evaluated over 100 rollouts per task under both Easy and Hard settings.

4.1 Main Results

LIBERO and LIBERO-Plus.

Standard LIBERO measures nominal manipulation performance, while LIBERO-Plus evaluates the same LIBERO-trained policy under controlled distribution shifts without additional fine-tuning. As shown in Table 1, ObjDream achieves the highest average success rate on LIBERO (98.6%) and ranks second overall on LIBERO-Plus (84.4%), with the best results under language (90.6%), lighting (98.4%), and background (98.5%) perturbations and second-best performance under sensor noise (88.6%). These results indicate strong robustness to semantic and appearance variations, while camera viewpoint and robot initialization remain the main challenges.

RoboTwin 2.0.

As shown in Fig. 4, we evaluate ObjDream on the fixed eight-task RoboTwin 2.0 subset under Easy and Hard settings. ObjDream achieves average success rates of 60.25% on Easy and 22.75% on Hard, demonstrating competitive robustness under randomized scene variations without large-scale pretraining on robot data. Compared with StarVLA, ObjDream improves the average success rate by 12.87 percentage points on Easy and 15.62 percentage points on Hard, with a larger gain under the more challenging Hard setting. The improvement is particularly evident on grab roller, open microwave, and click bell under Hard conditions. These results indicate improved robustness to substantial distribution shifts in bimanual manipulation.

4.2 Ablation and Analysis

We conduct ablations on LIBERO-Long and LIBERO-Plus to assess the object interface, object relation modeling (ObjRel), object dynamics (ObjDyn), factorized object injection (Fact. Injection), and identity-guided routing (Id. Routing). Table 2 includes a backbone baseline without the object interface, while “w/o ObjRel & ObjDyn” retains the object tokens, factorized injection, identity-guided routing, and current state supervision.

The full model outperforms the backbone baseline by 2.0 and 3.2 percentage points on LIBERO-Long and LIBERO-Plus, respectively. The object interface without ObjRel or ObjDyn provides a modest improvement over the backbone on LIBERO-Plus and comparable performance on LIBERO-Long. Among single-component removals, disabling factorized injection or identity-guided routing produces the largest decreases in LIBERO-Plus performance, from 84.4% to 82.5% and 82.6%, respectively. Removing ObjDyn reduces the overall success rate to 83.2%, while removing ObjRel results in a smaller decrease to 84.1%. Overall, factorized injection, identity-guided routing, and future object state supervision provide complementary gains.

4.3 Real-World Experiments

Real-world experiments.

We evaluate ObjDream on four real-world tasks: button pressing, block pick-and-place, two-fruit placement, and tabletop sweeping, covering contact interaction, object manipulation, and long-horizon control. We compare against \pi_{0.5} (Physical Intelligence et al., 2025) and ControlVLA (Li et al., 2025a). All methods use the same task demonstrations for adaptation and are evaluated under in-distribution (ID) and out-of-distribution (OOD) settings without additional fine-tuning.

Results. As shown in Table 3, ObjDream achieves average success rates of 74% and 55% under ID and OOD conditions, respectively, exceeding the strong \pi_{0.5} baseline by 8 percentage points under ID while achieving comparable aggregate performance under OOD. Compared with ControlVLA, ObjDream improves average success rates by 14 and 24 percentage points under ID and OOD conditions, respectively. Notably, under OOD conditions, ObjDream outperforms the best baseline by 10 percentage points on both two-fruit placement and tabletop sweeping, two tasks that require precise manipulation. These results demonstrate competitive generalization in real-world manipulation without large-scale pretraining on robot data.

5 Conclusion

We present ObjDream, an object-centric VLA framework combining structured object representations with action-conditioned future object state supervision. ObjDream represents objects with persistent identity cues and time-varying state features, and leverages identity-guided routing to direct VLM attention toward task-relevant object tokens. It further supervises object state evolution during training without requiring the decoding of future object states at inference. Experiments on LIBERO, LIBERO-Plus, a fixed 8-task RoboTwin 2.0 subset, and four real-world tasks demonstrate strong manipulation performance and robustness to scene variations.

References

1. Assran et al. (2025) M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. V-jepa 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985.
2. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.
3. Bai et al. (2026) Y. Bai, H. Wang, M. Dai, Q. Zhong, Y. Liu, and L. Lin Bridge-wa: predicting where and how the world changes for robotic action. arXiv preprint arXiv:2607.02195.
4. Bendikas et al. (2025) R. Bendikas, D. Dijkman, M. Peschl, S. Haresh, and P. Mazzaglia Focusing on What Matters: Object-Agent-Centric Tokenization for Vision Language Action Models. In Proceedings of The 9th Conference on Robot Learning, Vol. 305, pp. 3869–3887.
5. Black et al. (2024) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. \pi_{0}: a Vision-Language-Action flow model for general robot control. arXiv preprint arXiv:2410.24164.
6. Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818.
7. Brohan et al. (2022) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817.
8. Bu et al. (2025) Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li Univla: learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111.
9. Cai et al. (2026) R. Cai, J. Guo, X. He, P. Jin, J. Li, B. Lin, F. Liu, W. Liu, F. Ma, K. Ma, et al. Xiaomi-robotics-0: an open-sourced vision-language-action model with real-time execution. arXiv preprint arXiv:2602.12684.
10. Cen et al. (2025a) J. Cen, S. Huang, Y. Yuan, K. Li, H. Yuan, C. Yu, B. Hou, Y. Jiang, J. Guo, X. Li, et al. Rynnvla-002: a unified vision-language-action and world model. arXiv preprint arXiv:2511.17502.
11. Cen et al. (2025b) J. Cen, C. Yu, H. Yuan, Y. Jiang, S. Huang, J. Guo, X. Li, Y. Song, H. Luo, F. Wang, et al. Worldvla: towards autoregressive action world model. arXiv preprint arXiv:2506.21539.
12. Chen et al. (2025a) H. Chen, J. Guo, B. Wang, T. Zhang, X. Huang, B. Zheng, Y. Hou, C. Tie, J. Deng, and L. Shao Goal-vla: image-generative vlms as object-centric world models empowering zero-shot robot manipulation. arXiv preprint arXiv:2506.23919.
13. Chen et al. (2025b) T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, et al. Robotwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088.
14. Chi et al. (2025) C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), pp. 1684–1704.
15. Chung et al. (2026) N. Chung, T. Hanyu, T. Nguyen, H. Le, F. Bumgarner, D. M. H. Nguyen, K. Vo, K. Yamazaki, C. Rainwater, T. Kieu, A. Nguyen, and N. Le Rethinking progression of memory state in robotic manipulation: an object-centric perspective. Proceedings of the AAAI Conference on Artificial Intelligence 40 (5), pp. 3407–3415. External Links: Document
16. Community (2026) S. Community StarVLA: a lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014.
17. Du et al. (2023) Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel Learning universal policies via text-guided video generation. Advances in neural information processing systems 36, pp. 9156–9172.
18. Fei et al. (2026) S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu LIBERO-plus: a progressive robustness benchmark for visual-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 38574–38583.
19. Ferraro et al. (2025) S. Ferraro, P. Mazzaglia, T. Verbelen, and B. Dhoedt FOCUS: object-centric world models for robotic manipulation. Frontiers in Neurorobotics 19, pp. 1585386.
20. Guo et al. (2026) X. Guo, B. Xie, W. Chai, X. Deng, T. Wang, Z. Wu, and X. Chen Priorvla: prior-preserving adaptation for vision-language-action models. arXiv preprint arXiv:2605.10925.
21. Hafner et al. (2019) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555–2565.
22. Hancock et al. (2026) A. Hancock, X. Wu, L. Zha, O. Russakovsky, and A. Majumdar Actions as language: fine-tuning vlms into vlas without catastrophic forgetting. In International Conference on Learning Representations, Vol. 2026, pp. 75333–75360.
23. Hanyu et al. (2025) T. Hanyu, N. Chung, H. Le, T. Nguyen, Y. Ikebe, A. Gunderman, D. N. H. Minh, K. Vo, T. Kieu, K. Yamazaki, et al. Slotvla: towards modeling of object-relation representations in robotic manipulation. arXiv preprint arXiv:2511.06754.
24. Hou et al. (2025) Z. Hou, T. Zhang, Y. Xiong, H. Duan, H. Pu, R. Tong, C. Zhao, X. Zhu, Y. Qiao, J. Dai, et al. Dita: scaling diffusion transformer for generalist vision-language-action policy. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7686–7697.
25. Jeong et al. (2025) Y. Jeong, J. Chun, S. Cha, and T. Kim Object-centric world model for language-guided manipulation. arXiv preprint arXiv:2503.06170.
26. Kim et al. (2025) M. J. Kim, C. Finn, and P. Liang Fine-tuning vision-language-action models: optimizing speed and success. arXiv preprint arXiv:2502.19645.
27. Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246.
28. Kipf et al. (2021) T. Kipf, G. F. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594.
29. Li et al. (2026a) F. Li, W. Song, H. Zhao, J. Wang, P. Ding, D. Wang, L. Zeng, and H. Li Spatial forcing: implicit spatial representation alignment for vision-language-action model. In International Conference on Learning Representations, Vol. 2026, pp. 132324–132345.
30. Li et al. (2025a) P. Li, Y. Wu, Z. Xi, W. Li, Y. Huang, Z. Zhang, Y. Chen, J. Wang, S. Zhu, T. Liu, and S. Huang ControlVLA: Few-Shot Object-Centric Adaptation for Pre-Trained Vision-Language-Action Models. In Proceedings of The 9th Conference on Robot Learning, Vol. 305, pp. 1898–1913.
31. Li et al. (2026b) W. Li, P. Jia, Y. Ma, X. Jiang, T. Jiang, S. Sun, Y. Li, X. Wen, H. Hong, Z. Liu, et al. FoMoVLA: bridging visual foresight and motion guidance for vision-language-action models. arXiv preprint arXiv:2607.14739.
32. Li et al. (2026c) X. Li, Y. Zheng, S. Gu, H. Ma, F. Yu, X. Nie, S. Yuan, Y. Zang, W. Li, S. Tian, et al. Latent action as intention enables efficient future imagination for world action models. arXiv preprint arXiv:2608.24882.
33. Li et al. (2025b) X. Li, J. Xu, M. Zhang, J. Liu, Y. Shen, I. Ponomarenko, J. Xu, L. Heng, S. Huang, S. Zhang, et al. Object-centric prompt-driven vision-language-action model for robotic manipulation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 27638–27648.
34. Li et al. (2025c) Y. Li, Y. Deng, J. Zhang, J. Jang, M. Memmel, C. Garrett, F. Ramos, D. Fox, A. Li, A. Gupta, et al. Hamster: hierarchical action models for open-world robot manipulation. In International Conference on Learning Representations, Vol. 2025, pp. 24040–24068.
35. Liang et al. (2026) W. Liang, G. Sun, Y. He, J. Dong, S. Dai, I. Laptev, S. Khan, and Y. Cong Pixelvla: advancing pixel-level understanding in vision-language-action model. In International Conference on Learning Representations, Vol. 2026, pp. 110464–110481.
36. Lipman et al. (2022) Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. arXiv preprint arXiv:2210.02747.
37. Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, pp. 44776–44791.
38. Liu et al. (2024) S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38–55.
39. Liu et al. (2026) Y. Liu, P. Sun, S. Li, Y. Xie, L. Zhang, X. Chao, S. Dong, F. Chen, X. Zhang, and W. Ding Oa-wam: object-addressable world action model for robust robot manipulation. arXiv preprint arXiv:2605.06481.
40. Locatello et al. (2020) F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf Object-centric learning with slot attention. Advances in neural information processing systems 33, pp. 11525–11538.
41. Mosbach et al. (2024) M. Mosbach, J. N. Ewertz, A. Villar-Corrales, and S. Behnke Sold: slot object-centric latent dynamics models for relational manipulation learning from pixels. arXiv preprint arXiv:2410.08822.
42. NVIDIA et al. (2025) NVIDIA, J. Bjorck, F. Castaneda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. GR00T N1: an open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734.
43. Oquab et al. (2023) M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.
44. O’Neill et al. (2024) A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open x-embodiment: robotic learning datasets and rt-x models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892–6903.
45. Physical Intelligence et al. (2025) Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. \pi_{0.5}: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054.
46. Qu et al. (2025) D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. Spatialvla: exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830.
47. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, pp. 8748–8763.
48. Ravi et al. (2025) N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: segment anything in images and videos. In International Conference on Learning Representations, Vol. 2025, pp. 28085–28128.
49. Sun et al. (2026) J. Sun, W. Zhang, Z. Qi, S. Ren, Z. Liu, H. Zhu, G. Sun, X. Jin, and Z. Chen Vla-jepa: enhancing vision-language-action model with latent world model. arXiv preprint arXiv:2602.10098.
50. Team et al. (2024) O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213.
51. Team et al. (2026) X. R. Team, J. Guo, P. Jin, J. Li, P. Li, Y. Li, F. Liu, W. Peng, O. Qin, Y. Su, et al. Xiaomi-robotics-1: scaling vision-language-action models with over 100k hours of real-world trajectories. arXiv preprint arXiv:2607.15330.
52. Tran et al. (2026) S. Tran, D. Nguyen, H. Vo, K. Vo, and N. Le Dual-state slot attention: decoupling appearance and identity for video object-centric learning. arXiv preprint arXiv:2606.12601.
53. Villar-Corrales and Behnke (2025) A. Villar-Corrales and S. Behnke Playslot: learning inverse latent dynamics for controllable object-centric video prediction and planning. arXiv preprint arXiv:2502.07600.
54. Vo et al. (2025) K. Vo, T. Hanyu, Y. Ikebe, T. T. Pham, N. Chung, M. N. Vu, D. N. H. Minh, A. Nguyen, A. Gunderman, C. Rainwater, et al. Clutter-robust vision-language-action models through object-centric and geometry grounding. arXiv preprint arXiv:2512.22519.
55. Wang et al. (2026a) K. Wang, K. Fan, C. Qiu, Z. Shangguan, Y. Fu, Y. Fu, D. Seita, and X. Xue Oflow: injecting object-aware temporal flow matching for robust robotic manipulation. arXiv preprint arXiv:2604.17876.
56. Wang et al. (2026b) Q. Wang, M. Li, J. Guan, J. Ye, S. Xie, Y. Liu, J. Chen, Z. Liang, J. Zhang, X. Hu, et al. Qwen-vla: unifying vision-language-action modeling across tasks, environments, and robot embodiments. arXiv preprint arXiv:2605.30280.
57. Wang et al. (2026c) X. Wang, C. Wang, Y. Xu, M. Ye, F. Zhang, J. Tian, X. Zhan, L. Zhu, C. Lu, and L. Yang Lamp: learning vision-language-action policies with 3d scene flow as latent motion prior. arXiv preprint arXiv:2603.25399.
58. Wei et al. (2026) J. Wei, Y. Li, C. Lu, and J. Sun Physically ground commonsense knowledge for articulated object manipulation with analytic concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13519–13528.
59. Wu et al. (2023) P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg Daydreamer: world models for physical robot learning. In Conference on robot learning, pp. 2226–2240.
60. Wu et al. (2026a) W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, F. Wang, Q. Zhu, H. Sun, Y. Wang, S. Ma, et al. A pragmatic vla foundation model. arXiv preprint arXiv:2601.18692.
61. Wu et al. (2026b) W. Wu, F. Wang, F. Lu, H. Sun, S. Liu, Y. Wang, Y. Yan, Y. Wang, S. Ma, X. Wang, et al. From foundation to application: improving vla models in practice. arXiv preprint arXiv:2607.06403.
62. Wu et al. (2022) Z. Wu, N. Dvornik, K. Greff, T. Kipf, and A. Garg Slotformer: unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861.
63. Xiao et al. (2026) L. Xiao, J. Li, J. Gao, F. Ye, Y. Jin, J. Qian, J. Zhang, Y. Wu, and X. Yu Ava-vla: improving vision-language-action models with active visual attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13453–13463.
64. Yang et al. (2026) Y. Yang, S. Cheng, Y. Fang, H. Bharadhwaj, M. Ding, G. Bertasius, and D. Szafir Lilo-vla: compositional long-horizon manipulation via linked object-centric policies. arXiv preprint arXiv:2602.21531.
65. Ye et al. (2026) S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922.
66. Yu et al. (2026a) H. Yu, H. Lin, J. Zhang, W. Zhang, C. Gu, H. Li, and P. Tan Maskwam: unifying mask prompting and prediction for world-action models. arXiv preprint arXiv:2606.13515.
67. Yu et al. (2026b) Q. Yu, J. You, Y. Wang, J. Liang, B. Ping, Y. Tian, Y. Chen, M. Cai, Z. Gong, R. Wu, et al. AffordanceVLA: a vision-language-action model empowering action generation through affordance-aware understanding. arXiv preprint arXiv:2606.06155.
68. Yuan et al. (2026) T. Yuan, Z. Dong, Y. Liu, and H. Zhao Fast-wam: do world action models need test-time future imagination?. arXiv preprint arXiv:2603.16666.
69. Zhang et al. (2025) W. Zhang, H. Liu, Z. Qi, Y. Wang, X. Yu, J. Zhang, R. Dong, J. He, H. Wang, Z. Zhang, L. Yi, W. Zeng, and X. Jin DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge. In Advances in Neural Information Processing Systems, Vol. 38, pp. 24195–24228.
70. Zhao et al. (2025) Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. Cot-vla: visual chain-of-thought reasoning for vision-language-action models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1702–1713.
71. Zhao et al. (2023) T. Z. Zhao, V. Kumar, S. Levine, and C. Finn Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705.
72. Zheng et al. (2026) J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, et al. X-vla: soft-prompted transformer as scalable cross-embodiment vision-language-action model. In International Conference on Learning Representations, Vol. 2026, pp. 60580–60606.
73. Zhou et al. (2024a) G. Zhou, H. Pan, Y. LeCun, and L. Pinto Dino-wm: world models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983.
74. Zhou et al. (2024b) S. Zhou, Y. Du, J. Chen, Y. Li, D. Yeung, and C. Gan Robodreamer: learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377.
75. Zhu et al. (2023) Y. Zhu, A. Joshi, P. Stone, and Y. Zhu Viola: imitation learning for vision-based manipulation with object proposal priors. In Conference on Robot Learning, pp. 1199–1210.

Appendix A Offline Annotation Pipeline

Importantly, the object-level supervision is automatically generated from the demonstration trajectories and does not require manual frame-wise annotation of masks, boxes, geometry, or visibility. For each trajectory with instruction x and visual observations, we construct object-level supervision through phrase extraction, grounded localization, mask propagation, and automatic object-state annotation, yielding temporally aligned masks, geometry, visibility, and phrase–region correspondences. The offline annotation pipeline is illustrated in Fig. 5.

Instruction-conditioned object phrases.

During automatic offline annotation, Qwen3-VL-8B (Bai et al., 2025) takes the language instruction x together with the initial visual observation I_{0} and extracts the task-relevant phrase set P_{0}. The extracted phrases are fixed for the entire trajectory and used as consistent semantic references across frames. When multiple instances share the same category, coarse spatial cues, such as relative left/right positions, are included when needed to distinguish them. During offline annotation, the label robot gripper is used to localize robot regions and assign their robot role.

Phrase-grounded localization and temporal segmentation.

Given P_{0}, Grounding DINO (Liu et al., 2024) provides initial phrase-conditioned localization, and SAM2 (Ravi et al., 2025) propagates the corresponding masks across the demonstration trajectory. The phrase associated with each object remains fixed while its spatial support changes with motion, occlusion, or visibility. Boxes, centers, areas, and visibility labels are automatically derived from the propagated masks and normalized with respect to the image dimensions where applicable. DINOv2 (Oquab et al., 2023) features are pooled over the initial grounded object region to construct the visual identity reference during encoder pretraining. Thus, the masks, geometry, and visibility labels serve as automatically generated offline supervision and are not provided as policy inputs during rollout. Grounding DINO localization and SAM2 video mask propagation are used only for offline annotation and are not executed during rollout.

Appendix B Future Object State Prediction Analysis

We analyze whether the auxiliary future object state decoder learns nontrivial object state evolution rather than simply preserving the current state. We evaluate three representative LIBERO manipulation tasks covering object placement and articulated-object interaction. Only valid non-robot object slots are evaluated; robot and padding slots are excluded. For each prediction origin, the decoder predicts object centers, bounding boxes, and masks at every consecutive future step h\in\{1,\ldots,H_{f}\}, with H_{f}=8.

We compare the learned decoder against a current state persistence baseline that repeats the current object estimate at each future step. We report an Unclamped subset containing only predictions whose future targets are directly available within the trajectory, and a Displacement subset containing object–target pairs whose centroid changes by more than 1\% of the image diagonal. Center error is normalized by the image diagonal, while box and mask IoU measure predicted object extent.

The learned decoder consistently improves over current state persistence. On the Unclamped subset, the learned decoder has lower center error and higher box and mask IoU than current state persistence. The gains are more pronounced under object displacement, where center error decreases from 13.91\% to 2.03\% and box and mask IoU improve from 0.287 and 0.245 to 0.575 and 0.542. These results indicate that the auxiliary decoder captures nontrivial future object state changes rather than merely propagating the current state.

Qualitative future object state predictions.

Figure 6 shows representative LIBERO future object state predictions. Although the decoder predicts all horizons h=1,\ldots,H_{f}, we display only h=1,4,8 for clarity. Figures 8 and 7 show additional predictions on real-world tasks and RoboTwin 2.0, respectively.

Effect of action conditioning on future object state prediction.

To examine whether the decoder uses action information, we intervene on the action conditioning path using the same trained model and evaluation tasks. Model parameters, contextualized object tokens, and action conditioning gates are kept fixed. Native Predicted Action uses the original auxiliary action proposal; No Action sets the position-aware action embedding to zero; and Mismatched Action replaces it with the position-aware embedding of a predicted action chunk from another sample within the same task. As shown in Table 5, the native condition yields the lowest center error and the highest box and mask IoU on both subsets. On the Unclamped subset, center error is 1.60\% with the native condition, compared with 1.90\% without action conditioning and 2.02\% with mismatched actions. On the Displacement subset, center error increases from 2.03\% to 2.35\% and 2.71\%, respectively. Box IoU decreases from 0.575 to 0.546 and 0.532, while mask IoU decreases from 0.542 to 0.534 and 0.486. These interventions indicate that the future object state decoder uses the action condition, while not establishing counterfactual dynamics under arbitrary action sequences.