LTM：基于潜在转移的上下文记忆，为VLA模型补上记忆能力

1 Introduction

Vision-language-action (VLA) models combine pretrained visual and linguistic representations with action generation to follow diverse manipulation instructions (Brohan et al., 2023; Kim et al., 2024; Black et al., 2024; Physical Intelligence et al., 2025). Yet the current observation may omit information needed for the next action. A robot placing three objects into a bag must remember completed transfers; one tracking an occluded object must retain the effects of earlier interactions. Similar scenes can therefore require different actions depending on the preceding history, motivating explicit memory for robotic control (Dai et al., 2026; Chung et al., 2026).

Existing memory designs retain visual observations, summarize experience in language, or update a recurrent state (Shi et al., 2025; Qu et al., 2026; Torne et al., 2026; Cherepanov et al., 2026). Dense visual histories consume substantial context, whereas sparse sampling can omit intermediate interactions. Language summaries abstract away visual detail but may miss fine-grained motion. Recurrent representations support compact updates but must continually decide what to preserve in a bounded state.

These trade-offs raise a representational question: what should a VLA remember about its past interactions?

For many manipulation tasks, the relevant history is the sequence of changes induced by robot–environment interactions. Latent action models encode an observation pair into a compact code from which the later observation can be reconstructed conditioned on the earlier one (Bruce et al., 2024; Ye et al., 2024). This objective encourages the code to capture temporal changes. While latent actions have been used for policy pretraining and world-model control (Ye et al., 2024; Bu et al., 2025; Gao et al., 2026a), we study their use as an ordered episodic memory for action generation.

We introduce Latent Transition Memory (LTM), which encodes consecutive observations with a frozen latent action model and projects each code into one memory token. The policy receives a delayed transition prefix alongside the instruction and current observation. Each code depends only on an adjacent observation pair, allowing encoding to overlap action execution. We instantiate LTM on \pi_{0.5} as LTM-\pi, with a lightweight adapter and causal attention that permits persistent KV caching. Sparse initial-frame anchors provide visual grounding for video-conditioned tasks. The policy and adapter learn from robot demonstrations without memory-specific semantic annotations. At inference, the frozen encoder supplies transition codes directly to the policy, without an external reasoning model or intermediate language generation.

We evaluate LTM-\pi with two frozen latent encoders, CD-LAM and DreamDojo, on RoboMME and LIBERO-Mem. With the original benchmark data, the DreamDojo variant reaches 58.54% average success on RoboMME, compared with 17.93% for the reported memory-free \pi_{0.5} baseline, and 80.5% on LIBERO-Mem, compared with 42.5% for the strongest baseline included in our comparison. We then isolate the effects of memory content and inference lag. Replacing observation-dependent latent transitions with fixed random representations reduces RoboMME success from 57.67% to 44.50%, supporting the contribution of transition-specific content beyond temporal-progress cues. Using a five-step lag reduces estimated latent-encoder blocking from 114.25 ms to 9.47 ms relative to zero lag, with a small change in success (57.88% to 57.67%).

We next test how LTM benefits from broader demonstration coverage. We expand RoboMME from 100 to 1,000 episodes per task family and increase training from 40K to 120K steps. With the same encoder and policy architecture, success rises from 57.67% to 87.50% (Figure 1). All four tracks improve; Reference, the weakest base-data track, gains 47.50 percentage points. Extending training to 120K steps on the original data alone yields no appreciable gain, supporting the benefit of broader demonstration coverage. LTM-\pi exceeds the reported PonderPounce result with 9\times data by 11.96 points overall, under different data budgets and supervision assumptions.

Scaling does not fully resolve the challenge of composing local transitions into persistent object states: VideoUnmaskSwap and ButtonUnmaskSwap reach 74% and 62%, versus 100% for their non-swap counterparts. In a Cup Swap diagnostic, process supervision achieves 97.93% final-state recovery from latent transitions, compared with 2.40% under final-state-only supervision. These results demonstrate that latent transitions preserve recoverable multi-step interaction information and highlight the importance of intermediate supervision, motivating language-guided reasoning and process supervision to address remaining temporal-composition challenges.

To bound execution-memory growth, we merge the most similar adjacent projected tokens under the same quota during training and inference. On RoboMME-10x, a 200-token budget retains 84.00% success versus 87.50% uncompressed and reduces mean transition-token count by at least 38.26% on profiled training queries.

Finally, we evaluate LTM-\pi on a Galbot G1 robot in a real-world repeated pick-and-place counting task. LTM-\pi improves average success from 11.25% for the memory-free \pi_{0.5} baseline to 82.50%, demonstrating that transition memory also helps track completed interactions during physical manipulation.

Our contributions are threefold:

• We propose Latent Transition Memory (LTM), a memory representation that uses compact latent transitions between consecutive observations as historical context instead of retaining the full visual history. This focuses memory on interaction-induced changes and enables the policy to use past experience without external reasoning-model calls or memory-specific semantic annotations.
• We develop LTM-\pi, integrating transition memory into \pi_{0.5}’s inference context through a lightweight adapter. Causal attention, persistent KV caching, and asynchronous transition encoding support efficient online control. Experiments on RoboMME and LIBERO-Mem demonstrate strong memory-dependent manipulation performance, with a measured policy inference time of 66.71 ms on an H100.
• We characterize transition-memory information, data scaling, and context-budget trade-offs. Ablations distinguish transition content from temporal cues; tenfold RoboMME data expansion raises success from 57.67% to 87.50%; process-supervised Cup Swap decoding recovers final states with 97.93% accuracy; and a 200-token budget retains 84.00% success. Finally, real-robot counting experiments improve success from 11.25% to 82.50%.

2 Related Work

Vision-language-action models.

VLA models transfer pretrained visual and linguistic representations to robotic control. RT-2 (Brohan et al., 2023) predicts tokenized actions, and OpenVLA (Kim et al., 2024) provides an open-source generalist policy. \pi_{0} (Black et al., 2024) introduces flow matching for continuous action generation; \pi_{0.5} (Physical Intelligence et al., 2025) and LingBot-VLA (Wu et al., 2026) extend generalization through heterogeneous co-training and multi-embodiment data. These advances do not remove partial observability: the appropriate action may depend on events absent from the current observation. RoboMME (Dai et al., 2026) and LIBERO-Mem (Chung et al., 2026) evaluate this need for execution memory.

Memory mechanisms for VLAs.

RoboFlamingo (Li et al., 2023) models history through a sequential policy head; HAMLET (Koo et al., 2025) and \muVLA (Cherepanov et al., 2026) use learned memory tokens. MemoryVLA (Shi et al., 2025) and LaMem-VLA (Qu et al., 2026) retrieve evidence from explicit memory banks. SAM2Act+ (Fang et al., 2025) extends a multi-view manipulation policy with a spatial memory bank, encoder, and memory attention. Object-centric approaches retain a different unit of history: SlotVLA (Hanyu et al., 2025) models object slots and their relations, while E-SlotSSM (Chung et al., 2026) studies the progression of object-centric memory states. GMP (Gao et al., 2026b) learns when to activate cross-attention over cached image and action tokens. RoboMME also compares symbolic, sampled-frame, and recurrent memories with several policy interfaces (Dai et al., 2026). LTM instead retains an ordered sequence of changes between observations, rather than appearance tokens or a recurrent summary of the episode. Temporal prediction also informs memory-based control: HiF-VLA (Lin et al., 2025) incorporates motion representations, MemoryVLA++ (Shi et al., 2026) combines retrieved history with imagined futures, and MemoryVAM (Jiang et al., 2026) injects compressed visual memory into both video prediction and action decoding.

MEM (Torne et al., 2026) combines recent video with long-term textual memory, HiMe (Ji et al., 2026) separates working memory from long-term planning, and WeaveLA (Zhu et al., 2026) passes latent summaries of completed segments to subsequent subtasks. MemER (Sridhar et al., 2025) retrieves keyframes for a high-level vision-language model that generates instructions for a low-level policy. PonderPounce (Choi et al., 2026) uses a separate pretrained multimodal context model to asynchronously supply continuous cognition tokens to its action model. Its RoboMME training recipe includes transition, subgoal, and demonstration-reasoning supervision. In 2AM (Hu et al., 2026), an agent maintains episodic memory and guides an episodically stateless action model through subtask language and optional 2D spatial hints. These designs delegate historical interpretation to a higher-level model or interface. LTM instead exposes observed transitions directly to the action policy, without invoking a separate reasoning model or generating intermediate language during control.

SimpleMemVLA (Yin et al., 2026) retains timestamped video in its VLM’s native context and reuses the prefix cache through streaming prefill. Its action head receives representations of generated subtask text, with subtask annotations produced offline by a cloud VLM. This is a training-time annotation dependency, rather than a requirement for cloud-model calls during control. T2Mem (Liu et al., 2026) places a fast-weight memory between the VLM and action expert and updates it through self-supervised test-time gradients. Like LTM, it learns from action demonstrations without an external reasoning model or memory-specific annotations. Our distinction is therefore the memory representation and update rule: LTM appends frozen pairwise transition codes to a persistent sequence, without test-time gradient updates. Its causal attention and delayed prefix support incremental inference while preserving access to the current observation.

Latent action models.

LAMs infer compact transition codes from observation pairs, typically using reconstruction without action labels (Bruce et al., 2024; Ye et al., 2024). Genie (Bruce et al., 2024) uses discrete codes for controllable video generation, LAPA (Ye et al., 2024) uses them for policy pretraining, and CLAM (Liang et al., 2025) and UniVLA (Bu et al., 2025) extend latent actions to continuous and cross-embodiment control. DreamDojo (Gao et al., 2026a) uses latent actions for world-model pretraining; CD-LAM (Wei et al., 2026) reduces visual confounding in such codes. We use pretrained LAMs only to encode observed transitions into episodic memory; the deployed policy does not need to reconstruct past frames or generate future videos.

3 Method

3.1 Latent-Transition-Based In-Context Memory for VLAs

We represent interaction history using compact latent transitions between consecutive observations. A latent action model (LAM) encodes each transition from o_{i-1} to o_{i} into a latent code z_{i}, from which a decoder reconstructs o_{i} conditioned on o_{i-1}:

z_{i}=E_{\phi}(o_{i-1},o_{i}),\qquad\hat{o}_{i}=D_{\psi}(o_{i-1},z_{i}),\qquad\mathcal{L}_{\mathrm{LAM}}=d(\hat{o}_{i},o_{i}),

where E_{\phi} and D_{\psi} are the transition encoder and decoder, respectively, and d is a reconstruction distance. Conditioning the decoder on the preceding observation encourages the compact latent bottleneck to capture temporal changes rather than redundant appearance information.

Two properties make these representations useful for VLA memory: (i) Compactness. Each transition is represented by a low-dimensional vector, reducing the context budget required to retain interaction history. (ii) Locality. Each code depends only on two consecutive observations, allowing incremental memory construction without re-encoding earlier frames.

Let \mathbf{A}_{t} denote an action chunk, s_{t} the robot state, and \ell the language instruction. A current-observation policy predicts \mathbf{A}_{t}\sim\pi_{\theta}(\cdot\mid o_{t},s_{t},\ell). Existing memory mechanisms augment this policy in different ways: frame-based memory uses sampled observations, \mathbf{A}_{t}\sim\pi_{\theta}(\cdot\mid o_{t},s_{t},\ell,\{o_{i}\}_{i\in\mathcal{I}_{t}}), where \mathcal{I}_{t}\subseteq\{0,\ldots,t-1\}; language-based memory uses a summary m_{t}^{\mathrm{lang}}=f_{\mathrm{LLM}}(o_{<t}), giving \mathbf{A}_{t}\sim\pi_{\theta}(\cdot\mid o_{t},s_{t},\ell,m_{t}^{\mathrm{lang}}); and recurrent memory updates h_{t}=f_{\mathrm{rec}}(h_{t-1},o_{t}), giving \mathbf{A}_{t}\sim\pi_{\theta}(\cdot\mid o_{t},s_{t},\ell,h_{t}).

Our approach instead conditions the policy on an ordered sequence of latent transitions:

\mathcal{M}_{t}^{\mathrm{LAM}}=(z_{i})_{1\leq i\leq\tau_{t}},\qquad\mathbf{A}_{t}\sim\pi_{\theta}(\cdot\mid o_{t},s_{t},\ell,\mathcal{M}_{t}^{\mathrm{LAM}}).

Here, \tau_{t}=\max(0,t-k) excludes the latest k transitions, while o_{t} remains the current visual input. This representation retains transition-level information without dense historical RGB tokens or recurrent compression; video-conditioned tasks additionally use the sparse anchors described below.

3.2 LTM-\pi

We instantiate this memory mechanism on \pi_{0.5} and refer to the resulting model as LTM-\bm{\pi}. The policy and memory projector are trained on demonstration data using a frozen LAM encoder, without requiring language summaries or intermediate semantic supervision. The architecture of LTM-\bm{\pi} is shown in Figure 2.

Training Phase.

For each training sample, we use a fixed lag k, shared with inference, and set \tau_{t}=\max(0,t-k). Frozen DreamDojo (Gao et al., 2026a) or CD-LAM (Wei et al., 2026) encodes the retained observation pairs into 32-dimensional transition vectors:

z_{i}=E_{\phi}(o_{i-1},o_{i})\in\mathbb{R}^{32},\qquad\tilde{z}_{i}=\mathtt{stopgrad}\left(\frac{z_{i}-\mu_{\mathcal{D}}}{\sigma_{\mathcal{D}}}\right),\quad 1\leq i\leq\tau_{t},

where E_{\phi} denotes the LAM encoder, and \mu_{\mathcal{D}} and \sigma_{\mathcal{D}} are dataset-level normalization statistics.

We use a lightweight memory adapter to align latent actions with the semantic space of the VLM. A two-layer MLP projector maps each normalized transition into a memory token in the 2048-dimensional hidden space of the PaliGemma-3B backbone of \pi_{0.5}:

m_{i}=\mathrm{LN}\!\left(W_{2}\,\mathrm{GELU}(W_{1}\tilde{z}_{i}+b_{1})+b_{2}\right),\qquad W_{1}\in\mathbb{R}^{512\times 32},\quad W_{2}\in\mathbb{R}^{2048\times 512}.

We insert these tokens between the language and current visual tokens:

X_{t}=\big[X^{\mathrm{lang}}(\ell);m_{1},\ldots,m_{\tau_{t}};X^{\mathrm{vis}}(o_{t})\big].

The resulting context conditions the action expert. We apply causal attention within the language–memory prefix and allow the current visual tokens to attend to this prefix. Each prefix token therefore depends only on itself and preceding prefix tokens, enabling its KV representations to be reused when new memory tokens and visual inputs arrive.

Inference Phase.

Algorithm 1 summarizes online inference. At decision step t, the policy consumes transitions only through \tau_{t}=\max(0,t-k). Each transition is encoded once, asynchronously after its observation pair becomes available, and stored in a latent buffer; the lag delays only when it is admitted to policy memory. Thus, a transition has up to k observation steps of computation slack before it is required. At each decision, newly eligible latents are appended to the persistent language–memory KV cache, after which o_{t} is processed and an action chunk is generated. This overlaps LAM computation with action execution and reduces critical-path blocking.

For n cached prefix tokens and b newly processed memory and visual tokens, caching reduces the per-chunk backbone self-attention cost from \mathcal{O}((n+b)^{2}) to \mathcal{O}(nb+b^{2}). For fixed b, this cost is linear in the cached history length. In the append-only variant, the cache itself also grows linearly with the number of retained transitions. Section 4.1.6 evaluates an optional compression rule that bounds execution-memory length; merging existing tokens requires refreshing the affected cache entries.

Visual Anchors for Video Demonstrations.

For tasks conditioned on video demonstrations, the initial scene layout may differ between the demonstration and execution environments. To improve visual–semantic alignment, we complement latent transitions with sparse visual features from both sequences, inspired by frame-based memory sampling. A minimal choice uses the first frame of each sequence as a visual anchor:

X_{t}=\big[X^{\mathrm{lang}}(\ell);X^{\mathrm{vis}}(o_{0}^{\mathrm{demo}});m_{1}^{\mathrm{demo}},\ldots,m_{T_{\mathrm{demo}}}^{\mathrm{demo}};X^{\mathrm{vis}}(o_{0});m_{1},\ldots,m_{\tau_{t}};X^{\mathrm{vis}}(o_{t})\big].

Here, o_{0}^{\mathrm{demo}} and o_{0} are the initial demonstration and execution observations, respectively, and m_{i}^{\mathrm{demo}} encodes the transition from o_{i-1}^{\mathrm{demo}} to o_{i}^{\mathrm{demo}}. We add distinct learnable type embeddings to the demonstration and execution segments to distinguish their roles; these embeddings are omitted from the formula for clarity. The fixed visual anchors and demonstration tokens are retained in the persistent KV cache alongside the language and execution memory tokens.

4 Experiment

Our experiments examine the effectiveness, information content, and memory cost of LTM-\pi. In simulation (Section 4.1), we introduce the benchmarks and evaluation protocol (Section 4.1.1), report results on RoboMME and LIBERO-Mem (Section 4.1.2), and isolate the effects of memory components and lag (Section 4.1.3). We then examine demonstration scaling (Section 4.1.4), probe whether latent transitions encode swap information (Section 4.1.5), and test manipulation under a fixed token budget (Section 4.1.6). Finally, Section 4.2 presents the real-robot setup and repeated pick-and-place evaluation.

4.1 Evaluation in Simulation

4.1.1 Settings

Benchmarks.

We evaluate memory-dependent robotic control on two complementary benchmarks, RoboMME (Dai et al., 2026) and LIBERO-Mem (Chung et al., 2026). RoboMME contains 16 history-dependent manipulation tasks organized into four suites: Counting, Permanence, Reference, and Imitation, targeting temporal, spatial, object, and procedural memory, respectively. These tasks require the policy to retain information from earlier observations or demonstrations that may no longer be recoverable from the current observation. LIBERO-Mem further evaluates long-horizon memory under object-level partial observability. It contains 10 tasks spanning four object-centric memory dimensions: Object Motion (OM), Object Sequence (OS), Object Relations (OR), and Object Occlusion (OO). The benchmark introduces temporally dependent subgoals and visually ambiguous objects, requiring the policy to track object identities, interactions, and state changes over trajectories of up to several hundred frames.

Training and Evaluation Configuration.

We train the model on 8 H100 GPUs with a batch size of 128, using 40K steps on the original RoboMME data and 30K steps on LIBERO-Mem. The RoboMME-10x experiment uses 120K steps (Section 4.1.4). The learning rate decays from 2.5\times 10^{-5} to 2.5\times 10^{-6}, with 1K warm-up steps. Unless otherwise specified, we use a fixed lag k=5 during training and inference. For the RoboMME data-scaling and token-compression experiments, we use k=10. The policy predicts action chunks of 50 steps on RoboMME and 10 steps on LIBERO-Mem, and replans after executing the first 10 and 5 steps, respectively. We do not use proprioceptive inputs on RoboMME. Following the official evaluation protocols, we evaluate each task on 20 layouts in LIBERO-Mem and over 50 rollouts in RoboMME. For the base-data RoboMME results in Table 1, we evaluate with three random seeds and report mean success rates. The Full and k=5 rows in the ablation tables reproduce these reference scores; the remaining ablations use a single seed.

Baselines.

For RoboMME, we compare against DP-Unet, OpenVLA-OFT, \pi_{0.5} with and without past actions, and its symbolic, perceptual, and recurrent memory variants (Dai et al., 2026). We also include MemoryVLA (Shi et al., 2025), SAM2Act+ (Fang et al., 2025), and MemER (Sridhar et al., 2025) as reported in RoboMME, alongside WeaveLA (Zhu et al., 2026), PonderPounce (9B) (Choi et al., 2026), SimpleMemVLA (Yin et al., 2026), and T2Mem (Liu et al., 2026). To make the resource assumptions explicit, Table 1 separates methods that use neither an external reasoning model nor memory-specific annotations from methods that use either resource. All baseline results are taken directly from published reports, with oracle-assisted methods and human performance listed separately for reference. Table 1 uses the original data scale; Table 5 additionally includes PonderPounce with 9\times data. These comparisons retain the reported methods’ different training and supervision resources. For LIBERO-Mem, we compare against \pi_{0.5}, SlotVLA (Hanyu et al., 2025), and Naive E-SlotSSM as reported by Chung et al. (2026), MemoryVLA (Shi et al., 2025), GMP (Gao et al., 2026b), MemoryVAM with UNet and DiT backbones (Jiang et al., 2026), and 2AM together with its \pi_{0} reimplementation (Hu et al., 2026).

4.1.2 Main Results

Results on RoboMME.

As shown in Table 1, LTM-\pi achieves average success rates of 57.67% with CD-LAM and 58.54% with DreamDojo, improving over \pi_{0.5} by 39.74 and 40.61 percentage points, respectively. Within the block that uses neither external reasoning-model calls nor memory-specific annotations, the DreamDojo and CD-LAM variants rank first and second overall; DreamDojo exceeds the strongest baseline in this block, T2Mem (56.83%), by 1.71 percentage points. Both variants also outperform baselines, suggesting that compact transition representations provide an effective alternative to sampled-frame memory. The DreamDojo variant is particularly strong on Counting, achieving 90.33% on average and solving StopCube with 100% success. For context, SimpleMemVLA reaches 88.25% using cloud-VLM-generated sub-task annotations, while PonderPounce reaches 60.83% using a separate 9B context model and additional grounding and demonstration-reasoning supervision. LTM-\pi achieves its results without external reasoning-model calls or memory-specific annotations. Reference tasks remain a target for improving fine-grained object identity and visual correspondence. In terms of inference efficiency, the average latency of a LTM-\pi policy inference on an H100 is 66.71 ms. Figure 1 compares task-level success rates.

Results on LIBERO-Mem.

As shown in Table 2, LTM-\pi achieves average success rates of 74.5% with CD-LAM and 80.5% with DreamDojo, exceeding the strongest baseline included in this comparison, MemoryVAM (UNet), by 32.0 and 38.0 percentage points, respectively. DreamDojo improves over CD-LAM by 6.0 points overall, with particularly large gains on repeated-interaction tasks T3–T5, while both variants achieve 100% success on T8 and T9. These results support the effectiveness of latent-transition memory for history-dependent manipulation across different LAM encoders. The encoder choice nevertheless affects individual tasks: CD-LAM performs better on swapping two bowls (T6), achieving 80.0% versus 65.0%. Rotating three bowls (T7) remains challenging for both variants (20.0% and 25.0%), suggesting that tracking multiple objects through sequential interactions remains a limitation.

4.1.3 Ablations

We first isolate the contributions of transition content and visual anchors, then test how delaying memory admission affects success and online computation.

Effect of Latent Transitions and Visual Anchors.

In Table 3, we ablate the two sources of historical information in LTM-\pi: latent-transition memory and sparse visual anchors. We compare the full model with LT-only, which removes visual anchors; Anchor-only, which removes latent transitions; and Fixed Random LT, which replaces observation-dependent latent transitions with fixed random representations while preserving the temporal structure of the memory sequence.

Removing latent transitions reduces average success from 57.67% to 23.13%, while LT-only retains 49.13%, showing that the transition sequence is the stronger historical component and anchors mainly provide complementary visual grounding.

Fixed Random LT still reaches 44.50%, indicating that the ordered memory sequence itself provides useful cues about elapsed time and trajectory progress. Replacing these random tokens with observation-dependent latent transitions further improves performance to 57.67%, demonstrating that transition-specific content contributes beyond temporal cues. This gain is limited on swap-dependent tasks, motivating the diagnostic study below.

Effect of Lag k.

On RoboMME, we replan every 10 execution steps and use the same k for training and inference. As shown in Table 4, lag creates computation slack between observing a transition and admitting it to policy memory. Increasing k from 0 to 5 changes average success only from 57.88% to 57.67%, while reducing the estimated LAM blocking cost from 114.25 ms to 9.47 ms. At k=10, the measured LAM update fits within the available execution overlap, giving an estimated zero blocking cost, with 56.38% success. These point estimates indicate that a small non-zero lag can reduce blocking with a small change in success. The 66.71 ms policy time excludes LAM work; the blocking column is an overlap estimate, not a measured end-to-end control latency.

4.1.4 Data Scaling on RoboMME

The ablations support the contribution of transition content beyond temporal cues. We next ask whether broader demonstration coverage helps the policy use this information across diverse task families. RoboMME provides 100 episodes per task family, each spanning multiple scenes and language instructions. Following the data-expansion setting explored by PonderPounce (Choi et al., 2026), we use RoboMME’s generation pipeline to increase this to 1,000 episodes per family, yielding RoboMME-10x. Training-generation seeds are kept disjoint from validation and test seeds. We increase training from 40K to 120K steps and use a fixed lag k=10 during both training and inference. To assess whether longer training alone explains the improvement, we also train on the original data for 120K steps under the same configuration.

Table 5 shows that LTM-\pi improves from 57.67% to 87.50%, a gain of 29.83 percentage points. Counting, Permanence, Reference, and Imitation rise from 86.33%, 57.17%, 28.50%, and 58.67% to 95%, 84%, 76%, and 95%, respectively. Reference gains the most (+47.50 points), showing that the weakest base-data track benefits substantially from the expanded training set. The scaled model reaches at least 88% on every Imitation task and 100% on three of four Counting tasks. At the original 1\times data scale, extending training from 40K to 120K steps yields no appreciable improvement. This control indicates that longer training alone does not account for the gain at 10\times data, supporting the contribution of additional demonstrations.

For comparison, PonderPounce improves from 60.83% to 75.54% with 9\times data (+14.71 points). At 10\times data, LTM-\pi is 11.96 points higher overall, leading on 11 tasks and tying on ButtonUnmask; PonderPounce remains higher on the four Reference tasks. The descriptive gain ratios are 3.31 versus 1.84 percentage points per additional data multiple, respectively, under different training and supervision budgets. LTM-\pi also comes within 0.75 points of SimpleMemVLA’s reported 88.25%, without memory-specific annotations but using more demonstrations (Table 1).

Despite the broad gains, VideoUnmaskSwap and ButtonUnmaskSwap reach only 74% and 62%, compared with 100% on both non-swap counterparts. Expanded demonstration coverage therefore reduces, but does not remove, the difficulty of tracking state through swaps. We therefore turn to a controlled diagnostic to test whether the required swap information can be recovered from the transition sequence (Section 4.1.5).

4.1.5 Diagnostic Study: Can Latent Transitions Represent “Swap”?

Problem.

The scaling results show that broader demonstration coverage improves overall performance, yet swap-dependent state tracking remains difficult. At 10\times data, the non-swap Video/ButtonUnmask tasks reach 100%, while their swap counterparts reach 74% and 62% (Table 5). Our qualitative inspection also finds swap-related failures in VideoRepick. These results motivate a narrower question: is information needed to track swaps recoverable from the latent sequence under controlled conditions?

Cup Swap Test.

To test recoverability separately from robotic control, we construct the Cup Swap diagnostic shown in Figure 3. Each sequence contains six cups and one to five hidden balls, followed by 5–20 randomly generated cup swaps. The initial ball locations are provided explicitly, while only the post-reveal motion history is represented by latent transitions. We inject these transitions into a Qwen3-VL-2B backbone and compare two supervision schemes: Vanilla, which directly predicts the final cup occupancy without intermediate supervision, and Process supervision (CoT-like), which explicitly supervises swap tracking and the resulting intermediate/final state. The initial state is supplied as text, and this diagnostic uses a separate latent-fusion architecture rather than the LTM-\pi policy. Details are provided in Appendix A.

As shown in Table 6, the process-supervised model recognizes individual swaps with 99.78% accuracy and recovers final states with 97.93% accuracy using a frozen VLM backbone, while the final-state-only model reaches 2.40% best exact accuracy. These results demonstrate that latent transitions contain recoverable multi-step swap information and that structured intermediate supervision enables accurate temporal-state recovery in the diagnostic. The final-state-only model also has low training accuracy (approximately 12.5%; Appendix A), so optimization differences may contribute to the observed gap. The findings motivate process-guided temporal reasoning in future transition-memory control policies.

4.1.6 Token Compression: Can Transition Memory Remain Effective Under a Fixed Budget?

The diagnostic demonstrates that process-supervised decoding recovers multi-swap state from the latent-transition sequence. We now return to LTM-\pi control and ask whether a bounded context can retain enough information for effective manipulation. Although each transition occupies only one token, append-only execution memory still grows with episode duration. We evaluate DreamDojo-based LTM-\pi on RoboMME-10x with a quota of Q=200 execution-transition tokens and a fixed lag k=10, using the same quota and compression rule during training and inference. The quota applies to execution transitions only; language, current visual inputs, and any fixed demonstration context are outside this count.

Online adjacent-token merging.

Let \mathcal{U}=(u_{1},\ldots,u_{n}) denote the ordered list of retained tokens in the projector output space. When a new eligible transition token m=P(\tilde{z}) arrives, we append it to \mathcal{U}. If the length exceeds Q, we select the most similar adjacent pair and replace it with its arithmetic mean:

j^{*}=\underset{1\leq j<n}{\arg\max}\;\operatorname{cos}(u_{j},u_{j+1}),\qquad\bar{u}=\frac{u_{j^{*}}+u_{j^{*}+1}}{2}.

Here, n is the length after appending, and \operatorname{cos} is cosine similarity. The merged token occupies the earlier position and the later token is removed. We repeat this operation until the budget is satisfied (Algorithm 2). Restricting merges to adjacent tokens preserves the order of the remaining temporal segments, while similarity guides compression toward locally redundant representations. The rule operates after projection and before the policy backbone, without an additional learned compressor. During training, each eligible history prefix is constructed by replaying these updates in chronological order; inference applies the same updates as transitions become eligible under the lag.

Each update preserves a representative of every temporal segment instead of discarding the oldest tokens. However, the merge is lossy: it removes an event boundary and replaces two vectors with one. Repeated merges use equal weights for the two current representatives, without tracking their constituent transition counts, so the result is generally not the uniform mean of all original transitions in a segment. For a stream that is already within budget, appending one token triggers at most one merge; a direct scan costs \mathcal{O}(Qd) per update for projector dimension d, and retained projected-token storage is \mathcal{O}(Qd).

Token budget.

We profile uncompressed execution memory at policy queries under the execution-only, fixed-k=10 training convention. The mean is 323.93 transition tokens/query, the median is 265, and the P90/P95 values are 747/893; the maximum is 1,789. In total, 61.98% of queries exceed 200 tokens. After applying the rule, a prefix containing N eligible transitions has exactly \min(N,Q) retained tokens. Table 7 gives the resulting bounds for the same query population. The compressed mean is at most 200, implying a reduction of at least 1-200/323.93=38.26\% in mean execution-memory length; its exact value requires the full count distribution. At the uncompressed P95 and maximum, the counts fall from 893 and 1,789 to 200, reductions of 77.60% and 88.82%, respectively. These are token-count reductions, rather than measured inference speedups.

Manipulation results.

Table 8 compares the compressed model with the uncompressed RoboMME-10x reference from Table 5. Compression achieves 672 successes over 800 rollouts, or 84.00% average success, compared with 87.50% without compression. This is a decrease of 3.50 percentage points, retaining 96.00% of the reference success rate. Counting, Permanence, Reference, and Imitation average 91.50%, 80.00%, 74.00%, and 90.50%, respectively, versus 95.00%, 84.00%, 76.00%, and 95.00% without compression.

PickXtimes, SwingXtimes, VideoUnmask, and ButtonUnmask remain at 100%, indicating that a bounded context can support these tasks despite substantial compression. The largest decreases occur on StopCube (-18 points), VideoUnmaskSwap and InsertPeg (-12 each), and VideoRepick (-10). This pattern is consistent with sensitivity to precise temporal boundaries or interaction details that averaging may weaken. In particular, the two swap tasks decline while their non-swap counterparts remain saturated. Cosine similarity measures representation similarity, rather than task relevance, so a locally similar pair can still contain an important state change. BinFill and PickHighlight improve by 4 points and VideoPlaceButton by 2, although single-seed evaluation does not resolve these smaller differences. Overall, a 200-token budget retains 96.00% of the uncompressed success rate. Together with the swap diagnostic, they motivate preserving informative event boundaries through event-aware merging or explicit segment-duration information.

Compatibility with KV caching.

The fixed token budget bounds execution-memory length independently of episode duration, making persistent KV caching less critical for computational efficiency. Although merging modifies existing prefix tokens and invalidates subsequent cached entries under causal attention, the bounded context allows these entries to be recomputed at a limited cost. Therefore, our compression strategy prioritizes bounded memory and context length over KV-cache reuse, which remains an optional optimization rather than a necessity.

4.2 Real-World Evaluation

4.2.1 Settings

Platform.

We use the Galbot G1 humanoid robot as the physical platform for real-world evaluation. During all experiments, only the two arms and grippers are actuated, while the remaining joints and body components are kept fixed.

Task and Data.

We consider a pick-and-place x times task shown in Figure 4, where the robot is required to sequentially pick up x objects and place them into a packaging bag. We collect teleoperated demonstrations for x\in\{1,2,3,4\}, with 50 trajectories for each setting, resulting in 200 trajectories in total. The dataset contains 222,834 frames recorded at 30 FPS, corresponding to 2.06 hours of real-world data.

Implementation Details.

We run inference on a single RTX 4090 GPU. The robot is controlled with a control interval of \Delta t=0.05\,\mathrm{s} per step. For real-world evaluation, we do not use anchor frames. A trial is considered successful only if the robot transfers exactly the instructed number of objects, returns to its initial pose, and remains stationary for at least 10 s.

4.2.2 Results

Table 9 shows that LTM-\pi improves success rate across all four target quantities, achieving a success rate of 82.50% compared with 11.25% for the memory-free \pi_{0.5}, a gain of 71.25 percentage points. Qualitatively, the memory-free policy tends to continue picking until approximately N objects remain in the source box, where N is the mean remaining-object count observed during data collection, suggesting reliance on a familiar scene configuration rather than tracking completed transfers. By retaining interaction history, LTM-\pi more reliably executes the requested number of transfers and stops accordingly. Per-trial transferred-object counts and a link to representative real-robot videos are provided in Appendix B.

Additional Findings.

We also observe shorter motion-onset delays for LTM-\pi than for the memory-free policy in trials with long static segments at the start of demonstrations. This qualitative pattern is consistent with temporal-progress cues from the memory sequence; onset latency was not measured separately.

5 Limitation

Our method relies on a pretrained latent action model (LAM) to provide transition representations, making memory quality dependent on the information preserved by the encoder. While latent transitions effectively capture local interaction dynamics, they may not adequately represent substantial changes in scene content, such as the appearance of new objects, where visual appearance and semantic information become important. Although sparse visual anchors provide complementary information, they may not fully address this limitation. A promising direction is to combine latent transition memory with visual memory, exploring how to effectively integrate dynamic changes and visual context for more robust long-horizon manipulation.

6 Conclusion

We introduced Latent Transition Memory, which retains observed interactions as an ordered sequence of compact transition tokens. LTM-\pi combines frozen latent action encoders with a lightweight adapter and cache-compatible causal attention, learning from action demonstrations without memory-specific semantic annotations or an external reasoning model during control. With DreamDojo transitions, it reaches 58.54% success on RoboMME and 80.5% on LIBERO-Mem at the original data scale. Ablations support the value of observation-dependent transition content beyond temporal-progress cues, with sparse visual anchors providing complementary grounding. Delayed memory admission also reduces estimated encoder blocking while largely preserving success.

We then examined the effect of demonstration scaling. Increasing RoboMME data tenfold and training from 40K to 120K steps raises success from 57.67% to 87.50%, with gains across all four tracks and the largest improvement on Reference. Extending training on the original data to 120K steps yields no appreciable gain, indicating that the improvement is not explained by longer training alone. These results support the benefit of broader demonstration coverage for learning to use compact transition memory.

Swap-dependent state tracking remains challenging after data scaling. The Cup Swap diagnostic shows that process-supervised decoding recovers final states from latent transitions with 97.93% accuracy, compared with 2.40% under final-state-only supervision. This demonstrates the recoverability of complex interaction information from transition memory and highlights intermediate supervision as a promising direction for temporal composition.

We next addressed execution-memory growth. Merging similar adjacent projected tokens with the same rule during training and inference caps execution memory at 200 tokens. This retains 84.00% success on RoboMME-10x versus 87.50% without compression, a decrease of 3.50 percentage points, while reducing mean transition-token count by at least 38.26% on profiled training queries. These results demonstrate a trade-off between bounded context and manipulation performance. Cache refresh costs must also be considered when translating token-count reductions into runtime gains.

Finally, real-world experiments on a repeated pick-and-place counting task improve average success from 11.25% with memory-free \pi_{0.5} to 82.50% with LTM-\pi. Together, these results establish compact latent transitions as an effective form of episodic memory for the evaluated history-dependent manipulation tasks. Improving temporal composition and preserving informative event boundaries are promising directions for further development.

References

1. Black et al. (2024) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky \pi_{0}: A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164. External Links: Document, Link
2. Brohan et al. (2023) A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15818. External Links: Document, Link
3. Bruce et al. (2024) J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel Genie: Generative Interactive Environments. arXiv preprint arXiv:2402.15391. External Links: Document, Link
4. Bu et al. (2025) Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li UniVLA: Learning to Act Anywhere with Task-centric Latent Actions. arXiv preprint arXiv:2505.06111. External Links: Document, Link
5. Cherepanov et al. (2026) E. Cherepanov, N. Kachaev, D. Zelezetsky, A. Bulatov, A. Pshenitsyn, Y. Kuratov, A. Skrynnik, A. I. Panov, and A. K. Kovalev \muVLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models. arXiv preprint arXiv:2606.12497. External Links: Document, Link
6. Choi et al. (2026) S. Choi, J. Jung, S. Kim, Y. Lee, and Y. Yu PonderPounce: a pretrained mllm as an episode context engine for robot control. arXiv preprint arXiv:2608.24115. External Links: Document, Link
7. Chung et al. (2026) N. Chung, T. Hanyu, T. Nguyen, H. Le, F. Bumgarner, D. M. H. Nguyen, K. Vo, K. Yamazaki, C. Rainwater, T. Kieu, A. Nguyen, and N. Le Rethinking progression of memory state in robotic manipulation: an object-centric perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, pp. 3407–3415. External Links: Document
8. Dai et al. (2026) Y. Dai, H. Fu, J. Lee, Y. Liu, H. Zhang, J. Yang, C. Finn, N. Fazeli, and J. Chai RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies. arXiv preprint arXiv:2603.04639. External Links: Document, Link
9. Fang et al. (2025) H. Fang, M. Grotz, W. Pumacay, Y. R. Wang, D. Fox, R. Krishna, and J. Duan SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, pp. 15925–15942. External Links: Link
10. Gao et al. (2026a) S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W. Tseng, Y. Dong, K. Mo, C. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. R. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M. Liu, Y. Zhu, J. Jang, and L. ". Fan DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos. arXiv preprint arXiv:2602.06949. External Links: Document, Link
11. Gao et al. (2026b) Y. Gao, J. J. Liu, S. Li, and S. Song Gated memory policy: in-context memorization and adaptation. arXiv preprint arXiv:2604.18933. External Links: Link
12. Hanyu et al. (2025) T. Hanyu, N. Chung, H. Le, T. Nguyen, Y. Ikebe, A. Gunderman, Duy Nguyen Ho Minh, K. Vo, T. Kieu, K. Yamazaki, C. Rainwater, A. Nguyen, and N. Le SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulation. arXiv preprint arXiv:2511.06754. External Links: Document, Link
13. Hu et al. (2026) Y. Hu, F. Chen, X. Cao, and R. Detry 2AM: grounding agent-side memory as guidance for steerable action models in long-horizon manipulation. arXiv preprint arXiv:2609.11308. External Links: Link
14. Ji et al. (2026) L. Ji, S. Wang, P. Qian, X. Yu, Y. Tian, Z. Fei, J. Gong, and X. Qiu HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control. arXiv preprint arXiv:2607.03449. External Links: Document, Link
15. Jiang et al. (2026) Y. Jiang, C. Yu, Y. Chen, X. Feng, Y. Yang, N. Gite, and C. Jiang MemoryVAM: integrating memory into video action model for robot manipulation. arXiv preprint arXiv:2606.20679. External Links: Link
16. Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn OpenVLA: An Open-Source Vision-Language-Action Model. arXiv preprint arXiv:2406.09246. External Links: Document, Link
17. Koo et al. (2025) M. Koo, D. Choi, T. Kim, K. Lee, C. Kim, Y. Seo, and J. Shin HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy. arXiv preprint arXiv:2510.00695. External Links: Document, Link
18. Li et al. (2023) X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y. Jing, W. Zhang, H. Liu, H. Li, and T. Kong Vision-Language Foundation Models as Effective Robot Imitators. arXiv preprint arXiv:2311.01378. External Links: Document, Link
19. Liang et al. (2025) A. Liang, P. Czempin, M. M. Hong, Y. Zhou, J. Wang, E. Biyik, and S. Tu CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations. arXiv preprint arXiv:2505.04999. External Links: Document, Link
20. Lin et al. (2025) M. Lin, P. Ding, S. Wang, Z. Zhuang, Y. Liu, X. Tong, W. Song, S. Lyu, S. Huang, and D. Wang HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Models. arXiv preprint arXiv:2512.09928. External Links: Document, Link
21. Liu et al. (2026) Y. Liu, H. Huang, Y. Hong, Z. Du, Z. Cao, F. Li, and J. Wu T{}^{2}Mem: learning test-time memory for robotics. arXiv preprint arXiv:2609.36720. External Links: Link
22. Physical Intelligence et al. (2025) Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky \pi_{0.5}: a Vision-Language-Action Model with Open-World Generalization. arXiv preprint arXiv:2504.16054. External Links: Document, Link
23. Qu et al. (2026) H. Qu, J. Gao, X. Hu, S. Yang, X. Yu, R. Yan, W. Wang, X. Shu, and S. Yan Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation. arXiv preprint arXiv:2607.07608. External Links: Document, Link
24. Shi et al. (2026) H. Shi, W. Li, B. Xie, Y. Wang, R. Zhou, T. Wang, X. Zhang, P. Luo, and G. Huang MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models. arXiv preprint arXiv:2606.09827. External Links: Document, Link
25. Shi et al. (2025) H. Shi, B. Xie, Y. Liu, L. Sun, F. Liu, T. Wang, E. Zhou, H. Fan, X. Zhang, and G. Huang MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation. arXiv preprint arXiv:2508.19236. External Links: Document, Link
26. Sridhar et al. (2025) A. Sridhar, J. Pan, S. Sharma, and C. Finn MemER: Scaling Up Memory for Robot Control via Experience Retrieval. arXiv preprint arXiv:2510.20328. External Links: Document, Link
27. Torne et al. (2026) M. Torne, K. Pertsch, H. Walke, K. Vedder, S. Nair, B. Ichter, A. Z. Ren, H. Wang, J. Tang, K. Stachowicz, K. Dhabalia, M. Equi, Q. Vuong, J. T. Springenberg, S. Levine, C. Finn, and D. Driess MEM: Multi-Scale Embodied Memory for Vision Language Action Models. arXiv preprint arXiv:2603.03596. External Links: Document, Link
28. Wei et al. (2026) Y. Wei, K. Zhou, L. Mao, Z. Xu, S. Liang, Z. Zhang, Z. Xi, R. Han, Y. Yan, X. Wang, F. Feng, and B. Huang Causally Debiased Latent Action Model for Embodied Action-Conditioned World Models. arXiv preprint arXiv:2607.09185. External Links: Document, Link
29. Wu et al. (2026) W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, F. Wang, Q. Zhu, H. Sun, Y. Wang, S. Ma, Y. Ren, K. Zhang, H. Yu, J. Zhao, S. Zhou, Z. Qiu, H. Xiong, Z. Wang, Z. Wang, R. Cheng, Y. Li, Y. Huang, X. Zhu, Y. Shen, and K. Zheng A Pragmatic VLA Foundation Model. arXiv preprint arXiv:2601.18692. External Links: Document, Link
30. Ye et al. (2024) S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y. Chao, B. Y. Lin, L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo Latent Action Pretraining from Videos. arXiv preprint arXiv:2410.11758. External Links: Document, Link
31. Yin et al. (2026) C. Yin, W. Xu, J. Yang, S. Tam, H. Liu, Y. Yao, X. Zeng, J. Cui, Y. Wang, Z. Yin, and Y. Lin SimpleMemVLA: a simple but effective native-video memory for vision-language-action models. arXiv preprint arXiv:2609.05533. External Links: Link
32. Zhu et al. (2026) S. Zhu, Z. Liu, F. Wang, J. Wang, B. Yue, G. Liu, S. Wu, X. Xue, and T. Zeng WeaveLA: Event Driven Cross-Subtask Latent Memory Weaving for Repetitive Robot Manipulation. arXiv preprint arXiv:2606.17463. External Links: Document, Link

Appendix A Cup Swap Diagnostic Experiment Details

Task and dataset.

We construct a synthetic Cup Swap task to isolate motion-memory and temporal state-tracking ability from robotic control. Each video contains six cups and between one and five balls, with the initial balls placed under distinct cups. The initial configuration is revealed at frame 42, after which the sequence contains 5–20 randomly generated swaps. Videos are rendered at 30 FPS. The dataset contains 14,000 videos, split into 11,176 training, 1,375 validation, and 1,449 test examples using a fixed random seed of 42. The task is to recover the final ball occupancy after composing the complete swap sequence.

Latent-transition extraction.

We use the same frozen encoder family as in LTM-\pi. Each adjacent frame pair is encoded into a 32-dimensional latent transition with stride 1. To prevent the reveal event itself from leaking the initial state into the motion representation, we retain only transitions whose source frame is at or after the reveal frame. Letting r=42 denote the reveal-frame index, we keep a transition z_{i} only when its source-frame index satisfies \operatorname{src}(z_{i})\geq r. Thus, the latent stream excludes the reveal transition; the initial occupancy is supplied separately as text.

Input and latent-memory fusion.

To isolate motion reasoning from visual recognition of the reveal frame, the initial cup occupancy is provided as text (reveal_mode=text). The base model is Qwen3-VL-2B-Instruct. Every four consecutive latents are compressed into one memory token and directly projected to the Qwen hidden dimension. We add sinusoidal temporal embeddings, process the latent stream with 8 layers of latent self-attention (16 heads), and let the language states read the latent memory through 8 layers of cross-attention (16 heads). Attention dropout is set to zero. Training uses bfloat16 and gradient checkpointing.

Supervision variants.

We distinguish two objectives. Vanilla (final-state only) uses a cup-occupancy classification target and directly predicts whether each of the six cups contains a ball; it receives no supervision on the swap sequence or intermediate states. Process supervision (CoT-like) uses a process-query-tracking objective that supervises recovery of the swap process and the evolving state before predicting the final state. Although this objective does not generate a natural-language chain of thought, it explicitly supervises intermediate reasoning and is therefore CoT-like in function.

Training configuration.

For the process-supervised run, we train on 8 H100 80GB GPUs with a per-GPU batch size of 4 (global batch size 32). The main learning rate is 1\times 10^{-4}, the classification-head learning rate is 3\times 10^{-4}, weight decay is 1\times 10^{-2}, and gradient clipping is 1.0. The Qwen backbone is frozen.

Additional metrics.

For the process-supervised model, token accuracy is 99.67%, swap accuracy is 99.78%, swap-count accuracy is 100.00%, final-state accuracy is 97.93%, and full exact accuracy is 97.52%. For the Vanilla six-cup objective, the best exact accuracy is 2.40% (1.75% at the final checkpoint), with training exact accuracy of approximately 12.5%. A separate, substantially simpler three-cup, single-ball, fixed-five-swap classifier reaches 99.40% exact accuracy without process supervision, but we do not compare it directly because its state space and task difficulty are much smaller.

Appendix B Real-World Evaluation

Raw Results.

We report the actual number of objects transferred into the packaging bag in every trial in Table 10. A trial is counted as correct if this number exactly matches the instructed quantity x, and the robot subsequently returns to its initial pose and remains stationary for at least 10 s. Each model is evaluated for 20 trials per target quantity, except the memory-free \pi_{0.5} at x=3 and x=4: we conducted only 10 trials each because the policy repeatedly failed to stop after the requested number of transfers and continued picking, making individual trials excessively long. Unperformed trials are marked as “–” and excluded from the accuracy denominator.

Representative Real-Robot Videos.

Representative videos of our real-robot experiments, including successful LTM-\pi pick-and-place trials and a representative failure of the memory-free \pi_{0.5} baseline, are available in our project repository: https://github.com/buaa-colalab/LTM-pi.