Physical Time as a Missing Dimension of Robot Policy Memory Dynamics

Abstract

Existing robot policy memories typically evolve with observation sequences, implicitly treating each update as the same amount of physical time. Yet identical observation histories can span different durations and require different actions, exposing physical time as a missing dimension of policy memory. We introduce Temporal Flow Policy (TFP), which learns memory dynamics to retain past events and evolve the memory state over measured time intervals. Observations and prior memory determine the content and time scales of each transition; elapsed time determines how far the state advances. TFP conditions diffusion and flow-matching policies through this memory state and is trained with their native imitation objectives. We evaluate TFP with Diffusion Policy and \pi_{0.5} through controlled comparisons of memory designs. These experiments show that successful timing depends on both measured intervals and retained history. TFP achieves higher average success than the recurrent and state-space memories we equip with the same intervals and than a variant that feeds the interval to a learned gate. Memory readouts recover the time since an event and motion information after occlusion. Changing the supplied intervals systematically delays or advances actions, linking time inputs to behavior. Real-robot tasks requiring timing or memory, together with LIBERO evaluations, demonstrate benefits in physical and general manipulation. These results suggest viewing robot memory not as stored observation history, but as a state that retains past events and evolves with physical time.

1 Introduction

Consider a robot preparing tea. After it lowers an infuser into a cup, the scene may remain nearly unchanged, yet the correct action changes from waiting to lifting once the tea has steeped long enough. The robot needs to remember when steeping began and track how much time has passed. A robot picking up a cup from a moving conveyor faces a related challenge. It must estimate the cup’s velocity from earlier observations and retain that information while the cup is briefly hidden from view. The same observed displacement over different time intervals implies different speeds, requiring the robot to match the cup’s speed at contact. Policy memory must therefore use elapsed time both to respond to remembered events and to interpret motion from past observations (Figure 1).

Memory-augmented robot policies, including vision-language-action (VLA) models, retain earlier cues and track completed subtasks, allowing actions to depend on information beyond the current observation (Koo et al., 2026; Shi et al., 2026; Li et al., 2026). Yet memories driven only by observation sequences treat each observation as one step: the same sequence can span different physical durations and require different actions. This mismatch becomes apparent when observation intervals vary with observation rate or inference latency. Timestamped histories make recording times available to the policy (Yin et al., 2026). Continuous-time recurrent models provide a mechanism for evolving state over measured intervals (Hasani et al., 2021; Hasani et al., 2022). We therefore study physical time as a missing dimension of robot policy memory: how remembered events and measured intervals jointly shape memory evolution and guide action.

We introduce Temporal Flow Policy (TFP), which retains past events in a persistent memory state that evolves with physical time. At each query, separate heads use current observation features and previous memory to predict a candidate state and channel-wise time constants. A closed-form exponential update advances memory toward the candidate over the measured interval, so elapsed time directly governs state evolution. Prior memory shapes both predictions after visual cues disappear, while adaptive timescales support rapid updates around events and slower evolution during waiting. The updated memory conditions diffusion and flow-matching policies (Chi et al., 2023; Physical Intelligence et al., 2025) through adaptive normalization, alongside their native observation and task context. We jointly learn memory dynamics and conditioning from causal demonstration histories using each backbone’s native imitation objective, without event-time labels.

Controlled comparisons with Diffusion Policy and \pi_{0.5} show higher average success than recurrent and state-space memories supplied with the same intervals and a learned-gate alternative. Memory readouts recover event age and motion information, while scaling the supplied intervals delays or advances actions without changing the physical execution schedule. Real-robot timing and sequencing tasks and LIBERO demonstrate benefits in physical and general manipulation.

Our contributions are:

• We identify the ambiguity between observation order and elapsed physical time as a representation problem for robot policy memory.
• We introduce TFP, which uses learned event-dependent memory dynamics over measured intervals to condition diffusion and flow-matching policies.
• We connect temporal memory to behavior through time-aware controls, memory readouts, and clock interventions, and evaluate its utility in real-robot and benchmark tasks.

2 Related Work

Memory in robot policies.

Robot policies use history to recover information absent from the current observation, including earlier cues and completed subtasks. HAMLET aggregates compact moment tokens into memory features (Koo et al., 2026), while ReMem-VLA carries context through frame- and chunk-level recurrent queries (Li et al., 2026). MemoryVLA retrieves perceptual and cognitive representations from a consolidated memory bank (Shi et al., 2026), and MEM combines short-term visual history with long-term language memory (Torne et al., 2026). MemoryVLA uses episode timesteps to locate memory entries, and MEM evaluates duration-sensitive cooking behavior. These systems encode temporal context or evaluate timing behavior, but do not specify how elapsed physical time should shape memory evolution.

Temporal information for robot control.

The role of time depends on how it enters the policy. SimpleMemVLA timestamps visual history relative to the current decision (Yin et al., 2026). DAM-VLA maintains modality-specific latent buffers refreshed at different sensor rates, decoupling action generation from individual encoding schedules (Vanjani et al., 2026). Time as a Control Dimension conditions policies on remaining time and a time ratio that regulates the progression of an internal clock (Jia & Chen, 2026). These approaches make time available through history, sensing schedules, or control conditions, but leave open how elapsed time should enter memory updates: as an input feature or an interval of state evolution.

Continuous-time dynamics in policy memory.

Continuous-time recurrent models provide tools for modeling state evolution over elapsed time. LTC learns adaptive time constants (Hasani et al., 2021), while CfC provides explicit time-dependent updates without numerical ODE solvers (Hasani et al., 2022). S4D discretizes continuous-time linear state-space dynamics (Gu et al., 2022b), while Mamba learns input-dependent discretization steps for selection (Gu & Dao, 2023). M-CfC augments CfC with a gated memory pathway to support long-horizon object-goal navigation (Chen et al., 2025). TFP combines measured physical intervals with event-dependent memory dynamics to condition diffusion and flow-matching action generators.

3 Method

3.1 Memory for Control in Physical Time

TFP combines past-event evidence and measured intervals in a persistent memory state that conditions the action generator after events leave view (Figure 2).

At query n, the backbone encodes observation o_{n} and task specification g into compact features \mathbf{x}_{n} and its native action context C_{n}. The memory summarizes the timed history, while C_{n} provides direct access to the current scene and task:

\displaystyle\mathbf{h}_{n} \\ \displaystyle=\mathcal{U}_{\theta}(\mathbf{h}_{n-1},\mathbf{x}_{n},\Delta_{n}), \\ \displaystyle\mathbf{A}_{n} \\ \displaystyle\sim\pi_{\psi,b}(\,\cdot\mid C_{n},\mathbf{h}_{n}).

Here \mathbf{h}_{n}\in\mathbb{R}^{d_{h}} is a persistent state, \mathbf{A}_{n} is an action chunk, and b denotes the policy backbone. For observations captured at timestamps t_{n}, \Delta_{n}=t_{n}-t_{n-1} for n\geq 1. The memory state must infer how much time has passed since an event from the preceding observations and intervals. Memory is reset at episode boundaries; the first-query convention is specified in Appendix A.2.

3.2 Memory Dynamics in Physical Time

Remembering past events.

Following continuous-time recurrent models (Hasani et al., 2021; Hasani et al., 2022), TFP predicts a candidate state and positive, channel-wise time constants from the current features and previous memory:

\displaystyle\widehat{\mathbf{h}}_{n} \\ \displaystyle=\tanh\!\left(f_{\theta}^{h}([\mathbf{x}_{n};\mathbf{h}_{n-1}])\right), \\ \displaystyle\mathbf{\tau}_{n} \\ \displaystyle=f_{\theta}^{\tau}([\mathbf{x}_{n};\mathbf{h}_{n-1}])>\mathbf{0}.

TFP shares a local relaxation form with canonical LTC and parameterizes candidate content and time scales through separate heads. Canonical LTC couples these quantities through a shared drive (Appendix A.3). The separate heads let the model learn what memory should represent and how quickly it changes, with past events shaping both predictions through the prior state.

Evolving with elapsed time.

Elapsed time sets how far memory moves toward the candidate:

\displaystyle\mathbf{k}_{n} \\ \displaystyle=\exp(-\Delta_{n}/\mathbf{\tau}_{n}), \\ \displaystyle\mathbf{h}_{n} \\ \displaystyle=\mathbf{k}_{n}\odot\mathbf{h}_{n-1}+(1-\mathbf{k}_{n})\odot\widehat{\mathbf{h}}_{n}.

Operations are elementwise; \mathbf{\tau}_{n} and \Delta_{n} share physical units.

Unlike memories that receive elapsed time only as an additional feature, TFP uses it as the integration interval of memory evolution. At fixed observation and prior memory, \Delta_{n} sets this interval without changing the predicted candidate or time constants.

Local physical-time dynamics.

The transition is the exact solution of a local relaxation process,

\frac{d\mathbf{y}(u)}{du}=\frac{\widehat{\mathbf{h}}_{n}-\mathbf{y}(u)}{\mathbf{\tau}_{n}},\qquad\mathbf{y}(0)=\mathbf{h}_{n-1}.

Fixing the predicted coefficients, the transition at u=\Delta_{n} preserves the state at zero duration, approaches the candidate monotonically, and composes across intervals (Appendix B.1). A generic learned gate need not satisfy these constraints. The coefficients are recomputed at each query.

Under these fixed coefficients, a channel displaced from equilibrium provides a monotonic temporal trace, with sensitivity decreasing toward saturation. Different time constants provide complementary temporal responses, allowing different memory channels to retain task-relevant events over different physical timescales. Together, these channels form a multi-timescale temporal basis for the memory state, echoing exponential traces studied in neuroscientific models of temporal representation (Shankar & Howard, 2012). TFP’s adaptive rates change with events and interaction phases, supporting rapid updates at event onset and slower evolution during waiting.

The exponential update specifies how a given memory transition depends on elapsed time, including intervals absent from training. With the candidate and time constants held fixed, changing the interval changes the extent of relaxation through the same analytic rule. This provides a temporal inductive bias for transferring from a single training cadence to different observation schedules, which we evaluate through the controlled comparisons in Section 4.1.

3.3 Acting on Memory

Memory conditions action generation through adaptive normalization (Peebles & Xie, 2023; Perez et al., 2018). A learned projection M_{b} augments the native decoder condition:

\mathbf{c}_{n,s}=\mathbf{c}^{\mathrm{native}}_{n,s}+M_{b}(\mathbf{h}_{n}).

Here s is diffusion or flow solver time; the condition modulates normalized decoder activations.

Memory summarizes visual and robot-state features for both backbones. With Diffusion Policy (Chi et al., 2023), it augments the denoiser’s adaptive layer-normalization condition. For \pi_{0.5} (Physical Intelligence et al., 2025), it additionally summarizes language in controlled timing tasks, while LIBERO retains language in the backbone context; both variants augment the action expert’s adaptive RMS-normalization condition. Memory updates once per query and stays fixed during solver steps. Persistent state requires \mathcal{O}(d_{h}) storage per episode, independent of episode length. The prediction heads, state update, and conditioning projection together cost \mathcal{O}(d_{h}(d_{x}+d_{h}+d_{c})) per query, where d_{x} and d_{c} are feature and conditioning widths (Appendix C.2).

3.4 Learning Memory Dynamics from Demonstrations

The backbone’s native imitation objective jointly trains memory dynamics and the conditioning projection from demonstrated actions. Each action loss uses the recurrent state computed from the causal observation history and its measured intervals. Controlled timing models reconstruct memory from causal prefixes; LIBERO uses consecutive episode segments with gradients truncated between segments. Appendices A.1 and D.1 detail sequence training and task protocols.

4 Experiments

We test control across observation schedules, connect temporal memory to action timing, and evaluate real-robot and general manipulation (Figure 3).

4.1 Does Policy Memory Need Physical Time?

Separating duration from observation count.

Both Diffusion Policy and \pi_{0.5} models are trained with fixed 0.1 s observation intervals, without mixed-cadence sampling augmentation. Time-aware models receive the measured observation interval: it is 0.1 s during training and follows the actual observation schedule at evaluation. Each trained model is evaluated at 10 Hz, 5 Hz, and under jitter and dropout, with actions continuing between queries. Task durations and target speeds remain fixed, so equal physical durations can span different observation counts.

Timed Button Press requires continuous contact for 3.0–3.5 seconds before release and retraction. Delayed Passage presents a brief cue at a randomized onset and requires crossing 6–9 seconds later. Interception requires matching a previously observed target’s velocity after occlusion; the robot moves only after the target disappears. These tasks test action duration, event age, and motion estimation. Time since contact, cue timestamps, and target velocity are not policy inputs.

Evaluation.

We evaluate Diffusion Policy (Chi et al., 2023) and \pi_{0.5} (Physical Intelligence et al., 2025) with reactive and short-history controls, and GRU (Cho et al., 2014) and S4D (Gu et al., 2022b) both with and without measured intervals. We also evaluate LTC (Hasani et al., 2021) with measured intervals for both backbones. The +\Delta t suffix denotes measured intervals. We report mean success across three training seeds. Per seed and schedule, we evaluate 50 trials each for Button Press and Passage and 300 direction–speed cases for Interception, paired across methods within each backbone. Appendices D.1 and E.1 give protocols and task-average sample SDs.

Memory improves control.

Reactive policies fail Button Press under every schedule with both backbones (Table 1). Short history improves Interception but achieves at most 4.7% on Passage, where recurrent memory must retain the cue throughout the waiting interval.

Averaged equally across tasks and schedules, TFP achieves the highest reported success: 77.5% with Diffusion Policy versus 62.1% for the stronger time-aware control, S4D+\Delta t; with \pi_{0.5}, the corresponding rates are 73.3% and 66.8%. Passage performance remains high across schedules, supporting responses to remembered cues despite changes in observation cadence.

4.2 How Memory Controls Behavior

Frozen-weight interventions test how clock inputs and stored memory affect behavior; separately trained ablations assess architectural choices. Linear probes read out information from memory.

Action timing follows the supplied clock.

We scale the intervals supplied to trained policies while keeping weights and physical execution schedules fixed. Under jitter, halving the intervals delays button release and passage entry, whereas doubling them advances these actions (Figure 4). For \pi_{0.5} on Button Press and both backbones on Passage, each directional shift occurs in all 50 paired trials. Both scalings eliminate success on these tasks with either backbone (Appendix D.5).

Replacing measured intervals with 0.1 s reduces success, and clearing memory at every query eliminates success on all three tasks at 10 Hz (Appendix Table 7). Timing thus depends on both measured intervals and accumulated history.

Elapsed event time is recoverable from memory.

Linear probes recover event duration and motion from frozen Diffusion Policy’s TFP memory. For Passage, predicting time since the cue from memory reduces mean absolute error from about 0.83 s using episode time alone to 0.04 s, supporting event-relative timing beyond episode progress. After occlusion, memory also retains speed information beyond single-frame cues. Appendix D.6 details these comparisons and protocols.

Memory conditioning supports action generation.

Without memory conditioning throughout Diffusion Policy training and evaluation, success is 0.5% on Button Press and 3.7% on Passage, compared with the Full TFP means of 81.2% and 97.3%, respectively (Appendix D.3).

Three design choices for physical-time memory.

Separately trained ablations remove history from the candidate, use static time constants, or replace interval-based evolution with a learned \Delta t gate (Table 2). Each lowers mean success relative to Full TFP, supporting event-dependent content, adaptive timescales, and the update rule in Section 3.2. On Diffusion Policy Button and Passage, the learned gate achieves 14.5% and 15.0%, compared with 81.2% and 97.3% for TFP’s exponential update. Appendix D.2 gives additional ablations.

Figure 5 illustrates these dynamics around the passage cue. Different channels exhibit distinct temporal responses, with some rapidly adapting around events and others maintaining slower traces during waiting. Write gains, \mathbf{g}_{n}=\mathbf{1}-\mathbf{k}_{n}, show the resulting variation across channels. Appendix D.7 shows corresponding event-aligned visualizations for Button Press and Interception.

4.3 Does TFP Improve Real-Robot Control?

On a Galaxea A1, we compare TFP with each backbone’s standard reactive policy. Object swapping, counting pick-and-place, and water pouring require tracking action sequences. Tea steeping requires 3.0\pm 0.5 s from the infuser entering the water to its removal. Staged heating requires two such intervals: heater activation to food contact, then contact to heater deactivation. Success requires completing each sequence in order within its timing constraints. Each task uses 50 demonstrations and 20 evaluation trials per method. Event timestamps are annotated in evaluation videos with the same timing criteria for all methods.

TFP improves all five tasks, with mean gains of 47 and 50 percentage points for Diffusion Policy and \pi_{0.5}, respectively (Table 3). Both backbones gain 50 points on tea steeping and staged heating, extending the timing results to physical duration constraints. See Appendix D.10 for details.

4.4 Memory Beyond Explicit Timing Tasks

We evaluate \pi_{0.5} with TFP on LIBERO (Liu et al., 2023) and visually perturbed LIBERO-plus (Fei et al., 2025) (Table 4). Reproduced memory variants share demonstrations, execution settings, and evaluation protocols. Standard LIBERO uses three training seeds and 50 rollouts per task and seed.

TFP achieves 98.7% mean LIBERO success, compared with the reported reactive baseline of 96.9%. The largest gap is on Long-10: 96.9% versus 92.4% (+4.5 points), compared with gaps of 0.7–1.3 points on the other suites. On LIBERO-plus, success rises from 91.4% to 93.7%, leading reproduced variants in every condition. Appendices D.8 and E give variant settings and rollout uncertainty.

5 Conclusion

We identify physical time as a missing dimension of robot policy memory driven by observation sequences. TFP retains past events in a memory state that evolves over measured time intervals. Observations and history determine the content and timescales of the transition, while measured intervals determine how far the memory state advances. This state conditions existing diffusion and flow-matching action generators.

Comparisons with recurrent and state-space memories supplied with the same measured intervals support using elapsed time to govern memory evolution. Memory readouts recover elapsed event time beyond episode progress and motion information after occlusion. Clock-scaling interventions systematically advance or delay button release and passage entry, while memory resets and conditioning ablations support the role of accumulated history in control. Real-robot tea steeping and staged heating extend the evidence to physical duration constraints; memory-dependent tasks and LIBERO show broader benefits in manipulation. When actions depend on how long ago an event occurred, memory dynamics should follow physical time.

References

1. Che et al. (2018) Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1):6085, 2018. doi: 10.1038/s41598-018-24271-9. URL https://www.nature.com/articles/s41598-018-24271-9.
2. Chen et al. (2025) Zifan Chen, Jiahong Zhang, Sijun Shen, Wenlong Fu, and Lihong Cao. Memory-augmented continuous-time neural policy for vision-guided embodied navigation. In Proceedings of the 7th ACM International Conference on Multimedia in Asia, pp. 1–8. Association for Computing Machinery, 2025. doi: 10.1145/3769748.3773343. URL https://doi.org/10.1145/3769748.3773343.
3. Chi et al. (2023) Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin CM Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 10.15607/RSS.2023.XIX.026. URL https://roboticsproceedings.org/rss19/p026.html.
4. Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Çağlar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1724–1734, 2014.
5. Fei et al. (2025) Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. LIBERO-Plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626, 2025.
6. Gu & Dao (2023) Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
7. Gu et al. (2022a) Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022a.
8. Gu et al. (2022b) Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022b. URL https://arxiv.org/abs/2206.11893.
9. Hasani et al. (2021) Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7657–7666, 2021. doi: 10.1609/aaai.v35i9.16936. URL https://ojs.aaai.org/index.php/AAAI/article/view/16936.
10. Hasani et al. (2022) Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, and Daniela Rus. Closed-form continuous-time neural networks. Nature Machine Intelligence, 4(11):992–1003, 2022. doi: 10.1038/s42256-022-00556-7.
11. Jia & Chen (2026) Yinsen Jia and Boyuan Chen. Time as a control dimension in robot learning, 2026. URL https://arxiv.org/abs/2511.07654v2. Version 2; first posted in 2025.
12. Koo et al. (2026) Myungkyu Koo, Daewon Choi, Taeyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, and Jinwoo Shin. HAMLET: Switch your vision-language-action model into a history-aware policy. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2510.00695.
13. Li et al. (2026) Hang Li, Fengyi Shen, Dong Chen, Liudi Yang, Xudong Wang, Jinkui Shi, Zhenshan Bing, Ziyuan Liu, and Alois Knoll. ReMem-VLA: Empowering vision-language-action model with memory via dual-level recurrent queries. arXiv preprint arXiv:2603.12942, 2026. doi: 10.48550/arXiv.2603.12942. URL https://arxiv.org/abs/2603.12942.
14. Liu et al. (2023) Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023. doi: 10.48550/arXiv.2306.03310. URL https://arxiv.org/abs/2306.03310.
15. Peebles & Xie (2023) William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195–4205, October 2023.
16. Perez et al. (2018) Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, 2018.
17. Physical Intelligence et al. (2025) Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. \pi_{0.5}: A vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054.
18. Rubanova et al. (2019) Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent ODEs for irregularly-sampled time series. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://arxiv.org/abs/1907.03907.
19. Shankar & Howard (2012) Karthik H. Shankar and Marc W. Howard. A scale-invariant internal representation of time. Neural Computation, 24(1):134–193, 2012. doi: 10.1162/NECO_a_00212. URL https://doi.org/10.1162/NECO_a_00212.
20. Shi et al. (2026) Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. MemoryVLA: Perceptual-cognitive memory in vision-language-action models for robotic manipulation. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2508.19236.
21. Torne et al. (2026) Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dhabalia, Michael Equi, Quan Vuong, Jost Tobias Springenberg, Sergey Levine, Chelsea Finn, and Danny Driess. MEM: Multi-scale embodied memory for vision language action models. arXiv preprint arXiv:2603.03596, 2026. doi: 10.48550/arXiv.2603.03596. URL https://arxiv.org/abs/2603.03596.
22. Vanjani et al. (2026) Pankhuri Vanjani, Zhuoyue Li, Jakub Suliga, Moritz Reuss, Gianluca Geraci, Xinkai Jiang, and Rudolf Lioutikov. DAM-VLA: Decoupled asynchronous multimodal vision language action model, 2026. URL https://arxiv.org/abs/2606.12105.
23. Xiao et al. (2026) Lei Xiao, Jifeng Li, Juntao Gao, Feiyang Ye, Yan Jin, Jingjing Qian, Jing Zhang, Yong Wu, and Xiaoyuan Yu. AVA-VLA: Improving vision-language-action models with active visual attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13453–13463, June 2026. URL https://openaccess.thecvf.com/content/CVPR2026/html/Xiao_AVA-VLA_Improving_Vision-Language-Action_models_with_Active_Visual_Attention_CVPR_2026_paper.html.
24. Yin et al. (2026) Cheng Yin, Wang Xu, Junpeng Yang, Sikyuen Tam, Hanyu Liu, Yuan Yao, Xiangrui Zeng, Junbo Cui, Yequan Wang, Zhouping Yin, and Yankai Lin. SimpleMemVLA: A simple but effective native-video memory for vision-language-action models, 2026. URL https://arxiv.org/abs/2609.05533.

Appendix A Additional Method Details

A.1 Sequence Training

The controlled timing experiments reconstruct memory from causal prefixes. For \pi_{0.5}, the recurrence starts from zero and gradients flow through the full prefix to each of four supervised queries. Padded observations leave memory unchanged and padded actions are excluded from the loss. LIBERO instead uses consecutive episode segments with memory carried forward and gradients truncated between segments, as detailed below.

For the LIBERO \pi_{0.5} instantiation, each episode e has a hidden state \mathbf{h}_{n}^{(e)}=\mathcal{U}_{\theta}(\mathbf{h}_{n-1}^{(e)},\mathbf{x}_{n}^{(e)},\Delta_{n}^{(e)}), initialized to \mathbf{h}_{-1}^{(e)}=\mathbf{0} and reset only at episode boundaries. At optimization step r, we sample B active episodes and unroll each for K consecutive queries. For episode e_{i}, let n_{i} be the segment’s first query index and k=0,\ldots,K-1 index queries within the segment, so that \mathbf{x}_{i,k}=\mathbf{x}_{n_{i}+k}^{(e_{i})} and \Delta_{i,k}=\Delta_{n_{i}+k}^{(e_{i})}. Writing \tilde{\mathbf{h}} for the stored state, we retrieve \mathbf{h}_{i,-1}=\tilde{\mathbf{h}}_{n_{i}-1}^{(e_{i})} and update

\mathbf{h}_{i,k}=\mathcal{U}_{\theta}(\mathbf{h}_{i,k-1},\mathbf{x}_{i,k},\Delta_{i,k}),\qquad k=0,\dots,K-1.

The optimization objective is

\mathcal{L}_{r}=\frac{1}{BK}\sum_{i=1}^{B}\sum_{k=0}^{K-1}\ell_{\mathrm{flow}}^{(i,k)}.

After the parameter update, the final hidden state is written back with gradient truncation,

\tilde{\mathbf{h}}_{n_{i}+K-1}^{(e_{i})}\leftarrow\operatorname{stopgrad}(\mathbf{h}_{i,K-1}).

This carries memory across training segments while bounding the backpropagation length.

A.2 Memory Initialization and Parameterization

Using zero-based query indices, memory is initialized as \mathbf{h}_{-1}=\mathbf{0} at each episode boundary. The controlled timing tasks use a bootstrap interval \Delta_{0}=0.1 s; subsequent updates use timestamp differences. In experimental labels, \Delta t denotes this elapsed-time input.

Diffusion Policy.

The memory has 128 channels. Separate heads predict a \tanh-bounded candidate and positive time constants using \operatorname{softplus} with a 10^{-6} floor. For the controlled Button and Passage recipes, time-constant head weights start at zero with the default linear-layer bias; Interception retains the default linear-layer initialization.

\pi_{0.5}.

The memory has 256 channels and a \tanh-bounded candidate. For the controlled timing tasks, a sigmoid maps time constants to [\tau_{\min},\tau_{\max}], set by the median query interval and the longest episode duration in the training data, respectively. The time-constant head has zero initial weights; its biases initialize the channel time constants geometrically from 1.01\tau_{\min} to \tau_{\max}/1.01. The memory-to-action projection starts with its default weights scaled by 0.02 and zero bias. These TFP implementations use the exponential update in Equation 3.

A.3 Relation to Canonical LTC Dynamics

Canonical LTC dynamics (Hasani et al., 2021) take the form

\frac{d\mathbf{h}}{du}=-\mathbf{\tau}_{0}^{-1}\odot\mathbf{h}+\mathbf{f}_{\theta}(\mathbf{h},\mathbf{x})\odot(\mathbf{a}-\mathbf{h}),

with positive baseline time constants and a nonnegative drive. Writing

\mathbf{\lambda}=\mathbf{\tau}_{0}^{-1}+\mathbf{f}_{\theta},\qquad\mathbf{\mu}=\frac{\mathbf{f}_{\theta}\odot\mathbf{a}}{\mathbf{\lambda}},

gives the relaxation form

\frac{d\mathbf{h}}{du}=\mathbf{\lambda}\odot(\mathbf{\mu}-\mathbf{h}).

For frozen coefficients its exact update is

\mathbf{h}^{+}=e^{-\delta\mathbf{\lambda}}\odot\mathbf{h}^{-}+(1-e^{-\delta\mathbf{\lambda}})\odot\mathbf{\mu}.

The interpolation has the same form as TFP when \mathbf{\tau}_{n}=\mathbf{\lambda}^{-1} and \widehat{\mathbf{h}}_{n}=\mathbf{\mu}. However, canonical LTC couples these quantities through the same drive, whereas TFP predicts them independently. TFP shares this local relaxation form, with separate heads for candidate content and time constants.

As in CfC (Hasani et al., 2022), content and temporal modulation depend on inputs and prior state. TFP uses this dependence to retain task-relevant events in persistent memory, evolve that memory over measured physical intervals, and condition diffusion and flow-matching action generators.

Appendix B Properties of the Memory Update

The memory dynamics in Equation 3 admit a local relaxation interpretation. We analyze the update with its predicted candidate and time constants fixed over the interval of a single policy query.

B.1 Local Relaxation and Interval Composition

For fixed candidate \mathbf{\mu}, positive rate \mathbf{\lambda}, and initial state \mathbf{h}, the solution of d\mathbf{y}/du=\mathbf{\lambda}\odot(\mathbf{\mu}-\mathbf{y}) is

\Phi_{\delta}(\mathbf{h})=\mathbf{\mu}+e^{-\delta\mathbf{\lambda}}\odot(\mathbf{h}-\mathbf{\mu}).

Setting \mathbf{\mu}=\widehat{\mathbf{h}}_{n} and \mathbf{\lambda}=\mathbf{\tau}_{n}^{-1} recovers TFP’s update. At zero duration, \Phi_{0}(\mathbf{h})=\mathbf{h}. For \delta\geq 0, each coordinate interpolates between the initial state and the candidate, with distance

\left|[\Phi_{\delta}(\mathbf{h})]_{j}-\mu_{j}\right|=e^{-\delta\lambda_{j}}|h_{j}-\mu_{j}|.

Since \lambda_{j}>0, this distance decreases monotonically with duration unless h_{j}=\mu_{j}, in which case the coordinate remains unchanged. Direct substitution also gives

\Phi_{\delta_{2}}(\Phi_{\delta_{1}}(\mathbf{h}))=\Phi_{\delta_{1}+\delta_{2}}(\mathbf{h}).

Splitting an interval leaves the result unchanged if both parts use the same candidate and rate.

For scalar retention, the exponential form follows from this composition condition. If a continuous function k:[0,\infty)\rightarrow(0,1] satisfies k(0)=1 and k(\delta_{1}+\delta_{2})=k(\delta_{1})k(\delta_{2}), then \log k is continuous and additive, so \log k(\delta)=-\lambda\delta for some \lambda\geq 0. Consequently, k(\delta)=e^{-\lambda\delta}.

TFP recomputes the candidate and rate from the observation and memory at each query, then holds them fixed over the local integration interval.

Temporal inductive bias and unseen intervals.

Identity at zero duration, monotonic approach to a candidate, and interval composition constrain how elapsed time affects the transition. A generic learned gate that receives elapsed time as a feature does not structurally enforce these properties and must learn its temporal response from data. For a fixed candidate and positive time constants, TFP’s transition remains defined for every nonnegative interval, including intervals outside the training range. Each coordinate stays between its initial value and its candidate, and converges to the candidate as the interval tends to infinity. This specifies the transition’s behavior beyond observed intervals without requiring the model to learn those constraints from examples at every duration. Table 2 compares this exponential update with a learned temporal gate.

B.2 Duration Sensitivity

Differentiating Equation 13 with respect to the interval gives

\frac{\partial\Phi_{\delta}(\mathbf{h})}{\partial\delta}=\mathbf{\lambda}\odot e^{-\delta\mathbf{\lambda}}\odot(\mathbf{\mu}-\mathbf{h}).

The state changes with duration when it differs from the candidate, with the rate controlling how quickly that difference decays. At equilibrium, \mathbf{h}=\mathbf{\mu}, duration sensitivity vanishes.

Appendix C Relation to Alternative Temporal Models

C.1 Recurrent and Continuous-Time Models

A GRU (Cho et al., 2014) or LSTM supplied with \Delta_{n} can learn duration-dependent transitions from temporal inputs. Without that input, identical observation sequences produce the same recurrent trajectory even if they span different durations. TFP assigns elapsed duration a specific role in the transition: it sets the integration interval of the learned relaxation dynamics.

Time-aware recurrent models also handle irregular observations. GRU-D uses elapsed time in learned input and hidden-state decays (Che et al., 2018), while ODE-RNNs and Latent ODEs integrate hidden dynamics over irregular observation intervals (Rubanova et al., 2019). These models establish that elapsed time can shape hidden-state evolution beyond serving as an additional input.

A continuous-time relaxation baseline can also use a constant learned rate,

\frac{d\mathbf{h}}{du}=\mathbf{\lambda}_{0}\odot(\mathbf{q}_{\theta}(\mathbf{h},\mathbf{x})-\mathbf{h}),\qquad\mathbf{\lambda}_{0}>\mathbf{0}.

Comparing this baseline with TFP tests the value of adapting the update rate to state and input while preserving history-dependent candidate content.

C.2 State-Space Models and Computational Cost

Continuous-time state-space models (Gu et al., 2022a) can likewise represent transitions that depend on elapsed time, and input-dependent state-space models provide additional adaptive mechanisms (Gu & Dao, 2023). Our manipulation comparison includes S4D alongside recurrent variants within the same action-conditioning setup (Appendix D.8). Mamba’s learned discretization step controls selection and is not itself a measured physical interval.

Evaluating the candidate and time-constant heads costs \mathcal{O}(d_{h}(d_{x}+d_{h})) per query, followed by an \mathcal{O}(d_{h}) elementwise update. Projecting the memory costs \mathcal{O}(d_{h}d_{c}), with additional adapter work depending on the decoder. Here d_{x} and d_{c} denote the observation-feature and decoder-conditioning widths, respectively. Each active episode requires \mathcal{O}(d_{h}) persistent state, independent of episode length. These costs are incurred at the existing policy queries.

Appendix D Additional Experimental Details

D.1 Controlled Timing Protocols

Model configurations.

Table 5 summarizes the controlled-task models, trained separately for each task. Diffusion Policy uses a compact visual architecture (Chi et al., 2023), trained from scratch: convolutional stages of 16, 32, and 64 channels, 3\times 3 pooling, and a 128-dimensional projection. Its denoiser has three Transformer blocks of width 128, four attention heads, and feed-forward width 512. The \pi_{0.5} models freeze the vision-language backbone and train the action expert, memory, and conditioning projections. They use one RGB camera and a continuous state token. Both backbones retain current-observation conditioning and keep memory fixed throughout each query’s solver steps.

Baseline memory implementations.

Recurrent alternatives replace the memory core while retaining each backbone’s observation projections and action-conditioning pathway. Their memory outputs have width 128 for Diffusion Policy and 256 for \pi_{0.5}.

For GRU+\Delta t, Diffusion Policy adds a learned affine projection of the interval in seconds to the observation features before the GRU update. The \pi_{0.5} implementation instead concatenates \log(1+\Delta t/(0.1\,\mathrm{s})) with the observation and recurrent features used by the reset/update gates and candidate. Plain GRU is trained separately without this explicit time input at training or evaluation.

S4D+\Delta t uses a diagonal complex-valued state-space recurrence with zero-order-hold discretization. Measured intervals scale each channel’s learned nominal step by \Delta t/(0.1\,\mathrm{s}). A learned readout, input skip connection, and gated output projection produce the memory condition. For both Diffusion Policy and \pi_{0.5}, plain S4D is trained independently with a constant 0.1-second input interval throughout training and evaluation.

LTC+\Delta t receives elapsed seconds directly in its state updates. Diffusion Policy uses the ncps 1.0.1 cell; \pi_{0.5} uses the corresponding JAX implementation. LTC uses dense sensory and recurrent conductances with positive parameter constraints and six semi-implicit integration steps per observation interval. We compare LTC and TFP under the same backbone and action-conditioning interface. TFP uses the exponential update in Equation 3.

Temporal inputs.

TFP starts from zero memory with a common 0.1-second first interval. For both backbones, fixed-\Delta t evaluation replaces each input interval with 0.1 seconds while keeping the trained weights fixed. At 10 Hz, fixed and measured intervals coincide.

Training data.

Both Diffusion Policy and \pi_{0.5} use fixed 0.1-second observation intervals (10 Hz) during training, without mixed-cadence sampling augmentation; the other observation schedules are used at evaluation. The main controlled timing comparison uses three training seeds. For DP, Button uses 400/100 training/validation demonstrations; Passage uses 400/100 original plus 128/32 recovery demonstrations. Interception uses 60/12 motion profiles, formed from 10/two scenes crossed with two directions and three speeds.

Observation schedules and execution.

Reported capture intervals are 0.1 seconds (10 Hz), 0.2 seconds (5 Hz), uniform draws from \{0.05,0.10,0.15,0.20\} seconds (jitter), or draws from \{0.10,0.20,0.40\} seconds with probabilities \{0.6,0.2,0.2\} (dropout). A captured observation replaces the action queue; missing captures create no extra memory updates. Timeouts are 12, 22, and 6 seconds for button, passage, and interception.

Physical success criteria.

Button success requires a single continuous press lasting 3.0–3.5 seconds followed by retraction above 0.30 meters. Passage presents a 0.6-second cue at a randomized onset and permits crossing during [t_{\mathrm{cue}}+6,t_{\mathrm{cue}}+9]; the robot must clear the window and reach the goal without collision. Interception evaluations use target speeds of 0.08, 0.12, and 0.16 m/s in two directions under each observation schedule. The target disappears at 0.55 seconds for the remainder of the six-second episode, and the robot holds until 0.6 seconds. Success requires legal first contact with pre-collision planar velocity mismatch at most 0.04 m/s, sustained for at least 0.1 seconds.

Statistics.

Cross-schedule means weight the four schedules equally and are computed from unrounded success counts. The ablations in Tables 2 and 6 report three-seed means.

D.2 Memory-Structure Ablations

In Table 2, static time constants retain only a learned bias, and the learned \Delta t gate uses a sigmoid head on [\mathbf{x}_{n};\mathbf{h}_{n-1};\Delta_{n}]. Observation features, memory width, and action-conditioning pathways are held fixed within each comparison.

The Diffusion Policy candidate and static-time-scale comparisons share task data, training budgets, and common-parameter initialization with the time-scale-input variants in Table 6. The latter restrict the time-constant head to observation or memory inputs, or use learned, input-independent time constants. These comparisons use measured intervals.

D.3 Accumulated Memory and Action Conditioning

Table 6 compares full TFP with two separately trained ablations. No memory conditioning removes the recurrent contribution to the decoder condition. Reset memory each observation retains the memory update and measured interval but starts every update from zero, eliminating information carried across observations. Both preserve current-observation and diffusion-timestep conditioning. The ablations apply throughout training and evaluation under the task-specific protocols in Appendix D.1. Shape-compatible parameters share initialization.

Each task uses 50 paired test scenes per schedule and seed. Results average the four schedules equally within each of three training seeds, then average across seeds.

D.4 Update Rule

Table 6 compares two update variants using the task-specific protocols. Results average four schedules and three training seeds, using 50 scenes per task, schedule, and seed. Shared scalar \tau predicts one time constant for all memory channels. Rational retention replaces \exp(-\Delta_{n}/\tau_{n,j}) by (1+\Delta_{n}/\tau_{n,j})^{-1} in each channel j, retaining measured time and the candidate update.

D.5 Frozen-Weight Interventions

Each closed-loop intervention keeps the trained weights and physical task schedule fixed and changes either the supplied interval or stored memory. Time-input changes act at policy queries; memory resets occur at the times stated in the table.

Figure 4 complements the success rates in Table 7(b) with median Button and Passage event times under the same clock scaling. With \pi_{0.5}, halving the supplied intervals lengthens button contact in all 50 paired trials, whereas doubling them shortens it in all 50. Both backbones show later passage entry with halved intervals and earlier entry with doubled intervals in all 50 paired trials.

D.6 Temporal and Motion Information in Memory

For the Diffusion Policy duration readouts in Tables 8 and 9(a), we fit linear ridge probes to frozen full-TFP representations extracted from demonstration histories at a nominal 10 Hz cadence. Targets are continuous hold time during button contact and elapsed time since the passage cue after it disappears. These labels supervise only the probe. The features are the current encoded observation, recurrent memory, their concatenation, or elapsed time since episode start.

Each probe uses four causal queries per episode: 128 episodes (512 queries) for fitting and 32 episodes (128 queries) for validation. Standardization uses training statistics; the ridge penalty is 1.

Table 9(b) also compares readouts of time since the Passage cue from \pi_{0.5} memory variants, each augmented with elapsed episode time, against the episode clock alone.

Motion information after occlusion.

For Interception, we extract representations from frozen Diffusion Policy with TFP at 0.6 seconds, after the target disappears and before the robot moves. Ridge probes fit 210 prefixes from 35 scenes and are tested on 90 prefixes from 15 other scenes; all six direction–speed variants of a scene remain in the same split. Standardization and penalty selection use only fitting scenes, with five-fold grouped cross-validation. Table 9(c) reports signed-velocity MAE, independently fitted speed-magnitude MAE, and direction accuracy. Single-frame controls use encoded images and robot state.

Navigation memory visualization.

The demonstration in Figure 5 has a cue at 1.7 seconds. Its layout has the readout error for time since the cue closest to the median over the 32 analyzed layouts. Channels 112 and 142 are drawn from the three largest readout contributors on the probe-fitting layouts. Heatmap rows are ordered once by their median time constant over the demonstration.

D.7 Event-Aligned Memory in the Other Simulation Tasks

Figures 6 and 7 complement the Passage visualization in Figure 5. Both use frozen Diffusion Policy with TFP at 10 Hz. Heatmaps show write gains \mathbf{g}_{n}=\mathbf{1}-\mathbf{k}_{n} for all 128 channels, ordered once by their median time constant over the displayed sequence, with a fixed [0,1] color scale. Lower panels show selected raw-memory channels \mathbf{h}_{n}; the Passage panel shows time constants.

D.8 Manipulation Memory Ablations

The reproduced \pi_{0.5} manipulation variants in Table 4 share demonstrations, action horizon, optimizer, execution settings, and evaluation protocol.

Training cost.

The largest \pi_{0.5} runs used over 100 GB GPU memory and approximately 80 hours on four H200 GPUs. Memory storage scales with the number of active episodes; sequence processing costs depend on the training unroll length.

D.9 Memory Dynamics in LIBERO

Figure 8 visualizes memory write gains during a LIBERO manipulation sequence.

D.10 Real-World Setup and Evaluation Protocol

We use a single-arm Galaxea A1 with a parallel gripper, a wrist-mounted fisheye camera, and a static Intel RealSense D435 camera.

The real-robot Diffusion Policy experiments use our custom compact visual diffusion-policy implementation described in Appendix D.1, with the same backbone architecture, TFP memory update, and action-conditioning interface as in the controlled simulations.

The baseline for each backbone is its standard reactive policy without persistent memory. Each task uses 50 demonstrations for training. Both backbones train at fixed 0.1-second query intervals (10 Hz), matching the controlled timing experiments. We evaluate each method in 20 trials per task.

Tea-steeping success requires an immersion duration of 3.0\pm 0.5 s, measured from the infuser entering the water to its removal from the water, followed by successful task completion. For staged heating, the preheating interval runs from heater activation to the food contacting the heating surface; the cooking interval runs from that contact to heater deactivation. Both intervals must be 3.0\pm 0.5 s, and the robot must complete the operations in the prescribed order. Event timestamps are annotated in evaluation videos, using the same timing criteria for all methods.

Appendix E Statistical Reliability of Rollout Results

E.1 Controlled Timing: Variation Across Training Seeds

Tables 1 and 2 report mean success across three independent training seeds. Additional Diffusion Policy ablations in Table 6 use the same seed-level aggregation and report mean \pm sample SD. Table 10 summarizes task-average success as mean \pm sample standard deviation. For each seed, we first average success equally across 10 Hz, 5 Hz, jitter, and dropout, then compute the mean and standard deviation of the three seed-level averages. Overall success in the main text additionally weights the three tasks equally, irrespective of their different rollout counts.

LTC+\Delta t has overall success of 19.7\pm 2.3% for Diffusion Policy and 23.6\pm 2.4% for \pi_{0.5}, averaging tasks equally within each seed. The continuation of Table 10 gives its results for each observation schedule as mean \pm sample SD and rollout 95% confidence intervals.

E.2 LIBERO Seed Variability and Rollout Uncertainty

The reproduced LIBERO and LIBERO-plus results use three training seeds. Table 12 reports mean \pm sample SD. For each seed, Avg. weights the four LIBERO suites or three LIBERO-plus perturbation conditions equally; its SD is computed from the three seed-level averages. Statistics are rounded only after aggregation.

Each standard LIBERO suite has 10 tasks with 50 rollouts per task and seed for our evaluated variants: 500 per seed and 1,500 across three seeds. Table 12 also reports suite-level rollout 95% confidence intervals for TFP. These quantify finite-rollout uncertainty in each evaluated policy’s success rate. Reproduced variants share this protocol; published policies retain their reported settings.

Appendix F Task Sequences