Robot-WATCHDOG : failure detection through object-centric graph representation

Abstract

Reliable real-time failure detection is critical for deploying learned robotic policies, yet defining failures explicitly is infeasible in open-world settings. This motivates failure detection from nominal demonstrations alone. Existing methods, however, struggle to balance robustness to benign variations with the ability to capture semantic and interaction-level errors. We introduce Robot-WATCHDOG, a self-supervised framework that models manipulation scenes as object-centric spatio-temporal graphs. Building on this representation, we investigate two complementary failure scores: Trajectory Projection and Correlation (TPC), a non-parametric measure of geometric deviation and relational consistency, and GnnT, a Graph Attention Network with a Transformer that predicts future object states. We evaluate both on a public benchmark and on a new dataset of semantically complex manipulation tasks, across six tasks and ten random seeds. Both scores outperform image-based failure detectors, particularly on semantic and interaction-level failures. TPC provides the strongest performance in AUPR and MCC on both dataset averages, with low variance across seeds. Qualitative results are made available at https://cea-list.github.io/Robot-WATCHDOG/.

Keywords: Robot Learning, Failure Detection, Object-Centric Representations, Graph Neural Networks

1 INTRODUCTION

Learned robotic policies provide no guarantees beyond the distribution induced by training demonstrations [2]. Detecting deviations from expected behavior is therefore essential for accountable deployment, and increasingly emphasized in regulation for high-risk AI systems such as the EU AI Act [18, 37]. As imitation learning and vision-language-conditioned policies increasingly enable long-horizon, closed-loop manipulation [9, 42, 6, 24, 21, 30], policies are executed autonomously in real time and without step-by-step human intervention, motivating dedicated monitoring systems that operate online, require no failure examples, and can identify anomalous behavior before errors propagate.

Explicitly enumerating all possible failures, however, is infeasible in open-world robotic settings. The standard framing therefore casts failure detection as a one-class problem [41, 32, 33]: a monitor trained exclusively on nominal demonstrations must identify, at runtime, deviations from expected behavior. Existing approaches divide broadly into two families. Visual failure detectors [35, 15, 43, 32] flag deviations in raw image space or learned visual representations. They are sensitive to benign background changes, lighting shifts, and visual clutter, producing frequent false positives that erode operational efficiency. Kinematic and action-based monitors [27, 14, 38] instead operate on trajectory or joint-space signals; they are robust to visual noise but structurally blind to the environment. They only verify whether the movement itself is familiar, failing to detect if the robot is interacting with the wrong object or operating in an environment that mismatches training.

This division points to a broader limitation. A substantial class of failures in robotic manipulation does not arise from abnormal motion or corrupted observations, but from incorrect interactions between otherwise familiar objects: grasping the wrong item, violating task-order constraints, or placing an object in an incorrect spatial relation to others [23, 8]. These failures emerge naturally in manipulation tasks whose success depends on object identity, relative placement, and sequential consistency, yet they remain difficult to capture when monitoring is restricted to either image appearance or motion trajectories alone.

The central idea of this work is to recast robotic failure detection as monitoring the evolution of object interactions over time. Rather than treating visual and kinematic signals as separate modalities, we represent the scene as a spatio-temporal graph of interacting objects, where nodes encode object states and edges encode their pairwise spatial relationships throughout execution. This object-centric formulation generalizes prior monitoring approaches: visual detectors can be interpreted as operating on node appearance, while kinematic monitors correspond to tracking motion along trajectories. By jointly modeling objects, their relations, and their temporal evolution in a unified representation, the proposed framework captures both the signals targeted by prior work and the relational structure required to detect task-relevant interaction failures.

Failure detection then becomes a well-posed problem: deviations from reference relational dynamics constitute anomalies. We instantiate this principle in Robot-WATCHDOG, a self-supervised framework that abstracts a manipulation episode into an object-centric spatio-temporal graph and scores each timestep against expert demonstrations. Within this representation we study two scoring functions. TPC (Trajectory Projection and Correlation) is non-parametric: it measures orthogonal projection distances to expert trajectories together with deviations in the pairwise correlation structure between objects. GnnT couples a Graph Attention Network with a Transformer encoder to predict future object dynamics from a sliding observation window, and flags prediction errors. Both are converted into binary decisions by conformal prediction calibrated on held-out expert demonstrations, which sets the decision threshold from a user-specified level without tuning the threshold on failure examples.

We evaluate Robot-WATCHDOG on four tasks of the public BotFails benchmark [32] and on a newly introduced dataset featuring two relationally challenging manipulation tasks: structured table setting and waste sorting on a moving conveyor belt. All results are averaged over ten seeds, each redrawing the expert partition and the network initialisation. Both object-centric scores outperform image-based detectors, and TPC is consistently the stronger of the two: averaged per dataset it reaches AUPR 0.81 and maximum MCC 0.75 (BotFails) and 0.81 / 0.72 (Watchdog), against 0.62 / 0.49 and 0.65 / 0.63 for GnnT, while the best baseline reaches only 0.44 / 0.32 and 0.35 / 0.34. After conformal calibration, TPC attains MCC 0.68 and 0.64 where the strongest baseline reaches 0.22 and 0.33. The learned score only leads in the continuously dynamic conveyor scene. Methods that perform well on simple kinematic deviations struggle with relational failures involving object identity and spatial arrangement, the very setting Robot-WATCHDOG addresses.

Our main contributions are: (i) identifying relational anomalies as a key yet underexplored failure mode in robotic manipulation, arising from dependencies on object identity, spatial arrangement, and temporal consistency; (ii) introducing two scoring functions: TPC (Trajectory Projection and Correlation) a non-parametric, geometric method and GnnT, a spatio-temporal graph formulation ; and (iii) releasing a benchmark of relationally challenging manipulation tasks with annotated relational failures.

2 RELATED WORK

2.1 Graph oriented object representation

A growing line of work in robotics and visuomotor learning has explored object-centric graph representations, where nodes encode task-relevant entities and edges capture spatial, label-conditioned, or action-conditioned relations [44, 25, 4]. Such representations improve invariance to irrelevant visual variations and provide a structured way to model interactions between objects. Building on this paradigm, several approaches construct dynamic or action-conditioned scene graphs to model evolving environments, improving robustness and generalization in manipulation tasks [22, 19]. In parallel, graph neural networks have been leveraged for relational policy learning, planning, and dynamics prediction, demonstrating improved scalability in multi-object settings [20, 36]. More recently, Vosylius et al. [39] show that graph-structured representations can support in-context policy adaptation by modeling demonstrations over relational structures. Recent progress in vision-language models and object tracking has also made these representations increasingly practical in robotics, enabling automatic extraction of task-relevant objects and trajectories directly from RGB observations. To our knowledge, while object-centric graph representations have shown strong promise for policy learning and planning, they have received limited attention for online failure detection, motivating their investigation in this work.

2.2 Failure detection in autonomous robotics

Existing real-time monitors struggle to balance environmental awareness, robustness to benign variations, and sensitivity to task-relevant object interactions. Furthermore, many traditional failure classifiers [13, 28, 29] require collecting explicit failure data a priori, a time-consuming process that scales poorly to open-world robotic deployment. Consequently, recent efforts focus on one-class methods trained exclusively on nominal demonstrations. However, vision-based detectors like FIDeL [32] remain highly sensitive to visual clutter. Sentinel [1] attempts to bridge the modality gap by combining STAC : a statistical measure of kinematic action consistency, with a Vision-Language Model (VLM) for visual verification. Yet, STAC remains blind to environment-only failures, and the VLM fallback introduces severe inference latency unsuited for reactive continuous control. Similarly, FAIL-Detect [41] merges modalities through latent concatenation, while FIPER [33] relies on brittle logical conjunctions.

Crucially, these approaches often struggle to capture failures involving object identity, spatial arrangement, or task progression, such as grasping the wrong object, executing sub-actions out of order, or misplacing items. Such errors often produce nominal kinematic profiles and only subtle pixel-level changes, making them difficult to detect with action-consistency metrics or standard visual anomaly detectors.

Our framework, Robot-WATCHDOG, addresses these limitations through an object-centric representation of scene dynamics. By abstracting tracked entities into a spatio-temporal graph, the framework reduces sensitivity to irrelevant visual variation while preserving explicit environmental context unavailable to kinematic-only monitors. Representing the scene in a relational geometric space also enables anomaly detection based on changes in object interactions over time, either directly, by comparing object trajectories and their pairwise correlations to expert demonstrations (TPC), or through a predictive GNN-Transformer that jointly models object identities, spatial configurations, and temporal dependencies (GnnT). Both enable real-time detection of relational failure modes that remain challenging for existing multimodal baselines, without requiring failure demonstrations during training.

3 Method

As illustrated in Figure 2, we propose a self-supervised failure detection framework for multi-object robotic manipulation. The framework rests on a single representation: each episode is abstracted into a sequence of object-centric spatio-temporal graphs built from RGB observations and robot proprioception. Within this representation we define two scoring functions, both trained or referenced on nominal demonstrations only, and both turned into binary decisions by the same conformal calibration. TPC is non-parametric and scores an episode by its geometric and relational deviation from expert trajectories. GnnT learns the nominal dynamics and scores an episode by its future-prediction error. At every timestep each scorer produces one value per tracked object; the scalar failure score S_{\text{total}} is their sum over objects.

3.1 Problem Formulation and Graph Extraction

We consider robotic manipulation episodes involving C distinct entities (task-relevant objects together with the robot end-effector) evolving in a fixed workspace. Using a vision-language initialization model and a tracking pipeline [3, 31], each episode is converted from RGB observations into a sequence of 2D trajectories over tracked labeled objects. This yields a temporal graph sequence \{\mathcal{G}_{t}\}_{t=1}^{T}, which is the common input of both scoring functions defined below.

At each time step t, the scene is abstracted as \mathcal{G}_{t}=(V,E_{t},X_{t}). The vertex features X_{t}\in\mathbb{R}^{C\times d} of vertices V encode absolute positions \mathbf{p}_{i,t} and instantaneous velocities: \mathbf{x}_{i,t}=[\mathbf{p}_{i,t},\mathbf{p}_{i,t}-\mathbf{p}_{i,t-1}]. Edges E_{t} encode the explicit pairwise spatial relations. For any pair (i,j), the relative displacement vector \mathbf{o}_{ij,t}=\mathbf{p}_{j,t}-\mathbf{p}_{i,t} yields a Euclidean distance d_{ij,t} and an orientation \theta_{ij,t}. The edge feature is defined as : \mathbf{e}_{ij,t}=\big[\,o_{ij,t}^{(x)},\ o_{ij,t}^{(y)},\ d_{ij,t},\ \cos(\theta_{ij,t}),\ \sin(\theta_{ij,t})\,\big]\in\mathbb{R}^{5}

This representation has three useful properties for failure detection: (i) relative edge encoding makes pairwise relations invariant to global translations; (ii) explicit modeling of pairwise object interactions, allowing failures involving object identity or spatial arrangement to be represented directly; and (iii) temporal consistency through velocity features and graph sequences, enabling detection of both instantaneous and sequential deviations.

3.2 TPC: Geometric and Relational Consistency

Our first scorer is non-parametric: it compares the observed episode directly to the set of expert trajectories, without fitting any model. For each tracked object, TPC computes the orthogonal projection distance from the observed trajectory to the closest expert trajectory of the same label, aggregated over time with exponentially increasing weights so that recent deviations dominate. Since the per-point distance does not depend on the prefix, the score of a streaming episode is maintained incrementally at constant cost per timestep. To capture deviations in interaction structure rather than in individual kinematics, TPC adds a correlation penalty: the \ell_{1} divergence between the pairwise Pearson correlation matrices of the observed objects and of the nominal dataset. Implementation details are given in Appendix B.4.

S_{\text{total}}(t)=\sum_{i=1}^{C}\sum_{k=1}^{K(t)}\frac{e^{\gamma k}}{\sum_{j\leq K(t)}e^{\gamma j}}\cdot\frac{\bigl|\min_{m}d\bigl(p_{i,k},\,\tau^{(m)}_{\ell(i)}\bigr)-\mu_{\ell(i)}\bigr|}{\sigma_{\ell(i)}+\varepsilon}\;+\;\sum_{i<j}\bigl|\rho_{ij}(t)-\bar{\rho}_{ij}\bigr|

3.3 GnnT: Predictive Spatio-Temporal Dynamics

Our second scorer replaces the direct comparison to expert trajectories by a learned model of the nominal dynamics: failure detection is cast as a future-prediction problem, where deviations from predicted object states signal an anomaly. This formulation can represent action-reaction delays that a purely geometric comparison cannot, at the cost of requiring enough demonstrations to fit the model.

Spatial Encoding via Dynamic Graph Attention

Standard convolutional operators treat all neighbors equally. Instead, we employ GATv2 [5] to compute dynamic attention coefficients \alpha_{ij,t}, allowing the network to weight task-relevant interactions (e.g., an end-effector approaching a target object) more strongly than distant or weakly coupled entities. For a vertex i and its neighbors j\in\mathcal{N}_{i}, the updated spatial embedding \mathbf{h}_{i,t}\in\mathbb{R}^{D} is:

\mathbf{h}_{i,t}=\sigma\left(\sum_{j\in\mathcal{N}_{i}}\alpha_{ij,t}\mathbf{W}\mathbf{x}_{j,t}\right)

where \mathbf{W} is a learnable weight matrix applied to the input features, and \sigma denotes a non-linear activation function. This operation yields a set of spatially contextualized embeddings H_{t}=\{\mathbf{h}_{1,t},\dots,\mathbf{h}_{C,t}\}, where C is the total number of vertices in the graph.

Spatio-Temporal Transformer & Future Prediction

To model action-reaction delays across time, we process a sliding window of the past m spatial embeddings \mathcal{S}_{in}=\{H_{t-m+1},\dots,H_{t}\}. We flatten \mathcal{S}_{in} into a sequence of m\times C tokens. This fine-grained tokenization enables modeling dependencies between specific objects at specific past timesteps. We inject structural awareness via a composite positional encoding (PE) that accounts for relative time \tau within the m-window, object ID i, and global episode progress:

\mathbf{z}_{i,\tau}^{(0)}=\mathbf{h}_{i,t-(m-1)+\tau}+\text{PE}_{\text{time}}(\tau)+\text{PE}_{\text{obj}}(i)+\text{PE}_{\text{global}}\left(\frac{t}{T_{\max}}\right)

After processing through N Transformer encoder layers, the tokens corresponding to the current step t are projected via an MLP to predict the future sequence of positions over horizon L: \widehat{\mathbf{Y}}_{i,t}=\text{MLP}(\widehat{\mathbf{h}}_{i,t})\in\mathbb{R}^{L\times d_{out}}.

Training Objective

The network is optimized solely on nominal demonstrations using a dual-loss objective. We minimize the Mean Squared Error (MSE) of the predicted trajectories (\mathcal{L}_{\text{pos}}) and add an edge-consistency term (\mathcal{L}_{\text{edge}}) encouraging agreement between predicted and observed pairwise spatial relations: \mathcal{L}=\lambda_{p}\mathcal{L}_{\text{pos}}+\lambda_{e}\mathcal{L}_{\text{edge}}.

3.4 Uncertainty Calibration via Conformal Prediction

A fundamental challenge in failure detection is defining the decision threshold without relying on arbitrary, task-specific manual tuning or exposing the model to OOD failure data. We address this using Conformal Prediction (CP) [26, 11]. Using a held-out calibration set of nominal episodes \mathcal{Z}_{N}, we compute the non-conformity scores (i.e., the prediction error for GnnT, or the geometric distance for TPC). For a user-specified tolerance level \alpha\in(0,1), we build a time-varying threshold \eta_{t} from these calibration scores with the functional construction of [11] (Appendix B.5): its width is the (1-\alpha)-quantile of the maximum deviation of calibration episodes from the mean nominal profile. Under exchangeability, the corresponding finite-sample conformal construction would control the trajectory-level error rate; because we use the uncorrected empirical quantile, we treat the resulting level as approximate.

4 Evaluation

4.1 Dataset

We evaluate Robot-WATCHDOG on two complementary datasets: (i) a public benchmark from prior work and (ii) a new dataset designed to stress-test relational and dynamic failure detection.

BotFails.

We evaluate on BotFails [32], a recent benchmark for robotic failure detection collected with LeRobot [7]. The full dataset spans 10 manipulation tasks across domestic and industrial settings with diverse annotated anomaly types. To align with our object-centric formulation, we retain 4 tasks involving rigid and distinguishable objects: Table-setting, Dish storing, Vegetable sorting, and Groceries sorting. Tasks involving deformable or fluid interactions are excluded, as they violate the discrete object abstraction underlying our method. Additional details are provided in Appendix C.1.

Ours.

To complement BotFails, we introduce a new dataset comprising two manipulation tasks with increased relational and dynamic complexity. The first task is a table-setting scenario requiring structured object placement and ordering. The second is a waste-sorting task on a conveyor belt, where objects exhibit continuous motion, leading to variability in initial states and interaction timing. This dataset serves two purposes: (i) providing a larger number of demonstrations for training data-driven models, and (ii) evaluating robustness under increased variability and less constrained dynamics. Further details are provided in Appendix C.1.

4.2 Baselines

We compare Robot-WATCHDOG against a diverse set of anomaly detection methods spanning three complementary categories.

Trajectory matching. These methods detect kinematic and geometric deviations by comparing observed trajectories to expert demonstrations. To ensure invariance to execution speed and temporal misalignment, AP-RMS and AP-Fréchet rely on arc-length resampling and evaluate prefix-based distances using Root Mean Square error and the discrete Fréchet distance [12], respectively. Soft-DTW instead computes a global alignment cost using a differentiable soft-minimum formulation [10], making it more robust to local temporal distortions.

Learned representations. We evaluate density-based and reconstruction-based approaches trained exclusively on nominal data. Specifically, we consider Continuous Normalizing Flow models (logpZO, lopO) [41], which estimate the likelihood of observations either in latent space or via exact log-likelihood computation, and an AutoEncoder (AE-Recon) [34, 16, 40], which uses reconstruction error as an anomaly score.

Visual-semantic alignment. We include FIDeL [32], a recent two-stage method that combines patch-level feature alignment via optimal transport with a vision-language model to filter semantically irrelevant anomalies.

Ours. We compare against both variants of our framework: (i) TPC, a non-parametric method based on trajectory projection and relational correlation, and (ii) GnnT, a predictive spatio-temporal graph model that captures object interactions and temporal dependencies. Additional implementation details are provided in the Appendix C.2.

4.3 Evaluation Protocol

We evaluate Robot-WATCHDOG at the step level in two settings: Figure 3 reports threshold-independent anomaly scoring, reporting AUPR (favored under strong class imbalance) and maximum MCC (a balanced summary over all confusion matrix terms), while Figure 5 reports end-to-end detection after Conformal Prediction, reporting calibrated MCC (TPR and TNR per task are given in Appendix D.2), capturing respectively failure interception and robustness to benign variability. Every number is averaged over ten seeds, a seed redrawing the expert train/calibration partition, the network initialisation and the conformal split, so the reported deviations measure sensitivity to which demonstrations a method was given. Calibrated results use the modulation s_{1} of Appendix B.5 (inspired by [41]) and are averaged over \alpha\in[0.80,0.95].

4.4 Results and discussion

Figure 3 summarizes threshold-independent performance: TPC leads on both datasets (AUPR 0.811 / MCC 0.751 on BotFails, 0.810 / 0.719 on Watchdog), ahead of GnnT (0.623 / 0.488 and 0.648 / 0.631) and of the strongest baselines, AP-RMS on BotFails (0.442 / 0.320) and AP-Fréchet on Watchdog (0.346 / 0.344). Trajectory metrics evaluate objects independently, cannot express relational deviations, and are the least stable of all, with seed-to-seed standard deviations up to 0.19 AUPR on the Watchdog average (0.37 on its table-setting task), an order of magnitude above TPC (0.013); density, reconstruction and visual baselines (logpZO, lopO, AE-Recon, FIDeL) are steadier but plateau well below both of our scores. Figure 4 illustrates a relational failure (misplaced cup) flagged TPC although absolute kinematics remain nominal.

4.4.1 End-to-End Failure Detection

With CP converting scores into binary alerts (Fig. 5), TPC again leads on both datasets (0.675 on BotFails, 0.641 on Watchdog) ahead of GnnT (0.347, 0.571), the best online baselines reaching 0.220 (FIDeL, BotFails) and 0.329 (logpZO, Watchdog). Gains in raw scoring do not transfer mechanically: baseline rankings change between the two settings, and each method has its own useful range, TPC peaking at low tolerance (\alpha=0.05–0.15, up to MCC 0.845 on Aloha table-setting) and GnnT only becoming competitive at \alpha=0.85–0.99. The nominal level is not matched empirically either: \alpha=0.05 flags 0.2–3\% of nominal steps, \alpha=0.99 flags 10–15\%. Calibration is thus a design choice to report in its own right, hence our two regimes and full sweep.

5 Conclusion

We introduced Robot-WATCHDOG, a self-supervised framework for robotic failure detection based on object-centric spatio-temporal graphs. On this representation, we studied two failure scores: TPC, a non-parametric measure of geometric and relational deviation from expert demonstrations, and GnnT, a predictive graph-transformer. Across a public benchmark and a newly collected dataset, both outperform image- and density-based detectors, particularly on failures involving object interactions and task structure. TPC is the strongest score overall, in both data regimes we tested, while requiring no training and the lowest inference cost; GnnT only approaches it in the continuously dynamic conveyor task, which indicates where learned dynamics may justify their additional complexity. More broadly, these results highlight object-centric relational representations as a promising direction for scalable real-time monitoring of learned robotic policies.

6 Limitations

Robot-WATCHDOG relies on a discrete object-centric abstraction, which limits applicability to tasks involving highly deformable materials, fluids, or interaction domains without clear object decomposition (e.g., painting or soldering). Performance also depends on the upstream vision-language tracking pipeline: severe occlusions, missed detections, or identity switches can degrade graph quality and propagate to the anomaly detector. Finally, our current formulation models object centroids only, and therefore cannot capture failures involving orientation or fine-grained pose. Extending the graph representation with 6D pose estimates is a natural direction for future work.

Acknowledgments

The authors would like to thank Charlotte Sczwarc for the magnificent illustration of the dog figure 1, and Saad Aziz Alaoui for his valuable insights as well as Melanie Cassedanne for her unconditional support. This publication was made possible by the use of the CEA List FactoryAI supercomputer, financially supported by the Ile-de-France Regional Council. This work was partly funded by : the European Union’s Horizon Europe Research and Innovation program under grant agreement no 101135708 (JARVIS project) and no 101070596 (euROBIN project) - and by the France 2030 program through the PEPR O2R projects AS3 (ANR-22-EXOD-007).

References

1. [1] C. Agia, R. Sinha, J. Yang, Z. Cao, R. Antonova, M. Pavone, and J. Bohg (2024) Unpacking failure modes of generative policies: runtime monitoring of consistency and progress. External Links: 2410.04640, Link
2. [2] C. Agia (2026) Deployment-time reliability of learned robot policies. External Links: 2603.11400, Link
3. [3] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025) Qwen3-vl technical report. External Links: 2511.21631, Link
4. [4] P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu (2016) Interaction networks for learning about objects, relations and physics. External Links: 1612.00222, Link
5. [5] S. Brody, U. Alon, and E. Yahav (2022) How attentive are graph attention networks?. In International Conference on Learning Representations, External Links: Link
6. [6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich (2023) RT-2: vision-language-action models transfer web knowledge to robotic control. In arXiv preprint arXiv:2307.15818,
7. [7] R. Cadene, S. Alibert, A. Soare, Q. Gallouedec, A. Zouitine, and T. Wolf (2024) LeRobot: state-of-the-art machine learning for real-world robotics in pytorch. Note: https://github.com/huggingface/lerobot
8. [8] V. Chandola, A. Banerjee, and V. Kumar (2009) Anomaly detection: a survey. ACM Comput. Surv. 41 (3). External Links: ISSN 0360-0300, Link, Document
9. [9] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2024) Diffusion policy: visuomotor policy learning via action diffusion. External Links: 2303.04137, Link
10. [10] M. Cuturi and M. Blondel (2017) Soft-dtw: a differentiable loss function for time-series. In Proceedings of the 34th International Conference on Machine Learning, pp. 894–903.
11. [11] J. Diquigiovanni, M. Fontana, and S. Vantini (2021) The importance of being a band: finite-sample exact distribution-free prediction sets for functional data. External Links: 2102.06746, Link
12. [12] T. Eiter and H. Mannila (1994) Computing discrete fréchet distance. Technical report Technical Report CD-TR 94/64, Christian Doppler Laboratory for Expert Systems, TU Vienna, Austria.
13. [13] C. Gokmen, D. Ho, and M. Khansari (2023) Asking for help: failure prediction in behavioral cloning through value approximation. External Links: 2302.04334, Link
14. [14] N. Grambow, L. Fenner, F. Kempkes, P. Hotz, D. Wan, J. Krüger, and K. Haninger (2026) Anomaly detection for generic failure monitoring in robotic assembly, screwing and manipulation. External Links: 2509.26308, Document, Link
15. [15] Q. Gu, Y. Ju, S. Sun, I. Gilitschenski, H. Nishimura, M. Itkina, and F. Shkurti (2025) SAFE: multitask failure detection for vision-language-action models. arXiv preprint arXiv:2506.09937.
16. [16] M. Hasan et al. (2016) Learning temporal regularity in video sequences. In Proc. of CVPR,
17. [17] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
18. [18] A. Herrera-Poyatos, J. D. Ser, M. L. de Prado, F. Wang, E. Herrera-Viedma, and F. Herrera (2025) A framework for responsible ai systems: building societal trust through domain definition, trustworthy ai design, auditability, accountability, and governance. External Links: Link
19. [19] H. Huang, M. Cen, K. Tan, X. Quan, G. Huang, and H. Zhang (2025) GraphCoT-vla: a 3d spatial-aware reasoning vision-language-action model for robotic manipulation with ambiguous instructions. External Links: 2508.07650, Link
20. [20] Y. Huang, A. Conkey, and T. Hermans (2023) Planning for multi-object manipulation with graph neural network relational classifiers. External Links: 2209.11943, Link
21. [21] P. Intelligence, B. Ai, A. Amin, R. Aniceto, A. Balakrishna, G. Balke, K. Black, G. Bokinsky, S. Cao, T. Charbonnier, V. Choudhary, F. Collins, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, M. Dhaka, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Habeeb, H. Hancock, K. Hausman, G. Hussein, V. Hwang, B. Ichter, C. Jacobsen, S. Jakubczak, R. Jen, T. Jones, G. Kammerer, B. Katz, L. Ke, M. Khadikov, C. Kuchi, M. Lamb, D. LeBlanc, B. LeCount, S. Levine, X. Li, A. Li-Bell, V. Lialin, Z. Liang, W. Lim, Y. Lu, E. Luo, V. Mano, N. Marwaha, A. Mongush, L. Murphy, S. Nair, T. Patterson, K. Pertsch, A. Z. Ren, G. Schelske, C. Sharma, B. Shi, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, J. Tang, J. Tanner, S. Tekeste, M. Torne, K. Vedder, Q. Vuong, A. Walling, H. Wang, J. Wang, X. Wang, C. Whalen, S. Whitmore, B. Williams, C. Xu, S. Yoo, L. Yu, W. Zhang, Z. Zhang, and U. Zhilinsky (2026) {\pi}_{0.7}: A steerable generalist robotic foundation model with emergent capabilities. External Links: 2604.15483, Link
22. [22] H. Jiang, B. Huang, R. Wu, Z. Li, S. Garg, H. Nayyeri, S. Wang, and Y. Li (2024) RoboEXP: action-conditioned scene graph via interactive exploration for robotic manipulation. External Links: 2402.15487, Link
23. [23] E. Khalastchi, M. Kalech, and G. Kaminka (2015) Online data-driven anomaly detection in autonomous robots. External Links: Link
24. [24] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn (2024) OpenVLA: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246.
25. [25] T. Kipf, E. Fetaya, K. Wang, M. Welling, and R. Zemel (2018) Neural relational inference for interacting systems. External Links: 1802.04687, Link
26. [26] J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman (2017) Distribution-free predictive inference for regression. External Links: 1604.04173, Link
27. [27] E. D. Lello, M. Klotzbücher, T. D. Laet, and H. Bruyninckx (2013) Bayesian time-series models for continuous fault detection and recognition in industrial robotic tasks. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5827–5833. External Links: Link
28. [28] H. Liu, S. Dass, R. Martín-Martín, and Y. Zhu (2023) Model-based runtime monitoring with interactive imitation learning. External Links: 2310.17552, Link
29. [29] H. Liu, Y. Zhang, V. Betala, E. Zhang, J. Liu, C. Ding, and Y. Zhu (2024) Multi-task interactive robot fleet learning with visual world models. External Links: 2410.22689, Link
30. [30] NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. ”. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu (2025) GR00T N1: an open foundation model for generalist humanoid robots. In ArXiv Preprint, External Links: 2503.14734
31. [31] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer (2024) SAM 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714. External Links: Link
32. [32] Q. Rolland, F. Mayran de Chamisso, and J. Mouret (2026) Failure identification in imitation learning via statistical and semantic filtering. In IEEE International Conference on Robotics and Automation (ICRA),
33. [33] R. Römer, A. Kobras, L. Worbis, and A. P. Schoellig (2025) Failure prediction at runtime for generative robot policies. External Links: 2510.09459, Link
34. [34] Y. Shi, J. Yang, and Z. Qi (2021) Unsupervised anomaly segmentation via deep feature reconstruction. Neurocomputing.
35. [35] S. Thoduka, J. Gall, and P. G. Ploger (2021) Using visual anomaly detection for task execution monitoring. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4604–4610. External Links: Link, Document
36. [36] R. Vakhitov, L. Ugadiarov, and A. Panov (2025) Object-centric world models meet monte carlo tree search. In Proceedings of the Ninth International Scientific Conference “Intelligent Information Technologies for Industry” (IITI’25), Volume 2, pp. 481–491. External Links: ISBN 9783032136121, ISSN 2367-3389, Link, Document
37. [37] M. Valdenegro-Toro and R. Stoykova (2024) The dilemma of uncertainty estimation for general purpose ai in the eu ai act. External Links: 2408.11249, Link
38. [38] A.T. Vemuri, M.M. Polycarpou, and S.A. Diakourtis (1998) Neural network based fault detection in robotic manipulators. IEEE Transactions on Robotics and Automation 14 (2), pp. 342–348. External Links: Document
39. [39] V. Vosylius and E. Johns (2025) Instant policy: in-context imitation learning via graph diffusion. External Links: 2411.12633, Link
40. [40] T. Wang et al. (2018) Generative neural networks for anomaly detection in crowded scenes. IEEE TIFS.
41. [41] C. Xu, T. K. Nguyen, E. Dixon, C. Rodriguez, P. Miller, R. Lee, P. Shah, R. Ambrus, H. Nishimura, and M. Itkina (2025) Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies. External Links: 2503.08558, Link
42. [42] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware. External Links: 2304.13705, Link
43. [43] S. Zhou, B. Zhu, J. Yang, X. Zhao, J. Chen, and Y. Jiang (2026) RC-nf: robot-conditioned normalizing flow for real-time anomaly detection in robotic manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
44. [44] Y. Zhu, A. Lim, P. Stone, and Y. Zhu (2025) Vision-based manipulation from single human video with open-world object graphs. External Links: 2405.20321, Link

Appendix

Appendix A Ablation Study

We conduct a series of ablation experiments to quantify the contribution of each component of the proposed framework. Our analysis focuses on (i) the input representation, (ii) relational modeling, (iii) the spatio-temporal architecture, and (iv) the predictive formulation.

A.1 Impact of Object-Centric Graph Representation

We evaluate whether representing the scene as an object-centric graph provides a measurable advantage over alternative representations.

In Table 1, we compare three configurations:

Vision feature baseline.

To isolate the contribution of our Spatio-Temporal Graph Neural Network (GnnT) and its object-centric inductive bias, we introduce a purely sequential vision-based baseline that bypasses the graph construction phase. In this ablation, the demonstration episode is modeled as a flat sequence of global observations. At each timestep t, dense visual features extracted by a pre-trained encoder \mathcal{F}_{\theta} (ResNet18 [17]) are concatenated with proprioceptive data to form a state vector \mathbf{x}_{t}\in\mathbb{R}^{d_{in}}. These states are linearly projected into a hidden dimension D, augmented with sinusoidal positional encodings, and processed by a multi-layer Transformer Encoder to capture global temporal dependencies across the sequence. The resulting output tokens are then mapped back to the original feature space via an MLP head to reconstruct the input states \mathbf{x}_{t}. Trained exclusively on nominal data using a Mean Squared Error (MSE) objective, this auto-encoding baseline utilizes the state-wise reconstruction error as its failure score during inference. Results are presented in Table 1 under the name Vision features.

Trajectory-only representation.

We evaluate a graph-free ablation model, denoted as the Trajectory Transformer. This baseline deliberately removes the Graph Neural Network (GNN) encoder and all explicit edge-feature computations. Specifically, for each object c\in\{1,\dots,C\} at time step t\in\{1,\dots,m\}, its kinematic state x_{c}^{t}\in\mathbb{R}^{2} is concatenated with a learned semantic embedding e_{c}\in\mathbb{R}^{d_{sem}} and linearly projected to a hidden token representation: h_{c}^{t}=\mathbf{W}[x_{c}^{t}\parallel e_{c}]+\mathbf{b}, where \mathbf{W}\in\mathbb{R}^{d_{h}\times(2+d_{sem})} is the projection weight matrix and \parallel denotes concatenation. Following the addition of standard temporal positional encodings, the flattened sequence of independent object tokens is processed by a multi-head Transformer encoder. The future trajectories \hat{Y}\in\mathbb{R}^{L\times C\times 2} are regressed directly from the final sequence embeddings via a Multi-Layer Perceptron (MLP) head. Unlike the full model, this baseline is optimized exclusively using a Mean Squared Error (MSE) loss on absolute coordinates, without any topological constraints. By stripping away structural priors, this ablation forces the self-attention mechanism to independently discover and model inter-object physical correlations solely from raw trajectories.

Graph-based representation (ours).

Object trajectories are structured as spatio-temporal graphs and processed by the proposed GNN-Transformer architecture; see Section 3-Method of the paper for more details.

The results in Table 1 highlight the importance of structured, object-level inputs. The global vision feature baseline performs poorly on both tasks and fails entirely on Table-setting (TNR and MCC of 0), indicating that unstructured, dense visual features lack the precise spatial inductive biases required for failure detection. Extracting explicit object trajectories yields a large improvement over raw vision features. The additional benefit of the graph structure, however, is strongly task-dependent. On Table-setting, where failures are defined by incorrect spatial arrangements between objects, the graph representation outperforms the trajectory-only model on every metric, improving TPR by 29.1 points, AUPR by 25.5 points and MCC by 23.8 points. On Waste-sorting, the picture is more nuanced: the graph representation substantially increases sensitivity (TPR +23.1 points) at the cost of more false alarms (TNR -13.0 points), and the trajectory-only model retains a slightly higher AUPR (54.7\% vs. 49.7\%) and MCC (56.5\% vs. 53.1\%). This is consistent with the nature of Waste-sorting failures, which mostly involve deviations of individual objects that can already be detected from their own trajectories, leaving limited room for relational reasoning. Overall, explicit relational structure is most valuable precisely in tasks whose failures are defined by inter-object configurations.

A.2 Impact of Relational Modeling

We evaluate the importance of explicitly modeling interactions between objects for both the predictive (GnnT) and non-parametric (TPC) formulations.

For the GnnT method

To assess the contribution of relational inductive bias, we construct a randomized graph baseline by perturbing the edge structure while preserving node features. Formally, given a graph \mathcal{G}=(\mathcal{V},\mathcal{E}) with |\mathcal{E}|=E, we sample a randomized edge set \tilde{\mathcal{E}} by drawing E pairs (i,j) uniformly from \mathcal{V}\times\mathcal{V}, yielding a graph \tilde{\mathcal{G}}=(\mathcal{V},\tilde{\mathcal{E}}). The node representations are then computed as

\mathbf{h}_{i}^{(l+1)}=\sigma\left(\sum_{j\in\mathcal{N}_{\tilde{\mathcal{G}}}(i)}\alpha_{ij}^{(l)}\mathbf{W}^{(l)}\mathbf{h}_{j}^{(l)}\right),

where \alpha_{ij}^{(l)} denotes attention coefficients as in GATv2 [5]. This procedure preserves the computational structure of the model while destroying meaningful spatial correlations, enabling a controlled evaluation of the role of relational structure.

For the TPC method

To disentangle the contribution of geometric versus relational cues in TPC, we evaluate three variants of the anomaly score:

• Orthogonal projection only. Each object trajectory is compared independently to expert demonstrations using the weighted orthogonal projection distance. This captures local kinematic deviations but ignores interactions between objects.
• Correlation only. We discard absolute trajectories and instead compare the pairwise correlation structure between object motions. Concretely, for each pair of objects, we compute the Pearson correlation of their x and y trajectories, and penalize deviations from the correlation patterns observed in expert demonstrations. This captures relational consistency but is invariant to global geometric shifts.
• Full TPC (projection + correlation). The final score combines both terms, aggregating per-object projection errors with a global correlation penalty. This jointly captures absolute trajectory deviations and interaction-level inconsistencies.

Results and Interpretation

The results in Table 2 highlight distinct roles of geometric and relational signals in failure detection, as well as their task-dependent interplay.

For GnnT, removing the edge model degrades performance on both tasks, but the magnitude of the drop depends on the relational complexity of the task. On Table-setting, the degradation is severe on threshold-independent and balanced metrics (AUPR -20.7 points, MCC -11.2 points) and is accompanied by more false alarms (TNR -5.6 points), showing that the model cannot reliably distinguish benign variations from relational failures without explicit interactions. On Waste-sorting, the effect is concentrated on sensitivity (TPR -8.1 points, MCC -4.1 points), while TNR and AUPR remain comparable. Removing the edge model also makes the predictions noticeably less stable across seeds (e.g., TNR standard deviation of 8.3 vs. 3.5 on Waste-sorting), indicating that relational structure also acts as a regularizer.

For TPC, the orthogonal projection term alone achieves the best TPR, AUPR and MCC on Waste-sorting, suggesting that failures in this task are predominantly characterized by deviations from nominal trajectories. In contrast, the correlation-only variant attains the highest TPR on Table-setting, indicating that many failures arise from incorrect coordination between objects despite locally plausible motions. Combining both signals yields the best overall performance on Table-setting, with the highest TNR (97.1\%), AUPR (96.3\%) and MCC (88.8\%, i.e. +7.5 points over correlation only and +10.6 points over projection only), while keeping a TPR close to the correlation-only variant (92.3\% vs. 94.9\%). This confirms that geometric deviations and relational inconsistencies provide complementary information. On Waste-sorting, however, the combined score shifts towards a conservative operating point: it achieves the highest TNR (96.8\%) but a markedly lower TPR (47.7\%), resulting in a lower MCC than projection only (55.1\% vs. 61.0\%). Since its AUPR remains close to that of projection only (65.8\% vs. 67.7\%), the ranking ability of the score is largely preserved, and the drop in sensitivity mainly reflects the decision threshold rather than a loss of discriminative power. This suggests that, in simpler tasks dominated by spatial accuracy, the relational term brings limited additional information. Overall, these results indicate that effective failure detection requires capturing both absolute trajectory deviations and interaction-level structure, with their relative importance depending on the semantic complexity of the task.

A.3 Impact of Spatio-Temporal Modeling

To rigorously evaluate the contribution of our combined GNN–Transformer architecture, we conduct an ablation study isolating the spatial and temporal inductive biases. We compare the full model against three degraded baselines:

• Full model (GnnT): Our proposed architecture, which uses an Edge-aware Graph Attention Network (EGAT) to explicitly model spatial interactions and physical constraints between objects, followed by a Transformer encoder to capture long-term temporal dependencies.
• Transformer-only (Temporal without Spatial Priors): The graph encoder is completely removed. Raw object coordinates and semantic embeddings are linearly projected into a hidden space and fed directly to the Transformer. This baseline tests whether the self-attention mechanism alone is sufficient to implicitly discover inter-object physical correlations without the structural priors provided by the GNN.
• GNN-only (Spatial without Temporal Attention): The temporal sequence modeling is replaced by a static feed-forward network. Spatial embeddings are still extracted frame-by-frame via the GNN, but they are subsequently concatenated along the temporal axis and mapped to future predictions via a Multi-Layer Perceptron (MLP). This isolates the value of the Transformer’s sequence-awareness.
• MLP (No explicit priors): A naive baseline where both the GNN and the Transformer are removed. The kinematic data of the scene over the observation window is flattened and processed by a simple MLP. This serves as a lower bound, representing a purely data-driven regression without structural or sequential inductive biases.

Results and Discussion

The empirical results, summarized in Table 3, show that the naive MLP baseline yields the lowest AUPR and MCC on both tasks. Its high TPR on Table-setting (88.7\%) comes at the cost of a very low TNR (46.2\%), meaning that it flags a large fraction of nominal episodes as failures. Conversely, the full GnnT architecture achieves the highest MCC on both tasks and the highest AUPR on Table-setting. This gap validates our hypothesis that explicitly modeling spatio-temporal priors is essential for robust failure detection in multi-object manipulation.

The ablation further reveals that the benefit of each inductive bias depends on the task’s primary failure modes. In the highly structured Table-setting task, anomalies frequently stem from incorrect geometric configurations or collisions. Without explicit relational modeling, the Transformer-only baseline struggles to capture these spatial constraints and clearly underperforms the GNN-only model (AUPR of 66.2\% vs. 78.1\%, MCC of 66.6\% vs. 71.2\%), and adding temporal attention on top of the GNN further improves both metrics (80.0\% AUPR and 73.1\% MCC for GnnT). In the Waste-sorting task, where the topological structure is less critical and anomalies often manifest as kinematic or timing deviations (e.g., missing an object), the differences between the non-MLP variants are much smaller and largely within one standard deviation. The sequence-aware Transformer-only model reaches a TPR and MCC on par with GnnT (82.3\% / 52.2\% vs. 81.8\% / 53.1\%), whereas the temporally blind GNN-only model trades sensitivity for specificity (TPR of 70.8\%, TNR of 82.9\%) and obtains a lower MCC (48.6\%). By combining both components, GnnT obtains the best MCC on both tasks, with the lowest variance on Waste-sorting (\pm 1.9), making it the most reliable configuration across spatial and temporal failure regimes.

A.4 Prediction vs Density Estimation

We compare the predictive formulation with a likelihood-based alternative. Results are presented in Table 4.

• GnnT, Prediction-based (ours). The model predicts future trajectories and uses prediction error as the failure score.
• Density-based. A normalizing flow is trained on latent representations to estimate likelihood.

A.4.1 Latent Density Estimation via Normalizing Flows

Instead of directly predicting the future trajectories of objects, we formulate the failure detection task as estimating the exact likelihood of the observed spatio-temporal dynamics. Let \mathbf{h}_{t}\in\mathbb{R}^{D} be the latent representation of the scene at time t, extracted by the Spatio-Temporal Transformer from the observation window. The distribution of nominal interactions p_{H}(\mathbf{h}_{t}) is highly complex and intractable.

To model this density, we employ a Normalizing Flow (NF) parameterized by \theta. The flow applies a sequence of invertible, differentiable transformations f_{\theta} to map the complex latent space \mathbf{h}_{t} into a simple, tractable base distribution, typically an isotropic multivariate Gaussian \mathbf{z}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I}).

Using the change of variables formula, the exact log-likelihood of a nominal representation is given by:

\log p_{H}(\mathbf{h}_{t})=\log p_{Z}(f_{\theta}(\mathbf{h}_{t}))+\log\left|\det\left(\frac{\partial f_{\theta}(\mathbf{h}_{t})}{\partial\mathbf{h}_{t}}\right)\right|

where p_{Z} is the standard Gaussian density function, and the second term is the log-determinant of the Jacobian of the transformation, accounting for the change in volume induced by f_{\theta}.

Training Objective.

The model is trained end-to-end to maximize the log-likelihood of the nominal data representations. Since \mathbf{z}_{t}=f_{\theta}(\mathbf{h}_{t}) follows a standard normal distribution, maximizing \log p_{H}(\mathbf{h}_{t}) is equivalent to minimizing the negative log-likelihood loss \mathcal{L}_{\text{NF}}:

\mathcal{L}_{\text{NF}}=\frac{1}{2}\|f_{\theta}(\mathbf{h}_{t})\|_{2}^{2}-\log\left|\det\left(\frac{\partial f_{\theta}(\mathbf{h}_{t})}{\partial\mathbf{h}_{t}}\right)\right|

Inference and Failure Scoring.

During deployment, the Normalizing Flow acts as an explicit density estimator. For a new observation window, the model computes the latent representation \mathbf{h}_{t} and its corresponding negative log-likelihood. The failure score S_{t} is directly defined as S_{t}=\mathcal{L}_{\text{NF}}(\mathbf{h}_{t}). A high score S_{t} indicates that the observed spatio-temporal interaction pattern has a low probability under the learned nominal distribution.

As shown in Table 4, the predictive formulation (GnnT) matches or outperforms the density estimation baseline on the balanced metrics while being more stable. On Waste-sorting, GnnT detects substantially more failures (TPR of 81.8\% vs. 70.2\%) at an equivalent false positive rate (TNR of 79.9\% vs. 80.3\%), which translates into an 8.0-point gain in MCC. On Table-setting, both formulations reach comparable AUPR (80.0\% vs. 78.2\%) and MCC (73.1\% vs. 72.9\%), but operate at different points: the density-based model is slightly more sensitive (TPR of 87.2\% vs. 83.6\%), while GnnT produces fewer false alarms (TNR of 92.4\% vs. 90.4\%). The most marked difference lies in stability: the density-based model exhibits considerably higher variance across seeds, reaching \pm 11.7 in AUPR and \pm 9.2 in MCC on Table-setting, compared to \pm 7.3 and \pm 6.6 for GnnT. This suggests that fitting a normalizing flow to high-dimensional, complex spatio-temporal latents yields less reliable decision boundaries, whose quality strongly depends on the training run. Conversely, framing failure detection as future trajectory prediction enforces a strict structural prior on nominal behavior. By directly computing errors in the geometric observation space rather than relying on abstract latent likelihoods, the predictive formulation provides a more consistent separation between genuine failures and benign variations across tasks and seeds.

Appendix B Method Details

B.1 Trajectory Extraction

Object trajectories are extracted from RGB observations using a two-stage pipeline combining vision-language-based object detection and segmentation-based multi-object tracking. The pipeline is designed to be fully automated, requiring only a natural language description of the task to identify and track all task-relevant entities throughout an episode.

Stage 1: Object detection via VLM (Qwen3-VL)

The first frame of each episode is passed to a vision-language model (Qwen3-VL-8B-Instruct [3]) together with a task-specific prompt that enumerates the expected task-relevant objects. For each object, the model is asked to predict a bounding box in normalised coordinates [0,1000]^{2}, a fine-grained semantic label, and a coarse-grained label. To improve robustness to occasional parsing failures, the model is queried up to seven times with a temperature annealing schedule (T_{k}=0.10+(k-1)\times 0.15, k\in\{1,\ldots,7\}). A response is accepted as valid only if all expected objects are returned with non-degenerate bounding boxes and non-empty labels. If the maximum number of attempts is exceeded, missing detections are flagged. This initialisation step is performed offline prior to deployment and therefore imposes no latency on real-time execution. The prompts used for each task are provided below.

Stage 2: Multi-object tracking via SAM2

The bounding boxes produced in Stage 1 initialise a SAM2 video tracking session [31]. At each subsequent frame, the segmentation masks are propagated using the SAM2 recurrent memory mechanism, and the 2D centroid of each object’s mask is recorded as its spatial state \mathbf{p}_{i,t}\in\mathbb{R}^{2}. When the mask of an object degenerates (empty support), the last valid centroid is carried forward (forward-fill imputation) so that all trajectories remain temporally aligned, while a validity label is switch from 1 to 0 to notify the missing object. If the object reappears later, the trajectory is interpolated between the last known position and the new position. The robot end-effector is treated as an additional object and tracked through the same pipeline from an initial bounding box provided by the VLM. Illustration of an example of the process is presented figure 6.

Real-time deployment characteristics

The VLM initialisation is performed entirely offline; once bounding boxes are obtained, the real-time tracking pipeline relies solely on SAM2. In its optimised streaming configuration, SAM2 operates at approximately 44\text{\,}\mathrm{f}ps on an NVIDIA A100 GPU with 5 tracked objects, well above the 30\text{\,}\mathrm{f}ps acquisition rate of the camera used in our experiments. The resulting end-to-end perception latency is therefore bounded by the single-frame forward pass of SAM2 and does not accumulate over an episode.

Task-specific VLM prompts

For reproducibility, we report the exact prompts supplied to Qwen3-VL for each task in our benchmark. Each prompt follows the same schema: a brief scene description, the user task string, a numbered list of the expected objects with their semantic categories, and an instruction to return a strictly formatted JSON response.

Table 5 summarises the object lists used per task. Full prompt strings are reproduced verbatim in the supplementary material.

All prompts share the following preamble and JSON schema:

B.2 Trajectory Extraction Robustness

B.2.1 VLM Initialization Robustness

The first step of our pipeline relies on a Vision-Language Model (VLM) to perform open-vocabulary object detection and establish the initial bounding boxes for subsequent tracking. To evaluate its robustness, we tested the Qwen-VL model on our dataset (341 episodes) both with and without our standardization loop.

As shown in Table 6, the base Qwen model achieves a correct detection rate of 92.4%. However, it occasionally misses some of the objects (5.3%) and suffers from identity confusion (1.8%) with a wrong label attribution, and localization inaccuracies (0.6%). By introducing our standardization verification loop (Qwen_WithStd), which re-prompts the model if the expected number of objects is not found, we attenuate the missing objects rate of 2.1%, raising the overall success rate to 94.4%.

B.2.2 Tracking Robustness

To ensure the VLM errors do not impact the tracking evaluation, we evaluated the SAM2 tracker’s robustness using ground-truth initial bounding boxes. We categorize the tracking performance into four levels: Perfect (object never lost), Recovered (object temporarily lost due to occlusion but recovered before the end of the task), Missing (object definitively lost by the tracker), and Fusion (tracker confused the object with another nearby object).

B.3 GnnT - Model Architecture

The spatial encoder is implemented using a Graph Attention Network (GATv2), while temporal dependencies are modeled using a multi-layer Transformer encoder. The prediction head consists of a shared MLP applied independently to each object embedding.

Background

Spatio-temporal forecasting in multi-agent or multi-object scenarios requires simultaneously resolving relational structures and temporal dynamics. While standard Graph Neural Networks (GNNs) effectively capture spatial topologies, they lack mechanisms for long-term sequence modeling. Conversely, Transformers excel at sequential processing but lack structural priors for geometric arrangements. The GnnT (Graph Neural Network - Transformer) architecture marries these paradigms, utilizing an edge-aware spatial encoder to independently process intra-frame geometry, followed by a sequence model to capture inter-frame kinematics.

Hypothesis

We hypothesize that nominal robotic manipulation is governed by strict, predictable physical laws and interaction patterns (e.g., collisions, grasping semantics, kinematic constraints). By explicitly forcing a network to predict future states from an observation window, it should internalize these underlying rules. The dynamic attention mechanism of the GATv2 is expected to learn to attend heavily to interacting entities (e.g., the end-effector approaching a target), while the Transformer learns temporal delays, inertia, and task phases. Consequently, mechanical or procedural failures will intrinsically violate these learned spatio-temporal priors, manifesting as a sharp spike in the prediction error (\mathcal{L}_{\text{pos}} and \mathcal{L}_{\text{edge}}), which serves as a highly discriminative anomaly signal.

Implementation details

The model takes as input a sequence of m graphs containing C objects. Let the raw kinematic state of object i at time t be \mathbf{x}_{i,t}\in\mathbb{R}^{2}. To provide semantic identity, each object’s discrete category c_{i}\in\{1,\dots,C\} is mapped to a continuous embedding \mathbf{e}_{i}\in\mathbb{R}^{16}. During training, we inject Gaussian noise \epsilon\sim\mathcal{N}(0,\sigma^{2}\mathbf{I}) with \sigma=0.01 to the coordinates to prevent overfitting and encourage the learning of robust physical dynamics rather than trivial identity mappings. The initial node feature vector is thus the concatenation:

\mathbf{v}_{i,t}^{(0)}=[\,(\mathbf{x}_{i,t}+\epsilon)\parallel\mathbf{e}_{i}\,]\in\mathbb{R}^{18}

The spatial encoder employs a two-layer Edge-aware GATv2 (EGAT). Let \mathbf{V}_{t}^{(l)} and \mathbf{E}_{t} denote the set of node and edge features at time t for layer l. The spatial embedding is refined iteratively:

\mathbf{v}_{i,t}^{(1)}=\text{ELU}\left(\text{EGAT}^{(1)}\big(\mathbf{v}_{i,t}^{(0)},\mathbf{E}_{t}\big)\right),\quad\mathbf{v}_{i,t}^{(2)}=\text{ELU}\left(\text{EGAT}^{(2)}\big(\mathbf{v}_{i,t}^{(1)},\mathbf{E}_{t}\big)\right)

where \text{ELU}(\cdot) is the Exponential Linear Unit activation used to maintain non-linear representational capacity while preventing dying gradients, and \mathbf{v}_{i,t}^{(2)}\in\mathbb{R}^{d_{h}} (with d_{h}=128) represents the spatially contextualized node embedding.

To model sequence dynamics, standard sinusoidal temporal positional encodings \mathbf{p}_{t}\in\mathbb{R}^{d_{h}} are added to the spatial embeddings: \tilde{\mathbf{h}}_{i,t}=\mathbf{v}_{i,t}^{(2)}+\mathbf{p}_{t}. The sequence of m spatial graphs is then flattened into a single contiguous token sequence:

\mathcal{S}_{in}=\big[\tilde{\mathbf{h}}_{1,t-m+1},\dots,\tilde{\mathbf{h}}_{C,t-m+1},\dots,\tilde{\mathbf{h}}_{1,t},\dots,\tilde{\mathbf{h}}_{C,t}\big]\in\mathbb{R}^{mC\times d_{h}}

This flattened topology permits unrestricted, fine-grained self-attention across both the temporal and spatial axes. The sequence is processed by a 4-layer Transformer Encoder to yield \mathcal{S}_{out}=\text{Transformer}(\mathcal{S}_{in}).

Finally, we extract only the C tokens corresponding to the most recent observation at timestep t (i.e., the last C elements of \mathcal{S}_{out}). Let \mathbf{z}_{i,t}\in\mathbb{R}^{d_{h}} denote this output token for object i. The future trajectory over horizon L is directly regressed via a two-layer Multi-Layer Perceptron (MLP) with a ReLU activation:

\widehat{\mathbf{Y}}_{i,t}=\mathbf{W}_{2}\max(0,\mathbf{W}_{1}\mathbf{z}_{i,t}+\mathbf{b}_{1})+\mathbf{b}_{2}\quad\in\mathbb{R}^{L\times 2}

where \mathbf{W}_{1}\in\mathbb{R}^{2d_{h}\times d_{h}} and \mathbf{W}_{2}\in\mathbb{R}^{2L\times 2d_{h}} are the shared projection weights applied independently to each object token.

Hyperparameters

The architecture and training routine are governed by the hyperparameters detailed in Table 8. These values were selected empirically to balance the temporal receptive field, representational capacity, and computational efficiency.

B.4 Trajectory Projection and Correlation (TPC) Formulation

The TPC method evaluates kinematic and relational anomalies by comparing inference trajectories against a set of nominal expert demonstrations. Let a tracked object’s trajectory of length T be defined as P=\{p_{1},\dots,p_{T}\} with p_{t}\in\mathbb{R}^{2}, and let the set of nominal expert trajectories for the corresponding object class be \mathcal{E}=\{V^{(1)},\dots,V^{(K)}\}. Prior to distance computation, trajectories are filtered to remove static segments, ensuring the evaluation focuses strictly on kinematic progression.

1. Projection-Based Kinematic Scoring:

The spatial deviation of a point p_{t} from an expert trajectory V (modeled as a piecewise linear polyline) is defined by the orthogonal point-to-polyline distance, denoted d(p_{t},V). The spatial failure at time t is taken as the distance to the nearest expert:

d^{*}(p_{t})=\min_{V\in\mathcal{E}}d(p_{t},V)

To account for the inherent natural variability of the task, these distances are standard-normalized using the empirical mean \mu and standard deviation \sigma of the pairwise projection distances computed within the expert set itself:

\hat{d}(p_{t})=\frac{|d^{*}(p_{t})-\mu|}{\sigma+\epsilon}

where \epsilon is a small constant for numerical stability. The individual projection score aggregates these normalized deviations over time. To penalize errors that compound towards the end of the execution, temporal weights w_{t} (e.g., derived from an exponential decay function) are applied:

S_{\text{proj}}=\sum_{t=1}^{T}w_{t}\hat{d}(p_{t})

2. Relational Correlation Penalty:

To capture higher-level interaction failures, the method evaluates the synchronized movement between objects. For any pair of objects i and j resampled to a uniform length T, let \rho(x^{i},x^{j}) and \rho(y^{i},y^{j}) be the Pearson correlation coefficients of their respective spatial coordinates over time. The pairwise spatial correlation is averaged across dimensions:

C_{i,j}=\frac{1}{2}\big(\rho(x^{i},x^{j})+\rho(y^{i},y^{j})\big)

The relational failure penalty is computed as the \ell_{1} distance between the inference correlation matrix (C^{\text{inf}}) and the reference correlation matrix (C^{\text{exp}}) derived from the expert dataset:

S_{\text{corr}}=\sum_{i=1}^{N}\sum_{j>i}\left|C^{\text{inf}}_{i,j}-C^{\text{exp}}_{i,j}\right|

where N is the number of tracked objects.

3. Final failure Score:

Depending on the configuration, the method can evaluate purely projection-based errors, purely relational errors, or a combination of both. The combined failure score is the sum of the individual projection scores and the relational penalty:

S_{\text{total}}=S_{\text{proj}}+S_{\text{corr}}

B.5 Conformal Prediction

B.5.1 Calibration Data

Calibrating the decision band requires a reference set of nominal trajectories, denoted \mathcal{Z}_{N}=\{\tau^{(1)},\dots,\tau^{(N)}\}. The expert set is partitioned once per seed: one half trains the detector (or, for the training-free scorers, serves as the reference set of expert trajectories), and the complementary half forms \mathcal{Z}_{N}. No demonstration is used both to fit the detector and to calibrate its threshold.

At every timestep, the per-object non-conformity scores are summed across the tracked objects to yield a single scalar S_{t}, and non-finite values are mapped to zero before calibration.

B.5.2 Sequential Band Construction

Rather than a single global threshold, we calibrate a time-varying band, following the functional split-conformal construction of Diquigiovanni et al. [11] The calibration set is split a second time into two disjoint subsets, \mathcal{D}_{A} (a fraction \rho of \mathcal{Z}_{N}) and \mathcal{D}_{B} (the remainder). \mathcal{D}_{A} estimates the shape of the band and \mathcal{D}_{B} its width, so that the width is never fitted on the same data that defines the centre.

From \mathcal{D}_{A} we compute the pointwise mean score

\mu_{t}=\frac{1}{|\mathcal{D}_{A}|}\sum_{i\in\mathcal{D}_{A}}S_{t}^{(i)}

together with a modulation function s(t) that sets how the band width is distributed over time. We consider two variants:

s_{1}(t)=\frac{1}{T},\qquad s_{2}(t)=\max_{i\in\mathcal{D}_{A}}\left|S_{t}^{(i)}-\mu_{t}\right|.

The constant modulation s_{1} yields a band that tracks the mean expert profile with a uniform offset, whereas s_{2} widens the band wherever the expert demonstrations themselves disagree.

The width is then calibrated on \mathcal{D}_{B}. For each trajectory j\in\mathcal{D}_{B} we take the maximum deviation from the mean profile, normalised by the modulation function:

h^{(j)}=\max_{t\in[1,T]}\frac{\left|\mu_{t}-S_{t}^{(j)}\right|}{s(t)},

and set h to the empirical (1-\alpha)-quantile of \{h^{(j)}\}_{j\in\mathcal{D}_{B}}. Taking the maximum over the whole horizon before the quantile makes the coverage statement simultaneous over the rollout rather than pointwise at a single timestep. The resulting threshold is

\eta_{t}=\mu_{t}+h\cdot s(t).

B.5.3 Online Deployment and Evaluation

At test time the framework computes the instantaneous score S_{t} and applies the one-sided decision rule \mathbb{1}(S_{t}>\eta_{t}): only high scores indicate out-of-distribution behaviour, and a failure is flagged at the earliest timestep where the band is breached. Test rollouts may outlast the calibration demonstrations; beyond the calibration horizon the last band value \eta_{T} is held constant.

Because h is the empirical quantile of the maximum normalised deviation over \mathcal{D}_{B}, the fraction of nominal trajectories flagged at any point of their execution is controlled at approximately \alpha in the exchangeable regime. We use the plain empirical quantile rather than the finite-sample corrected \lceil(1-\alpha)(|\mathcal{D}_{B}|+1)\rceil order statistic, so the guarantee is asymptotic; with the calibration set sizes used here (|\mathcal{D}_{B}| of a few tens of demonstrations) the distinction is not negligible and the nominal level should be read as indicative.

We report s_{1} throughout, and sweep \alpha over \{0.05,0.10,\dots,0.95,0.99\} for both modulation variants; headline numbers average the operating points \alpha\in[0.80,0.95], and every (\alpha,s) configuration is reported in the supplementary material. All results are averaged over 10 seeds, each seed redrawing the expert partition, the \mathcal{D}_{A}/\mathcal{D}_{B} split and the network initialisation.

B.6 Pipeline Latency and Real-Time Capability

A critical requirement for our Robot-Watchdog pipeline is the ability to operate in real-time alongside the robot’s control loop. We profiled the latency of each component on an NVIDIA RTX 3090 GPU. Table 9 details the execution time of each module. The VLM initialization is an expensive operation (\sim7.0s) but is only executed once at the beginning of the task. For the continuous monitoring loop, SAM2 segmentation achieves highly efficient real-time performance at approximately 22.72 ms per frame (44 FPS). On the anomaly detection side, the Graph Neural Network Transformer (GnnT) inference takes 19.87 ms per frame, while TPC sliding window inference is extremely lightweight, requiring only 7.00 ms. This low computational overhead enables the overall pipeline to operate in true real-time. When using TPC, the end-to-end processing time is 29.72 ms per frame (33.6 FPS), comfortably exceeding standard 30 FPS control loop requirements. When using the more complex GnnT, the pipeline runs at 23.5 FPS, which remains fully suitable for standard 15-20 FPS robotic control setups.

Appendix C Experimental Details

C.1 Datasets

We evaluate Robot-WATCHDOG on two datasets: the BotFails benchmark [32] and a new dataset collected for this work. Together, they provide complementary coverage of controlled failure scenarios and more realistic, dynamically varying environments.

BotFails.

BotFails is a recently introduced dataset for robotic failure detection, collected via teleoperation on a real robotic platform. It includes multimodal observations (vision, proprioception, and language) across 10 tasks spanning domestic and industrial settings, with annotations for both general anomalies and task-relevant failures.

In our evaluation, we select a subset of tasks compatible with our object-centric assumptions. Specifically, we retain tasks involving rigid, non-deformable, and visually distinguishable objects, enabling consistent tracking and relational reasoning. Tasks involving fluids or deformable materials (e.g., pouring) are excluded, as they violate the assumption of persistent object identity.

Ours.

We introduce a new dataset, see figure 7, composed of two manipulation tasks designed to increase both relational and dynamic complexity:

• Table-setting: a structured assembly task where the robot must place objects (e.g., plates, utensils) in predefined spatial configurations and correct temporal order. This task emphasizes relational correctness (object identity, placement location, ordering).
• Waste sorting: a dynamic task where the robot sorts objects on a moving conveyor belt. Unlike static tabletop setups, object positions evolve continuously, introducing variability in initial conditions and interaction timing.

For each task, we collect two types of datasets: (i) expert demonstrations, used for training, and (ii) failure datasets, used for evaluation and containing both nominal and failure trajectories.

Data collection and modalities.

The table-setting task is performed using an ALOHA robot, while the waste-sorting task uses a SO-100 platform. Observations include RGB video streams and proprioceptive states (joint positions), along with corresponding action commands. The table-setting setup includes multiple camera viewpoints (external and wrist-mounted), whereas the sorting task uses a top-down camera.

Discussion.

Our dataset complements BotFails in two key aspects. First, it provides a larger number of expert demonstrations, which is beneficial for training predictive models of scene dynamics. Second, it introduces more challenging conditions for failure detection, particularly in the sorting task where object motion induces continuous distribution shift. This setup allows us to evaluate robustness to both relational or identity errors (e.g., wrong object or placement) and dynamic deviations (e.g., delayed or missed interactions).

C.2 Baselines

We compare our methods against classical and learning-based failure detection methods, which characteristics are presented in table 11.

C.2.1 Arc-Length Prefix RMS (AP-RMS)

The AP-RMS baseline detects kinematic deviations by evaluating the cumulative spatial error of a trajectory’s prefix relative to a set of nominal expert trajectories. To ensure the comparison is invariant to execution speed and sampling rates, all trajectories are spatially aligned via arc-length parameterization.

1. Arc-Length Resampling:

Let a raw trajectory consisting of N points be defined as P=\{p_{1},\dots,p_{N}\} with p_{i}\in\mathbb{R}^{2}. The cumulative arc-length s_{i} at step i is computed as:

s_{i}=\sum_{j=2}^{i}\|p_{j}-p_{j-1}\|_{2},\quad\text{with }s_{1}=0

The sequence is parameterized by normalized arc-length \bar{s}_{i}=s_{i}/s_{N}\in[0,1]. We construct a continuous trajectory function f(\bar{s}) via linear interpolation. The trajectory is then uniformly resampled into a fixed length T, producing the spatially aligned trajectory \tilde{P}=\{\tilde{p}_{1},\dots,\tilde{p}_{T}\}, where \tilde{p}_{\tau}=f\left(\frac{\tau-1}{T-1}\right). This procedure is applied identically to the set of expert trajectories to form the aligned expert set \tilde{\mathcal{E}}=\{\tilde{V}^{(1)},\dots,\tilde{V}^{(K)}\}.

2. Prefix RMS Distance:

To evaluate the deviation of the execution up to a given progression step t\in[1,T], we extract the trajectory prefix \tilde{P}_{1:t}=\{\tilde{p}_{1},\dots,\tilde{p}_{t}\}. The Root Mean Square (RMS) distance between the observed prefix and the corresponding prefix of an expert \tilde{V}^{(k)}_{1:t} is defined as:

d_{\text{RMS}}\big(\tilde{P}_{1:t},\tilde{V}^{(k)}_{1:t}\big)=\sqrt{\frac{1}{t}\sum_{\tau=1}^{t}\big\|\tilde{p}_{\tau}-\tilde{v}^{(k)}_{\tau}\big\|_{2}^{2}}

3. Final failure Score:

At each progression step t, the spatial failure is quantified as the distance to the closest expert prefix. This local deviation is given by:

d^{*}(t)=\min_{\tilde{V}\in\tilde{\mathcal{E}}}d_{\text{RMS}}\big(\tilde{P}_{1:t},\tilde{V}_{1:t}\big)

The overall failure score for the complete trajectory is the temporal average of these minimal prefix deviations, capturing the continuous divergence from the nominal behavior manifold:

S_{\text{total}}=\frac{1}{T}\sum_{t=1}^{T}d^{*}(t)

C.2.2 Arc-Length Prefix Fréchet (AP-Fréchet) Formulation

The AP-Fréchet baseline detects structural and geometric anomalies by evaluating the shape discrepancy of trajectory prefixes relative to nominal expert demonstrations. Similar to the AP-RMS method, this approach enforces temporal invariance by spatially aligning trajectories via arc-length parameterization.

1. Arc-Length Resampling:

We apply the identical arc-length normalization procedure defined for AP-RMS. Let the uniformly resampled observed trajectory of length T be \tilde{P}=\{\tilde{p}_{1},\dots,\tilde{p}_{T}\}, and the corresponding aligned expert set be \tilde{\mathcal{E}}=\{\tilde{V}^{(1)},\dots,\tilde{V}^{(K)}\}.

2. Discrete Fréchet Distance:

To evaluate the geometric divergence up to a progression step t\in[1,T], we extract the trajectory prefixes \tilde{P}_{1:t} and \tilde{V}^{(k)}_{1:t}. The Discrete Fréchet distance considers all possible ordered couplings between the points of the two polygonal curves. Let an order-preserving coupling sequence L be a sequence of index pairs (i,j) such that (i_{1},j_{1})=(1,1), (i_{|L|},j_{|L|})=(t,t), and successive pairs transition by (+1,0), (0,+1), or (+1,+1).

The Discrete Fréchet distance is defined as the minimum over all valid coupling sequences of the maximum Euclidean distance between coupled points:

d_{\text{F}}\big(\tilde{P}_{1:t},\tilde{V}^{(k)}_{1:t}\big)=\min_{L}\max_{(i,j)\in L}\big\|\tilde{p}_{i}-\tilde{v}^{(k)}_{j}\big\|_{2}

In practice, this distance is computed efficiently utilizing dynamic programming.

3. Prefix failure and Normalization:

At each progression step t, the geometric failure is evaluated against the closest expert prefix:

d^{*}(t)=\min_{\tilde{V}\in\tilde{\mathcal{E}}}d_{\text{F}}\big(\tilde{P}_{1:t},\tilde{V}_{1:t}\big)

To account for the natural geometric variance of the specific object’s class, d^{*}(t) is standardized. Let \mu and \sigma be the empirical mean and standard deviation of the pairwise Fréchet distances evaluated among the full-length resampled experts in \tilde{\mathcal{E}}. The normalized failure score at step t is formulated as:

\hat{d}(t)=\frac{d^{*}(t)-\mu}{\sigma+\epsilon}

where \epsilon is a small constant ensuring numerical stability. The final per-step scores form a trajectory of geometric divergence tracking the severity of the failure over the execution sequence.

C.2.3 Soft Dynamic Time Warping (Soft-DTW) Formulation

The Soft-DTW baseline detects anomalies by assessing the global spatio-temporal dissimilarity between a full observed trajectory and a set of nominal expert trajectories. By utilizing dynamic time warping, this approach is highly resilient to temporal scaling and non-linear shifts in execution speed.

1. Trajectory Preprocessing:

Let the observed trajectory be P=\{p_{1},\dots,p_{N}\} and the set of expert trajectories be \mathcal{E}=\{V^{(1)},\dots,V^{(K)}\}. To ensure computational tractability and enable batched parallel execution on GPU hardware, all trajectories are linearly downsampled to a maximum fixed length T_{max} (if their original length exceeds it) and zero-padded.

2. Soft-DTW Distance:

For a given test trajectory P and an expert trajectory V (both of length T\leq T_{max}), we define the pairwise Euclidean distance matrix \Delta\in\mathbb{R}^{T\times T}, where \Delta_{i,j}=\|p_{i}-v_{j}\|_{2}^{2}. Let \mathcal{A}(T,T) represent the set of all valid binary alignment matrices (paths) between two sequences of length T. The standard DTW distance minimizes the inner product \langle A,\Delta\rangle over all A\in\mathcal{A}.

Soft-DTW introduces a smoothing parameter \gamma>0 and replaces the non-differentiable \min operator with a soft-minimum, defined for a set of values \{x_{i}\} as:

\min{}^{\gamma}(x_{1},\dots,x_{n})=-\gamma\log\sum_{i=1}^{n}e^{-x_{i}/\gamma}

The Soft-DTW distance between P and V aggregates the costs of all possible alignment paths:

d_{\text{sDTW}}(P,V)=\min{}^{\gamma}_{A\in\mathcal{A}(T,T)}\langle A,\Delta\rangle

This softening ensures that the distance metric is less brittle to localized noise compared to the standard, strictly optimal DTW path.

3. Minimum Expert Distance and Normalization:

The raw trajectory failure is quantified as the minimum Soft-DTW distance to any expert demonstration in the nominal set:

d^{*}(P)=\min_{V\in\mathcal{E}}d_{\text{sDTW}}(P,V)

To determine statistical significance, this raw distance is standard-normalized. Let \mu and \sigma be the empirical mean and standard deviation of all pairwise Soft-DTW distances computed exclusively within the expert set \mathcal{E}:

\hat{d}(P)=\frac{|d^{*}(P)-\mu|}{\sigma+\epsilon}

where \epsilon is a small scalar added for numerical stability. The resulting scalar \hat{d}(P) serves as the final failure score for the evaluated trajectory.

C.3 Evaluation Metrics

We assess our framework in two stages: evaluating failure scores independently of decision thresholds, and evaluating the operational end-to-end binary failure detection. Unlike some prior works that evaluate failure detection solely at the episode level, our predictions are evaluated at a granular step level, where labels y\in\{0,1\} correspond to nominal behavior and task failure, respectively. Let TP, TN, FP, and FN denote the number of true positives, true negatives, false positives, and false negatives.

Raw Failure Scoring.

To evaluate the intrinsic discriminative power of the continuous anomaly scores S_{t}, we report the Area Under the Precision-Recall Curve (AUPR) and the maximum Matthews Correlation Coefficient (MCC). We favor AUPR because it is particularly informative under strong class imbalance and is sensitive to the precision-recall trade-off. Furthermore, to capture the optimal threshold-independent balance between precision and recall, we report the maximum MCC achieved across all possible thresholds. MCC evaluates the quality of binary classifications on skewed distributions, yielding a value between -1 and 1 (with 0 indicating random correlation and 1 indicating perfect correlation):

\text{MCC}=\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}

End-to-End Detection Pipeline.

To evaluate the practical deployment readiness of the complete pipeline, we assess the binary decisions produced by applying the Conformal Prediction (CP) thresholds. This measures the entire system’s ability to safely flag critical failures while remaining robust to benign variations. We report the True Positive Rate (TPR), True Negative Rate (TNR), and the threshold-calibrated MCC:

\text{TPR}=\frac{TP}{TP+FN},\quad\text{TNR}=\frac{TN}{TN+FP}

These metrics isolate distinct operational requirements: TPR (Recall) evaluates system safety by measuring the proportion of failures successfully intercepted, while TNR (Specificity) measures operational efficiency by quantifying the ability to ignore benign noise without triggering false alarms.

Appendix D Additional results

D.1 Threshold-independent results per task

Table 12 details Figure 3 per task for all nine methods, and adds AUROC.

Our two scores.

TPC is the best method on every task in AUPR and maximum MCC, except AUPR on Dish storing, where logpZO is higher (0.670 vs. 0.639). It outperforms GnnT on all six tasks in both metrics, including the table-setting task of the Watchdog dataset (AUPR 0.963 vs. 0.800) for which 100 expert demonstrations are available, so the number of demonstrations alone does not explain the gap. Waste sorting is the only task where GnnT comes close: its maximum MCC is within 0.02 of TPC (0.531 vs. 0.551), its AUROC is higher (0.854 vs. 0.762, on all 10 seeds) and it reaches the higher recall after calibration (Table 13), while TPC keeps a much higher AUPR (0.658 vs. 0.497, on all 10 seeds). We attribute this to the continuous motion of the belt: initial states never repeat, which penalizes a direct comparison to stored demonstrations more than a model that predicts object motion from the observed past.

Comparison with baselines.

GnnT outperforms every baseline on dataset averages but not on every task: logpZO is better on Dish storing (AUPR 0.670 vs. 0.548), AP-RMS on Vegetable sorting (AUPR 0.711 vs. 0.690) and AP-Fréchet on Groceries sorting in maximum MCC (0.527 vs. 0.497). No baseline, however, is strong across tasks. Trajectory-matching baselines are the most task-dependent: AP-RMS reaches an AUPR of 0.711 on Vegetable sorting, but all three fall below the chance level on Waste sorting (0.139–0.166 vs. a failure-step prevalence of 0.22), and they are the least stable across seeds (standard deviation up to 0.373 AUPR for AP-Fréchet on the Watchdog table-setting task), since their score depends on which demonstrations form the reference set. Because they compare each object independently, they cannot represent relational errors such as swapped placements. Density (logpZO, lopO), reconstruction (AE-Recon) and visual (FIDeL) baselines are stable across seeds (standard deviation at most 0.054 AUPR) but remain task-dependent: logpZO is the strongest baseline on Dish storing (0.670) yet drops to 0.189 on Groceries sorting, and FIDeL goes from 0.475 to 0.210 on the same two tasks.

D.2 End-to-end results per task

Table 13 details Figure 5 per task, with TPR and TNR. After calibration, TPC is the best method on five of the six tasks, with an MCC between 0.519 (Dish storing) and 0.819 (Vegetable sorting). On Waste sorting, AP-Fréchet (0.553) is ahead of both TPC (0.470) and GnnT (0.449), which are tied (TPC ahead on 5 of 10 seeds); GnnT trades a lower TNR (0.876 vs. 0.897) for a higher TPR (0.587 vs. 0.551). Trajectory matching benefits the most from calibration: its calibrated MCC is far above its maximum MCC (e.g., AP-Fréchet on Waste sorting, 0.553 vs. 0.192), because the time-varying band absorbs the drift of prefix-based scores along the episode, which a single global threshold cannot do. GnnT benefits less, and its calibrated MCC is below that of AP-Fréchet on both dataset averages (0.347 vs. 0.452 and 0.571 vs. 0.630).