HelpMeOut: A Neuropsychology-Grounded Benchmark for Error-Aware Monitoring of Household Task Execution

Abstract

We introduce HelpMeOut, an embodied AI benchmark for error-aware monitoring of household task execution. Existing embodied household benchmarks typically assume that humans execute tasks correctly, but this assumption does not hold for cognitively impaired persons or older adults whose everyday activities produce structured execution errors. To address this gap, HelpMeOut operationalizes research on everyday action errors into 20,000 paired activity episodes, covering nine neuropsychologically motivated error categories. In each episode, agents observe a streaming human activity trajectory and must monitor whether the execution is unfolding correctly, diagnosing any structured errors as early as possible. We evaluate learned-monitor, symbolic, change-point, and frontier LLM prompting baselines. Results show that these methods perform poorly as error monitors, often missing when execution goes wrong. These results suggest a gap left by standard goal-inference and task-completion benchmarks, making HelpMeOut a testbed for error-aware assistive agents.

Keywords: Error-Aware Monitoring, Assistive Robotics, Embodied AI Benchmark

1 Introduction

Consider a cognitively impaired person making a cup of instant coffee. She adds the powder and pours hot water, but never stirs—the stirring step does not come to mind. Later, she takes a fork to add sugar, unaware that a spoon is the appropriate tool. These look like minor lapses, but to an assistive robot they pose a distinct problem: the person is not confused about her goal, nor is she asking for help—she is executing an everyday task under degraded cognitive control. How can the robot recognize, from partial household actions, that execution is going wrong?

Neuropsychological studies of everyday action show that such breakdowns are not random noise: everyday tasks often fail in structured ways such as omissions, substitutions, sequence errors, and perseverations [1, 2]. Detecting such errors and intervening at the right time is a core requirement for assistive robots supporting people with cognitive impairment and older adults. This is a large and growing population: cognitive impairments such as dementia affect tens of millions worldwide—dementia alone reaches 55 million people and is projected to grow to 139 million by 2050 [3, 4], and a further 12–18% of adults over 60 live with mild cognitive impairment, a condition often associated with deficits in instrumental activities of daily living [5]. Yet this capability is rarely evaluated in existing embodied household benchmarks.

Most such benchmarks share a common assumption: humans act coherently toward well-formed goals, and the agent’s central problem is inferring what the human wants to do. For cognitively impaired persons and older adults, however, the harder question is not only what the person is trying to do, but whether their ongoing execution is failing. A separate line of smart-home and robot-assisted systems does target execution failures directly [6, 7, 8], but typically relies on fixed sensor setups or predefined activity scripts, yielding task-specific signals rather than standardized, large-scale data for agent evaluation. What is missing is a scalable benchmark that tests step-level error monitoring from partial household activity streams. Figure 1 illustrates this setup with an instant-coffee episode.

We therefore introduce HelpMeOut, an embodied AI benchmark for error-aware monitoring of household task execution. Guided by the neuropsychological evidence above, HelpMeOut grounds its episodes in a goal-control model [9] whose parameters are calibrated against the per-category error rates reported in naturalistic-action studies of control and clinical populations [2, 10]: error type and frequency vary by error category and degree of cognitive impairment, rather than being sampled uniformly at random. Agents must therefore recognize errors whose structure reflects impaired everyday action, rather than artifacts of the sampling procedure. As in these tests, the task is known; the monitor must judge whether its execution goes wrong.

HelpMeOut makes three contributions. First, we formalise online error-aware monitoring as a task of deciding whether, when, and what kind of execution error occurs while a household task unfolds. Second, we build a generator that turns Activities of Daily Living (ADL)/Instrumental Activities of Daily Living (IADL) tasks into paired correct and errorful episodes, driven by a goal-control model from the neuropsychology of naturalistic action, and release a dataset of 20,000 paired episodes1. Third, we cast the task as a streaming evaluation protocol that separately measures error discrimination, step-level detection, latency, and type diagnosis. Experiments show that learned baselines struggle with step-level error detection and diagnosis, symbolic baselines with additional domain knowledge primarily identify incomplete tasks at episode termination without distinguishing error types, and frontier LLMs remain limited in step-level error localization and timely alerting.

2 Related Works

Everyday action errors.

In the past few decades, clinical neuropsychology studied everyday action impairment using performance-based instruments such as the Naturalistic Action Test (NAT), the Multi-Level Action Test (MLAT), and the Performance Assessment of Self-Care Skills (PASS) [1, 2, 11, 10, 12]. These tests ask participants to perform familiar multi-object tasks and score errors such as omissions, sequence errors, substitutions, additions, and perseverations. The goal-control account [9] draws on the Supervisory Attentional System (SAS) framework [13] and contention-scheduling models [14, 15], linking omissions to weak or decaying task goals and commission errors to failures to suppress competing goal activations, especially around subgoal transitions [16, 17]. An alternative connectionist account models the same error categories

through recurrent dynamics without explicit schema hierarchies [26]. More recently, these error taxonomies have been extended to characterise subtle, sub-clinical errors in mild cognitive impairment [27].

Assistive error detection systems.

Smart-home and robot-assisted systems have used sensors or predefined activity scripts to detect deviations and prompting opportunities [6, 7, 8]. Deployed prompting systems include COACH, which uses a POMDP over video observations to guide dementia patients [28, 29], and SmartPrompt2, which targets common omission errors [30]. Related datasets capture erroneous kitchen and procedural activities: CaptainCook4D records egocentric cooking videos in which participants deviate from recipes to induce errors [31], while the Kitchen Task Assessment dataset records sensor data from participants simulating typical dementia errors [32]. Existing methods also address procedural error detection: PREGO compares recognized steps against LLM-predicted steps [33], differentiable task-graph learning checks steps against a learned task graph [34], and MistExit aims to identify erroneous pre-segmented key steps as early as possible [35]. These works demonstrate the importance of detecting execution failures, but are often constrained by specific tasks, sensor setups, or errors that participants induce or simulate. As a result, they do not provide a scalable benchmark for evaluating whether models can detect neuropsychologically grounded error types as they unfold, with explicit onset labels and first-alarm timing metrics. Large-scale data collection from cognitively impaired users is also constrained by privacy and ethical concerns. HelpMeOut addresses these limitations by converting the neuropsychological taxonomy and goal-control account into automatically labelled episodes, with clinically calibrated error rates and deterministic step-level onsets, for streaming agent evaluation.

Embodied household benchmarks and simulators.

Monitoring an ongoing task is closely related to goal and plan recognition from partial observations [36, 37]. Classical methods usually assume that the observed agent acts rationally; bounded-rational variants relax this assumption by allowing sub-optimal plans [38], which is closer to the impaired executions considered in HelpMeOut; such work infers goals from imperfect behavior, whereas HelpMeOut asks whether, when, and how the execution itself goes wrong. LLMs are also increasingly used as embodied household planners [39, 40], motivating their evaluation as error monitors in our benchmark.

Two adjacent lines of work share HelpMeOut’s household setting. The first evaluates agents that infer human intent or resolve instructions from observed behavior and language, including HandMeThat [24], FISER [41], BALI [25], and Watch-and-Help [21]. The second provides scalable household task simulators and trajectory generators, including TextWorld [18], BEHAVIOR and BEHAVIOR-1K [22, 23], VirtualHome [20], ALFRED [19], and Habitat rearrangement [42]. Across both lines, however, the human action stream is assumed competent and the resulting trajectories are correct by construction. HelpMeOut keeps the scalable household-generation setting but adds neuropsychologically grounded errorful trajectories, paired correct references, and step-level onset labels for error-aware monitoring (Table 1).

3 The HelpMeOut Benchmark

3.1 Preliminaries: Neuropsychology of Everyday Action

HelpMeOut uses the neuropsychological literature on everyday action impairment to define both its error labels and its generation model. Performance-based instruments such as NAT, MLAT, and PASS identify recurring failures in familiar multi-object tasks, including omissions, sequence errors, substitutions, additions, and perseverations [1, 2, 11, 10, 12]. These categories recur across clinical populations such as CVA, traumatic brain injury, and dementia, and provide the basis for our benchmark labels.

For generation, we adopt the goal-control account [9], which separates failures of goal activation from failures of executive control. In our benchmark, goal-activation failures produce omissions, while executive-control failures produce commission errors such as substitutions, additions, and captures. This abstraction gives the generator two interpretable control factors: whether a required subgoal is omitted, and whether a competing action is inserted or substituted. Sec. 3.3 describes how these factors are instantiated in the error generator. Figure 2 summarizes this generation process. Additional background on the neuropsychological literature is provided in the supplementary material.

3.2 Basic Monitoring Setting

HelpMeOut is built over a symbolic household domain with 156 object types and 38 grounded action operators covering object manipulation, tool use, and state transformations. At evaluation step t, an agent observes the task name, the initial symbolic scene representation, and the action prefix so far, including the current action. The task is known to the monitor, as in the naturalistic-action tests on which our error categories are based: in the NAT and MLAT, the examiner names the task and its requirements before the person begins, and errors are scored against those requirements [11, 10]. Unlike the examiner, the monitor receives only the task name, not its formal requirements, and must infer what the task calls for. From this prefix, the agent must answer three questions: whether execution has already gone wrong, when the observable error onset occurred, and which error categories have appeared.

Each HelpMeOut episode consists of paired trajectories. A symbolic planner first generates a correct household execution from an ADL/IADL task template, and the error process then modifies that execution to obtain an errorful counterpart. The templates cover familiar multi-object routines such as food preparation, packing, cleaning, laundry, medication, and personal routines, following task families used in clinical and functional assessments of everyday action [10, 12, 43, 44, 45]. Formally, we write each episode as

e=\langle s_{0},\Omega_{\mathrm{correct}},\Omega_{\mathrm{err}},\mathcal{S}_{0},\ y_{0:T^{\prime}-1},\ O\rangle.

Here, s_{0} is the initial household state, \Omega_{\mathrm{correct}} is the reference execution, and \Omega_{\mathrm{err}} is the realized errorful execution of length T^{\prime}. \mathcal{S}_{0} is the initial symbolic scene representation exposed to agents, encoding object identities, locations, states, affordances, and spatial relations. The sequence y_{0:T^{\prime}-1} gives sparse onset annotations: a step is labelled with has_error=1 only when an injected error first becomes observable. O=\{(\mathrm{type}_{j},t_{j})\}_{j} records each error category and onset step, which supports step-level error detection, first-alarm timing, and cumulative error-type diagnosis. Subgoal boundaries and generation goals are retained as dataset metadata for constructing and analysing episodes; they are not shown to agents. Every episode ends with an explicit task-end event, at which the person declares the whole task finished.

An error’s observable onset is the earliest step at which it can be identified from the observed execution prefix and task constraints, using category-specific rules: the first use of a substitute object or tool, an out-of-order action, an action repetition, committing an unprocessed item, placing an unrelated item in the task container, acting on an unrelated object, or leaving an unrelated storage location empty-handed. An omission removes an entire subgoal block. Since valid subgoal orders vary, a missing subgoal may still be completed later; thus, omission becomes observable only at the task-end event. An earlier consequence, such as storing an unprocessed item, is instead labelled as anticipation under MLAT coding [11]. Computed automatically from s_{0}, \Omega_{\text{err}}, and the injected category, onset annotations are deterministic and reproducible without human raters.

3.3 Structured Execution Errors

Goal-Control error process.

Inspired by the goal-control account [9], we treat omissions as goal-activation failures and commission errors as executive-control failures. We operationalise this distinction by computing, for each subgoal block b_{k}, an activation level:

u_{k}=\theta_{\mathrm{goal}}(1-\theta_{\mathrm{decay}})^{k}+\epsilon_{k},\qquad\epsilon_{k}\sim\mathcal{N}(0,\sigma^{2}),

where \theta_{\mathrm{goal}} controls baseline goal salience and \theta_{\mathrm{decay}} captures accumulating cognitive load across subgoals. If u_{k}<\tau_{\mathrm{omit}}, the subgoal is omitted; otherwise, with probability \theta_{\mathrm{ctrl}}, a commission error is sampled from a task-conditioned distribution modulated by per-category feasibility. We calibrate \theta_{\mathrm{goal}},\theta_{\mathrm{decay}},\theta_{\mathrm{ctrl}} and the category weights w_{c} so that the marginal error distribution produced by the generator matches the per-category error rates reported for control and clinical populations by Schwartz et al. [10] and Giovannetti et al. [2]. The resulting impaired profile (\theta_{\mathrm{goal}}{=}0.48, \theta_{\mathrm{decay}}{=}0.20, \theta_{\mathrm{ctrl}}{=}0.12) aggregates rates across left-CVA, right-CVA, TBI, and dementia subjects in those two studies. Numerical values of \tau_{\mathrm{omit}} and the per-category feasibility scores \phi_{c}(s_{k}) are reported in the supplementary material.

Error taxonomy.

The nine error categories are grounded in MLAT/NAT taxonomies summarized by the goal-control model [9]. We organise them into two levels: progression-level errors that disrupt the structure of task execution (omission, sequence, anticipation, perseveration), and current-action errors that primarily affect local action selection, object use, or spatial targeting (object_substitution, tool_substitution, spatial_misreach, action_addition, capture). Detailed definitions and examples for each error category are provided in the supplementary material. This distinction matters for assistance: progression-level errors require reactivation or redirection of the task plan, whereas current-action errors require immediate correction of the ongoing action. Consistent with clinical reports across neurological populations, omissions are treated as a prominent error class, while perseverations are sampled sparsely to reflect their lower base rate. Correct executions vary in subgoal order and intermediate placements, as the generator randomizes subgoal order and plans each subgoal separately. A reordering is labelled sequence only if it violates a required order (stirring before all ingredients are added, or a step-order norm such as buttering bread before toasting); otherwise, it remains correct.

Dual-standard validation.

A key challenge is that errorful trajectories must be semantically wrong yet physically executable: a naive PDDL validator would reject substitution errors, because using a fork where a spoon is required is the violation, while an unconstrained validator admits physically impossible traces. We resolve this by applying strict precondition replay to the seven non-substitution error types, and for substitution errors enforcing only the physical chain—navigation, accessibility, containment, hand occupancy—while relaxing the single semantic precondition that defines the substitution. This ensures that injected errors remain interpretable as human execution errors rather than simulator failures: every errorful trajectory is physically executable, and the person can carry out each of its actions.

Dataset statistics.

The HelpMeOut benchmark spans 50 ADL/IADL task templates and contains 20,000 episodes—12,000 errorful and 8,000 correct, with correct episodes paired with themselves and labeled error-free. Errorful trajectories average 24.7 steps, including the task-end event, with 1.07 injected errors each (6.8% have more than one), yielding 12,746 annotated onsets; omissions are the most frequent category (5,346), followed by sequence errors (1,478) and object substitutions (1,142). We hold out 500 test episodes (312 errorful, 188 correct) and a 2,022-episode validation split; the remaining 17,478 episodes are for training.

3.4 Prefix-Based Evaluation

We evaluate online error monitoring as a streaming task: at step t, the agent observes the task name, \mathcal{S}_{0}, and the action prefix a_{0:t}, and outputs p_{t}, the probability that an error occurs at step t, together with cumulative probabilities for the nine categories; no future action is visible. For a trajectory with N actions, the first \lfloor N/4\rfloor steps are a context window that is not scored, and all onsets fall in the remaining evaluation window, which ends with the task-end event. The four metrics correspond to the benchmark questions of whether something went wrong, where the error becomes observable, when the model raises its first alarm, and what type of error occurred:

M1—Trajectory-level discrimination. The score of a trajectory is s=\max_{t}p_{t} over the evaluation window, and AUROC measures whether errorful trajectories score above correct ones. M1 does not depend on when the alarm is raised; timeliness is measured by M2 and M3. M2—Step-level detection. Each evaluated step is scored against its label y_{t} with AUPRC. Only an onset step is positive; later steps, where the error’s consequences persist, are negative, so a late alarm earns no credit and counts as a false alarm. Because omissions are confirmed at the task-end step, we also report M2-end, the AUPRC at that step, and M2-mid, the AUPRC at the earlier steps where all other errors occur. M3—First-alarm timing. For the first-alarm step \hat{t}=\min\{t:p_{t}\geq\theta\} with \theta=0.5, latency is \Delta=\hat{t}-t_{\mathrm{onset}}. We group alarms as early (\Delta<0), on target (\Delta=0), late (\Delta>0), or missed, and report the false-alarm rate on correct episodes. Omissions, which can only be confirmed at task end, are reported separately. M4—Error-type diagnosis. The cumulative category probabilities are scored against the set of categories that have occurred so far (multi-label): in errorful trajectories from the first onset onward, and in correct trajectories at every evaluated step, where this set is empty and naming any category is a false positive. Each trajectory is weighted equally, and we report macro-F1. A monitor is thus credited for naming an error’s category only where an error has actually occurred.

4 Experimental Results

Evaluation protocol.

We evaluate three families of monitors. Learned: Seq2Seq, which encodes the initial scene and the observed action prefix; Seq2Seq+SG, which additionally encodes the symbolic scene graph replayed from the PDDL state (object identity, location, state, and containment; no geometry); and a scene-graph transformer (SGT) that reads the replayed graph before each action together with the action itself. Non-learned: a CUSUM change-point detector over action-bigram surprise fitted on correct training trajectories; a precondition checker that flags actions whose PDDL preconditions fail; and a re-planner that, given the task’s template-level goal, plans with the generator’s planner, tracks the plan, re-plans when the observed action deviates, and raises an alarm when the re-plan grows longer, when an action cannot execute, or when the goal is unmet at task end. LLMs: frontier models prompted step by step with the same prefix-only input. All learned and non-learned baselines run on the full 500-episode test set; because LLM evaluation is costly, LLMs and the Human reference use a 60-episode subset (30 errorful, 30 correct). For the Human reference, annotators saw the task setting (objects, initial scene, and error categories) and judged each prefix under the same whether, when, and what-kind questions. We report 95% confidence intervals from a trajectory-level bootstrap stratified by errorful and correct episodes, and we do not rank methods whose intervals overlap. Implementation details are in the supplementary material.

Learned and change-point baselines.

Table 3 summarizes the results. Seq2Seq and Seq2Seq+SG detect error steps well above chance on M2 (0.376 and 0.394 vs. 0.036), but most of this comes from the task-end step, where omissions are confirmed (end-step AUPRC 0.66 vs. 0.13–0.15 at the remaining steps), and their trajectory-level discrimination stays near chance (M1 0.496 and 0.554). Seq2Seq+SG improves over Seq2Seq on M1 (+0.058, 95% CI [+0.023,+0.092]) and M4 (+0.047, [+0.019,+0.077]), but not significantly on M2. SGT, which conditions every step on the current action, is the strongest learned monitor (M1 0.778, M2 0.656; 0.553 at non-final steps); its gains concentrate on errors with local action signatures (sequence, spatial misreach, perseveration), while substitutions and action additions remain near chance (Table 3). CUSUM reaches M1 0.645 only by alarming on 81% of correct episodes, with M2 at chance and no type diagnosis, which is why M1 alone cannot measure monitoring. All methods remain far below Human (M1 0.967, M2 0.748).

Symbolic baselines.

Unlike the learned monitors, the symbolic baselines use the PDDL domain model, and the re-planner additionally receives the task’s formal goal and the generator’s planner; they are knowledge-rich references rather than competitors, and they test whether the benchmark’s errors can be caught by construction. Because seven of the nine categories are executable under strict PDDL replay, the precondition checker fires on fewer than 5% of errorful episodes (M1 0.513). The re-planner attains the highest M1 (0.887), but almost entirely through its end-of-task check: once the person declares the task finished, an unmet goal confirms an omission with no false alarms, and this check alone gives M1 0.802. Restricted to steps before task end, its M1 drops to 0.669 [0.622,0.714], overlapping with the other baselines (0.59–0.62); its step-level AUPRC at non-final steps is 0.118 (SGT: 0.553); and 62% of correct episodes receive an alarm, because valid human executions are not shortest plans and deviation from a plan is not an error. A symbolic monitor can thus confirm after the fact that a task was left incomplete, but it does not detect or localize errors while execution unfolds.

Alarm timing and type diagnosis.

At \theta{=}0.5, the Seq2Seq models almost never raise an alarm (miss rate 0.99 and 0.98): their per-step scores rank error steps above other steps but rarely cross the threshold. SGT is the only monitor whose alarms are mostly timely—80% of its first alarms fall at the onset step, at a 12% false-alarm rate on correct episodes—but it misses 52% of errorful episodes, so only 38% receive an on-target alarm; the misses concentrate in the substitution, action-addition, and capture categories (70–89% missed), whereas 83% of spatial misreaches receive an on-target alarm (Table 3). CUSUM and the re-planner rarely miss, but their first alarm comes early in about three quarters of errorful episodes, and they alarm on 81% and 62% of correct episodes. On M4, the learned monitors reach macro-F1 0.50–0.55 (per-class F1 in Table 3 and the supplementary material). Their type heads also name an error category in most correct episodes (all 188 for both Seq2Seq models, 85% for SGT), which M4 penalizes; scored on errorful episodes alone, their macro-F1 would be 0.63–0.68. The Seq2Seq models thus name error categories confidently while almost never raising an alarm, so their type and alarm outputs are not consistent with each other.

LLM baselines do not close the monitoring gap.

Figure 3 compares thirteen LLM configurations on the 60-episode subset. The upper ends of the best LLM confidence intervals stay below Human on both M1 (0.837 vs. 0.967) and M2 (0.482 vs. 0.748), and on M2 no LLM exceeds SGT on the same episodes (0.423). Intervals overlap substantially across models, so we do not rank them. Reasoning matters most for localization and diagnosis: with reasoning enabled, the larger models (Claude Sonnet 4.6, DeepSeek V4 Pro, GPT-5.5, Gemini 3.1 Pro) reach M2 0.12–0.33 and M4 0.43–0.52, whereas with reasoning disabled M2 stays at or below 0.14 and M4 at or below 0.31. More reasoning effort does not help further (GPT-5.5 low vs. medium: M2 0.238 vs. 0.231). With reasoning enabled, these models also give the most on-target first alarms (42–50% of errorful episodes, vs. 23% for SGT on the same subset), but they alarm on 30–77% of correct episodes. Without reasoning, several models output near-zero scores at almost every step. LLMs can thus often recognize that something is off, but they still struggle to identify the step at which an error occurs and to raise a timely alarm, whereas the human reference raises an on-target alarm in 73% of errorful episodes at a 17% false-alarm rate.

5 Limitations and Broader Impact

HelpMeOut is symbolic: episodes are discrete action sequences over a PDDL-style state, which gives large-scale data with deterministic onsets but assumes that actions have already been recognized from video or sensors. It therefore captures neither recognition errors nor the multimodal cues through which errors manifest (hesitation, gaze, fumbling), and its results are an upper-bound proxy for symbolic-level monitoring; grounding the episodes in video or real ADL recordings is left to future work.

Two further limits follow from how episodes are generated. First, every errorful trajectory is physically executable, so failures in which an earlier lapse makes a later action impossible, such as pouring from a kettle that was never filled, are not represented. Second, correct executions are produced by a planner that completes one subgoal before starting the next; they vary in order and intermediate placements but do not interleave subgoals or contain the harmless redundancy of everyday human action, and insertion-type errors are therefore partly identifiable from surface patterns. The errors are generated rather than recorded: a label-blinded review by two clinical experts judged most generated errors plausible (supplementary material), but this is not patient-level validation.

Deployment raises risks the benchmark does not measure. False alarms interrupt the person and erode their autonomy and trust, whereas missed alarms can let an unsafe lapse, such as a stove left on, go unnoticed; thresholds should therefore be set with users and caregivers rather than by a benchmark score. In-home monitoring also requires informed consent, including from people whose capacity to consent may change, data minimization, and the person’s control over what is recorded. HelpMeOut contains only synthetic episodes and no personal data, and results on it are not evidence that a system is safe to deploy.

6 Conclusions

We presented HelpMeOut, a benchmark that reframes household assistance as online error monitoring rather than goal inference. By pairing correct executions with neuropsychologically grounded errorful ones and asking whether, when, and what kind of error occurs, it exposes a gap that goal-inference and task-completion benchmarks miss: learned monitors, symbolic references, and frontier LLMs can often tell that an episode went wrong, especially once it has ended, but rarely detect and localize errors as they happen, and errors that require knowing which object or tool the task calls for remain near chance. We hope HelpMeOut serves as a testbed for assistive agents that catch execution breakdowns before a task fails. Results of further expert validation will be released on the project website.

References

1. [1] L. J. Buxbaum. Ideational apraxia and naturalistic action. Cognitive Neuropsychology, 15(6-8):617–643, 1998.
2. [2] T. Giovannetti, D. J. Libon, L. J. Buxbaum, and M. F. Schwartz. Naturalistic action impairments in dementia. Neuropsychologia, 40(8):1220–1232, 2002.
3. [3] World Health Organization. Dementia. Fact sheet, 2025. URL https://www.who.int/news-room/fact-sheets/detail/dementia. Accessed: 2026-04-29.
4. [4] E. Nichols, J. D. Steinmetz, S. E. Vollset, K. Fukutaki, J. Chalek, F. Abd-Allah, A. Abdoli, A. Abualhasan, E. Abu-Gharbieh, T. T. Akram, et al. Estimation of the global prevalence of dementia in 2019 and forecasted prevalence in 2050: an analysis for the global burden of disease study 2019. The Lancet Public Health, 7(2):e105–e125, 2022.
5. [5] S. A. Olatunji, J. S. Shim, A. Syed, Y.-L. Tsai, A. E. Pereira, H. P. Mahajan, R. A. Mudar, and W. A. Rogers. Robotic support for older adults with cognitive and mobility impairments. Frontiers in Robotics and AI, 12:1545733, 2025.
6. [6] B. Das, D. J. Cook, N. C. Krishnan, and M. Schmitter-Edgecombe. One-class classification-based real-time activity error detection in smart homes. IEEE journal of selected topics in signal processing, 10(5):914–923, 2016.
7. [7] G. Wilson, C. Pereyda, N. Raghunath, G. De La Cruz, S. Goel, S. Nesaei, B. Minor, M. Schmitter-Edgecombe, M. E. Taylor, and D. J. Cook. Robot-enabled support of daily activities in smart home environments. Cognitive systems research, 54:258–272, 2019.
8. [8] C. Pereyda, N. Raghunath, B. Minor, G. Wilson, M. Schmitter-Edgecombe, and D. J. Cook. Cyber-physical support of daily activities: A robot/smart home partnership. ACM Transactions on Cyber-Physical Systems, 4(2):1–24, 2019.
9. [9] T. Giovannetti, R. Mis, K. Hackett, S. M. Simone, and M. B. Ungrady. The goal-control model: An integrated neuropsychological framework to explain impaired performance of everyday activities. Neuropsychology, 35(1):3, 2021.
10. [10] M. F. Schwartz, M. Segal, T. Veramonti, M. Ferraro, and L. J. Buxbaum. The naturalistic action test: A standardised assessment for everyday action impairment. Neuropsychological rehabilitation, 12(4):311–339, 2002.
11. [11] M. F. Schwartz, M. W. Montgomery, L. J. Buxbaum, S. S. Lee, T. G. Carew, H. Coslett, M. Ferraro, E. Fitzpatrick-DeSalme, T. Hart, and N. Mayer. Naturalistic action impairment in closed head injury. Neuropsychology, 12(1):13, 1998.
12. [12] M. B. Holm and J. C. Rogers. The performance assessment of self-care skills. In Assessments in occupational therapy mental health, pages 359–370. Routledge, 2024.
13. [13] D. A. Norman and T. Shallice. Attention to Action: Willed and Automatic Control of Behavior, pages 1–18. Springer US, 1986. ISBN 9781475706291. doi:10.1007/978-1-4757-0629-1_1. URL http://dx.doi.org/10.1007/978-1-4757-0629-1_1.
14. [14] R. Cooper and T. Shallice. Contention scheduling and the control of routine activities. Cognitive neuropsychology, 17(4):297–338, 2000.
15. [15] R. P. Cooper and T. Shallice. Hierarchical schemas and goals in the control of sequential behavior. Psychological Review, 113(4):887–916, 2006. ISSN 0033-295X. doi:10.1037/0033-295x.113.4.887. URL http://dx.doi.org/10.1037/0033-295X.113.4.887.
16. [16] M. F. Schwartz. The cognitive neuropsychology of everyday action and planning. Cognitive neuropsychology, 23(1):202–221, 2006.
17. [17] M. F. Schwartz, E. S. Reed, M. Montgomery, C. Palmer, and N. H. Mayer. The quantitative description of action disorganisation after brain damage: A case study. Cognitive Neuropsychology, 8(5):381–414, 1991.
18. [18] M.-A. Côté, A. Kádár, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, M. Hausknecht, L. El Asri, M. Adada, et al. Textworld: A learning environment for text-based games. In Workshop on Computer Games, pages 41–75. Springer, 2018.
19. [19] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740–10749, 2020.
20. [20] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018.
21. [21] X. Puig, T. Shu, S. Li, Z. Wang, Y.-H. Liao, J. B. Tenenbaum, S. Fidler, and A. Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890, 2020.
22. [22] S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on robot learning, pages 477–490. PMLR, 2022.
23. [23] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80–93. PMLR, 2023.
24. [24] Y. Wan, J. Mao, and J. Tenenbaum. Handmethat: Human-robot communication in physical and social environments. Advances in Neural Information Processing Systems, 35:12014–12026, 2022.
25. [25] D. Ghose, O. Gitelson, M. Vazquez, and B. Scassellati. Open-ended goal inference through actions and language for human-robot collaboration. In Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction, pages 68–77, 2026.
26. [26] M. Botvinick and D. C. Plaut. Doing without schema hierarchies: A recurrent connectionist approach to normal and impaired routine sequential action. Psychological Review, 111(2):395–429, 2004. ISSN 0033-295X. doi:10.1037/0033-295x.111.2.395. URL http://dx.doi.org/10.1037/0033-295X.111.2.395.
27. [27] S. C. Seligman, T. Giovannetti, J. Sestito, and D. J. Libon. A new approach to the characterization of subtle errors in everyday action: Implications for mild cognitive impairment. The Clinical Neuropsychologist, 28(1):97–115, Nov. 2013. ISSN 1744-4144. doi:10.1080/13854046.2013.852624. URL http://dx.doi.org/10.1080/13854046.2013.852624.
28. [28] A. Mihailidis, J. N. Boger, T. Craig, and J. Hoey. The coach prompting system to assist older adults with dementia through handwashing: An efficacy study. BMC Geriatrics, 8(1), Nov. 2008. ISSN 1471-2318. doi:10.1186/1471-2318-8-28. URL http://dx.doi.org/10.1186/1471-2318-8-28.
29. [29] J. Hoey, P. Poupart, A. v. Bertoldi, T. Craig, C. Boutilier, and A. Mihailidis. Automated handwashing assistance for persons with dementia using video and a partially observable markov decision process. Computer Vision and Image Understanding, 114(5):503–519, May 2010. ISSN 1077-3142. doi:10.1016/j.cviu.2009.06.008. URL http://dx.doi.org/10.1016/j.cviu.2009.06.008.
30. [30] K. Hackett, M. McKniff, S. Lehman, E. Pinsky, C. C. Tan, M. Kaplan, G. Vallecorsa, M. D. Serruya, and T. Giovannetti. A counterbalanced crossover pilot study of a personalized smartphone reminder application for older adults with cognitive impairment. Neuropsychological Rehabilitation, 36(1):1–35, Apr. 2025. ISSN 1464-0694. doi:10.1080/09602011.2025.2489125. URL http://dx.doi.org/10.1080/09602011.2025.2489125.
31. [31] R. Peddi, S. Arya, B. Challa, L. Pallapothula, A. Vyas, B. Gouripeddi, Q. Zhang, J. Wang, V. Komaragiri, E. Ragan, et al. Captaincook4d: A dataset for understanding errors in procedural activities. Advances in Neural Information Processing Systems, 37:135626–135679, 2024.
32. [32] K. Yordanova, A. Hein, and T. Kirste. Kitchen task assessment dataset for measuring errors due to cognitive impairments. In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pages 1–6. IEEE, 2020.
33. [33] A. Flaborea, G. Di Mattia, A. Furnari, G. M. Farinella, and F. Galasso. PREGO: Online mistake detection in PRocedural EGocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18483–18492, 2024.
34. [34] L. Seminara, G. M. Farinella, and A. Furnari. Differentiable task graph learning: Procedural activity representation and online mistake detection from egocentric videos. In Advances in Neural Information Processing Systems, volume 37, 2024.
35. [35] S. Majumder, A. Nethi, Z. Al-Halah, and K. Grauman. MistExit: Learning to exit for early mistake detection in procedural videos. arXiv preprint arXiv:2603.14252, 2026.
36. [36] M. Ramírez and H. Geffner. Probabilistic plan recognition using off-the-shelf classical planners. In Proceedings of the AAAI conference on artificial intelligence, volume 24, pages 1121–1126, 2010.
37. [37] C. L. Baker, R. Saxe, and J. B. Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):329–349, Dec. 2009. ISSN 0010-0277. doi:10.1016/j.cognition.2009.07.005. URL http://dx.doi.org/10.1016/j.cognition.2009.07.005.
38. [38] T. Zhi-Xuan, J. L. Mann, T. Silver, J. B. Tenenbaum, and V. K. Mansinghka. Online Bayesian goal inference for boundedly rational planning agents. In Advances in Neural Information Processing Systems, volume 33, pages 19238–19250, 2020.
39. [39] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287–318. PMLR, 2022.
40. [40] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pages 1769–1782. PMLR, 2022.
41. [41] Y. Wan, Y. Wu, Y. Wang, J. Mao, and N. Jaques. Infer human’s intentions before following natural language instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25309–25317, 2025.
42. [42] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251–266, 2021.
43. [43] M. P. Lawton and E. M. Brody. Assessment of older people: self-maintaining and instrumental activities of daily living. The gerontologist, 1969.
44. [44] C. M. Baum, L. T. Connor, T. Morrison, M. Hahn, A. W. Dromerick, and D. F. Edwards. Reliability, validity, and clinical utility of the executive function performance test: A measure of executive function in a sample of people with stroke. The American journal of occupational therapy, 62(4):446–455, 2008.
45. [45] S. Katz, A. B. Ford, R. W. Moskowitz, B. A. Jackson, and M. W. Jaffe. Studies of illness in the aged: the index of adl: a standardized measure of biological and psychosocial function. jama, 185(12):914–919, 1963.

This appendix provides the additional details for the main paper. Section A describes how the benchmark is built, from its neuropsychological grounding to the symbolic domain and the Goal-Control error generator. Section B reports dataset statistics and the data split. Section C covers the baseline and human-study implementations together with the complete LLM results.

Appendix A Benchmark Design

A.1 Neuropsychology of Everyday Action

HelpMeOut builds its error labels and generation model on prior work in clinical neuropsychology on naturalistic action. We draw on three lines of work.

Performance-based instruments.

The Naturalistic Action Test (NAT) and the Multi-Level Action Test (MLAT) ask participants to perform familiar multi-object tasks such as making toast and coffee, wrapping a gift, packing a lunchbox—and score the resulting errors against a standardized taxonomy of omissions, sequence errors, substitutions, additions, and perseverations [10, 11]. These instruments establish that everyday-action failures are structured and recurrent across clinical populations (left-CVA, right-CVA, traumatic brain injury, dementia); this structure makes everyday-action errors suitable for systematic monitoring.

The Supervisory Attentional System (SAS) and contention scheduling.

Contention-scheduling accounts describe routine action as competition among activated schemas, while the Supervisory Attentional System (SAS) provides top-down control when routine schema selection is insufficient [13, 14, 15]. Within this view, weak or decaying task goals can manifest as omissions, whereas poor suppression of competing schemas can manifest as commission errors, especially around subgoal transitions [16, 17].

The Goal-Control account.

Giovannetti et al. [9] synthesize these observations into a Goal-Control model that separates failures of goal activation from failures of executive control. We use this two-factor account in the generator (Section A.6), but the error taxonomy itself is organized by what an error disrupts: the progression of the task or the selection of the current action. Figure 4 summarizes this organizing axis, and the per-category clinical error rates we calibrate against are reported in Table 4.

A.2 Object Hierarchy and Symbolic Domain

HelpMeOut represents each household scene with an object-centric symbolic state. All entities are organized into a five-class hierarchy (Table 5). The location class contains the 31 non-movable entities an agent can move to, grouped by whether objects can be placed ontop of them or inside them; the remaining four classes contain the 156 movable object types referred to in the main text. Including locations, the domain defines 187 object types in total; the 156 “object types” quoted in the main text counts only movable objects (receptacle, food, tool, thing).

Predicates.

The state is described with spatial relations (inside, ontop, human-at, holding, hand-empty), boolean state attributes (open, toggled, cooked, frozen, sliced, soaked, dusty, stained, toasted, buttered, stirred, folded, ironed, hung, worn, wrapped, packed, taken, …), affordance meta-predicates (movable, openable, toggleable, cookable, freezable, sliceable, soakable, foldable, hangable, wearable, toastable, …), and per-type predicates (type-knife, type-spoon, type-iron, …) used to express tool/object identity constraints. Spatial relations are represented directly (only the immediate container/surface), so moving a container automatically carries its contents.

A.3 Action Operators

The domain defines 38 action schemas (Table 6), each specified in PDDL with full preconditions and effects; we give intuitive semantics here and release the complete domain file.

Action constraints.

Several operators carry tool/location constraints enforced through the per-type predicates: slicing requires a type-knife tool, stirring a type-spoon tool, ironing a type-iron tool; heating is only valid at a microwave/oven/stove, cooling only inside a refrigerator, soaking only at a sink, and cleaning only with a suitable cleaning tool. These constraints are exactly the preconditions whose violation defines the tool_substitution error class (Section A.5).

A.4 Task Templates

HelpMeOut spans 50 ADL/IADL task templates organized into eight families (Table 7). Each template is anchored in a clinical assessment, functional scale, or occupational-therapy framework for everyday action—NAT, PASS, the Lawton IADL scale, the Katz ADL index, the Executive Function Performance Test (EFPT), or the OTPF-4 occupational-therapy framework—and is annotated with a complexity tier (simple / medium / complex) reflecting the number of accomplishment steps. Each template is expressed as a set of first-order subgoals over the predicates of Section A.2; Figure 5 shows a worked example.

A.5 Error Taxonomy

The nine error categories are organized into two levels. Progression-level errors disrupt the structure of task execution; current-action errors affect local action selection, object use, or spatial targeting. For each category we give its definition, trigger condition, and where the onset is labelled; Figure 6 pairs a correct and an errorful micro-example per category.

Each definition below names the step at which the onset is labelled; onsets are recognizable from the observed prefix and the task, without the reference execution. An immediately repeated action without additional effect, or walking back and forth between places the task uses, is harmless redundancy and not an error.

Progression-level errors.

• Omission. An entire subgoal block is deleted. Because correct executions may perform subgoals in different orders, the missing subgoal could still be performed later; the omission is therefore confirmed at the task-end event. Onset: the task-end step. When the missing step has an earlier observable consequence, such as storing an item before it is processed, the error is labelled anticipation and the deleted step is not counted as an omission, following MLAT coding [11].
• Sequence. Two steps that must occur in a specific order are performed in the wrong order. Two constructions are used: stirring before the last ingredient is added (the MLAT example “stir mug of water, then add grinds”), and violating a step-order norm (buttering bread before toasting it, folding before ironing, taking a pill before pouring water, putting on a jacket before the required shirt). Other orders that reach the goal are correct. Onset: the step performed out of order.
• Anticipation. An item is stored, served, or packed before it receives the processing the task requires, or a gift box’s lid is closed without first closing its flaps, which are never closed afterwards. Onset: the step that commits the unprocessed item: putting it into its final place, closing the container that holds it, or delivering that container to its final place.
• Perseveration. After doing something else in between, the person repeats an action that should be done only once, such as adding sugar to the coffee again after buttering the toast, or taking an extra dose of a pill the task does not require. Onset: the repeated action.

Current-action errors.

• Object_substitution. A confusable object of the same subclass is used in place of the object the task needs (e.g. jam for butter). Onset: the first step that uses the substitute.
• Tool_substitution. A confusable tool is used in place of the appropriate one (e.g. a fork where a spoon is required). Onset: the first step that uses the substitute tool.
• Action_addition. An item unrelated to the task (not named in the goal and of a different subclass from every task object) is put into or onto the task’s target container, plate, or tray. Removing it later does not undo the error. Onset: the step that puts it there.
• Capture. An action the task needs is applied to an unrelated object of a different subclass instead of the task’s object, which is then left unprocessed. Onset: the action applied to the unrelated object (fetching it or walking to it does not count).
• Spatial_misreach. The person goes to a storage location that the task never uses, opens it, and leaves without taking anything. Onset: the step of leaving that location empty-handed, the first step at which the visit is recognizably unrelated to the task.

A.6 Error-Generation Model

Goal-Control process (supplementary parameter values).

The main text defines the Goal-Control process (Eq. 2): for each subgoal block b_{k} an activation u_{k}=\theta_{\mathrm{goal}}(1-\theta_{\mathrm{decay}})^{k}+\epsilon_{k} gates omission when u_{k}<\tau_{\mathrm{omit}}, and surviving blocks receive a commission with probability \theta_{\mathrm{ctrl}}. Here we report only the values deferred to this supplement: the omission threshold \tau_{\mathrm{omit}}{=}0.3 and the activation noise \sigma{=}0.1 (with \epsilon_{k}\sim\mathcal{N}(0,\sigma^{2})). The impaired profile (\theta_{\mathrm{goal}},\theta_{\mathrm{decay}},\theta_{\mathrm{ctrl}}){=}(0.48,0.20,0.12) and the omission-vs-commission split it targets are reported in Table 4; all generator hyperparameters are collected in Table 8.

Two-axis per-subgoal decision.

The generator walks the subgoal blocks of the reference trajectory in order and applies two decisions at each block. (i) Goal axis: it first tests the activation u_{k}; if u_{k}<\tau_{\mathrm{omit}} the block is dropped as an omission. (ii) Control axis: otherwise, with probability \theta_{\mathrm{ctrl}} a commission is injected, and its category is drawn from the weights w_{c} (Table 4(b)) restricted to the categories that are feasible at the current step. These two axes correspond to the two panels of Table 4: the goal-vs-control decision gives the omission-vs-commission split (a), and the weighted draw on the control axis gives the commission-category weights (b). Because a commission category is realized only when feasible in the current state, the dataset frequencies can differ from w_{c} (see Per-category feasibility below).

Per-category feasibility \phi_{c}(s_{k}).

Not every commission category can be injected at every step. Each injector requires a particular trajectory or scene configuration: for example, object_substitution requires a confusable alternative object in the scene, while sequence requires reorderable subgoal blocks. We represent this with a binary feasibility gate \phi_{c}(s_{k}), which is 1 only when category c can be realized at step k. As a result, the observed category frequencies are shaped by both the sampling weights w_{c} and the states in which each error is feasible. Table 9 summarizes these gates. Every injected episode must also pass the validation of Section A.7, match its category rule exactly once at the injected step, and the reference execution must not match the rule.

Calibration.

We calibrate \theta_{\mathrm{goal}},\theta_{\mathrm{decay}},\theta_{\mathrm{ctrl}} and w_{c} so that the marginal error distribution produced by the generator matches the per-category rates of Table 4. Figure 7 compares the generated distribution against the clinical targets. Because the feasibility gates bind unevenly, the raw generator output under-represents low-feasibility categories; the released episodes were therefore selected per category from a larger generated pool, so that the released distribution approximates the targets.

A.7 Dual-Standard Validation

A core requirement is that errorful trajectories be semantically wrong yet physically executable. We therefore validate with two standards.

Strict standard (7 categories).

omission, sequence, anticipation, perseveration, action_addition, capture_error, and spatial_misreach must pass a complete PDDL replay: every action’s preconditions must hold in the simulated state. These errors are “legal-but-wrong” executions and remain fully physically consistent.

Relaxed standard (2 categories).

tool_substitution and object_substitution would be rejected by a naive validator—using the wrong tool/object is the error. For these we enforce only the physical chain—navigation, accessibility, containment, and hand occupancy—while relaxing the single semantic precondition that defines the substitution (e.g. the type-spoon requirement of human-stir). Within the error block, only the semantic type constraint is relaxed; all physical preconditions still hold.

Confusable-alternate selection.

Substituted tools/objects are drawn from a hierarchy-constrained confusable set: tool/food/thing substitutions stay within the same subclass, receptacle substitutions may cross subclass, and no substitution ever crosses a parent class. This keeps each substitution interpretable as a human confusion rather than an arbitrary swap.

Appendix B Dataset Statistics

B.1 Corpus Overview

Figure 8 summarizes the released HelpMeOut corpus—its episode composition, task coverage, trajectory length, injected-error rate, annotated onsets, and dominant error categories.

B.2 Data Split

The released corpus contains 20,000 episodes over 50 templates: 17,478 for training, 2,022 for validation, and 500 for testing. The errorful test episodes are stratified by category in proportion to the full corpus and the test set covers all 50 templates (312 errorful, 188 correct; 121 simple, 291 medium, and 88 complex episodes by template tier). All learned and non-learned baselines are evaluated on the full test set. Because frontier-LLM evaluation is substantially more expensive, LLMs and the human reference are evaluated on a fixed 60-episode subset of the test set with 30 errorful episodes, covering every category (three to four per category), and 30 correct episodes from 30 different templates; the learned and non-learned baselines are re-scored on the same subset for comparison.

Appendix C Experimental Details

C.1 Learned Baselines

The learned baselines are trained as discriminative error monitors on the benchmark’s training corpus (disjoint from the held-out test set) and run as online monitors on the 500-episode test set. Every learned baseline receives the same input as the LLMs: the task name, the initial scene, and the action prefix; none receives the formal goal, the reference execution, or any label.

Seq2Seq.

At each step, the text-only baseline encodes the task description, the initial scene description, and the observed action prefix as token sequences. The shared sequence representation feeds three supervised prediction heads: a step-level error head producing p_{t} (the reported p_instant score), a 9-class multi-label error-type head producing p_type, and an auxiliary prefix-error head used only as training supervision. The auxiliary prefix head is not used by the reported metrics.

Training minimizes the sum of binary cross-entropy for the step-level error head, multi-label binary cross-entropy for the type head, and auxiliary binary cross-entropy for the prefix head. The step-level and prefix heads are trained without positive-class weights; the type head uses per-class positive weights equal to the negative-to-positive ratio. We train with batch size 32 (two accumulated batches of 16), learning rate 10^{-4}, and a single seed (42), evaluate on the validation split every 2,000 steps, and keep the checkpoint with the highest validation M2 AUPRC; the test set is scored once.

The reported metrics are derived directly from these outputs: M1 uses \max_{t}p_{t}, M2 is the AUPRC of p_{t} against the step label y_{t}, M3 is the first time step with p_{t}\geq\theta, and M4 uses the 9-class p_type predictions.

Seq2Seq+SG.

The +SG variant augments the same Seq2Seq backbone with an explicit scene-graph encoder. A relational graph is built from the symbolic scene state after replaying the observed prefix in the PDDL simulator (object identity, location, state, and containment; there is no geometry, hence the renaming from “+3D”) (build_local_graph); each node carries a 134-dimensional feature (a 32-d learned type embedding, plus class, subclass, affordance, and state-boolean one-hot blocks), and a two-layer R-GCN (RGCNConv(134,64) \to ReLU \to Dropout(0.2) \to RGCNConv(64,64) \to ReLU) with four relation types and mean-pool readout produces a graph-level embedding that is fused before the prediction heads. Training otherwise matches the Seq2Seq baseline (same discriminative multi-head loss). This isolates the contribution of explicit object/spatial state to error monitoring. Both supervised baselines serve as reference implementations rather than new model contributions. Seq2Seq+SG performs slightly better than Seq2Seq (main paper, Table 2).

Per-class error-type diagnosis (M4).

Table 11 reports per-class M4 F1 of the learned monitors on the 500-episode test set. M4 scores the cumulative category predictions in errorful episodes from the first onset onward and in correct episodes at every evaluated step, where naming any category is a false positive; each episode is weighted equally. The type heads of the learned monitors name an error category in most correct episodes (at the last step: all 188 for Seq2Seq and Seq2Seq+SG, 160 for SGT), mostly omission and object substitution, which lowers their macro-F1 from 0.63–0.68 when scored on errorful episodes alone to 0.50–0.55. Omission and sequence remain the best-diagnosed categories (F1 \geq 0.74). SGT is best on anticipation, perseveration, and spatial misreach, the Seq2Seq models on capture and action addition; object and tool substitution stay at or below 0.20 for all three models. Predicting categories from the task name alone, using their training frequencies per template, reaches at most 0.20 in the errorful-only scoring, so the type heads use the observed actions rather than a task prior.

Per-class step-level detection (M2).

Table 12 breaks the step-level AUPRC of the learned monitors down by error category. For each category we pool the evaluation-window steps of the test episodes of that category with those of all 188 correct episodes; for omissions all evaluated steps are scored, with the task-end step as the positive, and for all other categories the steps before task end. SGT’s advantage concentrates on categories with a local action signature—sequence (0.814), spatial_misreach (0.771), and perseveration (0.502)—which it can recognize from the current action and the scene graph that precedes it. Categories whose detection requires knowing which object, tool, or item the task calls for remain near chance for all three models: object_substitution (0.012–0.038), tool_substitution (0.003–0.020), and action_addition (0.005–0.029). The Seq2Seq models are strong only on omissions at the task-end step.

C.2 Scene-Graph Transformer and CUSUM

Scene-graph transformer (SGT).

At each step t, SGT encodes the task name, the symbolic scene graph obtained by replaying actions a_{0:t-1} in the PDDL simulator, and the current action a_{t}; a causally masked transformer encoder (model width 128, four heads) runs over the steps, and three heads predict the step-level error probability p_{t}, an auxiliary prefix probability, and the nine type probabilities. Because the graph precedes the action and the action is a separate input, each step can be scored by whether the current action fits the current state. Training uses losses analogous to Seq2Seq (the type loss is applied at error steps), without positive-class weights on the step heads and without truncating trajectories, for three epochs (6,552 steps of eight episodes) with AdamW at learning rate 3\times 10^{-4}; the checkpoint with the highest validation M2 AUPRC is kept. We verified causality on the trained model: changing any action after step t changes the outputs at steps \leq t by at most 2.9\times 10^{-6} (floating-point noise).

CUSUM.

A dedicated change-point detector without learning: a task-conditioned action-bigram model P(a_{t}\mid a_{t-1}) with add-k smoothing and unigram back-off is fitted on the correct training episodes; the per-step surprise s_{t}=-\log P(a_{t}\mid a_{t-1}) drives the CUSUM statistic S_{t}=\max(0,S_{t-1}+s_{t}-k), where k is the mean surprise on the training data, and p_{t}=S_{t}/(S_{t}+k). CUSUM makes no type prediction, so its M4 is 0.

C.3 Symbolic Baselines

Both symbolic baselines replay the observed actions in the PDDL simulator and therefore use the domain model, which the learned monitors do not receive.

Precondition checker.

An alarm (p_{t}=1) is raised at any step whose action violates its PDDL preconditions in the replayed state; all other steps receive p_{t}=0. Because seven of the nine categories pass strict replay by construction (Section A.7), it can fire only on the semantic precondition relaxed for substitutions, and it raises an alarm on fewer than 5% of errorful test episodes.

Re-planner.

The re-planner additionally receives the task’s template-level formal goal—the subgoals of the task template with object placeholders, e.g. “some bread is toasted and buttered and lies on some plate”—but not the episode’s grounded goal, so it does not know which objects the person will use or the correct final state. It uses the generator’s per-subgoal STRIPS planner. At every (re-)planning point it scores the admissible bindings of the template placeholders by the number of subgoals already satisfied and by the objects the person has touched, plans for the three best bindings, and keeps the shortest plan. It then tracks the plan: an action that matches the next planned action advances it (p_{t}=0); an action that violates its preconditions raises p_{t}=1; any other action triggers re-planning from the new state, and if the new plan is \Delta>0 steps longer than the remaining old plan minus one, p_{t}=\Delta/(\Delta+1), otherwise p_{t}=0. At the task-end event, p_{t}=1 if no admissible binding satisfies all subgoals. When the person walks to a location outside the planner’s task-relevant set, that location is added to the planner’s action space before re-planning. Each episode has a 30-minute budget; 60 test episodes (42 errorful, 18 correct) did not finish and are excluded from M1 and M3, as for any method with missing predictions.

Analysis.

The re-planner’s M1 (0.887) comes almost entirely from the end-of-task check: this check alone gives M1 0.802, and with the task-end step removed M1 drops to 0.669 [0.622,0.714]. On the steps before task end its AUPRC is 0.118, it raises the first alarm early in 84% of detected errorful episodes, and it alarms on 62% of correct episodes: correct executions are not shortest plans, so deviation from a plan is not an error. Its weakest categories are those that leave the goal satisfied, such as spatial misreach and action addition. For reference, the row marked ∗ in Tables 13–15 gives a privileged variant that receives the episode’s grounded goal, i.e. the correct final state (33 test episodes did not finish); it is an upper reference for the re-planner and not a monitor that could be deployed.

C.4 Additional Results

Table 13 gives 95% confidence intervals for the main results (trajectory-level bootstrap stratified by errorful and correct episodes, 1,000 resamples), Table 14 breaks M1 and M2-mid down by template complexity, and Table 15 gives the full first-alarm distribution. The learned monitors degrade with task complexity (SGT: M1 0.844 on simple, 0.573 on complex templates), whereas the re-planner does not, because its end-of-task goal check does not depend on task length.

C.5 LLM Baselines

Protocol.

LLMs are evaluated prefix by prefix with the same input as every other monitor. For each evaluated step t of an episode, one request contains the system prompt below and a user message with the task name, the initial scene (every object with its class, location, and initial states), and the actions a_{0:t}; the model returns a JSON object for step t only, with p_instant (p_{t}) and nine cumulative category probabilities (p_types). No future action, trajectory length, goal condition, reference execution, or label is included; the task-end event appears only at the final step. On the 60-episode subset this gives 1,201 requests per configuration. Before running, we checked all 1,201 user prompts automatically for these properties.

API settings.

All calls use OpenRouter with temperature 0. Because the default reasoning behaviour of the same model differs across providers (e.g. DeepSeek V4 Flash reasons by default on one provider and not on another), each configuration is pinned to a single provider with fallback disabled, and reasoning is set explicitly: disabled, an effort level, or a token budget (Table 16). Requests time out after 120–600 s and are retried twice; steps whose response cannot be parsed are re-requested once with a doubled output limit. All thirteen configurations completed every step. Every response is logged verbatim, together with the reasoning text when the provider returns it, token counts, cost, and the serving provider; the logs are released with the benchmark. The total cost of the thirteen runs was US$107.

System prompt.

Reproduced verbatim (line breaks added for display; an em dash is typeset as “–”).

You are monitoring a human performing a household task in real time, ONE STEP AT A TIME. You see the task name, the initial scene, and the actions from step 0 up to and including the current step. You CANNOT see any future action, and you are NOT told how long the task will last. The last step of every episode is the event ‘human-task-end h‘: the person declares the WHOLE task finished. It appears only when it happens. At that step you can judge whether some step the task needs was never done. For the CURRENT step, output two judgments: - p_instant (0.0-1.0): probability that an error occurs AT the current step (the current action is where an error happens). - p_types: for each of the 9 error categories, the probability that an error of that category has occurred at or before the current step (cumulative over the episode so far). IMPORTANT GUIDANCE: - You are NOT given the formal goal conditions, only the task name. Infer what the task requires from the task name and the scene, as a person watching would. - By default assume the current step is correct (p_instant near 0). Give p_instant > 0.5 only with clear evidence that an error happens AT this step. - A step is correct if it is a reasonable action toward the task, even if it is roundabout or inefficient. Harmless redundancy is NOT an error: walking back and forth between places the task uses, picking something up and putting it back, or repeating an action immediately when the repetition has no additional effect. - p_instant marks only the step where an error happens, not later steps that are merely its consequences. - p_types is cumulative: once an error of some category has happened, keep that category high at all later steps, including the final task-end step. Error categories (the step in brackets is where p_instant should be high): - OMISSION: a step the task requires was never performed. This can only be confirmed when the person declares the task finished. [the ‘human-task-end‘ step] - SEQUENCE: two steps that must happen in a specific order were done in the wrong order, e.g. applying butter to bread before toasting it. [the step done out of order] - ANTICIPATION: an item is stored, served or packed before it receives the processing the task requires. [the step that commits the unprocessed item: putting it into its final place, closing the container that holds it, or delivering that container to its final place] - PERSEVERATION: after doing something else in between, the person repeats an action that should be done only once, e.g. adds sugar to the coffee, butters the toast, then adds sugar to the coffee again. [the repeated action] - OBJECT_SUBSTITUTION: a similar object of the same kind is used in place of the object the task needs. [the first step that uses the substitute] - TOOL_SUBSTITUTION: a similar tool of the same kind is used in place of the appropriate tool. [the first step that uses the substitute tool] - ACTION_ADDITION: an item unrelated to the task is put into or onto the task’s target container, plate or tray. Removing it later does not undo the error. [the step that puts it there] - CAPTURE_ERROR: an action the task needs is applied to an unrelated object of a different kind instead of the task’s object. [the step that acts on the unrelated object] - SPATIAL_MISREACH: the person goes to a storage location that has nothing to do with the task, opens it, and leaves without taking anything. [the step of leaving] OUTPUT FORMAT -- respond with ONLY valid JSON (no markdown, no commentary): {"step": <int>, "p_instant": <float>, "p_types": {"OMISSION": <float>, "SEQUENCE": <float>, "ANTICIPATION": <float>, "PERSEVERATION": <float>, "OBJECT_SUBSTITUTION": <float>, "TOOL_SUBSTITUTION": <float>, "ACTION_ADDITION": <float>, "CAPTURE_ERROR": <float>, "SPATIAL_MISREACH": <float>}} Output exactly ONE such object, for the current step only.

User prompt template.

Task: <task name, underscores replaced by spaces> Initial scene: <object>:<CLASS>@<location>[<states>] ... Actions observed so far: Step 0: <action> h <arg1> <arg2> ... ... Step t: <action> h <arg1> ... Judge the current step (Step t). Output the JSON object for step t only.

C.6 Full LLM Results

Table 17 reports all thirteen configurations on the 60-episode subset (30 errorful, 30 correct; 1,201 evaluated steps, 33 of them error steps; M2 random 0.027, M2-end 0.067, M2-mid 0.025), with 95% bootstrap intervals, together with the other baselines re-scored on the same episodes; Table 18 gives the first-alarm distribution.

Key observations.

1. Models cannot be ranked on M1. With 60 episodes, the M1 intervals of all configurations overlap. The best upper bounds stay below the human reference on both M1 (0.837 vs. 0.967) and M2 (0.482 vs. 0.748).
2. Reasoning matters for localization and diagnosis. With reasoning enabled, the four larger models reach M2-mid 0.18–0.48 and M4 0.43–0.52; with reasoning disabled, M2-mid is at most 0.15 and M4 at most 0.31, and Qwen3.6 and DeepSeek V4 Flash output near-zero scores at almost every step. At the steps before task end, the best reasoning configurations (DeepSeek V4 Pro 0.478, Claude Sonnet 4.6 0.451) are on par with SGT on the same episodes (0.455); none of them confirms omissions at task end well (M2-end at most 0.22).
3. More effort does not help further. GPT-5.5 low and medium are indistinguishable on every metric (M2 0.238 vs. 0.231), at 65% higher cost for medium.
4. Reasoning trades misses for false alarms. The reasoning configurations give the most on-target first alarms (42–50% of errorful episodes) but alarm on 30–77% of correct episodes; without reasoning, most models rarely alarm at all.

C.7 Expert Review

To obtain an initial check of generator fidelity, two domain experts (neuropsychology; nursing care) jointly reviewed 24 randomly selected NAT-aligned episodes (12 errorful, 12 correct) without access to the labels. They judged 10 of the 12 injected errors plausible, identified 11 of the 12 correct episodes as correct, and matched the generator’s error category on 8 of the 12 errorful episodes. The experts noted that the NAT alignment yields realistic tasks, whereas the plausibility of an error depends on the scene, which is consistent with the scene-specific feasibility rules of Section A.6. Mapping the generated errors to NAT error instances gives three direct and four structural matches. These results provide initial expert support for the generator, but they are not a patient-level validation; results of further expert validation will be released on the project website.