MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models
Robot arms that insert, wipe or press usually notice a jam only after it happens. This Shanghai Jiao Tong University preprint predicts the force a planned motion will produce and warns about 25 to 28 control steps early; used to pick between candidate motions, it lifted a frozen pi0.5 policy from 52.6% to 61.5% task success on a real Franka arm.
Can a robot judge, before executing a planned action chunk, whether that motion will push a contact-rich task into an abnormal interaction such as jamming, slipping or losing contact?
Modern manipulation policies plan dozens of steps ahead, but monitoring is still reactive: a failure detector fires once a deviation is already visible, leaving little time to intervene. Cameras show geometry but only hint at contact force, friction and resistance, and a planned action describes intended motion, not how the object will push back. Existing failure datasets mostly label whole episodes as success or failure and rarely record force, so there was no way to train or score a model on when an anomaly starts.
Out-of-distribution detectors (PCA-KMeans, LogpZO) flag anything that looks unlike nominal runs, which also catches harmless variation in speed or pose. SAFE-LSTM reads a VLA's internal features and learns failure labels, but its balanced accuracy fell from 0.724 on seen tasks to 0.495 on new ones in this paper's tests. Progress reward models (Robo-Dopamine, Robometer) give indirect warnings with many false positives, and WCM, a world-model monitor, predicts only one step of dynamics ahead.
MAIA starts from the GE-Base video world model and adds an action and force pathway: future actions are encoded as tokens that attend to recent wrist force readings, and the result is injected into the video transformer through gated cross-attention adapters. The model is trained only on normal executions to predict video and force over the next 50 control steps (about 1.7 s at 30 Hz). A small anomaly head is then trained on frozen rollouts to output the probability that any anomaly occurs in the window and what share of the window it will cover. The team also built CRIA, a real-world dataset of 2,852 trajectories over 18 tasks in five categories (insertion, pressing, wiping, path-constrained, articulated objects), recorded on a Franka Research 3 with an ATI 6-axis force sensor and three RealSense cameras, with frame-level anomaly labels; 1,817 trajectories contain at least one anomaly.
On 13 seen tasks MAIA reached AUPRC 0.738, balanced accuracy 0.765 and a 0.464 anticipation rate with a median lead of 28.0 steps, against 0.669, 0.702, 0.242 and 17.7 steps for the best baseline, WCM. On five held-out tasks it kept 0.728 AUPRC and 0.710 balanced accuracy with a 25.1-step lead, while SAFE-LSTM fell to 0.495. Removing force input cut average balanced accuracy from 0.765 to 0.689 (vision only). With 25 trajectories from a new task, joint adaptation reached 0.762, up from 0.615 with no task data. On the real robot, choosing among eight pi0.5 candidate chunks by lowest predicted anomaly probability raised success over 18 tasks (30 trials each per method) from 52.6% to 61.5%; wiping rose from 46.7% to 66.7%, path-constrained tasks from 48.9% to 62.2%, while insertion stayed at 63.3%.
This is a preprint from one lab on one robot with one force sensor, and the authors say cross-embodiment and cross-sensor transfer are untested. The horizon is fixed at 50 steps with fixed history, so slow-building anomalies may be missed, and the method warns but does not diagnose causes or recover. We also note that the paper does not report the compute or latency of rolling a video world model for eight candidates at every planning step, which decides whether this runs on an industrial controller; and that anomaly labels are human-annotated, so the anticipation scores depend on annotator judgement of when an anomaly begins.
Force-aware failure prediction matters most for assembly, polishing, cable routing and kitting cells, where a jam costs a line stop. Policy vendors such as Physical Intelligence ship models like pi0.5 that integrators cannot retrain; a wrapper that only reranks candidate actions fits that model, and it gives sensor makers (wrist force-torque units from ATI and peers) a new reason to be on every arm. On a one to three year view this is a pattern for safety and quality layers above foundation-model policies rather than a product ready today.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.