STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction
STARS is a preprint method that learns compact, reusable social representations from short multi-agent navigation windows by treating agents as nodes in a spatiotemporal graph and training a self-supervised graph variational autoencoder. In downstream tests, its heterogeneous variant reaches 0.503 SNS macro F1 for pedestrian action classification versus about 0.30–0.33 for unstructured baselines, while remaining competitive on SEAN-T human perception tasks. This matters because it shows structured graph representations can offset scarce human-robot-interaction data, enabling data-efficient social navigation.
Can self-supervised, graph-structured representations of spatiotemporal multi-agent dynamics capture reusable social semantics and improve data-efficient downstream social-navigation tasks?
Robots must reason jointly about physical motion and social context, but existing spatial models lack social understanding, text and vision models struggle with continuous dynamics, and grounded HRI datasets are small, fragmented, and prone to overfitting when used with large unstructured models.
Prior work either used trajectory forecasting GNNs that optimize spatial displacement error rather than social semantics, or relied on large foundation and vision-language-action models trained on massive datasets; these approaches fail to generalize in low-data HRI settings and do not natively unify spatial and social reasoning.
STARS converts fixed-duration multi-agent scenarios into complete spatiotemporal graphs with agent node attributes and directed temporal edge features encoding relative SE(2) poses and deltas. A message-passing graph autoencoder, with homogeneous and heterogeneous variants, reconstructs temporal edges and available node features under a VAE objective, producing compact per-node latents. Pretraining is self-supervised on multiple datasets; frozen latents are then fitted with lightweight linear probes for downstream tasks.
On SEAN-T perception tasks, STARS-Ho reaches competence, surprise, and intention F1 scores of 0.799, 0.741, and 0.734 at 100% labels, versus MLP 0.797, 0.720, 0.703; temporal autoencoder 0.742, 0.739, 0.664; and random forest 0.718, 0.496, 0.698. The autoencoder remains competitive on some SEAN-T fractions. On SNS action classification, STARS-He reaches macro F1 of 0.362 at 1% labels, 0.447 at 5%, 0.470 at 10%, 0.493 at 25%, 0.501 at 50%, and 0.503 at 100%, versus MLP 0.294, 0.297, 0.300, 0.306, 0.295, 0.293 and autoencoder 0.308, 0.309, 0.307, 0.310, 0.309, 0.332; STARS-Ho reaches 0.344 at 1% and 0.475 at 100%. STARS-He at 100% beats the same heterogeneous graph architecture trained end-to-end on SNS labels, 0.503 vs 0.445. Cross-dataset transfer: SEAN-T-only pretraining yields 0.497 SNS macro F1 vs 0.503 joint; SNS-only pretraining yields 0.801, 0.710, 0.729 on SEAN-T competence, surprise, intention vs 0.794, 0.709, 0.721 joint. All results use 10 random seeds, frozen encoders, and linear probes; evaluations are on VR/navigation datasets, not physical robots.
The authors state that the autoencoder and message passing smooth high-frequency pairwise geometry, making the latents less suited to edge-centric tasks such as group detection that require exact relational geometry; unified graph construction can discard dataset-specific details; and the evaluation assumes fixed-duration windows and processed agent tracks on curated datasets. Evident limitations include no physical-robot validation, only two datasets, and linear-probe evaluation rather than end-to-end deployment.
Developers of social-navigation stacks for delivery robots, service robots, warehouse AMRs operating near people, robotaxis, and humanoids could use these representations to improve crowd interaction and intent prediction. Adoption is likely 3+ years out because the method must first be validated on real robot platforms, with noisy onboard perception and online segmentation, and scaled to more diverse HRI datasets.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.