Beyond Leaderboard Scores: A Deployment-Focused Protocol for Interpretable Tracking Evaluation in Pedestrian-Centric Environments
This preprint presents a deployment-focused evaluation protocol for 3D pedestrian multi-object tracking that holds detections fixed and reports trajectory-level success rates for initialization, detector-gap continuation, identity recovery, close-neighbor association, and load-dependent runtime on a Jetson Orin. On JRDB, non-GT-assisted trackers score only 24.26–29.67% HOTA; after a 1.0 s detection gap, no non-GT tracker maintains correct same-ID spatial output in more than 45.4% of cases, and most fall below 1%. The protocol shows aggregate leaderboard scores hide large behavioral differences that matter for real-time robots among pedestrians.
Can a shared-detection, trajectory-level evaluation protocol reveal deployment-relevant tracker behavior—initialization latency, gap continuation, identity recovery, near-neighbor association, and embedded runtime—beyond aggregate HOTA scores?
Mobile robots in pedestrian environments need trajectories that appear quickly, preserve identity through missed detections, stay spatially credible, and run within embedded compute. Existing MOT benchmarks are vehicle-centric, and aggregate scores such as HOTA/MOTA/IDF1 do not show when or how trajectories fail, while differing detector inputs can confound tracker quality with detector quality.
Prior work evaluates 3D MOT on KITTI, nuScenes, and Waymo with scalar metrics; open-source trackers include motion-model-plus-assignment systems (AB3DMOT, CenterPoint tracking), confidence-aware lifecycles (SimpleTrack, CBMOT), geometry-aware association (Poly-MOT, Fast-Poly), probabilistic methods (GNN-PMB, FGO-PMB), and learned association (3DMOTFormer, PolarMOT). Their weakness is that aggregate scores summarize whole-test-set performance and do not expose failure conditions, and evaluations often use different detector inputs.
The protocol fixes shared Person-MinkUNet detections (score ≥ 0.5) for all trackers, corrects the JRDB 3D-IoU evaluator, and defines tracker-only events from ground truth: 0.5 s stable initialization, 1.0 s detector-gap continuation, post-gap identity recovery, and 1.0 s close-neighbor windows stratified by nearest-neighbor distance. It also measures tracker-step FPS versus detection count on a Jetson Orin without GPU and adds GT-derived dropout and box-instability perturbations. The authors implement PedRefTrack, a lightweight constant-velocity Kalman tracker with score-aware confirmation, adaptive coasting based on recent prediction error and predicted close approach, and hidden identity retention; a GT-assisted variant idealizes association and motion to estimate remaining tracker-side headroom.
On the 27-sequence JRDB test split with fixed detections, HOTA for non-GT trackers is 24.26–29.67%; GT-assisted PedRefTrack achieves 33.33% vs 29.67% for PedRefTrack and 27.48% for Fast-Poly. LocA is narrow (65.49–66.15%), so differences come from association/output behavior. In 1.0 s detector-gap continuation, GT-assisted retains same identity spatially in 86.8% of gaps; SimpleTrack 45.4%, PedRefTrack 37.0%, all others below 1%. Identity recovery after a gap is stronger: at 0.5 s Fast-Poly 78.9%, CBMOT 75.1%, GNN-PMB 59.8%; at 1.5 s PedRefTrack 69.8%, Fast-Poly 55.8%, SimpleTrack 50.0%, CBMOT 41.9%, while AB3DMOT, GNN-PMB, and ELPTNet stay below 5%. Close-neighbor failure rises as distance falls: from 0.7–0.8 m to 0.3–0.4 m, failure increases 2.3 points for PedRefTrack and up to 13.5 points for GNN-PMB; the deficit to GT-assisted widens to 4.1–25.2 points in the closest bin. Initialization by 0.5 s reaches only 39.1–56.0% of starts; at 0.2 s Fast-Poly has 45.0% vs PedRefTrack 29.4%. On Jetson Orin, median tracker-step FPS ranges from 2.23 (SimpleTrack) to 315.25 (CBMOT); PedRefTrack runs 74.96 FPS on 1 CPU core. Fast-Poly falls below 10 Hz at roughly 45 detections, while AB3DMOT, GNN-PMB, and SimpleTrack do so at lower loads; PedRefTrack holds about 15.2 FPS at 75–80 detections. Under GT-derived perturbations, PedRefTrack and Fast-Poly show the smallest HOTA deficit growth as severity increases.
Authors state: one dataset (JRDB), one trained detector at score ≥ 0.5, HOTA-tuned configurations at selected matching thresholds and temporal horizons, GT-derived perturbations do not reproduce full learned detector error, runtime covers only tracker-step execution on one embedded platform and excludes the rest of the perception/navigation stack, and tests are deployment-oriented proxies rather than direct navigation measures. Also evident: the GT-assisted reference is a diagnostic rather than a deployable tracker; the non-GT comparison includes only six open-source systems; corrected evaluator makes results not directly comparable to the official JRDB leaderboard.
Mobile-robot and autonomous mobile robot developers targeting hospitals, airports, retail, campuses, or last-mile delivery can use this protocol and released code now to benchmark and select 3D pedestrian trackers under fixed detections and embedded compute, and to tune coasting/confirmation policies. For product teams, integration into routine regression testing could happen in 1–3 years once the protocol is validated across additional detectors, datasets, hardware, and downstream navigation metrics. More confident deployment in crowded human spaces would require replicating the gap-continuation and identity-recovery results across target environments and proving that improved tracker profiles improve planner or safety outcomes.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.