Real-time Event-camera Stereo Visual Odometry via Keytime Gaussian Process Regression

Can a continuous-time Gaussian process event-camera stereo visual odometry pipeline preserve asynchronous measurement times and still run in real time by using a sparse keytime state representation?

Event cameras produce asynchronous, microsecond-timestamped measurements, but most visual odometry pipelines group events into frames and lose temporal resolution; continuous-time Gaussian process estimators preserve timing but require a state for every measurement, causing prohibitively large state sizes and slow optimization.

Discrete event-based methods such as ES-PTAM, ESVO, and EVO accumulate events into frames or motion-compensated representations, discarding native asynchronous timestamps. Continuous-time approaches like GPCT place a Gaussian process state at every measurement time, which is accurate but not real time. B-spline continuous-time methods require prebuilt maps or numerical Jacobians and often do not achieve real-time performance.

The pipeline uses motion-compensated RANSAC for outlier rejection, then represents the SE(3) trajectory with a Gaussian process and a white-noise-on-acceleration prior at sparse keytimes chosen from event cluster endpoints. Measurements are evaluated at their exact timestamps by interpolating between bounding keytime states using WNOA interpolation matrices and analytical Jacobians. A producer-consumer architecture skips event clusters to maintain real-time estimation. The system is implemented in C++ with GTSAM and iSAM2.

On MVSEC the pipeline ran at about 22 Hz, and on DSEC at about 6 Hz. Compared with the not-real-time GPCT baseline on MVSEC indoor sequences, final translation error changed from 6.6% to 4.8% on indoor_1 and from 5.5% to 6.9% on indoor_3, while RMS relative errors were similar (4.1e-3 vs 4.7e-3 and 3.7e-3 vs 3.8e-3). Against ES-PTAM, global translation error was lower in all tested sequences except dsec_city_4b, where final translation was 156.7 cm versus 33.5 cm for ES-PTAM. Across valid sequences, the paper reports aggregated RMS relative translation and rotation of 0.46 cm and 0.038 degrees, which are 11x and 15x lower than ES-PTAM. For example, on dsec_city_4f, relative translation RMS was 0.4 cm versus 22.3 cm, and relative rotation RMS was 0.015 degrees versus 3.5 degrees.

The authors note only that future work should compare different event representations and feature front ends. Evident limitations include evaluation on two prerecorded datasets using a desktop CPU rather than an embedded or live robotic system; the pipeline provides only visual odometry without mapping; it uses a fixed appearance-based feature extractor; and the DSEC estimation frequency of 6 Hz may be low for high-speed control. Keytime selection is tied to event clusters rather than optimized, and no code or model release is reported.