RYOPO: Bringing End-to-End Category-Level Object Pose Estimation into Real Time
RYOPO is an end-to-end RGB-D model that jointly detects, segments, and estimates 9-DoF category-level object poses without CAD shape priors or an external instance segmentor. On NOCS REAL275 it reaches 52.5 AP at 5°5cm in joint evaluation and 51.1 AP in all-object evaluation, while running at 31.8 FPS on an RTX A6000. This is a preprint; the speed-accuracy balance matters for real-time manipulation where crop-based pipelines scale poorly with object count.
Can a single query-based RGB-D set predictor deliver accurate category-level 9-DoF pose estimates in real time without relying on CAD-derived shape priors or separately trained instance segmentation?
State-of-the-art category-level pose estimators usually run a separate instance segmentor, then crop each object and run a per-object pose network. This repeats encoding and pose prediction for every instance, so latency grows with the number of objects, limiting visual servoing and manipulation throughput. Joint methods often still depend on CAD-derived shape representations.
Prior per-object pipelines such as SPD, GCE-Pose, DPDN, AG-Pose, and CleanPose use supplied detections or masks and often learned shape priors or instance models. Joint detectors like CenterSnap, ShAPO, and FSD use CAD-derived shape representations; YOPO is RGB-only and query-based. Their precision is strong, but runtime scales with object count and many require CAD or instance-specific data.
RYOPO replaces a crop-based pose network with a DETR-style EdgeCrafter encoder and object queries. It back-projects depth and pools query-associated 3D point descriptors using mask confidence and depth-consistency weights. Geometry feedback then updates query content and positional representations via object cross-attention and sparse-voxel scene context. A final refinement module maintains an explicit pose state (rotation, translation, size), encodes it, attends to point descriptors, and applies recurrent residual pose corrections without test-time optimization.
On NOCS joint evaluation, RYOPO scores 85.7 AP at 5°5cm on CAMERA25 and 52.5 on REAL275, exceeding ShAPO by 19.1 and 3.7 points, with IoU50 of 98.6 and 93.0. In all-object scoring over 2,754 REAL275 frames, it reaches 51.1 AP at 5°5cm and 73.6 AP at 10°5cm, the best 10°5cm among compared methods; CleanPose and AG-Pose remain 1.6–4.0 points higher at 5°5cm. On HouseCat6D with common inputs it scores 21.5 AP at 5°5cm versus CleanPose 21.1 and AG-Pose 17.7. Ablation shows +18.2 points at 5°5cm over a base RGB-D predictor. Full-frame pose output runs at 31.8 FPS on RTX A6000, while per-object pipelines plus Mask R-CNN total 96.5–133.4 ms per frame at eight detected objects.
The authors note that at stricter 2 cm translation thresholds on REAL275, RYOPO falls below AG-Pose and CleanPose, and per-object methods remain stronger on cuboid alignment and HouseCat6D 10°5cm AP. The model processes frames independently without tracking or temporal smoothing. This is a preprint; no peer-review acceptance or code release is stated. Reported runtime is on a desktop RTX A6000, and evaluation is limited to NOCS and HouseCat6D categories.
Category-level pose could benefit pick-and-place and bin-picking in logistics and warehouse robots, manufacturing assembly, and domestic mobile manipulators that handle unseen objects. Because it avoids per-instance CAD models and external detectors, it suits GPU-equipped robot cells now; embedded or edge deployment likely needs 1–3 years and validation on more categories, occlusions, and lower-power GPUs. Products would still require grasp planning and robot integration.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.