ColoACT: Multi-Cue Action Chunking for Smooth Autonomous Colon Navigation on a Self-Propelled Endoscopic Robot
This is a preprint describing ColoACT, an imitation-learning navigation system for a compact self-propelled endoscopic robot. The policy fuses RGB, relative depth, and a gradient-based pseudo-elevation map with an Action Chunking Transformer and temporal ensembling. In ex-vivo porcine colons it reached 85.4% success on straight and 72.5% on curved segments, with smooth steering and success on 90-degree turns (70%) and double bends (60%), suggesting a path toward reducing physician burden in colonoscopy.
Can a multi-cue RGB-depth-elevation observation combined with action chunking and temporal ensembling enable smooth, robust autonomous navigation of a self-propelled colonoscopic robot through ex-vivo porcine colons, including tortuous bends?
Autonomous colon navigation must operate in deformable tissue with weak texture, specular highlights, and viscoelastic contact forces. Geometry-driven pipelines are brittle because they rely on handcrafted features and switching logic, while learned policies can be temporally inconsistent or jittery and suffer from sim-to-real gaps when trained in simulation.
Prior work includes geometry-driven lumen-center fitting and haustral fold detection, reinforcement learning trained in simulated colon models, monocular depth inference, and naive behavior cloning. Geometry methods fail under noisy visuals and non-ideal lumen shapes; simulation-trained RL transfers poorly due to unmodeled viscoelastic contact; step-wise behavior cloning produces high-frequency jitter and short-horizon errors.
ColoACT combines a custom 24×20×34 mm self-propelled Bevel-Gear-Based Endoscopic Robot (BGER) with PDMS treads and a bevel-gear transmission, and a CVAE-based Action Chunking Transformer. The policy takes 320×240 inputs with five channels: RGB, Depth Anything V2-Base relative depth, and a Scharr-gradient log-normalized pseudo-elevation map. A ResNet-18 backbone encodes observations; the decoder predicts 16-step action chunks of normalized linear and angular velocity, and overlapping predictions are fused with a boxcar temporal ensemble. Training used 214 teleoperated ex-vivo porcine trajectories.
On four unseen porcine colons (~60 cm), RGB-D-E ACT achieved 85.4% success on 48 straight trials and 72.5% on 40 curved trials, outperforming RGB-only ACT (35.4%/25.0%), RGB-D ACT (58.3%/50.0%), and RGB-D-E behavior cloning (45.8%/57.5%). In complex segments it succeeded in 70% of 90-degree turns (7/10), 60% of double bends (6/10), and 30% of triple bends (3/10). The ACT policy reduced normalized angular jerk from about 60 a.u. for behavior cloning to below 5 a.u. Offline ADE/FDE over 43 validation samples were 0.0835/0.1689 for RGB-D-E, 23.8% better ADE and 18.6% better FDE than RGB-D. Perception stability over 50 static frames showed depth std 0.0071 and elevation std 0.0250. Real-time control ran at 15 Hz on an RTX 4060.
The authors note the lack of force sensing and validation only in ex-vivo tissue; they plan in-vivo trials to address peristalsis and lubrication. Trial counts are modest (up to 48 per condition, 10 per complex bend), all tests use a single custom robot, and no code or data release is reported. Generalization to living human anatomy and clinical safety remain unproven.
This work is most relevant to medical-device companies building robotic colonoscopy or autonomous endoscopic platforms, including flexible robotic endoscope startups and established endoscopy firms. Research groups can use the policy now; preclinical and product prototyping may follow in 1–3 years if in-vivo and force-sensing results are positive. Clinical deployment is 3+ years away and would require larger animal/human trials, tissue-contact safety mechanisms, regulatory clearance, and evidence of reduced operator workload.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.