MEKAWIRE
IntermediatearXiv

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback

Kevin Avdiu, Seongjin Bien, Yannik Blei, Philine Meister, Johannes Hechtl, Philipp Schmitt, Kai Wurm, Georg von Wichert, Wolfram Burgard
In 30 seconds

This preprint presents RUMI, a handheld demonstration interface that matches the Robotiq 2F-85 gripper and combines tactile sensing, augmented reality and contact-triggered resistance. In robot-free data collection, AR visualization plus pose initialization reduced episodes with joint-limit violations to 3/50 from 16/50, suggesting a way to collect more robot-compatible training data.

Research question

Can a commercial-gripper-matched handheld interface collect robot-compatible, tactile-aware demonstrations and support object-dependent grasp-force control through haptic feedback?

Problem

Portable demonstration devices let operators collect data without a robot, but motions can exceed the target arm's limits. Differences between handheld and robot gripper mechanics also complicate transfer. During remote manipulation, missing contact cues make it difficult to regulate grasp force, especially when objects require different handling.

Previous approach

UMI-style devices decouple demonstration collection from robot access, but many use parallel-gripper mechanics and provide no live feasibility feedback. ActiveUMI already adds AR visualization; tactile interfaces such as UMI-FT capture finger forces, while MagiClaw and OmniUMI offer haptics. RUMI combines these capabilities around an existing commercial gripper rather than introducing each capability individually.

New approach

RUMI reproduces Robotiq 2F-85 nonlinear finger motion and mounting geometry through a scissor linkage, with interchangeable CoinFT force/torque or DIGIT tactile fingers. CoinFT measures six-axis loads at 200 Hz. A RealSense D405 supplies wrist images, and a Meta Quest 3 controller tracks pose. The extended IRIS application uses QR registration and inverse kinematics to display a MuJoCo robot shadow. At remote contact, a servo locks the linkage; subsequent squeezing becomes a measured grasp-force command. ACT policies use force-adjusted impedance targets for CoinFT tasks or tactile images from contrastively pretrained, frozen ResNet-18 encoders for DIGIT tasks. The authors provide a code and assets link.

Results

The robot-free study involved 5 participants, each collecting 10 episodes per condition, 100 total. AR plus initialization versus no AR yielded joint-limit violations in 3/50 vs 16/50 episodes and velocity violations in 11/50 vs 29/50; manipulability was 0.055 ± 0.022 vs 0.046 ± 0.0057. On a real Franka FR3 with Robotiq 2F-85, ACT successes with sensors versus without were gear assembly 20/20 vs 7/20, wiping 16/20 vs 7/20, screw tightening 9/20 vs 3/20 and box sorting 17/20 vs 9/20. Training used around 100 demonstrations per task, with 20 evaluations per variant. MuJoCo packing used around 100 demonstrations and 100 rollouts per object per feedback condition, with force amplification set to 1.5. With haptics, bulb/transformer success was 94%/97%; mean demonstration forces were 5.66/22.82 N and rollout forces 5.42/19.86 N. Without haptics, demonstration forces were 18.46/15.83 N; every bulb rollout exceeded the 10 N simulated fracture threshold.

Limitations

The authors identify the wrist-only camera view and missing external-force feedback as limitations. Mechanical blocking signals contact rather than reproducing an external six-axis wrench. Demonstration jitter contributed to slipping during screw tightening. Evaluation covers one robot platform, 5 study participants and 20 real-robot trials per policy variant. Quantitative evidence for haptic force control comes from simulation; the paper describes real-robot teleoperation support but does not establish equivalent real-world performance. The AR comparison combines visualization and initialization, so their separate contributions are unresolved.

Industry impact

Industrial automation integrators, gripper suppliers and robot-learning data providers could use the released materials for research pilots now, particularly around Robotiq 2F-85 installations. A plausible integration horizon is 1-3 years if broader real-robot trials establish repeatable force control, stable contact demonstrations and reliable workspace registration. This is an industry assessment; the paper provides no commercial deployment timetable.

Full paper

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback

Kevin Avdiu, Seongjin Bien, Yannik Blei, Philine Meister, Johannes Hechtl, Philipp Schmitt, Kai Wurm, Georg von Wichert, Wolfram Burgard

Published under CC BY 4.0. Reproduced with attribution; original at arXiv:2611.15802 (PDF).

Abstract

Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator’s arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers. We present RUMI, a handheld universal manipulation interface matched to the Robotiq 2F-85 with exchangeable fingertip sensors that preserves the target gripper geometry while enabling natural input before contact and force measurement after contact. By integrating the tracking system into a virtual reality headset, our method provides real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform. We further introduce a haptic feedback mechanism that supports force-aware teleoperation for simulated and real robots. A user study shows fewer episodes containing joint-limit violations with live robot visualization and pose initialization (3/50) than without augmented reality (16/50). We additionally train imitation policies from demonstrations, collected using two different fingertip modalities enabled by the gripper-matched hardware, and assess the haptic interface’s capability for collecting force-aware teleoperation datasets in simulation. The code and assets for RUMI are available at: https://utn-air.github.io/RUMI/.

Fig. 1: RUMI supports three demonstration modes: virtual reality (VR), teleoperation, and mixed reality (MR). The hardware features scissor input, selective blocking, Robotiq-matched kinematics, and interchangeable fingers, including CoinFT, standard, and DIGIT.

I Introduction

Learning from demonstrations reduces the effort required to program robots for complex or variable tasks [1, 2]. This is increasingly relevant for large visuomotor policies, whose performance depends on the scale and quality of their demonstration data [3]. Collaborative efforts such as Open X-Embodiment [4] aggregate standardized manipulation data across robot platforms, tasks, and institutions using traditional teleoperation approaches, which are typically time-consuming. This limitation has motivated the creation of handheld data collection devices that enable rapid data collection. Universal Manipulation Interfaces (UMIs) support portable, robot-free data collection and thereby decouple data acquisition from continuous access to the target robot [5, 6, 7]. These devices have also gained popularity in the tactile manipulation community, where collecting large-scale datasets remains challenging due to the inherent sparsity of tactile sensor signals [8, 9, 10].

Despite the scalability and speed-up they offer, the inherent robot-free nature of these approaches means that it becomes difficult to observe the validity of the trajectories which they generate. While an intuition for robot-compatible motion can be learned through practice, the lack of feedback necessitates careful post-processing to filter out any unusable data. Furthermore, most existing methods are designed around custom-built robotic grippers, which makes them unsuitable for applications that require more robust grippers.

Another underexplored potential of UMI devices is that they can serve as a more intuitive interface for teleoperation, given their similar appearances and kinematics to target grippers. However, most existing UMI designs lack a way to transmit contact information at time of grasp, which makes it difficult to regulate grasp forces in such scenarios.

To address these challenges, we present the Robotiq Universal Manipulation Interface (RUMI), a handheld platform matched to the nonlinear kinematics, finger attachments, and wrist-camera viewpoint of the Robotiq 2F-85. A scissor-like input drives the coupled fingers, while a rotary encoder measures the resulting grasp coordinate. Interchangeable fingers support force/torque and vision-based tactile sensing for contact-rich demonstrations. During simulated and real-robot teleoperation, a contact-triggered blocker provides low-resistance aperture control before contact and converts the subsequent squeeze into a measured force command. By integrating the handheld pose tracking into a virtual reality (VR) headset, we can project a digital shadow of the target robot platform in augmented reality (AR) fashion during data collection, allowing the operator to recognize infeasible poses while recording rather than filtering them only after collection.

In summary, we make the following contributions:

  • RUMI, a Robotiq 2F-85-matched handheld interface with nonlinear actuation and interchangeable fingertips;
  • An AR pipeline for visualizing and validating robot configurations during robot-free data collection;
  • A haptic feedback mechanism which allows for gripper force control during simulated and real teleoperation;
  • Releasing the device design and software framework as well as a dataset of demonstrations featuring force/torque and vision-based tactile sensor data.

II Related Work

TABLE I: Reported capabilities of handheld demonstration interfaces. \checkmark: reported; –: not reported. Grasp force excludes wrist-only loads; visual guidance is operator-facing; added haptics excludes natural contact.
  • Two-finger interfaces without fingertip contact sensors
    Contact sensing
    No data
    Grasp force
    No data
    Live robot visualization
    No data
    Gripper haptics
    No data
    Tracking / capture setup
    No data
    Robot counterpart
    No data
  • UMI [5]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Visuoinertial SLAM
    Robot counterpart
    WSG-50
  • FastUMI [6]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Realsense T265
    Robot counterpart
    Parallel gripper
  • ForceMimic [11]
    Contact sensing
    Wrist force
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Realsense T265
    Robot counterpart
    Parallel gripper
  • ManipForce [12]
    Contact sensing
    Wrist force
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    ArUco tracking
    Robot counterpart
    Parallel gripper
  • ActiveUMI [13]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    \checkmark AR Projection
    Gripper haptics
    –
    Tracking / capture setup
    Quest VR + Backpack
    Robot counterpart
    Custom bimanual system
  • UMIGen [14]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Realsense T265
    Robot counterpart
    Parallel gripper
  • XRZero-G0 [15]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    PICO VR + backpack
    Robot counterpart
    Pinch gripper
  • UMI-3D [16]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Onboard LiDAR
    Robot counterpart
    Parallel gripper
  • YUBI [17]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Static Quest VR
    Robot counterpart
    Pinch gripper
  • EgoGuide [18]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Quest VR
    Robot counterpart
    Flexiv Grav
  • HiFi-UMI [19]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Offl. SLAM + ArUco
    Robot counterpart
    Pinch gripper
  • Two-finger interfaces with fingertip contact sensors
    Contact sensing
    No data
    Grasp force
    No data
    Live robot visualization
    No data
    Gripper haptics
    No data
    Tracking / capture setup
    No data
    Robot counterpart
    No data
  • ViTaMIn [20]
    Contact sensing
    Optical tactile
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Visuoinertial SLAM
    Robot counterpart
    PGI-140-80-W-S
  • FreeTacMan [10]
    Contact sensing
    Optical tactile
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    NOKOV
    Robot counterpart
    Parallel gripper
  • Touch in the Wild [21]
    Contact sensing
    Resistive tactile
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Visuoinertial SLAM
    Robot counterpart
    Parallel gripper
  • exUMI [22]
    Contact sensing
    Modular tactile
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Quest
    Robot counterpart
    Grav / Robotiq 2F
  • MagiClaw [23]
    Contact sensing
    Optical tactile
    Grasp force
    \checkmark Estimated
    Live robot visualization
    –
    Gripper haptics
    \checkmark Motors
    Tracking / capture setup
    iPhone SLAM
    Robot counterpart
    Shared MagiClaw
  • FARM [9]
    Contact sensing
    Optical tactile
    Grasp force
    \checkmark Estimated
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    OptiTrack
    Robot counterpart
    Actuated UMI
  • ViTaMIn-B [24]
    Contact sensing
    Optical tactile
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Quest
    Robot counterpart
    Parallel gripper
  • UMI-FT [8]
    Contact sensing
    Finger force
    Grasp force
    \checkmark Direct
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    iPhone SLAM
    Robot counterpart
    WSG-50
  • TacUMI [25]
    Contact sensing
    Tactile + wrist
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Vive tracking stations
    Robot counterpart
    Parallel gripper
  • OmniUMI [26]
    Contact sensing
    Tactile + wrist
    Grasp force
    \checkmark Estimated
    Live robot visualization
    –
    Gripper haptics
    \checkmark Motors
    Tracking / capture setup
    Vive tracking stations
    Robot counterpart
    Parallel gripper
  • Multi-finger grippers and dexterous hands
    Contact sensing
    No data
    Grasp force
    No data
    Live robot visualization
    No data
    Gripper haptics
    No data
    Tracking / capture setup
    No data
    Robot counterpart
    No data
  • DexUMI [27]
    Contact sensing
    Tactile arrays
    Grasp force
    \checkmark Distributed
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    iPhone SLAM
    Robot counterpart
    Custom hand
  • MV-UMI [28]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    GoPro + overhead
    Robot counterpart
    Custom hand
  • RealDexUMI [29]
    Contact sensing
    Resistive tactile
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Pose tracker
    Robot counterpart
    Custom hand
  • Koala Gripper [30]
    Contact sensing
    –
    Grasp force
    –
    Live robot visualization
    –
    Gripper haptics
    –
    Tracking / capture setup
    Pose camera
    Robot counterpart
    Custom gripper
  • RUMI (ours)
    Contact sensing
    Force / optical tac.
    Grasp force
    \checkmark Direct
    Live robot visualization
    \checkmark Robot
    Gripper haptics
    \checkmark Blocker
    Tracking / capture setup
    Quest
    Robot counterpart
    Robotiq 2F-85 [31]

II-A Robot-Free Demonstration and Embodiment Transfer

We provide a comprehensive overview of representative robot-free data collection approaches in Tab. I. Existing UMI-style systems use portable handheld grippers with gripper-centric observations and pose tracking to collect demonstrations without an active target robot [5, 6, 7, 17, 28]. The original UMI design already supports transfer across parallel grippers through replaceable finger geometries, while later systems explore closer correspondence to specific robot embodiments, such as exUMI [22]. ActiveUMI [13] further incorporates AR-based robot visualization during robot-free collection.

RUMI follows these directions by matching the nonlinear finger kinematics and mounting geometry of the Robotiq 2F-85, enabling straightforward deployment on existing robot platforms without requiring a custom robot-side gripper or additional compatibility solution. It also uses live robot projection during collection for online kinematic checking.

II-B Multimodal Handheld Devices and Haptic Teleoperation

Several UMI systems extend pose and gripper-width measurements with tactile or force sensing for contact-rich manipulation [8, 9, 24, 25]. RUMI similarly supports interchangeable fingertip sensors through a compatible finger mount, allowing the sensing configuration to be adapted without changing the underlying gripper embodiment.

Haptic feedback and teleoperation have also been incorporated into UMI-style devices, including MagiClaw [23] and OmniUMI [26]. RUMI follows this direction with a selectively engaged gripper-haptic mechanism, allowing the same handheld device to support both in-the-wild collection and teleoperation.

Overall, prior work has explored robot-free collection, transferable gripper designs, tactile and force sensing, live robot visualization, and haptic teleoperation in different combinations. RUMI brings these capabilities together in a single handheld interface with direct correspondence to an existing commercial gripper.

III Method

III-A Design Requirements

RUMI is designed for industrial manipulation settings, where robots equipped with commercial grippers are commonly used for task automation. We derive four main requirements from this setting.

First, the interface should reproduce the geometry and grasp actuation of a commonly used industrial gripper, specifically the Robotiq 2F-85, such that demonstrations can be collected using an embodiment consistent with the target system. Second, the fingers should be modular to support different tactile sensing modalities depending on the task. Third, robot-free demonstrations should remain compatible with the target robot’s workspace and kinematic constraints despite the robot being absent from the physical interaction loop. Finally, data collection should be intuitive and minimally intrusive to the production environment, while the same interface should also support remote teleoperation when direct access to the robot workspace is restricted due to safety or spatial constraints.

III-B Hardware and Sensing

RUMI implements a crossed, scissor-like linkage that couples the motion of the operator’s thumb and index finger to the gripper fingers (Fig. 2). Its linkage geometry and finger attachment interface are derived from the official Robotiq 2F-85 CAD model. The operator actuates two rear input pads, while an AS5600 magnetic rotary encoder measures the resulting gripper aperture. The scissor-like input interface allows direct mapping between operator’s applied force to the gripper-side force, further complemented by a CoinFT [32] sensor on the index finger pad that captures the applied force.

The linkage can additionally be physically locked to provide haptic feedback (Fig. 2 c). A 9g Miuzei micro servo engages a toothed counterpart with the internal coupling gear, preventing further motion of the linkage. We selected this actuator because it is widely available and provides sufficient torque density for the compact mechanism.

The finger attachment points reproduce the 2F-85 mounting geometry, allowing the same finger modules to be transferred between RUMI and the robotic gripper, including the manufacturer’s original fingers. We additionally provide CAD files for fingertip modules compatible with two sensing modalities: the CoinFT sensor, which measures a six-axis wrench at 200 Hz, and the DIGIT [33] vision-based tactile sensor, which provides high-resolution tactile observations.

A Teensy 4.0 microcontroller coordinates the on-device sensing and actuation. It synchronizes measurements from the rotary encoder and any connected CoinFT sensors, transmits these measurements to the host computer via USB, and controls the servo used by the blocking mechanism. DIGIT sensors, when used, bypass the microcontroller and connect directly to the host computer.

A RealSense D405 is mounted at the wrist attachment point in the same pose used on the target robot. A Meta Quest 3 controller is rigidly attached to RUMI to provide six-DoF pose tracking relative to the headset.

III-C Robot-State Visualization

Robot-free data collection removes the target robot from the physical interaction loop and therefore also removes direct feedback about its workspace and kinematic limits. Without such feedback, an operator may generate motions that cannot be reproduced by the target robot.

To expose these constraints during demonstration, we extend the IRIS [34] AR application with support for RUMI. During recording, IRIS renders a MuJoCo-based [35] digital shadow of the target robot. The tracked RUMI pose is transformed into the robot base frame and used as the desired end-effector pose; an inverse-kinematics solver computes the corresponding robot configuration for visualization. The operator can therefore observe the robot configuration associated with the demonstrated motion in real time.

Providing meaningful feedback requires the virtual robot base to be accurately registered with the physical workspace. We use multiple QR codes to estimate the base pose and refine its orientation by fitting a plane to depth measurements of the deployment surface. This registration establishes the transformation between the headset tracking frame, the physical workspace, and the robot base frame. IRIS additionally supports configurable offsets from the detected base pose, allowing the virtual robot to be placed where its physical counterpart could not be accommodated because of spatial restrictions. An example is shown in Fig. 4a.

The same registration is used to express recorded RUMI trajectories in the robot base frame. Before starting an episode, the operator can additionally use a catch-to-teleop initialization procedure in which RUMI is aligned with the projected robot end-effector. This establishes a consistent initial end-effector pose while allowing the corresponding robot configuration to be inspected before and during the demonstration.

Fig. 2: Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T) sensing. (b) The device rests on the back of the hand, while the thumb and index finger operate the rear scissor pads. A human-side CoinFT sensor measures force at the index-side input. (c) The nonlinear transmission couples finger motion, while a selectively engaged blocker constrains further closure. (d) The same interface supports three contact domains: virtual contact in VR, remote robot contact during teleoperation, and direct contact with a local object in MR. In VR and teleoperation, contact engages the blocker, providing resistance at the input pads; subsequent squeeze is measured by the human-side CoinFT sensor. In MR, the local object supplies the physical reaction force, and fingertip F/T sensors measure the interaction while the blocker and human-side force-sensing path remain inactive. Mode scenes are schematic.

III-D Operating Modes and Contact Handling

RUMI uses the same grasp interface for direct in-the-wild demonstration, simulated teleoperation, and real-robot teleoperation (Fig. 2). The modes differ primarily in how contact is generated and communicated to the operator.

During in-the-wild demonstrations, contact occurs directly between RUMI’s fingers and the manipulated object. After the fingers make contact, the operator can continue squeezing the input pads, and the resulting force is measured by the actuation-side CoinFT sensor.

During teleoperation, contact occurs only at the remote or simulated gripper. Before contact, the blocking mechanism remains disengaged and the encoder measurement directly controls the commanded gripper aperture, allowing low-resistance motion of the RUMI linkage. When the remote gripper detects contact, the blocker engages and prevents further linkage closure. Subsequent squeezing is then measured by the actuation-side CoinFT sensor and mapped to a grasp-force command. An optional force amplification factor reduces the physical force required from the operator. When the operator releases the applied force, the blocker disengages and the interface returns to aperture-control mode.

The source of the contact signal depends on the teleoperation setting. In simulation, contact and force response are obtained from the simulated gripper; during real-robot teleoperation, contact is inferred from the physical gripper state and the measured RUMI force is sent as the gripper force command. The Meta Quest controller can additionally provide vibration feedback indicating remote contact and the approximate magnitude of the commanded load. This vibration complements the binary mechanical blocking cue but is not intended to reproduce an arbitrary external six-axis wrench.

III-E Data Recording and Policy Interface

We use Robot Control Stack [36] as the underlying software architecture for data acquisition and robot communication. At each timestep, the system records the RUMI pose, gripper state, camera observations, and measurements from the selected fingertip sensors when available.

For robot-free demonstrations, RUMI poses are stored in the registered robot base frame established by the AR system. Robot actions can then be derived from the recorded trajectory either by temporally shifting the pose sequence or by computing the relative motion between consecutive poses.

During teleoperation, the RUMI input is instead used online as the commanded action, while the measured robot state is used as the observation. Both acquisition modes therefore share the same episode structure, while differing in whether actions are reconstructed from a robot-free demonstration or issued directly during teleoperation.

The resulting episodes can be converted into different observation-action representations for imitation learning. Observations can include camera streams, robot or RUMI poses, gripper state, blocking-mechanism state, and available fingertip-sensor measurements, while actions contain the desired robot motion and gripper command. For robot-free demonstrations, the inverse-kinematics solution associated with the AR projection can additionally provide joint configurations along the demonstrated trajectory, allowing the recordings to support both Cartesian pose-based and joint-space policy representations.

IV Experimental Evaluation

Fig. 3: Real-world tasks used to evaluate our data acquisition approach: (a) whiteboard wiping, (b) gear insertion, (c) screw tightening, and (d) box sorting. Evaluation also includes one simulated task (not shown).

To validate the effectiveness of RUMI, we conduct a series of experiments, focusing on A) usability, B) transfer learning on an imitation policy, and C) haptic feedback utility. For experiments conducted in subsections A) and B), we set up an office table covered with wooden boards (Fig. 4a) to mimic the inference setup (Fig. 4c). We place one QR code at each table corner, and calibrate the base pose of the robot. Our inference environment consists of a Franka FR3 robot arm equipped with a Robotiq 2F-85 gripper and a RealSense D405 wrist camera. The haptic feedback utility experiment is conducted in a digital twin of the inference setup. The complete setup is shown in Fig. 4c.

IV-A Data quality comparison

To test whether our proposed AR-based digital twin projection provides meaningful assistance in the quality of demonstration data, we conduct a small user study. We recruit 5 participants with limited experience in teleoperating a robotic arm with VR controllers. The participants are asked to perform a simple box pick-and-place task. We collect demonstrations under two conditions. In the first condition, the participants perform the task without any sort of AR projection, and are informed beforehand to start each episode from the same starting pose as close as possible. We name this condition w/o AR. In the second condition, we enable the AR projection and the catch-to-teleop functionality. We call this condition AR. Each participant performs 10 demonstrations per condition, resulting in 100 total episodes.

With the collected data, we calculate 4 metrics: initial pose variance, average Cartesian space velocity of the VR controller, number of episodes containing joint limit and velocity violations, and average Yoshikawa manipulability across the whole dataset. Joint velocity and position limits are taken from FR3’s technical data. The results of the analysis are shown in Fig. 5 and Tab. II. Looking at Tab. II, the AR condition results in fewer number of joint limit- and velocity violations. The average manipulability is 19.6% higher than the w/o AR condition, though the standard deviation is also greater. Fig. 5 shows that the catch-to-teleop feature in the AR condition results in a much tighter cluster of initial poses. Additionally, the average Cartesian space velocity is lower, indicating that the projected robot encourages the participant to slow down until the digital shadow is able to catch up. Given that the participants are unfamiliar with VR-based robot teleoperation, the results suggest that our AR-based robot visualization can assist inexperienced users with collecting high-quality data.

Fig. 4: Experimental environments used to evaluate RUMI. (a) Robot-free data collection with QR-code markers and an augmented-reality robot overlay for recording-time validation. (b) Simulated packing task involving a fragile light bulb and a heavy transformer block. (c) Real-world setup for policy deployment using a Franka FR3 robot equipped with a Robotiq 2F-85 gripper.

IV-B Imitation Learning from RUMI Demonstrations

The second experiment is to test whether RUMI is capable of collecting a high-quality dataset that can be used for training an imitation policy on a real robot. Furthermore, to test RUMI’s versatility in using different fingertip sensors, we collect datasets for the CoinFT [32] and the DIGIT [33] optical tactile sensor. We design four contact-rich manipulation tasks, two for each sensor type, as shown in Fig. 3. Additionally, we divide each task into 3 broad stages: 1) grasp success, 2) task attempt, and 3) task success, in order to better assess where each policy variant fails. We use the Action Chunking Transformer (ACT) [37] for all trained models.

For the CoinFT sensor, we collected datasets for board wiping and gear insertion tasks (Fig. 3a,b). In the board wiping task, the robot must first pick up a whiteboard eraser, which is fitted into a 3D-printed holder for easier grasp. Then, it must erase the star pattern drawn on the board positioned in front of it. The position and shape of the star pattern is varied across each trial. The gear insertion task involves a small 3D-printed setup consisting of three gears, drawn from the ManipulationNet benchmark [38]. The robot must pick up a gear and insert it into the center position of the gear assembly and ensure that the teeth align for a full insertion. The positions of the main assembly plate and the gear are not randomized.

We evaluate the screw tightening and box sorting tasks for the DIGIT sensor, as the optical tactile sensor’s high-resolution tactile information makes it suitable for sensing both shear force and surface texture (Fig. 3c,d). For the screw tightening task, the robot must tighten a cap already mounted on a 3D-printed screw fixture. The number of rotations required for tightening varies across each demonstration. The box sorting task involves two 3D-printed boxes with different surface texture patterns. The robot must then sort these boxes into their respective plates positioned adjacent to them.

Fig. 5: Top: Initial poses for all episodes under the two user study conditions, split across translation (left) and rotation (right). Bottom: Average Cartesian space velocity under the two conditions.

We collect around 100 demonstrations per task with their appropriate fingertip sensors attached to the RUMI in the setup described above. For the CoinFT tasks, we train one model with force-adjusted impedance targets:

z^{\prime}=z+F_{z,\text{CoinFT}}/K,

where z is the recorded z-axis translation, K is an empirically chosen value of 1000, corresponding to 10\text{\,}\mathrm{N}\mathrm{/}\mathrm{c}\mathrm{m}, and F_{z,\text{CoinFT}} is the z-axis force measured by the CoinFT sensor. We train another model using the raw observed trajectories, with wrist camera images being available for both. For the DIGIT tasks, we follow the training recipe described in VITaL [39]: we first perform contrastive pretraining on a ResNet-18-based [40] tactile encoder backbone along with a ResNet-18-based vision encoder backbone. This combined backbone is then frozen to train the corresponding ACT policy with tactile images. Again, another variant is trained without the tactile image input using just the observed trajectories and the wrist camera images. This setup allows us to test whether tactile data collected with RUMI contribute meaningfully towards completing the tasks successfully. We evaluate each model variant 20 times. The results are shown in Tab. III.

The results show a clear correlation between task success rates and the availability of their respective fingertip tactile sensors. In the gear assembly task, we observe that the force-adjusted trajectories are able to reliably push the gear into alignment with its neighbors, also aided by the force-adjusted gripper strength that allows for passive compliance. In the no-tactile case, although the gear is often successfully inserted through the stick, the lack of further downward trajectory means that the policy often releases its grasp without pushing the gear into alignment. For the board task, the no-tactile variant is not able to completely wipe the pattern off the whiteboard despite passing over it with the eraser due to the lack of downward pressure.

The majority of failures with the screw tightening DIGIT variant comes from the fact that the policy struggles to maintain a stable contact with the cap for long enough to fully tighten it, resulting in slippage. This is expected, as RUMI operation creates a lot of jitter in the demonstration, which makes the contact stage of the task unstable. Nonetheless, the DIGIT variant outperforms the raw variant, which frequently released its grasp after barely rotating the cap, as it lacks the shear force signal to understand when the task is completed. For the box sorting task, the raw variant overfits to one particular trajectory, as it lacks information about the different boxes’ textures. The DIGIT variant was able to successfully sort the box correctly in all but one valid attempt. These results indicate that RUMI is capable of recording useful demonstrations with different fingertip sensors.

TABLE II: Ablation of data quality with and without AR projection. Joint limit- and velocity violations report affected episodes/total episodes, and average manipulability indicates the average value across the entire dataset.
  • Joint Limit Vio. \downarrow
    AR
    3/50
    w/o AR
    16/50
  • Joint Velocity Vio. \downarrow
    AR
    11/50
    w/o AR
    29/50
  • Avg. Manip. \uparrow
    AR
    0.055\pm 0.022
    w/o AR
    0.046\pm 0.0057

IV-C Evaluation in Simulation

We investigate whether RUMI can collect force-aware demonstrations in simulation and whether the contact-triggered blocker helps operators to provide object-dependent grasp-force commands. Using the MuJoCo simulator, the experiment reproduces an industrial packing task of transferring a fragile light bulb [41] and a substantially heavier transformer block into a shipping box. Here, we set the gripper force amplification factor to 1.5 to reduce the force required from the operator. The two objects require distinct handling strategies: the bulb must be grasped gently, whereas the transformer requires a stronger grasp to prevent slip.

We collect around 100 demonstrations in total and use them to train the same imitation policy described in the previous subsection. A rollout is successful when the selected object is transferred into the shipping box. Bulb rollouts additionally fail if the maximum grasp force exceeds 10\text{\,}\mathrm{N}, which serves as a simulated fracture threshold. For the transformer, insufficient grasp force leads to object slippage.

To isolate the effect of haptic contact feedback during data collection, we train a second policy using demonstrations recorded in the same setting with the contact trigger and blocker disabled. In this condition, operators command the gripper through a position controller, where grasp force arises implicitly from the position error. The operator then visually estimates the quality of the grasp and adjusts the gripper width as needed. We compare the mean grasp force in the demonstrations (F{}_{\text{DS}}), the mean force produced by policy rollouts (F{}_{\text{R}}), and the success rate based on 100 rollouts for each object and feedback condition.

As shown in Table IV, demonstrations collected with haptic feedback exhibit a clear object-dependent force strategy: operators apply 5.66\text{\,}\mathrm{N} to the bulb and 22.82\text{\,}\mathrm{N} to the transformer on average. The learned policies preserve this separation during rollout, producing mean forces of 5.42\text{\,}\mathrm{N} and 19.86\text{\,}\mathrm{N}, respectively, and achieve success rates of 94% and 97%. Without haptic feedback, the demonstration forces converge toward less task-appropriate values: bulb force rises to 18.46\text{\,}\mathrm{N}, causing every rollout to exceed the fracture threshold, while transformer force falls to 15.83\text{\,}\mathrm{N}. These results indicate that the blocker provides a useful contact reference for demonstrating force-sensitive behavior and that the learned policy reproduces the resulting object-dependent grasp strategy.

TABLE III: Task success rates across individual stages and overall execution. Bracket indicates the sensor used to train the model.
  • Gear Assembly (CoinFT)
    Grasp
    20/20
    Attempt
    20/20
    Success
    20/20
  • Gear Assembly (None)
    Grasp
    20/20
    Attempt
    20/20
    Success
    7/20
  • Whiteboard Wiping (CoinFT)
    Grasp
    20/20
    Attempt
    18/20
    Success
    16/20
  • Whiteboard Wiping (None)
    Grasp
    20/20
    Attempt
    17/20
    Success
    7/20
  • Screw Tightening (DIGIT)
    Grasp
    17/20
    Attempt
    17/20
    Success
    9/20
  • Screw Tightening (None)
    Grasp
    17/20
    Attempt
    17/20
    Success
    3/20
  • Box Sorting (DIGIT)
    Grasp
    18/20
    Attempt
    18/20
    Success
    17/20
  • Box Sorting (None)
    Grasp
    19/20
    Attempt
    19/20
    Success
    9/20

V Conclusions

In this work, we presented RUMI, a gripper-matched handheld interface for tactile-enabled data collection, modeled after the Robotiq 2F-85 gripper. The device combines gripper-pose tracking with compatibility to diverse fingertip sensors and user-side grasp-force measurement and provides haptic feedback during teleoperation. We demonstrated its usability for robot-free data collection, imitation policy training, and simulated teleoperation. By providing a consistent physical interface across these modes, RUMI enables the collection of embodiment-matched, tactile-aware demonstrations for robot learning. Our method’s current limitations include being restricted to only a wrist camera viewpoint and the lack of external force feedback to complement the blocking mechanism. Future work could include generative AI to overcome the single-view limitation as well as more informative visual cues about external forces acting upon the simulated or teleoperated robot.

Acknowledgment

TABLE IV: Simulated packing results. Haptic feedback produces object-dependent grasp forces in demonstrations (F{}_{\text{DS}}) that persist in policy rollouts (F{}_{\text{R}}).
  • Bulb
    Avg. F{}_{\text{DS}} [N]
    5.66
    Avg. F{}_{\text{R}} [N]
    5.42
    Success (\uparrow)
    94%
  • Block
    Avg. F{}_{\text{DS}} [N]
    22.82
    Avg. F{}_{\text{R}} [N]
    19.86
    Success (\uparrow)
    97%
  • Bulb (no haptic)
    Avg. F{}_{\text{DS}} [N]
    18.46
    Avg. F{}_{\text{R}} [N]
    –
    Success (\uparrow)
    0%
  • Block (no haptic)
    Avg. F{}_{\text{DS}} [N]
    15.83
    Avg. F{}_{\text{R}} [N]
    13.67
    Success (\uparrow)
    99%

All work related to the CoinFT sensors was performed by and at UTN. Siemens’ contribution was limited to the mechanical design of the device. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the BayernKI project v106be. BayernKI funding is provided by Bavarian state authorities. This work has been partially supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG). This work has been partially supported by the project GeniusRobot funded by the German Federal Ministry of Education and Research (BMBF grant no. 01IS24083). We further thank Volker Schneider and Devadas Sheela for their assistance with hardware development and setup. OpenAI ChatGPT and Claude Code were used to assist with debugging and the generation of limited boilerplate and data-processing code. All generated code was reviewed, adapted, and validated by the authors. The research methodology, algorithm design, analysis, and interpretation of results were performed by the authors. ChatGPT was further used to remove background elements from Figure 4 a.

References

  1. [1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robot. Auton. Syst., vol. 57, no. 5, pp. 469–483, 2009.
  2. [2] G. Liu, W. Sun, and P. Li, “Motion capture and AR based programming by demonstration for industrial robots using handheld teaching device,” Sci. Rep., vol. 14, 2024.
  3. [3] M. J. Kim et al., “OpenVLA: An open-source vision-language-action model,” in Proc. Conf. Robot Learn. (CoRL), 2024.
  4. [4] Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic learning datasets and RT-X models,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2024.
  5. [5] C. Chi et al., “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” 2024.
  6. [6] Zhaxizhuoma et al., “FastUMI: A scalable and hardware-independent universal manipulation interface with dataset,” in Proc. Conf. Robot Learn. (CoRL), 2025.
  7. [7] K. Liu et al., “FastUMI-100K: Advancing data-driven robotic manipulation with a large-scale UMI-style dataset,” in arXiv:2510.08022v1, 2025.
  8. [8] H. Choi et al., “In-the-wild compliant manipulation with UMI-FT,” in arXiv:2601.09988v1, 2026.
  9. [9] E. Helmut, N. Funk, T. Schneider, C. de Farias, and J. Peters, “Tactile-conditioned diffusion policy for force-aware robotic manipulation,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2026.
  10. [10] L. Wu et al., “FreeTacMan: Robot-free visuo-tactile data collection system for contact-rich manipulation,” in arXiv:2506.01941v4, 2026.
  11. [11] W. Liu, J. Wang, Y. Wang, W. Wang, and C. Lu, “ForceMimic: Force-centric imitation learning with force-motion capture system for contact-rich manipulation,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025.
  12. [12] G. Lee et al., “ManipForce: Force-guided policy learning with frequency-aware representation for contact-rich manipulation,” in arXiv:2509.19047v1, 2025.
  13. [13] Q. Zeng et al., “ActiveUMI: Robotic manipulation with active perception from robot-free human demonstrations,” in arXiv:2510.01607v1, 2025.
  14. [14] Y. Huang, S. Li, X. Li, and W. Ding, “UMIGen: A unified framework for egocentric point cloud generation and cross-embodiment robotic imitation learning,” in arXiv:2511.09302v1, 2025.
  15. [15] J. Wang et al., “XRZero-G0: Pushing the frontier of dexterous robotic manipulation with interfaces, quality and ratios,” in arXiv:2604.13001v2, 2026.
  16. [16] Z. Wang, “UMI-3D: Extending universal manipulation interface from vision-limited to 3D spatial perception,” in arXiv:2604.14089v1, 2026.
  17. [17] T. Ohkawa et al., “YUBI: Yielding universal bidigital interface for bimanual dexterous manipulation at scale,” in 1st ICRA Workshop on ”Beyond Teloperation: Learning from Diverse Human and Simulation Data”, 2026.
  18. [18] Y. Xu et al., “EgoGuide: Egocentric guidance for efficient robot-free demonstration collection and learning,” in arXiv:2606.14665v1, 2026.
  19. [19] Simple AI et al., “HiFi-UMI: Learning deployable manipulation policies from high-fidelity UMI data alone,” in arXiv:2607.25895v1, 2026.
  20. [20] F. Liu, C. Li, Y. Qin, J. Xu, P. Abbeel, and R. Chen, “ViTaMIn: Learning contact-rich tasks through robot-free visuo-tactile manipulation interface,” in arXiv:2504.06156v2, 2025.
  21. [21] X. Zhu, B. Huang, and Y. Li, “Touch in the wild: Learning fine-grained manipulation with a portable visuo-tactile gripper,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2025.
  22. [22] Y. Xu, L. Wei, P. An, Q. Zhang, and Y.-L. Li, “exUMI: Extensible robot teaching system with action-aware task-agnostic tactile representation,” in Proc. Conf. Robot Learn. (CoRL), 2025.
  23. [23] T. Wu et al., “MagiClaw: A dual-use, vision-based soft gripper for bridging the human demonstration to robotic deployment gap,” in arXiv:2509.19169v1, 2025.
  24. [24] C. Li et al., “ViTaMIn-B: A reliable and efficient visuo-tactile bimanual manipulation interface,” in arXiv:2511.05858v2, 2025.
  25. [25] T. Cheng et al., “TacUMI: A multi-modal universal manipulation interface for contact-rich tasks,” in arXiv:2601.14550v1, 2026.
  26. [26] S. Luo et al., “OmniUMI: Towards physically grounded robot learning via human-aligned multimodal interaction,” in arXiv:2604.10647v3, 2026.
  27. [27] M. Xu et al., “DexUMI: Using human hand as the universal manipulation interface for dexterous manipulation,” in Proc. Conf. Robot Learn. (CoRL), 2025.
  28. [28] O. Rayyan, J. Abanes, M. Hafez, A. Tzes, and F. Abu-Dakka, “Mv-umi: A scalable multi-view interface for cross-embodiment learning,” IEEE Access, vol. 14, pp. 104 545–104 555, 2026.
  29. [29] C. Xu et al., “RealDexUMI: A wearable universal manipulation interface for dexterous robot learning,” in arXiv:2606.06033v2, 2026.
  30. [30] A. Hajj-Ahmad et al., “Koala gripper: Co-designing robotic grippers and data-capture devices for scaling dexterous manipulation learning,” in arXiv:2608.20546v1, 2026.
  31. [31] Robotiq, “Adaptive grippers: 2F-85 and 2F-140,” Product documentation, accessed: Sep. 15, 2026. [Online]. Available: https://robotiq.com/products/adaptive-grippers
  32. [32] H. Choi et al., “Coinft: A coin-sized, capacitive 6-axis force torque sensor for robotic applications,” arXiv preprint arXiv:2503.19225, 2025.
  33. [33] M. Lambeta et al., “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” IEEE Robot. Autom. Lett., vol. 5, no. 3, pp. 3838–3845, 2020.
  34. [34] X. Jiang et al., “IRIS: An immersive robot interaction system,” in Proc. Conf. Robot Learn. (CoRL), 2025.
  35. [35] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2012.
  36. [36] T. Jülg et al., “Robot Control Stack: A lean ecosystem for robot learning at scale,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2026.
  37. [37] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proc. Robot.: Sci. Syst. (RSS), 2023.
  38. [38] Y. Chen et al., “Manipulationnet: An infrastructure for benchmarking real-world robot manipulation with physical skill challenges and embodied multimodal reasoning,” in arXiv:2603.04363, 2026.
  39. [39] A. George, S. Gano, P. Katragadda, and A. B. Farimani, “Vital pretraining: Visuo-tactile pretraining for tactile and non-tactile manipulation policies,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025.
  40. [40] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016.
  41. [41] Dileepkumar Khajana, “Spiral light bulb,” GrabCAD. https://grabcad.com/library/spiral-light-bulb-2, accessed: 2026-09-12.