RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback
This preprint presents RUMI, a handheld demonstration interface that matches the Robotiq 2F-85 gripper and combines tactile sensing, augmented reality and contact-triggered resistance. In robot-free data collection, AR visualization plus pose initialization reduced episodes with joint-limit violations to 3/50 from 16/50, suggesting a way to collect more robot-compatible training data.
Can a commercial-gripper-matched handheld interface collect robot-compatible, tactile-aware demonstrations and support object-dependent grasp-force control through haptic feedback?
Portable demonstration devices let operators collect data without a robot, but motions can exceed the target arm's limits. Differences between handheld and robot gripper mechanics also complicate transfer. During remote manipulation, missing contact cues make it difficult to regulate grasp force, especially when objects require different handling.
UMI-style devices decouple demonstration collection from robot access, but many use parallel-gripper mechanics and provide no live feasibility feedback. ActiveUMI already adds AR visualization; tactile interfaces such as UMI-FT capture finger forces, while MagiClaw and OmniUMI offer haptics. RUMI combines these capabilities around an existing commercial gripper rather than introducing each capability individually.
RUMI reproduces Robotiq 2F-85 nonlinear finger motion and mounting geometry through a scissor linkage, with interchangeable CoinFT force/torque or DIGIT tactile fingers. CoinFT measures six-axis loads at 200 Hz. A RealSense D405 supplies wrist images, and a Meta Quest 3 controller tracks pose. The extended IRIS application uses QR registration and inverse kinematics to display a MuJoCo robot shadow. At remote contact, a servo locks the linkage; subsequent squeezing becomes a measured grasp-force command. ACT policies use force-adjusted impedance targets for CoinFT tasks or tactile images from contrastively pretrained, frozen ResNet-18 encoders for DIGIT tasks. The authors provide a code and assets link.
The robot-free study involved 5 participants, each collecting 10 episodes per condition, 100 total. AR plus initialization versus no AR yielded joint-limit violations in 3/50 vs 16/50 episodes and velocity violations in 11/50 vs 29/50; manipulability was 0.055 ± 0.022 vs 0.046 ± 0.0057. On a real Franka FR3 with Robotiq 2F-85, ACT successes with sensors versus without were gear assembly 20/20 vs 7/20, wiping 16/20 vs 7/20, screw tightening 9/20 vs 3/20 and box sorting 17/20 vs 9/20. Training used around 100 demonstrations per task, with 20 evaluations per variant. MuJoCo packing used around 100 demonstrations and 100 rollouts per object per feedback condition, with force amplification set to 1.5. With haptics, bulb/transformer success was 94%/97%; mean demonstration forces were 5.66/22.82 N and rollout forces 5.42/19.86 N. Without haptics, demonstration forces were 18.46/15.83 N; every bulb rollout exceeded the 10 N simulated fracture threshold.
The authors identify the wrist-only camera view and missing external-force feedback as limitations. Mechanical blocking signals contact rather than reproducing an external six-axis wrench. Demonstration jitter contributed to slipping during screw tightening. Evaluation covers one robot platform, 5 study participants and 20 real-robot trials per policy variant. Quantitative evidence for haptic force control comes from simulation; the paper describes real-robot teleoperation support but does not establish equivalent real-world performance. The AR comparison combines visualization and initialization, so their separate contributions are unresolved.
Industrial automation integrators, gripper suppliers and robot-learning data providers could use the released materials for research pilots now, particularly around Robotiq 2F-85 installations. A plausible integration horizon is 1-3 years if broader real-robot trials establish repeatable force control, stable contact demonstrations and reliable workspace registration. This is an industry assessment; the paper provides no commercial deployment timetable.
Full paper
RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback
Published under CC BY 4.0. Reproduced with attribution; original at arXiv:2611.15802 (PDF).
Abstract
Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator’s arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers. We present RUMI, a handheld universal manipulation interface matched to the Robotiq 2F-85 with exchangeable fingertip sensors that preserves the target gripper geometry while enabling natural input before contact and force measurement after contact. By integrating the tracking system into a virtual reality headset, our method provides real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform. We further introduce a haptic feedback mechanism that supports force-aware teleoperation for simulated and real robots. A user study shows fewer episodes containing joint-limit violations with live robot visualization and pose initialization (3/50) than without augmented reality (16/50). We additionally train imitation policies from demonstrations, collected using two different fingertip modalities enabled by the gripper-matched hardware, and assess the haptic interface’s capability for collecting force-aware teleoperation datasets in simulation. The code and assets for RUMI are available at: https://utn-air.github.io/RUMI/.
I Introduction
Learning from demonstrations reduces the effort required to program robots for complex or variable tasks [1, 2]. This is increasingly relevant for large visuomotor policies, whose performance depends on the scale and quality of their demonstration data [3]. Collaborative efforts such as Open X-Embodiment [4] aggregate standardized manipulation data across robot platforms, tasks, and institutions using traditional teleoperation approaches, which are typically time-consuming. This limitation has motivated the creation of handheld data collection devices that enable rapid data collection. Universal Manipulation Interfaces (UMIs) support portable, robot-free data collection and thereby decouple data acquisition from continuous access to the target robot [5, 6, 7]. These devices have also gained popularity in the tactile manipulation community, where collecting large-scale datasets remains challenging due to the inherent sparsity of tactile sensor signals [8, 9, 10].
Despite the scalability and speed-up they offer, the inherent robot-free nature of these approaches means that it becomes difficult to observe the validity of the trajectories which they generate. While an intuition for robot-compatible motion can be learned through practice, the lack of feedback necessitates careful post-processing to filter out any unusable data. Furthermore, most existing methods are designed around custom-built robotic grippers, which makes them unsuitable for applications that require more robust grippers.
Another underexplored potential of UMI devices is that they can serve as a more intuitive interface for teleoperation, given their similar appearances and kinematics to target grippers. However, most existing UMI designs lack a way to transmit contact information at time of grasp, which makes it difficult to regulate grasp forces in such scenarios.
To address these challenges, we present the Robotiq Universal Manipulation Interface (RUMI), a handheld platform matched to the nonlinear kinematics, finger attachments, and wrist-camera viewpoint of the Robotiq 2F-85. A scissor-like input drives the coupled fingers, while a rotary encoder measures the resulting grasp coordinate. Interchangeable fingers support force/torque and vision-based tactile sensing for contact-rich demonstrations. During simulated and real-robot teleoperation, a contact-triggered blocker provides low-resistance aperture control before contact and converts the subsequent squeeze into a measured force command. By integrating the handheld pose tracking into a virtual reality (VR) headset, we can project a digital shadow of the target robot platform in augmented reality (AR) fashion during data collection, allowing the operator to recognize infeasible poses while recording rather than filtering them only after collection.
In summary, we make the following contributions:
- RUMI, a Robotiq 2F-85-matched handheld interface with nonlinear actuation and interchangeable fingertips;
- An AR pipeline for visualizing and validating robot configurations during robot-free data collection;
- A haptic feedback mechanism which allows for gripper force control during simulated and real teleoperation;
- Releasing the device design and software framework as well as a dataset of demonstrations featuring force/torque and vision-based tactile sensor data.
II Related Work
- Two-finger interfaces without fingertip contact sensors
- No data
- No data
- No data
- No data
- No data
- No data
- UMI [5]
- –
- –
- –
- –
- Visuoinertial SLAM
- WSG-50
- FastUMI [6]
- –
- –
- –
- –
- Realsense T265
- Parallel gripper
- ForceMimic [11]
- Wrist force
- –
- –
- –
- Realsense T265
- Parallel gripper
- ManipForce [12]
- Wrist force
- –
- –
- –
- ArUco tracking
- Parallel gripper
- ActiveUMI [13]
- –
- –
- \checkmark AR Projection
- –
- Quest VR + Backpack
- Custom bimanual system
- UMIGen [14]
- –
- –
- –
- –
- Realsense T265
- Parallel gripper
- XRZero-G0 [15]
- –
- –
- –
- –
- PICO VR + backpack
- Pinch gripper
- UMI-3D [16]
- –
- –
- –
- –
- Onboard LiDAR
- Parallel gripper
- YUBI [17]
- –
- –
- –
- –
- Static Quest VR
- Pinch gripper
- EgoGuide [18]
- –
- –
- –
- –
- Quest VR
- Flexiv Grav
- HiFi-UMI [19]
- –
- –
- –
- –
- Offl. SLAM + ArUco
- Pinch gripper
- Two-finger interfaces with fingertip contact sensors
- No data
- No data
- No data
- No data
- No data
- No data
- ViTaMIn [20]
- Optical tactile
- –
- –
- –
- Visuoinertial SLAM
- PGI-140-80-W-S
- FreeTacMan [10]
- Optical tactile
- –
- –
- –
- NOKOV
- Parallel gripper
- Touch in the Wild [21]
- Resistive tactile
- –
- –
- –
- Visuoinertial SLAM
- Parallel gripper
- exUMI [22]
- Modular tactile
- –
- –
- –
- Quest
- Grav / Robotiq 2F
- MagiClaw [23]
- Optical tactile
- \checkmark Estimated
- –
- \checkmark Motors
- iPhone SLAM
- Shared MagiClaw
- FARM [9]
- Optical tactile
- \checkmark Estimated
- –
- –
- OptiTrack
- Actuated UMI
- ViTaMIn-B [24]
- Optical tactile
- –
- –
- –
- Quest
- Parallel gripper
- UMI-FT [8]
- Finger force
- \checkmark Direct
- –
- –
- iPhone SLAM
- WSG-50
- TacUMI [25]
- Tactile + wrist
- –
- –
- –
- Vive tracking stations
- Parallel gripper
- OmniUMI [26]
- Tactile + wrist
- \checkmark Estimated
- –
- \checkmark Motors
- Vive tracking stations
- Parallel gripper
- Multi-finger grippers and dexterous hands
- No data
- No data
- No data
- No data
- No data
- No data
- DexUMI [27]
- Tactile arrays
- \checkmark Distributed
- –
- –
- iPhone SLAM
- Custom hand
- MV-UMI [28]
- –
- –
- –
- –
- GoPro + overhead
- Custom hand
- RealDexUMI [29]
- Resistive tactile
- –
- –
- –
- Pose tracker
- Custom hand
- Koala Gripper [30]
- –
- –
- –
- –
- Pose camera
- Custom gripper
- RUMI (ours)
- Force / optical tac.
- \checkmark Direct
- \checkmark Robot
- \checkmark Blocker
- Quest
- Robotiq 2F-85 [31]
II-A Robot-Free Demonstration and Embodiment Transfer
We provide a comprehensive overview of representative robot-free data collection approaches in Tab. I. Existing UMI-style systems use portable handheld grippers with gripper-centric observations and pose tracking to collect demonstrations without an active target robot [5, 6, 7, 17, 28]. The original UMI design already supports transfer across parallel grippers through replaceable finger geometries, while later systems explore closer correspondence to specific robot embodiments, such as exUMI [22]. ActiveUMI [13] further incorporates AR-based robot visualization during robot-free collection.
RUMI follows these directions by matching the nonlinear finger kinematics and mounting geometry of the Robotiq 2F-85, enabling straightforward deployment on existing robot platforms without requiring a custom robot-side gripper or additional compatibility solution. It also uses live robot projection during collection for online kinematic checking.
II-B Multimodal Handheld Devices and Haptic Teleoperation
Several UMI systems extend pose and gripper-width measurements with tactile or force sensing for contact-rich manipulation [8, 9, 24, 25]. RUMI similarly supports interchangeable fingertip sensors through a compatible finger mount, allowing the sensing configuration to be adapted without changing the underlying gripper embodiment.
Haptic feedback and teleoperation have also been incorporated into UMI-style devices, including MagiClaw [23] and OmniUMI [26]. RUMI follows this direction with a selectively engaged gripper-haptic mechanism, allowing the same handheld device to support both in-the-wild collection and teleoperation.
Overall, prior work has explored robot-free collection, transferable gripper designs, tactile and force sensing, live robot visualization, and haptic teleoperation in different combinations. RUMI brings these capabilities together in a single handheld interface with direct correspondence to an existing commercial gripper.
III Method
III-A Design Requirements
RUMI is designed for industrial manipulation settings, where robots equipped with commercial grippers are commonly used for task automation. We derive four main requirements from this setting.
First, the interface should reproduce the geometry and grasp actuation of a commonly used industrial gripper, specifically the Robotiq 2F-85, such that demonstrations can be collected using an embodiment consistent with the target system. Second, the fingers should be modular to support different tactile sensing modalities depending on the task. Third, robot-free demonstrations should remain compatible with the target robot’s workspace and kinematic constraints despite the robot being absent from the physical interaction loop. Finally, data collection should be intuitive and minimally intrusive to the production environment, while the same interface should also support remote teleoperation when direct access to the robot workspace is restricted due to safety or spatial constraints.
III-B Hardware and Sensing
RUMI implements a crossed, scissor-like linkage that couples the motion of the operator’s thumb and index finger to the gripper fingers (Fig. 2). Its linkage geometry and finger attachment interface are derived from the official Robotiq 2F-85 CAD model. The operator actuates two rear input pads, while an AS5600 magnetic rotary encoder measures the resulting gripper aperture. The scissor-like input interface allows direct mapping between operator’s applied force to the gripper-side force, further complemented by a CoinFT [32] sensor on the index finger pad that captures the applied force.
The linkage can additionally be physically locked to provide haptic feedback (Fig. 2 c). A 9g Miuzei micro servo engages a toothed counterpart with the internal coupling gear, preventing further motion of the linkage. We selected this actuator because it is widely available and provides sufficient torque density for the compact mechanism.
The finger attachment points reproduce the 2F-85 mounting geometry, allowing the same finger modules to be transferred between RUMI and the robotic gripper, including the manufacturer’s original fingers. We additionally provide CAD files for fingertip modules compatible with two sensing modalities: the CoinFT sensor, which measures a six-axis wrench at 200 Hz, and the DIGIT [33] vision-based tactile sensor, which provides high-resolution tactile observations.
A Teensy 4.0 microcontroller coordinates the on-device sensing and actuation. It synchronizes measurements from the rotary encoder and any connected CoinFT sensors, transmits these measurements to the host computer via USB, and controls the servo used by the blocking mechanism. DIGIT sensors, when used, bypass the microcontroller and connect directly to the host computer.
A RealSense D405 is mounted at the wrist attachment point in the same pose used on the target robot. A Meta Quest 3 controller is rigidly attached to RUMI to provide six-DoF pose tracking relative to the headset.
III-C Robot-State Visualization
Robot-free data collection removes the target robot from the physical interaction loop and therefore also removes direct feedback about its workspace and kinematic limits. Without such feedback, an operator may generate motions that cannot be reproduced by the target robot.
To expose these constraints during demonstration, we extend the IRIS [34] AR application with support for RUMI. During recording, IRIS renders a MuJoCo-based [35] digital shadow of the target robot. The tracked RUMI pose is transformed into the robot base frame and used as the desired end-effector pose; an inverse-kinematics solver computes the corresponding robot configuration for visualization. The operator can therefore observe the robot configuration associated with the demonstrated motion in real time.
Providing meaningful feedback requires the virtual robot base to be accurately registered with the physical workspace. We use multiple QR codes to estimate the base pose and refine its orientation by fitting a plane to depth measurements of the deployment surface. This registration establishes the transformation between the headset tracking frame, the physical workspace, and the robot base frame. IRIS additionally supports configurable offsets from the detected base pose, allowing the virtual robot to be placed where its physical counterpart could not be accommodated because of spatial restrictions. An example is shown in Fig. 4a.
The same registration is used to express recorded RUMI trajectories in the robot base frame. Before starting an episode, the operator can additionally use a catch-to-teleop initialization procedure in which RUMI is aligned with the projected robot end-effector. This establishes a consistent initial end-effector pose while allowing the corresponding robot configuration to be inspected before and during the demonstration.
III-D Operating Modes and Contact Handling
RUMI uses the same grasp interface for direct in-the-wild demonstration, simulated teleoperation, and real-robot teleoperation (Fig. 2). The modes differ primarily in how contact is generated and communicated to the operator.
During in-the-wild demonstrations, contact occurs directly between RUMI’s fingers and the manipulated object. After the fingers make contact, the operator can continue squeezing the input pads, and the resulting force is measured by the actuation-side CoinFT sensor.
During teleoperation, contact occurs only at the remote or simulated gripper. Before contact, the blocking mechanism remains disengaged and the encoder measurement directly controls the commanded gripper aperture, allowing low-resistance motion of the RUMI linkage. When the remote gripper detects contact, the blocker engages and prevents further linkage closure. Subsequent squeezing is then measured by the actuation-side CoinFT sensor and mapped to a grasp-force command. An optional force amplification factor reduces the physical force required from the operator. When the operator releases the applied force, the blocker disengages and the interface returns to aperture-control mode.
The source of the contact signal depends on the teleoperation setting. In simulation, contact and force response are obtained from the simulated gripper; during real-robot teleoperation, contact is inferred from the physical gripper state and the measured RUMI force is sent as the gripper force command. The Meta Quest controller can additionally provide vibration feedback indicating remote contact and the approximate magnitude of the commanded load. This vibration complements the binary mechanical blocking cue but is not intended to reproduce an arbitrary external six-axis wrench.
III-E Data Recording and Policy Interface
We use Robot Control Stack [36] as the underlying software architecture for data acquisition and robot communication. At each timestep, the system records the RUMI pose, gripper state, camera observations, and measurements from the selected fingertip sensors when available.
For robot-free demonstrations, RUMI poses are stored in the registered robot base frame established by the AR system. Robot actions can then be derived from the recorded trajectory either by temporally shifting the pose sequence or by computing the relative motion between consecutive poses.
During teleoperation, the RUMI input is instead used online as the commanded action, while the measured robot state is used as the observation. Both acquisition modes therefore share the same episode structure, while differing in whether actions are reconstructed from a robot-free demonstration or issued directly during teleoperation.
The resulting episodes can be converted into different observation-action representations for imitation learning. Observations can include camera streams, robot or RUMI poses, gripper state, blocking-mechanism state, and available fingertip-sensor measurements, while actions contain the desired robot motion and gripper command. For robot-free demonstrations, the inverse-kinematics solution associated with the AR projection can additionally provide joint configurations along the demonstrated trajectory, allowing the recordings to support both Cartesian pose-based and joint-space policy representations.
IV Experimental Evaluation
To validate the effectiveness of RUMI, we conduct a series of experiments, focusing on A) usability, B) transfer learning on an imitation policy, and C) haptic feedback utility. For experiments conducted in subsections A) and B), we set up an office table covered with wooden boards (Fig. 4a) to mimic the inference setup (Fig. 4c). We place one QR code at each table corner, and calibrate the base pose of the robot. Our inference environment consists of a Franka FR3 robot arm equipped with a Robotiq 2F-85 gripper and a RealSense D405 wrist camera. The haptic feedback utility experiment is conducted in a digital twin of the inference setup. The complete setup is shown in Fig. 4c.
IV-A Data quality comparison
To test whether our proposed AR-based digital twin projection provides meaningful assistance in the quality of demonstration data, we conduct a small user study. We recruit 5 participants with limited experience in teleoperating a robotic arm with VR controllers. The participants are asked to perform a simple box pick-and-place task. We collect demonstrations under two conditions. In the first condition, the participants perform the task without any sort of AR projection, and are informed beforehand to start each episode from the same starting pose as close as possible. We name this condition w/o AR. In the second condition, we enable the AR projection and the catch-to-teleop functionality. We call this condition AR. Each participant performs 10 demonstrations per condition, resulting in 100 total episodes.
With the collected data, we calculate 4 metrics: initial pose variance, average Cartesian space velocity of the VR controller, number of episodes containing joint limit and velocity violations, and average Yoshikawa manipulability across the whole dataset. Joint velocity and position limits are taken from FR3’s technical data. The results of the analysis are shown in Fig. 5 and Tab. II. Looking at Tab. II, the AR condition results in fewer number of joint limit- and velocity violations. The average manipulability is 19.6% higher than the w/o AR condition, though the standard deviation is also greater. Fig. 5 shows that the catch-to-teleop feature in the AR condition results in a much tighter cluster of initial poses. Additionally, the average Cartesian space velocity is lower, indicating that the projected robot encourages the participant to slow down until the digital shadow is able to catch up. Given that the participants are unfamiliar with VR-based robot teleoperation, the results suggest that our AR-based robot visualization can assist inexperienced users with collecting high-quality data.
IV-B Imitation Learning from RUMI Demonstrations
The second experiment is to test whether RUMI is capable of collecting a high-quality dataset that can be used for training an imitation policy on a real robot. Furthermore, to test RUMI’s versatility in using different fingertip sensors, we collect datasets for the CoinFT [32] and the DIGIT [33] optical tactile sensor. We design four contact-rich manipulation tasks, two for each sensor type, as shown in Fig. 3. Additionally, we divide each task into 3 broad stages: 1) grasp success, 2) task attempt, and 3) task success, in order to better assess where each policy variant fails. We use the Action Chunking Transformer (ACT) [37] for all trained models.
For the CoinFT sensor, we collected datasets for board wiping and gear insertion tasks (Fig. 3a,b). In the board wiping task, the robot must first pick up a whiteboard eraser, which is fitted into a 3D-printed holder for easier grasp. Then, it must erase the star pattern drawn on the board positioned in front of it. The position and shape of the star pattern is varied across each trial. The gear insertion task involves a small 3D-printed setup consisting of three gears, drawn from the ManipulationNet benchmark [38]. The robot must pick up a gear and insert it into the center position of the gear assembly and ensure that the teeth align for a full insertion. The positions of the main assembly plate and the gear are not randomized.
We evaluate the screw tightening and box sorting tasks for the DIGIT sensor, as the optical tactile sensor’s high-resolution tactile information makes it suitable for sensing both shear force and surface texture (Fig. 3c,d). For the screw tightening task, the robot must tighten a cap already mounted on a 3D-printed screw fixture. The number of rotations required for tightening varies across each demonstration. The box sorting task involves two 3D-printed boxes with different surface texture patterns. The robot must then sort these boxes into their respective plates positioned adjacent to them.
We collect around 100 demonstrations per task with their appropriate fingertip sensors attached to the RUMI in the setup described above. For the CoinFT tasks, we train one model with force-adjusted impedance targets:
z^{\prime}=z+F_{z,\text{CoinFT}}/K,where z is the recorded z-axis translation, K is an empirically chosen value of 1000, corresponding to 10\text{\,}\mathrm{N}\mathrm{/}\mathrm{c}\mathrm{m}, and F_{z,\text{CoinFT}} is the z-axis force measured by the CoinFT sensor. We train another model using the raw observed trajectories, with wrist camera images being available for both. For the DIGIT tasks, we follow the training recipe described in VITaL [39]: we first perform contrastive pretraining on a ResNet-18-based [40] tactile encoder backbone along with a ResNet-18-based vision encoder backbone. This combined backbone is then frozen to train the corresponding ACT policy with tactile images. Again, another variant is trained without the tactile image input using just the observed trajectories and the wrist camera images. This setup allows us to test whether tactile data collected with RUMI contribute meaningfully towards completing the tasks successfully. We evaluate each model variant 20 times. The results are shown in Tab. III.
The results show a clear correlation between task success rates and the availability of their respective fingertip tactile sensors. In the gear assembly task, we observe that the force-adjusted trajectories are able to reliably push the gear into alignment with its neighbors, also aided by the force-adjusted gripper strength that allows for passive compliance. In the no-tactile case, although the gear is often successfully inserted through the stick, the lack of further downward trajectory means that the policy often releases its grasp without pushing the gear into alignment. For the board task, the no-tactile variant is not able to completely wipe the pattern off the whiteboard despite passing over it with the eraser due to the lack of downward pressure.
The majority of failures with the screw tightening DIGIT variant comes from the fact that the policy struggles to maintain a stable contact with the cap for long enough to fully tighten it, resulting in slippage. This is expected, as RUMI operation creates a lot of jitter in the demonstration, which makes the contact stage of the task unstable. Nonetheless, the DIGIT variant outperforms the raw variant, which frequently released its grasp after barely rotating the cap, as it lacks the shear force signal to understand when the task is completed. For the box sorting task, the raw variant overfits to one particular trajectory, as it lacks information about the different boxes’ textures. The DIGIT variant was able to successfully sort the box correctly in all but one valid attempt. These results indicate that RUMI is capable of recording useful demonstrations with different fingertip sensors.
- Joint Limit Vio. \downarrow
- 3/50
- 16/50
- Joint Velocity Vio. \downarrow
- 11/50
- 29/50
- Avg. Manip. \uparrow
- 0.055\pm 0.022
- 0.046\pm 0.0057
IV-C Evaluation in Simulation
We investigate whether RUMI can collect force-aware demonstrations in simulation and whether the contact-triggered blocker helps operators to provide object-dependent grasp-force commands. Using the MuJoCo simulator, the experiment reproduces an industrial packing task of transferring a fragile light bulb [41] and a substantially heavier transformer block into a shipping box. Here, we set the gripper force amplification factor to 1.5 to reduce the force required from the operator. The two objects require distinct handling strategies: the bulb must be grasped gently, whereas the transformer requires a stronger grasp to prevent slip.
We collect around 100 demonstrations in total and use them to train the same imitation policy described in the previous subsection. A rollout is successful when the selected object is transferred into the shipping box. Bulb rollouts additionally fail if the maximum grasp force exceeds 10\text{\,}\mathrm{N}, which serves as a simulated fracture threshold. For the transformer, insufficient grasp force leads to object slippage.
To isolate the effect of haptic contact feedback during data collection, we train a second policy using demonstrations recorded in the same setting with the contact trigger and blocker disabled. In this condition, operators command the gripper through a position controller, where grasp force arises implicitly from the position error. The operator then visually estimates the quality of the grasp and adjusts the gripper width as needed. We compare the mean grasp force in the demonstrations (F{}_{\text{DS}}), the mean force produced by policy rollouts (F{}_{\text{R}}), and the success rate based on 100 rollouts for each object and feedback condition.
As shown in Table IV, demonstrations collected with haptic feedback exhibit a clear object-dependent force strategy: operators apply 5.66\text{\,}\mathrm{N} to the bulb and 22.82\text{\,}\mathrm{N} to the transformer on average. The learned policies preserve this separation during rollout, producing mean forces of 5.42\text{\,}\mathrm{N} and 19.86\text{\,}\mathrm{N}, respectively, and achieve success rates of 94% and 97%. Without haptic feedback, the demonstration forces converge toward less task-appropriate values: bulb force rises to 18.46\text{\,}\mathrm{N}, causing every rollout to exceed the fracture threshold, while transformer force falls to 15.83\text{\,}\mathrm{N}. These results indicate that the blocker provides a useful contact reference for demonstrating force-sensitive behavior and that the learned policy reproduces the resulting object-dependent grasp strategy.
- Gear Assembly (CoinFT)
- 20/20
- 20/20
- 20/20
- Gear Assembly (None)
- 20/20
- 20/20
- 7/20
- Whiteboard Wiping (CoinFT)
- 20/20
- 18/20
- 16/20
- Whiteboard Wiping (None)
- 20/20
- 17/20
- 7/20
- Screw Tightening (DIGIT)
- 17/20
- 17/20
- 9/20
- Screw Tightening (None)
- 17/20
- 17/20
- 3/20
- Box Sorting (DIGIT)
- 18/20
- 18/20
- 17/20
- Box Sorting (None)
- 19/20
- 19/20
- 9/20
V Conclusions
In this work, we presented RUMI, a gripper-matched handheld interface for tactile-enabled data collection, modeled after the Robotiq 2F-85 gripper. The device combines gripper-pose tracking with compatibility to diverse fingertip sensors and user-side grasp-force measurement and provides haptic feedback during teleoperation. We demonstrated its usability for robot-free data collection, imitation policy training, and simulated teleoperation. By providing a consistent physical interface across these modes, RUMI enables the collection of embodiment-matched, tactile-aware demonstrations for robot learning. Our method’s current limitations include being restricted to only a wrist camera viewpoint and the lack of external force feedback to complement the blocking mechanism. Future work could include generative AI to overcome the single-view limitation as well as more informative visual cues about external forces acting upon the simulated or teleoperated robot.
Acknowledgment
- Bulb
- 5.66
- 5.42
- 94%
- Block
- 22.82
- 19.86
- 97%
- Bulb (no haptic)
- 18.46
- –
- 0%
- Block (no haptic)
- 15.83
- 13.67
- 99%
All work related to the CoinFT sensors was performed by and at UTN. Siemens’ contribution was limited to the mechanical design of the device. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the BayernKI project v106be. BayernKI funding is provided by Bavarian state authorities. This work has been partially supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG). This work has been partially supported by the project GeniusRobot funded by the German Federal Ministry of Education and Research (BMBF grant no. 01IS24083). We further thank Volker Schneider and Devadas Sheela for their assistance with hardware development and setup. OpenAI ChatGPT and Claude Code were used to assist with debugging and the generation of limited boilerplate and data-processing code. All generated code was reviewed, adapted, and validated by the authors. The research methodology, algorithm design, analysis, and interpretation of results were performed by the authors. ChatGPT was further used to remove background elements from Figure 4 a.