LocoPilot: 1명이 팔 달린 유니트리 Go2를 조종하며 1.27N 힘까지 느낀다 (IIT)

Abstract

Learning loco-manipulation policies requires demonstrations that capture coordinated base and arm motion on the target embodiment. Robot-free interfaces allow demonstrations to be collected independently of the target robot, increasing flexibility but without directly capturing platform-specific commands, proprioception, and body-mounted observations. On-robot teleoperation allows to capture these signals directly, but requires an interface capable of commanding the complete system concurrently. This paper presents LocoPilot, a single-operator bilateral teleoperation and data-collection framework implemented on a 12-DoF Unitree Go2 quadruped equipped with a 6-DoF AgileX Piper L manipulator, an Intel RealSense D435 wrist camera, and an Intel RealSense D405 head camera. A second 6-DoF Piper L arm, used as the leader and two handheld controllers provide simultaneous control of the base, body posture, manipulator, and gripper. The follower arm interaction forces are estimated without a dedicated force/torque sensor and rendered to the operator through a passivity-constrained haptic channel. A ROS 2 pipeline records synchronized multimodal observations together with a unified 11D action shared by teleoperation and policy deployment. Compared to prior work, LocoPilot uniquely unifies robot-in-the-loop legged data collection, force feedback without motion-capture instrumentation, and the same action interface at deployment. We evaluate the interface across multiple loco-manipulation tasks and validate the complete collection-to-deployment pipeline by fine-tuning MolmoAct2 [1] on pick-and-place demonstrations and deploying on the physical robot without action retargeting or additional inverse kinematics.

I Introduction

Robots capable of both locomotion and manipulation extend physical interaction beyond the workspace of fixed-base manipulators. Arm-equipped quadruped robots are particularly suited to unstructured environments, where they combine legged mobility with physical interaction capabilities. Prior work has explored legged loco-manipulation through model-based whole-body control [2], or learned coordination of locomotion and manipulation [3, 4]. More recently, generalist Visual Language Action (VLA) models have gained increasing attention for learning manipulation behaviors from multimodal demonstrations, leveraging large-scale visual and language representations [5, 6, 7, 8]. For loco-manipulation, however, such demonstrations must capture both base and arm actions on the target embodiment. Ideally, the collection interface should therefore minimize the collection–deployment gap while preserving direct operator authority over all task-relevant commands.

Existing systems trade off these requirements in different ways:

Teleoperation systems place a human directly in the control loop of the physical robot, combining the operator task understanding with the constraints of the target embodiment. Leader-follower interfaces with direct joint correspondence provide intuitive control of fixed-base manipulators [9, 10], while mounting the same kind of interface on a wheeled base extends the concept to mobile manipulation, recording base motion alongside bimanual manipulation [11]. XR-based systems provide more general cross-embodiment interfaces [12, 13], and DROID demonstrates the use of teleoperation for standardized large-scale data collection [14]. Immersive interfaces additionally stream the robot’s own onboard imagery to the operator [12], allowing real-time inspection of the visual inputs that will directly condition the downstream policy. For arm-equipped quadrupeds, previous systems have used wearable motion capture, joysticks, haptic devices, or hand gestures [15, 16, 17, 18]. Collecting directly on the target robot reduces observation and action mismatch, but simultaneously commanding all task-relevant DoFs becomes increasingly difficult for a single operator.

Robot-free interfaces make the opposite trade-off. The Universal Manipulation Interface (UMI) [19] records demonstrations using a sensorized handheld gripper, allowing the operator to demonstrate manipulation without the kinematic constraints of the target robot. This improves collection flexibility, but omits embodiment-specific signals such as base commands, proprioception, and body-mounted observations. Deployment therefore requires an additional mapping from demonstrated end-effector motion to the target embodiment, as in UMI on Legs [20].

This work aims to combine the collection-deployment consistency of on-robot teleoperation with the flexibility of learning-based interfaces for arm-equipped quadrupeds. As illustrated in Fig. , both the human operator and the learned policy act through the same loco-manipulation interface, while the resulting demonstrations are collected directly on the target platform. Achieving this requires addressing two main challenges.

The first challenge is enabling a single operator to command the entire loco-manipulation action concurrently. Reaching and grasping may require the robot to translate, rotate, or change body posture while the arm is moving, so sequential base and manipulator control produces segmented demonstrations rather than the smooth continuous coordination desired from the final policy. Existing systems address this through multiple operators, wearable instrumentation, or partial autonomy, introducing additional coordination or removing task decisions from the demonstrator.

To retain these decisions under single-operator control, we propose a bimanual interface design where a handheld controller is mounted on the end-effector handle of the leader arm. The dominant hand articulates the manipulator while operating the mounted controller, and the non-dominant hand provides the remaining base commands through a second controller. Together, the two interfaces provide concurrent control of planar base velocity, yaw rate, body pitch, arm configuration, and gripper aperture. Interaction forces at the follower are additionally estimated online and rendered through the leader arm, providing contact feedback without a dedicated force/torque sensor.

The second challenge is maintaining consistency between collection and deployment. On a legged manipulator, high-level commands are executed through a locomotion and manipulation stack whose behavior also shapes the recorded motion. Replacing this stack at deployment can therefore shift the distribution of observations and executed motions away from the demonstrated one, even when the high-level action representation is unchanged. We avoid this by using the same 11-dimensional loco-manipulation action in both regimes and by keeping the same downstream execution modules: the locomotion policy and leg controller, the arm controller, and the gripper interface. During collection, the action is generated by the operator, whereas at deployment it is generated by the learned policy; the locomotion policy, arm controller, and gripper interface remain unchanged. Table I compares LocoPilot with the closest prior systems in terms of the teleoperation and data-collection properties discussed above. The contributions of this work can be summarized as follows:

• A single-operator bilateral teleoperation interface for arm-equipped quadrupeds, combining concurrent base, body, arm, and gripper control with sensorless force reflection and an operator-side passivity layer.
• A unified 11-dimensional action interface shared between teleoperation and learned-policy deployment, with the same locomotion and manipulation controllers used in both regimes.
• A synchronized learning-oriented data pipeline recording multi-camera RGB-D observations, robot state, odometry, and commanded actions, validated through multiple teleoperation tasks and real-robot deployment of a fine-tuned MolmoAct2 policy.

The remainder of this paper is organized as follows: Section II presents the hardware and software architecture. Section III the teleoperation mapping and unified command. Section IV reports the teleoperation experiments and validates the collected demonstrations through real-robot deployment. Section VI draws the final conclusions.

II System Overview

In this section we provide an overview of the hardware and software components that compose our framework.

II-A Hardware

Figure shows both the teleoperation setup and the system being controlled. In particular, the follower platform, shown on the right, consists of a 12-DoF Unitree Go2 quadruped equipped with a 6-DoF AgileX Piper L manipulator and parallel gripper. Two RGB-D cameras provide complementary observations: an Intel RealSense D435 mounted near the gripper captures close-range manipulation, while an Intel RealSense D405 mounted on the quadruped head, thanks to its larger field of view, observes the wider workspace in front of the robot. This platform is the same during both demonstration collection and autonomous deployment.

The leader station, on the left of Fig. 2, is used only during data collection. It is built around a second Piper L arm serving as a leader device. A custom handle at the end-effector improves ergonomics and holds one of two Nintendo Joy-Con controllers [21]. The operator grips this handle with the dominant hand and articulates the leader arm. With the thumb of the same hand, the operator reaches the controller’s analog stick and buttons, which respectively control the quadruped robot pitch and yaw rate and the episode markers, while the index finger controls the gripper through the trigger button. With the non-dominant hand the operator holds the second controller, which commands the base velocity. The configuration shown in Fig. 2 assumes a right-handed operator: for left-handed operation, a specular handle can be employed while swapping the controller assignment without changing the teleoperation scheme.

To ensure visual consistency, the operator drives the robot using the exact onboard camera streams recorded for downstream policy training, displayed in real time at the leader station. This visual alignment ensures the demonstrator and the policy share the exact same perspective.At the same time, placing all commands under a single operator makes it easy to execute base and arm movements concurrently instead of sequentially; for instance, the operator can steer and turn the quadruped while reaching toward an out-of-reach object in a single coordinated motion.

II-B Software Architecture

On the software side, the framework is composed of interconnected, distributed modules running on the quadruped’s onboard computer and an operator workstation, as shown in Fig. 3. In particular, the quadruped’s onboard computer runs: the reinforcement-learning base policy of [22], which converts base-velocity and body-pitch references into joint position targets for a low-level PD controller; the MUSE state estimator [23], which provides the state estimates used as input by the policy; the arm control stack; and the camera nodes, which publish the two RGB-D streams.

As a direct consequence of unifying the action and observation interfaces, the same software modules run on the robot both for data collection and for autonomous deployment. What changes between the two modes is the operator side. During demonstration collection the workstation runs: the leader arm control stack; the teleoperation module described in Sec. III-A that couples leader and follower and renders contact back to the operator; and the recording module that assembles episodes from the observation, state, and command streams. During autonomous deployment a single policy-inference node replaces all three, receiving the same observations and emitting the same commands the teleoperation loop previously produced. The switch between the operator and the policy therefore falls at one interface, and everything after it is unaffected by which of the two is in control.

Having defined explicit software interfaces that decouple the teleoperation and autonomous deployment stack from the underlying robot hardware, each hardware-dependent subsystem can be replaced independently, provided that it preserves the corresponding input-output contract. In particular, the locomotion subsystem, including the base, leg controller, reinforcement-learning policy, and state estimator, can be replaced by any implementation that accepts commanded planar velocity and absolute body pitch and exposes the leg states and base-velocity estimate required by the higher-level stack. Likewise, the manipulation subsystem can be replaced by any arm and controller implementation that supports joint-space impedance control and exposes joint position, velocity, and torque feedback. All components above these interfaces, including the teleoperation logic, action representation, and data-collection pipeline, remain unchanged.

This interface-based organization therefore reduces the engineering effort required to port the framework to a different platform and makes the overall architecture modular at both the software and hardware levels. The present implementation uses ROS 2 Jazzy with Cyclone DDS. Section III details the central elements of the software stack, i.e., the teleoperation and data collection modules.

III Teleoperation and Data-Collection Framework

III-A Leader-Follower Arm Mapping

A haptic bilateral teleoperation scheme is used to map the leader arm motion to the follower while reflecting interaction forces back to the operator, as shown in Fig. 2. Because leader and follower share the same kinematic structure, the coupling between them can be imposed directly in joint space: the follower directly tracks the leader configuration, and the latter renders the joint-space interaction torques experienced by the follower. We model both manipulators with the standard joint-space rigid-body dynamics

\left\{\begin{aligned} \mathbf{M}_{l}\ddot{\mathbf{q}}_{l}+\mathbf{C}_{l}\dot{\mathbf{q}}_{l}+\mathbf{g}_{l}+\mathbf{f}_{l}&=\bm{\tau}_{l}+\bm{\tau}_{h},\\
\mathbf{M}_{f}\ddot{\mathbf{q}}_{f}+\mathbf{C}_{f}\dot{\mathbf{q}}_{f}+\mathbf{g}_{f}+\mathbf{f}_{f}&=\bm{\tau}_{f}+\bm{\tau}_{\mathrm{ext}}.\end{aligned}\right.

where subscripts l and f denote leader and follower quantities, respectively. In the usual notation, \mathbf{q}_{*} is the joint configuration, \mathbf{M}_{*} the inertia matrix, \mathbf{C}_{*} the Coriolis/centrifugal matrix, and \mathbf{g}_{*} the gravity torque. The term \mathbf{f}_{*} denotes the friction, while \bm{\tau}_{*} is the actuator torque. Finally, \bm{\tau}_{h} and \bm{\tau}_{\mathrm{ext}} are the human operator and environment interaction torques. The two models are identified separately, starting from their nominal values, since the leader carries the operator handle while the follower carries the gripper and the wrist camera. As usual, \mathbf{M}_{*} is symmetric positive definite and \dot{\mathbf{M}}_{*}-2\mathbf{C}_{*} is skew-symmetric. Throughout, the dependence on time is omitted where unambiguous, and made explicit only for quantities evaluated at a specific instant.

III-A1 Position channel

The follower tracks the leader through a local PD law with gravity compensation,

\bm{\tau}_{f}^{d}=-\mathbf{K}_{p}\left(\mathbf{q}_{f}-\mathbf{q}_{l}\right)-\mathbf{K}_{d}\left(\dot{\mathbf{q}}_{f}-\dot{\mathbf{q}}_{l}\right)+\mathbf{g}_{f}(\mathbf{q}_{f}),

with \mathbf{K}_{p},\mathbf{K}_{d}\in\mathbb{R}^{6\times 6} diagonal and positive definite. The superscript d denotes a desired quantity; we assume the drives realize \bm{\tau}_{f}^{d} as the actuator torque \bm{\tau}_{f} of (1) without saturation.

III-A2 Force channel

The leader receives an estimate of the external joint torque induced by the interaction between the follower and the environment. The estimate is obtained using a generalized-momentum residual filter [24] on the identified follower model, which does not require dedicated force/torque sensors. Let \mathbf{p}=\mathbf{M}_{f}(\mathbf{q}_{f})\dot{\mathbf{q}}_{f} be the generalized momentum of the follower, \mathbf{p}(0) its value at the instant t=0 at which the observer is initialized, and \bm{\beta}=\mathbf{g}_{f}+\mathbf{f}_{f}-\mathbf{C}_{f}^{\top}\dot{\mathbf{q}}_{f} the sum of its gravity, friction and Coriolis terms. Differentiating \mathbf{p} and using the skew-symmetry of \dot{\mathbf{M}}_{f}-2\mathbf{C}_{f} gives \dot{\mathbf{p}}=\bm{\tau}_{f}-\bm{\beta}+\bm{\tau}_{\mathrm{ext}}, so the momentum change \mathbf{p}-\mathbf{p}(0) accumulated since initialization differs from the change predicted by the known torques, the integral term below, exactly by the contribution of \bm{\tau}_{\mathrm{ext}}. The residual evaluates this mismatch and feeds it back through \mathbf{K}_{I},

\mathbf{r}=\mathbf{K}_{I}\!\left[\mathbf{p}-\mathbf{p}(0)-\int_{0}^{t}\!\left(\bm{\tau}_{f}^{m}-\bm{\beta}+\mathbf{r}\right)\mathrm{d}s\right],

where the actuator torque is replaced by the current-derived joint-effort feedback \bm{\tau}_{f}^{m}\approx\bm{\tau}_{f} reported by the follower drives. The observer is initialized at teleoperation engagement, when the follower is at rest and \mathbf{p}(0)=\mathbf{0}. Under the rigid-body dynamics of (1), the residual obeys \dot{\mathbf{r}}=\mathbf{K}_{I}(\bm{\tau}_{\mathrm{ext}}-\mathbf{r}), so \mathbf{r} estimates the external joint torque to first order with bandwidth \mathbf{K}_{I} and without requiring joint acceleration. To suppress high-frequency residual components caused by measurement noise and unmodeled dynamics, we apply a low-pass filter on the residual at f_{c}=5 Hz; in the equations below, \mathbf{r} denotes the filtered residual.

On the leader side, gravity compensation and light damping are combined with the reflected estimate,

\bm{\tau}_{l}=\mathbf{g}_{l}(\mathbf{q}_{l})-\mathbf{D}_{l}\dot{\mathbf{q}}_{l}+\alpha\,\bm{\Lambda}\,\mathbf{r},

where \bm{\Lambda} is a diagonal reflection gain, \mathbf{D}_{l}\succ 0 a diagonal damping matrix, and \alpha\in[0,1] the modulation supplied by the passivity layer (described below). No position term appears in (4): the leader is free to move under damping alone, and the operator feels the follower only when the latter interacts with the environment.

The force channel is therefore designed to convey informative interaction cues rather than reproduce the environment impedance exactly. Wildenbeest et al. [25] compared teleoperated assembly under haptic feedback of increasing fidelity and found that most of the improvement in task time and control effort was already achieved by the lowest-fidelity condition, which conveyed only the low-frequency component of the interaction, while higher-fidelity conditions added little further benefit. Filtering the residual at f_{c} retains that component. In learning from demonstration, haptic feedback during teleoperated data collection has also been shown to improve demonstration throughput and the performance of the resulting autonomous policies [26].

III-A3 Passivity layer

Position–force coupling does not, in general, make the teleoperator passive by construction [27]. Classic bilateral frameworks constrain both sides of the chain [28]; we instead passivize only the operator-facing port, a single-port approach with precedent in Time-Domain Passivity Control [29], which leaves the follower command unmodified and therefore identical to the one used at deployment. Following [30], the active reflection \bm{\tau}_{r}^{d}=\bm{\Lambda}\mathbf{r} is budgeted by an energy tank H, filled by the power dissipated by the leader damping and drained by the power the reflection injects,

\dot{H}=\dot{\mathbf{q}}_{l}^{\top}\mathbf{D}_{l}\dot{\mathbf{q}}_{l}-\alpha\,\bm{\tau}_{r}^{d\top}\dot{\mathbf{q}}_{l},\qquad 0<H(0)=H_{0}\leq H_{\max},

with H upper-bounded by H_{\max} so that no unbounded reservoir accumulates, and initialized at the same instant t=0 as the residual. Writing P_{k}^{d}=\bm{\tau}_{r,k}^{d\top}\dot{\mathbf{q}}_{l,k} for the power the reflection would inject at control sample k, the modulation is

\alpha_{k}=\begin{cases}1,&P_{k}^{d}\leq 0,\\[4.0pt]
\displaystyle\min\!\left(1,\max\!\left(0,\frac{H_{k}}{T_{\delta}\,P_{k}^{d}}\right)\right),&P_{k}^{d}>0,\end{cases}

with T_{\delta} the control period. Reflection passes unchanged whenever it absorbs energy from the operator, and is scaled down when it would inject more than the tank holds over the next interval. Taking the storage S=\tfrac{1}{2}\dot{\mathbf{q}}_{l}^{\top}\mathbf{M}_{l}\dot{\mathbf{q}}_{l}+H, and using the skew-symmetry of \dot{\mathbf{M}}_{l}-2\mathbf{C}_{l} together with Eqs. (4) and (5), the damping and reflected-power terms cancel and \dot{S}=\bm{\tau}_{h}^{\top}\dot{\mathbf{q}}_{l}-\dot{\mathbf{q}}_{l}^{\top}\mathbf{f}_{l}\leq\bm{\tau}_{h}^{\top}\dot{\mathbf{q}}_{l}, since the leader joint friction is dissipative. The same bound holds when H_{\max} is reached and the excess energy is discarded. As S is non-negative, the haptic subsystem is passive at the operator port (\bm{\tau}_{h},\dot{\mathbf{q}}_{l}), and the guarantee does not extend to the complete operator–leader–follower–environment chain. Since the residual enters only through the injected power, errors in the follower model degrade the rendered cue but not the energy accounting.

III-A4 Engagement and safety

Teleoperation is engaged after an automatic procedure which aligns the follower and the leader configurations, preventing a position-error transient at coupling. The tank is initialized to H_{0} at engagement, providing an initial energy budget for active reflection. The operator can suspend coupling at any time, and loss of the leader state stream triggers the same transition, causing the follower to hold its current configuration. Commanded positions and torques are clamped to the hardware limits.

III-B Unified Loco-Manipulation and Learning Interface

In both regimes of Section II-B, the robot is commanded through the same 11-dimensional loco-manipulation action, written by the operator during collection and by the policy at deployment,

\mathbf{a}=\begin{bmatrix}v_{x}^{d}&v_{y}^{d}&\omega_{z}^{d}&\theta^{d}&\mathbf{q}_{\mathrm{arm}}^{d\top}&g^{d}\end{bmatrix}^{\top},

where v_{x}^{d} and v_{y}^{d} are the desired linear velocities of the base in the horizontal plane, expressed in a gravity-aligned frame that follows the base yaw, \omega_{z}^{d} the desired yaw rate, \theta^{d} the desired body pitch, \mathbf{q}_{\mathrm{arm}}^{d}\in\mathbb{R}^{6} the follower-arm joint references, and g^{d} the gripper aperture. The command is divided among three low-level interfaces: the base velocities and pitch references are sent to the locomotion policy, the arm references to the manipulator controller, and the gripper reference to the gripper interface. During deployment, the joint-position references produced by the policy are temporally interpolated by the arm trajectory interface, which samples the resulting trajectory to provide both position and velocity references to the low-level joint controller of (2). During demonstration collection, these references are provided directly by the leader position and velocity, while the remaining action entries come from the two handheld controllers.

III-B1 Locomotion policy

The base velocities and pitch reference are tracked by a reinforcement-learning policy, adapted from BinWalker [22], that outputs the 12 desired leg joint positions passed to the quadruped low-level controller. Since the arm changes the mass distribution of the platform and transmits a wrench to its base, the measured arm configuration is part of the proprioceptive observation,

\mathbf{o}=\left(\mathbf{v},\;\bm{\omega},\;\mathbf{g}_{b},\;\mathbf{q}_{\mathrm{leg}},\;\dot{\mathbf{q}}_{\mathrm{leg}},\;\mathbf{q}_{\mathrm{arm}},\;v_{x}^{d},\,v_{y}^{d},\,\omega_{z}^{d},\,\theta^{d}\right),

alongside the estimated base linear and angular velocities \mathbf{v} and \bm{\omega}, the gravity direction \mathbf{g}_{b}=\mathbf{R}_{b}^{\top}\hat{\mathbf{g}} expressed in the base frame, where \mathbf{R}_{b} maps base-frame vectors to the world frame, the leg joint positions and velocities, and the four commanded entries of (7) that the policy is asked to track. The term \mathbf{g}_{b} encodes the base roll and pitch and not the yaw, which matches the command convention of (7): the pitch reference \theta^{d} is absolute, whereas the yaw is commanded as a rate. The five most recent observations are stacked and given to the policy at each step. On hardware \mathbf{v}, \bm{\omega}, and the base orientation \mathbf{R}_{b} are supplied by the MUSE state estimator [23], in both modes. We also extend the arm-configuration randomization of BinWalker’s policy at training time to cover the workspace required by this platform. Specifically, this accounts for the forward-extended configurations that occur when reaching for and transporting objects.

The policy therefore adapts the leg motion to the current arm configuration, while never commanding the arm itself. Locomotion and manipulation remain modular, and the policy accounts for their physical coupling through the observation of (8).

III-B2 Policy learning interface

The recording module is designed so that demonstrations can be used for policy learning without redefining the robot interface. At each step, it pairs the synchronized camera images and proprioceptive state with the command issued by the operator at that instant. Complete episodes are exported to the LeRobot dataset format [31] and can be used for training without further conversion.

Two choices keep the exported data model-agnostic while preserving the collection–deployment interface. First, the recorded action is the command received by the low-level controllers, rather than the robot response measured afterwards. A policy trained on these labels therefore emits quantities that the same controllers already accept at deployment. Second, each episode includes colour and depth images from both cameras, the proprioceptive state, the action, and custom language instructions, allowing a candidate model to condition on any subset of these observations. The only requirement imposed by the framework is that the model transmits the loco-manipulation action of (7), or a subset of it. Its output rate is not constrained to match the low-level controllers, since the arm trajectory interface interpolates the joint references to the controller rate, and the remaining entries are held between updates.

IV Experimental Evaluation

IV-A Arm-to-arm Teleoperation

We evaluate the leader–follower teleoperation interface of Sec. III-A on a tabletop pick-and-place task containing both free-space motion and sustained object interaction. The experiment verifies that the two channels required for demonstration collection operate simultaneously: the follower tracks the leader motion, while contact with the environment produces an interaction estimate suitable for haptic rendering. To obtain a ground-truth reference for the force channel, the manipulated object is a known payload of 1.5\text{\,}\mathrm{kg}, so that during transport the expected vertical component of the estimated interaction force is known.

The position channel is evaluated through the Cartesian tracking error \mathbf{e} between the leader and follower end-effector positions, while the force channel is characterized by the norm of the estimated interaction force \mathbf{F}. Figure 4 reports both over a representative episode, in which the robot first moves in free space, then grasps and transports the payload, and finally releases it. The weight of the payload is drawn as a reference line, since it is the force the estimate should report while the object is held.

We compare the estimate against the value expected in each phase: zero while the robot moves in free space, and the weight of the payload while it is carried. Grasping and releasing take a short time, during which the load builds up and vanishes, so the second (1\text{\,}\mathrm{s}) before and after each of the two transitions is excluded from the comparison.

In free motion the estimated force has an RMSE of 1.27\text{\,}\mathrm{N}, which is the noise floor of the estimate and therefore the smallest interaction the operator can perceive. While the payload is held, the RMSE against its known weight is 2.25\text{\,}\mathrm{N}, or 15.3\text{\,}\mathrm{\%} of the reference value. Tracking error follows the same structure: 8.02\text{\,}\mathrm{mm} in free motion against 60.14\text{\,}\mathrm{mm} during interaction, as expected when environmental constraints oppose the configuration commanded by the leader. Larger tracking errors occur in the same interval with a mean of 24.55\text{\,}\mathrm{mm}, as expected when environmental constraints oppose the configuration commanded by the leader. The two signals remain complementary, with the position error quantifying the deviation between commanded and executed motion and the force estimate exposing the interaction responsible for it.

IV-B VLA Deployment

We evaluate the collected demonstrations by fine-tuning a vision-language-action model and deploying it on the robot. The purpose of the experiment is to verify that the recorded episodes are directly usable for policy learning, and that the coordinated base and arm motion they contain is reproduced by the learned policy. The episodes exported by the recorder are used to train the policy, which then drives the robot through the interface the operator employed, with no intermediate processing stage between collection and deployment. We instantiate the command interface of Sec. III-B with MolmoAct2 [1], a vision-language-action model that predicts action chunks from camera observations and a language instruction. The choice is motivated by openness: its authors release the training data, the training recipe, and the pretrained checkpoints, so the fine-tuning procedure reported here contains no proprietary component and can be repeated in full. In its original paper [1] the model is demonstrated only on fixed-base manipulators, so fine-tuning it here also accounts for instructing a new physical embodiment which includes commanding a mobile base.

The dataset collected with the framework covers two tasks, Grasp and Push. Grasp motion requires the operator to drive the quadruped toward a cube, close the gripper on it, and place it on a colored target region. In Push, the operator uses the end-effector to push the cube toward the target without lifting it, starting from a nearby initial position. In both tasks the target usually lies outside the workspace of the arm at the stance in which the episode starts, so the recorded actions contain coordinated base and arm motion rather than manipulation alone. Either a red or a white cube are used for the collection, and textual instructions are generated from templates over the object and target descriptions, which varies the language input across episodes.

The collection comprises 289 demonstrations, 259 of grasping and 30 of pushing, of which 258 are used for fine-tuning and 31 are held out for validation.

Fine-tuning starts from the released MolmoAct2 post-training checkpoint, of 5.5B parameters, and follows the strategies reported by its authors, adapting the vision-language backbone with rank-512 low-rank adapters [32] and training the action head in full. Table II reports the rest of the configuration. We select the best checkpoint based on its open-loop performance on held-out episodes, where predicted action chunks are compared against recorded commands, then deploy that checkpoint on the robot.

Deployment. On the robot the policy replaces the operator without any change below the interface: it receives the two camera streams, the proprioceptive state, and a language instruction, and writes the same command that the teleoperation layer produced during collection, at the same rate. Example instructions include “Move the red cube to the blue square” and “Pick up the red cube and place it in the blue square.” Figure 5 shows a successful rollout, in which the model drives the base toward the cube, pitches the robot’s body to extend the reach of the arm, closes the gripper, and carries the cube to the target region, coordinating base and arm within one continuous motion. The trials are assessed qualitatively, with respect to the ability to solve the task from different starting positions. The policy learns the pick-and-place motion in the simplest scenarios, specifically when both the cube and the target are visible and no complex reorientation is required. Complications arise when the target region leaves the camera’s field of view, since the policy must then reorient and navigate based solely on surrounding visual cues, and it sometimes walks and moves the cube in the wrong direction. Another challenging scenario occurs when the base stops too early or too far from the target region, so that the cube is released slightly away from it. In these situations the smooth transition between motions leads the policy to perform corrections, nudging the cube toward the target or re-approaching it for a second grasp until the task is completed.

V Limitations and Applicability

The residual of (3) is computed from a model of the arm alone. The floating base is treated as inertial, so whenever it accelerates the resulting inertial effects appear at the joints and the observer attributes them to the environment. A second error comes from the identified inertial and friction parameters, whose residual mismatch, dominated by the static friction of the joint drives, biases the estimate as a function of the configuration. Neither contribution can be removed entirely, and together they set a free-motion floor, which is the smallest interaction the operator can perceive. Compensating the base acceleration explicitly, from the base state already estimated on board, would reduce the noise in the force channel, and we leave it to future work.

These errors affect what the operator perceives but not the stability of the interface. The passivity layer of Sec. III-A3 accounts for measured power flow, so an inaccurate residual costs force feedback accuracy rather than safety. The force channel is designed as an extension of the position channel, such that disabling it does not change the recorded action nor the collection pipeline.

We found that the force channel is most useful when the environment constrains the end-effector motion in ways not visible to onboard cameras, and when the resulting interaction is sustained rather than impulsive. Opening a door or rotating a valve constrains the end effector along a path that the operator has to follow, and pushing or pulling a load requires regulating a force over the whole motion. In these cases the operator perceives the constraint directly instead of inferring it from the position error. Tasks dominated by free-space motion, or in which contact is brief and light with respect to the floor reported above, gain correspondingly less from the force channel.

The evaluation covers a single hardware configuration. The autonomous policy is trained on two task families and assessed qualitatively over a limited number of trials, and a quantitative study of the success rate against the dataset size, or of the contribution of contact feedback to the quality of the collected demonstrations, would require a controlled comparison that we leave to future work.

VI CONCLUSIONS

We presented a teleoperation and data-collection framework for an arm-equipped quadruped in which a single operator commands base velocity, body pitch, arm configuration, and gripper through one 11-dimensional action, and demonstrations are recorded in the same action the learned policy later writes. Holding it fixed across the two regimes removes the conversion stage that usually separates a recorded demonstration from a deployable policy: the exported episodes are used for fine-tuning as they are, and the resulting policy is executed by the controllers already in place. We additionally showed a successful fine-tuning of MolmoAct2 on pick-and-place data collected with the framework, obtaining a policy that coordinates base and arm motion on the robot. Beyond that, the framework simplifies the collection of new data and the addition of new tasks through the same recording pipeline, making extensions along this axis straightforward.

References

1. [1] H. Fang, J. Duan, D. Clay, S. Wang, S. Liu, W. Huang, X. Fan, W. Tsai, S. Chen, Y. R. Wang, S. Xing, J. Cho, J. S. Park, A. Eftekhar, P. Sushko, K. Farley, A. Wadhwa, C. Harrison, W. Han, Y. Lee, E. VanderBilt, R. Hendrix, S. Ellawela, L. Ngoo, J. Chai, Z. Ren, A. Farhadi, D. Fox, and R. Krishna (2026) MolmoAct2: action reasoning models for real-world deployment. External Links: 2605.02881, Link
2. [2] C. D. Bellicoso, K. Krämer, M. Stäuble, D. Sako, F. Jenelten, M. Bjelonic, and M. Hutter (2019) ALMA - articulated locomotion and manipulation for a torque-controllable robot. In 2019 International Conference on Robotics and Automation (ICRA), Vol. , pp. 8477–8483. External Links: Document
3. [3] Z. Fu, X. Cheng, and D. Pathak (2022) Deep whole-body control: learning a unified policy for manipulation and locomotion. In Conference on Robot Learning (CoRL),
4. [4] G. Pan, Q. Ben, Z. Yuan, G. Jiang, Y. Ji, S. Li, J. Pang, H. Liu, and H. Xu (2024) RoboDuet: learning a cooperative policy for whole-body legged loco-manipulation. External Links: 2403.17367, Link
5. [5] O. X. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang, G. Wang, H. Su, H. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Vakil, J. Bohg, J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang, J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. ”. Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, M. Z. Irshad, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. ”. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart’in-Mart’in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, V. Guizilini, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin (2023) Open X-Embodiment: robotic learning datasets and RT-X models. Note: https://arxiv.org/abs/2310.08864
6. [6] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn (2024) OpenVLA: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246.
7. [7] Y. Niu, Y. Zhang, M. Yu, C. Lin, C. Li, Y. Wang, Y. Yang, W. Yu, T. Zhang, Z. Li, J. Francis, B. Chen, J. Tan, and D. Zhao (2025) Human2LocoMan: learning versatile quadrupedal manipulation with human pretraining. In Robotics: Science and Systems (RSS),
8. [8] Z. Wu, Y. Zhou, X. Xu, Z. Wang, and H. Yan (2025) MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation . In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , Los Alamitos, CA, USA, pp. 1714–1723. External Links: ISSN , Document, Link
9. [9] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel (2023) GELLO: a general, low-cost, and intuitive teleoperation framework for robot manipulators.
10. [10] A. A. Team, J. Aldaco, T. Armstrong, R. Baruch, J. Bingham, S.-H. G. Chan, K. L. Draper, D. Dwibedi, C. Finn, P. Florence, S. Goodrich, W. Gramlich, T. Hage, A. Herzog, J. Hoech, T. Nguyen, I. Storz, B. Tabanpour, L. Takayama, J. Tompson, A. Wahid, T. Wahrburg, S. Xu, S. Yaroshenko, K. Zakka, and T. Zhao (2024) ALOHA 2: an enhanced low-cost hardware for bimanual teleoperation. ArXiv abs/2405.02292. External Links: Link
11. [11] Z. Fu, T. Z. Zhao, and C. Finn (2024) Mobile aloha: learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference on Robot Learning (CoRL),
12. [12] X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang (2024) Open-television: teleoperation with immersive active visual feedback. arXiv preprint arXiv:2407.01512.
13. [13] Z. Zhao, L. Yu, K. Jing, and N. Yang (2026) XRoboToolkit: a cross-platform framework for robot teleoperation. In International Symposium on System Integration,
14. [14] A. Khazatsky, K. Pertsch, S. Nair, et al. (2024) DROID: a large-scale in-the-wild robot manipulation dataset. In Proceedings of Robotics: Science and Systems,
15. [15] C. Zhou, C. Peers, Y. Wan, R. Richardson, and D. Kanoulas (2022) TeLeMan: teleoperation for legged robot loco-manipulation using wearable imu-based motion capture. External Links: 2209.10314, Document, Link
16. [16] D. Seidel, A. Schmidt, X. Luo, A. Raffin, L. Mayershofer, T. Ehlert, D. Calzolari, M. Hermann, T. Gumpert, F. Löffl, E. Den Exter, A. Köpken, R. Luz, A. Bauer, N. Batti, F. Lay, A. Manaparampil, A. Albu-Schäffer, D. Leidner, P. Schmaus, T. Krüger, and N. Y. Lii (2024) Toward space exploration on legs: iss-to-earth teleoperation experiments with a quadruped robot. In 2024 IEEE Conference on Telepresence, Vol. , pp. 10–15. External Links: Document
17. [17] Y. T. Tefera, I. Sarakoglou, S. N. Deore, Y. Kim, V. Barasuol, M. Villa, S. Anastasi, D. G. Caldwell, N. Tsagarakis, C. Semini, and N. Deshpande (2024) ROBOT teleoperativo: collaborative cybernetic systems for immersive remote teleoperation. In 2024 IEEE Conference on Telepresence, Vol. , pp. 1–4. External Links: Document
18. [18] X. J. Z. X. J. Z. Y. Gao and K. Hashimoto (2025) Human–robot interaction using dynamic hand gesture for teleoperation of quadruped robots with a robotic arm. Electronics.
19. [19] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song (2024) Universal manipulation interface: in-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS),
20. [20] H. Ha, Y. Gao, Z. Fu, J. Tan, and S. Song (2024) UMI on legs: making manipulation policies mobile with manipulation-centric whole-body controllers. In Proceedings of the 2024 Conference on Robot Learning,
21. [21] Nintendo (2026) Joy-con controllers. Note: Nintendo Switch SupportAccessed: 2026-08-13 External Links: Link
22. [22] G. Turrisi, A. Bratta, G. Minelli, G. F. Abati, A. H. Rad, J. C. Virgolino Soares, and C. Semini (2026) BinWalker: development and field evaluation of a quadruped manipulator platform for sustainable litter collection. In 2026 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Vol. , pp. 1–8. External Links: Document
23. [23] Y. Nisticò, J. C. V. Soares, L. Amatucci, G. Fink, and C. Semini (2025) MUSE: a real-time multi-sensor state estimator for quadruped robots. IEEE Robotics and Automation Letters 10 (5), pp. 4620–4627. External Links: Document
24. [24] A. De Luca and R. Mattone (2005) Sensorless robot collision detection and hybrid force/motion control. In Proc. IEEE Int. Conf. Robotics and Automation, pp. 999–1004. External Links: Document
25. [25] J. G. W. Wildenbeest, D. A. Abbink, C. J. M. Heemskerk, F. C. T. van der Helm, and H. Boessenkool (2013) The impact of haptic feedback quality on the performance of teleoperated assembly tasks. IEEE Transactions on Haptics 6 (2), pp. 242–252. External Links: Document
26. [26] C. Cuan, A. M. Okamura, and M. Khansari (2024) Leveraging haptic feedback to improve data quality and quantity for deep imitation learning models. IEEE Transactions on Haptics 17 (4), pp. 984–991. External Links: Document
27. [27] D. A. Lawrence (1993) Stability and transparency in bilateral teleoperation. IEEE Transactions on Robotics and Automation 9 (5), pp. 624–637. External Links: Document
28. [28] M. Franken, S. Stramigioli, S. Misra, C. Secchi, and A. Macchelli (2011) Bilateral telemanipulation with time delays: a two-layer approach combining passivity and transparency. IEEE Transactions on Robotics 27 (4), pp. 741–756. External Links: Document
29. [29] B. Hannaford and J. Ryu (2002) Time-domain passivity control of haptic interfaces. IEEE Transactions on Robotics and Automation 18 (1), pp. 1–10. External Links: Document
30. [30] F. Ferraguti, C. Secchi, and C. Fantuzzi (2013) A tank-based approach to impedance control with variable stiffness. In Proc. IEEE Int. Conf. Robotics and Automation, pp. 4948–4953. External Links: Document
31. [31] R. Cadène, S. Aliberts, F. Capuano, M. Aractingi, A. Zouitine, P. Kooijmans, J. Choghari, M. Russi, C. Pascal, S. Palma, M. Shukor, J. Moss, A. Soare, D. Aubakirova, Q. Lhoest, Q. Gallou’edec, and T. Wolf (2026) LeRobot: an open-source library for end-to-end robot learning. ArXiv abs/2602.22818. External Links: Link
32. [32] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link