Learning Force-Reactive Humanoid Loco-Manipulation
A humanoid policy that predicts and commands contact forces, not just positions, completed 80% of real-world trials on three pushing, rotating and wiping tasks against 26.7% for a position-only policy, in a preprint from Amazon's Frontier AI & Robotics team with NYU, CMU and the University of Chicago. The training data came from handheld force-sensing grippers, with no robot in the loop.
Can a humanoid learn tasks that depend on how hard it pushes, not only where its hands go, from demonstrations recorded without a robot, and can a whole-body controller actually deliver the commanded force while the robot walks and balances?
Robot-free data collection with handheld grippers has made imitation learning cheap, but it records motion only. Pushing a heavy cart, turning a box with two hands or wiping a board needs the right force; a policy that copies positions either loses contact or pushes too hard, and standard humanoid whole-body controllers accept no force command at all.
Whole-body trackers such as SONIC follow reference motions for the hands and body but have no force input. Adding an admittance loop around them lets a force be requested, but the servo works outside the balance policy and delivers only part of it. Diffusion policies trained on motion-only demonstrations ignore contact force.
FLoRe records demonstrations with two UMI-style handheld grippers fitted with force-torque sensing plus wearable trackers for the body, so each demonstration carries hand motion and contact wrench. A hybrid compliance whole-body controller is trained to track both motion and a commanded hand force, integrating the measured force error so it does not need to know the stiffness of what it touches. The high-level policy reads force and outputs pose plus a force command, and the system runs on a Unitree G1.
In simulation the controller tracks 5 to 25 N commands with a gain of 1.03 to 1.06 and about 1 N of in-contact error per hand, against a gain of 0.67 to 0.70 and 7.9 N of error for SONIC with an admittance loop. On the real robot, pressing a mounted force gauge, it reached about 91% of the commanded force along the forward axis with 1.1 N mean error and settled within 1 N in 1.8 s. On three real tasks (pushing a gantry carrying another robot, rotating a box with both hands, wiping a board) the full system succeeded in 24 of 30 trials (80%). Withholding the force command from the controller dropped success to 36.7%, removing force input to the policy to 66.7%, and a position-only diffusion policy reached 26.7%.
Each task was run 10 times, so the success rates carry wide uncertainty, and only three tasks on one robot model were tested. Sideways and downward force commands reach a lower gain and leak into other axes; commands are capped at 25 N and zeroed below 1.5 N because the sensor reads about 1.3 to 1.5 N at rest. Force tracking costs some motion accuracy: 1.3 to 3.2 cm more root error than the same controller without force. This is a preprint.
Contact-rich work such as pushing carts, handling large boxes and cleaning is where warehouse and service humanoids earn their keep, and it is exactly what motion-only imitation learning misses. That Amazon's own robotics research group built this on a Unitree G1 with low-cost handheld data collection matters for Agility, Figure, Apptronik and Chinese makers: force-annotated, robot-free data could become a standard data product within one to two years, and controllers will need a force input to use it.
Full paper
Learning Force-Reactive Humanoid Loco-Manipulation
Published under CC BY 4.0. Reproduced with attribution; original at arXiv:2610.35862 (PDF).
Abstract
Many everyday tasks require controlling contact force as well as motion. On a humanoid, this force must often be coordinated across both hands and the whole body, and cannot be specified by kinematic trajectories alone. Therefore, we present FLoRe, a framework for learning force-reactive humanoid loco-manipulation. We collect robot-free demonstrations using two force-sensing UMI-style handheld grippers for hand motion and contact wrenches, and wearable trackers for body motion. A force-reactive diffusion policy takes vision, proprioception, and measured wrenches as input, then predicts actions and per-hand force commands. A hybrid compliance whole-body controller with force tracking executes these commands while maintaining balance: an integral of the force error measured by real force/torque (F/T) sensors lets each hand track force accurately along the commanded direction and remain compliant along the others. In simulation, the controller tracks per-hand force commands of varied magnitude and direction, with little loss in whole-body motion accuracy. Across three contact-rich tasks on the robot, the force-reactive policy succeeds in 80% of rollouts, compared with 26.7% for a position-only policy trained on the same demonstrations and executed by the same controller. For more details, please visit our website: https://louis-leee.github.io/FLoRe/.
I Introduction
Humanoid robots offer a general-purpose embodiment for working in environments built for people. Many tasks in these environments, such as pushing a cart, moving a heavy box, and wiping a surface, require the robot to control contact force as well as motion. However, learned humanoid loco-manipulation is commonly formulated as generating and tracking motion references [1, 2, 3]. Tracking a demonstrated hand trajectory alone does not reproduce the demonstrated force, which also depends on contact geometry, stiffness, and object motion. Unlike a fixed-base arm, a humanoid must stay balanced under these contact loads, coordinating its hands, torso, and legs while regulating force.
Two difficulties stand in the way. The first is data: imitation learning requires scalable demonstrations that record contact force alongside motion. Existing robot-free interfaces for whole-body demonstration collection record motion but not force [4, 5]. The second is execution: the robot must accurately apply the desired force. Current learned humanoid whole-body controllers track motion alone, or treat force as a disturbance to reject or a stiffness to lower [6, 7]. Tracking a commanded contact force has been shown only on quadrupedal manipulators, using estimated rather than measured force [8, 9], which is not accurate enough. A humanoid instead needs to track force at both hands together with whole-body motion while maintaining balance.
We present FLoRe, a framework for learning force-reactive humanoid loco-manipulation. Its robot-free force-annotated data collection system records contact force with each demonstration; the policy learns to predict this force, and the controller tracks it. The system pairs two UMI-style handheld grippers [10] with a chest tracker and two ankle trackers [4, 11]. Each gripper carries a wrist camera, a pose tracker, and a six-axis F/T sensor [12, 13]. The applied wrench is thus recorded alongside hand, torso, and foot motion. The robot uses the same gripper–sensor assembly.
For the high-level planner, we propose a force-reactive diffusion policy [14]. It observes wrist images, robot state, and the measured wrench at each hand, then predicts a chunk [15] of action, gripper widths, and per-hand desired force. Force enters as both an input and an output. The input lets the policy react to interactions that images do not reveal. The output is commanded to the controller [13], rather than used only as an auxiliary target [16]. This connects the force measured during demonstration to the deployed force.
To track the commanded forces, we develop a whole-body controller with hybrid compliance. It outputs only joint positions, so each hand’s force command becomes a shift of its position target. In contact, an integral of the force error measured at the wrist adjusts this shift until the hand exerts the commanded force, whatever the contact stiffness. Along uncommanded axes, the same integral lets the hand yield to external pushes. Each hand thus tracks force along the commanded direction and complies along the others, without switching modes. We train the controller in simulation [17] and deploy it on the robot using wrist F/T measurements.
We deploy FLoRe on a Unitree G1 and evaluate three whole-body contact-rich tasks: pushing a gantry hanging another robot, bimanual box rotation, and board wiping. In simulation, the controller realises in-contact force error of about 1 N per hand while tracking whole-body motion. On the real robot, FLoRe succeeds in 80% of rollouts, compared with 26.7% for a position-only policy trained on the same demonstrations and executed by the same controller with force commands disabled. Withholding only the force command reduces success to 36.7%, while removing only the force observation reduces it to 66.7%. Both contribute to the final performance. Our key contributions are:
- A robot-free pipeline for learning force-reactive humanoid loco-manipulation from demonstrations that record both motion and contact wrench.
- A learned whole-body controller that tracks per-hand force together with keypoint motion while remaining compliant to external forces and maintaining balance.
- Evaluation in simulation and on three contact-rich tasks on a Unitree G1, where force commands and force observations both improve task success over position-only control.
II Related Work
II-A Force Sensing for Loco-Manipulation Data Collection and Policy Learning
Robot-free data collection builds on UMI [10]. Force-aware variants add wrist or fingertip sensing, but deploy on fixed-base arms with analytic impedance or admittance control [12, 13, 18]. Humanoid extensions instead collect whole-body motion and execute it with learned trackers, but do not record contact force [4, 11]. WT-UMI is the closest interface for humanoids, although it measures only a scalar normal force at each palm and relies on paired teleoperation data [19]. Our interface records a six-axis wrench at each hand together with torso motion, and its demonstrations require no robot-side data.
Measured force has been used to condition visuomotor policies [20, 21]. Other work predicts stiffness, admittance gains, or virtual position targets that are executed by an impedance controller [22, 23, 24, 12]. More closely related methods predict a wrench trajectory together with pose, but execute it on fixed-base arms through analytic force controllers [13, 25, 26]. On humanoids, WT-UMI passes a scalar normal-force prediction to an admittance rule [19]. UniFP is the closest formulation. It predicts 3-D force commands, but deploys end-effector force control only on a quadrupedal manipulator. Its humanoid interface exposes force only at the base [9]. Our planner instead predicts per-hand pose, gripper, and wrench trajectories from handheld demonstrations, then sends the predicted force to a learned humanoid whole-body controller.
II-B Force for Low-Level Control
Learned legged and humanoid controllers usually either withstand external force or yield to it through commanded stiffness or damping [6, 27, 28, 29, 7, 30], but neither approach directly specifies how hard to press. Direct force tracking has mainly been studied on quadrupedal manipulators, as a separate control mode or through a target-shifting spring law, both using force estimated from proprioception [8, 9]. WT-UMI regulates a scalar palm force on a humanoid, but through an analytic admittance rule and inverse kinematics above a separate locomotion controller [19]. In contrast, our hybrid compliance whole-body controller with force tracking integrates the force error measured at each wrist. One learned policy tracks position in free space and force along the commanded direction, while remaining compliant along the other directions and maintaining balance.
III Method
In FLoRe (Fig. 1), a hybrid compliance whole-body controller with force tracking executes motion references and per-hand force commands (Sec. III-A); a robot-free force-annotated data collection system records vision, motion, and contact wrenches with handheld devices (Sec. III-B); and a force-reactive diffusion policy, trained on these demonstrations, predicts keypoint motion and force commands for the controller (Sec. III-C).
Notation. The robot is a humanoid with a parallel-jaw gripper on each hand. The controller tracks keypoints k\in\mathcal{K} (both hand tool center point (TCP), the torso, and both feet) with poses \mathbf{T}_{k}=(\mathbf{p}_{k},\mathbf{R}_{k}) expressed in the pelvis frame. For each hand h\in\mathcal{H}, \mathbf{f}^{\mathrm{cmd}}_{h} is the force the hand should exert on the environment, \mathbf{f}^{\mathrm{meas}}_{h} the reaction force measured by its wrist F/T sensor, and \mathbf{f}^{\mathrm{ext}}_{h} the true contact force available only in simulation; all are expressed in the pelvis frame and act at the TCP, so that perfect tracking gives \mathbf{f}^{\mathrm{meas}}_{h}=-\mathbf{f}^{\mathrm{cmd}}_{h}. The high-level policy predicts chunks of H steps; the low-level controller runs at a higher rate and sends joint-position targets to joint PD controllers. Hardware, rates, and all numerical settings are given in Sec. IV-A.
III-A Hybrid Compliance Whole-body Controller with Force Tracking Capability
The controller is a single learned policy that tracks whole-body motion together with a commanded force at each hand. We describe its control law, the contact model and simulated sensor it is trained with, and its training objective.
Control law and unified reference generator. The controller outputs only joint positions and is rewarded only on keypoint poses, so a force command must become a position target. For each hand h, a unified reference generator combines the motion reference \mathbf{p}^{*}_{h} and the force command \mathbf{f}^{\mathrm{cmd}}_{h} into a single target (see Fig. 2):
\displaystyle\mathbf{p}^{\mathrm{tgt}}_{h} \\ \displaystyle=\underbrace{\mathbf{p}^{*}_{h}}_{\text{motion reference}}+\underbrace{\frac{\mathbf{f}^{\mathrm{cmd}}_{h}}{K}}_{\text{force command}}+\underbrace{\frac{\mathbf{I}_{h}}{K}}_{\text{force feedback}}, \\ \displaystyle\mathbf{I}_{h} \\ \displaystyle\leftarrow c_{h}\,\mathrm{clip}\big(\mathbf{I}_{h}+K_{I}\underbrace{(\mathbf{f}^{\mathrm{cmd}}_{h}+\mathbf{f}^{\mathrm{meas}}_{h})}_{\text{force error}}\Delta t,\,\pm I_{\max}\big),with nominal stiffness K, integral gain K_{I}, bound I_{\max} and time step \Delta t; \Delta\mathbf{x}_{h}=(\mathbf{f}^{\mathrm{cmd}}_{h}+\mathbf{I}_{h})/K is the total shift. The command moves the target \mathbf{f}^{\mathrm{cmd}}_{h}/K along the force direction; where the target lies inside the surface, the hand cannot follow it and presses instead, and tracking means \mathbf{f}^{\mathrm{meas}}_{h}=-\mathbf{f}^{\mathrm{cmd}}_{h}. How hard a given shift presses depends on the contact stiffness, which the robot does not know. The integral of the force error therefore moves the target further in, or back, until the measured force balances the command, independent of K and K_{\mathrm{env}}; unlike the spring law of UniFP [9], which shifts the target in proportion to an estimated contact force, the shift is thus closed on the measured force. The soft contact phase c_{h}\in[0,1], computed from the measured force magnitude with onset and release hysteresis thresholds and a first-order low-pass filter, lets the integral build up only in contact and clears it on release, while the clip bounds windup. It requires no privileged contact information, and also enters the observation and weights the force-related rewards. \mathbf{p}^{\mathrm{tgt}}_{h} replaces the hand target in the tracking reward, and \mathbf{I}_{h} and \Delta\mathbf{x}_{h} enter the observation; the joint command is untouched, so the policy remains the sole executor.
\mathbf{p}^{\mathrm{tgt}}_{h} is shifted from the motion reference \mathbf{p}^{*}_{h} into the surface by the force command and the force feedback, until the measured force \mathbf{f}^{\mathrm{meas}}_{h} balances the command. Along the compliance direction, the gripper yields to \mathbf{f}^{\mathrm{ext}}_{h} (dashed gripper).The law acts on every axis of each hand, and a measured force need not lie along the command. Dropping the hand index, let a force be commanded along x and none along z. A push along z is then measured too, f^{\mathrm{meas}}_{z}=f^{\mathrm{ext}}_{z}:
\displaystyle x \\ \displaystyle:\ f^{\mathrm{cmd}}_{x}\neq 0\Rightarrow f^{\mathrm{meas}}_{x}\to-f^{\mathrm{cmd}}_{x}\quad\text{(force tracking)}, \\ \displaystyle z \\ \displaystyle:\ f^{\mathrm{cmd}}_{z}=0\Rightarrow\dot{I}_{z}=K_{I}f^{\mathrm{meas}}_{z}=K_{I}f^{\mathrm{ext}}_{z}\quad\text{(compliance)}.Along z the integral grows with the push and moves the target with it at a rate K_{I}f^{\mathrm{ext}}_{z}/K, so the hand gives way instead of resisting, for as long as the push lasts and within the bound I_{\max}. Each hand thus tracks force along the commanded direction and remains compliant along the others, with no switching between them.
Contact model and simulated sensor. In training each hand faces a virtual compliant plane with outward normal \mathbf{n}_{h} and randomized stiffness K_{\mathrm{env}} that reacts only to penetration,
\mathbf{f}^{\mathrm{ext}}_{h}=K_{\mathrm{env}}\,\max\!\big(0,\ \mathbf{n}_{h}^{\!\top}(\mathbf{x}^{\mathrm{wall}}_{h}-\mathbf{p}_{h})\big)\,\mathbf{n}_{h},applied at the TCP, with penetration capped at d_{\max} and the force capped as well. The policy observes a simulated F/T reading instead of \mathbf{f}^{\mathrm{ext}}_{h},
\mathbf{f}^{\mathrm{meas}}_{h}=\mathbf{f}^{\mathrm{ext}}_{h}+m_{d}\,\ddot{\mathbf{p}}_{h}^{\mathrm{wrist}}+\mathbf{b}_{h}+\boldsymbol{\epsilon}_{h},with randomized distal mass m_{d}, per-episode bias \mathbf{b}_{h} and white noise \boldsymbol{\epsilon}_{h}. At each reset the plane is placed at a random distance beyond the hand’s reset position with a random orientation, and \mathbf{f}^{\mathrm{cmd}}_{h} follows trapezoidal ramp–hold–release pulses with random peaks, clamped to what the plane can deliver, K_{\mathrm{env}}d_{\max}. The motion reference is the unmodified clip; where it conflicts with the force, the shifted target of (1) wins. The plane is frictionless, so training commands are purely normal; on the robot the same per-axis law applies along tangential axes as above, although tangential loads are never seen in training.
Training objective. The reward extends the motion-tracking reward of [1], r_{\mathrm{track}}, with terms for the hands:
\displaystyle r=r_{\mathrm{track}} \\ \displaystyle+\,r_{\mathrm{pos}}+\,r_{\mathrm{tcp}}+\,r_{\mathrm{orn}}+\,r_{\mathrm{force}} \\ \displaystyle+\,r_{\mathrm{contact}}+\,r_{\mathrm{approach}}-\,r_{\mathrm{rate}}.All tracking terms use the shifted target of (1): r_{\mathrm{pos}}, r_{\mathrm{tcp}} and r_{\mathrm{orn}} are exponential kernels on the hand-position, TCP-position and hand-orientation errors, the latter two constraining the wrist-to-TCP offset. Weighted by c_{h} while a force is commanded, r_{\mathrm{force}} scores the force error against the clean simulated force and r_{\mathrm{contact}} rewards contact below the force cap; weighted by 1-c_{h}, r_{\mathrm{approach}}=\exp(-d_{h}/d_{0}) pulls the hand toward the plane, with d_{h} its distance and d_{0} a length scale. r_{\mathrm{rate}} penalizes changes in contact force between steps and softens impacts.
III-B Force-annotated Robot-free Data Collection System
Device. Each handheld UMI-style gripper [10] carries an F/T sensor1 between handle and gripper, a wrist camera, a pose tracker, and a jaw-width encoder; the demonstrator holds one in each hand and wears trackers on the chest and both ankles (Fig. 3). The robot carries the same gripper–sensor assembly on each wrist, so sensor frame, sensor-to-TCP transform and gripper geometry are identical in collection and deployment. The wrench is kept in the sensor frame, gravity-compensated with a per-device payload and tared at the start of each recording; on the robot, inertial loads are also compensated.
Streams and synchronization. Images, tracker poses, wrenches, and gripper widths arrive at different rates and are stamped on arrival with one host clock; the labels below are built from the raw streams, not the camera grid. Missing wrench readings and tare transients are masked rather than zero-filled, since a tared sensor also reads zero in free space.
III-C Force-Reactive High-level Policy Learning
The high-level policy \pi_{\theta} is a diffusion policy [14] that maps the current observation to a chunk of H future actions [15]. We augment both its inputs and outputs with per-hand contact wrenches.
Observation. The observation \mathbf{o}_{t} contains the wrist images of both hands, \mathbf{I}^{L}_{t} and \mathbf{I}^{R}_{t}, the measured wrenches \{\mathbf{w}^{\mathrm{meas}}_{h,t}\}_{h\in\mathcal{H}}, and a low-dimensional state:
\displaystyle\mathbf{s}_{t}=\big[ \\ \displaystyle(\mathbf{p}_{H_{L}}-\mathbf{p}_{H_{R}})_{xy},\ (\mathbf{p}_{H_{L}}-\mathbf{p}_{T})_{xy},\ (\mathbf{p}_{H_{R}}-\mathbf{p}_{T})_{xy}, \\ \displaystyle\mathrm{rpy}(\mathbf{R}_{H_{L}}),\ \mathrm{rpy}(\mathbf{R}_{H_{R}}),\ g_{H_{L}},\ g_{H_{R}}\big].The state contains pairwise horizontal position differences among the hands and torso, hand orientations in the pelvis frame, and gripper widths g_{h}. Here, (\cdot)_{xy} selects the horizontal components, and \mathrm{rpy}(\mathbf{R}) returns the extrinsic-XYZ Euler angles (roll, pitch, yaw) of \mathbf{R} in radians. Using position differences makes the state invariant to absolute translation. We omit absolute poses and joint angles because they differ between the completed demonstration motion and the robot.
Action. Each step j of the chunk stacks the pose, gripper width and force command \mathbf{f}^{\mathrm{cmd}}_{h} of each hand, together with the poses of the torso and both feet:
\displaystyle\mathbf{a}_{t+j} \\ \displaystyle=[\{\tilde{\mathbf{T}}_{h},g_{h},\mathbf{f}^{\mathrm{cmd}}_{h}\}_{h\in\mathcal{H}},\{\tilde{\mathbf{T}}_{k}\}_{k\in\mathcal{K}\setminus\mathcal{H}}], \\ \displaystyle\tilde{\mathbf{T}}_{k} \\ \displaystyle=\mathbf{T}_{k,t}^{-1}\,\mathbf{T}_{k,t+j}.Poses are anchor-relative, in each keypoint’s own frame at the observation time, so the chunk is invariant to the world frame and to where the demonstrator stood [4, 11]; rotations use the continuous rotation representation of [31].
Model and training. Each image is encoded by a CLIP-initialized vision transformer [32], the state by a linear layer, and each hand’s wrench history by a small causal convolutional encoder, in which a learned embedding replaces masked windows. These condition a temporal U-Net [33] denoiser through FiLM [34], trained with the DDPM objective [35] and sampled with DDIM [36]. Actions are quantile-normalized per column and the loss is a masked Mean Squared Error (MSE) Loss.
Closing the loop. At each policy step the chunk is decoded into absolute pelvis-frame keypoint targets, re-based onto the currently commanded keypoints so that a re-plan never jumps, and streamed to the controller either sequentially (execute N steps, then re-plan) or asynchronously with temporal ensembling [15, 37]. The predicted force of each hand becomes its \mathbf{f}^{\mathrm{cmd}}_{h} (Sec. III-A).
IV Experiments
We evaluate whether a humanoid can learn force-reactive loco-manipulation from robot-free demonstrations. Experiments in simulation and on the robot test four capabilities:
- Whole-body force tracking: tracking per-hand force commands and whole-body motion with one learned controller.
- Sustained contact: making contact and holding an appropriate force while the hands move with or along the object.
- Bimanual and whole-body coordination: applying force with both hands while the torso and legs balance and walk.
- Robustness to force variation: completing tasks when resistance changes within or across episodes but is not visible in the images, e.g., a heavier load or box mass.
We first compare our hybrid compliance whole-body controller with force tracking (Sec. III-A) to three baselines:
- SONIC [3]: a large-scale whole-body motion tracker, run from its released checkpoint. It tracks motion references and accepts no force command.
- SONIC + admittance: SONIC followed by an analytic force servo that adds
\alpha\,\mathbf{J}_{h}^{\!\top}(\mathbf{f}^{\mathrm{cmd}}_{h}+\mathbf{f}^{\mathrm{meas}}_{h})to its arm joint targets, a correction that SONIC does not observe. - FLoRe w/o Force: our controller trained with the force channel removed (
\mathbf{f}^{\mathrm{cmd}}\equiv 0, no\mathbf{I}_{h},\Delta\mathbf{x}_{h},c_{h}observed).
The second comparison isolates how the high-level policy uses force. FLoRe observes the measured wrench (F-in), learns to predict the demonstrated force (F-out), and sends this prediction to the controller (F-cmd). w/o F-cmd and w/o F-in remove one pathway each, and DP all three:
- w/o F-cmd: the FLoRe policy with its predicted force withheld (
\mathbf{f}^{\mathrm{cmd}}\equiv 0), which turns off the force channel (c_{h},\mathbf{I}_{h},\Delta\mathbf{x}_{h}\equiv 0) so that the controller tracks pose only. - w/o F-in: the FLoRe policy trained without the measured wrench in its observation. It still commands force but cannot sense whether contact is made or lost.
- DP: the position-only diffusion policy of UMI-style humanoid systems [4, 11], trained with the same architecture, data and schedule as FLoRe but without force input or output, and executed by our controller with
\mathbf{f}^{\mathrm{cmd}}\equiv 0.
We train one policy per task and evaluate each in 10 rollouts, with test conditions paired across policies. We report the success rate, compared using Fisher’s exact test, and the offline validation error of the predicted pose, force and torque on held-out demonstrations.
IV-A Implementation Details
- Controller
- \mathbf{f}^{\mathrm{cmd}}
- Global Root (m) \downarrow
- Wrist Pos (mm) \downarrow
- Wrist Ori (rad) \downarrow
- a_{1} \rightarrow1
- |e_{F}| (N) \downarrow
- F{=}0 floor (N) \downarrow
- ROOT-90 [38]
- No data
- No data
- No data
- No data
- No data
- No data
- No data
- SONIC [3]
- ✗
- 0.317
- 47.16
- 0.2787
- N/A
- N/A
- N/A
- SONIC + admittance [3]
- ✓
- 0.317
- 47.16
- 0.2787
- 0.70
- 7.89
- 2.62
- FLoRe w/o Force
- ✗
- 0.342
- 36.75
- 0.2239
- N/A
- N/A
- N/A
- FLoRe (Ours)
- ✓
- 0.374
- 44.70
- 0.1176
- 1.06
- 1.05
- 1.50
- VLK-Pick-100 [39]
- No data
- No data
- No data
- No data
- No data
- No data
- No data
- SONIC
- ✗
- 0.072
- 21.45
- 0.2658
- N/A
- N/A
- N/A
- SONIC + admittance [3]
- ✓
- 0.072
- 21.45
- 0.2658
- 0.67
- 7.87
- 2.62
- FLoRe w/o Force
- ✗
- 0.071
- 23.54
- 0.2141
- N/A
- N/A
- N/A
- FLoRe (Ours)
- ✓
- 0.084
- 23.16
- 0.1165
- 1.03
- 1.00
- 1.25
Hybrid Compliance Whole-body Controller. The reference generator uses K=200 N/m, K_{I}=0.5\,\mathrm{s}^{-1} and I_{\max}=40 N; the soft contact phase uses hysteresis thresholds of 2.0 N for onset and 1.5 N for release and a 50 ms low-pass time constant. We sample K_{\mathrm{env}}\sim\mathrm{LogUniform}(100,800) N/m and cap penetration at d_{\max}=8 cm and contact force at 35 N. Command pulses peak at up to 25 N. The reward kernels have widths of 0.1 m (TCP position), 0.25 rad (hand orientation) and 25 N (force). r_{\mathrm{contact}} rewards contact below 35 N.
Policy. The two 224\times 224 wrist images are encoded by a CLIP-initialized ViT-B/16; the state has 14 dimensions, and rotations use the first two rows of the rotation matrix. The 1-D U-Net denoiser is sampled with 16 DDIM steps, the image encoder is trained with a 10\times lower learning rate, and each policy is trained for 10^{5} steps with H=50 (1.67 s).
IV-B Hybrid Compliance Whole-body Low-level Controller
Simulation protocol. Following [1], we train in Isaac Lab [17] and evaluate sim-to-sim in MuJoCo (Fig. 4). Motion/Force tracking is measured with \mathbf{f}^{\mathrm{cmd}}=0 on two held-out sets of retargeted clips: ROOT-90 [38] and VLK-Pick-100 [39]. For force tracking, the dose-response test applies forward commands of 5, 10, 15, 20, and 25 N to each hand in turn, with F{=}0 as the baseline; we also test 10 N commands along \pm x, \pm y, and \pm z, and simultaneous commands to both hands. Statistics are computed over clips in which both paired conditions complete without a fall.
Metrics. For motion tracking, we report global root error and wrist position and orientation errors [40]. For force tracking, the gain a_{1} is the slope of a least-squares fit, with free intercept, of the realized hold-phase force against the true contact force \mathbf{f}^{\mathrm{ext}}. The in-contact error \left\lVert e_{F}\right\rVert_{1}=\left\lVert\mathbf{f}^{\mathrm{cmd}}_{h}+\mathbf{f}^{\mathrm{meas}}_{h}\right\rVert_{1} is averaged over the hold phase of each pulse. The F{=}0 floor is the force read back when nothing is commanded; it bounds the smallest verifiable command.
Adding a force channel costs little motion tracking. Table I compares FLoRe with FLoRe w/o Force, SONIC [3], and SONIC + admittance, all evaluated with \mathbf{f}^{\mathrm{cmd}}=0 on two held-out motion sets. Relative to FLoRe w/o Force, FLoRe adds 1.3–3.2 cm of global root error and at most 8.0 mm of wrist position error, while its wrist orientation error roughly halves; it also tracks orientation better than SONIC, at a higher root error. When a force is commanded, the hand target is shifted by design (\mathbf{f}^{\mathrm{cmd}}/K alone is 12.5 cm at 25 N), and the hand departs from the motion reference only as far as needed to realize the force.
Hybrid compliance controller tracks commanded force accurately. Neither SONIC nor FLoRe w/o Force accepts a force command. SONIC + admittance does, but its servo acts outside the balance policy and realizes only 0.67–0.70 of the commanded force, with an in-contact error of about 7.9 N. FLoRe tracks 5–25 N commands with a gain of 1.03–1.06 and about 1 N of in-contact error per hand; lateral and downward commands reach a lower gain and leak partly into the other axes. With nothing commanded the sensor still reads 1.25–1.50 N, so at deployment we zero commands below 1.5 N and cap them at 25 N. Integrating the measured force error removes the dependence on the unknown contact stiffness: Spring-Law, the target-shift law of [9] without its force-feedback term, cannot detect contact, and its force scales with that stiffness relative to the assumed K. The same integral makes the hand compliant: while it tracks a force, a push along an uncommanded axis is read by the wrist sensor and builds up the integral, which moves the target with it, so the hand gives way instead of resisting, with no mode switch (real robot, Fig. 4 bottom).
- Policy
- F-in
- F-out
- F-cmd
- push_gantry
- rotate_box
- wipe_board
- Mean
- Pose (cm)
- Force (N)
- Torque (N m)
- Ours
- ✓
- ✓
- ✓
- 7/10
- 9/10
- 8/10
- 80%
- 1.18
- 1.29
- 0.148
- w/o F-cmd†
- ✓
- ✓
- ✗
- 3/10
- 4/10
- 4/10
- 36.7%
- 1.18
- 1.29
- 0.148
- w/o F-in
- ✗
- ✓
- ✓
- 6/10
- 7/10
- 7/10
- 66.7%
- 1.78
- 2.51
- 0.277
- DP
- ✗
- ✗
- ✗
- 2/10
- 3/10
- 3/10
- 26.7%
- 2.23
- N/A
- N/A
Real-world force tracking. We verify force tracking on the robot against an external reference: a rigidly mounted digital force gauge that the hand presses along each of \pm x, \pm y, and \pm z while the humanoid stands freely. The command follows a staircase of 4, 6, 8, and 10 N, repeated across directions and hands. Along the x axis, the measured steady-state force reaches approximately 91\% of the commanded magnitude, with a mean absolute steady-state error of about 1.1 N. The response enters a \pm 1 N band in 1.8 s on average, while the steady-state fluctuation remains approximately \pm 0.3–0.5 N. The controller consistently recovers the commanded contact force without loss of stability.
IV-C Push Gantry
Task. As shown in Fig. b, the robot places both hands on a heavy wheeled gantry, from which a second humanoid hangs, and walks it forward to a target distance. The demonstrated hand motion does not specify how hard to push, yet the gantry does not move until the push overcomes its static friction. This is the only task recorded with ankle trackers, so the policy predicts all five keypoints. A rollout succeeds if the gantry reaches the target distance without a stop.
Capabilities. Bimanual and whole-body coordination: the legs and torso produce the push while the two hands balance their forces; otherwise the gantry drifts from its heading. Sustained contact: both hands must keep contact as the gantry accelerates and the robot steps. Robustness to force variation: the hanging humanoid swings as the gantry rolls, so the resistance changes over time.
Test scenarios. We run 10 rollouts per policy over two gantry loads and five initial poses that vary in distance and heading.
Performance. Ours reaches the target in 7/10 rollouts (Table II). It fails when a strong push coincides with a step. DP succeeds in 2/10 rollouts; in the others the gantry either does not move or drifts from its heading as the two hands push unequally (Fig. 5, top). w/o F-cmd succeeds in 3/10 and fails similarly, as it senses the resistance but cannot command force. w/o F-in succeeds in 6/10. Without force feedback, it stalls against the heavier load and keeps walking after losing contact with the lighter one. Offline, Ours has a pose error of 0.82 cm and a force error of 0.77 N per hand.
IV-D Rotate Box
Task. As shown in Fig. c, the robot pinches the rim of a box with both hands. It rotates the box by about 50^{\circ}, drags it about 0.5 m across the table, and releases it. The box remains on the table, so the measured wrench reflects table friction, box inertia, and the internal squeeze between the hands. A rollout succeeds if the box rotates by more than 40^{\circ} and remains upright after release.
Capabilities. Bimanual coordination: the hands share an internal force and rotate about a common axis. In the demonstrations, the right hand applies about 1.4\times the force of the left. If one hand leads the other, the box tips. Sustained contact: the pinch must hold through the drag and the yaw without slipping or lifting the box. Robustness to force variation: friction changes with the box mass and when the box begins to slide.
Test scenarios. We run 10 rollouts per policy over two box masses and five initial yaw angles.
Performance. Ours succeeds in 9/10 rollouts and has an offline pose error of 0.70 cm. DP succeeds in 3/10 and exhibits two failure modes (Fig. 5, bottom left). It either fails to drag the heavy box or leans into it and loses balance. w/o F-cmd succeeds in 4/10 and fails similarly because its predicted squeeze is not executed. w/o F-in succeeds in 7/10 but cannot react when the box begins to slide.
IV-E Wipe Board
Task. As shown in Fig. d, the robot presses an eraser against a whiteboard and wipes a marked strip. Success depends on normal force, which is not visible in the images. A rollout succeeds if the strip is erased without a stop.
Capabilities. Sustained contact: the robot must maintain normal force during each stroke and re-establish it after losing contact. Images alone do not distinguish hovering from pressing. Robustness to force variation: we displace the board by \pm 3 cm along its normal. The demonstrated trajectory then produces either no contact or excessive force if tracked directly.
Test scenarios. We run 10 rollouts per policy over two board offsets (\pm 3 cm along the normal) and two surfaces.
Performance. Ours erases the strip in 8/10 rollouts. DP succeeds in 3/10. It usually hovers above the board (Fig. 5, bottom right). This directly shows that observed motion can be valid supervision in free space but not in contact [41]. w/o F-cmd succeeds in 4/10. It detects contact but cannot increase a force that is too low. w/o F-in succeeds in 7/10 but continues wiping in free space when the board is farther away. This task also has the largest offline errors: 2.02 cm for pose and 1.59 N for force.
IV-F Findings Summary
F1: The force channel costs little motion tracking. Relative to FLoRe w/o Force, FLoRe adds 1.3–3.2 cm of root error and at most 8.0 mm of wrist position error while halving wrist orientation error (Table I); in return it tracks commanded force with a gain of 1.03–1.06 and about 1 N of in-contact error. The 1.25–1.50 N F{=}0 floor bounds the smallest verifiable command, and lateral and downward commands remain weaker.
F2: Force-reactive policies resolve failures that motion alone cannot distinguish. With the same demonstrations and controller, DP succeeds in 26.7% of rollouts and FLoRe in 80% (Table II). The position-only policy stalls when a reached pose does not move the object, hovers or presses too hard when the board is displaced, and loses balance against the heavy box (Fig. 5). Force also composes with coordination: both hands hold the internal force on Rotate Box, and the push is sustained while walking on Push Gantry.
F3: Force commands give the larger, significant gain; force observations show a smaller trend. Withholding the predicted force from the controller reduces success from 80% (24/30) to 36.7% (11/30; Fisher’s exact test, p=0.001): the policy senses resistance but can respond only through pose. Removing the force observation reduces success to 66.7%.
V Conclusion
We presented FLoRe, a framework for learning force-reactive humanoid loco-manipulation from robot-free demonstrations that record both motion and contact wrench. Our hybrid compliance whole-body controller tracks the learned force commands, allowing the humanoid to sustain contact, coordinate both hands, and apply force while walking and balancing. On three tasks, the force-reactive policy succeeds in 80% of rollouts, compared with 26.7% for a position-only policy. By making force a shared interface between demonstrations, policies, and whole-body control, FLoRe extends robot-free imitation learning to contact-rich skills that motion alone cannot specify.