FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity
A Stanford-led preprint, with Fei-Fei Li and Jiajun Wu among the authors, shows that human demonstrations recorded with force-sensing gloves can teach robot hands fiddly jobs such as pipetting and opening test tubes with only 10 robot demonstrations per task. Its model beat Physical Intelligence's pi-0.5 and the tactile policy T-Rex on ten real-robot tasks, in some cases by 60 percentage points.
Can large amounts of human manipulation data that include measured fingertip force replace most of the expensive robot teleoperation data needed to train dexterous, contact-sensitive robot hands?
Human video datasets are cheap to scale but record only what hands look like, not how hard they press, so robots trained on them miss the force control that tasks like turning a pipette dial or plucking a harp string depend on. Collecting robot demonstrations with force data instead is slow and costly.
Earlier tactile policies such as T-Rex fed compressed force history into the policy as an input and refined actions afterwards, which leaves the link between motion and contact implicit. General robot foundation models such as pi-0.5 do not model force at all.
The team built EgoForce, about 2,000 hours, 40,000 episodes and 1,000 tasks from 1,000 people, recorded with three egocentric cameras, tactile gloves on both hands and automatic force calibration. FACT then works in two jointly trained stages: a planner built on the Wan2.2 video model predicts future video, visual point tracks and a coarse action chunk, and a refiner co-denoises the remaining action together with predicted future fingertip force so that each shapes the other.
On a bimanual station with two 22-joint Sharpa Wave hands, fine-tuned per task on 10 robot plus 90 human trajectories, FACT reached 40% on lab pipetting, 80% on opening a test-tube cap and 40% on multi-object grasping, against 0/20/0% for T-Rex and 0/10/0% for pi-0.5; it sounded 6.7 of 10 harp notes cleanly against 2.1 and 0.5. Removing force supervision from the human pretraining data cut test-tube opening from 80% to 0% and raised pipette-dial error from 6.4 to 20.5 microlitres, and tactile supervision lowered validation loss by 10 to 14% at every data scale from 100 to 2,000 hours.
This is a preprint evaluated on ten tasks with ten rollouts each, so single-task gaps of 10 to 20 points rest on one or two trials. Absolute success rates stay low (20% on shoelaces, 30% on card shuffling), everything runs on one hand type and one station, and training used 240 Alibaba T-Head accelerators, so the recipe is expensive to reproduce; we also see no test of how the glove data transfers to hands with different finger layouts.
The result supports the business case of firms selling human-data capture, such as glove and egocentric-camera data operations, and of hand makers such as Sharpa whose fingertips sense six-axis force. For laboratory automation and electronics assembly, where force matters more than reach, it suggests a path in which a few robot demonstrations per task are enough once a large human tactile corpus exists, on a two-to-four-year horizon before such policies are robust enough for production lines.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.