GPT-6 Astra Is a Native Visuo-Tactile Policy for Contact-Rich Manipulation
A general multimodal model, GPT-6 Astra, controlled a simulated robot hand from a few demonstrations and reached 93.8% average success on eight UniVTAC contact-rich tasks when it was shown images from the gripper's touch sensors, 22.9 points above the best published specialist policy. The preprint from Tsinghua University and Alibaba's Amap suggests touch-heavy factory tasks such as insertion may not need a separately trained tactile model, but every test was in simulation.
Can a general multimodal model that was never trained on tactile data read images from vision-based touch sensors and use them to control a robot in contact-rich tasks, and does that beat policies built specifically for touch?
Insertion, stable grasping and judging an object's hardness or weight depend on contact information that a camera view of the scene often does not show. The field has assumed that such tasks need specialist policies trained on paired visual and tactile data.
Specialist tactile policies such as SimpleTouch, OpenVTLA and FTP-1 learn joint visual and tactile representations, and generalist vision-language-action models are trained on robot data. Earlier work had shown GPT-6 Astra can act as a robot policy through in-context learning, but only from ordinary camera images.
The authors feed the left and right GelSight-style tactile images through the model's existing image input, alongside fixed and wrist camera views, task text and joint states. A few demonstration segments, each pairing actions with the observations before and after, stay in the context window at every step, and the model outputs seven-dimensional arm and gripper commands in a closed loop. The same scenes and demonstrations are run with and without tactile images.
On 19 simulated tasks (8 UniVTAC, 11 OpenViTac; 210 scene seeds per setting, a 10% stratified sample), Astra with touch averaged 93.8% on UniVTAC and 81.8% on OpenViTac, against 70.9% for SimpleTouch and 68.7% for OpenVTLA, the strongest reported baselines (gaps of 22.9 and 13.1 points). Vision alone already reached 76.3% and 80.6%. It beat FTP-1 by 30.9 and 20.6 points. With touch it reached at least 90% on 13 of 19 tasks and 100% on nine. The two runs used 335.97 million tokens and cost CNY 7,505.05 in provider fees; touch input used about 59% more tokens and 42% more money. Median API response time was 15.4 s with vision and 14.0 s with touch.
Everything ran in simulation, so sensor noise, calibration drift and real contact dynamics are untested. Most tasks had only ten episodes per setting, so one outcome moves a task score by ten points, and baselines are published numbers rather than reruns on the same sampled scenes. Astra struggles with sideways (tangential) forces, fine corrective moves and probing that fails to converge. By MEKAWIRE's arithmetic the evaluation cost about CNY 17.9 per episode on average, and responses of 14 to 15 s are far too slow for real-time control on a production line.
If the result carries to hardware, makers of tactile sensors gain a route to market through general models rather than per-task training, and integrators could prototype insertion or fragile-handling cells from a handful of demonstrations. The latency and per-episode cost mean the near-term use is data collection, task planning and supervision on top of a faster low-level controller, on a two to four year horizon. The authors also warn that the benchmarks are close to saturated, so harder tactile tests will matter for anyone comparing policies.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.