Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning
Training the pi0.5 robot policy inside PhysCanvas, a world model that predicts camera images and fingertip touch together, raised its average real-robot success on four contact-heavy tasks from 59.1% to 70.8%, a team from the Chinese Academy of Sciences' Institute of Automation and Anyverse Robotics reports in a preprint. The model's simulated success rates also tracked real ones far more closely, with the mean task-wise correlation rising from 0.602 to 0.885.
Can a learned world model that predicts both vision and touch stand in for a real robot when testing and improving manipulation policies, so that its verdicts match what happens on hardware?
Testing and fine-tuning robot policies on real hardware is slow and costly, so labs turn to action-conditioned world models as simulators. Video models can produce plausible-looking frames while getting contact wrong, which makes their pass or fail judgements unreliable for tasks such as unscrewing a cap.
Earlier robot world models, including the WoVR baseline compared in the paper, predict camera video from actions and often add separate networks for touch or task state. Visual quality alone does not guarantee that the predicted outcome agrees with the real one.
PhysCanvas places multi-view RGB, tactile images, tactile marker displacement, task state and actions on one shared image canvas, so every modality runs through the same video autoencoder and a pretrained video backbone (Wan 2.2). Actions are drawn as motion images obtained from forward kinematics and camera projection. After pre-training on 3,805 real policy rollouts from an ARX AC One dual-arm robot with three cameras and four Xense fingertip tactile sensors, the model is post-trained with three frozen reward models that check scene preservation, vision and touch agreement, and agreement with the real task outcome. The resulting simulator scores closed-loop rollouts and supplies imagined interactions and rewards for GRPO policy updates.
On four tasks (test-tube placement, bottle-cap unscrewing, block relocation, Rubik's Cube manipulation, 80 held-out test rollouts each) the state-reward post-training raised the mean task-wise Pearson correlation between world-model and real success rates from 0.602 to 0.885, improving it on every task. With GRPO in the base world model, pi0.5 rose from 59.1% to 67.5% mean real-world success, and to 70.8% in the post-trained model, a gain of 11.7 points (MEKAWIRE subtraction); a flow-matching policy rose from 11.2% to 16.6% and then 19.1%. The extra gains of the post-trained simulator for pi0.5 were largest on Blocks (6.6 points) and Rubik's Cube (10.0 points).
This is a preprint on one dual-arm robot, one tactile sensor type and four tasks. Benefits vary by task: pi0.5's bottle-cap success fell from 90.0% to 86.6%, and the flow-matching policy stayed at zero on test tubes and the Rubik's Cube. ACT was left out of the reinforcement-learning tests because the method assumes diffusion or flow-matching policies. Correlation of success rates is measured over a limited set of evaluation batches, and the reward models are trained on the same task family, so transfer to new tasks is untested.
If world models can be trusted to rank policies, robot makers can cut the hardware hours spent on evaluation and fine-tuning, which is a cost line for every humanoid and dual-arm programme training VLA models such as Physical Intelligence's pi0.5. Adding touch matters for assembly, lab automation and household tasks where vision misses contact. The dual-arm platform (ARX) and fingertip sensors (Xense) used here are off-the-shelf research hardware; the commercial question over one to two years is whether such simulators become part of standard VLA post-training stacks.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.