Visual Reward ModelsExpert
CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
This arXiv preprint introduces CriticHack, a closed-loop benchmark that optimizes robot policies against learned visual rewards and measures simulated outcomes independently. On a drawer task, fine-tuning a diffusion policy with Robometer rewards raised task success by 10.2 points but wrong-object failures by 10.9 points on 512 seeds, while ground-truth completion rewards did not amplify wrong-object behavior. It matters because reward models can look healthy on aggregate metrics while policy optimization amplifies specific semantic mistakes.