PhysicsLENS: Diagnosing Physical Property Blindness in Video Generation Models

Does high physical plausibility in generated robot videos imply that the model correctly grounded text-specified physical properties, and can an automated pipeline with measured motion and VLM questions approximate human judgments of this distinction?

Video world models can generate realistic robot interactions that violate hidden physical constraints such as object weight, viscosity, friction, or deformability, and current benchmarks only score visual fidelity or general physics plausibility. This makes it impossible to tell whether a model obeyed a property given in text, limiting the reliability of generated videos as predictive environments for robot learning, planning, and evaluation.

Prior physics and robotics benchmarks, including VideoPhy-2, PhyGenBench, Physion-Eval, RoboWM-Bench, RBench, and RoboGaze, evaluate visual quality, motion plausibility, task adherence, or localized glitches rather than whether a text-specified hidden property was followed. Latent-property datasets such as Physion++ and ComPhy ask models to infer mass or friction from observed interactions, which is the reverse of what generative grounding requires. VLM judges used in these benchmarks often score near chance on intuitive-physics tests.

PhysicsLENS constructs matched scenario pairs from 80 public robot video frames. Each pair has the same initial frame and task; the observable description matches the frame, while the unobservable description changes one latent property (viscosity, surface condition, elasticity, or mass). Four video generators produce one clip per prompt, giving 439 videos. Human annotators separately rate physical plausibility, task completion, and hidden-property adherence. An automated evaluator combines first-five-second motion measurements (frame differences, optical flow, camera motion, flow-direction entropy, DINOv2 embedding change), point tracking, and targeted VLM questions about specific physics errors and property adherence. The evaluator is fitted on human labels with cross-validation, keeping each scenario out of its training fold.

The benchmark has 110 prompts (80 observable, 30 unobservable) and 439 generated clips. On the 119 unobservable clips, 47 were rated plausible (P≥3), but 34 of those 47 (72%) did not follow the stated property (H≤2); MAGI accounted for 21 of the 34, and excluding MAGI still left 13 of 26 plausible clips (50%) ungrounded. Conversely, 15 of 28 clips that did follow the property were rated physically implausible. Across matched pairs, stating a latent property lowered mean plausibility by 0.13 to 0.38 points depending on generator; pooled decrease was 0.28 with p=0.085, not statistically significant. The full automated evaluator reached AUC 0.72 for overall plausibility, 0.81 for task completion, 0.73 for violation detection, and 0.71 for hidden-property adherence. VLM-only plausibility AUC was 0.53, while VLM plus motion signals reached 0.61. Hidden-property judgment remained weak: the best VLM scored 0.57 AUC and signals-only scored 0.69. Violation attribution across physics families was 0.53–0.58 except fluids at 0.82; 66% of violation videos contained multiple violation types.

The authors state the study is limited to 80 initial frames, four video generators, and one generation per prompt/model, so seed variability is not measured. The automated evaluator cannot yet reliably determine whether a specific hidden property was realized, and precise violation attribution remains weak. Human labels are not time-localized. The benchmark's 30 matched unobservable scenarios cover only four property types, and computational cost is not reported in the paper.