Skip to main content
Back

PhyAVBench Exposes How Sora 2 and Veo 3.1 Fail Basic Physics

A new benchmark exposes how even Sora 2 and Veo 3.1 fail at generating physically plausible sounds, scoring under 0.46 on audio-physics tests.

  • PhyAVBench is the first benchmark testing whether audio-video models understand acoustic physics, not just synchronization. Paper
  • Ships with PhyAV-Sound-11K: 25.5 hours, 11,605 newly recorded videos from 184 participants, zero data leakage.
  • 337 paired-prompt groups isolate single physical variables across 6 dimensions and 41 fine-grained test points.
  • New CPRS metric scores directional and magnitude alignment of generated audio against ground-truth physical transitions.
  • Sora 2 tops T2AV at CPRS 0.4512; Veo 3.1 scores 0.3563; all models fail at fluid dynamics and propagation.
  • CPRS correlates 0.92 with human ratings. Code at github.com/imxtx/PhyAVBench.

Current audio-video generators often time a generic impact sound accurately while missing the acoustic differences between wood and metal, or between empty and water-filled bottles. These errors matter in generated Foley, simulation, embodied AI, and synthetic training data, where material, force, distance, and environment should produce predictable changes in sound.

PhyAVBench measures that physical sensitivity with controlled prompt pairs, a newly recorded dataset, and a metric that compares generated acoustic changes with real recordings. Its authors describe it as the first systematic benchmark focused on physical behavior in joint audio-video generation.

A synced impact can still sound wrong

Systems including Sora 2, Veo 3.1, Kling v2.6, Wan 2.6, and Seedance 1.5 Pro can generate visual and audio tracks together. Existing evaluation suites such as TAVGBench, JavisBench, and VABench primarily measure timing and semantic consistency: whether an impact sound occurs when a hammer lands, for example, and whether the sound belongs to the expected category.

Synchronization and semantic metrics can reward a generic impact even when its timbre, resonance, loudness, or decay conflicts with the depicted material and force. Benchmarks assembled from datasets such as AudioSet also contain uncontrolled recording conditions and may overlap with model training corpora.

PhyAVBench addresses those gaps by changing one physical factor at a time. A model receives closely matched conditions, such as striking wood and metal with the same action, and its audio should change in the same direction as the corresponding real recordings.

11,605 recordings isolate physical variables

The accompanying PhyAV-Sound-11K dataset contains 11,605 newly recorded videos totaling 25.5 hours. Its 184 participants captured 337 paired-prompt groups covering six physics dimensions and 41 detailed test points. Each pair changes one target variable while holding the remaining conditions as constant as practical.