Skip to main content
Back

StreamEval Opens the First Benchmark Testing Video AI Mid-Session

StreamEval scores interactive streaming video models across five tasks, with incremental inputs, 15 to 300 second sessions, and an open harness.

  • StreamEval is an open benchmark for interactive streaming video generation across five distinct tasks.
  • Sessions run 15 to 300 seconds with inputs arriving incrementally to test context and instruction-following.
  • Five tasks: digital-human dialogue, avatar performance, world generation, instruction editing, reference editing.
  • Metrics are 0 to 100, per-task; tasks are never merged into a single leaderboard number.
  • Dataset is 3.1 GB on Hugging Face and ModelScope; Apache 2.0 code, CC BY 4.0 data.
  • Models plug in via a WebSocket protocol; GitHub repo ships a CPU smoke test.

Most video-generation benchmarks score a finished clip against one prompt, leaving them unable to measure systems that receive instructions while rendering. StreamEval targets those systems with five interactive tasks lasting 15 to 300 seconds. Its open release includes benchmark data, a live leaderboard, and a reproducible evaluation harness.

Five workflows, no catch-all rank

During each session, text, speech, images, source video, or camera controls arrive incrementally along a timeline. The harness records generated output and timestamps, then sends the completed session to an automated judge. Scores remain separate by task, allowing developers to distinguish avatar, world-generation, and editing performance.

StreamEval tasks, inputs, and scoring dimensions
Task Inputs Metrics
Digital-human dialogue Portrait, persona, and user speech or text turns Quality, A-V Sync, Stability, Dialogue, Identity
Avatar performance Portrait, speech to perform, and live directions Quality, A-V Sync, Stability, Following, Identity
World generation Opening prompt or image, live prompts, and camera controls Quality, Stability, Following, Continuity
Instruction editing Live source video and text editing instructions Quality, Stability, Following, Edit Success, Preservation