StreamEval Opens the First Benchmark Testing Video AI Mid-Session
StreamEval scores interactive streaming video models across five tasks, with incremental inputs, 15 to 300 second sessions, and an open harness.
Takeaways
- StreamEval is an open benchmark for interactive streaming video generation across five distinct tasks.
- Sessions run 15 to 300 seconds with inputs arriving incrementally to test context and instruction-following.
- Five tasks: digital-human dialogue, avatar performance, world generation, instruction editing, reference editing.
- Metrics are 0 to 100, per-task; tasks are never merged into a single leaderboard number.
- Dataset is 3.1 GB on Hugging Face and ModelScope; Apache 2.0 code, CC BY 4.0 data.
- Models plug in via a WebSocket protocol; GitHub repo ships a CPU smoke test.
Most video-generation benchmarks score a finished clip against one prompt, leaving them unable to measure systems that receive instructions while rendering. StreamEval targets those systems with five interactive tasks lasting 15 to 300 seconds. Its open release includes benchmark data, a live leaderboard, and a reproducible evaluation harness.
Five workflows, no catch-all rank
During each session, text, speech, images, source video, or camera controls arrive incrementally along a timeline. The harness records generated output and timestamps, then sends the completed session to an automated judge. Scores remain separate by task, allowing developers to distinguish avatar, world-generation, and editing performance.
| Task | Inputs | Metrics |
|---|---|---|
| Digital-human dialogue | Portrait, persona, and user speech or text turns | Quality, A-V Sync, Stability, Dialogue, Identity |
| Avatar performance | Portrait, speech to perform, and live directions | Quality, A-V Sync, Stability, Following, Identity |
| World generation | Opening prompt or image, live prompts, and camera controls | Quality, Stability, Following, Continuity |
| Instruction editing | Live source video and text editing instructions | Quality, Stability, Following, Edit Success, Preservation |