Tencent's MMAE Benchmark Exposes AI Audio Editing at Under 5% Accuracy
MMAE exposes a brutal truth: current AI audio editors score below 5% on exact edits, revealing a massive gap between generation and true editing
Takeaways
- MMAE is the first comprehensive benchmark for instruction-based audio editing, released by Tencent Hunyuan and academic collaborators. [Paper]
- Current AI audio editors score below 5% Exact Match Rate -- meaning they almost never make the right edit without breaking something else.
- The benchmark contains 2,000 real-world audio samples and 17,741 fine-grained rubric criteria across 7 modalities and 6 complexity levels.
- Evaluation uses Qwen3-Omni as a judge model, answering multiple-choice questions about input/output audio pairs rather than comparing to ground-truth files.
- Tasks range from simple removals to multi-hop reasoning ("delete the sentence before each laugh") and multi-round editing across conversation turns.
- Dataset, evaluation code, and audio files are fully open-source on GitHub and HuggingFace.
AI can generate audio from scratch with impressive fidelity. But can it actually edit existing audio -- changing only what you ask while leaving everything else untouched? A new benchmark from Tencent Hunyuan and collaborators across SJTU, NTU, PKU, FDU, and others says the answer is: barely. MMAE (Massive Multitask Audio Editing Benchmark) is the first comprehensive evaluation testbed for instruction-based audio editing, and its headline finding is damning -- current models achieve an Exact Match Rate (EMR) below 5%.
Generation is not editing
The distinction matters enormously in practice. Audio generation means producing something new from a text prompt. Audio editing means taking a real clip -- a podcast recording, a music track, a sound effect reel -- and making a targeted modification based on a natural language instruction, while preserving everything else. Think: "Remove the background music but keep the speech," or "Replace the word 'cold' with emphasis on it," or "Extract only the guitar from this mix."
Despite progress in the field, there has been no unified standard for evaluating audio editing tasks, and the absence of a widely accepted open-source benchmark has led to fragmented evaluation protocols and a lack of reliable objective metrics. The most commonly used metric has been the CLAP Score, which measures similarity between the edited audio and the target prompt in an embedding space. That proxy metric tells you almost nothing about whether untouched regions were actually preserved -- which is the whole point of editing.
What MMAE actually tests
MMAE is the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing, extending to a broad spectrum of real-world scenarios across 7 distinct audio modalities including sound, speech, music, and their mixtures. The benchmark was built through a systematic five-stage pipeline: brainstorming, taxonomy construction, instruction-centric data collection, rubric annotation, and quality inspection -- all done through human-agent collaboration.

The scale and structure of the benchmark is what sets it apart:
- 2,000 high-fidelity audio samples drawn from real-world scenarios
- 17,741 fine-grained rubric evaluation items decomposed from free-form tasks into verifiable criteria
- 7 modality settings: pure sound, speech, music, and four mixture types
- 6 complexity levels: from single basic modifications up to multi-hop reasoning (e.g., "delete the sentence before each laugh") and multi-round editing (e.g., "add guitar in round 1, then remove all guitar except what you just added in round 2")
- 8 operation types: addition, removal, replacement, extraction, alteration, foreground change, background change, and reordering
- 2 granularity levels: local (targeting a specific segment or event) and global (affecting the whole clip)
The rubric-based evaluation engine
The most technically interesting contribution is how MMAE scores outputs. Rather than comparing against a ground-truth audio file (which is expensive to produce and often ambiguous), MMAE decomposes free-form tasks into 17,741 verifiable criteria, enabling a precise, multi-dimensional assessment of both instruction following and context consistency. Each rubric item is a multiple-choice question answered by an audio judge model.
The judge is Qwen3-Omni, an omnimodal LLM that listens to both the input and output audio and answers questions like: "In the output clip from 0s to 3s, can the English commentary still be clearly heard?" or "Compare the output and input: which one has a more tense background music?" Three independent judge responses are collected per rubric item, and the majority vote determines the score. This produces three metrics per sample:
- Instruction Following Rate (IFR): did the model make the requested change?
- Consistency Rate (CR): did the model leave untouched regions intact?
- Exact Match Rate (EMR): did it nail both simultaneously?
EMR is the brutal one. Getting both right -- making the right change AND not breaking anything else -- is where current models collapse. No tested model crossed 5% EMR. That is not a rounding error; it is a fundamental capability gap.
Why this is so hard
The interaction between textual instructions and source audio is limited to cross-attention in most generation models. In complex audio scenes, accurately editing a specific event requires not only interpreting the instruction but also locating the event within the source audio -- a capability that text-only semantic encoders inherently lack, making such decoupled designs inadequate for context-aware understanding of complex editing instructions.
The dataset samples make the challenge concrete. Consider a multi-hop task: "Delete the sentence immediately preceding each instance of laughter." The model must first locate all laughter events, then find the sentence before each one, then surgically remove those sentences while leaving the laughter and everything else untouched. Or a multi-round task: "Add guitar to the background during speech [Round 1], then remove all guitar sounds except those added in Round 1 [Round 2]." That requires the model to track its own prior edits across conversation turns.
Running the benchmark yourself
The full benchmark is open. The dataset is available on HuggingFace with 2,000 rows totaling 5.55 GB of audio. The evaluation pipeline is on GitHub and requires deploying Qwen3-Omni via vLLM. The workflow is straightforward:
- Deploy two Qwen3-Omni instances (tensor-parallel=4 each) on 8 GPUs
- Run your audio editing model on the
MMAE-meta.jsoninputs and save output WAV paths - Run the scorer with
python -m eval.score --predictions your_predictions.json
The scorer outputs three files: results.jsonl (per-rubric judge responses), per_sample.json (IFR, CR, EMR per sample), and taxonomy.json (scores broken down by modality, complexity, and operation type). That last file is where the diagnostic value lives -- it tells you exactly which operation types or modality combinations your model struggles with.
What this unlocks
The practical use cases for reliable audio editing are significant: post-production for video and podcasts, accessibility tools that can reformat audio on demand, game audio engines that respond to player instructions, voice assistant pipelines that can surgically modify recorded speech. All of these require the same core capability MMAE is testing -- targeted modification without collateral damage.
The benchmark also fills a gap that has been quietly acknowledged across the research community. The absence of an open and unified benchmark has exacerbated the challenge, leading to inconsistent evaluation and limited reproducibility across different studies. MMAE gives the field a shared yardstick. A model that scores well here is not just generating plausible-sounding audio -- it is demonstrating genuine understanding of audio structure and the ability to act on it precisely. That is a meaningfully higher bar, and right now, no model is anywhere close to clearing it.