Skip to main content
Back

Surrey's RFM-Editing Rewrites Audio Clips 10x Faster With Plain-English Instructions

A new diffusion framework edits sounds inside an audio clip from a short natural-language instruction, without full captions or masks.

Paper, Full Access
  • RFM-Editing performs add, remove, and replace audio edits from a short natural-language instruction, no caption or mask needed.
  • Built on rectified flow matching with a LoRA-tuned Flan-T5 encoder and a VAE-U-Net latent pipeline over log-mel spectrograms.
  • Original audio latent is concatenated channel-wise to the noisy latent so the model preserves unedited regions.
  • Trained on a new AudioCaps2-derived dataset of 234,639 overlapping multi-event samples across three task types.
  • Matches best training-free baseline on CLAP alignment (0.4398 vs 0.4579) at roughly 11 seconds versus 102 seconds per clip.
  • Paper on arXiv; no code or weights released yet.

Editing audio with plain-English instructions like remove the drill or replace the dog barking with pigeon wings flapping sounds simple, but doing it well has proven stubbornly hard. A team from the University of Surrey and collaborators has proposed RFM-Editing, a rectified flow matching diffusion framework that performs add, remove, and replace operations directly from a short instruction, with no full target caption and no time-aligned mask.

The system runs an order of magnitude faster than the strongest training-free baseline while matching it on semantic alignment, and it holds up even when the sound to edit overlaps with other events in the clip.

Why audio editing has lagged behind image editing

Text-to-audio generation has moved fast with models like AudioLDM, Make-An-Audio, and TangoFlux, but text-guided audio editing has been stuck between two awkward options. Training-free methods based on DDIM or DDPM inversion, such as Zero-Shot audio editing and AudioEditor, produce high-fidelity results but need the full caption of the source audio, sometimes the indices of the tokens you want to change, and expensive null-text optimization at inference time. In practice, audio clips rarely come with those clean captions.

Training-based methods like AUDIT accept instructions directly, but they struggle to localize the edit region when multiple sounds overlap, and they are limited by the scarcity of paired editing data. RFM-Editing aims squarely at that gap: instruction-only input, no per-sample optimization, and robustness to overlapping events.