HKUST's AudioX Collapses Music, Foley, and Sound Into One Open Model
HKUST's new open-source Diffusion Transformer unifies text, video, and audio into a single model for anything-to-audio generation, accepted to ICLR 2026.
Takeaways
- HKUST released AudioX, a unified Diffusion Transformer for text, video, and audio-to-audio generation, accepted to ICLR 2026.
- Single model handles text-to-audio, text-to-music, video-to-audio, video-to-music, inpainting, and music completion.
- Core innovation: Multimodal Adaptive Fusion module plus multi-modal masked training for cross-modal alignment.
- Ships with IF-caps, a curated 7-million-sample multimodal audio dataset on Hugging Face.
- Three pretrained checkpoints available: AudioX, AudioX-MAF, and AudioX-MAF-MMDiT with Synchformer video sync.
- Licensed CC-BY-NC with watermarked weights, non-commercial use only; Gradio demo included.
A team from HKUST just open-sourced AudioX, a single Diffusion Transformer that generates sound effects, foley, and music from almost any input you throw at it: text prompts, silent video, reference audio, or combinations of all three. Accepted to ICLR 2026, the project tackles two long-standing problems in audio generation, the lack of a unified multimodal modeling framework and the shortage of large-scale, high-quality training data, by integrating text, video, and audio signals under one architecture built around a Multimodal Adaptive Fusion module.
Rather than stitching together a text-to-audio model, a video-to-music model, and a foley model, AudioX collapses those workflows into one checkpoint. The team also released the underlying 7-million-sample dataset, three pretrained models on Hugging Face, and a Gradio demo you can run locally.
One model, six generation tasks
The framework covers a matrix of tasks depending on which inputs you provide. All are handled by the same conditioning interface, so switching between them is a matter of passing or omitting the relevant tensor.
| Task | Video | Text prompt | Audio |
|---|---|---|---|
| Text-to-Audio | None | "Typing on a keyboard" | None |
| Text-to-Music | None | "A music with piano and violin" | None |
| Video-to-Audio | video.mp4 | "Generate general audio for the video" | None |
| Video-to-Music | video.mp4 | "Generate music for the video" | None |
| Text+Video-to-Audio | video.mp4 | "Ocean waves crashing with people laughing" | None |
| Text+Video-to-Music | video.mp4 | "Generate music with piano instrument" | None |
Beyond these, the demo page shows the same checkpoint doing audio inpainting (filling in masked regions of an existing clip) and music completion (continuing a musical phrase). That versatility is the real headline, since most open-source audio models only do one of these jobs well.