Skip to main content
Back

HKUST's AudioX Collapses Music, Foley, and Sound Into One Open Model

HKUST's new open-source Diffusion Transformer unifies text, video, and audio into a single model for anything-to-audio generation, accepted to ICLR 2026.

  • HKUST released AudioX, a unified Diffusion Transformer for text, video, and audio-to-audio generation, accepted to ICLR 2026.
  • Single model handles text-to-audio, text-to-music, video-to-audio, video-to-music, inpainting, and music completion.
  • Core innovation: Multimodal Adaptive Fusion module plus multi-modal masked training for cross-modal alignment.
  • Ships with IF-caps, a curated 7-million-sample multimodal audio dataset on Hugging Face.
  • Three pretrained checkpoints available: AudioX, AudioX-MAF, and AudioX-MAF-MMDiT with Synchformer video sync.
  • Licensed CC-BY-NC with watermarked weights, non-commercial use only; Gradio demo included.

A team from HKUST just open-sourced AudioX, a single Diffusion Transformer that generates sound effects, foley, and music from almost any input you throw at it: text prompts, silent video, reference audio, or combinations of all three. Accepted to ICLR 2026, the project tackles two long-standing problems in audio generation, the lack of a unified multimodal modeling framework and the shortage of large-scale, high-quality training data, by integrating text, video, and audio signals under one architecture built around a Multimodal Adaptive Fusion module.

Rather than stitching together a text-to-audio model, a video-to-music model, and a foley model, AudioX collapses those workflows into one checkpoint. The team also released the underlying 7-million-sample dataset, three pretrained models on Hugging Face, and a Gradio demo you can run locally.

One model, six generation tasks

The framework covers a matrix of tasks depending on which inputs you provide. All are handled by the same conditioning interface, so switching between them is a matter of passing or omitting the relevant tensor.

TaskVideoText promptAudio
Text-to-AudioNone"Typing on a keyboard"None
Text-to-MusicNone"A music with piano and violin"None
Video-to-Audiovideo.mp4"Generate general audio for the video"None
Video-to-Musicvideo.mp4"Generate music for the video"None
Text+Video-to-Audiovideo.mp4"Ocean waves crashing with people laughing"None
Text+Video-to-Musicvideo.mp4"Generate music with piano instrument"None

Beyond these, the demo page shows the same checkpoint doing audio inpainting (filling in masked regions of an existing clip) and music completion (continuing a musical phrase). That versatility is the real headline, since most open-source audio models only do one of these jobs well.