Meta's AudioCraft Adds JASCO and MusicGen-Style for Precise Music Control
Meta's open-source audio generation library bundles MusicGen, AudioGen, EnCodec, MAGNeT, JASCO and AudioSeal into one PyTorch toolkit for text-to-music research.
Takeaways
- AudioCraft bundles MusicGen, AudioGen, EnCodec, MAGNeT, MBD, AudioSeal, MusicGen-Style and JASCO in one PyTorch library.
- JASCO adds flow-matching text-to-music with chord, drum and melody timing conditions.
- JASCO ships in 400M and 1B sizes, trained on roughly 16K hours of licensed music.
- MusicGen-Style conditions a 1.5B transformer on a 1.5 to 4.5 second audio reference plus text.
- Code is MIT licensed; model weights are CC-BY-NC 4.0 and restricted to non-commercial use.
- Requires Python 3.9 and PyTorch 2.1.0; MusicGen-Style needs a 16 GB GPU for inference.
Meta’s Fundamental AI Research team has expanded the AudioCraft repository into a PyTorch stack for music generation, sound generation, neural audio compression, diffusion decoding, and watermarking. Its newer JASCO and MusicGen-Style models add time-aligned musical controls and reference-audio conditioning.
Developers can use shared audio processing, configuration, training, and inference infrastructure across several generation architectures. That common foundation reduces the work required to compare models, build new conditioning methods, or train compatible weights.
Eight components, one codebase
The repository groups eight components around common PyTorch and EnCodec tooling:
| Component | Purpose | Key detail |
|---|---|---|
| MusicGen | Text-to-music generation | Autoregressive Transformer available in 300M, 1.5B, and 3.3B parameter sizes |
| AudioGen | Text-to-sound generation | Produces sound effects and environmental audio |
| EnCodec | Neural audio compression | Converts waveforms into compact tokens used by the generators |
| Multi Band Diffusion | Audio decoding | Uses diffusion to decode EnCodec representations |
| MAGNeT | Music and sound generation | Uses non-autoregressive masked modeling to generate multiple tokens in parallel |
| AudioSeal | Audio watermarking | Embeds and detects localized watermarks in generated audio |
| MusicGen-Style | Text-and-style-to-music generation | Conditions generation on text and a short reference clip |
| JASCO | Structurally controlled music generation | Accepts text, timed chords, drum tracks, and melodies |
JASCO puts controls on the timeline
JASCO accepts a global text description alongside time-aligned local conditions. A prompt can specify instrumentation or character, while chord events, drums, and melody define musical structure at particular points in the clip. This design supports experiments that measure whether generated music follows both semantic and temporal instructions.
Its architecture pairs EnCodec tokenization with a Transformer trained through flow matching, a method that learns a continuous path from noise to audio-token representations. Meta publishes 400M and 1B parameter variants with chord and drum controls, plus variants that also accept melody. Training used roughly 16,000 hours of licensed music.
Condition dropout randomly removes inputs during training, allowing inference to omit one or more controls. The Python API can therefore generate a sample from text and a timed chord progression even when the selected checkpoint supports drums and melody:
from audiocraft.data.audio import audio_write
from audiocraft.models import JASCO
model = JASCO.get_pretrained(
"facebook/jasco-chords-drums-melody-1B",
chords_mapping_path="assets/chord_to_index_mapping.pkl",
)
model.set_generation_params(cfg_coef_all=1.5, cfg_coef_txt=0.5)
text = "Strings, woodwind, orchestral, symphony."
chords = [
("C", 0.0),
("D", 2.0),
("F", 4.0),
("Ab", 6.0),
("Bb", 7.0),
("C", 8.0),
]
output = model.generate_music(
descriptions=[text],
chords=chords,
progress=True,
)
audio_write(
"output",
output.cpu().squeeze(0),
model.sample_rate,
strategy="loudness",
loudness_compressor=True,
)
With all controls supplied, the JASCO 1B checkpoint reports a chord Intersection over Union score of 0.690 and a melody cosine similarity of 0.427. Chord IoU measures temporal overlap between requested and generated chord labels, while melody cosine similarity measures alignment between melody representations. Higher values indicate closer adherence on both metrics.
MusicGen-Style learns from a reference clip
MusicGen-Style accepts a reference excerpt lasting 1.5 to 4.5 seconds and extracts features through a style conditioner. A text prompt can then specify the requested content while the reference features guide the musical characteristics of the output.
A 1.5B parameter autoregressive Transformer generates the music token by token, using EnCodec for tokenization and separate conditioners for text and style. Inference requires a GPU with at least 16 GB of memory. The MusicGen-Style guide documents the available checkpoints and API.
One pipeline, four generation strategies
Audio generation experiments often duplicate dataset loaders, resampling logic, tokenization, checkpoint handling, and training loops. AudioCraft centralizes those layers through EnCodec, Hydra configuration, shared audio utilities, and common training infrastructure. Each architecture retains model-specific settings, while the shared layers make comparisons and extensions easier to maintain.
Published training recipes cover EnCodec, MusicGen, Multi Band Diffusion, and JASCO. Researchers can use them for several concrete workflows:
- Architecture comparisons: evaluate autoregressive, masked, diffusion, and flow-matching methods with compatible audio representations.
- Controllability studies: measure adherence to text, chords, drums, melodies, and reference audio.
- Codec research: train EnCodec variants or reuse its tokens in a separate model.
- Provenance experiments: apply AudioSeal to generated clips and test watermark detection or localization.
Deployment constraints arrive early
- JASCO duration: Current checkpoints generate fixed 10-second clips, limiting experiments with long-term musical structure.
- Environment versions: The project targets Python 3.9 and pins PyTorch 2.1.0. Integration with newer environments may require dependency changes.
- xFormers compatibility: The project’s issue tracker includes recurring installation and xFormers reports.
- GPU memory: MusicGen-Style requires at least 16 GB for inference, and larger models may need additional memory depending on generation settings.
Code and weights follow different rules
The repository code uses the MIT license. Published model weights use CC BY-NC 4.0, which permits attributed use and adaptation for noncommercial purposes. Serving commercial generations from those checkpoints requires separate permission or differently licensed weights, and fine-tuning does not remove the original checkpoint terms.
Included training pipelines make new-weight training possible for several models. Commercial use still depends on compatible rights for training data, reference audio, third-party dependencies, and the resulting outputs.
A research stack built to extend
AudioCraft gives generative-audio researchers a common base for comparing model families and testing structured conditioning. JASCO adds time-indexed musical control, while MusicGen-Style adds reference-audio guidance. Fixed clip length, pinned dependencies, GPU requirements, and noncommercial checkpoint licenses currently position the project for research, evaluation, and custom model training.