Skip to main content
Back

F5-TTS Clones Any Voice in Seconds and Runs 25x Faster Than Real Time

F5-TTS clones any voice from 5 seconds of audio using flow matching, no phoneme alignment needed, fully open-source and MIT licensed

  • Zero-shot voice cloning: F5-TTS clones any voice from 1-15 seconds of reference audio with no fine-tuning required.
  • No phoneme alignment needed: replaces duration models, text encoders, and aligners with a simple text-infilling approach using filler tokens.
  • Fast inference: achieves RTF of 0.15 out of the box; drops to 0.039 with TensorRT-LLM on an L20 GPU.
  • Fully open-source: MIT code license, pip-installable, supports NVIDIA/AMD/Apple Silicon, with a Gradio UI and Docker image.
  • Active research base: used as the foundation for RL-enhanced F5R-TTS (29.5% WER reduction) and cross-lingual voice cloning extensions.
  • Limitations: zero-shot output can flatten prosody and expressiveness; cross-lingual duration estimation breaks down; model weights are CC-BY-NC licensed.

F5-TTS has quietly become one of the most-starred open-source TTS projects on GitHub, sitting at nearly 15,000 stars and 2,100 forks. The pitch is simple: give it a few seconds of someone's voice, and it generates new speech in that voice from any text you provide. No speaker enrollment pipeline, no fine-tuning required. It just works, out of the box, for free.

The problem it actually solves

Most production-grade voice cloning systems before F5-TTS relied on a stack of specialized components: a phoneme aligner to map text to sounds, a duration predictor to control timing, and a separate text encoder. Each of these adds complexity, training cost, and failure modes. F5-TTS throws all of that out.

F5-TTS is a fully non-autoregressive text-to-speech system based on flow matching with a Diffusion Transformer. Without requiring complex designs such as a duration model, text encoder, or phoneme alignment, the text input is simply padded with filler tokens to the same length as the input speech, and denoising is performed for speech generation. The idea was originally shown to be feasible by E2 TTS, but the original E2 TTS design made it hard to follow due to slow convergence and low robustness. F5-TTS addresses these issues by modeling the input with ConvNeXt to refine the text representation, making it easier to align with speech.

How it works under the hood

Flow matching is the core generative mechanism. Think of it as a cleaner, more direct version of diffusion: instead of adding and removing noise through hundreds of steps, flow matching learns a smooth path from random noise to real speech, guided by an ordinary differential equation (ODE) solver. The model learns to follow this path conditioned on both the reference audio and the target text.

The architecture has three key pieces working together:

  • Diffusion Transformer (DiT) backbone: F5-TTS uses a latent Diffusion Transformer with zero-initialized adaptive Layer Norm (adaLN-zero), and also leverages ConvNeXt V2 blocks to enhance the model's alignment ability.
  • Text-as-filler training: The input text is converted to a character sequence, padded with filler tokens to the same length as input speech, and refined by ConvNeXt V2 blocks before concatenation with the speech input. This lets the model jointly learn semantic and acoustic alignment without any explicit supervision.