Baidu's PaddleSpeech Adds Whisper large-v3-turbo and Real-Time Multilingual Streaming ASR
Baidu's PaddleSpeech adds Whisper large-v3-turbo support and a real-time code-switching ASR model to its open-source speech toolkit
Takeaways
- Whisper large-v3 and turbo added: PaddleSpeech now supports OpenAI's Whisper large-v3 and the faster turbo variant in its ASR pipeline.
- Code-switch online ASR: A new streaming model handles speakers who mix two languages mid-sentence, with a server demo included.
- Turbo = 4 decoder layers: Whisper large-v3-turbo cuts decoding layers from 32 to 4, delivering much faster inference with minimal accuracy loss.
- r1.5.0 upgraded to Paddle 3.0: The latest release migrated the full toolkit from Paddle 2.5.1 to Paddle 3.0-beta, covering 80+ models.
- Full speech toolkit: PaddleSpeech covers ASR, TTS, speaker verification, diarization, translation, and keyword spotting under one Apache 2.0 repo.
- Free to use: Available via
pip install paddlespeechwith CLI, server, and streaming server modes out of the box.
PaddleSpeech, Baidu's open-source speech toolkit built on the PaddlePaddle deep learning framework, has logged two notable additions to its develop branch: support for OpenAI's Whisper large-v3 and turbo models, and a new code-switch online ASR model with a server demo. The project, which won the NAACL 2022 Best Demo Award, now covers the full spectrum of production speech tasks from streaming transcription to voice cloning , and these updates push its ASR capabilities meaningfully forward.
Two additions that matter
The changelog tells the story clearly. PaddleSpeech added Whisper large-v3 and turbo model support. Shortly before that, a code-switch online model and server demo were added. Together, these two additions address two very different but equally real problems in production ASR: raw transcription quality at scale, and handling multilingual speakers in real time.
What Whisper large-v3-turbo actually is
Whisper large-v3-turbo is a finetuned version of a pruned Whisper large-v3 , the exact same model, except the number of decoding layers has been reduced from 32 to 4. As a result, the model is way faster, at the expense of a minor quality degradation. Think of it as the large model's encoder doing all the heavy lifting, with a much leaner decoder that doesn't slow things down.
This work is inspired by Distil-Whisper, where the authors observed that using a smaller decoder can greatly improve transcription speed while causing minimal degradation in accuracy. Unlike Distil-Whisper, which used distillation to train a smaller model, Whisper turbo was fine-tuned for two more epochs over the same multilingual transcription data used for training large-v3.