MOSI.AI's MOSS-Audio Beats 30B Models With an 8B Open-Source Audio AI
MOSS-Audio is an open-source 4B/8B model that unifies speech, sound, and music understanding with native timestamp awareness, outperforming 30B models on audio reasoning benchmarks.
Takeaways
- MOSS-Audio is an open-source unified audio-language model (4B and 8B) handling speech, environmental sound, and music understanding in one autoregressive framework.
- 8B Thinking variant scores 71.08 average on MMAU/MMAU-Pro/MMAR/MMSU benchmarks, beating the 30B Qwen3-Omni on general audio reasoning.
- Timestamp ASR is a standout: 35.77ms average error on AISHELL-1 vs. 833ms for Qwen3-Omni and 708ms for Gemini-3.1-Pro.
- DeepStack cross-layer injection feeds intermediate encoder features into early LLM layers, preserving prosody, timbre, and transient events lost by top-layer-only approaches.
- Event-preserving data pipeline segments audio at natural acoustic boundaries and routes clips through speech, music, and general audio annotation branches before merging into unified captions.
- All four models are available on Hugging Face under Apache 2.0, with fine-tuning and SGLang serving support on GitHub.
Audio AI has long been a patchwork of specialized systems: one model for transcription, another for speaker identification, yet another for sound event detection. MOSS-Audio, from the OpenMOSS team at MOSI.AI and Shanghai Innovation Institute, takes a different bet -- a single autoregressive model that handles all of it, and does so with explicit awareness of when things happen in the audio stream.
The Problem with Specialized Pipelines
Conventional audio analysis requires stitching together disparate systems for transcription, emotion detection, and scene analysis, which often leads to synchronization errors and high computational overhead. Beyond the engineering pain, there is a deeper issue: the encoder produces 12.5 Hz temporal representations that most off-the-shelf speech frontends simply discard. ASR-trained encoders are optimized for mapping acoustics to words -- they throw away prosody, timbre, transient events, and musical structure. Building a voice agent on top of that is building on a narrow foundation.
The temporal problem is equally thorny. Most audio-language models have no native sense of when something happened. They can tell you a dog barked, but not at second 4.2. Time is a critical dimension in audio understanding. To enhance explicit temporal awareness, MOSS-Audio adopts a time-marker insertion strategy during pretraining: explicit time tokens are inserted between audio frame representations at fixed time intervals to indicate temporal positions. This design enables the model to learn "what happened when" within a unified text generation framework, naturally supporting timestamp ASR, event localization, time-based QA, and long-audio retrospection.
The Architecture: Three Pieces, Two Key Ideas
MOSS-Audio adopts a modular three-stage design: audio encoder, modality adapter, and language model backbone. Raw audio is encoded into a continuous temporal representation at 12.5 Hz, projected into the LLM embedding space, and then processed through autoregressive text generation. The LLM backbone is Qwen3 (4B or 8B), keeping the model competitive in size while benefiting from a strong language prior.

Two architectural choices separate MOSS-Audio from most audio-language models:
- DeepStack Cross-Layer Feature Injection: Instead of feeding only the encoder's final layer to the LLM, MOSS-Audio extracts intermediate hidden states from multiple encoder depths and injects them into the early layers of the decoder. The intuition is that the top encoder layer is dominated by lexical/semantic content, while lower layers hold prosody, timbre, and transient acoustic events. The DeepStack Cross-Layer Feature Injection module preserves prosody and timbre by injecting intermediate encoder layer features into the LLM's early layers. Ablation studies confirm this: DeepStack consistently improves music, pure sound, and environmental acoustics scores, with only a marginal trade-off on pure speech tasks.