Skip to main content
Back

Alibaba's Qwen-Audio-3.1-Realtime Cuts Background Speech Responses by 60%

Alibaba's new voice model raises task success to 82% and cuts background-speech interruptions from 73% to 13% through a Think-Act-Speak framework.

  • Qwen-Audio-3.1-Realtime organizes voice interaction around Think, Act, and Speak-and-Coordinate layers.
  • Task success on half-duplex τ-Voice rises from 78.4% to 82.0% versus version 3.0.
  • Background-speech response rate drops from 73% to 13% on Full-Duplex-Bench v1.5.
  • Trained via M²-OPD on-policy distillation plus GRPO in self-evolving executable environments with three validation gates.
  • Available through Alibaba Cloud Model Studio; runtime open-sourced as qwen-audio-agent.
  • Chinese multi-turn attack success drops from 80.5% to 26.0%; FLEURS WER falls from 9.01 to 3.98.

Real-time voice assistants often interrupt background speakers, invent tool calls, or claim success before a task completes. Alibaba’s Tongyi team has released Qwen-Audio-3.1-Realtime, a follow-up to its 3.0 speech model that organizes voice interaction around three layers: Think, Act, and Speak-and-Coordinate. The release focuses on reliability during noisy, evolving, multi-turn conversations.

Full-duplex systems listen while producing speech, which allows interruption and fluid turn-taking. It also forces the model to decide whether incoming audio addresses the assistant, whether a tool should run, and whether execution has finished. Alibaba’s Model Studio docs position the realtime line for assistants, customer service, and AI companions, with WebSocket, WebRTC, and the proprietary AOQ protocol for unstable networks.

Release at a glance
Area Details
Primary goals Reasoning, grounded tool execution, turn-taking, interruption handling, and refusal
Delivery Hosted through Alibaba Cloud Model Studio
Interaction modes Acoustic voice activity detection, semantic turn detection, and manual push-to-talk
Interfaces WebSocket, WebRTC, and AOQ
Released artifacts Hosted model and paper; no training code or in-house evaluation sets

A control problem hiding inside voice

Consider a driver who requests a route, adds constraints, asks the assistant to wait, and changes the destination while a query is running. The agent must preserve the evolving goal, execute tools against live feedback, recognize who is speaking, and choose an appropriate moment to respond. Response time accounts for only one part of that sequence.

Qwen-Audio-3.0 already supported streaming speech interaction. Version 3.1 targets the deployment failures left exposed by that capability: reacting to nearby conversations, accepting unsupported requests, generating inaccurate tool arguments, and speaking before an action reaches a verified state.

Diagram showing the Listen, Think, Act, and Speak interaction loop
The model separates reasoning, action execution, and conversational timing within a continuous interaction loop.

Think: distillation from two directions

Foundation training uses Multimodality and Multi-Teacher On-Policy Distillation, abbreviated M2-OPD. During on-policy distillation, the student generates its own attempts and frozen teachers evaluate those trajectories token by token. A text teacher guides reasoning and instruction-following, while an audio reference supplies information tied to emotion, prosody, and acoustic setting. Thinking Machines Lab provides a useful