Skip to main content
Back

Sesame Unveils the Most Human-Like AI Speech Model

TRY NOW

March 3, 2025 5 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 5 min 52 sec

🎖️ Top News

⚡️ Trending Signals

📌  AskUI


📝  Top Tutorials

🧠  How To

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Conversational AI

Sesame unveils CSM, a near-human speech model that adapts tone and pace, outperforming traditional models

⇧ 16,495 Likes

What's New

Oculus co-founder Brendan Iribe’s new startup Sesame introduces its Conversational Speech Model (CSM), designed for generating context-aware speech in real-time conversations. The model integrates emotional intelligence and adjusts tone based on conversation dynamics, creating a more natural dialogue flow. CSM outperforms traditional models with significant improvements in conversational responsiveness and emotional context handling.


Key Features

CSM excels in real-time conversational adjustments.

  • Adjusts tone, pace, and rhythm based on conversational context and emotions.

  • Processes entire conversation history, improving coherence and flow.

  • Implements RVQ-based tokenizers for precise speech generation.

Technical Details

It uses two autoregressive transformers to process text and speech together.

  • First transformer models zeroth codebook from interleaved text and audio inputs.

  • Second transformer reconstructs remaining codebooks for complete speech output.

  • A decoder reconstructs high-fidelity speech from Residual Vector Quantization (RVQ) tokens.

  • Single-stage processing eliminates post-generation steps, improving real-time inference speed.

  • 2048-token sequence length allows handling of longer conversational inputs.

Training Data and Model Sizes
CSM was trained on a large English speech dataset and offers multiple model sizes.

  • 1 million hours of publicly available transcribed and diarized speech used for training.

  • Three model sizes: 1B (Tiny), 3B (Small), 8B (Medium) backbone parameters.

  • Training spanned five epochs, using 2048-token sequences (~2 minutes of audio per sample).

Benchmark Performance and Evaluation
CSM achieves near-human naturalness but still lags in full conversational prosody.

  • In Comparative Mean Opinion Score (CMOS) tests, listeners could not consistently distinguish CSM-generated speech from real recordings.

  • With 90 seconds of conversational context, human speech was preferred over CSM-generated output.

  • Evaluated on word error rate, homograph disambiguation, and pronunciation consistency.


Sesame plans to open-source key components under Apache 2.0 license in future, for now demo is available for everyone to try.

Community Feedback

Kyle Balmer

"Just gave it a shot and you know what impressed me? When it didn’t know something it just said so Instead of hallucinating in something reasonable sounding Straight up “oh I’m not sure” So refreshing"

Rowan Zeller

"Impressive demo. Worth noting, it has memory -- call it twice and it remembers what was talked about in the last call. Personalization and memory are so critical for voice models to feel "human", and it feels like we're only at the beginning here."


Xander Dunn

"I tried Sesame and I felt like I was talking to a pushy salesperson, or a first date who was trying too hard to impress me. Still doesn't handle thinking silence well: "The name of the movie was... <thinking, silence, please don't start talking, omg it's talking>""

Try now

TRENDING SIGNALS

AI Infrastructure

DeepSeek achieves 545% profit margin on inference costs with dynamic resource allocation, while charging 25x less than OpenAI

⇧ 20,583 Likes

AI Game Development

Electronic Arts releases four Command & Conquer games as open-source and introduces a support pack for modding and preservation

⇧ 15,385 Likes

AI Tool

Perplexity unveils GPT-4.5 for Pro users, balancing demand and computational resources with 10 daily questions

⇧ 2,395 Likes

AI Optimization

Anthropic introduces token-efficient tool use for Claude 3.7 Sonnet, reducing token usage by 14% on average, cutting costs and latency

⇧ 1,469 Likes

Agents

LangChain publishes a lightweight Python library for multi-agent swarms with streaming, memory, and human-in-the-loop support

⇧ 785 Like

Build AI Agents that see and interact with any computer interface

AskUI automates testing and processes for native apps on Windows, Linux, MacOS, Android, and iOS. Use natural language prompts in Python to build and run agents across any application.


The latest update includes step-by-step guidance, templates, and a streamlined installer for faster setup.

Start automating now ↗️

TOP LECTURES

Coding Assistance

Build Apps with Windsurf's AI Coding Agents

⇧ 1,593 Likes

In this course by AndrewNg's Deeplearning.ai, learn how AI coding agents track intent, integrate tools, and manage context to improve coding efficiency. Use Windsurf to analyze large codebases, refactor outdated code, and build applications. Implement multi-step code search, automate edits, and optimize workflows. Apply AI-driven UI generation and caching techniques to enhance performance.

Agents

Build a multi-agent best flight finder app

⇧ 2,359 Likes

Build a local multi-agent system to find the best flights with CrewAI, Browserbase, and DeepSeek-R1. Parse queries, extract flight data, and summarize results automatically. Learn to integrate a headless browser for human-like interactions and automate complex workflows. Use a structured agent workflow for orchestration.

AI Evaluation

Trace & Evaluate your Agent

⇧ 210 Likes

Trace and evaluate your AI agent with Arize Phoenix, an open-source observability platform. Track decision-making, optimize tool usage, and measure performance. Implement OpenTelemetry for tracing, analyze execution spans, and use GPT-4o to evaluate response relevance. Improve accuracy, detect hallucinations, and refine outputs using pre-built evaluation templates. Run local or cloud-based monitoring for real-time insights.

HOW TO

Coding Assistance

Customize AI behavior in Cursor

⇧ 999 Likes

Using rules in Cursor, you can control the behavior of the underlying model. You can think of it as instructions and/or a system prompt for LLMs. Rules can be applied globally or to specific files and folders using pattern matching.

Follow these steps to create and apply them:

Step 1

Press Cmd + Shift + P and search for New Cursor Rule.


Step 2: Define Rule Behaviour

Name the rule and specify its behavior using semantic descriptions.


Step 3: Set File Matching Patterns

Use glob patterns (e.g., **/*.tsx) to apply rules to specific files or folders.


Step 4: (Optional)

Use @file to include them as context when the rule is applied.


Step 5

Rules are stored in '.cursor/rules', ensuring they are version-controlled.

Learn more

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us