Skip to main content
Back

Google’s Gemini 2.5 Pro leads coding + reasoning benchmarks

TRY NOW

June 6, 2025 4 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 4 min 18 sec

🎖️ Top News

› Google upgrades Gemini 2.5 Pro, leads LMArena and WebDevArena.

7,394 Likes

📌 AugmentCode

› Automate code cleanup and testing with Augment’s Remote Agents.

⚡️ Trending Signals

› ElevenLabs releases a new TTS model focused on expressiveness, lets you control it through tags in prompts.

11,384 Likes

› Anthropic updates Claude Projects to access 10x more knowledge using automatic RAG-based retrieval.

4,795 Likes

› Mistral unveils new coding assistant for enterprise with support for completion, retrieval, and agentic tasks.

2,048 Likes

› Perplexity adds SEC data search feature, allowing you to compare company filings.

6,573 Likes

› Anthropic launches custom models for U.S. security agencies, under its Claude Gov project.

1,946 Likes

  📄 Trending Papers

› DeepMind and NVIDIA researchers measure memorization in LLMs.

5,028 Likes

› Meta researchers introduce an open-source safety infrastructure for AI agents. 

482 Likes

› Stanford researcher shows how SFT impacts knowledge accuracy and reasoning quality.

466 Likes

🧠 Top Lecture

› Automate Prompt Tuning and Build Modular GenAI Apps with DSPy.


If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Gemini Update

Google releases Gemini 2.5 Pro preview, lets you manage costs with thinking budgets

7,394 Likes

Google has released a preview update of Gemini 2.5 Pro, which leads the LM Arena leaderboard with a score of 1470. Its score is up 24 points from the version released during Google IO last month. It adds a new “thinking budget” feature. 


Benchmark Improvements

  • 21.6% on Humanity’s Last Exam (HLE), highest among all listed models beating OpenAI's o3

  • 86.4% on GPQA Diamond, benchmark for Science, above o3 and Claude Opus 4


  • 88.0% on AIME 2025 math benchmark beating DeepSeek R1


New “thinking budget”

  • You can now control response latency and compute use via a thinking budget parameter.

  • Adjusts token-level reasoning steps before responses

Access and Availability

  • Gemini 2.5 Pro (06-05) now in preview, with general availability coming later

  • Available via the Gemini app, AI Studio, and Vertex AI

  • The updated model keeps the same pricing as previous versions

Community Feedback

xlr8harder

"Google's Gemini 2.5 Pro Preview 06 05 has improved substantially on SpeechMap's free speech evaluation, up to 89%, nearly a 20% increase from the previous 2.5 Pro Preview, and well above average for US labs. Great job, Google. I hope this is your new standard."

Laurence Bremner
"Wow..those are impressive benchmarks! 21.6% on HLE and 86.4% on GPQA?? And already in the API too?"

Logan Kilpatrick
"building models is a juggling act, we optimized 05-06 for certain code tasks at the cost of other tasks, good lessons learned that the small regressions we saw in evals tracked to how people felt about the model, but we closed the gaps with 06-05!"

Try now

AI Coding Agents Can Now Clear Your Backlog Automatically

Every team has a graveyard of “we’ll get to it” tickets: flaky tests, outdated docs, low-impact bugs, tiny refactors.


Augment's Remote Agents pick up the slack. They can:

  • Fix doc rot: update READMEs, code comments, API examples

  • Refactor safely: isolate legacy code, preserve green tests

  • Migrate configs in bulk without locking your IDE

  • Boost test coverage with real unit and integration tests

This isn’t a generic LLM. Augment loads your full codebase: millions of lines, deep dependencies, team patterns, so every suggestion fits your stack.

Log off. Come back to ready-to-merge PRs.

Try for free

partner with us

TRENDING PAPERS

LLM Memory

New Study Estimates How Much LLMs Memorize

5,028 Likes

The paper introduces a way to measure how much a language model memorizes specific data by removing its ability to generalize. It finds GPT-style models store about 3.6 bits per parameter. 

AI Agent Security

Secure your AI agents with Meta’s Open-Source Framework

482 Likes

LlamaFirewall is an open-source tool by Meta that protects AI agents from security risks. It blocks jailbreaks, checks agent reasoning, and scans for unsafe code, helping you enforce safety rules in real time.

LLM Reasoning

Stanford Study shows SFT Boosts Answers but Weakens Reasoning

466 Likes

This paper by Stanford researchers introduces a framework to separate and evaluate knowledge recall and reasoning logic in LLMs. It shows SFT can hurt reasoning quality, while RL improves medical reasoning by filtering poor knowledge.

TOP LECTURE

AI Agents

Automate Prompt Tuning and Build Modular GenAI Apps with DSPy

4,939 Likes

DSPy is an open-source tool that helps you build and improve AI apps using simple, modular blocks. In this course, you’ll learn to build with DSPy, debug with MLflow, and auto-tune prompts with DSPy Optimizer.


You’ll learn:

  • How to compose agents using modules like Predict, ChainOfThought, and React

  • How to trace and debug programs using MLflow

  • How to tune prompts with DSPy Optimizer

  • How to optimize model performance when switching LLMs

  • How to build applications like a sentiment classifier, a celebrity guessing game, and a Wikipedia RAG agent

Start now

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us