Skip to main content
Back

Google Adds New Gemini Models: Dethrones GPT-4.1 on Video Benchmarks

TRY NOW

May 12, 2025 6 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 6 min 50 sec

🎖️ Top News

📌  Encord


⚡️ Trending Signals

📝  Top Substack Read

🧠  Deep Dive

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

AI Models

Google unveils two new Gemini models with multimodal video understanding and YouTube-to-app workflows

⇧ 7,385 Likes

What's New

Google releases Gemini 2.5 Pro (05-06), a multimodal model with advanced video analysis capabilities. It surpasses GPT-4.1 on standard benchmarks using the same input format. Its ability to process 6 hours of video with 2M token context is the most notable upgrade.

It achieves state-of-the-art results on dense captioning (e.g., YouCook2) and moment retrieval (e.g., QVHighlights). It matches specialized fine-tuned models on these challenging datasets despite being a general-purpose multimodal model.


Benchmark performance

Gemini 2.5 Pro outperforms baseline LLMs and matches specialized video models on key tasks.

  • 84.7% accuracy on VideoMME using low-resolution mode, only 0.5% behind full resolution

  • Beats GPT-4.1 in controlled tests with same video frames and prompts

  • Matches fine-tuned systems on YouCook2 dense captioning and QVHighlights moment retrieval

  • Handles both short and long video benchmarks under consistent evaluation settings


Multimodal Integration

Gemini 2.5 Pro accepts diverse input formats and combines them in a single interaction.

  • Processes YouTube links, video files, and text-based queries via Gemini API or AI Studio

  • Parses audio-visual cues and structured instructions jointly

  • Responds with executable code, structured data, or natural language depending on task

  • Integrates visual content with programming workflows, e.g., p5.js or UI specs

Application generation from video input

Gemini 2.5 Pro turns video analysis into working code for learning or simulation tools.

  • Extracts structured specs from YouTube videos based on instructional prompts

  • Generates complete learning apps and simulators directly from video content

  • Uses video timeline to align generated outputs with visual sequences

  • Supports both Pro and Flash models for this functionality

Temporal reasoning and segment detection

The model handles moment retrieval, object tracking, and counting within long videos.

  • Identifies 16 product segments in a 10-minute keynote using visual and audio signals

  • Counts 17 instances of repeated action across timeline with frame-level precision

  • Breaks video into labeled intervals without manual annotation

  • Uses full context window to reason over entire video duration

Developer access and API capabilities
Gemini 2.5 Pro and Flash are available through multiple Google developer platforms.

  • Available in Google AI Studio, Gemini API, and Vertex AI

  • Supports YouTube integration directly from API

  • “Low” media resolution mode enables ~6 hours of video within a 2M token limit

  • Includes controls for resolution to manage cost-performance trade-offs

Community Feedback


elvis

"Wow. The timing of this is crazy. I was just testing YT video understanding of technical talks on Google AI Studio. I think the token limit on Gemini 2.5 Pro Preview 05-06 limits the length of video supported, but I will check out the API."

Freshski
"Gemini is unbelievably good at multi-modality. Uploaded a residential construction estimate PDF. I kid you not, I was able to cut $4K (10%) off the estimate due to discrepancies that Gemini pointed out."

Ashish Rawat
"The only problem with gemini models that it output too much code and comments, otherwise it's solves the problem other models can't (not even claude 3.7)"

Try now

Want to Build Smarter Physical AI Systems That Adapt in the Real World?

Deploying ML in real-world systems isn’t easy. You deal with noisy sensor data, unpredictable edge cases, and models that don’t generalize well.


“Breakthroughs in generative AI are bringing 3D perception, control, skill planning and intelligence to robots and Physical AI systems.”

– Rev Lebaredian, VP, Omniverse & Simulation, NVIDIA


Join Encord’s ML research team on May 22 to explore how the latest World Models (like GR00T N1 and Dreamer) can be integrated into agentic data workflows—to help you go from simulation to real-world deployment faster.

What You’ll Learn:

  • Multi-step reasoning: Use world models to enable adaptive, goal-directed behavior.

  • Edge-case handling: Supervise and correct failure modes in petabyte-scale sensor data.

  • Clean data pipelines: Align vision-language-action (VLA) datasets for robotics and industrial AI.

This one’s built for ML teams solving real deployment problems.

Save your seat

partner with us →

TRENDING SIGNALS

Generative AI

Tencent launches HunyuanCustom, an open-source video model with subject consistency and multimodal input support

⇧ 1,294 Likes

AI Assistant

Alibaba introduces Web Dev to Qwen Chat, enabling production-ready webpages from text prompts with instant code generation

⇧ 1,837 Likes

AI Assistant

Hugging Face presents Open Computer Agent, a cloud-hosted virtual assistant for full Linux VM automation with smolagents and Qwen 2

⇧ 1,802 Likes

Coding Assistance

GitHub rolls out OpenAI's GPT-4.1 as the default model in GitHub Copilot for better instruction following and code generation

⇧ 581 Likes

AI Architecture

Sakana AI unveils CTM which encodes information through timing patterns, pushing beyond static transformer-based representations

⇧ 2,827 Like

TOP SUBSTACK READS

LLM Systems

Build your Second Brain AI assistant

Breaks down how to build a Notion-like Second Brain AI assistant using agents, RAG, fine-tuned LLMs, and LLMOps. Walks through custom data pipelines, system architecture, and open-source tools to teach you how to create a production-grade assistant that understands your digital resources and avoids hallucinations.

⇧ 856 Likes

AI Tools

Claude Code: Anthropic's Agent in Your Terminal

Explains how Anthropic built Claude Code as a Unix-style AI agent for terminal-based coding. It runs bash commands, reads files, and integrates with tools like MCP for automation. Built for power users, it offers direct model access without a complex UI, aligning with Anthropic’s "do the simple thing first" philosophy.

⇧ 39 Likes

AI Agents

The Open-Source Toolkit for Building AI Agents

Outlines the top open-source tools for building AI agents, covering frameworks, memory, voice, document understanding, and observability. Highlights include CrewAI for multi-agent orchestration, Open Interpreter for code execution, Qwen2-VL for document analysis, and Letta for stateful memory.

⇧ 591 Likes

DEEP DIVE

Prompt Engineering

Unlocking System Prompt Learning for LLM Behavior Beyond Pretraining

⇧ 9,294 Likes

This lecture highlighted by Andrej Karpathy introduces system prompt learning—a proposed third paradigm for LLM behavior learning, alongside pretraining and fine-tuning. Unlike parameter updates, this approach involves persistent, editable, textual instructions inside the system prompt.

It helps LLMs accumulate general problem-solving strategies through self-generated guidance, similar to how humans “write mental notes” after solving a task.

You will learn:

  • The distinction between knowledge, habit, and instruction layers in LLMs

  • Structure and components of Claude's 17k-word system prompt

  • Role of MCP servers and tool invocation protocols

  • Differences between hand-written vs. learned system prompts

  • How Anthropic encodes behavior fixes as XML-like hotpatches

  • Limitations of finetuning for encoding explicit reasoning steps

  • Open questions in editable system prompt training (e.g., edit algorithms, long-term retention, hybridization with weights)

  • Implications for future LLM memory architectures and agent design

Read more

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us