Skip to main content
Back

OpenAI Releases First Models to "Think with Images"

TRY NOW

April 18, 2025 5 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 5 min 33 sec

🎖️ Top News

⚡️ Trending Signals

💻  Trending Repos

  • code-server: Run full VS Code in the browser using remote servers to save local resources and boost performance.

  • Stagehand lets you build reliable AI-powered browser workflows using LLMs and Playwright in production.

  • FastMCP v2 allows building MCP servers with minimal code, supporting OpenAPI, LLM sampling, and proxies.

🧠  How To

  • Install OpenAI CLI, set your API key, and run text completions directly from terminal.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

LLMs

OpenAI introduces models that think with images and chain tools to solve complex coding and math tasks

⇧ 63,375 Likes

What's New

OpenAI released two models, o3 and o4-mini, optimized for multi-step reasoning and structured tool use. They run separately from the GPT line and integrate directly with the Chat Completions API. The most notable result: o3 outperforms GPT-4-turbo on long-form reasoning benchmarks.


Key Features

  • Structured Thought: Supports multi-step logic with accurate intermediate state tracking

  • Dynamic Tool Use: Chooses tools and formats arguments without manual prompt engineering

  • Chain-of-Thought Support: Improved outcomes when using explicit reasoning traces

  • Vision + Text Input: Accepts mixed-modality inputs, including screenshots and diagrams

  • Codex Integration: Works with terminal-based tools for generating and editing code, images, and text

Thiking with images

The models analyze diagrams, screenshots, and complex layouts with improved spatial and textual understanding.

  • Chart Interpretation: Extracts insights from graphs, plots, and structured visual data

  • UI Navigation: Understands interface layouts to generate actions or explain steps

  • Math Reasoning: Processes handwritten or typeset equations within images

  • OCR + Context: Combines text extraction with reasoning grounded in surrounding visuals

  • Sequential Vision Tasks: Follows step-by-step visual changes, including slide transitions or tool outputs


Reasoning and performance benchmarks
o3 ranks first on reasoning-heavy benchmarks and performs well on structured tasks.

  • Tops SWE-bench, MMMU, MathVista, Codeforces, and Charxiv-r without scaffolding

  • Outperforms GPT-4-turbo on long-form math, code, and multi-step logic

  • Uses reinforcement learning as the main training method

  • Scales with compute during fine-tuning

  • Delivers more accurate outputs with chain-of-thought prompting

Optimized model

o4-mini targets lower-latency, high-throughput scenarios while retaining reasoning capabilities.

  • Ranks #1 on AIME 2024 and AIME 2025 among small models

  • Best used where GPT-3.5 or Gemini Flash is currently deployed

  • Supports structured reasoning in math and vision tasks

  • Intended for low-cost applications with logic-heavy prompts

Tool use and custom integrations

Both models support dynamic tool use with improved argument selection and context awareness.

  • Integrates with function calling and custom tools through the OpenAI API

  • Determines tool use based on system and user messages

  • Refines arguments more accurately during multi-turn tasks

  • Supports mixed-modality workflows including vision, text, and code

Availability and usage details

You can access both models through ChatGPT and OpenAI APIs starting today.

  • Free-tier users can use o4-mini via the “Think” composer mode

  • Plus, Pro, and Team users see o3 and o4-mini in model selector

  • o4-mini supports higher usage limits and faster responses than o3

  • Responses API adds support for reasoning token preservation and integrated tool calls

Community Feedback


cedric

"Surprisingly, they didn't provide a comparison to Gemini 2.5 Pro and Sonnet 3.7 (thinking). That’s likely because, although o3 and o4-mini are impressive, the improvement is only marginal."

Kelsey Pipe
"o4-mini-high is the first AI to pass my personal secret benchmark for hallucinations and complex reasoning, so I guess now I can tell you all what that benchmark is. It's simple: I post a complex midgame chessboard and 'mate in one'. The chessboard does not have a mate in one."

Simon Willison

"Interesting OpenAI-insider tip on Hacker News: "o4-mini is actually a considerably better vision model than o3, despite the benchmarks"

Try now

TRENDING SIGNALS

AI Agents

Microsoft adds computer use to Copilot Studio, enabling agentic UI automation across web and desktop apps

⇧ 6,503 Likes

Generative AI

Google integrates its SOTA text-to-video generation model, Veo 2 into Gemini API, AI Studio and mobile app 

⇧ 6,429 Likes

VLMs

Meta releases PLM, a fully open vision-language model for detailed video understanding, trained on 2.8M newly human-labeled video QA

⇧ 683 Likes

AI Evaluation

AI2 unveils DataDecide: using small-scale models, it identifies cost-effective benchmarks; MMLU and ARC Easy rank highest

⇧ 828 Likes

Generative AI

ByteDance presents Seaweed-7B, a model that supports real-time, long-form, and high-res video generation from text, images, or audio

⇧ 2,359 Like

TRENDING REPOS

Cloud IDE

code-server

☆ 71,124 Stars

Run VS Code in the browser from any machine. code-server helps you write, test, and debug remotely with full VS Code functionality. Use cloud resources to reduce local CPU and battery usage. Fast startup (<1s), minimal latency. Supports Linux, Docker, and custom cloud deploys.

Agents

stagehand

☆ 10,248 Stars

Use natural language or Playwright to automate browser actions with high control and reliability. Stagehand supports OpenAI, Anthropic, and Gemini 2.5 models. Run agents with one line of code. Preview or cache actions to reduce token cost. Extract structured data with zod schemas.

MCP

fastmcp

☆ 5,475 Stars

Build and run Model Context Protocol (MCP) servers and clients with minimal Python code. FastMCP v2 supports OpenAPI/FastAPI conversion, LLM sampling, proxying, and server composition. Use decorators to expose tools, resources, and prompts. Supports Claude, Anthropic, and custom transports.

HOW TO

AI Tools

Use OpenAI CLI for Text Completion

Step 1
Install OpenAI CLI


pip install openai-cli

Step 2

Get your OpenAI API key. Go to OpenAI API Keys acreate a new key.


Step 3
Authenticate. Set your token as an environment variable:


export OPENAI_API_KEY=

Or pass it directly in the command.


Step 4

Run a prompt:


echo "Explain LLMs in simple terms" | openai complete -

Step 5

Try interactive mode:


openai repl

You can now chat with the OpenAI model from your terminal. Hit Ctrl+C to exit anytime.

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us