Skip to main content
Back

Anthropic releases open-source political-bias eval with 1,350 tests

NotebookLM uses Images as Source, Claude gets structured outputs, Google's Veo 3.1 gets improved reference, Alibaba's Qw

November 17, 2025 4 min read

Hey
Welcome to AlphaSignal, the most read source of news by AI engineers and researchers.

Every day, we identify and summarize the top 1% of news, papers, models, and repos, so you’re always up to date.

Here’s today’s roundup:
Summary

Read time: 4 min 23 sec

Top News

▸ Anthropic open-sources the evaluation method used to test Claude's political bias

WorkOS

▸ Ship secure API key infrastructure fast using WorkOS API Keys today

Signals

▸ Google lets NotebookLM read handwritten notes, screenshots, and document images
▸ Anthropic releases structured outputs so Claude always matches JSON definitions
▸ Google adds multi-image reference support to Veo 3.1 video generation
▸ Alibaba ships Qwen Code update with web search and improved IDE support
▸ Perplexity upgrades Comet with permission-based autonomous browsing

Trending Reads

▸ OpenAI’s guide to unlocking GPT-5.1’s full prompting potential
▸ Andrej Karpathy explains why verifiable work disappears first to AI
▸ Nathan Lambert on open models out-pacing closed labs faster

Coding Tip

▸ Use git worktree to run multiple branches without extra clones
Top News
Anthropic publishes open-source evalution method to measure political bias with structured scoring
1,248 Likes
claude

Anthropic introduces a new evaluation that measures political balance with numbers instead of impressions. The test covers 1,350 paired prompts across 150 topics, giving you a wide map of how a model behaves when pushed from opposite ideological angles. The scope makes this one of the most detailed public neutrality evaluations yet.

The Setup: What Anthropic Built

Anthropic designs a test that checks whether a model treats two conflicting prompts with equal depth.

  • Each topic includes one stance and its opposite.

  • The grader inspects clarity, depth, tone, and structure.

  • Scoring rules define “even-handedness,” “opposing perspectives,” and “refusals.”

  • You see how consistently a model handles friction in political framing.

The Problem: Subtle Bias Is Hard to See

Models rarely show bias through obvious statements.

  • Uneven detail reveals preference even when wording stays neutral.

  • Higher refusal rates skew outcomes by cutting off one side.

  • Missing caveats or counterpoints weaken balance.

  • You need a numeric way to inspect these patterns.

The Breakthrough: Everything Is Open Source

Anthropic releases the full pipeline so you can run it end-to-end.

  • The repo includes all prompts, grader rubrics, and scripts.

  • You run your model across the paired prompts to generate outputs.

  • You pass both outputs to the grader to score each metric.

  • The process requires no Anthropic model unless you choose to use one.

The Results: How Models Compare

Anthropic reports clear behavioral differences across the tested models.

  • Claude Opus 4.1 gives both sides of a prompt nearly identical depth and structure.

  • Claude Sonnet 4.5 maintains consistent balance but shows slightly less symmetry than Opus.

  • Gemini 2.5 Pro produces the most evenly matched responses across opposing viewpoints.

  • Grok 4 stays close behind Gemini with stable, symmetrical arguments on both sides.

  • GPT-5 shows uneven depth between viewpoints, with occasional shifts in tone or caution.

  • Llama 4 Maverick diverges the most, with shorter explanations or refusals on one side.

What You Can Do Now

You can test your own models under the same conditions.

  • Run the paired prompts.

  • Score with the grader.

  • Compare your results to Anthropic’s published baselines.

Try now
Stop Rebuilding API Key Infrastructure With WorkOS API Keys
Sponsored
Work OS

WorkOS provides infrastructure for authentication, access control, and enterprise features.

The team built WorkOS API Keys to remove the recurring work of secure key generation, rotation, revocation, and scoped permissions that appears as soon as customers ask for API access.

What you’ll get:

  • Production-ready API key infrastructure you ship in minutes

  • Drop-in widget for customer-facing key management

  • API endpoints for programmatic key creation and revocation

  • Keys tied to WorkOS Organizations and Roles for consistent permissions

  • A system that scales without ongoing maintenance overhead

Start building today
partner with us
Signals
1 Google updates NotebookLM to process photos, diagrams, and document screenshots as research sources 5,666 Likes
2 Anthropic introduces structured outputs to Claude API, ensuring every response matches defined JSON schemas 2,158 Likes
3 Google rolls out multi-image support in Veo 3.1 for more controlled visual results 2,538 Likes
4 Alibaba upgrades Qwen Code with with free web search, fuzzy matching, and cleaner responses 1,749 Likes
5 Perplexity adds transparent browsing and permission prompts to Comet for balanced autonomy and oversight 957 Likes
Trending Reads
OpenAI’s guide to unlocking GPT-5.1’s full prompting potential 1,885 Likes
It emphasises that while GPT-5.1 works out of the box, iterative prompt refinement delivers major reliability gains. You’ll learn structure, tool use, and experiment mindset to leverage the model fully.
Andrej Karpathy explains why verifiable work disappears first to AI 10,864 Likes
The essay reframes AI’s impact by focusing on verifiability. It describes how neural networks improve through practice, why structured tasks advance rapidly, and how this predicts the jagged, uneven frontier of automation.
Nathan Lambert on open models out-pacing closed labs faster 871 Likes
He highlights three insights: faster open-model releases, benchmark supremacy ahead of real-world user feedback, and the platform shift where Chinese labs challenge U.S. dominance in open models.
Coding Tip
Use git worktree to run multiple branches without extra clones
Git's worktree feature lets you check out multiple branches at once without extra clones. It helps you test model variants, debug pipelines, or run experiments in parallel without touching your main branch.

How to use it:
Run
git worktree add ../exp-new feature-branch
to create a new working directory.Repeat for any other branch you want to isolate.

Why use it?
It keeps experiments separate, avoids stash conflicts, and removes the need for repeated cloning when you switch between branches.
Know more
At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.

Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.
Looking to promote your company, product, service, or event to 250,000+ AI developers?
Work with us