Skip to main content
Back

Sorbonne Finds LLMs Are Secretly Biased Against Far-Right Text Before Training Ends

A new study finds LLMs consistently struggle with far-right and nationalist political text across 10 models and 37 languages, and instruction-tuning does nothing to fix it.

Paper, Full Access
  • Core finding: LLMs are systematically more perplexed by far-right and nationalist political text than by social-democratic text, across 10 models and 37 languages.
  • New measurement approach: Uses perplexity on real political corpora instead of prompt-based surveys, making it applicable to base models and harder to game.
  • Pretraining is the culprit: Base and instruction-tuned models show highly correlated political perplexity, meaning RLHF and instruction-tuning do not fix the underlying bias.
  • Real downstream harm: Perplexity scores correlate with translation quality gaps, confirming the bias affects practical LLM tasks like multilingual translation.
  • Breadth of evidence: Three datasets, 37 languages, and 10 LLMs including Llama-3.1 and Qwen3 variants all show the same pattern.
  • Implication: Fixing political fairness requires changes to pretraining data curation, not post-training alignment techniques.

Most research on political bias in AI asks models to fill out a survey or answer leading questions, then checks which way they lean. A new paper from Sorbonne Université takes a fundamentally different approach: instead of asking what a model says, it measures what a model finds surprising. The result is an uncomfortable finding that cuts across every major model family tested.

The researchers assessed political fairness using perplexity, positing that a fair model should give equal probability to all political groups. Across ten LLMs and three datasets covering 37 languages, they found that LLMs are more perplexed by the texts of far-right and nationalist parties than of social-democratic parties.

Why Perplexity Is the Right Ruler

Perplexity (PPL) is a measure of how surprised a language model is by a piece of text. Formally, it is the exponentiated average negative log-likelihood of each token in a sequence. A low perplexity means the model finds the text natural and predictable; a high perplexity means the text feels foreign to it. The key insight here is that whether LLMs are used for summarization, translation, or plain chatting, the underlying computations rely at their core on probability distributions defined over texts, making perplexity a natural tool for assessing political fairness.

This matters because most prior approaches to measuring political bias are brittle. Following works have demonstrated the brittleness of the survey approach, as the computation of LLMs' answers is sensitive to the exact phrasing of the prompt, also yielding a lack of overall coherence. Perplexity sidesteps this entirely. You do not need to prompt the model at all. You just feed it real political text and measure how well it fits.