Skip to main content
Back

UK Ministry of Justice's Splink Merges Messy Records Without Shared IDs

Splink is a Python library from the UK Ministry of Justice that deduplicates millions of records without unique IDs, running on a laptop.

Repo, Full Access
  • Splink is an open-source Python library for probabilistic record linkage without unique IDs, from the UK Ministry of Justice.
  • Links a million records on a laptop in a minute, or 100M+ on Spark and Athena backends.
  • Implements the Fellegi-Sunter model with EM-based unsupervised training and term frequency adjustments.
  • Runs the same code across DuckDB, Spark, Athena, and PostgreSQL via SQL compilation.
  • Used in production by NHS England, UK MoD, and the Office for National Statistics for census linkage.
  • MIT licensed, 2.2k stars, installable via pip install splink, with a full tutorial site.

Anyone who has tried to join two customer tables, patient databases, or fraud lists knows the pain: names are misspelled, dates of birth are inconsistent, and there is no shared primary key to fall back on. Splink, an open-source Python library maintained by the UK Ministry of Justice's analytical services team, tackles exactly this problem, and it has quietly become one of the most capable tools in the entity-resolution space with more than 2,100 GitHub stars and an MIT license.

The package handles probabilistic record linkage, letting you deduplicate and join datasets that lack unique identifiers. Adoption spans government, academia, and industry. Under the hood, Splink implements the Fellegi-Sunter statistical model with modern enhancements for accuracy and scalability, the same framework used by the R fastLink package and most serious probabilistic linkage tooling.

Example of duplicate records with inconsistent name, date of birth and city fields

From messy rows to a single entity ID

Given a table with fuzzy, inconsistent columns like name, date of birth, and city, Splink predicts which rows refer to the same real-world entity, then clusters those pairwise links into a single estimated identifier. The workflow has three stages:

  1. Compare record pairs across configured columns using exact, fuzzy, or custom logic.
  2. Estimate match probabilities using the Expectation-Maximization algorithm.