Meta's McMetric Catches Hidden Model Bias Across Subpopulations With One Number
Researchers at Meta propose a Kuiper-statistic metric weighted by signal-to-noise ratios, arguing that unweighted multi-calibration scores are mostly noise.
Takeaways
- Meta researchers propose a new scalar metric for measuring multi-calibration across subpopulations
- Built on the Kuiper statistic, avoiding binning and kernel density estimation problems of ECE and ICI
- Weights each subpopulation by its signal-to-noise ratio, then takes the maximum
- Ablation shows unweighted version is dominated by noise from tiny subpopulations
- Open-source Python package at facebookresearch/McMetric reproduces all experiments
- Isotonic regression with cross-validation emerges as a strong calibration baseline across benchmarks
Calibration is one of those quiet properties that separates a usable classifier from a misleading one. If your model says 0.8, you want it to rain on roughly 80% of those days. A new paper from Meta's Central Applied Science and FAIR teams tackles the harder version of that question: how do you put a single, trustworthy number on calibration across many subpopulations at once, and why do most existing attempts produce noise dressed up as a metric?
The authors (Ido Guy, Daniel Haimovich, Fridolin Linder, Nastaran Okati, Lorenzo Perini, Niek Tax, and Mark Tygert) propose a scalar metric for multi-calibration built on the classical Kuiper statistic, and back it up with an open-source package that reproduces every experiment in the paper.
Why global calibration lies to you
Ensuring good calibration for many specified subpopulations simultaneously has been known as "multi-calibration" since Hebert-Johnson et al. introduced the concept in 2018. Calibration measured on the full population alone can obscure signals from subgroups suspected to affect accuracy. A model can look perfectly calibrated overall while being badly miscalibrated for particular geographies, income brackets, or demographic slices.
The natural response is to compute a calibration score per subpopulation and combine them. That is where things break. The two dominant metrics in practice are the ECE (expected calibration error) and the ICI (integrated calibration index), both of which rely on binning or kernel density estimation. Lee et al. (2023) proved rigorously that an undesirable trade-off is unavoidable in methods that leverage histograms or kernel density estimation. Widen the bins and you smear away the signal. Narrow them and you drown in noise.