Google DeepMind's MIND Metric Beats FID With 10x Fewer Samples
The torch-fidelity library now includes Monge Inception Distance, a metric that replaces 50k-sample FID with a 5k-sample sliced Wasserstein alternative.
Takeaways
- torch-fidelity added Monge Inception Distance (MIND), a new metric for evaluating generative models.
- MIND uses sliced Wasserstein distance via random projections and sorting, avoiding covariance estimation.
- Achieves FID-quality evaluation with 5k samples instead of 50k, matching correlations closely.
- Runs over 100x faster than FID and uses roughly 10x less memory during computation.
- As a proper distance, MIND resists moment-matching attacks that can artificially deflate FID scores.
- Available via
pip install torch-fidelitywith a new--mindCLI flag, from the MIND paper.
The torch-fidelity library, a widely used PyTorch toolkit for evaluating generative models, has added support for a new metric called Monge Inception Distance (MIND). It joins Inception Score, Fréchet Inception Distance, Kernel Inception Distance, Precision and Recall, and Perceptual Path Length, giving practitioners a faster and more robust alternative to FID for tracking generative model quality.
MIND was proposed by researchers at Google DeepMind to address well-known statistical and computational issues with FID. The metric leverages the sliced Wasserstein distance to compare distributions by averaging one-dimensional optimal transport distances, efficiently computed via sorting. That sidesteps the estimation of high-dimensional means and covariance matrices that underlie FID's poor sample complexity and vulnerability to adversarial attacks.
Where FID falls down
FID has been the default benchmark for image generation for years, but it carries hidden costs. It estimates a d-by-d covariance matrix on Inception-v3 embeddings, which sits at d = 2,048 dimensions, so evaluations typically need 50,000 samples just to stabilize the covariance estimate. That adds up to a lot of generation time when you want to check quality after every training epoch.