Skip to main content
Back

Polymathic AI Drops 15TB Open-Source Physics Dataset to Train Smarter AI Simulators

PolymathicAI releases 15TB of physics simulation data across 16 domains, giving neural PDE researchers a serious benchmark to work with

  • 15TB dataset: 16 physics simulation datasets spanning fluid dynamics, astrophysics, acoustics, and more, released by Polymathic AI at NeurIPS 2024.
  • Free and open: BSD-3 licensed, available via GitHub and Hugging Face, installable with pip install the_well.
  • Unified PyTorch interface: Standard DataLoader-compatible API with HuggingFace streaming or local download via CLI.
  • Built-in benchmark: Includes FNO, TFNO, U-Net, and CNextU-Net baselines with pre-trained checkpoints on HuggingFace.
  • Gaining traction: Already used as the evaluation suite for new architectures like PhysiX and KANO neural operators.
  • Limitation: Uniform grids only, fixed resolutions, and 3D datasets require significant compute and disk space (up to 5.1TB per dataset).

The Well is a 15TB open-source collection of physics simulation datasets built specifically for training and benchmarking machine learning surrogate models. Produced by Polymathic AI in collaboration with researchers from the Flatiron Institute, NYU, Princeton, Cambridge, Los Alamos National Laboratory, and eight other institutions, it was published at NeurIPS 2024 and has since become a go-to benchmark in the neural PDE space.

The gap it fills

ML-based surrogate models offer powerful tools for accelerating simulation-based workflows, but standard datasets in this space often cover small classes of physical behavior, making it difficult to evaluate the efficacy of new approaches. The Well is a direct answer to that. Before it, researchers building neural operators or PDE surrogates had to either generate their own data from scratch or work with narrow single-physics benchmarks that couldn't tell you how well a model generalizes.

Modern machine learning relies on massive, curated, and diverse datasets. Natural language processing is built on internet-scale datasets, while vision models have grown to utilize sets containing billions of text-image pairs. Physics ML has been lagging behind that curve. The Well is an attempt to close that gap.

What's inside

The Well draws from domain experts and numerical software developers to provide 15TB of data across 16 datasets covering diverse domains such as biological systems, fluid dynamics, acoustic scattering, as well as magneto-hydrodynamic simulations of extra-galactic fluids or supernova explosions. Individual datasets range from 6.9GB to 5.1TB.

The breadth is genuinely unusual. Here's a sample of what's included:

  • Fluid dynamics: Rayleigh-Bénard convection, Rayleigh-Taylor instability, shear flow, turbulent radiative layers (2D and 3D)
  • Astrophysics: Supernova explosions (64³ and 128³), post-neutron-star merger, convective envelope of red supergiant stars, magneto-hydrodynamics (MHD at 64³ and 256³)