Skip to main content
Back

Lance Ships v8.0.0 to Replace Parquet With 100x Faster AI Data Access

Lance v8.0.0 lands with FM-Index substring search, vectorized RaBitQ, branch-aware versioning, and a sweeping index architecture overhaul

Repo, Full Access
  • Lance v8.0.0 released as the latest stable version of the open-source multimodal AI lakehouse format, with 40+ contributors and hundreds of changes.
  • FM-Index and ngram index added for exact substring search and accelerated regex/infix LIKE queries on string columns, no full scans needed.
  • RaBitQ gets AVX-512 SIMD kernels and a new approx mode, delivering faster vector search with tunable recall-latency tradeoffs.
  • Segment-based index architecture replaces bitmap indexes, enabling independent merge and compaction without full index rebuilds (breaking change at Rust API level).
  • Branch-aware versioning extends Lance's zero-copy time travel to support git-style branch operations across directory and REST namespaces.
  • Free and open source (Apache 2.0), installable via pip install pylance; GitHub repo | docs.

Lance, the open-source lakehouse format purpose-built for multimodal AI, has shipped v8.0.0 as its latest stable release. The project has been moving fast: the tweet that seeded this story called out "Lance 6.0," and by the time you read this, the repo has already rolled past it to v8. That pace is intentional. Lance adopted semantic versioning with a community-driven release process, and major bumps come whenever the ecosystem demands breaking changes, not on a fixed calendar.

If you have not heard of Lance before, here is the one-sentence pitch: Lance is an open lakehouse format for multimodal AI. It contains a file format, table format, and catalog spec that allows you to build a complete lakehouse on top of object storage to power your AI workflows. Think of it as Parquet's successor for AI workloads, with vector search, full-text search, and zero-copy versioning baked in rather than bolted on.

The problem Parquet was never designed to solve

Existing formats like Parquet, Iceberg, and Delta Lake excel at SQL analytics, but they require additional specialized systems for AI capabilities. Lance brings these AI-first features directly into the lakehouse format. The gap shows up everywhere in the ML lifecycle: you need a vector database for similarity search, a separate object store for images and audio, another tool for feature versioning, and yet another for full-text retrieval. Lance's bet is that all of this belongs in one format.

The performance numbers back that up. Lance delivers 100x faster random access than Parquet or Iceberg without sacrificing scan performance. On analytics queries using the Oxford Pet dataset, Lance format v2.2 cuts storage by 50%+, beats Parquet on compression, and delivers up to 68x faster blob reads, while preserving the scan and random access patterns that multimodal training depends on.