Skip to main content
Back

Breakthrough: Pinecone's New Serverless Vector DB

GO SERVERLESS

January 27, 2024 4 min read

Hey 

Welcome to today’s edition of AlphaSignal.


This week, we're examining Pinecone's Serverless Vector DB, a new approach to database technology optimized for Generative AI applications. It's engineered to significantly reduce operational costs, by up to 50x, while providing scalable solutions for managing and querying large, complex datasets in data-intensive environments.


Let's get into it,


Lior

Reading time: 4 min 47 sec

Serverless Vector DB

An entirely new AI “tech stack” has emerged in the past year. 


Large Language Models (LLMs) have emerged as the intelligence and orchestration layer for the stack, and vector databases are the knowledge layer. 


In today’s newsletter, we’ll share how Pinecone serverless created a breakthrough solution to the problem of knowledge in the AI era.


We have observed a few key use cases that require vector databases to upgrade for the evolving demands.

  • Labeling

  • Multi-tenant search

  • Retrieval Augmented Generation (RAG)

Features

50x Cost Reduction: Separates read/write/storage costs, optimizing resource usage for cost-effective scaling.


Instant Setup: Eliminates infrastructure management, allowing direct API interaction for streamlined data indexing and querying.


Efficient Performance: Multi-tenant architecture ensures low-latency, accurate vector searches, ideal for on-demand, large-scale ML applications.


Try Pinecone serverless now and unlock $100 in free usage credits.

Go serverless

Architecture

The Pinecone serverless has solved three critical pain points inherent in vector databases to achieve freshness, elasticity, and cost at scale.

  • Separation of storage from compute that significantly reduces costs for all types and sizes of workloads.

  • Multitenancy for powerful yet efficient indexing and retrieval performance with 100s of thousands of namespaces in an index.

  • Freshness to provide a fresh and up-to-date index to the tune of a few seconds.

Labeling 

Labeling allows Pinecone to efficiently search and label the most relevant data for model fine-tuning. For example, our customer Gong, a revenue intelligence platform, uses Pinecone to develop an innovative “active learning system” called Smart Trackers that detect and track complex concepts in conversations. 


This is an on-demand use case where queries are infrequent and aren’t sensitive to latency — several seconds is good enough since it is a human labeling activity. The critical challenge is to run cost-efficient searches over billions of vectors.


In the classic vector database search engine architecture, scatter-gather, data is sharded across many smaller individual indexes, and queries are sent to all shards. In this architecture, we don’t know what parts to page into memory until we touch all shards.


Such an architecture doesn’t make sense for on-demand queries over large datasets where only a portion of your corpus is relevant for any query. To improve efficiency, we need a new design that can effectively page portions of the index as needed from persistent, low-cost storage. That is, we need true decoupling of storage from compute for vector search.



Multitenant search

Notion, the productivity tool that allows users to create, update, and search for recommended content, represents another popular use case. The usage - data sizes and update frequencies of individual users - varies dramatically but still demands sub-100ms latencies.


Pinecone’s solution to this multi-tenancy within an index is namespaces. Customers like Notion leverage thousands of namespaces within a single index for cost-effectiveness at scale while ensuring isolation. 


To improve cost efficiencies for this workflow, we need to effectively collocate and cache namespaces so that the frequently-used-namespaces can be queried with low latency, while the rest can be efficiently paged from persistent low-cost storage on demand.


Retrieval Augmented Generation (RAG)

RAG combines LLMs alongside a vector database like Pinecone to augment the LLM with up-to-date and domain-specific knowledge to reduce hallucinations.


Our research on RAG shows that you can dramatically improve the performance of LLMs on many tasks, by leveraging a vector database that stores the embeddings generated from their billion-scale training corpus.

(The faithfulness of popular LLMs with and without RAG. Adding RAG means smaller models like Llama-2-70b can outperform larger SotA models like GPT-4.)


However, for RAG to be truly useful, vector databases must also be economical, which is only possible if we can decouple storage from compute. Without it, one would have to pay the cost of storing the entire billion scale index on some fast local storage like SSDs or RAM.


The demands of vector databases

All the above and many other use cases show that vector databases have high demands on freshness, elasticity, and cost at scale. 


Those areas are challenging. All existing vector search algorithms including graph algorithms like HNSW struggle to deal with keeping indexes fresh on dynamically evolving corpora. 


The traditional search engine architecture makes elastically scaling indexes hard because horizontal scaling requires provisioning additional shards and moving data around, which at scale can be manual, slow, and expensive.

Geometric partitioning

Geometric partitioning can be thought of as dividing the space of vectors into regions, and the centroid vectors as representatives of each partition. For dynamic production workloads, we need a mechanism to build and rebuild partitions elastically as needed. 

The index builder maintains a hierarchy of partitions starting from coarse-grained to more refined as it sees more data. This process allows the index to be incrementally maintained and stay efficient for search as data evolves.

This unlocks a critical feature of serverless: you only need to load segments that have been newly added and are relevant for a given query. 


This feature allows RAG applications to continue to serve fresh results at scale as users add, edit, and redact documents.

The freshness layer 

In parallel to the index builders, the freshness layer tails the same log and builds a compact fresh index over the most recent data. The freshness layer only cares about data that hasn’t been indexed yet, and it figures this out using the max version number of the index manifest. 


The freshness layer is what allows customers to edit, add, or redact documents and have them available for search within seconds.


Learn more

There are a lot more innovations to our serverless architecture, dive deeper with our VP of Engineering.

Try Pinecone serverless now and unlock $100 in free usage credits.

Get serverless

Not Great   Good    Amazing

Want to promote your company, product, job, or event to 150,000+ AI researchers and engineers? You can reach out here.

Thank You