Shopify's CEO Built qmd to Cut AI Token Usage by 96% Locally
qmd brings production-grade hybrid search — BM25, vector embeddings, and LLM reranking — to your local markdown files, with zero cloud dependencies
Takeaways
- What it is: qmd is a fully local CLI search engine combining BM25, vector embeddings, and LLM reranking for markdown files.
- Token savings: Reduces agent token usage by up to 96% by returning only relevant snippets instead of full file contents.
- Three local models: Auto-downloads a 300MB embedder, 640MB reranker, and 1.1GB fine-tuned query expansion model — no cloud APIs required.
- MCP + SDK support: Ships with a Model Context Protocol server for Claude integration and a full Node.js/Bun SDK for embedding in your own apps.
- Smart chunking: Splits documents at markdown heading and code block boundaries; AST-aware chunking for TypeScript, Python, Go, and Rust files.
- Free and open source: MIT license, installable via
npm install -g @tobilu/qmd, requires Node.js 22+ or Bun 1.0+.
qmd is a fully local, CLI-first search engine for your personal knowledge base. Point it at a folder of markdown notes, meeting transcripts, or documentation, and it indexes everything using a three-stage pipeline: BM25 keyword search, vector semantic search, and LLM-based reranking. Everything runs on-device via GGUF models , no API keys, no cloud calls, no data leaving your machine.
The problem it actually solves
When AI agents need to find something in your notes, the naive approach is to dump files into the context window. That gets expensive fast. With a 600+ note Obsidian vault, an AI assistant reading files whole burns roughly 15,000 tokens just to answer a single question. qmd indexes your markdown locally using BM25 and vector embeddings, returning just the relevant snippets , the same query costs around 500 tokens. That's a 96% reduction in token usage, and it's the core value proposition: move retrieval out of the LLM and onto your machine.
How the pipeline works
qmd exposes three search modes with increasing quality and latency tradeoffs:
qmd search, pure BM25 full-text search, fast, no model neededqmd vsearch, vector semantic search using local embeddingsqmd query, the full hybrid pipeline: query expansion + BM25 + vector + reranking
The query command is where the interesting engineering lives.
It expands the query, runs both full-text and vector retrieval in parallel, fuses the candidate lists, and then reranks the top results with a local reranker model.
The fusion step uses Reciprocal Rank Fusion (RRF) , a technique that merges multiple ranked lists by giving each document a score based on its rank position rather than its raw score, making it robust when different backends use incompatible scoring scales.
The position-aware blending at the end is a subtle but important detail. Rather than letting the reranker override everything, qmd blends reranker scores with RRF scores based on where a document landed in the initial retrieval: