Skip to main content
Back

Cactus Compute's Needle Runs AI Tool-Calling on a Phone at 6,000 Tokens/sec

Cactus Compute distilled Gemini into a 26M-parameter model that runs function calling on phones and watches at 1,200 tokens/sec, beating models 10x its size

Repo, Full Access
  • Needle is a 26M-parameter, MIT-licensed function-calling model from Cactus Compute, distilled from Gemini 3.1 Flash Lite. GitHub
  • It runs at 6,000 tokens/sec prefill and 1,200 tokens/sec decode on consumer devices, fitting in a 14 MB INT4 footprint.
  • Needle beats FunctionGemma-270M, Qwen-0.6B, Granite-350M, and LFM2.5-350M on single-shot function calling despite being 10x smaller.
  • The architecture (Simple Attention Network) removes all FFN/MLP layers entirely, keeping only attention and gating, cutting parameters by ~2/3 per layer.
  • Training used 16 TPU v6e for 27 hours of pretraining on 200B tokens, then 45 minutes of fine-tuning on 2B Gemini-generated synthetic tool-call examples.
  • Free to use commercially; weights on Hugging Face, local fine-tuning via needle finetune data.jsonl in minutes.

Most AI agents today assume a large model is somewhere in the loop for every decision, including the simplest one: which tool to call. Needle, a new open-source release from Cactus Compute, challenges that assumption directly. It is a 26-million-parameter model built for exactly one job: read a user query, look at a list of available tools, and emit a valid JSON function call. Nothing else. And it does it fast enough to run on a phone.

Smaller than anything that should work this well

Needle is a 26-million-parameter open-source model distilled from Google's Gemini 3.1 Flash Lite and trained specifically for single-shot function calling. Given a natural-language query and a list of available tools, it emits a JSON-formatted function call. Released under the MIT license, Needle quantizes to a 14 MB INT4 footprint and is designed to run AI agents entirely on phones, smartwatches, and other consumer devices, a sharp contrast to the cloud round-trips that power most tool-calling assistants today.

In production, Needle runs on Cactus at 6,000 tokens/sec prefill and 1,200 decode speed. To put that in perspective, a cloud API call to a frontier model takes 300-800ms in the best case. Needle responds in single-digit milliseconds, on-device, with no network round-trip. It beats FunctionGemma-270M, Qwen-0.6B, Granite-350M, and LFM2.5-350M on single-shot function calling for personal AI, despite being 10x smaller than the smallest of those competitors.

The architecture bet: throw out the FFN

Needle is built on what Cactus calls a Simple Attention Network (SAN), a transformer-style model with the feed-forward (MLP) layers removed entirely. The encoder-decoder design uses 12 encoder layers without FFNs and 8 decoder layers with masked self-attention plus cross-attention.

To understand why that matters, you need to know what FFN layers normally do. In a standard transformer block, the attention layer mixes information across positions in the sequence, and the feed-forward network (FFN) transforms each position individually. The FFN is where the model stores most of its learned knowledge, and it accounts for roughly two-thirds of a typical model's parameters. Cactus's argument is that for function calling, that's dead weight.

Cactus was frustrated by the little effort made towards building agentic models that run on budget phones, and their investigation led to an observation: agentic experiences are built upon tool calling, and massive models are overkill for it. Tool calling is fundamentally retrieval-and-assembly: match query to tool name, extract argument values, emit JSON. Cross-attention is the right primitive for this, and FFN parameters are wasted at this scale. The entire model is just attention and gating, no MLPs anywhere.