Skip to main content
Back

Google's Gemini API Drops Schema Overhead for Agents With Hundreds of Tools

Gemini's new Tool Retrieval loads function schemas on demand, letting agents scale to hundreds of tools without blowing up context windows.

  • Google launched Tool Retrieval in preview in the Gemini Interactions API.
  • Agents can scale to hundreds of tools by loading parameter schemas only on demand.
  • Enabled by adding a tool_search tool and setting defer_loading: true per function.
  • Works with Gemini 3.6 and later, including all Gemini 4 models.
  • Preserves prefix caching since deferred schemas stay out of the prompt start.
  • No extra charge, and MCP servers can be deferred wholesale.

Google has added Tool Retrieval to the Gemini API, allowing agents to discover hundreds of functions while loading parameter schemas only when needed. The approach reduces input tokens, limits prompt growth, and gives the model a smaller set of schemas to evaluate before calling a function.

Detail Availability
API Interactions API preview
Models Gemini 3.6 and later, including all Gemini 4 models
Retrieval modes Server-side, client-side, and deferred MCP loading
Billing No separate Tool Retrieval fee; standard model input and output rates apply

Function schemas create a scaling tax

When an application exposes a function to a model, it usually sends a schema describing the function’s name, arguments, types, and required fields. A catalog containing dozens or hundreds of functions can place thousands of schema tokens in every request, even when the model uses only one function.

Large catalogs increase input costs and latency while giving the model more candidates to evaluate. Similar function names and overlapping descriptions can also reduce selection accuracy as the catalog expands.

Deferred schemas keep discovery lightweight

Tool Retrieval exposes the name and description of each deferred function while keeping its parameter schema out of the initial context. After Gemini selects a function, the API loads its schema by name and emits the function call. Server-side retrieval completes that sequence within one request.

This design separates two tasks that previously shared the same prompt space: discovering the appropriate function and generating valid arguments for it. Gemini can scan the full catalog, then inspect the detailed schema for the selected function.

Server-side retrieval requires a tool_search entry and defer_loading: true on each function whose schema should load on demand. Google runs the lookup, so the application does not need to build or host an embedding index.

tools = [
    {"type": "tool_search"},
    {
        "type": "function",
        "name": "issue_refund",
        "description": "Refund an order to the original payment method.",
        "defer_loading": True,
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"}
            },
            "required": ["order_id"]
        }
    }
]

Frequently called functions can remain fully declared, while infrequently used integrations can be deferred. The response’s interaction.steps sequence records the tool_search_call, tool_search_result, and resulting function_call, giving developers a trace for diagnosing weak descriptions, incorrect matches, and argument-generation failures.

Deferred schemas preserve cached prefixes

Prefix caching reuses model computation when consecutive requests begin with the same token sequence. Deferred schemas appear later in the interaction, leaving the initial prompt stable across turns and preserving more opportunities for cache reuse.

Application-managed retrieval can change the schemas placed near the beginning of each request, which alters the cached prefix. Tool Retrieval keeps the catalog’s compact names and descriptions stable while adding selected schemas after the search step.

MCP servers support the same pattern. Setting defer_loading: true on an mcp_server tool makes the server’s functions available on demand, allowing an agent to connect to a large MCP catalog without declaring every function separately.

Applications with an existing embedding index, tenant-specific catalog, or permission layer can set "execution": "client" on tool_search. Gemini then requests a search, the application queries its registry, and the application returns the matching tools.

Client-side execution adds one application-managed round trip for each search. It also allows tools to appear during an active conversation, such as when a user connects a new integration or enables a plugin.

Per-user retrieval should return only tools the user may access. Applications still need execution-time authorization, argument validation, and audit logging for every function call.

Token savings favor large, sparse catalogs

Tool Retrieval changes token usage in three places:

  • tool_search adds a small, fixed block of input tokens.
  • Each loaded schema counts as input for the remainder of the conversation.
  • Deferred schemas that remain unused add no schema tokens.

Net savings appear when the omitted schemas outweigh the search overhead and the schemas eventually loaded. A small agent with four short functions is unlikely to benefit. An agent with five frequently used functions and fifty occasional integrations is a stronger candidate.

The Tool Retrieval docs recommend keeping common functions loaded and deferring the rest. This hybrid configuration avoids search overhead for routine calls while reducing the cost of the broader catalog.

Benchmark the full retrieval path

Tool Retrieval brings semantic search over function descriptions into Gemini’s server-side request flow. Teams using custom vector indexes can remove that component when Google’s retrieval meets their ranking, authorization, and observability requirements; client-side execution remains available for specialized registries.

A representative evaluation should compare full-schema requests with deferred retrieval across several measures:

  • Input tokens and total model cost
  • Median and tail latency
  • Tool-selection accuracy
  • Function argument validity
  • Prefix-cache hit rate
  • Client-side search frequency, when applicable

Because the feature remains in preview, production evaluations should also include model-version pinning where available, regression tests for tool selection, and monitoring for changes to response steps or retrieval behavior.