Skip to main content
Back

How to Speed Up Local Inference (e.g. Qwen3.8-27B)

Ollama picks the window, flash attention, and an MTP draft before you type. Here is the one number that judges each, on a 24GB card and a Mac.

  • On a 24GB NVIDIA card, Ollama reserves a 32,768-token context window for Qwen3.8-27B, using about 2 GiB of KV cache, alongside flash attention set to auto and an MTP draft guessing 4 tokens ahead.
  • In SOTAAZ's benchmark on Qwen3-8B (80GB A100), switching the KV cache from 16-bit to 8-bit was only 6% slower at 128 tokens but dropped generation speed to 55% of baseline (44.7 vs 81.6 tok/s) once the cache held 64K tokens.
  • On a 16GB RTX 5060 Ti, enabling MTP at draft depth 2 raised Japanese prose generation from 27.3 to 37.2 tok/s, a +36% gain costing +890 MiB VRAM, while on one RTX 4090 switching from MTP to DFlash 2 boosted speed from 61.5 to 105.3 tok/s.
  • Memory bandwidth caps unaided throughput: an RTX 4090 delivers 1,008 GB/s (versus just 89.6 GB/s on an Intel i9-14900K CPU), implying a ceiling near 59 tok/s for the 17GB model without a draft.
  • On the DFlash 2 vendor's single H200 GPU test with SGLang, using 7-token draft guesses yielded 2.67x to 3.43x speedups (184.0 to 236.1 tok/s versus about 69 tok/s baseline), compared to 1.96x to 2.59x for MTP.

Qwen3.8-27B arrives in Ollama with three speed choices already made. On a 24GB NVIDIA card, it reserves a 32,768-token context window, sets flash attention to auto, and runs a helper (a draft) that guesses up to 4 tokens ahead.

Flash attention is a faster way to compute attention: it works in small tiles inside the GPU's fastest on-chip memory, which mostly shortens the wait before a reply starts.

While the draft is a small add-on (here multi-token prediction, or MTP, which ships with the model) that guesses the next few tokens, so the full 27B model checks all of them in one pass instead of writing them one at a time.

When your coding agent still feels slow, you have to tune your local model before downloading a smaller quantized version of it. The same five settings apply to other local models, while the numbers here are Qwen3.8-27B's.

A published benchmark on a smaller sibling (Qwen3-8B on an 80GB A100 GPU) shows the trap with the key-value (KV) cache, the memory of every past token. Storing it at 8-bit instead of 16-bit looked almost free on short 128-token replies (6% slower), yet with 64K tokens already in the cache, generation ran at only 55% of its 16-bit speed.

One knob, one number is the rule that avoids that trap. Tune five settings in order:

  • Context window: how many tokens of conversation Ollama reserves memory for when the model loads.
  • KV cache: the stored keys and values of every past token, kept at 16-bit or packed to 8-bit to save memory.
  • Draft: the helper that guesses tokens ahead for the big model to check in one pass.
  • Prompt reuse: keeping the processed start of a chat so the next turn skips re-reading it.
  • Server: an engine that runs several requests at once on the same weights.

Judge each change only by the one number it owns: the measurement you read right after the change, listed in the table below.

What to expect: if your window was spilling onto CPU, getting back to 100% GPU moves those layers from system RAM (89.6 GB/s on a high-end desktop CPU) back to the card's own memory (1,008 GB/s on an RTX 4090).

A draft brings the biggest reported gains on consumer cards (+36% from MTP on a 16GB RTX 5060 Ti, and 61.5 to 105.3 tok/s moving from MTP to DFlash 2, a separate helper model, on one 4090), but it can also lose. Prompt reuse shortens the wait on repeated long prompts, and an 8-bit cache mostly buys back memory (about 1 GiB at 32K on this model).

What follows: the window that fits, why an 8-bit cache rarely helps here, Ollama's draft versus llama.cpp's DFlash 2, why a vendor's 3x stays on an H200, and a run log.

Which settings change Qwen3.8-27B's speed?

Each of the five settings in the table owns one number to check, and your runtime prints two speeds:

  • Prompt speed (Ollama's prompt eval rate, pp in llama-bench) for reading your input.
  • Generation speed (eval rate, or tg) for new tokens streaming back (output).

On Ollama, the "before" in each test (your starting point) is already tuned: the three choices above (32,768 window, flash attention auto, MTP draft at 4) plus a 16-bit (f16) KV cache, with MTP as the draft.

Each section changes one setting against that starting point.

I covered the 4-bit file, engine, and Mac coding setup in How to Run Qwen 3.8 27B as a Local Coding Agent, so this piece starts after the download. For how the KV cache is built and why agents pay to rebuild it when they switch models, see Ben Dickson's KV cache techniques for faster, cheaper multi-model AI agents.

KnobIts numberOllama on a 24GB NVIDIA cardOllama on a MacRead it in
Context window100% GPU at your window32,768 tokensMost likely 4,096 on 24GB and 32,768 on 48GB (from Metal's working-set limit)ollama ps PROCESSOR, CONTEXT
KV cacheGeneration speed at depthf16No cache-type switch on the MLX build--verbose eval rate, llama-bench -d
DraftAcceptance and generation speedMTP on at 4 tokensMTP automatic, no switch--verbose eval rate, llama.cpp draft acceptance line, Mac server.log
Prompt reuseTime to first tokenFlash attention auto, prompt cache onPrompt cache on (MLX prefix cache), no flash-attention setting--verbose prompt eval cached
ServerTotal throughput1 request at a time for this model1 request at a timeAggregate tok/s at N requests

How much context fits Qwen3.8-27B on a 24GB GPU or a Mac?

On a 24GB card, your context window is whatever fits while ollama ps command still shows 100% GPU at your window.

The context window (context length) is chat memory reserved at load, and if it plus the weights exceeds VRAM, extra layers spill onto the CPU and slow every token.

This model's window is light. Per the model card, 48 of its 64 layers use Gated DeltaNet linear attention (a layer type that keeps a fixed-size running summary of the conversation instead of storing every past token), so only 16 full-attention layers (4 KV heads, head dimension 256) store a per-token cache.

The math for those 16 layers: 16 layers × 2 (key and value) × 4 heads × 256 dimensions × 2 bytes (16-bit) = 65,536 bytes, or 64 KiB per token.

Reminder: KiB = 1024 bytes. KB = 1000 bytes. Memory uses KiB / MiB / GiB (1024-based)

That puts 4,096 tokens at 256 MiB, Ollama's 32,768-token default (applied from 23 GiB in code) at 2 GiB, and the full 262,144 at 16 GiB. A