How to Speed Up Local Inference (e.g. Qwen3.8-27B)
Ollama picks the window, flash attention, and an MTP draft before you type. Here is the one number that judges each, on a 24GB card and a Mac.
Takeaways
- On a 24GB NVIDIA card, Ollama reserves a
32,768-token context window forQwen3.8-27B, using about 2 GiB of KV cache, alongside flash attention set toautoand an MTP draft guessing4tokens ahead. - In SOTAAZ's benchmark on
Qwen3-8B(80GB A100), switching the KV cache from 16-bit to 8-bit was only 6% slower at 128 tokens but dropped generation speed to 55% of baseline (44.7 vs 81.6 tok/s) once the cache held64Ktokens. - On a 16GB RTX 5060 Ti, enabling MTP at draft depth 2 raised Japanese prose generation from 27.3 to 37.2 tok/s, a
+36%gain costing+890 MiBVRAM, while on one RTX 4090 switching from MTP toDFlash 2boosted speed from 61.5 to 105.3 tok/s. - Memory bandwidth caps unaided throughput: an RTX 4090 delivers
1,008 GB/s(versus just89.6 GB/son an Intel i9-14900K CPU), implying a ceiling near 59 tok/s for the 17GB model without a draft. - On the DFlash 2 vendor's single H200 GPU test with SGLang, using 7-token draft guesses yielded 2.67x to 3.43x speedups (184.0 to 236.1 tok/s versus about 69 tok/s baseline), compared to 1.96x to 2.59x for MTP.
Qwen3.8-27B arrives in Ollama with three speed choices already made. On a 24GB NVIDIA card, it reserves a 32,768-token context window, sets flash attention to auto, and runs a helper (a draft) that guesses up to 4 tokens ahead.
Flash attention is a faster way to compute attention: it works in small tiles inside the GPU's fastest on-chip memory, which mostly shortens the wait before a reply starts.
While the draft is a small add-on (here multi-token prediction, or MTP, which ships with the model) that guesses the next few tokens, so the full 27B model checks all of them in one pass instead of writing them one at a time.
When your coding agent still feels slow, you have to tune your local model before downloading a smaller quantized version of it. The same five settings apply to other local models, while the numbers here are Qwen3.8-27B's.
A published benchmark on a smaller sibling (Qwen3-8B on an 80GB A100 GPU) shows the trap with the key-value (KV) cache, the memory of every past token. Storing it at 8-bit instead of 16-bit looked almost free on short 128-token replies (6% slower), yet with 64K tokens already in the cache, generation ran at only 55% of its 16-bit speed.
One knob, one number is the rule that avoids that trap. Tune five settings in order:
- Context window: how many tokens of conversation Ollama reserves memory for when the model loads.
- KV cache: the stored keys and values of every past token, kept at 16-bit or packed to 8-bit to save memory.
- Draft: the helper that guesses tokens ahead for the big model to check in one pass.
- Prompt reuse: keeping the processed start of a chat so the next turn skips re-reading it.
- Server: an engine that runs several requests at once on the same weights.
Judge each change only by the one number it owns: the measurement you read right after the change, listed in the table below.
What to expect: if your window was spilling onto CPU, getting back to 100% GPU moves those layers from system RAM (89.6 GB/s on a high-end desktop CPU) back to the card's own memory (1,008 GB/s on an RTX 4090).
A draft brings the biggest reported gains on consumer cards (+36% from MTP on a 16GB RTX 5060 Ti, and 61.5 to 105.3 tok/s moving from MTP to DFlash 2, a separate helper model, on one 4090), but it can also lose. Prompt reuse shortens the wait on repeated long prompts, and an 8-bit cache mostly buys back memory (about 1 GiB at 32K on this model).
What follows: the window that fits, why an 8-bit cache rarely helps here, Ollama's draft versus llama.cpp's DFlash 2, why a vendor's 3x stays on an H200, and a run log.
Which settings change Qwen3.8-27B's speed?
Each of the five settings in the table owns one number to check, and your runtime prints two speeds:
- Prompt speed (Ollama's
prompt eval rate,ppin llama-bench) for reading your input. - Generation speed (
eval rate, ortg) for new tokens streaming back (output).
On Ollama, the "before" in each test (your starting point) is already tuned: the three choices above (32,768 window, flash attention auto, MTP draft at 4) plus a 16-bit (f16) KV cache, with MTP as the draft.
Each section changes one setting against that starting point.
I covered the 4-bit file, engine, and Mac coding setup in How to Run Qwen 3.8 27B as a Local Coding Agent, so this piece starts after the download. For how the KV cache is built and why agents pay to rebuild it when they switch models, see Ben Dickson's KV cache techniques for faster, cheaper multi-model AI agents.
| Knob | Its number | Ollama on a 24GB NVIDIA card | Ollama on a Mac | Read it in |
|---|---|---|---|---|
| Context window | 100% GPU at your window | 32,768 tokens | Most likely 4,096 on 24GB and 32,768 on 48GB (from Metal's working-set limit) | ollama ps PROCESSOR, CONTEXT |
| KV cache | Generation speed at depth | f16 | No cache-type switch on the MLX build | --verbose eval rate, llama-bench -d |
| Draft | Acceptance and generation speed | MTP on at 4 tokens | MTP automatic, no switch | --verbose eval rate, llama.cpp draft acceptance line, Mac server.log |
| Prompt reuse | Time to first token | Flash attention auto, prompt cache on | Prompt cache on (MLX prefix cache), no flash-attention setting | --verbose prompt eval cached |
| Server | Total throughput | 1 request at a time for this model | 1 request at a time | Aggregate tok/s at N requests |
How much context fits Qwen3.8-27B on a 24GB GPU or a Mac?
On a 24GB card, your context window is whatever fits while ollama ps command still shows 100% GPU at your window.
The context window (context length) is chat memory reserved at load, and if it plus the weights exceeds VRAM, extra layers spill onto the CPU and slow every token.
This model's window is light. Per the model card, 48 of its 64 layers use Gated DeltaNet linear attention (a layer type that keeps a fixed-size running summary of the conversation instead of storing every past token), so only 16 full-attention layers (4 KV heads, head dimension 256) store a per-token cache.
The math for those 16 layers: 16 layers × 2 (key and value) × 4 heads × 256 dimensions × 2 bytes (16-bit) = 65,536 bytes, or 64 KiB per token.
Reminder: KiB = 1024 bytes. KB = 1000 bytes. Memory uses KiB / MiB / GiB (1024-based)
That puts 4,096 tokens at 256 MiB, Ollama's 32,768-token default (applied from 23 GiB in code) at 2 GiB, and the full 262,144 at 16 GiB. A