KTransformers Runs DeepSeek V4 Flash at 1M Tokens on Two Consumer GPUs
KTransformers now runs DeepSeek V4 Flash at 1M context on 2 consumer GPUs by splitting MoE experts between GPU VRAM and system RAM
Takeaways
- 1M context on consumer hardware: DeepSeek V4 Flash runs at 1,048,576 token context on 2x RTX Pro 6000 + 256 GB RAM using KTransformers' hybrid GPU/CPU inference.
- MoE expert splitting: KTransformers pins "hot" experts on GPU VRAM and offloads the rest to system RAM, with dynamic rebalancing after prefill based on routing statistics.
- 6-12x fine-tuning speedup: The v0.6.1 SFT backend beats ZeRO-Offload by 6-12x, enabling DeepSeek-671B LoRA fine-tuning on just 2-4 RTX 4090s.
- Broad model support: Day-0 support for DeepSeek V4 Flash, Qwen3.5 (400B MoE), Kimi-K2.5, MiniMax-M2.5, and GLM-5 with BF16/FP8 precision backends.
- Simplified install: SGLang fork now ships as
sglang-kton PyPI; single./install.shreplaces a multi-step manual process. - Open source & free: Apache 2.0 license, available on GitHub with tutorials for each supported model.
KTransformers, the open-source heterogeneous inference framework from KVCache-AI, just got a lot more interesting. A community benchmark showed DeepSeek V4 Flash running at a full 1M token context window on just two RTX Pro 6000 GPUs plus 256 GB of DDR5 RAM, using KTransformers' forked SGLang engine. The trick: offloading most of the model's MoE (Mixture-of-Experts) layers to system RAM while keeping the hot ones on GPU VRAM. This is the kind of setup that was previously only practical on multi-node H100 clusters.
The Problem KTransformers Solves
Modern frontier models like DeepSeek V4 Flash are MoE architectures. MoE means the model has a large pool of "expert" sub-networks, but only activates a small subset per token. DeepSeek V4 Flash has 284B total parameters but only ~13B active per token. The catch is that all 284B parameters still need to live somewhere in memory at inference time. KTransformers offers a way to run V4-Flash using a mix of GPU and CPU resources, offloading MoE expert layers to system RAM while keeping attention layers and active experts on the GPU.
This matters because GPU VRAM is the most expensive and scarce resource in local inference. System RAM, by contrast, is cheap and plentiful. Many researchers and developers are constrained by expensive GPUs and models with tens or even hundreds of billions of parameters, making it hard to fine-tune or run very large models under limited resources.
How the Expert Splitting Actually Works
KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. Its core mechanism is expert placement: you decide how many of the model's experts live on GPU vs. CPU, and the framework handles routing tokens to the right device at runtime.
The key flags that control this are:
--kt-num-gpu-experts: how many experts to pin on GPU VRAM