KAIST's ELMoE-3D Runs 30B AI Models 6.6x Faster on Desktop Workstations
A hardware-software co-design fuses expert caching and speculative decoding on 3D-stacked memory, cutting MoE inference latency 6.6x on desktop-class hardware.
Takeaways
- ELMoE-3D unifies expert caching and speculative decoding on 3D-stacked hybrid-bonded memory for on-premise MoE serving.
- Achieves 6.6x speedup and 4.4x energy efficiency over xPU baseline, 2.2x over best prior accelerator.
- Elastic Self-SD uses MSB slices of hot experts as both cache and self-draft model, no duplicated storage.
- LSB-augmented bit-slice architecture enables bit-nested quantization with just a single added multiplexer.
- Evaluated on Qwen3-30B, GLM-Flash, DeepSeek-V2-Lite, GPT-OSS-20B across MT-Bench, GSM8K, Alpaca, HumanEval.
- Long-context KV cache pressure remains the main limitation; simulation-only, no released hardware or code.
Running a 30B-parameter Mixture-of-Experts model on a desktop workstation is a memory nightmare. Even though only a fraction of experts fire per token, batching a handful of requests together activates nearly every expert in the model, turning what looks like sparse compute into dense memory traffic. A team from KAIST, GIST, and Samsung has proposed ELMoE-3D, a full hardware-software stack that attacks this bottleneck by treating expert caching and speculative decoding as a single, unified design problem on top of 3D-stacked hybrid-bonded memory.
Across four MoE models and four datasets at batch sizes 1 to 16, ELMoE-3D delivers a 6.6x average speedup and 4.4x energy efficiency gain over a naive xPU baseline, and 2.2x speedup with 1.4x energy improvement over the best-performing prior accelerator.
Why MoE Breaks the Usual Playbook
MoE architectures sparsely gate feed-forward layers so each token routes to a top-k subset of experts, scaling parameter count without scaling per-token compute. The catch: because expert selection is independent across tokens, different tokens in a batch activate different experts simultaneously, so per-token computation stays sparse while memory activation becomes effectively dense at the batch level.
Two families of fixes exist, and both fall short on their own:
- Memory-centric hardware (processing-in-memory, near-memory processing) raises bandwidth, but MoE's low arithmetic intensity leaves the internal compute underutilized, and once batch size grows, workloads get forced back onto the host through a narrow external interconnect.
- Speculative decoding drafts several tokens with a small model and verifies them in a single forward pass. In MoE, verification has to load experts for every drafted token including the ones ultimately rejected, inflating memory traffic especially at low batch sizes.