Skip to main content
Back

KAIST's ELMoE-3D Runs 30B AI Models 6.6x Faster on Desktop Workstations

A hardware-software co-design fuses expert caching and speculative decoding on 3D-stacked memory, cutting MoE inference latency 6.6x on desktop-class hardware.

Paper, Full Access
  • ELMoE-3D unifies expert caching and speculative decoding on 3D-stacked hybrid-bonded memory for on-premise MoE serving.
  • Achieves 6.6x speedup and 4.4x energy efficiency over xPU baseline, 2.2x over best prior accelerator.
  • Elastic Self-SD uses MSB slices of hot experts as both cache and self-draft model, no duplicated storage.
  • LSB-augmented bit-slice architecture enables bit-nested quantization with just a single added multiplexer.
  • Evaluated on Qwen3-30B, GLM-Flash, DeepSeek-V2-Lite, GPT-OSS-20B across MT-Bench, GSM8K, Alpaca, HumanEval.
  • Long-context KV cache pressure remains the main limitation; simulation-only, no released hardware or code.

Running a 30B-parameter Mixture-of-Experts model on a desktop workstation is a memory nightmare. Even though only a fraction of experts fire per token, batching a handful of requests together activates nearly every expert in the model, turning what looks like sparse compute into dense memory traffic. A team from KAIST, GIST, and Samsung has proposed ELMoE-3D, a full hardware-software stack that attacks this bottleneck by treating expert caching and speculative decoding as a single, unified design problem on top of 3D-stacked hybrid-bonded memory.

Across four MoE models and four datasets at batch sizes 1 to 16, ELMoE-3D delivers a 6.6x average speedup and 4.4x energy efficiency gain over a naive xPU baseline, and 2.2x speedup with 1.4x energy improvement over the best-performing prior accelerator.

Why MoE Breaks the Usual Playbook

MoE architectures sparsely gate feed-forward layers so each token routes to a top-k subset of experts, scaling parameter count without scaling per-token compute. The catch: because expert selection is independent across tokens, different tokens in a batch activate different experts simultaneously, so per-token computation stays sparse while memory activation becomes effectively dense at the batch level.

Two families of fixes exist, and both fall short on their own:

  • Memory-centric hardware (processing-in-memory, near-memory processing) raises bandwidth, but MoE's low arithmetic intensity leaves the internal compute underutilized, and once batch size grows, workloads get forced back onto the host through a narrow external interconnect.
  • Speculative decoding drafts several tokens with a small model and verifies them in a single forward pass. In MoE, verification has to load experts for every drafted token including the ones ultimately rejected, inflating memory traffic especially at low batch sizes.