HKU's Sandwich Doubles CPU Inference Speed by Exposing Hidden Cache Structures
A new compiler-plus-runtime approach squeezes 2x more throughput out of x86 and ARM CPUs by switching execution plans between prefill and decode.
Takeaways
- Sandwich is a full-stack CPU LLM serving system integrated with vLLM, delivering 2.01x average speedup.
- Runtime hot-switches between separate prefill and decode execution plans without duplicating model weights.
- TopoTree abstraction discovers hidden cache-sharing structures like AMD LLC slices and Kunpeng L3-Tags.
- Fast-start-then-finetune kernel compiler matches TVM quality with three orders of magnitude less tuning time.
- Evaluated on Xeon 6151/6230/8272CL, EPYC 7H12, and Kunpeng 920 with Llama 160M through 8B models.
- Sustains 90% SLO attainment at up to 3.40x tighter latency than OpenVINO and vLLM baselines. Read the paper.
Serving large language models on CPUs has always felt like a compromise. You get availability, cost, and edge deployability, but you leave a lot of performance on the table because the two halves of inference want opposite things from the hardware. A new system called Sandwich, from researchers at the University of Hong Kong, tackles that split head-on and reports an average 2.01x end-to-end speedup with up to 3.40x latency reduction across five x86 and ARM platforms.
The trick comes in three parts, wrapped into a full-stack serving engine that plugs into vLLM: a runtime that hot-switches between two different execution plans mid-request, a tree-based abstraction of CPU topology that discovers cache-sharing structures the OS never tells you about, and a tensor program generator that matches static compiler kernel quality with three orders of magnitude less tuning time.
Two phases, one machine, opposite demands
LLM inference splits into a prefill phase that is compute-bound and processes variable-length input sequences to populate the KV cache, and a decode phase that is memory-bound and generates output tokens one at a time while accessing that cache. On GPUs, recent systems disaggregate the two phases onto separate replicas. On CPUs, especially in edge or memory-constrained deployments, that option is off the table because you cannot afford to keep two copies of the weights around.
The authors quantify the mismatch by profiling BF16 Llama3.2-1B and 3B on a dual-socket Xeon Platinum 8275CL. Prefill exhibits higher IPC and lower bus cycles per thousand instructions, indicating a compute-bound workload where memory accesses are spaced apart by computation. Decode shows lower IPC with significantly higher bus and memory-controller traffic, reflecting memory-bound behavior. Existing serving stacks like the vLLM CPU backend, llama.cpp, and Intel's xFasterTransformer paper over this with hand-tuned kernels and even model partitioning across NUMA nodes, plus a light touch of core deactivation. That helps decode but throttles prefill's parallelism, and it ignores sub-NUMA sharing entirely.