HKU's Multi-SPIN Boosts Edge AI Inference Goodput by 88% on Weak Devices
Multi-SPIN turns every user's phone into a draft engine, boosting edge LLM token throughput by up to 88% through smart bandwidth and draft-length co-optimization.
Takeaways
- What happened: Researchers from HKU proposed Multi-SPIN, a distributed speculative decoding system where user devices draft tokens and an edge server verifies them in batches.
- Key result: Up to 88% improvement in token goodput over heterogeneity-agnostic baselines, tested on Llama-2 and Qwen3.5 model pairs.
- Core insight: Draft length per user is the most important control variable -- getting it right matters more than just allocating more bandwidth.
- Homogeneous vs heterogeneous: Letting each device use a different draft length (heterogeneous mode) outperforms forcing all users to the same length, especially under bandwidth constraints.
- Bandwidth allocation flip: In homogeneous mode, weaker devices get more spectrum; in heterogeneous mode, devices with higher acceptance rates get rewarded instead.
- No code released: The paper is theoretical/experimental with closed-form algorithms; no open-source implementation has been published yet.
Running large language models at the edge is one of the harder unsolved problems in applied AI. The devices users carry are too weak to run a full LLM, but shipping every prompt to a remote server adds latency, cost, and privacy risk. A new paper from the University of Hong Kong proposes a clever middle ground: let the device do the cheap part, and let a nearby edge server do the expensive part, in a tightly coordinated loop.
The architecture is called Multi-SPIN (Multi-Access Speculative Inference), and it extends the well-known speculative decoding technique into a multi-user wireless setting. The result is a system that can serve multiple users simultaneously with full LLM quality, while distributing the compute burden intelligently across heterogeneous hardware.
Why edge LLM inference is so hard
The growing gap between the increasing complexity of large language models and the limited computational budgets of edge devices poses a key challenge for efficient on-device inference, despite gradual improvements in hardware capabilities. The standard playbook has been to either compress the model aggressively or offload everything to the cloud. Existing strategies, such as aggressive quantization, pruning, or remote inference, trade accuracy for efficiency or lead to substantial cost burdens.
There is a third option that has gained traction: speculative decoding. Speculative decoding accelerates LLM inference by allowing a lightweight draft model to predict multiple future tokens that are subsequently verified by a larger target model. The key insight is that the verification step can be done in a single parallel forward pass, so if the small model guesses right most of the time, you get multiple tokens for the price of one. Speculative decoding has demonstrated remarkable improvements in the generative pipeline on high-end GPUs, achieving speedups up to 6.5x compared to traditional autoregressive decoding.
But almost all prior work assumes the draft model and target model live on the same machine. Multi-SPIN asks: what if the draft model runs on the user's phone, and the target model runs on a nearby edge server?
The architecture: phones draft, server verifies
The Multi-SPIN architecture utilizes on-device small language models to generate and upload candidate token drafts, while an edge server operates the LLM to verify them in parallel batches. This is a natural split: small language models (SLMs) are cheap enough to run on a phone, and the edge server's GPU can verify drafts from many users at once by batching them together.